Flytekit remote updates for notebooks and versions #3109

kumare3 · 2025-02-05T05:36:20Z

Problems:

Version had to be provided when registering certain entities from the notebook
Execute worked, but register did not work in the same way
In cases when same entity is registered in non interactive mode, remote entity was auto selected to be latest registered if version field was empty. This is not explicitly called latest as a special version
Makes version handling uniform across all entities

Example cases, now works

Summary by Bito

Implements standardized version handling in Flytekit's remote operations with 'latest' as default, enhancing consistency across interactive and notebook environments. Improves documentation for execution_name_prefix parameter and clarifies behavior of 'latest' version parameter. Includes code cleanup by removing unused elements and replacing print statements with proper logging. Updates documentation for improved clarity and conciseness.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 2

Signed-off-by: Ketan Umare <[email protected]>

flyte-bot · 2025-02-05T05:36:34Z

Code Review Agent Run #7eedad

Actionable Suggestions - 1

flytekit/remote/remote.py - 1
- Consider splitting version resolution logic · Line 2055-2067

Additional Suggestions - 1

flytekit/remote/remote.py - 1
- Consider consolidating version check conditions · Line 166-167

Review Details

Files reviewed - 1 · Commit Range: 9e0af8a..9e0af8a
- flytekit/remote/remote.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

flyte-bot · 2025-02-05T05:39:42Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - Enhanced Version Handling in Remote Operations	- `remote.py` - Standardized version handling with 'latest' as default, improved version resolution logic, and added better logging

flyte-bot · 2025-02-05T05:39:44Z

flytekit/remote/remote.py

+    def _resolve_version(
+        self, version: typing.Optional[str], entity: typing.Any, ss: SerializationSettings
+    ) -> typing.Tuple[str, typing.Optional[PickledEntity]]:
+        if version is None and self.interactive_mode_enabled:
+            md5_bytes, pickled_target_dict = _get_pickled_target_dict(entity)
+            return self._version_from_hash(
+                md5_bytes, ss, entity.python_interface.default_inputs_as_kwargs, *self._get_image_names(entity)
+            ), pickled_target_dict
+        elif version is not None:
+            return version, None
+        raise ValueError(
+            "Version must be provided when not in interactive mode. If you want to use latest version pass 'latest'"
+        )


Consider splitting version resolution logic

Consider extracting the version resolution logic into a separate method for better code organization. The _resolve_version method currently handles both version resolution and pickled entity management, which could be split for better maintainability.

Code suggestion

Check the AI-generated fix before applying

Suggested change

def _resolve_version(

self, version: typing.Optional[str], entity: typing.Any, ss: SerializationSettings

) -> typing.Tuple[str, typing.Optional[PickledEntity]]:

if version is None and self.interactive_mode_enabled:

md5_bytes, pickled_target_dict = _get_pickled_target_dict(entity)

return self._version_from_hash(

md5_bytes, ss, entity.python_interface.default_inputs_as_kwargs, *self._get_image_names(entity)

), pickled_target_dict

elif version is not None:

return version, None

raise ValueError(

"Version must be provided when not in interactive mode. If you want to use latest version pass 'latest'"

)

def _get_pickled_entity(self, entity: typing.Any) -> typing.Optional[PickledEntity]:

if not self.interactive_mode_enabled:

return None

md5_bytes, pickled_target_dict = _get_pickled_target_dict(entity)

return pickled_target_dict

def _resolve_version(

self, version: typing.Optional[str], entity: typing.Any, ss: SerializationSettings

) -> typing.Tuple[str, typing.Optional[PickledEntity]]:

if version is not None:

return version, None

if not self.interactive_mode_enabled:

raise ValueError(

"Version must be provided when not in interactive mode. If you want to use latest version pass 'latest'"

)

md5_bytes, pickled_target_dict = _get_pickled_target_dict(entity)

version = self._version_from_hash(

md5_bytes, ss, entity.python_interface.default_inputs_as_kwargs, *self._get_image_names(entity)

)

return version, pickled_target_dict

Code Review Run #7eedad

Is this a valid issue, or was it incorrectly flagged by the Agent?

it was incorrectly flagged

Signed-off-by: Samhita Alla <[email protected]>

flyte-bot · 2025-02-07T00:42:12Z

Code Review Agent Run #dda473

Actionable Suggestions - 2

flytekit/remote/remote.py - 2
- Consider adding specific type hints · Line 809-821
- Consider moving version resolution earlier · Line 1078-1078

Review Details

Files reviewed - 1 · Commit Range: 9e0af8a..e020bd1
- flytekit/remote/remote.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

flyte-bot · 2025-02-07T00:45:25Z

flytekit/remote/remote.py

+    def _resolve_version(
+        self, version: typing.Optional[str], entity: typing.Any, ss: SerializationSettings
+    ) -> typing.Tuple[str, typing.Optional[PickledEntity]]:
+        if version is None and self.interactive_mode_enabled:
+            md5_bytes, pickled_target_dict = _get_pickled_target_dict(entity)
+            return self._version_from_hash(
+                md5_bytes, ss, entity.python_interface.default_inputs_as_kwargs, *self._get_image_names(entity)
+            ), pickled_target_dict
+        elif version is not None:
+            return version, None
+        raise ValueError(
+            "Version must be provided when not in interactive mode. If you want to use latest version pass 'latest'"
+        )


Consider adding specific type hints

Consider adding type hints for entity parameter in _resolve_version(). The current implementation accepts typing.Any which could lead to runtime errors if invalid types are passed.

Code suggestion

Check the AI-generated fix before applying

- def _resolve_version(self, version: typing.Optional[str], entity: typing.Any, ss: SerializationSettings + def _resolve_version(self, version: typing.Optional[str], entity: typing.Union[WorkflowBase, PythonTask], ss: SerializationSettings

Code Review Run #dda473

Consider consolidating version handling logic

Consider consolidating the version handling logic. The conditions version is not None and ss.version is not None could be combined since they have the same outcome of returning version, None or ss.version, None respectively.

Code suggestion

Check the AI-generated fix before applying

- elif version is not None: - return version, None - elif ss.version is not None: - return ss.version, None + elif version is not None or ss.version is not None: + return version or ss.version, None

Code Review Run #0c0f7f

Is this a valid issue, or was it incorrectly flagged by the Agent?

it was incorrectly flagged

flyte-bot · 2025-02-07T00:45:26Z

flytekit/remote/remote.py

                project=self.default_project,
                domain=self.default_domain,
            )

+        version, _ = self._resolve_version(version, entity, serialization_settings)


Consider moving version resolution earlier

Consider moving the version resolution logic before the serialization settings initialization to avoid potential inconsistencies between version and settings.

Code suggestion

Check the AI-generated fix before applying

- serialization_settings = SerializationSettings( - image_config=ImageConfig.auto_default_image(), - project=self.default_project, - domain=self.default_domain, - ) - - version, _ = self._resolve_version(version, entity, serialization_settings) + version, _ = self._resolve_version(version, entity, None) + serialization_settings = SerializationSettings( + image_config=ImageConfig.auto_default_image(), + project=self.default_project, + domain=self.default_domain, + version=version + )

Code Review Run #dda473

Is this a valid issue, or was it incorrectly flagged by the Agent?

it was incorrectly flagged

wild-endeavor · 2025-02-07T01:49:56Z

There's a breaking change here I wanted to highlight which may or may not be okay. I think it's technically more correct, but it is a change in behavior.

Assuming you have a simple file like this

from flytekit import task, workflow, ImageSpec, FlyteRemote, Config

image = ImageSpec(
    name="yt_public",
    builder="default",
    registry="ghcr.io/wild-endeavor",
    packages=["flytekit==v1.15.0b2"],
)

@task(container_image=image, enable_deck=True)
def print_hello():
    print("hello")

@workflow
def wf():
    print_hello()

if __name__ == "__main__":
    remote = FlyteRemote(Config.for_sandbox(), default_project="flytesnacks", default_domain="development")
    remote.register_workflow(wf, version="from_main_v4_pr")

the behavior on master is as follows

if the user does pyflyte register, fast register is used, and the code is copied into the tarball. Code is not built into the image.
if the user does python file.py, fast register is not used, and the code is built into the image.

The reason the code is built into the image in the second case is because the code currently is setting the project root. When the register workflow call triggers the downstream register task call, it gets set on the image spec, causing the user code to be built in.

With this pr, without setting project root, pyflyte register remains the same, but python file.py will no longer copy code - when the workflow is run, the task will fail.

If the user wants to make this work, they'd have to change to fast_register_workflow, which actually feels more correct.

I think we can make a better experience actually - the register_workflow function right now never does fast register... even if the user passes in a serialization settings object with fast register enabled, so there's some discrepancy here to clean up.

Signed-off-by: Kevin Su <[email protected]>

flyte-bot · 2025-02-07T19:56:31Z

Code Review Agent Run #0c0f7f

Actionable Suggestions - 1

flytekit/remote/remote.py - 1
- Consider consolidating version handling logic · Line 809-821

Review Details

Files reviewed - 1 · Commit Range: e020bd1..8368ca8
- flytekit/remote/remote.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Signed-off-by: Yee Hing Tong <[email protected]>

flyte-bot · 2025-02-11T06:10:56Z

Code Review Agent Run #b0d8cc

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 8368ca8..f9a9b8d
- flytekit/remote/remote.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Signed-off-by: Yee Hing Tong <[email protected]>

wild-endeavor · 2025-02-11T17:43:30Z

@kumare3 pushed a nit to get the monodocs build to pass. should we merge? Keep in mind there is a small breaking change (detailed above). I'm okay with this given the fact that this function is a bit broken to begin with (fast settings not being respected), and the behavior now actually might better align with expectations.

flyte-bot · 2025-02-11T18:19:56Z

Code Review Agent Run #37064e

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: f9a9b8d..394fc86
- flytekit/remote/remote.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

WIP: Fixes version handling in flytekit remote

9e0af8a

Signed-off-by: Ketan Umare <[email protected]>

kumare3 requested review from wild-endeavor, eapolinario, pingsutw, cosmicBboy, samhita-alla, thomasjpfan and Future-Outlier as code owners February 5, 2025 05:36

flyte-bot reviewed Feb 5, 2025

View reviewed changes

make version optional in register methods

e020bd1

Signed-off-by: Samhita Alla <[email protected]>

flyte-bot reviewed Feb 7, 2025

View reviewed changes

pingsutw added 3 commits February 7, 2025 10:54

fix test

42aea78

Signed-off-by: Kevin Su <[email protected]>

nit

988ff13

Signed-off-by: Kevin Su <[email protected]>

nit

8368ca8

Signed-off-by: Kevin Su <[email protected]>

pingsutw previously approved these changes Feb 7, 2025

View reviewed changes

try adding missing param doc

f9a9b8d

Signed-off-by: Yee Hing Tong <[email protected]>

wild-endeavor dismissed pingsutw’s stale review via f9a9b8d February 11, 2025 05:09

monodocsgit add -A

394fc86

Signed-off-by: Yee Hing Tong <[email protected]>

wild-endeavor approved these changes Feb 12, 2025

View reviewed changes

wild-endeavor merged commit b2dbd24 into master Feb 12, 2025
108 of 110 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flytekit remote updates for notebooks and versions #3109

Flytekit remote updates for notebooks and versions #3109

kumare3 commented Feb 5, 2025 •

edited by flyte-bot

Loading

flyte-bot commented Feb 5, 2025 •

edited

Loading

Code Review Agent Run #7eedad

flyte-bot commented Feb 5, 2025 •

edited

Loading

Changelist by Bito

flyte-bot Feb 5, 2025

flyte-bot commented Feb 7, 2025 •

edited

Loading

Code Review Agent Run #dda473

flyte-bot Feb 7, 2025 •

edited

Loading

flyte-bot Feb 7, 2025

wild-endeavor commented Feb 7, 2025

flyte-bot commented Feb 7, 2025 •

edited

Loading

Code Review Agent Run #0c0f7f

flyte-bot commented Feb 11, 2025 •

edited

Loading

Code Review Agent Run #b0d8cc

wild-endeavor commented Feb 11, 2025

flyte-bot commented Feb 11, 2025 •

edited

Loading

Code Review Agent Run #37064e

Flytekit remote updates for notebooks and versions #3109

Flytekit remote updates for notebooks and versions #3109

Conversation

kumare3 commented Feb 5, 2025 • edited by flyte-bot Loading

Summary by Bito

flyte-bot commented Feb 5, 2025 • edited Loading

Code Review Agent Run #7eedad

flyte-bot commented Feb 5, 2025 • edited Loading

Changelist by Bito

flyte-bot Feb 5, 2025

Choose a reason for hiding this comment

flyte-bot commented Feb 7, 2025 • edited Loading

Code Review Agent Run #dda473

flyte-bot Feb 7, 2025 • edited Loading

Choose a reason for hiding this comment

flyte-bot Feb 7, 2025

Choose a reason for hiding this comment

wild-endeavor commented Feb 7, 2025

flyte-bot commented Feb 7, 2025 • edited Loading

Code Review Agent Run #0c0f7f

flyte-bot commented Feb 11, 2025 • edited Loading

Code Review Agent Run #b0d8cc

wild-endeavor commented Feb 11, 2025

flyte-bot commented Feb 11, 2025 • edited Loading

Code Review Agent Run #37064e

kumare3 commented Feb 5, 2025 •

edited by flyte-bot

Loading

flyte-bot commented Feb 5, 2025 •

edited

Loading

flyte-bot commented Feb 5, 2025 •

edited

Loading

flyte-bot commented Feb 7, 2025 •

edited

Loading

flyte-bot Feb 7, 2025 •

edited

Loading

flyte-bot commented Feb 7, 2025 •

edited

Loading

flyte-bot commented Feb 11, 2025 •

edited

Loading

flyte-bot commented Feb 11, 2025 •

edited

Loading