Introducing a pythonic CLI #5

marcromeyn · 2024-08-21T13:49:42Z

No description provided.

github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

hemildesai

Can you try rebasing on the latest main? That should update some of the deleted docstrings.

docs/img/task-help.png

hemildesai · 2024-08-22T02:50:18Z

src/nemo_run/core/runners/fdl_runner.py

@@ -1,72 +1,50 @@
 # SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
-#


Why did we remove this header?

src/nemo_run/core/runners/fdl_runner.py

src/nemo_run/run/experiment.py

hemildesai · 2024-08-22T02:56:01Z

src/nemo_run/run/experiment.py

@@ -317,7 +320,13 @@ def _save_tasks(self):
        with open(os.path.join(self._exp_dir, self.__class__._TASK_FILE), "w+") as f:
            json.dump(serialized_tasks, f)

+        if "__main__" in sys.modules:


Will this work for remote executors as well?

This is called on the node that's configuring the experiment. The output is put inside the experiment dir, which I am under the impression is accessible for the remote executor?

Only the necessary files are accessible remotely, which currently just includes the configs and the code. You'll have to edit the executors to include the copying of this file.

src/nemo_run/config.py

src/nemo_run/api.py

src/nemo_run/cli/api.py

+    is_target_default: bool = False,
+    name: Optional[str] = None,
+    namespace: Optional[str] = None,
+) -> F: ...


src/nemo_run/cli/api.py

+    is_target_default: bool = False,
+    name: Optional[str] = None,
+    namespace: Optional[str] = None,
+) -> F: ...


src/nemo_run/cli/api.py

@@ -95,6 +448,20 @@
    return app


+@runtime_checkable
+class EntrypointProtocol(Protocol):
+    def cli_entrypoint(self) -> "Entrypoint": ...


src/nemo_run/cli/api.py

src/nemo_run/cli/devspace.py

@@ -61,7 +62,9 @@
 def create() -> typer.Typer:
    app = typer.Typer()

-    CLI(launch).cli(app)
+    from nemo_run.cli.api import Entrypoint


src/nemo_run/run/experiment.py

test/cli/test_api.py

+from rich.console import Console
+from typer.testing import CliRunner
+
+import nemo_run as run


src/nemo_run/cli/cli_parser.py

+# limitations under the License.
+
+import ast
+import functools


Signed-off-by: Marc Romeyn <[email protected]>

src/nemo_run/config.py

src/nemo_run/api.py

src/nemo_run/core/lark_parser.py

hemildesai · 2024-08-26T14:48:59Z

src/nemo_run/run/experiment.py

@@ -317,7 +320,13 @@ def _save_tasks(self):
        with open(os.path.join(self._exp_dir, self.__class__._TASK_FILE), "w+") as f:
            json.dump(serialized_tasks, f)

+        if "__main__" in sys.modules:


Only the necessary files are accessible remotely, which currently just includes the configs and the code. You'll have to edit the executors to include the copying of this file.

src/nemo_run/core/runners/fdl_runner.py

Signed-off-by: Marc Romeyn <[email protected]>

src/nemo_run/cli/api.py

+        for ep in entrypoints:
+            ep.load()
+    except Exception:
+        ...


Signed-off-by: Marc Romeyn <[email protected]>

github-advanced-security bot found potential problems Aug 21, 2024

View reviewed changes

hemildesai reviewed Aug 22, 2024

View reviewed changes

marcromeyn force-pushed the pythonic-cli branch from 4af05bd to c2c565d Compare August 22, 2024 11:38

github-advanced-security bot found potential problems Aug 22, 2024

View reviewed changes

src/nemo_run/cli/cli_parser.py

# limitations under the License.

import ast

import functools

Check notice

Code scanning / CodeQL

Module is imported with 'import' and 'import from' Note

Module 'functools' is imported with both 'import' and 'import from'.

marcromeyn force-pushed the pythonic-cli branch from 62de001 to fb81aeb Compare August 22, 2024 15:04

marcromeyn added 12 commits August 22, 2024 17:13

Introducing a pythonic CLI

d6b4378

Signed-off-by: Marc Romeyn <[email protected]>

Remove commented out code

2d5383a

Signed-off-by: Marc Romeyn <[email protected]>

Small fix in readme

840d22d

Signed-off-by: Marc Romeyn <[email protected]>

Adding --interactive

c593485

Signed-off-by: Marc Romeyn <[email protected]>

Starting to support plugins

5106d3b

Signed-off-by: Marc Romeyn <[email protected]>

Using RunContext to simplify things

a10cc9f

Signed-off-by: Marc Romeyn <[email protected]>

Fix some bugs + improved example

b0bfaa3

Signed-off-by: Marc Romeyn <[email protected]>

Fix typo

b1146c1

Signed-off-by: Marc Romeyn <[email protected]>

Make sure code in README is the same as task.py

02f44bb

Signed-off-by: Marc Romeyn <[email protected]>

Fixing failing tests

31555da

Signed-off-by: Marc Romeyn <[email protected]>

Fix linting errors

005af4e

Signed-off-by: Marc Romeyn <[email protected]>

Run fmt

bca6553

Signed-off-by: Marc Romeyn <[email protected]>

marcromeyn force-pushed the pythonic-cli branch from 476bcee to bca6553 Compare August 22, 2024 15:13

marcromeyn added 9 commits August 22, 2024 17:17

Fix some more issues

d64cdcf

Signed-off-by: Marc Romeyn <[email protected]>

Run formatting again

3e4d537

Signed-off-by: Marc Romeyn <[email protected]>

Fix spelling issue

c284dcc

Signed-off-by: Marc Romeyn <[email protected]>

Change API to create an experiment entrypoint

2bdfa0a

Signed-off-by: Marc Romeyn <[email protected]>

Fix docstring

6c47eaa

Signed-off-by: Marc Romeyn <[email protected]>

Adding default_factory arg to entrypoint

a50d81d

Signed-off-by: Marc Romeyn <[email protected]>

Also expose default_factory in main

2fbaab6

Signed-off-by: Marc Romeyn <[email protected]>

Adding default_executor

3a0adfe

Signed-off-by: Marc Romeyn <[email protected]>

Adding default_plugins

7be0996

Signed-off-by: Marc Romeyn <[email protected]>

hemildesai reviewed Aug 26, 2024

View reviewed changes

src/nemo_run/core/runners/fdl_runner.py Show resolved Hide resolved

Some fixes

e76d67f

Signed-off-by: Marc Romeyn <[email protected]>

github-advanced-security bot found potential problems Aug 26, 2024

View reviewed changes

src/nemo_run/cli/api.py

for ep in entrypoints:

ep.load()

except Exception:

...

Check notice

Code scanning / CodeQL

Statement has no effect Note

This statement has no effect.

marcromeyn added 4 commits August 26, 2024 18:52

Fix linting issues

a07ae58

Signed-off-by: Marc Romeyn <[email protected]>

List -> list

677e59a

Signed-off-by: Marc Romeyn <[email protected]>

Copy over __main__.py in slurm packaging

c15e98e

Signed-off-by: Marc Romeyn <[email protected]>

Copy over __main__.py in slurm packaging

aadd121

Signed-off-by: Marc Romeyn <[email protected]>

marcromeyn requested a review from hemildesai August 26, 2024 17:18

Add copyright header to fdl_runner

809a697

Signed-off-by: Marc Romeyn <[email protected]>

hemildesai approved these changes Aug 26, 2024

View reviewed changes

marcromeyn merged commit d614a8a into main Aug 26, 2024
7 checks passed

marcromeyn deleted the pythonic-cli branch August 26, 2024 17:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing a pythonic CLI #5

Introducing a pythonic CLI #5

marcromeyn commented Aug 21, 2024

github-advanced-security bot left a comment

hemildesai left a comment

hemildesai Aug 22, 2024

hemildesai Aug 22, 2024

marcromeyn Aug 22, 2024

hemildesai Aug 26, 2024

hemildesai Aug 26, 2024

Introducing a pythonic CLI #5

Introducing a pythonic CLI #5

Conversation

marcromeyn commented Aug 21, 2024

github-advanced-security bot left a comment

Choose a reason for hiding this comment

hemildesai left a comment

Choose a reason for hiding this comment

hemildesai Aug 22, 2024

Choose a reason for hiding this comment

hemildesai Aug 22, 2024

Choose a reason for hiding this comment

marcromeyn Aug 22, 2024

Choose a reason for hiding this comment

hemildesai Aug 26, 2024

Choose a reason for hiding this comment

hemildesai Aug 26, 2024

Choose a reason for hiding this comment