Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update mermaid entrypoint diagrams with current state #4339

Merged
merged 8 commits into from
Nov 25, 2024
Merged

Update mermaid entrypoint diagrams with current state #4339

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
dd53319
Update mermaid entrypoint diagrams with current state
merelcht Nov 19, 2024
9efaf70
Update diagram kedro with project
merelcht Nov 19, 2024
3f713bc
Update diagram kedro run
merelcht Nov 19, 2024
54aa08d
Update diagram kedro-plugin
merelcht Nov 19, 2024
f6287e4
Update kedro-ipython diagram
merelcht Nov 19, 2024
602bc26
Merge branch 'main' into update-diagrams
merelcht Nov 19, 2024
3b68b9c
Remove unused component
merelcht Nov 22, 2024
41ec71d
Update docs/diagrams/installed-kedro-project.md
merelcht Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/diagrams/installed-kedro-project.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
```mermaid
sequenceDiagram
title "Installed Kedro project"
title Installed Kedro project

participant script as "third-party Python script"
participant curr_dir as "Directory with Kedro conf/ in it"
participant session as "KedroSession"
participant script as third-party Python script
merelcht marked this conversation as resolved.
Show resolved Hide resolved
participant curr_dir as Directory with Kedro conf/ in it
participant session as KedroSession

script->>script: run third-party script
script->>curr_dir: get path to the project config
script->>session: create a session with Kedro project name and project config dir
session->>run: run a pipeline and/or nodes
script->>session: create a session with Kedro project config dir
session->>session: run a pipeline and/or nodes
```
28 changes: 12 additions & 16 deletions docs/diagrams/kedro-ipython.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,30 +1,26 @@
```mermaid
sequenceDiagram
title "$ kedro ipython"
title $ kedro ipython

participant cli as "$ kedro ipython"
participant env as "Environment variables"
participant ipython as "IPython"
participant entrypoint as "00-kedro-init.py\nreload_kedro"
participant hook_manager as "Hook manager"
participant project as "Kedro project directory"
participant session as "KedroSession"
participant context as "KedroContext"
participant cli as $ kedro ipython
participant env as Environment variables
participant ipython as IPython
participant entrypoint as ipython/__init__.py <br> reload_kedro
participant hook_manager as Hook manager
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like hook manager is not connected to anything in this diagram and can be removed?

participant project as Kedro project directory
participant session as KedroSession
participant context as KedroContext

cli->>cli: Check if IPython is importable
cli->>env: Set IPYTHONDIR to metadata.project_path / ".ipython"
cli->>env: Set KEDRO_ENV to the chosen Kedro environment
cli->>cli: Print an info message
cli->>ipython: Start ipython
ipython->>entrypoint: load startup script
entrypoint->>entrypoint: import Kedro
entrypoint->>hook_manager: clear the hook manager
ipython->>entrypoint: load ipython extension
entrypoint->>project: find Kedro project
entrypoint->>project: bootstrap the project
entrypoint->>entrypoint: remove imported project package modules
entrypoint->>session: create a KedroSession
entrypoint->>session: activate the session
entrypoint->>session: load KedroContext
entrypoint->>context: get the data catalog
entrypoint->>entrypoint: expose session, context and catalog variables
entrypoint->>entrypoint: expose session, context, catalog and pipelines variables
entrypoint->>entrypoint: register reload_kedro line magic
```
16 changes: 8 additions & 8 deletions docs/diagrams/kedro-no-project.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
```mermaid
sequenceDiagram
title "$ kedro\ndirectory without Kedro project"
title $ kedro in directory without Kedro project

participant kedro as "$ kedro"
participant entrypoint as "setup.py\nkedro = kedro.framework.cli:main"
participant init_plugins as "Kedro Plugins\nentry_point = kedro.init"
participant kedro_cli as "Kedro CLI\nglobal commands\ninfo, new, docs, starter"
participant global_plugins as "Kedro Plugins\nentry_point = kedro.global_commands"
participant pyproject.toml as "Current directory\npyproject.toml"
participant click as "Click"
participant kedro as $ kedro
participant entrypoint as pyproject.toml <br> kedro = "kedro.framework.cli:main"
participant init_plugins as Kedro Plugins <br> [project.entry-points."kedro.init"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a valid entrypoint? I was trying to find info on it but I couldn't.

Copy link
Member Author

@merelcht merelcht Nov 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took me a while to figure this out as well. It's part of the entrypoint groups:

ENTRY_POINT_GROUPS = {
    "global": "kedro.global_commands",
    "project": "kedro.project_commands",
    "init": "kedro.init",
    "line_magic": "kedro.line_magic",
    "hooks": "kedro.hooks",
    "cli_hooks": "kedro.cli_hooks",
    "starters": "kedro.starters",
}

It's used to initialise plugins: https://github.com/kedro-org/kedro/blob/main/kedro/framework/cli/cli.py#L121 which is called when kedro.framework.cli:main is used as entrypoint.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh TIL! Thanks!

participant kedro_cli as Kedro CLI <br> global commands <br> info, new, docs, starter
participant global_plugins as Kedro Plugins <br> [project.entry-points."kedro.global_commands"]
participant pyproject.toml as Current directory <br> pyproject.toml
participant click as Click

kedro->>entrypoint: Python calls this

Expand Down
14 changes: 7 additions & 7 deletions docs/diagrams/kedro-plugin.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
```mermaid
sequenceDiagram
title "$ kedro plugin"
title $ kedro plugin

participant cli as "$ kedro plugin"
participant prelude as "See kedro-with-project.puml for details"
participant project_plugin as "Kedro Plugin\nentry_point = kedro.project_commands"
participant click as "Click context"
participant session as "KedroSession"
participant cli as $ kedro plugin
participant prelude as See kedro-with-project.md for details
participant project_plugins as Kedro Plugins <br> [project.entry-points."kedro.project_commands"]
participant session as KedroSession
participant kedro_cli as KedroCLI

cli->>prelude: prepare click commands as prelude to this
prelude->>project_plugin: execute plugin click command
project_plugin->>click: get ProjectMetadata from the click context
project_plugin->>kedro_cli: get ProjectMetadata from the KedroCLI command collection group
project_plugin->>project_plugin: plugin code
project_plugin->>session: need to create KedroSession for all runtime config and info
```
22 changes: 11 additions & 11 deletions docs/diagrams/kedro-run.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
```mermaid
sequenceDiagram
title "$ kedro run"
title $ kedro run

participant cli as "$ kedro run"
participant prelude as "See kedro-with-project.puml for details"
participant project_cli as "Project directory\ncli.py"
participant session as "KedroSession"
participant context as "KedroContext"
participant runner as "Runner"
participant hooks as "Hook manager"
participant cli as $ kedro run
participant prelude as See kedro-with-project.md for details
participant project_cli as Project directory <br> cli.py
participant session as KedroSession
participant context as KedroContext
participant runner as Runner
participant hooks as Hook manager

cli->>prelude: prepare click commands as prelude to this
prelude->>project_cli: run
project_cli->>session: create KedroSession
session->>hooks: initialise hook manager
session->>session: run
session->>session: load KedroContext
session->>context: get the selected pipeline
context->>context: filter the pipeline based on command line arguments
session->>session: get the selected pipeline
session->>session: filter the pipeline based on command line arguments
session->>context: get catalog with load version / save version
session->>runner: create runner
session->>hooks: get hook manager
hooks->>hooks: before_pipeline_run
runner->>runner: run the filtered pipeline with the catalog
hooks->>hooks: on_pipeline_error (if runner fails)
Expand Down
21 changes: 11 additions & 10 deletions docs/diagrams/kedro-with-project.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
```mermaid
sequenceDiagram
title "$ kedro\ndirectory without Kedro project"
title $ kedro in directory with Kedro project

participant kedro as "$ kedro"
participant entrypoint as "setup.py\nkedro = kedro.framework.cli:main"
participant init_plugins as "Kedro Plugins\nentry_point = kedro.init"
participant kedro_cli as "Kedro CLI\nglobal commands\ninfo, new, docs, starter"
participant global_plugins as "Kedro Plugins\nentry_point = kedro.global_commands"
participant pyproject.toml as "Current directory\npyproject.toml"
participant project_plugins as "Kedro Plugins\nentry_point = kedro.project_commands"
participant kedro_project as "Current directory\nKedro Project: cli.py"
participant kedro as $ kedro
participant entrypoint as pyproject.toml <br> kedro = "kedro.framework.cli:main"
participant init_plugins as Kedro Plugins <br> [project.entry-points."kedro.init"]
participant kedro_cli as Kedro CLI <br> global commands <br> info, new, docs, starter
participant global_plugins as Kedro Plugins <br> [project.entry-points."kedro.global_commands"]
participant pyproject.toml as Current directory <br> pyproject.toml
participant project_plugins as Kedro Plugins <br> [project.entry-points."kedro.project_commands"]
participant kedro_project as "Current directory <br> Kedro Project: cli.py"
participant click as "Click"

kedro->>entrypoint: Python calls this
entrypoint->>init_plugins: load and run all installed

entrypoint->>init_plugins: load and run all installed plugins
entrypoint->>kedro_cli: collect built-in commands
entrypoint->>global_plugins: load and collect global plugin commands
entrypoint->>pyproject.toml: check current dir for a Kedro project
Expand Down
12 changes: 7 additions & 5 deletions docs/diagrams/python-m-project.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
```mermaid
sequenceDiagram
title "$ <project>\n$ python -m <project>.run"
title python -m {project}.run

participant cli as "$ <project>\n$ python -m <project>"
participant entrypoint as "setup.py\n<project> = <project>.__main__"
participant session as "KedroSession"
participant cli as $ <project> <br> $ python -m <project>
participant entrypoint as pyproject.toml <br> <project> = <project>.__main__:main
participant find_run as find_run_command()
participant session as KedroSession

cli->>entrypoint: Python calls the entrypoint
entrypoint->>session: create session
entrypoint->>find_run: Find run command
find_run->>session: create session (for default run)
session->>session: run
```