Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/remove session store #2219

Merged
Merged
8 changes: 0 additions & 8 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,14 +210,6 @@ make run PROJECT_PATH=<path-to-your-test-project>/new-kedro-project

> **Note**: Once the backend development server is launched at port 4142, the local app will always pull data from that server. To prevent this, you can comment out the proxy setting in `package.json` and restart the dev server at port 4141.

#### Launch the development server with the `SQLiteSessionStore`

Kedro-Viz provides a `SQLiteSessionStore` that users can use in their project to enable experiment tracking functionality. If you want to use this session store with the development server, make sure you don't use a relative path when specifying the store's location in `settings.py`. For example, `demo-project` specifies the local `data` directory within a project as the session store's location as follows:

```python
from kedro_viz.integrations.kedro.sqlite_store import SQLiteStore
SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data")}
```
Copy link
Contributor

@ravi-kumar-pilla ravi-kumar-pilla Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to make changes to all the starter templates (may be a note saying kedro-viz v11 does not support ET) as a follow up - https://github.com/kedro-org/kedro-starters/blob/main/spaceflights-pandas-viz/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/%7B%7B%20cookiecutter.python_package%20%7D%7D/settings.py

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey, @merelcht has already raised this concern, and I’ll create a ticket for it. For now, we’ve added a deprecation warning for Kedro datasets to alert users about this change coming in the next release.


Owing to this coupling between the project settings and Kedro-Viz, if you wish to execute any Kedro commands on `demo-project` (including `kedro run`), you will need to install the Kedro-Viz Python package. To install your local development version of the package, run:

Expand Down
19 changes: 0 additions & 19 deletions demo-project/src/demo_project/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,6 @@
# List the installed plugins for which to disable auto-registry
# DISABLE_HOOKS_FOR_PLUGINS = ("kedro-viz",)

from pathlib import Path

# Define where to store data from a KedroSession. Defaults to BaseSessionStore.
# from kedro.framework.session.store import ShelveStore
from kedro_viz.integrations.kedro.sqlite_store import SQLiteStore

SESSION_STORE_CLASS = SQLiteStore
SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data")}

# Setup for collaborative experiment tracking.
# SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data"),
# "remote_path": "s3://{path-to-session_store}" }

# Define custom context class. Defaults to `KedroContext`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can keep the code from line 18-23 as is, as this is not related to ET

# CONTEXT_CLASS = KedroContext

# Define the configuration folder. Defaults to `conf`
# CONF_ROOT = "conf"

from kedro.config import OmegaConfigLoader # NOQA

CONFIG_LOADER_CLASS = OmegaConfigLoader
360 changes: 0 additions & 360 deletions docs/source/experiment_tracking.md

This file was deleted.

4 changes: 1 addition & 3 deletions docs/source/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

</p>

Kedro-Viz is an interactive development tool for visualising data science pipelines built with [Kedro](https://github.com/kedro-org/kedro). Kedro-Viz also enables users to view and compare different experiment runs within their Kedro project.
Kedro-Viz is an interactive development tool for visualising data science pipelines built with [Kedro](https://github.com/kedro-org/kedro).

Kedro-Viz features include:

Expand All @@ -18,7 +18,6 @@ Kedro-Viz features include:
🎨 Rich metadata side panel to display parameters, plots, etc.
📊 Support for all types of [Plotly charts](https://plotly.com/javascript/).
♻️ Autoreload on code change.
🧪 Support for experiment tracking and comparing runs in a Kedro project.

Take a look at the <a href="https://demo.kedro.org/" target="_blank" rel="noopener noreferrer">live demo</a> for a preview of Kedro-Viz.

Expand All @@ -30,7 +29,6 @@ kedro-viz_visualisation
share_kedro_viz
preview_datasets
slice_a_pipeline
experiment_tracking
```

```{toctree}
Expand Down
1 change: 0 additions & 1 deletion docs/source/kedro-viz_visualisation.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,6 @@ Some of the known limitations while using `--lite` flag:

* If the datasets are not resolved, they will be defaulted to a custom dataset `UnavailableDataset`.
* The flowchart will not show the layers information for the datasets.
* Experiment Tracking will not work if the pre-requisite of having kedro-datasets version 2.1.0 and above is not met.

## Automatic visualisation updates

Expand Down
27 changes: 0 additions & 27 deletions package/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,33 +205,6 @@ Options:
-h, --help Show this message and exit.
```

### Experiment Tracking usage

To enable [experiment tracking](https://docs.kedro.org/en/stable/experiment_tracking/index.html) in Kedro-Viz, you need to add the Kedro-Viz `SQLiteStore` to your Kedro project.

This can be done by adding the below code to `settings.py` in the `src` folder of your Kedro project.

```python
from kedro_viz.integrations.kedro.sqlite_store import SQLiteStore
from pathlib import Path
SESSION_STORE_CLASS = SQLiteStore
SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data")}
```

Once the above set-up is complete, tracking datasets can be used to track relevant data for Kedro runs. More information on how to use tracking datasets can be found in the [experiment tracking documentation](https://docs.kedro.org/en/stable/experiment_tracking/index.html)

**Notes:**

- Experiment Tracking is only available for Kedro-Viz >= 4.0.2 and Kedro >= 0.17.5
- Prior to Kedro 0.17.6, when using tracking datasets, you will have to explicitly mark the datasets as `versioned` for it to show up properly in Kedro-Viz experiment tracking tab. From Kedro >= 0.17.6, this is done automatically:

```yaml
train_evaluation.r2_score_linear_regression:
type: tracking.MetricsDataset
filepath: ${base_location}/09_tracking/linear_score.json
versioned: true
```

### Standalone React component usage

To use Kedro-Viz as a standalone React component, you can follow the example below. However, please note that Kedro-Viz does not support server-side rendering (SSR). If you're using Next.js or another SSR framework, you should be aware of this limitation.
Expand Down
18 changes: 0 additions & 18 deletions package/kedro_viz/database.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,21 +21,3 @@ def configure_wal_for_azure(engine):
if is_azure_ml:
with engine.connect() as conn:
conn.execute(text("PRAGMA journal_mode=WAL;"))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is for handling SQLite which is not needed anymore, so I guess we can delete the file entirely


def make_db_session_factory(session_store_location: str) -> sessionmaker:
"""SQLAlchemy connection to a SQLite DB"""
database_url = f"sqlite:///{session_store_location}"
engine = create_engine(database_url, connect_args={"check_same_thread": False})
# TODO: making db session factory shouldn't depend on models.
# So want to move the table creation elsewhere ideally.
# But this means returning engine as well as session class.

# Check if we are running in an Azure ML environment if so enable WAL mode.
configure_wal_for_azure(engine)

# Create the database tables if they do not exist.
Base.metadata.create_all(bind=engine)

# Return a session factory bound to the engine.
return sessionmaker(bind=engine)
12 changes: 4 additions & 8 deletions package/kedro_viz/integrations/kedro/data_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
from kedro import __version__
from kedro.framework.project import configure_project, pipelines
from kedro.framework.session import KedroSession
from kedro.framework.session.store import BaseSessionStore
from kedro.framework.startup import bootstrap_project
from kedro.io import DataCatalog
from kedro.pipeline import Pipeline
Expand Down Expand Up @@ -73,8 +72,7 @@ def _load_data_helper(
configuration.
is_lite: A flag to run Kedro-Viz in lite mode.
Returns:
A tuple containing the data catalog, pipeline dictionary, session store
and dataset stats dictionary.
A tuple containing the data catalog, pipeline dictionary and dataset stats dictionary.
"""

with KedroSession.create(
Expand All @@ -88,7 +86,6 @@ def _load_data_helper(
session._hook_manager = _VizNullPluginManager() # type: ignore

context = session.load_context()
session_store = session._store

# patch the AbstractDataset class for a custom
# implementation to handle kedro.io.core.DatasetError
Expand All @@ -110,7 +107,7 @@ def _load_data_helper(
# Useful for users who have `get_current_session` in their `register_pipelines()`.
pipelines_dict = dict(pipelines)
stats_dict = _get_dataset_stats(project_path)
return catalog, pipelines_dict, session_store, stats_dict
return catalog, pipelines_dict, stats_dict


def load_data(
Expand All @@ -120,7 +117,7 @@ def load_data(
package_name: Optional[str] = None,
extra_params: Optional[Dict[str, Any]] = None,
is_lite: bool = False,
) -> Tuple[DataCatalog, Dict[str, Pipeline], BaseSessionStore, Dict]:
) -> Tuple[DataCatalog, Dict[str, Pipeline], Dict]:
"""Load data from a Kedro project.
Args:
project_path: the path where the Kedro project is located.
Expand All @@ -134,8 +131,7 @@ def load_data(
configuration.
is_lite: A flag to run Kedro-Viz in lite mode.
Returns:
A tuple containing the data catalog, pipeline dictionary, session store
and dataset stats dictionary.
A tuple containing the data catalog, pipeline dictionary,and dataset stats dictionary.
"""
if package_name:
configure_project(package_name)
Expand Down
201 changes: 0 additions & 201 deletions package/kedro_viz/integrations/kedro/sqlite_store.py

This file was deleted.

Loading
Loading