Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move kedro-catalog JSON schema to kedro-datasets #4359

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
1 change: 1 addition & 0 deletions .github/styles/Kedro/ignore-names.txt
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ Puneet
Rashida
Ravi
Richard
Schopp
Schwarzmann
Sorokin
Stichbury
Expand Down
1 change: 1 addition & 0 deletions .github/styles/Kedro/ignore.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
Kedro
Kedro's
Kedroids
Kubeflow
Databricks
Conda
Expand Down
3 changes: 3 additions & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,13 @@
* Added `node` import to the pipeline template.
* Update error message when executing kedro run without pipeline.
* Safeguard hooks when user incorrectly registers a hook class in settings.py.
* Moved `kedro-catalog` JSON schema to `kedro-datasets`.

## Breaking changes to the API
## Documentation changes
## Community contributions
Many thanks to the following Kedroids for contributing PRs to this release:

Check warning on line 19 in RELEASE.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Kedro.weaselwords] 'Many' is a weasel word! Raw Output: {"message": "[Kedro.weaselwords] 'Many' is a weasel word!", "location": {"path": "RELEASE.md", "range": {"start": {"line": 19, "column": 1}}}, "severity": "WARNING"}
* [Chris Schopp](https://github.com/chrisschopp)

# Release 0.19.10

Expand Down
2 changes: 1 addition & 1 deletion docs/source/data/how_to_create_a_custom_dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -603,6 +603,6 @@ kedro-plugins/kedro-datasets/kedro_datasets/image
There are two special considerations when contributing a dataset:

1. Add the dataset to `kedro_datasets.rst` so it shows up in the API documentation.
2. Add the dataset to `static/jsonschema/kedro-catalog-X.json` for IDE validation.
2. Add the dataset to `kedro-plugins/kedro-datasets/static/jsonschema/kedro-catalog-X.json` for IDE validation.

```
4 changes: 2 additions & 2 deletions docs/source/development/set_up_pycharm.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,10 +163,10 @@ You can enable the Kedro catalog validation schema in your PyCharm IDE to enable

![](../meta/images/pycharm_edit_schema_mapping.png)

Add a new mapping using the "+" button in the top left of the window and select the name you want for it. Enter this URL `https://raw.githubusercontent.com/kedro-org/kedro/develop/static/jsonschema/kedro-catalog-0.19.json` in the "Schema URL" field and select "JSON Schema Version 7" in the "Schema version" field.
Add a new mapping using the "+" button in the top left of the window and select the name you want for it. Enter this URL `https://raw.githubusercontent.com/kedro-org/kedro-plugins/main/kedro-datasets/static/jsonschema/kedro-catalog-0.19.json` in the "Schema URL" field and select "JSON Schema Version 7" in the "Schema version" field.
Copy link
Author

@chrisschopp chrisschopp Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming this should point to main.


Add the following file path pattern to the mapping: `conf/**/*catalog*`.

![](../meta/images/pycharm_catalog_schema_mapping.png)

> [Different schemas for different Kedro versions can be found in the Kedro repository](https://github.com/kedro-org/kedro/tree/main/static/jsonschema).
> [Different schemas for different Kedro versions can be found in the `kedro-datasets` repository](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-datasets/static/jsonschema).
4 changes: 2 additions & 2 deletions docs/source/development/set_up_vscode.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,11 +260,11 @@ Enter the following in your `settings.json` file:
```json
{
"yaml.schemas": {
"https://raw.githubusercontent.com/kedro-org/kedro/develop/static/jsonschema/kedro-catalog-0.19.json": "conf/**/*catalog*"
"https://raw.githubusercontent.com/kedro-org/kedro-plugins/main/kedro-datasets/static/jsonschema/kedro-catalog-0.19.json": "conf/**/*catalog*"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming this should point to main.

}
}
```

and start editing your `catalog` files.

> [Different schemas for different Kedro versions can be found in the Kedro repository](https://github.com/kedro-org/kedro/tree/main/static/jsonschema).
> [Different schemas for different Kedro versions can be found in the Kedro repository](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-datasets/static/jsonschema).
102 changes: 102 additions & 0 deletions static/img/kedro-catalog-0.19.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
{
"type": "object",
"patternProperties": {
"^[a-z0-9-_]+$": {
"required": [
"type"
],
"properties": {
"type": {
"type": "string",
"enum": [
"CachedDataset",
"MemoryDataset",
"LambdaDataset"
]
}
},
"allOf": [
{
"if": {
"properties": {
"type": {
"const": "CachedDataset"
}
}
},
"then": {
"required": [
"dataset"
],
"properties": {
"dataset": {
"pattern": ".*",
"description": "A Kedro Dataset object or a dictionary to cache."
},
"copy_mode": {
"type": "string",
"description": "The copy mode used to copy the data. Possible\nvalues are: \"deepcopy\", \"copy\" and \"assign\". If not\nprovided, it is inferred based on the data type."
}
}
}
},
{
"if": {
"properties": {
"type": {
"const": "MemoryDataset"
}
}
},
"then": {
"required": [],
"properties": {
"data": {
"pattern": ".*",
"description": "Python object containing the data."
},
"copy_mode": {
"type": "string",
"description": "The copy mode used to copy the data. Possible\nvalues are: \"deepcopy\", \"copy\" and \"assign\". If not\nprovided, it is inferred based on the data type."
}
}
}
},
{
"if": {
"properties": {
"type": {
"const": "LambdaDataset"
}
}
},
"then": {
"required": [
"load",
"save"
],
"properties": {
"load": {
"pattern": ".*",
"description": "Method to load data from a data set."
},
"save": {
"pattern": ".*",
"description": "Method to save data to a data set."
},
"exists": {
"pattern": ".*",
"description": "Method to check whether output data already exists."
},
"release": {
"pattern": ".*",
"description": "Method to release any cached information."
}
}
}
}
]
}
}
}

Loading
Loading