From e830b19046942426903cb7b0f521cfee56669a60 Mon Sep 17 00:00:00 2001 From: Huong Nguyen <32060364+Huongg@users.noreply.github.com> Date: Thu, 18 Apr 2024 18:55:08 +0100 Subject: [PATCH 01/22] Fix/gitpod (#1862) * install python 3.9 Signed-off-by: huongg * update python to 3.9 Signed-off-by: huongg * remove --force Signed-off-by: huongg * install python 3.9 global Signed-off-by: huongg * install python 3.9 in BE Signed-off-by: huongg * added the necessary environment setup commands (export PATH, eval) to ensure pyenv is available in the environment. Signed-off-by: huongg * set up a custom dockerfile to install python Signed-off-by: huongg * fix typo Signed-off-by: huongg * use custom dockerfile Signed-off-by: huongg * test dockerfile Signed-off-by: ravi-kumar-pilla * fix build issue gitpod Signed-off-by: ravi-kumar-pilla * remove docker file Signed-off-by: huongg * update gitpod image Signed-off-by: huongg * update release note Signed-off-by: huongg * update python note to 3.12 Signed-off-by: huongg * Update to a different image Co-authored-by: Ravi Kumar Pilla Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * Update CONTRIBUTING.md with steps to use gitpod Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * update python version in release note Signed-off-by: huongg * update contributing guidline Signed-off-by: huongg * include note Co-authored-by: Nok Lam Chan Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: rashidakanchwala <37628668+rashidakanchwala@users.noreply.github.com> Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * update contributing note Co-authored-by: Ravi Kumar Pilla Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * update copy for contributing Signed-off-by: huongg --------- Signed-off-by: huongg Signed-off-by: ravi-kumar-pilla Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> Co-authored-by: ravi-kumar-pilla Co-authored-by: Nok Lam Chan Co-authored-by: rashidakanchwala <37628668+rashidakanchwala@users.noreply.github.com> --- .gitpod.yml | 6 +++--- CONTRIBUTING.md | 17 +++++++++++++++++ RELEASE.md | 7 +++++++ 3 files changed, 27 insertions(+), 3 deletions(-) diff --git a/.gitpod.yml b/.gitpod.yml index c9ffc8cfbc..c7986d0e48 100644 --- a/.gitpod.yml +++ b/.gitpod.yml @@ -1,5 +1,5 @@ -image: gitpod/workspace-full:2023-01-16-03-31-28 - +image: gitpod/workspace-full:2023-09-29-11-03-42 # image with Python 3.11 and Node 18 + tasks: - name: frontend init: | @@ -14,7 +14,7 @@ tasks: - name: backend before: echo PIP_USER=no >> ~/.bashrc && export PIP_USER=no init: | - pip install "pip==23.0.1" --user + pip install "pip==23.3.1" --user pip install -r package/test_requirements.txt -r demo-project/src/docker_requirements.txt --user # Install latest kedro and all its dependencies. pip install https://github.com/kedro-org/kedro/archive/main.zip --user diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 08673a0f62..89c575d110 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -318,6 +318,23 @@ Python dependencies in Kedro-Viz are usually updated automatically through tools If the lower-bound e2e test fails, it indicates that some dependencies may not work correctly with Kedro-Viz. To resolve this, update the problematic dependency in both `requirements.txt` and `lower-requirements.txt`. +### Testing UI using Gitpod +**Please ensure you have installed Gitpod on your browser extensions - [Gitpod browser extension](https://www.gitpod.io/docs/configure/user-settings/browser-extension)** + +**1. Once you open a pull request, click on the `Open` icon on the right** +Screenshot 2024-04-17 at 15 54 36 + + +**2. It will open the gitpod workspace automatically** +Screenshot 2024-04-17 at 15 57 41 + +**3. Please wait until the frontend and backend builds are successful. You can check it by clicking on `frontend` and `backend` tabs in the terminal. (Estimated build time : 5 minutes)** +Screenshot 2024-04-17 at 15 59 46 +Screenshot 2024-04-17 at 16 04 38 + +**4. Unblock pop-up ads and refresh your browser. Kedro-Viz should be running on your gitpod workspace** +Screenshot 2024-04-17 at 16 06 05 + # Release guidelines - Practice frequent, staggered releases. diff --git a/RELEASE.md b/RELEASE.md index 48cba0cad2..b81494519d 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -5,6 +5,13 @@ Please follow the established format: - Use present tense (e.g. 'Add new feature') - Include the ID number for the related PR (or PRs) in parentheses --> +# Next release + +## Major features and improvements + +## Bug fixes and other changes +- Upgrade the gitpod workspace-full to a newer version which includes both Node 18 and Python 3.11.5. (#1862) + # Release 9.0.0 ## Major features and improvements From 92860cd3a9c922abafd3d81ff767cbc2266e993b Mon Sep 17 00:00:00 2001 From: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Date: Mon, 22 Apr 2024 10:48:07 +0100 Subject: [PATCH 02/22] Move the `Expand all modular pipelines` toggle from the settings panel (#1858) * Update config.js Signed-off-by: Sajid Alam * Update config.js Signed-off-by: Sajid Alam * remove experiment and expand modular pipeline from settings menu Signed-off-by: Sajid Alam * implement the expandallpipelines Signed-off-by: Sajid Alam * Update flowchart-primary-toolbar.test.js Signed-off-by: Sajid Alam * improve tests Signed-off-by: Sajid Alam * changes based on review Signed-off-by: Sajid Alam * remove query from settings-modal Signed-off-by: Sajid Alam * Update settings-modal.js Signed-off-by: Sajid Alam * design changes make icon always hoverable Signed-off-by: Sajid Alam * update css Signed-off-by: Sajid Alam --------- Signed-off-by: Sajid Alam --- cypress/tests/ui/toolbar/global-toolbar.cy.js | 36 ++---- src/components/app/app.js | 1 + .../run-details-modal/run-details-modal.scss | 12 ++ .../flowchart-primary-toolbar.js | 121 ++++++++++++------ .../flowchart-primary-toolbar.test.js | 23 +++- .../global-toolbar/global-toolbar.scss | 4 + .../global-toolbar/global-toolbar.test.js | 1 + src/components/icons/collapse-pipelines.js | 9 ++ src/components/icons/expand-pipelines.js | 9 ++ .../settings-modal/settings-modal.js | 74 +++++------ src/components/ui/toggle/toggle.scss | 5 +- src/config.js | 6 - src/store/initial-state.js | 1 + src/store/initial-state.test.js | 1 + 14 files changed, 184 insertions(+), 119 deletions(-) create mode 100644 src/components/icons/collapse-pipelines.js create mode 100644 src/components/icons/expand-pipelines.js diff --git a/cypress/tests/ui/toolbar/global-toolbar.cy.js b/cypress/tests/ui/toolbar/global-toolbar.cy.js index 95f07b127d..a0bcc80e91 100644 --- a/cypress/tests/ui/toolbar/global-toolbar.cy.js +++ b/cypress/tests/ui/toolbar/global-toolbar.cy.js @@ -146,38 +146,24 @@ describe('Global Toolbar', () => { }); }); - it('verifies that users can expand all modular pipelines on first load. #TC-7', () => { + it('verifies that users can expand all modular pipelines directly from the toolbar. #TC-7', () => { const modularPipelineChildNodeText = 'Create Derived Features'; - // Alias - cy.get('[data-test="pipeline-toggle-input-expandAllPipelines"]').as( - 'isExpandAllPipelinesCheckBox' - ); + // Alias for better readability + cy.get('[data-test="expand-all-pipelines-toggle"]').as('expandAllPipelinesToggle'); // Assert before action - cy.get('@isExpandAllPipelinesCheckBox').should('not.be.checked'); - cy.get('.pipeline-node__text').should( - 'not.contain', - modularPipelineChildNodeText - ); - cy.get('[role="treeitem"]') - .should('have.attr', 'aria-expanded') - .should('eq', 'false'); + cy.get('@expandAllPipelinesToggle').should('not.be.checked'); + cy.get('.pipeline-node__text').should('not.contain', modularPipelineChildNodeText); + cy.get('[role="treeitem"]').should('have.attr', 'aria-expanded', 'false'); - // Action - cy.get('@isExpandAllPipelinesCheckBox').check({ force: true }); - cy.get('[data-test="Apply changes and close in Settings Modal"]').click({ - force: true, - }); + // Action - toggling the expand all pipelines directly from the toolbar + cy.get('@expandAllPipelinesToggle').click(); // Assert after action - cy.get('[role="treeitem"]', { timeout: 5000 }) - .should('have.attr', 'aria-expanded') - .should('eq', 'true'); - cy.get('.pipeline-node__text').should( - 'contain', - modularPipelineChildNodeText - ); + cy.get('[role="treeitem"]') + .should('have.attr', 'aria-expanded', 'true'); + cy.get('.pipeline-node__text').should('contain', modularPipelineChildNodeText); }); }); }); diff --git a/src/components/app/app.js b/src/components/app/app.js index 6948da6558..ba70cfb8af 100644 --- a/src/components/app/app.js +++ b/src/components/app/app.js @@ -97,6 +97,7 @@ App.propTypes = { labelBtn: PropTypes.bool, layerBtn: PropTypes.bool, exportBtn: PropTypes.bool, + pipelineBtn: PropTypes.bool, sidebar: PropTypes.bool, }), /** diff --git a/src/components/experiment-tracking/run-details-modal/run-details-modal.scss b/src/components/experiment-tracking/run-details-modal/run-details-modal.scss index ed6b438fbc..c43b8b4dae 100644 --- a/src/components/experiment-tracking/run-details-modal/run-details-modal.scss +++ b/src/components/experiment-tracking/run-details-modal/run-details-modal.scss @@ -18,3 +18,15 @@ text-align: right; width: 100%; } + +.version-reminder-and-run-details-button-wrapper { + align-items: baseline; + display: flex; + justify-content: space-between; + width: 100%; + margin-top: 50px; + + .button:first-of-type { + margin-right: 20px; + } +} diff --git a/src/components/flowchart-primary-toolbar/flowchart-primary-toolbar.js b/src/components/flowchart-primary-toolbar/flowchart-primary-toolbar.js index 2d7932fc57..844fc7ec90 100644 --- a/src/components/flowchart-primary-toolbar/flowchart-primary-toolbar.js +++ b/src/components/flowchart-primary-toolbar/flowchart-primary-toolbar.js @@ -5,13 +5,18 @@ import { toggleLayers, toggleSidebar, toggleTextLabels, + changeFlag, } from '../../actions'; +import { loadInitialPipelineData } from '../../actions/pipelines'; import IconButton from '../ui/icon-button'; import LabelIcon from '../icons/label'; import ExportIcon from '../icons/export'; import LayersIcon from '../icons/layers'; import PrimaryToolbar from '../primary-toolbar'; import { getVisibleLayerIDs } from '../../selectors/disabled'; +import ExpandPipelinesIcon from '../icons/expand-pipelines'; +import CollapsePipelinesIcon from '../icons/collapse-pipelines'; +import { useGeneratePathname } from '../../utils/hooks/use-generate-pathname'; /** * Main controls for filtering the chart data @@ -28,47 +33,76 @@ export const FlowchartPrimaryToolbar = ({ textLabels, visible, visibleLayers, -}) => ( - <> - - onToggleTextLabels(!textLabels)} - visible={visible.labelBtn} - /> - onToggleLayers(!visibleLayers)} - visible={visible.layerBtn} - /> - onToggleExportModal(true)} - visible={visible.exportBtn} - /> - - -); + expandedPipelines, + onToggleExpandAllPipelines, +}) => { + const { toSetQueryParam } = useGeneratePathname(); + + const handleToggleExpandAllPipelines = () => { + const isExpanded = !expandedPipelines; + onToggleExpandAllPipelines(isExpanded); + toSetQueryParam('expandAllPipelines', isExpanded.toString()); + }; + + return ( + <> + + onToggleTextLabels(!textLabels)} + visible={visible.labelBtn} + /> + onToggleLayers(!visibleLayers)} + visible={visible.layerBtn} + /> + + onToggleExportModal(true)} + visible={visible.exportBtn} + /> + + + ); +}; export const mapStateToProps = (state) => ({ disableLayerBtn: !state.layer.ids.length, @@ -76,6 +110,7 @@ export const mapStateToProps = (state) => ({ textLabels: state.textLabels, visible: state.visible, visibleLayers: Boolean(getVisibleLayerIDs(state).length), + expandedPipelines: state.flags.expandAllPipelines, }); export const mapDispatchToProps = (dispatch) => ({ @@ -91,6 +126,10 @@ export const mapDispatchToProps = (dispatch) => ({ onToggleTextLabels: (value) => { dispatch(toggleTextLabels(Boolean(value))); }, + onToggleExpandAllPipelines: (isExpanded) => { + dispatch(changeFlag('expandAllPipelines', isExpanded)); + dispatch(loadInitialPipelineData()); + }, }); export default connect( diff --git a/src/components/flowchart-primary-toolbar/flowchart-primary-toolbar.test.js b/src/components/flowchart-primary-toolbar/flowchart-primary-toolbar.test.js index f9c332ca2e..732389f33c 100644 --- a/src/components/flowchart-primary-toolbar/flowchart-primary-toolbar.test.js +++ b/src/components/flowchart-primary-toolbar/flowchart-primary-toolbar.test.js @@ -6,10 +6,16 @@ import ConnectedFlowchartPrimaryToolbar, { } from './flowchart-primary-toolbar'; import { mockState, setup } from '../../utils/state.mock'; +jest.mock('../../utils/hooks/use-generate-pathname', () => ({ + useGeneratePathname: () => ({ + toSetQueryParam: jest.fn(), + }), +})); + describe('PrimaryToolbar', () => { it('renders without crashing', () => { const wrapper = setup.mount(); - expect(wrapper.find('.pipeline-icon-toolbar__button').length).toBe(4); + expect(wrapper.find('.pipeline-icon-toolbar__button').length).toBe(5); }); it('hides all buttons (except menu button) when visible prop is false for each of them', () => { @@ -17,6 +23,7 @@ describe('PrimaryToolbar', () => { labelBtn: false, layerBtn: false, exportBtn: false, + pipelineBtn: false, }; const wrapper = setup.mount(, { visible, @@ -31,7 +38,7 @@ describe('PrimaryToolbar', () => { const wrapper = setup.mount(, { visible, }); - expect(wrapper.find('.pipeline-icon-toolbar__button').length).toBe(3); + expect(wrapper.find('.pipeline-icon-toolbar__button').length).toBe(4); }); const functionCalls = [ @@ -39,6 +46,7 @@ describe('PrimaryToolbar', () => { ['.pipeline-menu-button--labels', 'onToggleTextLabels'], ['.pipeline-menu-button--export', 'onToggleExportModal'], ['.pipeline-menu-button--layers', 'onToggleLayers'], + ['.pipeline-menu-button--pipeline', 'onToggleExpandAllPipelines'], ]; test.each(functionCalls)( @@ -70,6 +78,7 @@ describe('PrimaryToolbar', () => { settingsModal: expect.any(Boolean), labelBtn: expect.any(Boolean), layerBtn: expect.any(Boolean), + pipelineBtn: expect.any(Boolean), sidebar: expect.any(Boolean), }), visibleLayers: expect.any(Boolean), @@ -113,5 +122,15 @@ describe('PrimaryToolbar', () => { type: 'TOGGLE_TEXT_LABELS', }); }); + + it('onToggleExpandAllPipelines', () => { + const dispatch = jest.fn(); + mapDispatchToProps(dispatch).onToggleExpandAllPipelines(true); + expect(dispatch.mock.calls[0][0]).toEqual({ + name: 'expandAllPipelines', + type: 'CHANGE_FLAG', + value: true, + }); + }); }); }); diff --git a/src/components/global-toolbar/global-toolbar.scss b/src/components/global-toolbar/global-toolbar.scss index 5a0cb30971..b117e90442 100644 --- a/src/components/global-toolbar/global-toolbar.scss +++ b/src/components/global-toolbar/global-toolbar.scss @@ -46,6 +46,10 @@ } } +.pipeline-menu-button--pipeline svg { + opacity: 0.7; +} + .pipeline-global-routes-toolbar a.active .pipeline-menu-button--link { background-color: var(--color-global-toolbar-active-btn); border-right: 1px solid var(--color-border-line); diff --git a/src/components/global-toolbar/global-toolbar.test.js b/src/components/global-toolbar/global-toolbar.test.js index f591299f08..74e9012844 100644 --- a/src/components/global-toolbar/global-toolbar.test.js +++ b/src/components/global-toolbar/global-toolbar.test.js @@ -56,6 +56,7 @@ describe('GlobalToolbar', () => { miniMapBtn: true, modularPipelineFocusMode: null, metadataModal: false, + pipelineBtn: true, settingsModal: false, shareableUrlModal: false, sidebar: true, diff --git a/src/components/icons/collapse-pipelines.js b/src/components/icons/collapse-pipelines.js new file mode 100644 index 0000000000..e23687c878 --- /dev/null +++ b/src/components/icons/collapse-pipelines.js @@ -0,0 +1,9 @@ +import React from 'react'; + +const CollapsePipelinesIcon = ({ className }) => ( + + + +); + +export default CollapsePipelinesIcon; diff --git a/src/components/icons/expand-pipelines.js b/src/components/icons/expand-pipelines.js new file mode 100644 index 0000000000..d280fa2638 --- /dev/null +++ b/src/components/icons/expand-pipelines.js @@ -0,0 +1,9 @@ +import React from 'react'; + +const ExpandPipelinesIcon = ({ className }) => ( + + + +); + +export default ExpandPipelinesIcon; diff --git a/src/components/settings-modal/settings-modal.js b/src/components/settings-modal/settings-modal.js index c3f081f8e3..a4847d04e7 100644 --- a/src/components/settings-modal/settings-modal.js +++ b/src/components/settings-modal/settings-modal.js @@ -8,14 +8,9 @@ import { } from '../../actions'; import { getFlagsState } from '../../utils/flags'; import SettingsModalRow from './settings-modal-row'; -import { - settings as settingsConfig, - localStorageName, - params, -} from '../../config'; +import { settings as settingsConfig, localStorageName } from '../../config'; import { saveLocalStorage } from '../../store/helpers'; import { localStorageKeyFeatureHintsStep } from '../../components/feature-hints/feature-hints'; -import { useGeneratePathname } from '../../utils/hooks/use-generate-pathname'; import Button from '../ui/button'; import Modal from '../ui/modal'; @@ -47,8 +42,6 @@ const SettingsModal = ({ useState(showFeatureHints); const [toggleFlags, setToggleFlags] = useState(flags); - const { toSetQueryParam } = useGeneratePathname(); - useEffect(() => { setShowFeatureHintsValue(showFeatureHints); }, [showFeatureHints]); @@ -66,9 +59,6 @@ const SettingsModal = ({ const updatedFlags = Object.entries(toggleFlags); updatedFlags.map((each) => { const [name, value] = each; - if (name === params.expandAll) { - toSetQueryParam(params.expandAll, value); - } return onToggleFlag(name, value); }); @@ -95,7 +85,6 @@ const SettingsModal = ({ onToggleIsPrettyName, showSettingsModal, toggleFlags, - toSetQueryParam, ]); const resetStateCloseModal = () => { @@ -115,7 +104,6 @@ const SettingsModal = ({ >
-
General
Name
State
@@ -149,9 +137,6 @@ const SettingsModal = ({ } }} /> -
-
-
Experiments
{flagData.map(({ name, value, description }) => ( ))} +
+
{isRunningLocally() ? ( isOutdated ? (
@@ -190,33 +177,34 @@ const SettingsModal = ({
) ) : null} -
-
- - +
+ + +
diff --git a/src/components/ui/toggle/toggle.scss b/src/components/ui/toggle/toggle.scss index cbb1af584e..c7c900600a 100644 --- a/src/components/ui/toggle/toggle.scss +++ b/src/components/ui/toggle/toggle.scss @@ -8,8 +8,8 @@ } .kui-theme--dark { - --color-toggle-on: #{colors.$blue-300}; - --color-toggle-on-bar: #{colors.$ocean-400}; + --color-toggle-on: #{colors.$blue-0}; + --color-toggle-on-bar: #{colors.$blue-0}; --color-toggle-off: #{colors.$white-0}; --color-toggle-off-bar: #{colors.$black-400}; } @@ -75,6 +75,7 @@ .pipeline-toggle-label--checked::before { background-color: var(--color-toggle-on-bar); + opacity: 0.3; } .pipeline-toggle-label--checked::after { diff --git a/src/config.js b/src/config.js index 0dca77dcd1..9de91e6263 100644 --- a/src/config.js +++ b/src/config.js @@ -63,12 +63,6 @@ export const flags = { default: true, icon: '🐳', }, - expandAllPipelines: { - name: 'Expand all modular pipelines', - description: 'Expand all modular pipelines on first load', - default: false, - icon: '🔛', - }, }; export const settings = { diff --git a/src/store/initial-state.js b/src/store/initial-state.js index 5279bdd49b..8bc03021ea 100644 --- a/src/store/initial-state.js +++ b/src/store/initial-state.js @@ -40,6 +40,7 @@ export const createInitialState = () => ({ miniMap: true, miniMapBtn: true, modularPipelineFocusMode: null, + pipelineBtn: true, settingsModal: false, shareableUrlModal: false, sidebar: window.innerWidth > sidebarWidth.breakpoint, diff --git a/src/store/initial-state.test.js b/src/store/initial-state.test.js index 240bfabd54..56b3458e93 100644 --- a/src/store/initial-state.test.js +++ b/src/store/initial-state.test.js @@ -123,6 +123,7 @@ describe('getInitialState', () => { exportBtn: true, labelBtn: true, layerBtn: true, + pipelineBtn: true, }, }); }); From 65acdf96ba32b44f43b5aad4f98b8fd583275d2e Mon Sep 17 00:00:00 2001 From: Jitendra Gundaniya <38945204+jitu5@users.noreply.github.com> Date: Mon, 22 Apr 2024 12:08:39 +0100 Subject: [PATCH 03/22] CircleCI to Github actions migration for Kedro-Viz (test) (#1846) * CircleCI to Github actions migration for test * trufflehog-ignore updated * matrix for js removed * Matrix removed for js in frontend * build-frontend workflow merged with js-lint-test workflow * Reusable action for tests created and kept node version and package path as input * action path fix * Action path fix in workflow * action file correction setup_tests * checkout moved from actions * hashFiles format added * name updated to install_kedro_and_python_dependencies * Condition for PR to main with windows updated * Code review fix added * Merge Gatekeeper added Signed-off-by: Jitendra Gundaniya * All CircleCI Pipeline name added to ignore list. Signed-off-by: Jitendra Gundaniya * ci/circleci pipeline renamed * Node version updated Signed-off-by: Jitendra Gundaniya * Windows run for PR to main is updated Signed-off-by: Jitendra Gundaniya --------- Signed-off-by: Jitendra Gundaniya --- .../action.yml | 15 +++++ .../install_node_dependencies/action.yml | 37 +++++++++++ .github/actions/setup_tests/action.yml | 46 ++++++++++++++ .github/workflows/all-checks.yml | 44 +++++++++++++ .github/workflows/build-backend.yml | 44 +++++++++++++ .github/workflows/e2e-tests.yml | 39 ++++++++++++ .../workflows/javascript-lint-and-tests.yml | 61 +++++++++++++++++++ .github/workflows/lint.yml | 45 ++++++++++++++ .github/workflows/merge-gatekeeper.yml | 26 ++++++++ .github/workflows/unit-tests.yml | 39 ++++++++++++ trufflehog-ignore.txt | 3 +- 11 files changed, 398 insertions(+), 1 deletion(-) create mode 100644 .github/actions/install_kedro_and_python_dependencies/action.yml create mode 100644 .github/actions/install_node_dependencies/action.yml create mode 100644 .github/actions/setup_tests/action.yml create mode 100644 .github/workflows/all-checks.yml create mode 100644 .github/workflows/build-backend.yml create mode 100644 .github/workflows/e2e-tests.yml create mode 100644 .github/workflows/javascript-lint-and-tests.yml create mode 100644 .github/workflows/lint.yml create mode 100644 .github/workflows/merge-gatekeeper.yml create mode 100644 .github/workflows/unit-tests.yml diff --git a/.github/actions/install_kedro_and_python_dependencies/action.yml b/.github/actions/install_kedro_and_python_dependencies/action.yml new file mode 100644 index 0000000000..3874ccdc20 --- /dev/null +++ b/.github/actions/install_kedro_and_python_dependencies/action.yml @@ -0,0 +1,15 @@ +name: Install Kedro and other Python Dependencies +description: Installs Kedro from the main branch and other Python dependencies, then prints the Python version and installed packages. +runs: + using: composite + steps: + - name: Install Python dependencies + run: |- + pip install git+https://github.com/kedro-org/kedro@main + pip install -r package/test_requirements.txt -r demo-project/src/docker_requirements.txt -U + shell: bash + - name: Echo package versions + run: |- + python -V + pip freeze + shell: bash \ No newline at end of file diff --git a/.github/actions/install_node_dependencies/action.yml b/.github/actions/install_node_dependencies/action.yml new file mode 100644 index 0000000000..b36a41e83a --- /dev/null +++ b/.github/actions/install_node_dependencies/action.yml @@ -0,0 +1,37 @@ +name: Setup Node.js and Install Dependencies +description: Sets up a specific Node.js version, caches Node modules, and installs Node dependencies. + +inputs: + node-version: + description: 'Node.js version' + required: false + default: '18.20.0' + + package-path: + description: 'Path to package.json file' + required: false + default: '.' + +runs: + using: composite + steps: + - name: Setup Node.js + uses: actions/setup-node@v4.0.0 + with: + node-version: ${{ inputs.node-version }} + + - name: Get NPM Cache Directory + id: npm-cache-dir + run: echo "dir=$(npm config get cache)" >> $GITHUB_OUTPUT + shell: bash + + - name: Cache Node.js packages + uses: actions/cache@v2 + with: + path: "${{ steps.npm-cache-dir.outputs.dir }}" + key: "${{ runner.os }}-node-${{ hashFiles(format('{0}/package-lock.json', inputs.package-path)) }}" + restore-keys: "${{ runner.os }}-node-" + + - name: Install Node Dependencies + run: npm install + shell: bash diff --git a/.github/actions/setup_tests/action.yml b/.github/actions/setup_tests/action.yml new file mode 100644 index 0000000000..4de29e785a --- /dev/null +++ b/.github/actions/setup_tests/action.yml @@ -0,0 +1,46 @@ +name: Setup Tests +description: Sets up the testing environment by setting up Python and Node.js, caching Python packages, installing Kedro and other Python dependencies, and building the React application. + +inputs: + os: + description: 'Operating system' + required: false + default: 'ubuntu-latest' + python-version: + description: 'Python version' + required: false + default: '3.9' + +runs: + using: "composite" + steps: + - name: Set up Python ${{inputs.python-version}} + uses: actions/setup-python@v5 + with: + python-version: ${{inputs.python-version}} + + - name: Cache python packages for Linux + if: inputs.os == 'ubuntu-latest' + uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{inputs.os}}-python-${{inputs.python-version}} + + - name: Cache python packages for Windows + if: inputs.os == 'windows-latest' + uses: actions/cache@v4 + with: + path: ~\AppData\Local\pip\Cache + key: ${{inputs.os}}-python-${{inputs.python-version}} + + - name: Install Kedro and other Python Dependencies + uses: "./.github/actions/install_kedro_and_python_dependencies" + + - name: Setup Node.js and Install Dependencies + uses: "./.github/actions/install_node_dependencies" + + - name: Build React application + run: |- + node -v + make build + shell: bash \ No newline at end of file diff --git a/.github/workflows/all-checks.yml b/.github/workflows/all-checks.yml new file mode 100644 index 0000000000..4c981d6c8b --- /dev/null +++ b/.github/workflows/all-checks.yml @@ -0,0 +1,44 @@ +name: Run all checks on Kedro-Viz +# Runs end-to-end tests, unit tests, linting and JavaScript +# linting & tests on Kedro-Viz for different +# operating systems and Python versions. + +on: + workflow_call: + workflow_dispatch: + schedule: + # Run every day at 1:00 AM(UTC time) + - cron: 0 1 * * * +jobs: + e2e_tests: + strategy: + matrix: + os: [ windows-latest, ubuntu-latest ] + python-version: [ "3.9", "3.10", "3.11" ] + uses: ./.github/workflows/e2e-tests.yml + with: + os: ${{ matrix.os }} + python-version: ${{ matrix.python-version }} + + unit_tests: + strategy: + matrix: + os: [ windows-latest, ubuntu-latest ] + python-version: [ "3.9", "3.10", "3.11" ] + uses: ./.github/workflows/unit-tests.yml + with: + os: ${{ matrix.os }} + python-version: ${{ matrix.python-version }} + + lint: + strategy: + matrix: + os: [ ubuntu-latest ] + python-version: [ "3.9", "3.10", "3.11" ] + uses: ./.github/workflows/lint.yml + with: + os: ${{ matrix.os }} + python-version: ${{ matrix.python-version }} + + javascript_lint_and_tests: + uses: ./.github/workflows/javascript-lint-and-tests.yml diff --git a/.github/workflows/build-backend.yml b/.github/workflows/build-backend.yml new file mode 100644 index 0000000000..70a8691854 --- /dev/null +++ b/.github/workflows/build-backend.yml @@ -0,0 +1,44 @@ +name: Build backend +# Runs end-to-end tests, unit tests, and linting on the backend code +# for different operating systems and Python versions. + +on: + push: + paths: + - 'package/**' + - '.github/**' + pull_request: + paths: + - 'package/**' + - '.github/**' + workflow_dispatch: +jobs: + e2e_tests: + strategy: + matrix: + os: [ windows-latest, ubuntu-latest ] + python-version: [ "3.9", "3.10", "3.11" ] + uses: ./.github/workflows/e2e-tests.yml + with: + os: ${{ matrix.os }} + python-version: ${{ matrix.python-version }} + + unit_tests: + strategy: + matrix: + os: [ windows-latest, ubuntu-latest ] + python-version: [ "3.9", "3.10", "3.11" ] + uses: ./.github/workflows/unit-tests.yml + with: + os: ${{ matrix.os }} + python-version: ${{ matrix.python-version }} + + lint: + strategy: + matrix: + os: [ ubuntu-latest ] + python-version: [ "3.9", "3.10", "3.11" ] + uses: ./.github/workflows/lint.yml + with: + os: ${{ matrix.os }} + python-version: ${{ matrix.python-version }} diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml new file mode 100644 index 0000000000..5eee3d98cd --- /dev/null +++ b/.github/workflows/e2e-tests.yml @@ -0,0 +1,39 @@ +name: Run e2e tests on Kedro-Viz +# Runs end-to-end tests on Kedro-Viz for different +# operating systems and Python versions. + +on: + workflow_call: + inputs: + os: + type: string + python-version: + type: string +jobs: + e2e_tests: + runs-on: ${{ inputs.os }} + + # below condition checks if the operating system is Ubuntu, or + # if the operating system is Windows and the branch is main/demo + if: > + inputs.os == 'ubuntu-latest' || + ( + ( + github.ref == 'refs/heads/main' || + github.ref == 'refs/heads/demo' + ) && + inputs.os == 'windows-latest' + ) + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Tests + uses: "./.github/actions/setup_tests" + with: + os: ${{ inputs.os }} + python-version: ${{ inputs.python-version }} + + - name: Run all end to end tests + run: make e2e-tests \ No newline at end of file diff --git a/.github/workflows/javascript-lint-and-tests.yml b/.github/workflows/javascript-lint-and-tests.yml new file mode 100644 index 0000000000..0cff820c9b --- /dev/null +++ b/.github/workflows/javascript-lint-and-tests.yml @@ -0,0 +1,61 @@ +name: Run javascript linters and tests on Kedro-Viz +# Runs JavaScript linting, unit tests, and end-to-end tests on +# Kedro-Viz for ubuntu-latest operating systems and Python 3.9. + +on: + push: + paths-ignore: + - 'package/**' + pull_request: + paths-ignore: + - 'package/**' + workflow_dispatch: + workflow_call: + +jobs: + javascript_lint_and_tests: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up Python "3.9" + uses: actions/setup-python@v5 + with: + python-version: "3.9" + + - name: Cache python packages for Linux + uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ubuntu-latest-python-3.9 + + - name: Install Kedro and other Python Dependencies + uses: "./.github/actions/install_kedro_and_python_dependencies" + + - name: Setup Node.js and Install Dependencies + uses: "./.github/actions/install_node_dependencies" + + - name: Setup Cypress requirements + run: |- + sudo sed -i 's/archive.ubuntu.com/us-east-1.ec2.archive.ubuntu.com/g' /etc/apt/sources.list + sudo apt-get update + sudo apt-get install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb + + - name: Test lib transpilation + run: npm run lib + + - name: Test JS library imports + run: |- + npm run lib-test:setup + cd tools/test-lib/react-app + npm run test:ci + + - name: Run Eslint + run: npm run lint + + - name: Run JavaScript tests + run: npm run test:ci + + - name: Run Javascript end to end tests + run: npm run cy:ci diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml new file mode 100644 index 0000000000..02489d36dd --- /dev/null +++ b/.github/workflows/lint.yml @@ -0,0 +1,45 @@ +name: Run linters on Kedro-Viz +# Runs secret scan, security scan, GraphQL schema check, +# and Python formatters and linters on Kedro-Viz for +# different operating systems and Python versions. + +on: + workflow_call: + inputs: + os: + type: string + python-version: + type: string +jobs: + lint: + runs-on: ${{ inputs.os }} + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up Python ${{inputs.python-version}} + uses: actions/setup-python@v5 + with: + python-version: ${{inputs.python-version}} + + - name: Cache python packages for Linux + if: inputs.os == 'ubuntu-latest' + uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{inputs.os}}-python-${{inputs.python-version}} + + - name: Install Kedro and other Python Dependencies + uses: "./.github/actions/install_kedro_and_python_dependencies" + + - name: Run secret scan + run: make secret-scan + + - name: Run security scan + run: make security-scan + + - name: Verify GraphQL schema is up to date + run: make schema-check + + - name: Run Python formatters and linters + run: make format-check lint-check diff --git a/.github/workflows/merge-gatekeeper.yml b/.github/workflows/merge-gatekeeper.yml new file mode 100644 index 0000000000..abc5c013fe --- /dev/null +++ b/.github/workflows/merge-gatekeeper.yml @@ -0,0 +1,26 @@ +name: Merge Gatekeeper + +on: + pull_request: + branches: + - main + +jobs: + merge-gatekeeper: + runs-on: ubuntu-latest + # Restrict permissions of the GITHUB_TOKEN. + # Docs: https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs + permissions: + checks: read + statuses: read + steps: + - name: Run Merge Gatekeeper + # NOTE: v1 is updated to reflect the latest v1.x.y. Please use any tag/branch that suits your needs: + # https://github.com/upsidr/merge-gatekeeper/tags + # https://github.com/upsidr/merge-gatekeeper/branches + uses: upsidr/merge-gatekeeper@v1 + with: + token: ${{ secrets.GITHUB_TOKEN }} + timeout: 3600 + interval: 30 + ignored: 'ci/circleci: win_unit_tests-3.9,ci/circleci: win_unit_tests-3.10,ci/circleci: win_unit_tests-3.8,ci/circleci: unit_tests-3.10,ci/circleci: unit_tests-3.8,ci/circleci: unit_tests-3.9,ci/circleci: win_e2e_tests-3.10,ci/circleci: win_e2e_tests-3.9,ci/circleci: win_e2e_tests-3.8,ci/circleci: e2e_tests-3.8,ci/circleci: e2e_tests-3.9,ci/circleci: e2e_tests-3.10,ci/circleci: lint-3.8,ci/circleci: lint-3.9,ci/circleci: lint-3.10,ci/circleci: javascript_lint_and_tests,ci/circleci: check-updated-files,ci/circleci: all_circleci_checks_succeeded' \ No newline at end of file diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml new file mode 100644 index 0000000000..fe970683c3 --- /dev/null +++ b/.github/workflows/unit-tests.yml @@ -0,0 +1,39 @@ +name: Run unit-tests on Kedro-Viz +# Runs unit tests on Kedro-Viz across different +# OS and Python versions after environment setup. + +on: + workflow_call: + inputs: + os: + type: string + python-version: + type: string +jobs: + unit_tests: + runs-on: ${{ inputs.os }} + + # below condition checks if the operating system is Ubuntu, or + # if the operating system is Windows and the branch is main/demo + if: > + inputs.os == 'ubuntu-latest' || + ( + ( + github.ref == 'refs/heads/main' || + github.ref == 'refs/heads/demo' + ) && + inputs.os == 'windows-latest' + ) + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Tests + uses: "./.github/actions/setup_tests" + with: + os: ${{ inputs.os }} + python-version: ${{ inputs.python-version }} + + - name: Run Python tests + run: make pytest \ No newline at end of file diff --git a/trufflehog-ignore.txt b/trufflehog-ignore.txt index da31b5fafa..470cd17fce 100644 --- a/trufflehog-ignore.txt +++ b/trufflehog-ignore.txt @@ -7,4 +7,5 @@ README.md LAYOUT_ENGINE.md src/utils/random-utils.js src/components/update-reminder/update-reminder-content.js -src/components/feature-hints/feature-hints-content.js \ No newline at end of file +src/components/feature-hints/feature-hints-content.js +cypress/fixtures/graphql/ \ No newline at end of file From d5a8b83fbe598f1b011252594d2afbb12d7b4a33 Mon Sep 17 00:00:00 2001 From: rashidakanchwala <37628668+rashidakanchwala@users.noreply.github.com> Date: Wed, 24 Apr 2024 17:50:40 +0100 Subject: [PATCH 04/22] Refactor backend integration with Kedro by replacing bootstrap_project with configure_project (#1796) * Try remove bootstrap_project() Signed-off-by: Merel Theisen * Try set the package name for spawn process Signed-off-by: Merel Theisen * move configure project before kedro integration Signed-off-by: ravi-kumar-pilla * include package name in jupyter and dev run Signed-off-by: ravi-kumar-pilla * fix tests and lint Signed-off-by: ravi-kumar-pilla * update release note Signed-off-by: ravi-kumar-pilla * revert permission change Signed-off-by: ravi-kumar-pilla * fix PR comments Signed-off-by: ravi-kumar-pilla * working draft with parent project search Signed-off-by: ravi-kumar-pilla * update error message to be consistent with other viz commands Signed-off-by: ravi-kumar-pilla * reverting project not found bug fix Signed-off-by: ravi-kumar-pilla * update release note Signed-off-by: ravi-kumar-pilla --------- Signed-off-by: Merel Theisen Signed-off-by: ravi-kumar-pilla Co-authored-by: Merel Theisen Co-authored-by: ravi-kumar-pilla --- RELEASE.md | 2 ++ .../integrations/kedro/data_loader.py | 17 ++++++++----- package/kedro_viz/launchers/cli.py | 15 ++++++++++-- package/kedro_viz/launchers/jupyter.py | 3 +++ package/kedro_viz/server.py | 24 ++++++++++++++----- package/tests/test_launchers/test_cli.py | 19 +++++++++++---- package/tests/test_launchers/test_jupyter.py | 3 +++ 7 files changed, 65 insertions(+), 18 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index b81494519d..e9dc534e79 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -10,7 +10,9 @@ Please follow the established format: ## Major features and improvements ## Bug fixes and other changes + - Upgrade the gitpod workspace-full to a newer version which includes both Node 18 and Python 3.11.5. (#1862) +- Refactor backend integration with Kedro by replacing bootstrap_project with configure_project. (#1796) # Release 9.0.0 diff --git a/package/kedro_viz/integrations/kedro/data_loader.py b/package/kedro_viz/integrations/kedro/data_loader.py index f3506d74fb..4662951781 100644 --- a/package/kedro_viz/integrations/kedro/data_loader.py +++ b/package/kedro_viz/integrations/kedro/data_loader.py @@ -3,7 +3,7 @@ load data from projects created in a range of Kedro versions. """ -# pylint: disable=import-outside-toplevel, protected-access +# pylint: disable=protected-access import json import logging @@ -11,8 +11,10 @@ from typing import Any, Dict, Optional, Tuple from kedro import __version__ +from kedro.framework.project import configure_project, pipelines from kedro.framework.session import KedroSession from kedro.framework.session.store import BaseSessionStore +from kedro.framework.startup import bootstrap_project from kedro.io import DataCatalog from kedro.pipeline import Pipeline @@ -69,14 +71,16 @@ def load_data( project_path: Path, env: Optional[str] = None, include_hooks: bool = False, + package_name: Optional[str] = None, extra_params: Optional[Dict[str, Any]] = None, ) -> Tuple[DataCatalog, Dict[str, Pipeline], BaseSessionStore, Dict]: """Load data from a Kedro project. Args: - project_path: the path whether the Kedro project is located. + project_path: the path where the Kedro project is located. env: the Kedro environment to load the data. If not provided. it will use Kedro default, which is local. include_hooks: A flag to include all registered hooks in your Kedro Project. + package_name: The name of the current package extra_params: Optional dictionary containing extra project parameters for underlying KedroContext. If specified, will update (and therefore take precedence over) the parameters retrieved from the project @@ -85,10 +89,11 @@ def load_data( A tuple containing the data catalog and the pipeline dictionary and the session store. """ - from kedro.framework.project import pipelines - from kedro.framework.startup import bootstrap_project - - bootstrap_project(project_path) + if package_name: + configure_project(package_name) + else: + # bootstrap project when viz is run in dev mode + bootstrap_project(project_path) with KedroSession.create( project_path=project_path, diff --git a/package/kedro_viz/launchers/cli.py b/package/kedro_viz/launchers/cli.py index a97d01e60c..87fa5b2676 100644 --- a/package/kedro_viz/launchers/cli.py +++ b/package/kedro_viz/launchers/cli.py @@ -9,6 +9,7 @@ from click_default_group import DefaultGroup from kedro.framework.cli.project import PARAMS_ARG_HELP from kedro.framework.cli.utils import KedroCliError, _split_params +from kedro.framework.project import PACKAGE_NAME from packaging.version import parse from watchgod import RegExpWatcher, run_process @@ -153,6 +154,7 @@ def run( "env": env, "autoreload": autoreload, "include_hooks": include_hooks, + "package_name": PACKAGE_NAME, "extra_params": params, } if autoreload: @@ -268,6 +270,7 @@ def create_shareableviz_process( endpoint, bucket_name, include_hooks, + PACKAGE_NAME, process_completed, exception_queue, ), @@ -338,11 +341,19 @@ def create_shareableviz_process( def load_and_deploy_viz( - platform, endpoint, bucket_name, include_hooks, process_completed, exception_queue + platform, + endpoint, + bucket_name, + include_hooks, + package_name, + process_completed, + exception_queue, ): """Loads Kedro Project data, creates a deployer and deploys to a platform""" try: - load_and_populate_data(Path.cwd(), include_hooks=include_hooks) + load_and_populate_data( + Path.cwd(), include_hooks=include_hooks, package_name=package_name + ) # Start the deployment deployer = DeployerFactory.create_deployer(platform, endpoint, bucket_name) diff --git a/package/kedro_viz/launchers/jupyter.py b/package/kedro_viz/launchers/jupyter.py index d44493f424..f51f6ce7eb 100644 --- a/package/kedro_viz/launchers/jupyter.py +++ b/package/kedro_viz/launchers/jupyter.py @@ -1,6 +1,7 @@ """`kedro_viz.launchers.jupyter` provides line_magic to launch the viz server from a jupyter notebook. """ + # pragma: no cover import logging import multiprocessing @@ -12,6 +13,7 @@ import IPython from IPython.display import HTML, display +from kedro.framework.project import PACKAGE_NAME from watchgod import RegExpWatcher, run_process from kedro_viz.launchers.utils import _check_viz_up, _wait_for @@ -140,6 +142,7 @@ def run_viz( # pylint: disable=too-many-locals "env": env, "autoreload": autoreload, "include_hooks": include_hooks, + "package_name": PACKAGE_NAME, "extra_params": params, "project_path": project_path, } diff --git a/package/kedro_viz/server.py b/package/kedro_viz/server.py index 2c5b205576..eb31f7a9c9 100644 --- a/package/kedro_viz/server.py +++ b/package/kedro_viz/server.py @@ -53,14 +53,19 @@ def load_and_populate_data( path: Path, env: Optional[str] = None, include_hooks: bool = False, - extra_params: Optional[Dict[str, Any]] = None, + package_name: Optional[str] = None, pipeline_name: Optional[str] = None, + extra_params: Optional[Dict[str, Any]] = None, ): """Loads underlying Kedro project data and populates Kedro Viz Repositories""" # Loads data from underlying Kedro Project catalog, pipelines, session_store, stats_dict = kedro_data_loader.load_data( - path, env, include_hooks, extra_params + path, + env, + include_hooks, + package_name, + extra_params, ) pipelines = ( @@ -83,6 +88,7 @@ def run_server( project_path: Optional[str] = None, autoreload: bool = False, include_hooks: bool = False, + package_name: Optional[str] = None, extra_params: Optional[Dict[str, Any]] = None, ): # pylint: disable=redefined-outer-name """Run a uvicorn server with a FastAPI app that either launches API response data from a file @@ -101,15 +107,24 @@ def run_server( project_path: the optional path of the Kedro project that contains the pipelines to visualise. If not supplied, the current working directory will be used. include_hooks: A flag to include all registered hooks in your Kedro Project. + package_name: The name of the current package extra_params: Optional dictionary containing extra project parameters for underlying KedroContext. If specified, will update (and therefore take precedence over) the parameters retrieved from the project configuration. """ + path = Path(project_path) if project_path else Path.cwd() if load_file is None: - load_and_populate_data(path, env, include_hooks, extra_params, pipeline_name) + load_and_populate_data( + path, + env, + include_hooks, + package_name, + pipeline_name, + extra_params, + ) if save_file: save_api_responses_to_fs(save_file, fsspec.filesystem("file")) @@ -127,8 +142,6 @@ def run_server( if __name__ == "__main__": # pragma: no cover import argparse - from kedro.framework.startup import bootstrap_project - parser = argparse.ArgumentParser(description="Launch a development viz server") parser.add_argument("project_path", help="Path to a Kedro project") parser.add_argument( @@ -140,7 +153,6 @@ def run_server( args = parser.parse_args() project_path = (Path.cwd() / args.project_path).absolute() - bootstrap_project(project_path) run_process_kwargs = { "path": project_path, diff --git a/package/tests/test_launchers/test_cli.py b/package/tests/test_launchers/test_cli.py index 83baeb917f..df53b70ca5 100755 --- a/package/tests/test_launchers/test_cli.py +++ b/package/tests/test_launchers/test_cli.py @@ -87,6 +87,7 @@ def mock_project_path(mocker): "env": None, "autoreload": False, "include_hooks": False, + "package_name": None, "extra_params": {}, }, ), @@ -101,6 +102,7 @@ def mock_project_path(mocker): "env": None, "autoreload": False, "include_hooks": False, + "package_name": None, "extra_params": {}, }, ), @@ -120,6 +122,7 @@ def mock_project_path(mocker): "env": None, "autoreload": False, "include_hooks": False, + "package_name": None, "extra_params": {}, }, ), @@ -150,6 +153,7 @@ def mock_project_path(mocker): "env": "local", "autoreload": False, "include_hooks": False, + "package_name": None, "extra_params": {"extra_param": "param"}, }, ), @@ -164,6 +168,7 @@ def mock_project_path(mocker): "env": None, "autoreload": False, "include_hooks": True, + "package_name": None, "extra_params": {}, }, ), @@ -284,6 +289,7 @@ def test_kedro_viz_command_with_autoreload( "autoreload": True, "project_path": mock_project_path, "include_hooks": False, + "package_name": None, "extra_params": {}, }, "watcher_cls": RegExpWatcher, @@ -579,6 +585,7 @@ def test_create_shareableviz_process( endpoint, bucket_name, include_hooks, + None, mock_process_completed.return_value, mock_exception_queue.return_value, ), @@ -614,22 +621,24 @@ def test_create_shareableviz_process( @pytest.mark.parametrize( - "platform, endpoint, bucket_name, include_hooks", + "platform, endpoint, bucket_name, include_hooks, package_name", [ ( "azure", "https://example-bucket.web.core.windows.net", "example-bucket", False, + "demo_project", ), ( "aws", "http://example-bucket.s3-website.us-east-2.amazonaws.com/", "example-bucket", True, + "demo_project", ), - ("gcp", "http://34.120.87.227/", "example-bucket", False), - ("local", None, None, True), + ("gcp", "http://34.120.87.227/", "example-bucket", False, "demo_project"), + ("local", None, None, True, "demo_project"), ], ) def test_load_and_deploy_viz_success( @@ -637,6 +646,7 @@ def test_load_and_deploy_viz_success( endpoint, bucket_name, include_hooks, + package_name, mock_DeployerFactory, mock_load_and_populate_data, mock_process_completed, @@ -651,12 +661,13 @@ def test_load_and_deploy_viz_success( endpoint, bucket_name, include_hooks, + package_name, mock_process_completed, mock_exception_queue, ) mock_load_and_populate_data.assert_called_once_with( - mock_project_path, include_hooks=include_hooks + mock_project_path, include_hooks=include_hooks, package_name=package_name ) mock_DeployerFactory.create_deployer.assert_called_once_with( platform, endpoint, bucket_name diff --git a/package/tests/test_launchers/test_jupyter.py b/package/tests/test_launchers/test_jupyter.py index af41f9b884..dd489778ca 100644 --- a/package/tests/test_launchers/test_jupyter.py +++ b/package/tests/test_launchers/test_jupyter.py @@ -35,6 +35,7 @@ def test_run_viz(self, mocker, patched_check_viz_up): "env": None, "autoreload": False, "include_hooks": False, + "package_name": None, "extra_params": "", }, ) @@ -59,6 +60,7 @@ def test_run_viz(self, mocker, patched_check_viz_up): "env": None, "autoreload": False, "include_hooks": True, + "package_name": None, "extra_params": "", }, ) @@ -105,6 +107,7 @@ def test_run_viz_on_databricks(self, mocker, patched_check_viz_up, monkeypatch): "env": None, "autoreload": False, "include_hooks": False, + "package_name": None, "extra_params": "", }, ) From be7236e11022d0bb44a416ca7e08a139b96bd5d8 Mon Sep 17 00:00:00 2001 From: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Date: Thu, 25 Apr 2024 16:57:31 +0100 Subject: [PATCH 05/22] Update settings panel design (#1875) * update settings-modal design Signed-off-by: Sajid Alam * PrettyName off by default Signed-off-by: Sajid Alam * update colors from rgb to hex Signed-off-by: Sajid Alam --------- Signed-off-by: Sajid Alam --- .../settings-modal/settings-modal.js | 7 ------ .../settings-modal/settings-modal.scss | 23 ++++++++++--------- src/components/ui/button/button.scss | 4 ++-- src/components/ui/modal/modal.scss | 5 ++-- src/components/ui/toggle/toggle.scss | 18 +++++++++++---- src/styles/_variables.scss | 7 ++++++ 6 files changed, 38 insertions(+), 26 deletions(-) diff --git a/src/components/settings-modal/settings-modal.js b/src/components/settings-modal/settings-modal.js index a4847d04e7..aeb0fb5656 100644 --- a/src/components/settings-modal/settings-modal.js +++ b/src/components/settings-modal/settings-modal.js @@ -104,13 +104,6 @@ const SettingsModal = ({ >
-
-
Name
-
State
-
- Description -
-
Date: Tue, 30 Apr 2024 15:52:29 -0500 Subject: [PATCH 06/22] Fix Kedro-Viz waiting for valid Kedro Project (#1871) * Try remove bootstrap_project() Signed-off-by: Merel Theisen * Try set the package name for spawn process Signed-off-by: Merel Theisen * move configure project before kedro integration Signed-off-by: ravi-kumar-pilla * include package name in jupyter and dev run Signed-off-by: ravi-kumar-pilla * fix tests and lint Signed-off-by: ravi-kumar-pilla * update release note Signed-off-by: ravi-kumar-pilla * revert permission change Signed-off-by: ravi-kumar-pilla * fix PR comments Signed-off-by: ravi-kumar-pilla * working draft with parent project search Signed-off-by: ravi-kumar-pilla * update error message to be consistent with other viz commands Signed-off-by: ravi-kumar-pilla * update release note: Signed-off-by: ravi-kumar-pilla * update release note Signed-off-by: ravi-kumar-pilla * update error message Signed-off-by: ravi-kumar-pilla --------- Signed-off-by: Merel Theisen Signed-off-by: ravi-kumar-pilla Co-authored-by: Merel Theisen --- RELEASE.md | 1 + package/kedro_viz/launchers/cli.py | 18 +++++-- package/kedro_viz/launchers/utils.py | 24 +++++++++- package/tests/test_launchers/test_cli.py | 55 ++++++++++++++++++++-- package/tests/test_launchers/test_utils.py | 50 ++++++++++++++++++++ 5 files changed, 141 insertions(+), 7 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index e9dc534e79..2010bdf855 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -13,6 +13,7 @@ Please follow the established format: - Upgrade the gitpod workspace-full to a newer version which includes both Node 18 and Python 3.11.5. (#1862) - Refactor backend integration with Kedro by replacing bootstrap_project with configure_project. (#1796) +- Fix Kedro-Viz waiting for valid Kedro project. (#1871) # Release 9.0.0 diff --git a/package/kedro_viz/launchers/cli.py b/package/kedro_viz/launchers/cli.py index 87fa5b2676..5c2bcac75f 100644 --- a/package/kedro_viz/launchers/cli.py +++ b/package/kedro_viz/launchers/cli.py @@ -23,7 +23,9 @@ from kedro_viz.integrations.deployment.deployer_factory import DeployerFactory from kedro_viz.integrations.pypi import get_latest_version, is_running_outdated_version from kedro_viz.launchers.utils import ( + _PYPROJECT, _check_viz_up, + _find_kedro_project, _start_browser, _wait_for, viz_deploy_progress_timer, @@ -129,6 +131,17 @@ def run( """Launch local Kedro Viz instance""" from kedro_viz.server import run_server + kedro_project_path = _find_kedro_project(Path.cwd()) + + if kedro_project_path is None: + display_cli_message( + "ERROR: Failed to start Kedro-Viz : " + "Could not find the project configuration " + f"file '{_PYPROJECT}' at '{Path.cwd()}'. ", + "red", + ) + return + installed_version = parse(__version__) latest_version = get_latest_version() if is_running_outdated_version(installed_version, latest_version): @@ -152,16 +165,15 @@ def run( "save_file": save_file, "pipeline_name": pipeline, "env": env, + "project_path": kedro_project_path, "autoreload": autoreload, "include_hooks": include_hooks, "package_name": PACKAGE_NAME, "extra_params": params, } if autoreload: - project_path = Path.cwd() - run_server_kwargs["project_path"] = project_path run_process_kwargs = { - "path": project_path, + "path": kedro_project_path, "target": run_server, "kwargs": run_server_kwargs, "watcher_cls": RegExpWatcher, diff --git a/package/kedro_viz/launchers/utils.py b/package/kedro_viz/launchers/utils.py index 1ade2b426c..bf79602349 100644 --- a/package/kedro_viz/launchers/utils.py +++ b/package/kedro_viz/launchers/utils.py @@ -3,12 +3,14 @@ import logging import webbrowser +from pathlib import Path from time import sleep, time -from typing import Any, Callable +from typing import Any, Callable, Union import requests logger = logging.getLogger(__name__) +_PYPROJECT = "pyproject.toml" class WaitForException(Exception): @@ -105,3 +107,23 @@ def viz_deploy_progress_timer(process_completed, timeout): ) sleep(1) elapsed_time += 1 + + +def _is_project(project_path: Union[str, Path]) -> bool: + metadata_file = Path(project_path).expanduser().resolve() / _PYPROJECT + if not metadata_file.is_file(): + return False + + try: + return "[tool.kedro]" in metadata_file.read_text(encoding="utf-8") + # pylint: disable=broad-exception-caught + except Exception: + return False + + +def _find_kedro_project(current_dir: Path) -> Any: + paths_to_check = [current_dir] + list(current_dir.parents) + for project_dir in paths_to_check: + if _is_project(project_dir): + return project_dir + return None diff --git a/package/tests/test_launchers/test_cli.py b/package/tests/test_launchers/test_cli.py index df53b70ca5..4ad1b6dd86 100755 --- a/package/tests/test_launchers/test_cli.py +++ b/package/tests/test_launchers/test_cli.py @@ -9,6 +9,7 @@ from kedro_viz import __version__ from kedro_viz.constants import SHAREABLEVIZ_SUPPORTED_PLATFORMS, VIZ_DEPLOY_TIME_LIMIT from kedro_viz.launchers import cli +from kedro_viz.launchers.utils import _PYPROJECT from kedro_viz.server import run_server @@ -85,6 +86,7 @@ def mock_project_path(mocker): "save_file": None, "pipeline_name": None, "env": None, + "project_path": "testPath", "autoreload": False, "include_hooks": False, "package_name": None, @@ -100,6 +102,7 @@ def mock_project_path(mocker): "save_file": None, "pipeline_name": None, "env": None, + "project_path": "testPath", "autoreload": False, "include_hooks": False, "package_name": None, @@ -120,6 +123,7 @@ def mock_project_path(mocker): "save_file": None, "pipeline_name": None, "env": None, + "project_path": "testPath", "autoreload": False, "include_hooks": False, "package_name": None, @@ -151,6 +155,7 @@ def mock_project_path(mocker): "save_file": "save_dir", "pipeline_name": "data_science", "env": "local", + "project_path": "testPath", "autoreload": False, "include_hooks": False, "package_name": None, @@ -166,6 +171,7 @@ def mock_project_path(mocker): "save_file": None, "pipeline_name": None, "env": None, + "project_path": "testPath", "autoreload": False, "include_hooks": True, "package_name": None, @@ -185,6 +191,11 @@ def test_kedro_viz_command_run_server( runner = CliRunner() # Reduce the timeout argument from 600 to 1 to make test run faster. mocker.patch("kedro_viz.launchers.cli._wait_for.__defaults__", (True, 1, True, 1)) + # Mock finding kedro project + mocker.patch( + "kedro_viz.launchers.cli._find_kedro_project", + return_value=run_server_args["project_path"], + ) with runner.isolated_filesystem(): runner.invoke(cli.viz_cli, command_options) @@ -195,8 +206,30 @@ def test_kedro_viz_command_run_server( assert run_server_args["port"] in cli._VIZ_PROCESSES +def test_kedro_viz_command_should_log_project_not_found( + mocker, mock_project_path, mock_click_echo +): + # Reduce the timeout argument from 600 to 1 to make test run faster. + mocker.patch("kedro_viz.launchers.cli._wait_for.__defaults__", (True, 1, True, 1)) + # Mock finding kedro project + mocker.patch("kedro_viz.launchers.cli._find_kedro_project", return_value=None) + runner = CliRunner() + with runner.isolated_filesystem(): + runner.invoke(cli.viz_cli, ["viz", "run"]) + + mock_click_echo_calls = [ + call( + "\x1b[31mERROR: Failed to start Kedro-Viz : " + "Could not find the project configuration " + f"file '{_PYPROJECT}' at '{mock_project_path}'. \x1b[0m" + ) + ] + + mock_click_echo.assert_has_calls(mock_click_echo_calls) + + def test_kedro_viz_command_should_log_outdated_version( - mocker, mock_http_response, mock_click_echo + mocker, mock_http_response, mock_click_echo, mock_project_path ): installed_version = parse(__version__) mock_version = f"{installed_version.major + 1}.0.0" @@ -209,6 +242,10 @@ def test_kedro_viz_command_should_log_outdated_version( # Reduce the timeout argument from 600 to 1 to make test run faster. mocker.patch("kedro_viz.launchers.cli._wait_for.__defaults__", (True, 1, True, 1)) + # Mock finding kedro project + mocker.patch( + "kedro_viz.launchers.cli._find_kedro_project", return_value=mock_project_path + ) runner = CliRunner() with runner.isolated_filesystem(): runner.invoke(cli.viz_cli, ["viz", "run"]) @@ -228,7 +265,7 @@ def test_kedro_viz_command_should_log_outdated_version( def test_kedro_viz_command_should_not_log_latest_version( - mocker, mock_http_response, mock_click_echo + mocker, mock_http_response, mock_click_echo, mock_project_path ): requests_get = mocker.patch("requests.get") requests_get.return_value = mock_http_response( @@ -238,6 +275,10 @@ def test_kedro_viz_command_should_not_log_latest_version( mocker.patch("kedro_viz.server.run_server") # Reduce the timeout argument from 600 to 1 to make test run faster. mocker.patch("kedro_viz.launchers.cli._wait_for.__defaults__", (True, 1, True, 1)) + # Mock finding kedro project + mocker.patch( + "kedro_viz.launchers.cli._find_kedro_project", return_value=mock_project_path + ) runner = CliRunner() with runner.isolated_filesystem(): runner.invoke(cli.viz_cli, ["viz", "run"]) @@ -248,7 +289,7 @@ def test_kedro_viz_command_should_not_log_latest_version( def test_kedro_viz_command_should_not_log_if_pypi_is_down( - mocker, mock_http_response, mock_click_echo + mocker, mock_http_response, mock_click_echo, mock_project_path ): requests_get = mocker.patch("requests.get") requests_get.side_effect = requests.exceptions.RequestException("PyPI is down") @@ -256,6 +297,10 @@ def test_kedro_viz_command_should_not_log_if_pypi_is_down( mocker.patch("kedro_viz.server.run_server") # Reduce the timeout argument from 600 to 1 to make test run faster. mocker.patch("kedro_viz.launchers.cli._wait_for.__defaults__", (True, 1, True, 1)) + # Mock finding kedro project + mocker.patch( + "kedro_viz.launchers.cli._find_kedro_project", return_value=mock_project_path + ) runner = CliRunner() with runner.isolated_filesystem(): runner.invoke(cli.viz_cli, ["viz", "run"]) @@ -272,6 +317,10 @@ def test_kedro_viz_command_with_autoreload( # Reduce the timeout argument from 600 to 1 to make test run faster. mocker.patch("kedro_viz.launchers.cli._wait_for.__defaults__", (True, 1, True, 1)) + # Mock finding kedro project + mocker.patch( + "kedro_viz.launchers.cli._find_kedro_project", return_value=mock_project_path + ) runner = CliRunner() with runner.isolated_filesystem(): runner.invoke(cli.viz_cli, ["viz", "run", "--autoreload"]) diff --git a/package/tests/test_launchers/test_utils.py b/package/tests/test_launchers/test_utils.py index e9dead2f79..04425cfd09 100644 --- a/package/tests/test_launchers/test_utils.py +++ b/package/tests/test_launchers/test_utils.py @@ -1,3 +1,4 @@ +from pathlib import Path from unittest import mock from unittest.mock import Mock, call, patch @@ -7,6 +8,8 @@ from kedro_viz.constants import VIZ_DEPLOY_TIME_LIMIT from kedro_viz.launchers.utils import ( _check_viz_up, + _find_kedro_project, + _is_project, _start_browser, viz_deploy_progress_timer, ) @@ -69,3 +72,50 @@ def test_viz_deploy_progress_timer(capsys): for second in range(1, VIZ_DEPLOY_TIME_LIMIT + 1): expected_output = f"...Creating your build/deploy Kedro-Viz ({second}s)" assert expected_output in captured.out + + +class TestIsProject: + project_path = Path.cwd() + + def test_no_metadata_file(self, mocker): + mocker.patch.object(Path, "is_file", return_value=False) + + assert not _is_project(self.project_path) + + def test_toml_invalid_format(self, tmp_path): + """Test for loading context from an invalid path.""" + toml_path = tmp_path / "pyproject.toml" + toml_path.write_text("!!") # Invalid TOML + + assert not _is_project(tmp_path) + + def test_non_kedro_project(self, mocker): + mocker.patch.object(Path, "is_file", return_value=True) + mocker.patch.object(Path, "read_text", return_value="[tool]") + + assert not _is_project(self.project_path) + + def test_valid_toml_file(self, mocker): + mocker.patch.object(Path, "is_file", return_value=True) + pyproject_toml_payload = "[tool.kedro]" # \nproject_name = 'proj'" + mocker.patch.object(Path, "read_text", return_value=pyproject_toml_payload) + + assert _is_project(self.project_path) + + def test_toml_bad_encoding(self, mocker): + mocker.patch.object(Path, "is_file", return_value=True) + mocker.patch.object(Path, "read_text", side_effect=UnicodeDecodeError) + + assert not _is_project(self.project_path) + + +@pytest.mark.parametrize( + "project_dir, is_project_found, expected", + [ + ("/path/to/valid/project", True, Path("/path/to/valid/project")), + ("/path/to/nonexistent/project", False, None), + ], +) +def test_find_kedro_project(project_dir, is_project_found, expected, mocker): + mocker.patch("kedro_viz.launchers.utils._is_project", return_value=is_project_found) + assert _find_kedro_project(Path(project_dir)) == expected From dd3a9894abaf268d0b8fdb8311fbe44348f89e83 Mon Sep 17 00:00:00 2001 From: Huong Nguyen <32060364+Huongg@users.noreply.github.com> Date: Thu, 2 May 2024 12:20:30 +0100 Subject: [PATCH 07/22] Enhancement/doc-integration (#1874) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * setup vale.ini Signed-off-by: huongg * set linter check on docs or md file Signed-off-by: huongg * add styling for kedro-viz docs Signed-off-by: huongg * testing Signed-off-by: huongg * remove python v 3.8 as we don't support it anymore Signed-off-by: huongg * remove python 3.12 Signed-off-by: huongg * Try docs build with build-docs.sh Signed-off-by: Ankita Katiyar * testing the doc build Signed-off-by: huongg * update release note Signed-off-by: huongg * Update .github/styles/Kedro-viz/Spellings.yml Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * Update .github/styles/Kedro-viz/Spellings.yml Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * Update .github/workflows/docs-only-checks.yml Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * Update .github/workflows/docs-only-checks.yml Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * Update .readthedocs.yml Co-authored-by: Juan Luis Cano Rodríguez Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> * remove test line Signed-off-by: huongg * remove ignore-name file as its only used in kedro framework Signed-off-by: huongg --------- Signed-off-by: huongg Signed-off-by: Ankita Katiyar Signed-off-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> Co-authored-by: Ankita Katiyar Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Juan Luis Cano Rodríguez --- .github/styles/Kedro-viz/Spellings.yml | 5 + .github/styles/Kedro-viz/abbreviations.yml | 11 + .github/styles/Kedro-viz/gender.yml | 13 ++ .github/styles/Kedro-viz/headings.yml | 56 +++++ .github/styles/Kedro-viz/ignore.txt | 33 +++ .github/styles/Kedro-viz/inclusive.yml | 12 ++ .github/styles/Kedro-viz/links.yml | 12 ++ .github/styles/Kedro-viz/oxfordcomma.yml | 6 + .github/styles/Kedro-viz/pronouns.yml | 11 + .github/styles/Kedro-viz/sentencelength.yml | 11 + .github/styles/Kedro-viz/toowordy.yml | 218 ++++++++++++++++++++ .github/styles/Kedro-viz/ukspelling.yml | 28 +++ .github/styles/Kedro-viz/weaselwords.yml | 207 +++++++++++++++++++ .github/styles/Kedro-viz/words.yml | 53 +++++ .github/workflows/docs-language-linter.yml | 17 ++ .github/workflows/docs-only-checks.yml | 26 +++ .readthedocs.yml | 8 +- .vale.ini | 7 + RELEASE.md | 1 + docs/Makefile | 20 -- docs/build-docs.sh | 14 ++ docs/make.bat | 35 ---- 22 files changed, 748 insertions(+), 56 deletions(-) create mode 100644 .github/styles/Kedro-viz/Spellings.yml create mode 100644 .github/styles/Kedro-viz/abbreviations.yml create mode 100644 .github/styles/Kedro-viz/gender.yml create mode 100644 .github/styles/Kedro-viz/headings.yml create mode 100644 .github/styles/Kedro-viz/ignore.txt create mode 100644 .github/styles/Kedro-viz/inclusive.yml create mode 100644 .github/styles/Kedro-viz/links.yml create mode 100644 .github/styles/Kedro-viz/oxfordcomma.yml create mode 100644 .github/styles/Kedro-viz/pronouns.yml create mode 100644 .github/styles/Kedro-viz/sentencelength.yml create mode 100644 .github/styles/Kedro-viz/toowordy.yml create mode 100644 .github/styles/Kedro-viz/ukspelling.yml create mode 100644 .github/styles/Kedro-viz/weaselwords.yml create mode 100644 .github/styles/Kedro-viz/words.yml create mode 100644 .github/workflows/docs-language-linter.yml create mode 100644 .github/workflows/docs-only-checks.yml create mode 100644 .vale.ini delete mode 100644 docs/Makefile create mode 100644 docs/build-docs.sh delete mode 100644 docs/make.bat diff --git a/.github/styles/Kedro-viz/Spellings.yml b/.github/styles/Kedro-viz/Spellings.yml new file mode 100644 index 0000000000..6a1c9c9b64 --- /dev/null +++ b/.github/styles/Kedro-viz/Spellings.yml @@ -0,0 +1,5 @@ +extends: spelling +message: "Did you really mean '%s'?" +level: warning +ignore: + - Kedro-viz/ignore.txt \ No newline at end of file diff --git a/.github/styles/Kedro-viz/abbreviations.yml b/.github/styles/Kedro-viz/abbreviations.yml new file mode 100644 index 0000000000..986b14c67b --- /dev/null +++ b/.github/styles/Kedro-viz/abbreviations.yml @@ -0,0 +1,11 @@ +extends: substitution +message: "Use '%s' instead of abbreviations like '%s'." +ignorecase: true +level: warning +nonword: true +action: + name: replace +swap: + '\b(?:eg|e\.g\.|eg\.)[\s,]': for example + '\b(?:ie|i\.e\.|ie\.)[\s,]': that is + '\b(?:etc)[\s\n,.]': and more \ No newline at end of file diff --git a/.github/styles/Kedro-viz/gender.yml b/.github/styles/Kedro-viz/gender.yml new file mode 100644 index 0000000000..2e00326f0f --- /dev/null +++ b/.github/styles/Kedro-viz/gender.yml @@ -0,0 +1,13 @@ +extends: existence +message: "Use a gender-neutral pronoun instead of '%s'." +level: error +ignorecase: true +tokens: + - he/she + - s/he + - \(s\)he + - \bhe\b + - \bhim\b + - \bhis\b + - \bshe\b + - \bher\b \ No newline at end of file diff --git a/.github/styles/Kedro-viz/headings.yml b/.github/styles/Kedro-viz/headings.yml new file mode 100644 index 0000000000..e250d1df25 --- /dev/null +++ b/.github/styles/Kedro-viz/headings.yml @@ -0,0 +1,56 @@ +extends: capitalization +message: "'%s' should use sentence-style capitalization." +level: warning +scope: heading +match: $sentence +exceptions: + - Kedro + - Kedro-Viz + - Airflow + - Amazon + - Amazon Web Services + - AWS Step Functions + - AWS Systems Manager + - Azure + - Azure App Service + - Azure App Service Plan + - Azure Blob Storage + - Azure Event Hub + - CI/CD + - DataCatalog + - Data Catalog + - Docker + - Docker Compose + - Docker Swarm + - Dockerfile + - GitHub + - GitHub Actions + - Google + - Google Analytics + - Google Cloud + - Google Cloud Functions + - GraphQL + - Hook + - Hooks + - IDs + - Jenkins + - JFrog + - JFrog Artifactory + - Jira + - Kafka + - Kubernetes + - Kubernetes Engine + - Kubernetes Pod + - Kubernetes Service + - Lambda + - Linux + - MySQL + - Python + - QuantumBlack + - QuantumBlack Labs + - Red Hat + - Redis + - Slack + - Ubuntu + - Unix + - URLs \ No newline at end of file diff --git a/.github/styles/Kedro-viz/ignore.txt b/.github/styles/Kedro-viz/ignore.txt new file mode 100644 index 0000000000..21d52adda3 --- /dev/null +++ b/.github/styles/Kedro-viz/ignore.txt @@ -0,0 +1,33 @@ +Kedro +kedro +Kedro's +kedro's +Kubeflow +Databricks +Conda +conda +Cookiecutter +config +fsspec +Kaggle +namespace +namespaces +namespaced +regressors +repo +Repo +dbx +MLflow +csv +yaml +matplotlib +Matplotlib +IPython +APIs +networkx +Plotly +Pylint +SQLAlchemy +Astro +Xebia +pytest \ No newline at end of file diff --git a/.github/styles/Kedro-viz/inclusive.yml b/.github/styles/Kedro-viz/inclusive.yml new file mode 100644 index 0000000000..aa79a3f2c1 --- /dev/null +++ b/.github/styles/Kedro-viz/inclusive.yml @@ -0,0 +1,12 @@ +extends: substitution +message: "Use '%s' instead of '%s'." +ignorecase: true +level: error +action: + name: replace +swap: + # bad: good + 'black ?list': 'disallow list|exclude list' + 'master': primary + 'slave': secondary + 'white ?list': 'allow list|include list' \ No newline at end of file diff --git a/.github/styles/Kedro-viz/links.yml b/.github/styles/Kedro-viz/links.yml new file mode 100644 index 0000000000..aa79a3f2c1 --- /dev/null +++ b/.github/styles/Kedro-viz/links.yml @@ -0,0 +1,12 @@ +extends: substitution +message: "Use '%s' instead of '%s'." +ignorecase: true +level: error +action: + name: replace +swap: + # bad: good + 'black ?list': 'disallow list|exclude list' + 'master': primary + 'slave': secondary + 'white ?list': 'allow list|include list' \ No newline at end of file diff --git a/.github/styles/Kedro-viz/oxfordcomma.yml b/.github/styles/Kedro-viz/oxfordcomma.yml new file mode 100644 index 0000000000..5b9c7dbff8 --- /dev/null +++ b/.github/styles/Kedro-viz/oxfordcomma.yml @@ -0,0 +1,6 @@ +extends: existence +message: "Use the Oxford comma in '%s'." +scope: sentence +level: suggestion +tokens: + - '(?:[^,]+,){1,}\s\w+\s(?:and|or)' \ No newline at end of file diff --git a/.github/styles/Kedro-viz/pronouns.yml b/.github/styles/Kedro-viz/pronouns.yml new file mode 100644 index 0000000000..d6ad378172 --- /dev/null +++ b/.github/styles/Kedro-viz/pronouns.yml @@ -0,0 +1,11 @@ +extends: existence +message: "Avoid first-person singular pronouns such as '%s'." +level: warning +nonword: true +tokens: + - (?<=^|\s)I(?=\s) + - (?<=^|\s)I,(?=\s) + - \bI'm\b + - (?<=\s)[Mm]e\b + - (?<=\s)[Mm]y\b + - (?<=\s)[Mm]ine\b \ No newline at end of file diff --git a/.github/styles/Kedro-viz/sentencelength.yml b/.github/styles/Kedro-viz/sentencelength.yml new file mode 100644 index 0000000000..4a8042ea13 --- /dev/null +++ b/.github/styles/Kedro-viz/sentencelength.yml @@ -0,0 +1,11 @@ +extends: occurrence +message: "Try to keep your sentence length to 30 words or fewer." +level: suggestion +# Here, we're counting the number of words +# in a sentence. +# +# If there are more than 30, we'll flag it. +scope: sentence +ignorecase: false +max: 30 +token: (\w+) \ No newline at end of file diff --git a/.github/styles/Kedro-viz/toowordy.yml b/.github/styles/Kedro-viz/toowordy.yml new file mode 100644 index 0000000000..320bfaf3fe --- /dev/null +++ b/.github/styles/Kedro-viz/toowordy.yml @@ -0,0 +1,218 @@ +# Write Good's "Too wordy" rule https://github.com/testthedocs/vale-styles/blob/master/write-good/TooWordy.yml +extends: existence +message: "'%s' is too wordy" +ignorecase: true +level: warning +tokens: + - a number of + - abundance + - accede to + - accelerate + - accentuate + - accompany + - accomplish + - accorded + - accrue + - acquiesce + - acquire + - adjacent to + - adjustment + - admissible + - advantageous + - adversely impact + - advise + - aforementioned + - aggregate + - aircraft + - all of + - all things considered + - alleviate + - allocate + - along the lines of + - already existing + - alternatively + - amazing + - ameliorate + - anticipate + - apparent + - appreciable + - as a matter of fact + - as a means of + - as far as I'm concerned + - as of yet + - as to + - as yet + - ascertain + - assistance + - at the present time + - at this time + - attain + - attributable to + - because of the fact that + - belated + - benefit from + - bestow + - by means of + - by virtue of the fact that + - by virtue of + - cease + - close proximity + - commence + - comply with + - concerning + - consequently + - consolidate + - constitutes + - demonstrate + - depart + - designate + - discontinue + - due to the fact that + - each and every + - economical + - eliminate + - elucidate + - employ + - endeavor + - enumerate + - equitable + - equivalent + - evaluate + - evidenced + - exclusively + - expedite + - expend + - expiration + - facilitate + - factual evidence + - feasible + - finalise + - first and foremost + - for all intents and purposes + - for the most part + - for the purpose of + - forfeit + - formulate + - have a tendency to + - honest truth + - however + - if and when + - impacted + - implement + - in a manner of speaking + - in a timely manner + - in a very real sense + - in accordance with + - in addition + - in all likelihood + - in an effort to + - in between + - in excess of + - in lieu of + - in light of the fact that + - in many cases + - in my opinion + - in order to + - in regard to + - in some instances + - in terms of + - in the case of + - in the event that + - in the final analysis + - in the nature of + - in the near future + - in the process of + - inception + - incumbent upon + - indicate + - indication + - initiate + - irregardless + - is applicable to + - is authorised to + - is responsible for + - it is essential + - it seems that + - it was + - magnitude + - maximum + - methodology + - minimise + - modify + - monitor + - multiple + - necessitate + - nevertheless + - not certain + - not many + - not often + - not unless + - not unlike + - notwithstanding + - null and void + - numerous + - objective + - obligate + - obtain + - on the contrary + - on the other hand + - one particular + - optimum + - overall + - owing to the fact that + - participate + - particulars + - pass away + - pertaining to + - point in time + - portion + - possess + - preclude + - previously + - prior to + - prioritise + - procure + - proficiency + - provided that + - purchase + - put simply + - readily apparent + - refer back + - regarding + - relocate + - remainder + - remuneration + - requirement + - reside + - residence + - retain + - satisfy + - shall + - should you wish + - similar to + - solicit + - span across + - strategise + - subsequent + - substantial + - successfully complete + - sufficient + - terminate + - the month of + - the point I am trying to make + - therefore + - time period + - took advantage of + - transmit + - transpire + - type of + - until such time as + - utilisation + - utilise + - validate + - various different + - what I mean to say is + - whether or not + - with respect to + - with the exception of + - witnessed \ No newline at end of file diff --git a/.github/styles/Kedro-viz/ukspelling.yml b/.github/styles/Kedro-viz/ukspelling.yml new file mode 100644 index 0000000000..10c8439ca8 --- /dev/null +++ b/.github/styles/Kedro-viz/ukspelling.yml @@ -0,0 +1,28 @@ +extends: existence +message: "In general, use UK English spelling instead of '%s'." +link: 'https://github.com/kedro-org/kedro/wiki/Kedro-documentation-style-guide' +ignorecase: true +level: warning +tokens: + - '(?:\w+)zation' + - '(?:\w+)izing' + - '(?:\w+)izer' + - '(?:\w+)ized' + - '(?:\w+)ize' + - '(?:\w+)log' + - '(?:\w+)lor' + - '(?:\w+)lyze' +exceptions: + - backlog + - blog + - capsize + - catalog + - Catalog + - DataCatalog + - dialog + - log + - maize + - prize + - seize + - size + - tailor \ No newline at end of file diff --git a/.github/styles/Kedro-viz/weaselwords.yml b/.github/styles/Kedro-viz/weaselwords.yml new file mode 100644 index 0000000000..bb8001fbc1 --- /dev/null +++ b/.github/styles/Kedro-viz/weaselwords.yml @@ -0,0 +1,207 @@ +# Write Good's Weasel Words rule https://github.com/testthedocs/vale-styles/blob/master/write-good/Weasel.yml +extends: existence +message: "'%s' is a weasel word!" +ignorecase: true +level: warning +tokens: + - absolutely + - accidentally + - additionally + - allegedly + - alternatively + - angrily + - anxiously + - approximately + - awkwardly + - badly + - barely + - beautifully + - blindly + - boldly + - bravely + - brightly + - briskly + - bristly + - bubbly + - busily + - calmly + - carefully + - carelessly + - cautiously + - cheerfully + - clearly + - closely + - coldly + - completely + - consequently + - correctly + - courageously + - crinkly + - cruelly + - crumbly + - cuddly + - currently + - daily + - daringly + - deadly + - definitely + - deliberately + - doubtfully + - dumbly + - eagerly + - easily + - elegantly + - enormously + - enthusiastically + - equally + - especially + - eventually + - exactly + - exceedingly + - exclusively + - extremely + - fairly + - faithfully + - fatally + - fiercely + - finally + - fondly + - few + - foolishly + - fortunately + - frankly + - frantically + - generously + - gently + - giggly + - gladly + - gracefully + - greedily + - happily + - hardly + - hastily + - healthily + - heartily + - helpfully + - honestly + - hourly + - hungrily + - hurriedly + - immediately + - impatiently + - inadequately + - ingeniously + - innocently + - inquisitively + - interestingly + - irritably + - jiggly + - joyously + - justly + - kindly + - largely + - lately + - lazily + - likely + - literally + - lonely + - loosely + - loudly + - loudly + - luckily + - madly + - many + - mentally + - mildly + - monthly + - mortally + - mostly + - mysteriously + - neatly + - nervously + - nightly + - noisily + - normally + - obediently + - occasionally + - only + - openly + - painfully + - particularly + - patiently + - perfectly + - politely + - poorly + - powerfully + - presumably + - previously + - promptly + - punctually + - quarterly + - quickly + - quietly + - rapidly + - rarely + - really + - recently + - recklessly + - regularly + - remarkably + - relatively + - reluctantly + - repeatedly + - rightfully + - roughly + - rudely + - sadly + - safely + - selfishly + - sensibly + - seriously + - sharply + - shortly + - shyly + - significantly + - silently + - simply + - sleepily + - slowly + - smartly + - smelly + - smoothly + - softly + - solemnly + - sparkly + - speedily + - stealthily + - sternly + - stupidly + - substantially + - successfully + - suddenly + - surprisingly + - suspiciously + - swiftly + - tenderly + - tensely + - thoughtfully + - tightly + - timely + - truthfully + - unexpectedly + - unfortunately + - usually + - very + - victoriously + - violently + - vivaciously + - warmly + - waverly + - weakly + - wearily + - weekly + - wildly + - wisely + - worldly + - wrinkly + - yearly \ No newline at end of file diff --git a/.github/styles/Kedro-viz/words.yml b/.github/styles/Kedro-viz/words.yml new file mode 100644 index 0000000000..0a9a1d3433 --- /dev/null +++ b/.github/styles/Kedro-viz/words.yml @@ -0,0 +1,53 @@ +extends: substitution +message: "Use '%s' instead of '%s'." +ignorecase: false +level: warning +action: + name: replace +swap: + # bad: good + 'acknowledgement': 'acknowledgment' + 'auto-complete': 'autocomplete' + 'a number of': 'few|several|many' + 'and/or': 'and|or|either or' + 'back end': 'backend' + 'bear in mind': 'keep in mind' + 'culprit': 'cause' + 'data set': 'dataset' + 'drill down|drilling down|drill into|drilling into': 'examine|investigate|analyze' + 'figure out': 'determine' + 'fine tune|fine-tune': 'customize|optimize|refine' + 'for the most part': 'generally|usually' + 'front end': 'frontend' + 'highly|very': '' + 'hit': 'click|select' + 'in order to': 'to' + 'keep in mind': 'consider' + 'left up to': 'determined by' + 'leverage': 'use' + 'multi-alert': 'multi alert' + 'Note that': '**Note**:' + 'obviously|obvious': '' + 'on the fly': 'real-time|real time' + 'once': 'after' + 'play a hand': 'influence' + 'please|just': '' + 'easily|easy': '' + 'quickly|quick': '' + 'screen board': 'screenboard' + 'simply|simple': '' + 'stand for': 'represents|means' + 'reenable': 're-enable' + 'run time': 'runtime' + 'refer to|visit': 'see|read|follow' + 'time board': 'timeboard' + 'time series': 'timeseries' + 'toplist': 'top list' + 'tradeoff': 'trade-off' + 'turnkey': 'ready to use' + 'under the hood': '' + 'utilize': 'use' + 'via': 'with|through' + 'visit': 'see|read' + 'webserver': 'web server' + 'web site': 'website' \ No newline at end of file diff --git a/.github/workflows/docs-language-linter.yml b/.github/workflows/docs-language-linter.yml new file mode 100644 index 0000000000..20980c3f49 --- /dev/null +++ b/.github/workflows/docs-language-linter.yml @@ -0,0 +1,17 @@ +name: Language Linter for Kedro-viz Docs +on: + workflow_dispatch: + pull_request: + paths: + - "docs/**" + - '**.md' + +jobs: + vale: + name: runner / vale + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: errata-ai/vale-action@reviewdog + with: + reporter: github-pr-check \ No newline at end of file diff --git a/.github/workflows/docs-only-checks.yml b/.github/workflows/docs-only-checks.yml new file mode 100644 index 0000000000..bd09e387e8 --- /dev/null +++ b/.github/workflows/docs-only-checks.yml @@ -0,0 +1,26 @@ +name: Run linter on Kedro-viz Docs + +on: + push: + branches: + - main + paths: + - "docs/**" + - '**.md' + pull_request: + branches: + - main + paths: + - "docs/**" + - '**.md' + +jobs: + lint: + strategy: + matrix: + os: [ ubuntu-latest ] + python-version: ["3.9", "3.10", "3.11"] + uses: ./.github/workflows/lint.yml + with: + os: ${{ matrix.os }} + python-version: ${{ matrix.python-version }} \ No newline at end of file diff --git a/.readthedocs.yml b/.readthedocs.yml index 57e08a91fc..4faaef3ab6 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -8,7 +8,13 @@ build: os: ubuntu-22.04 tools: python: "3.9" - nodejs: "19" + jobs: + post_checkout: + - git fetch --unshallow || true + pre_build: + - pip freeze + - python -m sphinx -WETan -j auto -D language=en -b linkcheck -d _build/doctrees docs/source _build/linkcheck + sphinx: builder: html diff --git a/.vale.ini b/.vale.ini new file mode 100644 index 0000000000..5f4e3c3a7c --- /dev/null +++ b/.vale.ini @@ -0,0 +1,7 @@ +StylesPath = .github/styles + +MinAlertLevel = suggestion + +[*.md] +BasedOnStyles = Vale, Kedro-viz +Vale.Spelling = NO \ No newline at end of file diff --git a/RELEASE.md b/RELEASE.md index 2010bdf855..1fa28a68d5 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -13,6 +13,7 @@ Please follow the established format: - Upgrade the gitpod workspace-full to a newer version which includes both Node 18 and Python 3.11.5. (#1862) - Refactor backend integration with Kedro by replacing bootstrap_project with configure_project. (#1796) +- Enhance kedro-viz doc integration. (#1874) - Fix Kedro-Viz waiting for valid Kedro project. (#1871) # Release 9.0.0 diff --git a/docs/Makefile b/docs/Makefile deleted file mode 100644 index d0c3cbf102..0000000000 --- a/docs/Makefile +++ /dev/null @@ -1,20 +0,0 @@ -# Minimal makefile for Sphinx documentation -# - -# You can set these variables from the command line, and also -# from the environment for the first two. -SPHINXOPTS ?= -SPHINXBUILD ?= sphinx-build -SOURCEDIR = source -BUILDDIR = build - -# Put it first so that "make" without argument is like "make help". -help: - @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - -.PHONY: help Makefile - -# Catch-all target: route all unknown targets to Sphinx using the new -# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). -%: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/build-docs.sh b/docs/build-docs.sh new file mode 100644 index 0000000000..89cecc80cb --- /dev/null +++ b/docs/build-docs.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash + +set -e + +# Exit script if you try to use an uninitialized variable. +set -o nounset + +action=$1 + +if [ "$action" == "linkcheck" ]; then + sphinx-build -WETan -j auto -D language=en -b linkcheck -d docs/build/doctrees docs/source docs/build/linkcheck +elif [ "$action" == "docs" ]; then + sphinx-build -WETa --keep-going -j auto -D language=en -b html -d docs/build/doctrees docs/source docs/build/html +fi diff --git a/docs/make.bat b/docs/make.bat deleted file mode 100644 index 747ffb7b30..0000000000 --- a/docs/make.bat +++ /dev/null @@ -1,35 +0,0 @@ -@ECHO OFF - -pushd %~dp0 - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set SOURCEDIR=source -set BUILDDIR=build - -%SPHINXBUILD% >NUL 2>NUL -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.https://www.sphinx-doc.org/ - exit /b 1 -) - -if "%1" == "" goto help - -%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% -goto end - -:help -%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% - -:end -popd From 9e661f6140e77c77dbced96107aef8d9abdf6ca7 Mon Sep 17 00:00:00 2001 From: Jitendra Gundaniya <38945204+jitu5@users.noreply.github.com> Date: Tue, 7 May 2024 10:18:59 +0100 Subject: [PATCH 08/22] Inherit tags from nodes for modular pipelines (#1878) * Inherit tags from nodes for modular pipelines Signed-off-by: Jitendra Gundaniya * Lint fix Signed-off-by: <> * Unit test fix Signed-off-by: <> * Unit test fix Signed-off-by: <> * Unit test fix Signed-off-by: <> * unit test fix Signed-off-by: <> * tags removed Signed-off-by: Jitendra Gundaniya * done * pipeline tree * new approach for tags Signed-off-by: Jitendra Gundaniya * lint fix Signed-off-by: Jitendra Gundaniya * Lint fix Signed-off-by: Jitendra Gundaniya * lint fix Signed-off-by: Jitendra Gundaniya * None check for pipeline_id added Signed-off-by: Jitendra Gundaniya * Comment removed Signed-off-by: Jitendra Gundaniya * New approach for modular pipeline tag * Update test_modular_pipelines.py Signed-off-by: Jitendra Gundaniya <38945204+jitu5@users.noreply.github.com> --------- Signed-off-by: Jitendra Gundaniya Signed-off-by: <> Signed-off-by: Jitendra Gundaniya <38945204+jitu5@users.noreply.github.com> Co-authored-by: rashidakanchwala --- .../data_access/repositories/modular_pipelines.py | 3 +++ package/kedro_viz/services/modular_pipelines.py | 2 ++ package/tests/test_api/test_rest/test_responses.py | 10 +++++----- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/package/kedro_viz/data_access/repositories/modular_pipelines.py b/package/kedro_viz/data_access/repositories/modular_pipelines.py index b887d020b8..b32b62ecb8 100644 --- a/package/kedro_viz/data_access/repositories/modular_pipelines.py +++ b/package/kedro_viz/data_access/repositories/modular_pipelines.py @@ -1,5 +1,6 @@ """`kedro_viz.data_access.repositories.modular_pipelines` defines repository to centralise access to modular pipelines data.""" + from typing import Dict, Optional, Union from kedro_viz.constants import ROOT_MODULAR_PIPELINE_ID @@ -215,6 +216,8 @@ def extract_from_node(self, node: GraphNode) -> Optional[str]: # so does the modular pipeline. modular_pipeline.pipelines.update(node.pipelines) + modular_pipeline.tags.update(node.tags) + # Since we extract the modular pipeline from the node's namespace, # the node is by definition a child of the modular pipeline. self.add_child( diff --git a/package/kedro_viz/services/modular_pipelines.py b/package/kedro_viz/services/modular_pipelines.py index e15f381000..bc568e99ff 100644 --- a/package/kedro_viz/services/modular_pipelines.py +++ b/package/kedro_viz/services/modular_pipelines.py @@ -1,6 +1,7 @@ """`kedro_viz.services.modular_pipelines` defines modular pipelines-related business logic. The service layer consist of pure functions operating on domain models. """ + from typing import Dict from kedro_viz.constants import ROOT_MODULAR_PIPELINE_ID @@ -78,6 +79,7 @@ def expand_tree( ) expanded_tree[parent_id].pipelines.update(modular_pipeline_node.pipelines) + expanded_tree[parent_id].tags.update(modular_pipeline_node.tags) expanded_tree[parent_id].children.add( ModularPipelineChild( id=f"{parent_id}.{chunks[i]}", diff --git a/package/tests/test_api/test_rest/test_responses.py b/package/tests/test_api/test_rest/test_responses.py index 5e5d32a4c2..1842ee51f1 100644 --- a/package/tests/test_api/test_rest/test_responses.py +++ b/package/tests/test_api/test_rest/test_responses.py @@ -184,7 +184,7 @@ def assert_example_data(response_data): { "id": "uk.data_processing", "name": "uk.data_processing", - "tags": [], + "tags": ["split"], "pipelines": ["__default__"], "type": "modularPipeline", "modular_pipelines": None, @@ -195,7 +195,7 @@ def assert_example_data(response_data): { "id": "uk.data_science", "name": "uk.data_science", - "tags": [], + "tags": ["train"], "pipelines": ["__default__"], "type": "modularPipeline", "modular_pipelines": None, @@ -206,7 +206,7 @@ def assert_example_data(response_data): { "id": "uk", "name": "uk", - "tags": [], + "tags": ["split", "train"], "pipelines": ["__default__"], "type": "modularPipeline", "modular_pipelines": None, @@ -731,7 +731,7 @@ def test_get_pipeline(self, client): { "id": "uk", "name": "uk", - "tags": [], + "tags": ["train"], "pipelines": ["data_science"], "type": "modularPipeline", "modular_pipelines": None, @@ -742,7 +742,7 @@ def test_get_pipeline(self, client): { "id": "uk.data_science", "name": "uk.data_science", - "tags": [], + "tags": ["train"], "pipelines": ["data_science"], "type": "modularPipeline", "modular_pipelines": None, From cb100c7e5000a7d16e0e06e0c8ac5749abeba74d Mon Sep 17 00:00:00 2001 From: Huong Nguyen <32060364+Huongg@users.noreply.github.com> Date: Fri, 10 May 2024 11:55:15 +0100 Subject: [PATCH 09/22] Docs/use kedro sphinx theme (#1898) * remove layout.html Signed-off-by: huongg * refers to kedro-sphinx-theme in setup Signed-off-by: huongg * refers to kedro-sphinx-theme in confy.py Signed-off-by: huongg * update release note Signed-off-by: huongg * update conf.py in doc/source Signed-off-by: huongg * update setup.py in demo project Signed-off-by: huongg * revert changes for demo-project/docs/source Signed-off-by: huongg --------- Signed-off-by: huongg --- RELEASE.md | 1 + docs/source/_templates/layout.html | 76 ------------------------------ docs/source/conf.py | 2 +- package/setup.py | 6 +-- 4 files changed, 3 insertions(+), 82 deletions(-) delete mode 100644 docs/source/_templates/layout.html diff --git a/RELEASE.md b/RELEASE.md index 1fa28a68d5..3d2420e1bb 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -15,6 +15,7 @@ Please follow the established format: - Refactor backend integration with Kedro by replacing bootstrap_project with configure_project. (#1796) - Enhance kedro-viz doc integration. (#1874) - Fix Kedro-Viz waiting for valid Kedro project. (#1871) +- Enhance Kedro-Viz documentation by using Kedro-sphinx-theme. (#1898) # Release 9.0.0 diff --git a/docs/source/_templates/layout.html b/docs/source/_templates/layout.html deleted file mode 100644 index 5e2cd48b45..0000000000 --- a/docs/source/_templates/layout.html +++ /dev/null @@ -1,76 +0,0 @@ -{% extends "!layout.html" %} {%- block extrabody %} - -
- - -
- Kedro - Kedro-Viz - Kedro-Datasets -
- - {%- include "searchbox.html" %} -
- -{% endblock %} {%- block extrahead %} - - - -{% endblock %} diff --git a/docs/source/conf.py b/docs/source/conf.py index be246d7c16..1d3edbd09d 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -62,7 +62,7 @@ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = "sphinx_rtd_theme" +html_theme = "kedro-sphinx-theme" # Theme options are theme-specific and customise the look and feel of a theme # further. For a list of options available for each theme, see the diff --git a/package/setup.py b/package/setup.py index d2257d6f6d..e04cda700f 100644 --- a/package/setup.py +++ b/package/setup.py @@ -52,11 +52,7 @@ }, extras_require={ "docs": [ - "sphinx>=5.3,<7.3", - "sphinx_copybutton==0.5.2", - "sphinx-notfound-page", - "sphinx_rtd_theme==1.3.0", - "myst-parser>=1.0,<2.1", + "kedro-sphinx-theme==2024.4.0", ], "aws": ["s3fs>=2021.4"], "azure": ["adlfs>=2021.4"], From aabb3a2063b416d35d1906e0b9ff2f0307dd1ce0 Mon Sep 17 00:00:00 2001 From: Ravi Kumar Pilla Date: Wed, 15 May 2024 09:39:30 -0500 Subject: [PATCH 10/22] Fix for schema change in strawberry-graphql JSON scalar (#1903) * merge main from remote Signed-off-by: ravi-kumar-pilla * hot fix for strawberry graphql schema change Signed-off-by: ravi-kumar-pilla * update release note and file permissions Signed-off-by: ravi-kumar-pilla --------- Signed-off-by: ravi-kumar-pilla --- RELEASE.md | 1 + src/apollo/schema.graphql | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index 3d2420e1bb..3ab55e08b4 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -16,6 +16,7 @@ Please follow the established format: - Enhance kedro-viz doc integration. (#1874) - Fix Kedro-Viz waiting for valid Kedro project. (#1871) - Enhance Kedro-Viz documentation by using Kedro-sphinx-theme. (#1898) +- Fix for schema change in strawberry-graphql JSON scalar. (#1903) # Release 9.0.0 diff --git a/src/apollo/schema.graphql b/src/apollo/schema.graphql index e48274ebb0..2fd62c4204 100644 --- a/src/apollo/schema.graphql +++ b/src/apollo/schema.graphql @@ -1,7 +1,7 @@ """ -The `JSON` scalar type represents JSON values as specified by [ECMA-404](http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf). +The `JSON` scalar type represents JSON values as specified by [ECMA-404](https://ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf). """ -scalar JSON @specifiedBy(url: "http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf") +scalar JSON @specifiedBy(url: "https://ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf") """Metric data""" type MetricPlotDataset { From a8f57adb304bc8abda5922ce6d121cad1de80067 Mon Sep 17 00:00:00 2001 From: Ravi Kumar Pilla Date: Wed, 15 May 2024 13:59:51 -0500 Subject: [PATCH 11/22] Fix messaging level when package compatibility is not satisfied (#1904) * merge main from remote Signed-off-by: ravi-kumar-pilla * initial draft Signed-off-by: ravi-kumar-pilla * revert permissions Signed-off-by: ravi-kumar-pilla * update release note Signed-off-by: ravi-kumar-pilla --------- Signed-off-by: ravi-kumar-pilla --- RELEASE.md | 1 + package/kedro_viz/api/rest/responses.py | 9 +-- package/kedro_viz/constants.py | 12 +++- .../test_api/test_rest/test_responses.py | 61 ++++++++++++++----- 4 files changed, 63 insertions(+), 20 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index 3ab55e08b4..c3cd6e3edf 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -17,6 +17,7 @@ Please follow the established format: - Fix Kedro-Viz waiting for valid Kedro project. (#1871) - Enhance Kedro-Viz documentation by using Kedro-sphinx-theme. (#1898) - Fix for schema change in strawberry-graphql JSON scalar. (#1903) +- Fix messaging level when package compatibility is not satisfied. (#1904) # Release 9.0.0 diff --git a/package/kedro_viz/api/rest/responses.py b/package/kedro_viz/api/rest/responses.py index 13aaa05a6d..7850d091ad 100644 --- a/package/kedro_viz/api/rest/responses.py +++ b/package/kedro_viz/api/rest/responses.py @@ -372,16 +372,17 @@ def get_selected_pipeline_response(registered_pipeline_id: str): def get_package_compatibilities_response( - package_requirements: Dict[str, str], + package_requirements: Dict[str, Dict[str, str]], ) -> List[PackageCompatibilityAPIResponse]: """API response for `/api/package_compatibility`.""" package_requirements_response = [] - for package_name, compatible_version in package_requirements.items(): + for package_name, package_info in package_requirements.items(): + compatible_version = package_info["min_compatible_version"] try: package_version = get_package_version(package_name) - except PackageNotFoundError as exc: - logger.exception("Failed to get package version. Error: %s", str(exc)) + except PackageNotFoundError: + logger.warning(package_info["warning_message"]) package_version = "0.0.0" is_compatible = packaging.version.parse( diff --git a/package/kedro_viz/constants.py b/package/kedro_viz/constants.py index 9afb7eddd4..620fa951e3 100644 --- a/package/kedro_viz/constants.py +++ b/package/kedro_viz/constants.py @@ -14,4 +14,14 @@ SHAREABLEVIZ_SUPPORTED_PLATFORMS = ["aws", "azure", "gcp"] -PACKAGE_REQUIREMENTS = {"fsspec": "2023.9.0", "kedro-datasets": "2.1.0"} +PACKAGE_REQUIREMENTS = { + "fsspec": { + "min_compatible_version": "2023.9.0", + "warning_message": "Publish and share Kedro-Viz requires fsspec >= 2023.9.0", + }, + "kedro-datasets": { + "min_compatible_version": "2.1.0", + "warning_message": "Experiment Tracking is exclusively supported " + "for users with kedro-datasets >= 2.1.0", + }, +} diff --git a/package/tests/test_api/test_rest/test_responses.py b/package/tests/test_api/test_rest/test_responses.py index 1842ee51f1..79c05092e4 100644 --- a/package/tests/test_api/test_rest/test_responses.py +++ b/package/tests/test_api/test_rest/test_responses.py @@ -1,4 +1,5 @@ # pylint: disable=too-many-lines +import logging import operator from pathlib import Path from typing import Any, Dict, Iterable, List @@ -7,7 +8,6 @@ import pytest from fastapi.testclient import TestClient -from importlib_metadata import PackageNotFoundError from kedro_viz.api import apps from kedro_viz.api.rest.responses import ( @@ -829,10 +829,30 @@ class TestPackageCompatibilities: @pytest.mark.parametrize( "package_name, package_version, package_requirements, expected_compatibility_response", [ - ("fsspec", "2023.9.1", {"fsspec": "2023.0.0"}, True), - ("fsspec", "2023.9.1", {"fsspec": "2024.0.0"}, False), - ("kedro-datasets", "2.1.0", {"kedro-datasets": "2.1.0"}, True), - ("kedro-datasets", "1.8.0", {"kedro-datasets": "2.1.0"}, False), + ( + "fsspec", + "2023.9.1", + {"fsspec": {"min_compatible_version": "2023.0.0"}}, + True, + ), + ( + "fsspec", + "2023.9.1", + {"fsspec": {"min_compatible_version": "2024.0.0"}}, + False, + ), + ( + "kedro-datasets", + "2.1.0", + {"kedro-datasets": {"min_compatible_version": "2.1.0"}}, + True, + ), + ( + "kedro-datasets", + "1.8.0", + {"kedro-datasets": {"min_compatible_version": "2.1.0"}}, + False, + ), ], ) def test_get_package_compatibilities_response( @@ -854,16 +874,27 @@ def test_get_package_compatibilities_response( assert package_response.package_version == package_version assert package_response.is_compatible is expected_compatibility_response - def test_get_package_compatibilities_exception_response( - self, - mocker, - ): - mocker.patch( - "kedro_viz.api.rest.responses.get_package_compatibilities_response", - side_effect=PackageNotFoundError("random-package"), - ) - package_name = "random-package" - response = get_package_compatibilities_response({package_name: "1.0.0"}) + def test_get_package_compatibilities_exception_response(self, caplog): + mock_package_requirement = { + "random-package": { + "min_compatible_version": "1.0.0", + "warning_message": "random-package is not available", + } + } + + with caplog.at_level(logging.WARNING): + response = get_package_compatibilities_response(mock_package_requirement) + + assert len(caplog.records) == 1 + + record = caplog.records[0] + + assert record.levelname == "WARNING" + assert ( + mock_package_requirement["random-package"]["warning_message"] + in record.message + ) + expected_response = PackageCompatibilityAPIResponse( package_name="random-package", package_version="0.0.0", is_compatible=False ) From c9142cd34867e9e0d3be2a09a3a99f15d4027d1b Mon Sep 17 00:00:00 2001 From: Ravi Kumar Pilla Date: Thu, 16 May 2024 11:07:42 -0500 Subject: [PATCH 12/22] Fix build failure due to incompatible seaborn and matplotlib versions (#1910) * merge main from remote Signed-off-by: ravi-kumar-pilla * fix build issue due to matplotlib new update Signed-off-by: ravi-kumar-pilla * update release note Signed-off-by: ravi-kumar-pilla * update release note Signed-off-by: ravi-kumar-pilla --------- Signed-off-by: ravi-kumar-pilla --- demo-project/src/demo_project/requirements.in | 2 +- demo-project/src/docker_requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/demo-project/src/demo_project/requirements.in b/demo-project/src/demo_project/requirements.in index af486a91a6..dff89fa8d9 100644 --- a/demo-project/src/demo_project/requirements.in +++ b/demo-project/src/demo_project/requirements.in @@ -16,4 +16,4 @@ wheel>=0.35, <0.37 pillow~=9.0 matplotlib==3.5.0 pre-commit~=1.17 -seaborn~=0.11.2 +seaborn>=0.13.0 diff --git a/demo-project/src/docker_requirements.txt b/demo-project/src/docker_requirements.txt index 407cdfb038..5f220c35cf 100644 --- a/demo-project/src/docker_requirements.txt +++ b/demo-project/src/docker_requirements.txt @@ -2,4 +2,4 @@ kedro>=0.18.0 kedro-datasets[pandas.CSVDataset,pandas.ExcelDataset, pandas.ParquetDataset, plotly.PlotlyDataset, matplotlib.MatplotlibWriter]>=2.1.0 scikit-learn~=1.0 pillow~=9.0 -seaborn~=0.11.2 +seaborn>=0.13.0 From c8c21b1b4c933eb07aef12c910cd5b2c14871e53 Mon Sep 17 00:00:00 2001 From: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Date: Fri, 17 May 2024 16:51:45 +0100 Subject: [PATCH 13/22] Update intersphinx mapping (#1911) Signed-off-by: Sajid Alam --- docs/source/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 1d3edbd09d..7a21f6e72e 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -54,7 +54,7 @@ intersphinx_mapping = { "kedro": ("https://docs.kedro.org/en/stable/", None), - "kedro-datasets": ("https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-1.7.1/", None), + "kedro-datasets": ("https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-3.0.0/", None), } # -- Options for HTML output ------------------------------------------------- From e9096305211f84a742f310f351f25b5c98160d5b Mon Sep 17 00:00:00 2001 From: Huong Nguyen <32060364+Huongg@users.noreply.github.com> Date: Mon, 20 May 2024 09:24:45 +0100 Subject: [PATCH 14/22] Remove default props from functional components (#1906) * remove default props to clear warning in the console Signed-off-by: huongg * set visible true for iconButton Signed-off-by: huongg * set default props Signed-off-by: huongg * update release note Signed-off-by: huongg * update missing default props Signed-off-by: huongg * remove default props from App.js Signed-off-by: huongg --------- Signed-off-by: huongg --- RELEASE.md | 1 + src/components/ui/button/button.js | 17 ++++++------ .../ui/dropdown/dropdown-renderer.js | 27 ++++--------------- src/components/ui/dropdown/dropdown.js | 20 +++----------- src/components/ui/icon-button/icon-button.js | 22 +++------------ src/components/ui/menu-option/menu-option.js | 13 ++------- .../ui/search-bar/search-bar-renderer.js | 5 ---- src/components/ui/search-bar/search-bar.js | 16 ++--------- .../search-bar/search-input/search-input.js | 15 ++--------- src/components/ui/tooltip/tooltip.js | 27 ++++++------------- tools/test-lib/react-app/app.js | 6 +---- 11 files changed, 37 insertions(+), 132 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index c3cd6e3edf..bfb4d2d918 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -16,6 +16,7 @@ Please follow the established format: - Enhance kedro-viz doc integration. (#1874) - Fix Kedro-Viz waiting for valid Kedro project. (#1871) - Enhance Kedro-Viz documentation by using Kedro-sphinx-theme. (#1898) +- Remove default props from functional components. (#1906) - Fix for schema change in strawberry-graphql JSON scalar. (#1903) - Fix messaging level when package compatibility is not satisfied. (#1904) diff --git a/src/components/ui/button/button.js b/src/components/ui/button/button.js index 3bc486ce35..84f009a7ce 100644 --- a/src/components/ui/button/button.js +++ b/src/components/ui/button/button.js @@ -6,7 +6,14 @@ import './button.scss'; /** * Generic Kedro Button */ -const Button = ({ children, dataTest, disabled, onClick, size, mode }) => ( +const Button = ({ + children, + dataTest = 'TestDefaultDataValue', + disabled = false, + onClick, + size = 'regular', + mode = 'primary', +}) => (