notebook_examples/pipelines-ml_lifecycle.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "35952005",
   "metadata": {},
   "source": [
    "@notebook{pipelines-ml_lifecycle.ipynb,\n",
    "    title: Working with Pipelines,\n",
    "    summary: Create and use ML pipelines through the entire machine learning lifecycle,\n",
    "    developed on: generalml_p38_cpu_v1,\n",
    "    keywords: pipelines, pipeline step, jobs pipeline, \n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdc165b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "\n",
    "!pip install -U oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b301ce",
   "metadata": {},
   "source": [
    "<font color=gray>Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2022 Oracle, Inc.  All rights reserved.\n",
    "Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.\n",
    "</font>\n",
    "\n",
    "***\n",
    "\n",
    "# <font color=red>Working with Pipelines</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by <font color=\"teal\">The Oracle Data Science Team</font></p>\n",
    "\n",
    "***\n",
    "\n",
    "## Overview:\n",
    "\n",
    "The machine learning lifecycle is composed of several steps: data acquisition and extraction, data preparation, featurization, model training (including algorithm selection and hyper-parameter tuning), model evaluation, deployment, and monitoring and possibly retraining the deployed model. Oracle Cloud Infrastructure (OCI) Data Science Machine Learning (ML) Pipeline enables you to define and run an end-to-end machine learning orchestration covering the entire machine learning lifecycle. Thus you can execute in a repeatable, and continuous ML pipeline with a few simple commands. \n",
    "\n",
    "This notebook uses the Accelerated Data Science (ADS) SDK to construct, control, and leverage pipelines within the Oracle Data Science service.\n",
    "\n",
    "Compatible conda pack: [General Machine Learning](https://docs.oracle.com/en-us/iaas/data-science/using/conda-gml-fam.htm) for CPU on Python 3.8 (version 1.0)\n",
    "\n",
    "---\n",
    "\n",
    "## Contents:\n",
    "\n",
    "- <a href=\"#concepts\">Introduction</a>\n",
    "  - <a href='#preliminary'>Setup</a>\n",
    "    - <a href='#policy'>Policy</a>\n",
    "    - <a href='#var'>Variables</a>\n",
    "- <a href=\"#construct\">Construct a Pipeline</a>\n",
    "  - <a href=\"#option1\">With PipelineSteps</a>\n",
    "  - <a href=\"#option2\">With YAML</a>\n",
    "- <a href='#create'>Create Pipeline</a>\n",
    "- <a href='#run'>Create a Pipeline run</a>\n",
    "  - <a href=\"#watch_status\">Watch Status</a>\n",
    "  - <a href=\"#monitor_logs\">Monitor Logs</a>\n",
    "  - <a href=\"#cancel_run\">Cancel Pipeline Run</a>\n",
    "  - <a href=\"#delete_run\">Delete Pipeline Run</a>\n",
    "- <a href='#load'>Load an Existing Pipeline</a>\n",
    "- <a href='#delete'>Delete Pipeline</a>\n",
    "- <a href='#clean-up'>Clean Up</a>\n",
    "- <a href='#magic'>Magic Commands</a>\n",
    "  - <a href=\"#magic_install\">Install</a>\n",
    "  - <a href=\"#magic_create\">Create</a>\n",
    "  - <a href=\"#magic_visualize\">Visualize</a>\n",
    "  - <a href=\"#magic_watch\">Watch</a>\n",
    "  - <a href=\"#magic_monitor\">Monitor</a>\n",
    "  - <a href=\"#magic_cancel\">Cancel</a>\n",
    "  - <a href=\"#magic_delete\">Delete</a>\n",
    "- <a href='#ref'>References</a>\n",
    "\n",
    "---\n",
    "\n",
    "**Important:**\n",
    "\n",
    "Placeholder text for required values are surrounded by angle brackets that must be removed when adding the indicated content. For example, when adding a database name to `database_name = \"<database_name>\"` would become `database_name = \"production\"`.\n",
    "\n",
    "---\n",
    "\n",
    "Datasets are provided as a convenience. Datasets are considered third-party content and are not considered materials under your agreement with Oracle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "612cf96f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ads\n",
    "import oci\n",
    "import os\n",
    "\n",
    "from ads.jobs import DataScienceJob\n",
    "from ads.pipeline import (\n",
    "    Pipeline,\n",
    "    PipelineStep,\n",
    "    PipelineRun,\n",
    "    ScriptRuntime,\n",
    "    NotebookRuntime,\n",
    "    CustomScriptStep,\n",
    ")\n",
    "from tempfile import mkdtemp\n",
    "\n",
    "ads.set_auth(\"resource_principal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2622765a",
   "metadata": {},
   "source": [
    "<a id=\"concepts\"></a>\n",
    "# Introduction\n",
    "\n",
    "A pipeline is a workflow of tasks, called steps. Steps can be run in sequence or in parallel, creating a [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) of the steps.\n",
    "\n",
    "In a machine learning context, pipelines usually provide a workflow for data import → data transformation → model training → model evaluation. In addition, the model can also be registered to a model catalog and deployed to serve predictions.\n",
    "\n",
    "The following are some key terms that will help you understand OCI Data Science ML Pipelines:\n",
    "\n",
    "* **Directed acyclic graph (DAG)**: A graph of the steps in a workflow. It defines the dependencies of each step on the other steps in the pipeline. The dependencies create a logical workflow in the form of an acyclic, there or no loops, graph. The pipeline will strive to execute steps in parallel to optimize the pipeline completion time unless the dependencies force steps to run sequentially. For example, the training steps must be completed before running the model evaluation steps. However multiple models can be trained in parallel.\n",
    "\n",
    "* **Pipeline lifecycle state**: This defines the lifecycle state of a pipeline. A pipeline can be in various states such as created, constructed, and even deleted. It is important to note that after the pipeline creation, it will still be in the CREATING state and can't be executed (run) until all steps have an artifact or job to run. In which case the pipeline will change to an ACTIVE state.\n",
    "\n",
    "* **Pipeline run**: The execution instance of a pipeline. Each pipeline run will include its step runs. A pipeline run can be configured to override some of the pipeline's defaults before starting the execution.\n",
    "\n",
    "* **Pipeline step**: a task in a pipeline. A pipeline step can be either a Data Science Job step or a Custom Script step.\n",
    "\n",
    "    - Data Science Job: the OCID of an existing Data Science Job must be provided.\n",
    "    - Custom Script: the artifact of the Python script and the execution configuration must be specified.\n",
    "    \n",
    "* **Step artifact**: The Python code to be used for the step. This code will be executed when the pipeline step is run.\n",
    "\n",
    "<a id=\"preliminary\"></a>\n",
    "## Setup\n",
    "\n",
    "<a id='policy'></a>\n",
    "### Policy\n",
    "\n",
    "Before using this notebook, your tenancy must be configured to use the ML Pipeline service.\n",
    "\n",
    "* Create or use an existing VCN Private Subnet with a Service Gateway attached to your Private Subnet Routing Table.\n",
    "* Set required policies.\n",
    "* Add users to the group's policies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83e74974",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Provide and OCID of existing DataScience Job\n",
    "job_id = \"<job_id>\"\n",
    "\n",
    "# The log group OCID\n",
    "log_group_id = \"<log_group_id>\"\n",
    "\n",
    "# The log OCID\n",
    "log_id = \"<log_id>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1d5832a",
   "metadata": {},
   "outputs": [],
   "source": [
    "compartment_id = os.environ[\"NB_SESSION_COMPARTMENT_OCID\"]\n",
    "project_id = os.environ[\"PROJECT_OCID\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab4700a3",
   "metadata": {},
   "source": [
    "<a id=\"construct\"></a>\n",
    "# Construct a Pipeline\n",
    "\n",
    "In an ADS pipeline module, you can either use the Python API or YAML to define a pipeline. In addition to the configuration of the pipeline, you provide the details of the Pipeline Steps and the DAG. The DAG is the pipeline steps and it defines dependencies between the steps.\n",
    "\n",
    "The following symbols are used in the DAG to define the dependencies between the steps. If a DAG is not provided, all steps will run in parallel.\n",
    "\n",
    "- ``>>`` denotes the tasks running in sequence, ``A >> B`` means that A is followed by B.\n",
    "- ``()`` denotes the tasks running in parallel.\n",
    "\n",
    "In the following example, `step_2` will start after `step_1` complete. `step_3` will start after both `step_1` and `step_2` are complete.\n",
    "\n",
    "```YAML\n",
    "dag:\n",
    "- step_1 >> step_2\n",
    "- (step_1, step_2) >> step_3\n",
    "```\n",
    "\n",
    "Both the log OCID and corresponding log group OCID can be specified in the ``Pipeline`` instance. If you specify only the log group OCID and no log OCID, a new Log resource is automatically created within the log group to store the logs.\n",
    "\n",
    "There are two types of logs for pipeline runs, service log and custom log. When defining a pipeline:\n",
    "\n",
    "- To enable custom log, specify ``log_id`` and ``log_group_id``.\n",
    "- To enable service log, specify ``log_group_id`` and set ``enable_service_log`` to ``True``.\n",
    "- To enable both types of logs, specify ``log_id`` and ``log_group_id``, and set ``enable_service_log`` to ``True``.\n",
    "\n",
    "With the specified DAG and pre-created pipeline steps, you can define a pipeline and give it a name.\n",
    "\n",
    "<a id=\"option1\"></a>\n",
    "## With Pipeline Steps\n",
    "\n",
    "To create a Pipeline, first, define a series of ``Pipeline Steps`` and then construct the pipeline object by providing the list of steps and DAG details. A pipeline step can be either a Data Science Job or a custom script. A Custom Script step can have different types of ``runtime`` depending on the source code you run:\n",
    "\n",
    "* ``GitPythonRuntime``: This allows you to run source code from a Git repository.\n",
    "* ``NotebookRuntime``: Allows you to run a JupyterLab Python notebook.\n",
    "* ``PythonRuntime``: This allows you to run Python code with additional options, including setting a working directory, adding Python paths, and copying output files.\n",
    "* ``ScriptRuntime`` allows you to run Python, Bash, and Java scripts from a single source file (``.zip`` or ``.tar.gz``) or code directory.\n",
    "\n",
    "The following example shows creating and running a pipeline with multiple steps. Where the steps, ``step1`` and ``step_2``are represented as a custom script, and ``step_3``is represented as a DataScience Job. The steps, ``step_1`` and ``step_2`` run in parallel and ``step_3`` runs after ``step_1`` and ``step_2`` are complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "336cb1f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare a simple script to be run with in a Pipeline step\n",
    "script_dir = mkdtemp()\n",
    "pipeline_step_script = os.path.join(script_dir, \"pipeline_step_script.py\")\n",
    "with open(pipeline_step_script, \"w\") as f:\n",
    "    f.write(\"print('Hello World!')\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b5dd8b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "infrastructure = CustomScriptStep(\n",
    "    block_storage_size=200,\n",
    "    shape_name=\"VM.Standard3.Flex\",\n",
    "    shape_config_details={\"ocpus\": 4, \"memory_in_gbs\": 32},\n",
    ")\n",
    "\n",
    "script_runtime = ScriptRuntime(\n",
    "    script_path_uri=pipeline_step_script,\n",
    "    conda={\"type\": \"service\", \"slug\": \"tensorflow26_p37_cpu_v2\"},\n",
    ")\n",
    "\n",
    "notebook_runtime = NotebookRuntime(\n",
    "    notebook_path_uri=\"https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/customization/basics.ipynb\",\n",
    "    conda={\"type\": \"service\", \"slug\": \"tensorflow26_p37_cpu_v2\"},\n",
    ")\n",
    "\n",
    "pipeline_step_1 = PipelineStep(\n",
    "    name=\"step_1\",\n",
    "    description=\"A step running a python script\",\n",
    "    infrastructure=infrastructure,\n",
    "    runtime=script_runtime,\n",
    ")\n",
    "\n",
    "pipeline_step_2 = PipelineStep(\n",
    "    name=\"step_2\",\n",
    "    description=\"A step running a notebook\",\n",
    "    infrastructure=infrastructure,\n",
    "    runtime=notebook_runtime,\n",
    ")\n",
    "\n",
    "pipeline_step_3 = PipelineStep(\n",
    "    name=\"step_3\", description=\"A step running a Data Science Job\", job_id=job_id\n",
    ")\n",
    "\n",
    "pipeline = Pipeline(\n",
    "    name=\"An example pipeline\",\n",
    "    compartment_id=compartment_id,\n",
    "    project_id=project_id,\n",
    "    step_details=[pipeline_step_1, pipeline_step_2, pipeline_step_3],\n",
    "    dag=[\"(step_1, step_2) >> step_3\"],\n",
    "    log_group_id=log_group_id,\n",
    "    log_id=log_id,\n",
    "    enable_service_log=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75cbd31d",
   "metadata": {},
   "source": [
    "<a id=\"option2\"></a>\n",
    "## With YAML\n",
    "A pipeline can also be constructed from a YAML string or a YAML file.\n",
    "\n",
    "* `Pipeline.from_yaml(<YAML string>)`\n",
    "* `Pipeline.from_yaml(uri=\"/path/to/file.yaml\")`\n",
    "* `Pipeline.from_yaml(uri=\"oci://<bucket_name>@<namespace>/<prefix>/file.yaml\")`\n",
    "\n",
    "In the previous section, a pipeline was created using Pipeline Steps. The `.to_yaml()` method can be used to convert the pipeline into a YAML format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "060afc0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline.from_yaml(pipeline.to_yaml())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd8a4858",
   "metadata": {},
   "source": [
    "Once the Pipeline object has been created, it can be printed in a YAML format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0d658e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bfb80d2",
   "metadata": {},
   "source": [
    "Use the ```.show()``` method on the Pipeline instance to visualize the pipeline in a graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aac7fe71",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "297663cf",
   "metadata": {},
   "source": [
    "<a id=\"create\"></a>\n",
    "# Create Pipeline\n",
    "\n",
    "Call the ```.create()``` method of the Pipeline instance to create a pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26184973",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.create()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9399616",
   "metadata": {},
   "source": [
    "If you print the ```pipleine``` object now, you will notice that the pipleine has an OCID value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89a24b8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f31f41e4",
   "metadata": {},
   "source": [
    "<a id=\"run\"></a>\n",
    "# Create a Pipeline Run\n",
    "A Pipeline Run is the execution instance of a Pipeline. Each Pipeline Run includes its step runs. A Pipeline Run can be configured to override some of the pipeline's defaults before starting the execution.\n",
    "\n",
    "You can call the ``.run()`` method of the ``Pipeline`` instance to launch a new Pipeline Run.\n",
    "It returns a ``PipelineRun`` instance. With a ``PipelineRun`` instance, you can watch the status of the run and stream logs for the pipeline run and the step runs.\n",
    "\n",
    "The ``.run()`` method gives you the option to override the configurations in a pipeline run. It takes the following optional parameters:\n",
    "\n",
    "- ``compartment_id: str, optional``. Defaults to ``None``. The compartment id overrides the one defined previously.\n",
    "- ``configuration_override_details: dict, optional``. Defaults to ``None``.\n",
    "The configuration details the dictionary to override the one defined previously. The ``configuration_override_details`` contains the following keys:\n",
    "\n",
    "    - ``command_line_arguments``: str, the command line arguments.\n",
    "    - ``environment_variables``: dict, the environment variables.\n",
    "    - ``maximum_runtime_in_minutes``: int, the maximum runtime allowed in minutes.\n",
    "    - ``type``: str, only ``DEFAULT`` is allowed.\n",
    "\n",
    "- ``defined_tags: dict(str, dict(str, object)), optional``. Defaults to ``None``. The defined tags dictionary to override the one defined previously.\n",
    "- ``display_name: str, optional``. Defaults to ``None``. The display name of the run.\n",
    "- ``free_form_tags: dict(str, str), optional``. Defaults to ``None``. The free-form tags dictionary overrides the one defined previously.\n",
    "- ``log_configuration_override_details: dict, optional``. Defaults to ``None``. The log configuration details the dictionary to override the one defined previously.\n",
    "- ``project_id: str, optional``. Defaults to ``None``. The project id to override the one defined previously.\n",
    "- ``step_override_details: list[PipelineStepOverrideDetails], optional``. Defaults to ``None``. The step details list overrides the one defined previously.\n",
    "- ``system_tags: dict(str, dict(str, object)), optional``. Defaults to ``None``. The system tags the dictionary to override the one defined previously.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3ca9143",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run a pipeline, a pipeline run will be created and started\n",
    "pipeline_run = pipeline.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1063b8cb",
   "metadata": {},
   "source": [
    "<a id=\"watch_status\"></a>\n",
    "## Watch Status\n",
    "\n",
    "Use the ``.show()`` method of the ``PipelineRun`` instance to retreive the current status of the pipeline run as well as each of the step runs. \n",
    "\n",
    "The ``.show()`` method takes the following optional parameter:\n",
    "- ``mode: (str, optional)``. Defaults to ``graph``. The allowed values are ``text`` or ``graph``. This parameter renders the current status of pipeline run as either text or a graph.\n",
    "\n",
    "- ``wait: (bool, optional)``. Defaults to ``False`` and it only renders the current status of each step run in graph. If set to ``True``, it renders the current status of each step run until the entire pipeline is complete.\n",
    "\n",
    "- ``rankdir: (str, optional)``. Defaults to ``TB``. The allowed values are ``TB`` or ``LR``. This parameter is applicable only for graph mode and it renders the direction of the graph as either top to bottom (TB) or left to right (LR).\n",
    "\n",
    "Render the current pipeline run status in text until the entire pipeline is complete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c01739ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run.show(mode=\"text\", wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1c75f9c",
   "metadata": {},
   "source": [
    "```\n",
    "  Step                Status\n",
    "  ------------------  ---------\n",
    "  step_1:             Succeeded\n",
    "  step_2:             Succeeded\n",
    "  step_3:             In Progress\n",
    "```\n",
    "\n",
    "Render the current pipeline run status in graph until the entire pipeline is complete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58e32a28",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run.show(wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3982ef3",
   "metadata": {},
   "source": [
    "<a id=\"monitor_logs\"></a>\n",
    "## Monitor Logs\n",
    "\n",
    "Use the ``.watch()`` method on the ``PipelineRun`` instance to stream the service log or custom log of the pipeline run.\n",
    "The ``.watch()`` method takes the following optional parameters:\n",
    "\n",
    "- ``steps: (list, optional)``. Defaults to ``None`` and streams the log of the pipeline run. If a list of the step names is provided, the method streams the log of the specified pipeline step runs.\n",
    "- ``log_type: (str, optional)``. Defaults to ``None``. The allowed values are ``custom_log``, ``service_log``, or ``None``. If ``None`` is provided, the method streams both the custom and service log of the pipeline run.\n",
    "- ``interval: (float, optional)``. The default value is ``3``. Time interval in seconds between each request to update the logs.\n",
    "\n",
    "Stream the custom and service log of the pipeline run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6020ab27",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run.watch()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e584a911",
   "metadata": {},
   "source": [
    "Stream the custom log of the specified steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f54a8b54",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run.watch(steps=[\"step_1\", \"step_2\"], log_type=\"custom_log\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c39d766f",
   "metadata": {},
   "source": [
    "<a id=\"cancel_run\"></a>\n",
    "## Cancel Pipeline Run\n",
    "Use the ``.cancel()`` method on the ``PipelineRun`` instance to cancel a pipeline run.\n",
    "\n",
    "Pipeline Runs can only be canceled when they are in the ACCEPTED or IN_PROGRESS state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46aeabe3",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run.cancel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42895b5a",
   "metadata": {},
   "source": [
    "<a id=\"delete_run\"></a>\n",
    "## Delete Pipeline Run\n",
    "Use the ``.delete()`` method on the ``PipelineRun`` instance to delete a pipeline run. It takes the following optional parameter:\n",
    "\n",
    "- ``delete_related_job_runs: (bool, optional)``. Specify whether to delete related JobRuns or not. Defaults to ``True``.\n",
    "- ``max_wait_seconds: (int, optional)``. The maximum time to wait in seconds. Defaults to ``1800``.\n",
    "\n",
    "Pipeline runs can only be deleted when they are in the SUCCEEDED, FAILED, or CANCELED state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23afd3f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d0143d8",
   "metadata": {},
   "source": [
    "<a id=\"load\"></a>\n",
    "# Load an Existing Pipeline\n",
    "\n",
    "Pipelines can be loaded by specifying their ``OCID`` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4195f03",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline.from_ocid(pipeline.id)\n",
    "pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f607af5",
   "metadata": {},
   "source": [
    "<a id=\"delete\"></a>\n",
    "# Delete Pipeline\n",
    "\n",
    "Use the ``.delete()`` method on the ``Pipeline`` instance to delete a pipeline. It takes the following optional parameters:\n",
    "\n",
    "- ``delete_related_job_runs: (bool, optional)``. Specify whether to delete related JobRuns or not. Defaults to ``True``.\n",
    "- ``delete_related_pipeline_runs: (bool, optional)``. Specify whether to delete related PipelineRuns or not. Defaults to ``True``.\n",
    "- ``max_wait_seconds: (int, optional)``. The maximum time to wait, in seconds. Defaults to ``1800``.\n",
    "\n",
    "A pipeline can only be deleted when its associated pipeline runs are all deleted, \n",
    "or alternatively, set the parameter ``delete_related_pipeline_runs`` to delete all associated runs in the same operation.\n",
    "Delete fails if a PipelineRun is in progress. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f308bfa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cd73947",
   "metadata": {},
   "source": [
    "<a id='clean-up'></a>\n",
    "# Clean Up\n",
    "The following code removes the all artifacts, Pipeline and Pipleine Run onbjects created in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f1d6707",
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.exists(pipeline_step_script):\n",
    "    os.remove(pipeline_step_script)\n",
    "\n",
    "if pipeline:\n",
    "    pipeline.delete()\n",
    "\n",
    "if os.path.exists(\"test_pipeline.yaml\"):\n",
    "    os.remove(\"test_pipeline.yaml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358ecce4",
   "metadata": {},
   "source": [
    "<a id='magic'></a>\n",
    "# Magic Commands\n",
    "Use magic commands of ``ads.pipeline`` module to construct, control, and leverage pipelines within the Oracle Data Science service."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9798fbf7",
   "metadata": {},
   "source": [
    "<a id=\"magic_install\"></a>\n",
    "## Install\n",
    "Install the pipeline extension by running the following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2346b4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext ads.pipeline.extension"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "583e07c2",
   "metadata": {},
   "source": [
    "Run ``-h`` to see supported subcommands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8971eae4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3399edf",
   "metadata": {},
   "source": [
    "```\n",
    "Usage: pipeline [SUBCOMMAND]\n",
    "Subcommand:\n",
    "    run, run a pipeline from YAML or an existing ocid.\n",
    "    log, stream the logs from pipeline run.\n",
    "    cancel, cancel a pipeline run.\n",
    "    delete, delete pipeline or pipeline run.\n",
    "    show, show the pipeline orchestration.\n",
    "    status, show the real-time status of a pipeline run.\n",
    "\n",
    "Run pipeline [SUBCOMMAND] -h to see more details.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d4052be",
   "metadata": {},
   "source": [
    "<a id=\"magic_create\"></a>\n",
    "## Create\n",
    "Use the ``run`` subcommand to create and run pipeline. Run ``-h`` to see allowed options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07ffca35",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline run -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12bff251",
   "metadata": {},
   "source": [
    "```\n",
    "Usage: pipeline run [OPTIONS]\n",
    "Options:\n",
    "    -f, --file, optional, uri to the YAML.\n",
    "    -o, --ocid, optional, ocid of existing pipeline.\n",
    "    -w, --watch, optional, a flag indicating that pipeline run will be watched after submission.\n",
    "    -l, --log-type, optional, should be custom_log, service_log or None. default is None.\n",
    "    -h, show this help message.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "825f1561",
   "metadata": {},
   "source": [
    "To create a brand new Data Science Pipeline and run it, provide the path to pipeline YAML file for the ``--file`` option"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b45161dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline run --file <path_to_pipeline_yaml>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a3a7b68",
   "metadata": {},
   "source": [
    "To run an existing pipeline, provide the pipeline OCID for the ``--ocid`` option"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "657c9c02",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline run --ocid <pipeline_ocid>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ee113b4",
   "metadata": {},
   "source": [
    "<a id=\"magic_visualize\"></a>\n",
    "## Visualize\n",
    "To visualize a pipeline in a graph, run the ``show`` subcommand and provide the pipeline OCID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d33062aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline show <pipeline_ocid>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae101744",
   "metadata": {},
   "source": [
    "<a id=\"magic_watch\"></a>\n",
    "## Watch\n",
    "Use the ``status`` subcommand to watch the current status of the pipeline run as well as each of the step runs. Run ``-h`` to see allowed options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5e910ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline status -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b991fef2",
   "metadata": {},
   "source": [
    "```\n",
    "Usage: pipeline status [OPTIONS] [RUN_ID]\n",
    "Options:\n",
    "    -x, --text, optional, a flag to show the status in text format.\n",
    "    -w, --watch, optional, a flag to wait until the completion of the pipeline run.\n",
    "    If set, the rendered graph will be updated until the completion of the pipeline run,\n",
    "    otherwise will render one graph with the current status.\n",
    "    -h, show this help message.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84d4127c",
   "metadata": {},
   "source": [
    "To watch the status of pipeline run in graph mode until it finishes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c80fd03b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline status <pipeline_run_ocid> -w"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40081345",
   "metadata": {},
   "source": [
    "To watch the status of pipeline run in text mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b47c58b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline status <pipeline_run_ocid> -x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "919737d1",
   "metadata": {},
   "source": [
    "<a id=\"magic_monitor\"></a>\n",
    "## Monitor\n",
    "Use the ``log`` subcommand to monitor the pipeline run. Run ``-h`` to see allowed options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8e7358c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline log -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc3d8772",
   "metadata": {},
   "source": [
    "```\n",
    "Usage: pipeline log [OPTIONS] [RUN_ID]\n",
    "Options:\n",
    "    -l, --log-type, optional, should be either custom_log, service_log or None. default is None.\n",
    "    -t, --tail, a flag to show the most recent log records.\n",
    "    -d, --head, a flag to show the preceding log records.\n",
    "    -n, --number, number of lines of logs to be printed. Defaults to 100.\n",
    "    -h, show this help message.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dfe10a9",
   "metadata": {},
   "source": [
    "To stream the ``custom_log``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bbfe20e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline log <pipeline_run_ocid> -l custom_log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e21215a",
   "metadata": {},
   "source": [
    "To tail the last 10 consolidated logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d55db21d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline log <pipeline_run_ocid> -t -n 10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bef8e5c9",
   "metadata": {},
   "source": [
    "<a id=\"magic_cancel\"></a>\n",
    "## Cancel\n",
    "To cancel a pipeline run, use the ``cancel`` subcommand and provide the pipeline run OCID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6df9be8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline cancel <pipeline_run_ocid>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f15c4b2",
   "metadata": {},
   "source": [
    "<a id=\"magic_delete\"></a>\n",
    "## Delete\n",
    "Use the ``delete`` subcommand to delete a pipeline or pipeline run. Run ``-h`` to see allowed options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "469d485e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline delete -h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fb08c82",
   "metadata": {},
   "source": [
    "```\n",
    "Usage: pipeline delete [OCID]\n",
    "Options:\n",
    "    -j, --no-delete-related-job-runs, a flag to not delete the related job runs.\n",
    "    -p, --no-delete-related-pipeline-runs, a flag to not delete related pipeline runs.\n",
    "    -m, --max-wait-seconds, integer, maximum wait time in second for delete to complete. Defaults to 1800.\n",
    "    -s, --succeeded-on-not-found, to flag to return successfully if the data we're waiting on is not found.\n",
    "    -h, show this help message.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f367ab2",
   "metadata": {},
   "source": [
    "To delete a pipeline run, provide the pipeline run OCID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "683b4705",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline delete <pipeline_run_ocid>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efda5e6d",
   "metadata": {},
   "source": [
    "To delete a pipeline, provide the pipeline OCID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1960b77e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pipeline delete <pipeline_ocid>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f6eec0",
   "metadata": {},
   "source": [
    "<a id='ref'></a>\n",
    "# References\n",
    "\n",
    "- [ADS Library Documentation](https://docs.cloud.oracle.com/en-us/iaas/tools/ads-sdk/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}