ModelOps/14_ModelOps_GIT_PIMA_Python_osml_LightGBM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bc5ef2d4-514a-4c8a-8741-4336263ec7ec",
   "metadata": {},
   "source": [
    "<header>\n",
    "   <p  style='font-size:36px;font-family:Arial; color:#F0F0F0; background-color: #00233c; padding-left: 20pt; padding-top: 20pt;padding-bottom: 10pt; padding-right: 20pt;'>\n",
    "       ModelOps demo: PIMA Predictions with teradataml OpenSourceML LightGBM using Git\n",
    "  <br>\n",
    "       <img id=\"teradata-logo\" src=\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\" alt=\"Teradata\" style=\"width: 125px; height: auto; margin-top: 20pt;\">\n",
    "    </p>\n",
    "</header>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "260bf2ce-6b27-468d-9200-1e928d8709c2",
   "metadata": {
    "tags": []
   },
   "source": [
    "![image](images/git_meth.png) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e628a88",
   "metadata": {
    "tags": []
   },
   "source": [
    "<p style ='font-size:18px;font-family:Arial'><b>Introduction</b></p>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>This Notebook will show you how to work with ClearScape Analytics OpenSourceML functions from teradataml with ModelOps. With teradataml OpenSourceML functions you can solve your scalable challenges by using Vantage to train and score your models. Whether you have a big volume of data or you want to avoid the data movement implementation to train models outside Vantage, you can use ModelOps to manage your Catalog of Models from multiple platforms including teradataml OpenSourecML functions.<br>To know more about teradataml OpenSouceML functions review teradata official documentation.</p>\n",
    " \n",
    "<p style='font-size:16px;font-family:Arial'>This notebook will cover the Operationalization of the PIMA diabetes use case with Python using the Teradata OpenSouceML LightGBM model. A <strong>decision tree</strong> is a flowchart-like tree structure where an internal node represents a feature(or attribute), the branch represents a decision rule, and each leaf node represents the outcome.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e89a61f-22f2-4ea0-8189-aee3e232a487",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial'><b>Steps in this Notebook</b></p>\n",
    "\n",
    "<ol style = 'font-size:16px;font-family:Arial'>\n",
    "    <li>Configure the Environment </li>\n",
    "    <li>Connect to Vantage</li>\n",
    "    <li>Define Training function </li>\n",
    "    <li>Define Evaluate function </li>\n",
    "    <li>Define Scoring function</li>\n",
    "    <li>Define Model Metadata</li>\n",
    "    <li>Commit and Push to Git to let ModelOps manage</li>\n",
    "    <li>ModelOps full lifecycle till deployment</li>\n",
    "    <li>ModelOps Monitoring</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c675a412-ac1c-4dbb-b4e3-21e847c3295b",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>1. Configure the Environment</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>Here, we import the required libraries, set environment variables and environment paths (if required).</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be62e431-e39a-4baa-b02d-1dc25a36f559",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial'><b>1.1 Libraries installation</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b499f2f4-45a0-49ba-bdf8-7a786cf01e53",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install teradataml==20.0.0.3 teradatamodelops==7.0.6 matplotlib==3.8.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50aebf72-36bd-4532-96b0-ec96b7ead902",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install --upgrade pandas \"dask[complete]\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87d9a3d-623f-45a5-891e-e3ab95b245d5",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'><b>A restart of the Kernel is needed to bring the new libraries into the session</b>. We use <b>%%capture</b> to hide the details of the installation log of the librries.  You may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02739b13-3145-48c5-a153-12a8e7ca13d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install pandas==1.5.2 numpy==1.23.5 scikit-learn==1.1.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fddd25d-9b92-46d1-aa08-cdde2030a29d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install --force-reinstall lightgbm==3.3.3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1dd211c-fb80-45f5-8113-6109d22c8a48",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'><b>Another restart of the Kernel is needed to bring these last libraries into the session</b>. We use <b>%%capture</b> to hide the details of the installation log of the librries.  You may remove this parameter if you want to know all the steps of the pip installation.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5691c81c-09e7-4acc-ae80-f4007cfc4522",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'><b>Hint:</b><i>The easy way to restart the kernel to bring the above installed software into memory is to type zero zero (<b> 0 0 </b>). </i></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b392c630-2106-4619-b4a8-ceaba6cb7ce2",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;\">\n",
    "<p style = 'font-size:18px;font-family:Arial'><b>1.2 Libraries import</b></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6fbde24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from teradataml import (\n",
    "    create_context,\n",
    "    remove_context,\n",
    "    get_context,\n",
    "    get_connection,\n",
    "    DataFrame,\n",
    "    configure,\n",
    "    execute_sql,\n",
    "    db_drop_table,\n",
    "    td_lightgbm\n",
    ")\n",
    "import os\n",
    "import getpass\n",
    "import logging\n",
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "245c1005-e835-430e-a48f-c3962b72f3c8",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>2. Connect to Vantage</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4a1f46c-7798-4fc9-a710-750c56e8a299",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>You will be prompted to provide the password. Enter your password, press the Enter key, then use down arrow to go to next cell. Begin running steps with Shift + Enter keys.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "702a57c1-6013-4f92-bbac-b4278f1fb86b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%run -i ../UseCases/startup.ipynb\n",
    "eng = create_context(host = 'host.docker.internal', username='demo_user', password = password)\n",
    "print(eng)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce03f507-ff4c-4d5d-902f-8c362fe59fa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "execute_sql('''SET query_band='DEMO=14_ModelOps_GIT_PIMA_Python_osml_LightGBM.ipynb;' UPDATE FOR SESSION; ''')\n",
    "\n",
    "# configure byom/val installation\n",
    "configure.val_install_location = \"VAL\"\n",
    "configure.byom_install_location = \"MLDB\"\n",
    "\n",
    "# set the path to the local project repository for this model demo\n",
    "model_local_path = '~/modelops-demo-models/model_definitions/python_pima_osml_LightGBM'\n",
    "res = os.system(f'mkdir -p {model_local_path}/model_modules')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49269685-e702-4f35-9a84-fc8b545397e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model_local_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16dd3af9-17a6-40e8-9dec-d2023cbaed5a",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial'><b>Getting Data for This Demo</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial'>We have provided data for this demo on cloud storage. You can either run the demo using foreign tables to access the data without any storage on your environment or download the data to local storage, which may yield faster execution. Still, there could be considerations of available storage. Two statements are in the following cell, and one is commented out. You may switch which mode you choose by changing the comment string.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c65dde8-c2e9-4bc6-b1f8-b331a8d6ae19",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_cloud');\"        # Takes 10 seconds\n",
    "%run -i ../UseCases/run_procedure.py \"call get_data('DEMO_ModelOps_local');\"        # Takes 30 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5abc3f95-2694-4f13-af04-3de523a04430",
   "metadata": {},
   "source": [
    "<hr style=\"height:1px;border:none;\">\n",
    "<p style = 'font-size:18px;font-family:Arial'><b>Creating predictions and model table</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial'>We will create a predictions table where we get our model predictions and the model table where we will upload the model created.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07a3e8f5-2ee8-4ace-83b7-e56c083effab",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for Aoa_Byom_Models \n",
    "query = '''CREATE SET TABLE DEMO_USER.Aoa_Byom_Models\n",
    "     (\n",
    "      model_version VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      model_type VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      project_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      deployed_at TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6),\n",
    "      model BLOB(2097088000))\n",
    "UNIQUE PRIMARY INDEX ( model_version );\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    execute_sql('DROP TABLE DEMO_USER.Aoa_Byom_Models;')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3d6c0b9-f6b9-416c-b788-11cd8a6a9723",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ddl for PIMA_Predictions\n",
    "query = '''CREATE MULTISET TABLE Pima_Patient_Predictions\n",
    "     (\n",
    "      job_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,\n",
    "      PatientId BIGINT,\n",
    "      HasDiabetes BIGINT,\n",
    "      json_report CLOB(1048544000) CHARACTER SET LATIN)\n",
    "PRIMARY INDEX ( job_id );;\n",
    "'''\n",
    " \n",
    "try:\n",
    "    execute_sql(query)\n",
    "except:\n",
    "    db_drop_table('Pima_Patient_Predictions')\n",
    "    execute_sql(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4c1c700-ba57-4c31-9fd2-3434023e6640",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>Next is an optional step – if you want to see the status of databases/tables created and space used.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be4b98a7-ed3a-4873-89b9-538bfbdc51a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i ../UseCases/run_procedure.py \"call space_report();\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acfea1e5-d74c-4b41-88f3-b40ed80f0d9b",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>3. Define Training Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>The training function takes the following shape </p>\n",
    "\n",
    "```python\n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "    \n",
    "    # your training code using teradataml indDB function\n",
    "    model = <InDB Function>(...)\n",
    "    \n",
    "    # save your model\n",
    "    model.result.to_sql(f\"model_${context.model_version}\", if_exists=\"replace\")  \n",
    "    \n",
    "    record_training_stats(...)\n",
    "```\n",
    "<p style = 'font-size:16px;font-family:Arial'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a7fb724",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/training.py\n",
    "from teradataml import (\n",
    "    DataFrame,\n",
    "    ScaleFit,\n",
    "    ScaleTransform,\n",
    ")\n",
    "from teradataml import td_sklearn as osml\n",
    "from teradataml import td_lightgbm\n",
    "\n",
    "from aoa import (\n",
    "    record_training_stats,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "\n",
    "from collections import Counter\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "warnings.simplefilter(action='ignore', category=DeprecationWarning)\n",
    "warnings.simplefilter(action='ignore', category=UserWarning)\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "\n",
    "def compute_feature_importance(features, importances):\n",
    "    feat_dict = {}\n",
    "    for i in range(len(importances)):\n",
    "        feat_dict[features[i]] = importances[i]\n",
    "    feat_df = pd.DataFrame({'Feature':feat_dict.keys(),'Importance':feat_dict.values()})\n",
    "    return feat_df\n",
    "\n",
    "def plot_feature_importance(fi, img_filename):\n",
    "    feat_importances = fi.sort_values(['Importance'], ascending = False)\n",
    "    feat_importances.plot(kind='barh', x='Feature', y='Importance').set_title('Feature Importance')\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "    \n",
    "def train(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "    \n",
    "    # Extracting feature names, target name, and entity key from the context\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    # Load the training data from Teradata\n",
    "    train_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "\n",
    "    print (\"Scaling using InDB Functions...\")\n",
    "    X_train = train_df.drop(['HasDiabetes','PatientId'], axis = 1)\n",
    "    y_train = train_df.select([\"HasDiabetes\"])\n",
    "    # Scale the training data using the ScaleFit and ScaleTransform functions\n",
    "    scaler = ScaleFit(\n",
    "        data=train_df,\n",
    "        target_columns = feature_names,\n",
    "        scale_method=\"STD\",\n",
    "        global_scale=False\n",
    "    )\n",
    "\n",
    "    scaled_train = ScaleTransform(\n",
    "        data=train_df,\n",
    "        object=scaler.output,\n",
    "        accumulate = [target_name,entity_key]\n",
    "    )\n",
    "    \n",
    "    scaler.output.to_sql(f\"scaler_${context.model_version}\", if_exists=\"replace\")\n",
    "    print(\"Saved scaler\")\n",
    "\n",
    "    # Dataset creation.\n",
    "    LightGBM_Classifier = td_lightgbm.Dataset(X_train, y_train, silent = True, free_raw_data=False)\n",
    "    \n",
    "    print(\"Starting training using teradata osml...\")\n",
    "    \n",
    "    model = td_lightgbm.train(params={}, train_set = LightGBM_Classifier, num_boost_round=30)\n",
    "    \n",
    "    model.save_model(f\"{context.artifact_output_path}/light_gbm\")\n",
    "        \n",
    "    print(\"Complete osml training...\")\n",
    "    \n",
    "    # Calculate feature importance and generate plot\n",
    "    imp = model.feature_importance()\n",
    "    feature_importance = compute_feature_importance(feature_names, imp)\n",
    "    \n",
    "    # Plot feature importance using Gain\n",
    "    plot_feature_importance(feature_importance, f\"{context.artifact_output_path}/feature_importance\")\n",
    "    # td_lightgbm.plot_importance(model, importance_type=\"gain\", figsize=(7,6), title=\"LightGBM Feature Importance (Gain)\")\n",
    "    # plt.savefig(f\"{context.artifact_output_path}/feature_importance\")\n",
    "    \n",
    "    record_training_stats(\n",
    "        train_df,\n",
    "        features=feature_names,\n",
    "        targets=[target_name],\n",
    "        categorical=[target_name],\n",
    "        feature_importance=feature_importance,\n",
    "        context=context\n",
    "    )\n",
    "    \n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c437521",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the training dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes\n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN DEMO_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 <> 0\n",
    "\"\"\"\n",
    "\n",
    "feature_metadata =  {\n",
    "    \"database\": \"DEMO_ModelOps\",\n",
    "    \"table\": \"aoa_statistics_metadata\"\n",
    "}\n",
    "\n",
    "hyperparams = {\n",
    "    \"random_state\":32,\n",
    "    \"n_estimators\":100,\n",
    "    \"num_leaves\":4,\n",
    "    \"max_depth\":4\n",
    "}\n",
    "\n",
    "\n",
    "entity_key = \"PatientId\"\n",
    "target_names = [\"HasDiabetes\"]\n",
    "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n",
    "\n",
    "from aoa import ModelContext, DatasetInfo\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   model_version=\"osml_LightGBM_v1\",\n",
    "                   model_table=\"model_osml_LightGBM_v1\")\n",
    "\n",
    "sys.path.append(os.path.expanduser(f\"{model_local_path}/model_modules\"))\n",
    "import training\n",
    "training.train(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15cf5f8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33498ab1-a54c-4694-950d-ad3d3b6fd207",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>4. Define Evaluation Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>The evaluation function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model from Vantage\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_evaluation_stats(...)\n",
    "```\n",
    "<p style = 'font-size:16px;font-family:Arial'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d751b6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/evaluation.py\n",
    "from sklearn.metrics import confusion_matrix, roc_curve, auc\n",
    "import matplotlib.pyplot as plt\n",
    "from teradataml import td_sklearn as osml\n",
    "from teradataml import(\n",
    "    DataFrame,\n",
    "    copy_to_sql,\n",
    "    get_context,\n",
    "    get_connection,\n",
    "    ScaleTransform,\n",
    "    ConvertTo,\n",
    "    ClassificationEvaluator,\n",
    "    ROC,\n",
    "    td_lightgbm,\n",
    "    INTEGER\n",
    ")\n",
    "from aoa import (\n",
    "    record_evaluation_stats,\n",
    "    save_plot,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "\n",
    "import joblib\n",
    "import json\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "warnings.simplefilter(action='ignore', category=DeprecationWarning)\n",
    "warnings.simplefilter(action='ignore', category=UserWarning)\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "\n",
    "def compute_feature_importance(features, importances):\n",
    "    feat_dict = {}\n",
    "    for i in range(len(importances)):\n",
    "        feat_dict[features[i]] = importances[i]\n",
    "    feat_df = pd.DataFrame({'Feature':feat_dict.keys(),'Importance':feat_dict.values()})\n",
    "    return feat_df\n",
    "\n",
    "def plot_feature_importance(fi, img_filename):\n",
    "    feat_importances = fi.sort_values(['Importance'], ascending = False)\n",
    "    feat_importances.plot(kind='barh', x='Feature', y='Importance').set_title('Feature Importance')\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "\n",
    "# Define function to plot a confusion matrix from given data\n",
    "def plot_confusion_matrix(cf, img_filename):\n",
    "    import matplotlib.pyplot as plt\n",
    "    fig, ax = plt.subplots(figsize=(7.5, 7.5))\n",
    "    ax.matshow(cf, cmap=plt.cm.Blues, alpha=0.3)\n",
    "    for i in range(cf.shape[0]):\n",
    "        for j in range(cf.shape[1]):\n",
    "            ax.text(x=j, y=i,s=cf[i, j], va='center', ha='center', size='xx-large')\n",
    "    ax.set_xlabel('Predicted labels');\n",
    "    ax.set_ylabel('True labels'); \n",
    "    ax.set_title('Confusion Matrix');\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=500)\n",
    "    plt.clf()\n",
    "\n",
    "\n",
    "# Define function to plot ROC curve from ROC output data \n",
    "def plot_roc_curve(roc_out, img_filename):\n",
    "    import matplotlib.pyplot as plt\n",
    "    from sklearn import metrics\n",
    "    fpr, tpr, thresholds = metrics.roc_curve(roc_out['HasDiabetes'], roc_out['booster_predict_1'])\n",
    "    auc = metrics.roc_auc_score(roc_out['HasDiabetes'], roc_out['booster_predict_1'])\n",
    "    plt.plot(fpr,tpr,label=\"ROC curve AUC=\"+str(auc), color='darkorange')\n",
    "    plt.plot([0, 1], [0, 1], color='darkblue', linestyle='--') \n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver Operatingmetrics. Characteristic (ROC) Curve')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    fig = plt.gcf()\n",
    "    fig.savefig(img_filename, dpi=200)\n",
    "    plt.clf()\n",
    "    \n",
    "    \n",
    "\n",
    "def evaluate(context: ModelContext, **kwargs):\n",
    "\n",
    "    aoa_create_context()\n",
    "    \n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "\n",
    "    # Load the test data from Teradata\n",
    "    test_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "    X_test = test_df.drop(['HasDiabetes','PatientId'], axis = 1)\n",
    "    y_test = test_df.select([\"HasDiabetes\"])\n",
    "    # Scaling the test set\n",
    "    print (\"Loading scaler...\")\n",
    "    scaler = DataFrame(f\"scaler_${context.model_version}\")\n",
    "\n",
    "    scaled_test = ScaleTransform(\n",
    "        data=test_df,\n",
    "        object=scaler,\n",
    "        accumulate = [target_name,entity_key]\n",
    "    )\n",
    "    \n",
    "    print(\"Evaluating osml...\")\n",
    "    LightGBM_Classifier = td_lightgbm.Booster(model_file=f\"{context.artifact_input_path}/light_gbm\")\n",
    "    predict_df = LightGBM_Classifier.predict(data=X_test, label=y_test, num_iteration=20)\n",
    "    \n",
    "    # accuracy_dt = LightGBM_Classifier.score(X_test, y_test)\n",
    "    # print('*'*20)\n",
    "    # temp = predict_df.to_pandas()\n",
    "    # print(min(temp['booster_predict_1']), max(temp['booster_predict_1']))\n",
    "    # print('*'*20)\n",
    "    \n",
    "    predict_df = predict_df.assign(prediction = predict_df.booster_predict_1.round(0))\n",
    "    predict_df = predict_df.assign(prediction = predict_df.prediction.cast(type_=INTEGER))\n",
    "    \n",
    "    # Evaluate classification metrics using ClassificationEvaluator\n",
    "    ClassificationEvaluator_obj = ClassificationEvaluator(\n",
    "        data=predict_df,\n",
    "        observation_column=target_name,\n",
    "        prediction_column='prediction',\n",
    "        num_labels=2\n",
    "    )\n",
    "\n",
    "#      # Extract and store evaluation metrics\n",
    "    metrics_pd = ClassificationEvaluator_obj.output_data.to_pandas()\n",
    "      \n",
    "         \n",
    "    evaluation = {\n",
    "        'Accuracy': '{:.4f}'.format(metrics_pd.MetricValue[0]),\n",
    "        'Micro-Precision': '{:.4f}'.format(metrics_pd.MetricValue[1]),\n",
    "        'Micro-Recall': '{:.4f}'.format(metrics_pd.MetricValue[2]),\n",
    "        'Micro-F1': '{:.4f}'.format(metrics_pd.MetricValue[3]),\n",
    "        'Macro-Precision': '{:.4f}'.format(metrics_pd.MetricValue[4]),\n",
    "        'Macro-Recall': '{:.4f}'.format(metrics_pd.MetricValue[5]),\n",
    "        'Macro-F1': '{:.4f}'.format(metrics_pd.MetricValue[6]),\n",
    "        'Weighted-Precision': '{:.4f}'.format(metrics_pd.MetricValue[7]),\n",
    "        'Weighted-Recall': '{:.4f}'.format(metrics_pd.MetricValue[8]),\n",
    "        'Weighted-F1': '{:.4f}'.format(metrics_pd.MetricValue[9]),\n",
    "    }\n",
    "\n",
    "     # Save evaluation metrics to a JSON file\n",
    "    with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n",
    "        json.dump(evaluation, f)\n",
    "        \n",
    "    # Generate and save confusion matrix plot\n",
    "    cm = confusion_matrix(predict_df.to_pandas()['HasDiabetes'], predict_df.to_pandas()['prediction'])\n",
    "    plot_confusion_matrix(cm, f\"{context.artifact_output_path}/confusion_matrix\")\n",
    "    \n",
    "    # Generate and save ROC curve plot\n",
    "    # Predictions are coming out to be negative at times and hence ROC is failing. Need to check.\n",
    "    roc_df = predict_df.loc[predict_df.booster_predict_1 >=0]\n",
    "    roc_out = ROC(\n",
    "        data=roc_df,\n",
    "        probability_column='booster_predict_1',\n",
    "        observation_column=target_name,\n",
    "        positive_class='1',\n",
    "        num_thresholds=1000\n",
    "    )\n",
    "    \n",
    "    plot_roc_curve(roc_df.to_pandas(), f\"{context.artifact_output_path}/roc_curve\")\n",
    "    \n",
    "    predictions_table = \"predictions_tmp\"\n",
    "    copy_to_sql(df=predict_df, table_name=predictions_table, index=False, if_exists=\"replace\", temporary=True)\n",
    "    \n",
    "    \n",
    "    # calculate stats if training stats exist\n",
    "    if os.path.exists(f\"{context.artifact_input_path}/data_stats.json\"):\n",
    "        record_evaluation_stats(\n",
    "            features_df=test_df,\n",
    "            predicted_df=DataFrame.from_query(f\"SELECT * FROM {predictions_table}\"),\n",
    "            # feature_importance=feature_importance,\n",
    "            context=context\n",
    "        )\n",
    "\n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12ba7a9a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the evaluation dataset \n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*, D.hasdiabetes \n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "JOIN DEMO_ModelOps.PIMA_PATIENT_DIAGNOSES D\n",
    "ON F.patientid = D.patientid\n",
    "    WHERE D.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   artifact_input_path=\"./artifacts\",\n",
    "                   model_version=\"osml_LightGBM_v1\",\n",
    "                   model_table=\"model_osml_LightGBM_v1\")\n",
    "\n",
    "import evaluation\n",
    "evaluation.evaluate(context=ctx)\n",
    "\n",
    "# view evaluation results\n",
    "import json\n",
    "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n",
    "    print(json.load(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ce4ba2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the generated files\n",
    "!ls -lh artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ae441da-9858-4a91-8f70-d51b92e0ee90",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>5. Define Scoring Function</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>The scoring function takes the following shape</p>\n",
    "\n",
    "```python\n",
    "def score(context: ModelContext, **kwargs):\n",
    "    aoa_create_context()\n",
    "\n",
    "    # read your model\n",
    "    model = DataFrame(f\"model_${context.model_version}\")\n",
    "    \n",
    "    # your evaluation logic\n",
    "    \n",
    "    record_scoring_stats(...)\n",
    "```\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>You can execute this from the CLI or directly within the notebook as shown.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb4d4c1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/scoring.py\n",
    "from teradataml import td_sklearn as osml\n",
    "from teradataml import (\n",
    "    copy_to_sql,\n",
    "    DataFrame,\n",
    "    ScaleTransform,\n",
    "    td_lightgbm,\n",
    "    INTEGER\n",
    ")\n",
    "from aoa import (\n",
    "    record_scoring_stats,\n",
    "    aoa_create_context,\n",
    "    ModelContext\n",
    ")\n",
    "import pandas as pd\n",
    "\n",
    "import json\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "warnings.simplefilter(action='ignore', category=DeprecationWarning)\n",
    "warnings.simplefilter(action='ignore', category=UserWarning)\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "\n",
    "def score(context: ModelContext, **kwargs):\n",
    "    \n",
    "    aoa_create_context()\n",
    "\n",
    "    \n",
    "    # Extract feature names, target name, and entity key from the context\n",
    "    feature_names = context.dataset_info.feature_names\n",
    "    target_name = context.dataset_info.target_names[0]\n",
    "    entity_key = context.dataset_info.entity_key\n",
    "    \n",
    "    # Load the test dataset\n",
    "    test_df = DataFrame.from_query(context.dataset_info.sql)\n",
    "    copy_to_sql(\n",
    "        df=test_df,\n",
    "        schema_name=context.dataset_info.predictions_database,\n",
    "        table_name='test_df',\n",
    "        index=False,\n",
    "        if_exists=\"replace\"\n",
    "    )\n",
    "    X_test = test_df.drop(['PatientId'], axis = 1)\n",
    "    # y_test = test_df.select([\"anomaly_int\"])\n",
    "    features_tdf = DataFrame.from_query(context.dataset_info.sql)\n",
    "    features_pdf = features_tdf.to_pandas(all_rows=True)\n",
    "    test_df.set_index(\"PatientId\")\n",
    "\n",
    "    print(\"Scoring using osml...\")\n",
    "    LightGBM_Classifier = td_lightgbm.Booster(model_file=f\"{context.artifact_input_path}/light_gbm\")\n",
    "    predict_df = LightGBM_Classifier.predict(X_test)\n",
    "    \n",
    "    predict_df = predict_df.assign(prediction = predict_df.booster_predict_1.round(0))\n",
    "    predict_df = predict_df.assign(prediction = predict_df.prediction.cast(type_=INTEGER))\n",
    "    # Convert predictions to pandas DataFrame and process\n",
    "    df_pred = predict_df.to_pandas(all_rows=True).rename(columns={\"prediction\": target_name})\n",
    "    print(\"Finished Scoring\")\n",
    "\n",
    "    # store the predictions\n",
    "   \n",
    "    predictions_pdf = pd.DataFrame(df_pred, columns=[target_name])\n",
    "    predictions_pdf[entity_key] = test_df.get(['PatientId']).to_pandas().reset_index().values.flatten()\n",
    "    \n",
    "    # add job_id column so we know which execution this is from if appended to predictions table\n",
    "    predictions_pdf[\"job_id\"] = context.job_id\n",
    "    predictions_pdf[\"json_report\"] = \"\"\n",
    "    predictions_pdf = predictions_pdf[[\"job_id\", entity_key, target_name, \"json_report\"]]\n",
    "    \n",
    "    copy_to_sql(\n",
    "        df=predictions_pdf,\n",
    "        schema_name=context.dataset_info.predictions_database,\n",
    "        table_name=context.dataset_info.predictions_table,\n",
    "        index=False,\n",
    "        if_exists=\"replace\"\n",
    "    )\n",
    "        \n",
    "    print(\"Saved predictions in Teradata\")\n",
    "\n",
    "    # calculate stats\n",
    "    predictions_df = DataFrame.from_query(f\"\"\"\n",
    "        SELECT \n",
    "            * \n",
    "        FROM {context.dataset_info.get_predictions_metadata_fqtn()} \n",
    "            WHERE job_id = '{context.job_id}'\n",
    "    \"\"\")\n",
    "    \n",
    "    record_scoring_stats(features_df=features_tdf, predicted_df=predictions_df, context=context)\n",
    "\n",
    "    print(\"All done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0edf735d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n",
    "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n",
    "\n",
    "# define the scoring dataset \n",
    "\n",
    "sql = \"\"\"\n",
    "SELECT \n",
    "    F.*\n",
    "FROM DEMO_ModelOps.PIMA_PATIENT_FEATURES F \n",
    "    WHERE F.patientid MOD 5 = 0\n",
    "\"\"\"\n",
    "\n",
    "# where to store predictions\n",
    "predictions = {\n",
    "    \"database\": \"demo_user\",\n",
    "    \"table\": \"pima_patient_predictions_tmp\"\n",
    "}\n",
    "\n",
    "import uuid\n",
    "job_id=str(uuid.uuid4())\n",
    "\n",
    "dataset_info = DatasetInfo(sql=sql,\n",
    "                           entity_key=entity_key,\n",
    "                           feature_names=feature_names,\n",
    "                           target_names=target_names,\n",
    "                           feature_metadata=feature_metadata,\n",
    "                           predictions=predictions)\n",
    "\n",
    "ctx = ModelContext(hyperparams=hyperparams,\n",
    "                   dataset_info=dataset_info,\n",
    "                   artifact_output_path=\"./artifacts\",\n",
    "                   artifact_input_path=\"./artifacts\",\n",
    "                   model_version=\"osml_randomforest_v1\",\n",
    "                   model_table=\"model_osml_randomforest_v1\",\n",
    "                   job_id=job_id)\n",
    "\n",
    "import scoring\n",
    "scoring.score(context=ctx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e27b2370",
   "metadata": {},
   "outputs": [],
   "source": [
    "DataFrame.from_query(f\"SELECT * FROM pima_patient_predictions_tmp WHERE job_id='{job_id}'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06db76e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up\n",
    "\n",
    "os.system('rm -f artifacts/*')\n",
    "\n",
    "try:\n",
    "    get_context().execute(f\"DROP TABLE model_osml_LightGBM_v1\")\n",
    "except: \n",
    "    pass\n",
    "\n",
    "try:\n",
    "    get_context().execute(f\"DROP TABLE pima_patient_predictions_tmp\")\n",
    "except: \n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b5019e9-654f-4260-8a95-848539f709ba",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>6. Define Model Metadata</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>Now let's create the configuration files.<br>Requirements file with the dependencies and versions:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "157dafbf-a633-43f6-94d7-68c0e89668b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model_modules/requirements.txt\n",
    "teradataml==20.0.0.3\n",
    "teradatamodelops==7.0.6\n",
    "matplotlib==3.8.2\n",
    "PyYAML==6.0\n",
    "scikit-learn==1.1.3\n",
    "pandas==1.5.2\n",
    "numpy==1.23.5\n",
    "lightgbm==3.3.3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f9f7d1",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>The hyper parameter configuration (default values):</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8711bb8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/config.json\n",
    "{\n",
    "   \"hyperParameters\": {\n",
    "    \"random_state\":32,\n",
    "    \"n_estimators\":100,\n",
    "    \"num_leaves\":4,\n",
    "    \"max_depth\":4\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ed3d03",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>The model configuration:</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "224d5f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $model_local_path/model.json\n",
    "{\n",
    "    \"id\": \"7794bd1a-e932-450e-944b-8c6c3ed4db5e\",\n",
    "    \"name\": \"Diabetes Prediction using teradataml OpenSource LightGBMClassifier\",\n",
    "    \"description\": \"teradataml OpenSource LightGBMClassifier for Diabetes Predictions\",\n",
    "    \"language\": \"python\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "336a5344-9860-4835-a156-19547b5f6930",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>7. Commit and Push to Git to let ModelOps manage</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>Run the command below to commit and push changes to our forked repository, so ModelOps can fetch the changes to the model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0299348e-8679-4d39-8159-6aaf74b297a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd $model_local_path/../.. \\\n",
    "&& git add . \\\n",
    "&& git commit -m \"Added additional Diabetes Prediction using osml demo model\" \\\n",
    "&& git push"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c7d023",
   "metadata": {},
   "source": [
    "<p style = 'font-size:16px;font-family:Arial'>Now that changes are pushed, you can make the lifecycle inside <strong>ModelOps User Interface</strong>, plan for new trainings, evaluations, scorings. Compare models and operationalize into Production with automated Monitoring and alerting capabilities.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04c9013c-31bb-4324-a1b4-1cb6065b4294",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>8. ModelOps full lifecycle till deployment</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a34b63",
   "metadata": {},
   "source": [
    "<p style='font-size:16px;font-family:Arial'>Use or Create a Project with the git code repository with the model code, then you should see the model in the catalog already created</p>\n",
    "\n",
    "<img src=\"images/08_01.png\" alt=\"Model Catalog with inDB\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>Select the Model and then click Train a new Model. Use default hyper-parameters. This will launch the training job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_02.png\" alt=\"Train\"/>\n",
    "\n",
    "<img src=\"images/08_03.png\" alt=\"Train job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_04.png\" alt=\"Train finished\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>When Model is trained a new Model Id is created and you can get inside the Model Lifecycle screen to review artifacts and other details</p>\n",
    "\n",
    "<img src=\"images/08_06.png\" alt=\"Model lifecycle\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>Now, let's evaluate the Model, click the button and select the evaluation dataset. This will launch the evaluation job with the training script we generated and pushed to Git.</p>\n",
    "\n",
    "<img src=\"images/08_07.png\" alt=\"Evaluation\" width=\"500\" height=\"500\"/> <img src=\"images/08_08.png\" alt=\"Evaluation job\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>When evaluation job is finished a Model evaluation Report is generated with the metrics and charts that evaluation script generates</p>\n",
    "\n",
    "<img src=\"images/08_26.png\" alt=\"Model Report\" />\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>Now, let's approve the model and provide an approval description</p>\n",
    "\n",
    "<img src=\"images/08_09.png\" alt=\"Approval\" />\n",
    "\n",
    "<img src=\"images/08_10.png\" alt=\"Approval description\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<p style='font-size:16px;font-family:Arial'>The model is ready to be deployed. Let's deploy using a Batch scheduling option - Run it manual</p>\n",
    "\n",
    "<img src=\"images/08_11.png\" alt=\"Deployment Engine\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_12.png\" alt=\"Deployment Publish\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_13.png\" alt=\"Deployment Schedule\" width=\"500\" height=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fe60441-f62e-46c1-8baa-ff3cb0f1b809",
   "metadata": {},
   "source": [
    "<p style='font-size:16px;font-family:Arial'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b69248e-606c-4e46-99ea-dbed58ed3935",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>9. ModelOps Monitoring</b></p>\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>Now the model is deployed and a new Deployment appears in the deployment screen</p>\n",
    "\n",
    "\n",
    "<img src=\"images/08_15.png\" alt=\"Deploymet\" />\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>You can run jobs manually from here, review history of executions and view the predictions for a specific job</p>\n",
    "\n",
    "<img src=\"images/08_16.png\" alt=\"Deployment Run\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_17.png\" alt=\"Deployment Jobs\" />\n",
    "\n",
    "<img src=\"images/08_18.png\" alt=\"Deployment view\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<img src=\"images/08_19.png\" alt=\"Deployment predictions\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "<img src=\"images/08_20.png\" alt=\"Deployment\" width=\"500\" height=\"500\"/>\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>From the Feature Drift and Prediction Drift tabs you can check on the monitoring of the data drift</p>\n",
    "\n",
    "<img src=\"images/08_22.png\" alt=\"Feature Drift\" />\n",
    "\n",
    "<img src=\"images/08_21.png\" alt=\"Prediction Drift\" />\n",
    "\n",
    "<img src=\"images/08_23.png\" alt=\"Performance Monitoring\" />\n",
    "\n",
    "\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>From the Performance Drift, you can review multiple evaluations, let's evaluate the model with a new dataset. We create a new evaluation dataset with this query:</p>\n",
    "    \n",
    "```sql\n",
    "SELECT * FROM pima_patient_diagnoses F WHERE F.patientid MOD 8 <> 0  \n",
    "```\n",
    "\n",
    "<img src=\"images/08_24.png\" alt=\"Evaluate\" width=\"500\" height=\"500\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>and now see the evolution of the metrics</p>\n",
    "\n",
    "<img src=\"images/08_25.png\" alt=\"Metrics monitoring\" />\n",
    "\n",
    "<p style = 'font-size:16px;font-family:Arial'>\n",
    "With ModelOps you can close the cycle and review make decisions when you need to replace yor model in production, For example, You could get alerting from Data Drift of Performance Drift and you can create multiple versions and compare them, select a champion and deploy new versions that replace existing in Production.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97beb6b4-ce0a-4787-bb8f-1653b9794dcd",
   "metadata": {},
   "source": [
    "<p style='font-size:16px;font-family:Arial'>Go and try this Step by yourself. Launch ModelOps from this button below:</p>\n",
    "<a href=\"/modelops\"><img src=\"images/launchModelOps.png\" alt=\"Launch ModelOps\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9bd6e3e-d0b4-43ff-a4d3-b8a369c0a19c",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px;border:none;\">\n",
    "<p><b style = 'font-size:20px;font-family:Arial'>10. Cleanup</b></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d1318f-fe9e-4c73-9428-9734ec1fbb8f",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<p style = 'font-size:16px;font-family:Arial'>If you are done with ModelOps usecase, please uncomment and run the below cleanup section.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a71990bc-09a9-4e58-bd20-5969c83523c0",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial'><b>Work Tables</b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial'>Cleanup work tables to prevent errors next time.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "420dd53e-f099-4f1b-a3f9-1757b47a4dda",
   "metadata": {},
   "outputs": [],
   "source": [
    "# db_drop_table(table_name = 'aoa_byom_models', schema_name = 'demo_user')\n",
    "# db_drop_table(table_name = 'pima_patient_predictions', schema_name = 'demo_user')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16b6f7b6-d739-4035-aef1-555f3bbf46c6",
   "metadata": {},
   "source": [
    "<p style = 'font-size:18px;font-family:Arial'> <b>Databases and Tables </b></p>\n",
    "<p style = 'font-size:16px;font-family:Arial'>The following code will clean up tables and databases created above.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0377bf79-cd6c-41c7-8684-84e7eb432b1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %run -i ../UseCases/run_procedure.py \"call remove_data('DEMO_ModelOps');\"        # Takes 10 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f1e45aa-131d-45c8-b494-7014d32a988b",
   "metadata": {},
   "outputs": [],
   "source": [
    "remove_context()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc01570c-13b8-4ba3-b07a-2245959a5a51",
   "metadata": {},
   "source": [
    "[<< Back to Git PIMA R GBM](./10_ModelOps_GIT_PIMA_R_GBM.ipynb) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b46c6d-eab2-4fa4-9e40-2d9064305c46",
   "metadata": {},
   "source": [
    "<footer style=\"padding:10px;background:#f9f9f9;border-bottom:3px solid #394851\">Copyright © Teradata Corporation - 2023. All Rights Reserved.</footer>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}