sdk/python/jobs/automl-standalone-jobs/automl-forecasting-recipes-univariate/automl-forecasting-recipe-univariate-run.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "**AutoML Forecasting Recipe Univariate**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "- A basic understanding of Machine Learning\n",
    "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace. [Check this notebook for creating a workspace](../../../resources/workspace/workspace.ipynb) \n",
    "- A Compute Cluster. [Check this notebook to create a compute cluster](../../../resources/compute/compute.ipynb)\n",
    "- A python environment\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../README.md) - check the getting started section\n",
    "\n",
    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
    "- Connect to your AML workspace from the Python SDK\n",
    "- Create an `AutoML time-series forecasting Job` with the 'forecasting()' factory-fuction\n",
    "- Train the model using AmlCompute by submitting/running the AutoML forecasting training job\n",
    "- Obtain the model and use it to generate forecast\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### Running AutoML experiments\n",
    "See the `automl-forecasting-recipe-univariate-experiment-settings` notebook on how to determine settings for seasonal features, target lags and whether the series needs to be differenced or not. To make experimentation user-friendly, the user has to specify several parameters: DIFFERENCE_SERIES, TARGET_LAGS and STL_TYPE. Once these parameters are set, the notebook will generate correct transformations and settings to run experiments, generate forecasts, compute inference set metrics and plot forecast vs actuals. It will also convert the forecast from first differences to levels (original units of measurement) if the DIFFERENCE_SERIES parameter is set to True before calculating inference set metrics.\n",
    "\n",
    "The output generated by this notebook is saved in the `experiment_outputfolder`."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500001276
    }
   },
   "outputs": [],
   "source": [
    "# Import required libraries\n",
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.ai.ml.constants import AssetTypes, InputOutputModes\n",
    "from azure.ai.ml import automl\n",
    "from azure.ai.ml import Input\n",
    "\n",
    "import json\n",
    "import pandas as pd\n",
    "import os\n",
    "import shutil\n",
    "import yaml"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Configure workspace details and get a handle to the workspace\n",
    "\n",
    "As part of the setup you have already created a Workspace. To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../configuration.ipynb) for more details on how to configure credentials and connect to a workspace.\n",
    "\n",
    " You will also need to create a compute target for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
    "\n",
    "<b>Note </b>that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show Azure ML Workspace information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500007437
    }
   },
   "outputs": [],
   "source": [
    "credential = DefaultAzureCredential()\n",
    "ml_client = None\n",
    "try:\n",
    "    ml_client = MLClient.from_config(credential)\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "    # Enter details of your AML workspace\n",
    "    subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "    resource_group = \"<RESOURCE_GROUP>\"\n",
    "    workspace = \"<AML_WORKSPACE_NAME>\"\n",
    "    ml_client = MLClient(credential, subscription_id, resource_group, workspace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500010359
    }
   },
   "outputs": [],
   "source": [
    "workspace = ml_client.workspaces.get(name=ml_client.workspace_name)\n",
    "\n",
    "output = {}\n",
    "output[\"Workspace\"] = ml_client.workspace_name\n",
    "output[\"Subscription ID\"] = ml_client.connections._subscription_id\n",
    "output[\"Resource Group\"] = workspace.resource_group\n",
    "output[\"Location\"] = workspace.location\n",
    "output"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data\n",
    "\n",
    "We will load the data into DataFrame objects, split the data to train and test datasets, then create the Azure Machine Learning MLTable objects to prepare for the later training and inference steps."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Load the data file into DataFrame\n",
    "\n",
    "We will load the data on local machine in dataframe objects. Only for this experiment, the data is in data folder, uploaded manually. We will drop the Covid period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500016696
    }
   },
   "outputs": [],
   "source": [
    "TARGET_COLNAME = \"S4248SM144SCEN\"\n",
    "TIME_COLNAME = \"observation_date\"\n",
    "COVID_PERIOD_START = (\n",
    "    \"2020-03-01\"  # start of the covid period. To be excluded from evaluation.\n",
    ")\n",
    "\n",
    "df = pd.read_csv(\"./data/S4248SM144SCEN.csv\", parse_dates=[TIME_COLNAME])\n",
    "df.sort_values(by=TIME_COLNAME, inplace=True)\n",
    "\n",
    "# remove the Covid period\n",
    "df = df.query('{} <= \"{}\"'.format(TIME_COLNAME, COVID_PERIOD_START))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set parameters\n",
    "\n",
    "The first set of parameters is based on the analysis performed in the `auto-ml-forecasting-univariate-recipe-experiment-settings` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500018986
    }
   },
   "outputs": [],
   "source": [
    "# set parameters based on the settings notebook analysis\n",
    "DIFFERENCE_SERIES = True\n",
    "TARGET_LAGS = None\n",
    "STL_TYPE = None"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Next, define additional parameters to be used in the <a href=\"https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig?view=azure-ml-py\"> AutoML config </a> class.\n",
    "\n",
    "<ul> \n",
    "    <li> FORECAST_HORIZON:  The forecast horizon is the number of periods into the future that the model should predict. Here, we set the horizon to 12 periods (i.e. 12 quarters). For more discussion of forecast horizons and guiding principles for setting them, please see the <a href=\"https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-energy-demand\"> energy demand notebook </a>. \n",
    "    </li>\n",
    "    <li> TIME_SERIES_ID_COLNAMES: The names of columns used to group a timeseries. It can be used to create multiple series. If time series identifier is not defined, the data set is assumed to be one time-series. This parameter is used with task type forecasting. Since we are working with a single series, this list is empty.\n",
    "    </li>\n",
    "    <li> BLOCKED_MODELS: Optional list of models to be blocked from consideration during model selection stage. At this point we want to consider all ML and Time Series models.\n",
    "        <ul>\n",
    "            <li> See the following <a href=\"https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.forecasting?view=azure-ml-py\"> link </a> for a list of supported Forecasting models</li>\n",
    "        </ul>\n",
    "    </li>\n",
    "</ul>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500024383
    }
   },
   "outputs": [],
   "source": [
    "# set other parameters\n",
    "FORECAST_HORIZON = 12\n",
    "TIME_SERIES_ID_COLNAMES = []\n",
    "BLOCKED_MODELS = [\"ExtremeRandomTrees\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500026457
    }
   },
   "outputs": [],
   "source": [
    "# difference data and test for unit root\n",
    "if DIFFERENCE_SERIES:\n",
    "    df_delta = df.copy()\n",
    "    df_delta[TARGET_COLNAME] = df[TARGET_COLNAME].diff()\n",
    "    df_delta.dropna(axis=0, inplace=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the data into train and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500028916
    }
   },
   "outputs": [],
   "source": [
    "from helper_functions import ts_train_test_split\n",
    "\n",
    "# split the data into train and test set\n",
    "if DIFFERENCE_SERIES:\n",
    "    # generate train/inference sets using data in first differences\n",
    "    df_train, df_test = ts_train_test_split(\n",
    "        df_input=df_delta,\n",
    "        n=FORECAST_HORIZON,\n",
    "        time_colname=TIME_COLNAME,\n",
    "        ts_id_colnames=TIME_SERIES_ID_COLNAMES,\n",
    "    )\n",
    "else:\n",
    "    df_train, df_test = ts_train_test_split(\n",
    "        df_input=df,\n",
    "        n=FORECAST_HORIZON,\n",
    "        time_colname=TIME_COLNAME,\n",
    "        ts_id_colnames=TIME_SERIES_ID_COLNAMES,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500038680
    }
   },
   "outputs": [],
   "source": [
    "# Save the DataFrame objects to files\n",
    "train_data_path = \"./data/train_S4248SM144SCEN.csv\"\n",
    "test_data_path = \"./data/test_S4248SM144SCEN.csv\"\n",
    "df_train.to_csv(train_data_path, index=False)\n",
    "df_test.to_csv(test_data_path, index=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Azure Machine Learning MLTable"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With Azure Machine Learning MLTables you can keep a single copy of data in your storage, easily access data during model training, share data and collaborate with other users. \n",
    "Below, we will upload the data by creating an MLTable to be used for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500041908
    }
   },
   "outputs": [],
   "source": [
    "def create_folder_and_ml_table(csv_file, output, delimiter=\",\", encoding=\"ascii\"):\n",
    "    os.makedirs(output, exist_ok=True)\n",
    "    fname = os.path.split(csv_file)[-1]\n",
    "\n",
    "    mltable = {\n",
    "        \"paths\": [{\"file\": f\"./{fname}\"}],\n",
    "        \"transformations\": [\n",
    "            {\"read_delimited\": {\"delimiter\": delimiter, \"encoding\": encoding}}\n",
    "        ],\n",
    "    }\n",
    "    with open(os.path.join(output, \"MLTable\"), \"w\") as f:\n",
    "        f.write(yaml.dump(mltable))\n",
    "    shutil.copy(csv_file, os.path.join(output, fname))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500049325
    }
   },
   "outputs": [],
   "source": [
    "train_mltable_path = \"./data/training-mltable-folder\"\n",
    "create_folder_and_ml_table(train_data_path, train_mltable_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500052359
    }
   },
   "outputs": [],
   "source": [
    "# Training MLTable defined locally, with local data to be uploaded\n",
    "my_training_data_input = Input(type=AssetTypes.MLTABLE, path=train_mltable_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "To create data input from TabularDataset created using V1 sdk, set the `type` to `AssetTypes.MLTABLE`, `mode` to `InputOutputModes.DIRECT` and `path` to the following format `azureml:<tabulardataset_name>` or `azureml:<tabulardataset_name:<version>`(in case we want to use specific version of the registered dataset).\n",
    "To run the following cell, remove `\"\"\"` at start and end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "# Training MLTable with v1 TabularDataset\n",
    "my_training_data_input = Input(\n",
    "    type=AssetTypes.MLTABLE, path=\"azureml:train:1\", mode=InputOutputModes.DIRECT\n",
    ")\n",
    "\"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we upload the directory with the test set data which will be used in the batch end point inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500064335
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "os.makedirs(\"test_dataset\", exist_ok=True)\n",
    "shutil.copy(\n",
    "    \"data/test_S4248SM144SCEN.csv\",\n",
    "    \"test_dataset/test_S4248SM144SCEN.csv\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500066276
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "my_test_data_input = Input(\n",
    "    type=AssetTypes.URI_FOLDER,\n",
    "    path=\"test_dataset/\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "To use TabularDataset created in V1 sdk as a test data on the batch end point inference we need to convert it to V2 Input. \n",
    "To run the following cell, remove `\"\"\"` at start and end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "from mltable import load\n",
    "os.makedirs(\"test_dataset\", exist_ok=True)\n",
    "filedataset_asset = ml_client.data.get(name=\"test\",version=1)\n",
    "test_df = load(f\"azureml:/{filedataset_asset.id}\").to_pandas_dataframe()\n",
    "test_df.to_csv(\"test_dataset/test_S4248SM144SCEN.csv\")\n",
    "my_test_data_input = Input(\n",
    "    type=AssetTypes.URI_FOLDER,\n",
    "    path=\"test_dataset/\",\n",
    ")\n",
    "\"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Create or Attach existing AmlCompute\n",
    "\n",
    "[Azure Machine Learning Compute](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute) is a managed-compute infrastructure that allows the user to easily create a single or multi-node compute. In this tutorial, you will create and an AmlCompute cluster as your training compute resource.\n",
    "\n",
    "<b>Creation of AmlCompute takes approximately 5 minutes.</b>\n",
    "\n",
    "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
    "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500104600
    }
   },
   "outputs": [],
   "source": [
    "from azure.core.exceptions import ResourceNotFoundError\n",
    "from azure.ai.ml.entities import AmlCompute\n",
    "\n",
    "cluster_name = \"recipe-cluster\"\n",
    "\n",
    "try:\n",
    "    # Retrieve an already attached Azure Machine Learning Compute.\n",
    "    compute = ml_client.compute.get(cluster_name)\n",
    "except ResourceNotFoundError as e:\n",
    "    compute = AmlCompute(\n",
    "        name=cluster_name,\n",
    "        size=\"STANDARD_DS12_V2\",\n",
    "        type=\"amlcompute\",\n",
    "        min_instances=0,\n",
    "        max_instances=4,\n",
    "        idle_time_before_scale_down=120,\n",
    "    )\n",
    "    poller = ml_client.begin_create_or_update(compute)\n",
    "    poller.wait()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Configure and run the AutoML Forecasting training job\n",
    "\n",
    "In this section we will configure and run the AutoML job to train the model.\n",
    "\n",
    "### 4.1 Configure the job through the forecasting() factory function\n",
    "\n",
    "#### forecasting() function parameters:\n",
    "\n",
    "The `forecasting()` factory function allows user to configure AutoML for the forecasting task for the most common scenarios with the following properties.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**target_column_name**|The name of the label column.|\n",
    "|**primary_metric**|This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>|\n",
    "|**training_data**|The training data to be used for this experiment. You can use a registered MLTable in the workspace using the format `<mltable_name>:<version>` OR you can use a local file or folder as a MLTable. For e.g `Input(mltable='my_mltable:1')` OR `Input(mltable=MLTable(local_path=\"./data\"))` The parameter 'training_data' must always be provided.|\n",
    "|**compute**|The compute on which the AutoML job will run. In this example we are using a compute called 'cpu-cluster' present in the workspace. You can replace it with any other compute in the workspace.|\n",
    "|**n_cross_validations**|Number of cross-validation folds to use for model/pipeline selection. The default value is \"auto\", in which case AutoMl determines the number of cross-validations automatically, if a validation set is not provided. Or, users could specify an integer value.|\n",
    "|**name**|The name of the Job/Run. This is an optional property. If not specified, a random name will be generated.\n",
    "|**experiment_name**|The name of the Experiment. An Experiment is like a folder with multiple runs in Azure ML Workspace that should be related to the same logical machine learning experiment. For example, if a user runs this notebook multiple times, there will be multiple runs associated with the same Experiment name.|\n",
    "|**enable_model_explainability**|If set to true, the explanations such as feature importance for the best model will be generated.|\n",
    "\n",
    "#### set_limits() parameters:\n",
    "This is an optional configuration method to configure limits parameters such as timeouts.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**timeout_minutes**|Maximum amount of time in minutes that the whole AutoML job can take before the job terminates. This timeout includes setup, featurization and training runs but does not include the ensembling and model explainability runs at the end of the process since those actions need to happen once all the trials (children jobs) are done. If not specified, the default job's total timeout is 6 days (8,640 minutes). To specify a timeout less than or equal to 1 hour (60 minutes), make sure your dataset's size is not greater than 10,000,000 (rows times column) or an error results. It is hard to say what the timeout limit should be because the runtimes depend on multiple factors such as number of unique time series in the dataset, length of time series, statistical properties of the data, etc. If your dataset is less than 10,000,000 observations, you can try to set the experiment to 1 hour. If you are seeing less than 30 child jobs completed in this time frame, increase the timeout limit and re-run the experiment.|\n",
    "|**trial_timeout_minutes**|Maximum time in minutes that each trial (child job) can run for before it terminates. If not specified, a value of 1 month or 43200 minutes is used.|\n",
    "|**max_trials**|The maximum number of trials/runs each with a different combination of algorithm and hyperparameters to try during an AutoML job. If not specified, the default is 1000 trials. If you are setting the `enable_early_termination=True` the number of trials will be smaller.|\n",
    "|**max_concurrent_trials**|Represents the maximum number of trials (children jobs) that would be executed in parallel. It's a good practice to set this number equal to the number of nodes in your cluster.|\n",
    "|**enable_early_termination**|Whether to enable early termination if the score is not improving over 10 iterations. Early stopping window starts only after first 20 iterations. This means that the first iteration where stopping can occur is the 31st.|\n",
    "\n",
    "#### Specialized Forecasting Parameters\n",
    "To define forecasting parameters for your experiment training, you can leverage the .set_forecast_settings() method. \n",
    "The table below details the forecasting parameters we will be passing into our experiment.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**time_column_name**|The name of your time column.|\n",
    "|**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).|\n",
    "|**frequency**|Forecast frequency. This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information.\n",
    "|**cv_step_size**|Number of periods between two consecutive cross-validation folds. The default value is \"auto\", in which case AutoML determines the cross-validation step size automatically, if a validation set is not provided. Or users could specify an integer value.|\n",
    "|**target_lags**|The target_lags specifies how far back we will construct the lags of the target variable.|\n",
    "|**country_or_region_for_holidays**|The country/region used to generate holiday features. These should be ISO 3166 two-letter country/region codes (i.e. 'US', 'GB').|\n",
    "\n",
    "#### Using lags\n",
    "This training is also using the **target lags** which are lagged values of the target variable. Since we generate lags with respect to horizon, we therefore must still specify the `forecast_horizon` that the model will learn to forecast. The `target_lags` keyword specifies how far back we will construct the lags of the target variable. The `target_lags` is set to auto in this notebook. Therefore, this value will be computed automatically.\n",
    "\n",
    "This notebook uses the `.set_training(blocked_training_algorithms=...)` parameter to exclude some models that take a longer time to train on this dataset.  You can choose to remove models from the blocked_training_algorithms list but you may need to increase the trial_timeout_minutes parameter value to get results.\n",
    "\n",
    "To run AutoML, you need to create an `Experiment`. An `Experiment` corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500105198
    }
   },
   "outputs": [],
   "source": [
    "# choose a name for the run history container in the workspace\n",
    "if isinstance(TARGET_LAGS, list):\n",
    "    TARGET_LAGS_STR = (\n",
    "        \"-\".join(map(str, TARGET_LAGS)) if (len(TARGET_LAGS) > 0) else None\n",
    "    )\n",
    "else:\n",
    "    TARGET_LAGS_STR = TARGET_LAGS\n",
    "\n",
    "experiment_desc = \"diff-{}_lags-{}_STL-{}\".format(\n",
    "    DIFFERENCE_SERIES, TARGET_LAGS_STR, STL_TYPE\n",
    ")\n",
    "\n",
    "# General job parameters.\n",
    "max_trials = 5\n",
    "exp_name = \"univariate_recipe_{}\".format(experiment_desc)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Create the AutoML forecasting job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500120998
    }
   },
   "outputs": [],
   "source": [
    "# Create the AutoML forecasting job with the related factory-function.\n",
    "\n",
    "forecasting_job = automl.forecasting(\n",
    "    compute=cluster_name,\n",
    "    experiment_name=exp_name,\n",
    "    training_data=my_training_data_input,\n",
    "    target_column_name=TARGET_COLNAME,\n",
    "    primary_metric=\"NormalizedRootMeanSquaredError\",\n",
    "    n_cross_validations=3,\n",
    "    enable_model_explainability=True,\n",
    "    tags={\"my_custom_tag\": \"My custom value\"},\n",
    ")\n",
    "\n",
    "# Limits are all optional\n",
    "forecasting_job.set_limits(\n",
    "    timeout_minutes=20,\n",
    "    trial_timeout_minutes=5,\n",
    "    enable_early_termination=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500123750
    }
   },
   "outputs": [],
   "source": [
    "# Specialized properties for Time Series Forecasting training\n",
    "forecasting_job.set_forecast_settings(\n",
    "    time_column_name=TIME_COLNAME,\n",
    "    forecast_horizon=FORECAST_HORIZON,\n",
    "    target_lags=TARGET_LAGS,\n",
    "    time_series_id_column_names=TIME_SERIES_ID_COLNAMES,\n",
    "    use_stl=STL_TYPE,\n",
    ")\n",
    "\n",
    "# Training properties are optional\n",
    "forecasting_job.set_training(blocked_training_algorithms=BLOCKED_MODELS)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Train the AutoML model\n",
    "\n",
    "Using the `MLClient` created earlier, we will execute the following commands to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684500133149
    }
   },
   "outputs": [],
   "source": [
    "# Submit the AutoML job\n",
    "returned_job = ml_client.jobs.create_or_update(\n",
    "    forecasting_job, experiment_name=exp_name\n",
    ")  # submit the job to the backend\n",
    "\n",
    "print(f\"Created job: {returned_job}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684501835464
    }
   },
   "outputs": [],
   "source": [
    "# Wait until AutoML training runs are finished\n",
    "ml_client.jobs.stream(returned_job.name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## 5. Retrieve the Best Trial (Best Model's trial/run)\n",
    "\n",
    "Use the MLFLowClient to access the results (such as Models, Artifacts, Metrics) of a previously completed AutoML Trial."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 5.1 Initialize MLFlow Client\n",
    "The models and artifacts that are produced by AutoML can be accessed via the MLFlow interface. \n",
    "Initialize the MLFlow client here, and set the backend as Azure ML, via. the MLFlow Client.\n",
    "\n",
    "*IMPORTANT*, you need to have installed the latest MLFlow packages with:\n",
    "\n",
    "    pip install azureml-mlflow\n",
    "\n",
    "    pip install mlflow"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Obtain the tracking URI for MLFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684502446887
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "import mlflow\n",
    "\n",
    "# Obtain the tracking URL from MLClient\n",
    "MLFLOW_TRACKING_URI = ml_client.workspaces.get(\n",
    "    name=ml_client.workspace_name\n",
    ").mlflow_tracking_uri\n",
    "\n",
    "print(MLFLOW_TRACKING_URI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684502450358
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# Set the MLFLOW TRACKING URI\n",
    "mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)\n",
    "\n",
    "print(\"\\nCurrent tracking uri: {}\".format(mlflow.get_tracking_uri()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684502452800
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "from mlflow.tracking.client import MlflowClient\n",
    "\n",
    "# Initialize MLFlow client\n",
    "mlflow_client = MlflowClient()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Get the AutoML parent Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684502578515
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "job_name = returned_job.name\n",
    "\n",
    "# Example if providing an specific Job name/ID\n",
    "# job_name = \"591640e8-0f88-49c5-adaa-39b9b9d75531\"\n",
    "\n",
    "# Get the parent run\n",
    "mlflow_parent_run = mlflow_client.get_run(job_name)\n",
    "\n",
    "print(\"Parent Run: \")\n",
    "print(mlflow_parent_run)\n",
    "\n",
    "# Print parent run tags. 'automl_best_child_run_id' tag should be there.\n",
    "print(mlflow_parent_run.data.tags)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### Get the AutoML best child run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684502656168
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# Get the best model's child run\n",
    "best_child_run_id = mlflow_parent_run.data.tags[\"automl_best_child_run_id\"]\n",
    "print(\"Found best child run id: \", best_child_run_id)\n",
    "\n",
    "best_run = mlflow_client.get_run(best_child_run_id)\n",
    "\n",
    "print(\"Best child run: \")\n",
    "print(best_run)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Model Evaluation and Forecasting"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 6.1 Download the best model\n",
    "Access the results (such as models, artifacts, metrics) of a previously completed AutoML Run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684502663520
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# Create local folder\n",
    "local_dir = \"./artifact_downloads\"\n",
    "if not os.path.exists(local_dir):\n",
    "    os.mkdir(local_dir)\n",
    "# Download run's artifacts/outputs\n",
    "local_path = mlflow_client.download_artifacts(\n",
    "    best_run.info.run_id, \"outputs\", local_dir\n",
    ")\n",
    "print(\"Artifacts downloaded in: {}\".format(local_path))\n",
    "print(\"Artifacts: {}\".format(os.listdir(local_path)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Forecasting using batch endpoint"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have retrieved the best pipeline/model, it can be used to make predictions on test data. We will do batch inferencing on the test dataset which must have the same schema as training dataset.\n",
    "\n",
    "The inference will run on a remote compute. In this example, it will re-use the training compute."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2.1 Create a model endpoint\n",
    "First, we need to register the model, environment and the batch endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684503462626
    }
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import (\n",
    "    Environment,\n",
    "    BatchEndpoint,\n",
    "    BatchDeployment,\n",
    "    BatchRetrySettings,\n",
    "    Model,\n",
    ")\n",
    "from azure.ai.ml.constants import BatchDeploymentOutputAction\n",
    "\n",
    "model_name = \"forecast-recipe-univariate-run4\"\n",
    "batch_endpoint_name = \"forecast-recipes-univariate-run4\"\n",
    "\n",
    "model = Model(\n",
    "    path=f\"azureml://jobs/{best_run.info.run_id}/outputs/artifacts/outputs/model.pkl\",\n",
    "    name=model_name,\n",
    "    description=\"Forecasting Recipe Univariate Model\",\n",
    ")\n",
    "registered_model = ml_client.models.create_or_update(model)\n",
    "\n",
    "env = Environment(\n",
    "    name=\"automl-tabular-env\",\n",
    "    description=\"environment for automl inference\",\n",
    "    image=\"mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210727.v1\",\n",
    "    conda_file=\"artifact_downloads/outputs/conda_env_v_1_0_0.yml\",\n",
    ")\n",
    "\n",
    "endpoint = BatchEndpoint(\n",
    "    name=batch_endpoint_name,\n",
    "    description=\"this is a sample batch endpoint\",\n",
    ")\n",
    "ml_client.begin_create_or_update(endpoint).wait()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a batch deployment, we will use the forecasting_script.py which will load the model and will call the forecast method each time we will envoke the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684503463103
    }
   },
   "outputs": [],
   "source": [
    "output_file = \"forecasting_recipe_univariate_output.csv\"\n",
    "batch_deployment = BatchDeployment(\n",
    "    name=\"univariate-non-mlflow-deployment\",\n",
    "    description=\"this is a sample non-mlflow deployment\",\n",
    "    endpoint_name=batch_endpoint_name,\n",
    "    model=registered_model,\n",
    "    code_path=\"./forecast\",\n",
    "    scoring_script=\"forecasting_script.py\",\n",
    "    environment=env,\n",
    "    environment_variables={\n",
    "        \"TARGET_COLUMN_NAME\": TARGET_COLNAME,\n",
    "    },\n",
    "    compute=cluster_name,\n",
    "    instance_count=4,\n",
    "    max_concurrency_per_instance=4,\n",
    "    mini_batch_size=10,\n",
    "    output_action=BatchDeploymentOutputAction.APPEND_ROW,\n",
    "    output_file_name=output_file,\n",
    "    retry_settings=BatchRetrySettings(max_retries=3, timeout=300),\n",
    "    logging_level=\"info\",\n",
    "    properties={\"include_output_header\": \"true\"},\n",
    "    tags={\"include_output_header\": \"true\"},\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, start a model deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684503612609
    }
   },
   "outputs": [],
   "source": [
    "ml_client.begin_create_or_update(batch_deployment).wait()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to create the Input, representing URI folder, because the batch endpoint is intended to process multiple files at a time. In this example we will use only one test file, which we have uploaded to the blob storage earlier. This file must be available through the url link.\n",
    "\n",
    "#### Create an inference job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684503956647
    }
   },
   "outputs": [],
   "source": [
    "job = ml_client.batch_endpoints.invoke(\n",
    "    endpoint_name=batch_endpoint_name,\n",
    "    input=my_test_data_input,\n",
    "    deployment_name=\"univariate-non-mlflow-deployment\",  # name is required as default deployment is not set\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will stream the job output to monitor the execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684504337555
    }
   },
   "outputs": [],
   "source": [
    "job_name = job.name\n",
    "batch_job = ml_client.jobs.get(name=job_name)\n",
    "print(batch_job.status)\n",
    "# stream the job logs\n",
    "ml_client.jobs.stream(name=job_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download the prediction result for metrics calculation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of forecast output is saved in JSON format. You can use it to calculate test set metrics and plot predictions and actuals over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684498234924
    }
   },
   "outputs": [],
   "source": [
    "ml_client.jobs.download(job_name, download_path=\".\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684498437408
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "output_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684499715806
    }
   },
   "outputs": [],
   "source": [
    "pred_df = pd.read_csv(output_file)\n",
    "pred_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684499729871
    }
   },
   "outputs": [],
   "source": [
    "def convert_fcst_diff_to_levels(fcst, yt, df_orig):\n",
    "    \"\"\"Convert forecast from first differences to levels.\"\"\"\n",
    "    fcst = fcst.reset_index(drop=False, inplace=False)\n",
    "    fcst[TIME_COLNAME] = fcst[TIME_COLNAME].apply(pd.to_datetime)\n",
    "    fcst[\"predicted_level\"] = fcst[\"predicted\"].cumsum()\n",
    "    fcst[\"predicted_level\"] = fcst[\"predicted_level\"].astype(float) + float(yt)\n",
    "    # merge actuals\n",
    "    out = pd.merge(\n",
    "        fcst, df_orig[[TIME_COLNAME, TARGET_COLNAME]], on=[TIME_COLNAME], how=\"inner\"\n",
    "    )\n",
    "    out.rename(columns={TARGET_COLNAME + \"_y\": \"actual_level\"}, inplace=True)\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684499739381
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "if DIFFERENCE_SERIES:\n",
    "    # convert forecast in differences to the levels\n",
    "    INFORMATION_SET_DATE = max(df_train[TIME_COLNAME])\n",
    "    YT = df.query(\"{} == @INFORMATION_SET_DATE\".format(TIME_COLNAME))[TARGET_COLNAME]\n",
    "    fcst_df = convert_fcst_diff_to_levels(fcst=pred_df, yt=YT, df_orig=df)\n",
    "else:\n",
    "    fcst_df = pred_df.copy()\n",
    "    fcst_df[\"actual_level\"] = y_test\n",
    "    fcst_df[\"predicted_level\"] = y_predictions"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate metrics and save output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684499754257
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "from metrics_helper import calculate_metrics\n",
    "\n",
    "metrics_df = calculate_metrics(fcst_df[\"actual_level\"], fcst_df[\"predicted_level\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684499764098
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# create output directory\n",
    "output_dir = \"experiment_output/{}\".format(experiment_desc)\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "# save output\n",
    "metrics_file_name = \"{}_metrics.csv\".format(exp_name)\n",
    "fcst_file_name = \"{}_forecst.csv\".format(exp_name)\n",
    "plot_file_name = \"{}_plot.pdf\".format(exp_name)\n",
    "\n",
    "metrics_df.to_csv(os.path.join(output_dir, metrics_file_name), index=True)\n",
    "fcst_df.to_csv(os.path.join(output_dir, fcst_file_name), index=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generate and save forecast versus actuals plot \n",
    "We will join actual observations of test data with the predictions to plot predictions and actuals on a time series plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684499771642
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684499777578
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "plot_df = df.query('{} > \"2010-01-01\"'.format(TIME_COLNAME))\n",
    "plot_df.set_index(TIME_COLNAME, inplace=True)\n",
    "fcst_df.set_index(TIME_COLNAME, inplace=True)\n",
    "\n",
    "# generate and save plots\n",
    "plt.figure(dpi=180)\n",
    "plt.plot(plot_df[TARGET_COLNAME], \"-g\", label=\"Historical\")\n",
    "plt.plot(fcst_df[\"actual_level\"], \"-b\", label=\"Actual\")\n",
    "plt.plot(fcst_df[\"predicted_level\"], \"-r\", label=\"Forecast\")\n",
    "plt.legend()\n",
    "plt.title(\"Forecast vs Actuals\")\n",
    "plt.xlabel(TIME_COLNAME)\n",
    "plt.ylabel(TARGET_COLNAME)\n",
    "\n",
    "plt.xticks(rotation=45)\n",
    "plt.savefig(os.path.join(output_dir, plot_file_name))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1684499793362
    }
   },
   "outputs": [],
   "source": [
    "# Delete the batch endpoint and compute. Do not do it occasionally.\n",
    "ml_client.batch_endpoints.begin_delete(name=batch_endpoint_name)\n",
    "ml_client.compute.begin_delete(name=cluster_name)"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python310-sdkv2"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  },
  "microsoft": {
   "host": {
    "AzureML": {
     "notebookHasBeenCompleted": true
    }
   },
   "ms_spell_check": {
    "ms_spell_check_language": "en"
   }
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}