examples/ML-IRIS-Import-and-Cache-Models-From-S3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing the IRIS Models from S3 and Caching them\n",
    "\n",
    "This notebook demonstrates how to import a machine learning model file from the S3 and store the Models + Analysis in the redis cache named \"**CACHE**\".\n",
    "\n",
    "## Overview\n",
    "\n",
    "Download the S3 IRIS Model archive from the configured S3 Bucket + Key and decompress + extract the pickled historcial analysis and previously built models. This includes examples from the IRIS sample dataset and requires you to have a valid S3 Bucket storing the models and are comfortable paying for the download costs for downloading the objects from S3 (https://aws.amazon.com/s3/pricing/).\n",
    "\n",
    "Once uploaded to the S3 Bucket you should be able to view the files have a similar disk size:\n",
    "\n",
    "![S3 Bucket Screenshot](https://raw.githubusercontent.com/jay-johnson/sci-pype/master/examples/images/scipype_s3_bucket_with_xgb_classifier_and_regressor_models_as_pickled_object_files.png \"S3 Bucket Example\")\n",
    "\n",
    "After importing, the Models and Analysis are available to any other Sci-pype instance with connectivity to the same redis cache.\n",
    "\n",
    "### Command-line Versions\n",
    "\n",
    "I built this notebook from the importer examples:\n",
    "\n",
    "https://github.com/jay-johnson/sci-pype/tree/master/bins/ml/importers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "source": [
    "### 1) Setup the Environment for Importing the IRIS Classifier Model File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Setup the Sci-pype environment\n",
    "import sys, os\n",
    "\n",
    "# Only redis is needed for this notebook:\n",
    "os.environ[\"ENV_DEPLOYMENT_TYPE\"] = \"JustRedis\"\n",
    "\n",
    "# Load the Sci-pype PyCore as a named-object called \"core\" and environment variables\n",
    "from src.common.load_ipython_env import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Setup the Request\n",
    "\n",
    "Import the Models from S3 and store the extracted Models + Analysis in the Cache.\n",
    "\n",
    "Please make sure the environment variables are set correctly and the S3 Bucket exists:\n",
    "\n",
    "   ```\n",
    "   ENV_AWS_KEY=<AWS API Key>\n",
    "   ENV_AWS_SECRET=<AWS API Secret>\n",
    "   ```\n",
    "   \n",
    "For docker containers make sure to set these keys in the correct Jupyter env file and restart the container:\n",
    "\n",
    "   ```\n",
    "   <repo base dir>/justredis/jupyter.env\n",
    "   <repo base dir>/local/jupyter.env\n",
    "   <repo base dir>/test/jupyter.env\n",
    "   ```   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "- What's the dataset name?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ds_name     = \"iris_classifier\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Where is the downloaded file getting stored?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_dir    = str(os.getenv(\"ENV_DATA_SRC_DIR\", \"/opt/work/data/src\"))\n",
    "if not os.path.exists(data_dir):\n",
    "    os.mkdir(data_dir, 0777)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What's the S3 Location (Unique Bucket Name + Key)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s3_bucket   = \"unique-bucket-name-for-datasets\" # name this something under your AWS Account (This might be open to the public in the future...stay tuned)\n",
    "s3_key      = \"dataset_\" + core.to_upper(ds_name) + \".cache.pickle.zlib\"\n",
    "s3_loc      = str(s3_bucket) + \":\" + str(s3_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Where will the downloaded file be stored?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ml_file     = data_dir + \"/\" + str(s3_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Check if the Model File needs to be Downloaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------\n",
      "Importing Models and Analysis from S3 into Caching Models from CACHE - S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib) File(/opt/work/data/src/dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "\n",
      "Downloading ModelFile S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n",
      "\n",
      "\u001b[32mDone Downloading ModelFile S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib) File(/opt/work/data/src/dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "lg(\"-------------------------------------------------\", 6)\n",
    "lg(\"Importing Models and Analysis from S3 into Caching Models from CACHE - S3Loc(\" + str(s3_loc) + \") File(\" + str(ml_file) + \")\", 6)\n",
    "lg(\"\", 6)\n",
    "\n",
    "if os.path.exists(ml_file) == False:\n",
    "\n",
    "    s3_loc              = str(s3_bucket) + \":\" + str(s3_key)\n",
    "    lg(\"Downloading ModelFile S3Loc(\" + str(s3_loc) + \")\", 6)\n",
    "    download_results    = core.s3_download_and_store_file(s3_loc, ml_file, core.get_rds(), core.get_dbs(), debug)\n",
    "\n",
    "    if download_results[\"Status\"] != \"SUCCESS\":\n",
    "        lg(\"ERROR: Stopping processing for errror: \" + str(download_results[\"Error\"]), 0)\n",
    "    else:\n",
    "        lg(\"\", 6)\n",
    "        lg(\"Done Downloading ModelFile S3Loc(\" + str(s3_loc) + \") File(\" + str(download_results[\"Record\"][\"File\"]) + \")\", 5)\n",
    "        ml_file         = download_results[\"Record\"][\"File\"]\n",
    "else:\n",
    "    lg(\"\", 6)\n",
    "    lg(\"Continuing with the existing file.\", 5)\n",
    "    lg(\"\", 6)\n",
    "# end of downloading from s3 if it's not locally available"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Start the Importer to Download + Cache the Models out of S3 or Locally if the file already exists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing(/opt/work/data/src/dataset_IRIS_CLASSIFIER.cache.pickle.zlib) Models and Analysis into Redis(CACHE)\n",
      "Loading DSName(IRIS_CLASSIFIER) ModelFile(/opt/work/data/src/dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python2/lib/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compressing Models(5)\n",
      "Compressing Model(0/5) Type(XGBRegressor)\n",
      "Done Compressing(0/5) Type(XGBRegressor) Size(347522) Decompressed(1081031)\n",
      "Caching Model(0) ID(_MD_IRIS_REGRESSOR_e50b12_0) RLoc(CACHE:_MD_IRIS_REGRESSOR_SepalLength)\n",
      "Done Caching Model(0) ID(_MD_IRIS_REGRESSOR_e50b12_0) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Compressing Model(1/5) Type(XGBRegressor)\n",
      "Done Compressing(1/5) Type(XGBRegressor) Size(374016) Decompressed(1136927)\n",
      "Caching Model(1) ID(_MD_IRIS_REGRESSOR_e50b12_1) RLoc(CACHE:_MD_IRIS_REGRESSOR_PetalLength)\n",
      "Done Caching Model(1) ID(_MD_IRIS_REGRESSOR_e50b12_1) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Compressing Model(2/5) Type(XGBRegressor)\n",
      "Done Compressing(2/5) Type(XGBRegressor) Size(195314) Decompressed(650161)\n",
      "Caching Model(2) ID(_MD_IRIS_REGRESSOR_e50b12_2) RLoc(CACHE:_MD_IRIS_REGRESSOR_PetalWidth)\n",
      "Done Caching Model(2) ID(_MD_IRIS_REGRESSOR_e50b12_2) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Compressing Model(3/5) Type(XGBRegressor)\n",
      "Done Compressing(3/5) Type(XGBRegressor) Size(410953) Decompressed(1240355)\n",
      "Caching Model(3) ID(_MD_IRIS_REGRESSOR_e50b12_3) RLoc(CACHE:_MD_IRIS_REGRESSOR_SepalWidth)\n",
      "Done Caching Model(3) ID(_MD_IRIS_REGRESSOR_e50b12_3) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Compressing Model(4/5) Type(XGBRegressor)\n",
      "Done Compressing(4/5) Type(XGBRegressor) Size(59797) Decompressed(301562)\n",
      "Caching Model(4) ID(_MD_IRIS_REGRESSOR_e50b12_4) RLoc(CACHE:_MD_IRIS_REGRESSOR_ResultTargetValue)\n",
      "Done Caching Model(4) ID(_MD_IRIS_REGRESSOR_e50b12_4) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Caching AccuracyResults RLoc(CACHE:_MD_IRIS_REGRESSOR_Accuracy)\n",
      "Done Caching AccuracyResults\n",
      "Caching PredictionsDF RLoc(CACHE:_MD_IRIS_REGRESSOR_PredictionsDF)\n",
      "Done Caching PredictionsDF\n",
      "Decompressing Analysis Dataset\n",
      "Finding ManifestKey(Accuracy) Records in RLoc(CACHE:_MD_IRIS_CLASSIFIER_Accuracy)\n",
      "Decompressing Key(Accuracy)\n",
      "Done Decompressing Key(Accuracy)\n",
      "Finding ManifestKey(PredictionsDF) Records in RLoc(CACHE:_MD_IRIS_CLASSIFIER_PredictionsDF)\n",
      "Decompressing Key(PredictionsDF)\n",
      "Done Decompressing Key(PredictionsDF)\n",
      "Done Decompressing Analysis Dataset\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_SepalLength)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_0) Type(XGBClassifier) Target(SepalLength) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_PetalLength)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_1) Type(XGBClassifier) Target(PetalLength) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_PetalWidth)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_2) Type(XGBClassifier) Target(PetalWidth) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_SepalWidth)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_3) Type(XGBClassifier) Target(SepalWidth) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_CLASSIFIER_ResultTargetValue)\n",
      "Found Model(_MD_IRIS_CLASSIFIER_bf9214_4) Type(XGBClassifier) Target(ResultTargetValue) FeatureNames(4)\n",
      "Sorting Predictions\n",
      "Done Decompressing Models(5)\n",
      "\n",
      "\u001b[32mDone Loading Model File for DSName(iris_classifier) S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_CLASSIFIER.cache.pickle.zlib)\u001b[0m\n",
      "\n",
      "\u001b[32mImporting and Caching Completed\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ra_name        = \"CACHE\"\n",
    "\n",
    "lg(\"Importing(\" + str(ml_file) + \") Models and Analysis into Redis(\" + str(ra_name) + \")\", 6)\n",
    "\n",
    "cache_req      = {\n",
    "                    \"RAName\"    : ra_name,\n",
    "                    \"DSName\"    : str(ds_name),\n",
    "                    \"TrackingID\": \"\",\n",
    "                    \"ModelFile\" : ml_file,\n",
    "                    \"S3Loc\"     : s3_loc\n",
    "               }\n",
    "\n",
    "upload_results = core.ml_load_model_file_into_cache(cache_req, core.get_rds(), core.get_dbs(), debug)\n",
    "if upload_results[\"Status\"] == \"SUCCESS\":\n",
    "    lg(\"\", 6)\n",
    "    lg(\"Done Loading Model File for DSName(\" + str(ds_name) + \") S3Loc(\" + str(cache_req[\"S3Loc\"]) + \")\", 5)\n",
    "    lg(\"\", 6)\n",
    "    lg(\"Importing and Caching Completed\", 5)\n",
    "    lg(\"\", 6)\n",
    "else:\n",
    "    lg(\"\", 6)\n",
    "    lg(\"ERROR: Failed Loading Model File(\" + str(cache_req[\"ModelFile\"]) + \") into Cache for DSName(\" + str(ds_name) + \")\", 6)\n",
    "    lg(upload_results[\"Error\"], 6)\n",
    "    lg(\"\", 6)\n",
    "# end of if success"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Setup to Import and Cache the IRIS Regressor Models and Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ds_name        = \"iris_regressor\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Import the IRIS Regressor Models from S3 and Store them in Redis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------\n",
      "Importing Models and Analysis from S3 into Caching Models from CACHE - S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_REGRESSOR.cache.pickle.zlib) File(/opt/work/data/src/dataset_IRIS_REGRESSOR.cache.pickle.zlib)\n",
      "\n",
      "Downloading ModelFile S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_REGRESSOR.cache.pickle.zlib)\n",
      "\n",
      "\u001b[32mDone Downloading ModelFile S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_REGRESSOR.cache.pickle.zlib) File(/opt/work/data/src/dataset_IRIS_REGRESSOR.cache.pickle.zlib)\u001b[0m\n",
      "Importing(/opt/work/data/src/dataset_IRIS_REGRESSOR.cache.pickle.zlib) Models and Analysis into Redis(CACHE)\n",
      "Loading DSName(IRIS_REGRESSOR) ModelFile(/opt/work/data/src/dataset_IRIS_REGRESSOR.cache.pickle.zlib)\n",
      "Compressing Models(5)\n",
      "Compressing Model(0/5) Type(XGBRegressor)\n",
      "Done Compressing(0/5) Type(XGBRegressor) Size(347522) Decompressed(1081031)\n",
      "Caching Model(0) ID(_MD_IRIS_REGRESSOR_8c4c5b_0) RLoc(CACHE:_MD_IRIS_REGRESSOR_SepalLength)\n",
      "Done Caching Model(0) ID(_MD_IRIS_REGRESSOR_8c4c5b_0) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Compressing Model(1/5) Type(XGBRegressor)\n",
      "Done Compressing(1/5) Type(XGBRegressor) Size(374016) Decompressed(1136927)\n",
      "Caching Model(1) ID(_MD_IRIS_REGRESSOR_8c4c5b_1) RLoc(CACHE:_MD_IRIS_REGRESSOR_PetalLength)\n",
      "Done Caching Model(1) ID(_MD_IRIS_REGRESSOR_8c4c5b_1) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Compressing Model(2/5) Type(XGBRegressor)\n",
      "Done Compressing(2/5) Type(XGBRegressor) Size(195314) Decompressed(650161)\n",
      "Caching Model(2) ID(_MD_IRIS_REGRESSOR_8c4c5b_2) RLoc(CACHE:_MD_IRIS_REGRESSOR_PetalWidth)\n",
      "Done Caching Model(2) ID(_MD_IRIS_REGRESSOR_8c4c5b_2) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Compressing Model(3/5) Type(XGBRegressor)\n",
      "Done Compressing(3/5) Type(XGBRegressor) Size(410953) Decompressed(1240355)\n",
      "Caching Model(3) ID(_MD_IRIS_REGRESSOR_8c4c5b_3) RLoc(CACHE:_MD_IRIS_REGRESSOR_SepalWidth)\n",
      "Done Caching Model(3) ID(_MD_IRIS_REGRESSOR_8c4c5b_3) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Compressing Model(4/5) Type(XGBRegressor)\n",
      "Done Compressing(4/5) Type(XGBRegressor) Size(59797) Decompressed(301562)\n",
      "Caching Model(4) ID(_MD_IRIS_REGRESSOR_8c4c5b_4) RLoc(CACHE:_MD_IRIS_REGRESSOR_ResultTargetValue)\n",
      "Done Caching Model(4) ID(_MD_IRIS_REGRESSOR_8c4c5b_4) RLoc(CACHE:_MODELS_IRIS_REGRESSOR_LATEST)\n",
      "Caching AccuracyResults RLoc(CACHE:_MD_IRIS_REGRESSOR_Accuracy)\n",
      "Done Caching AccuracyResults\n",
      "Caching PredictionsDF RLoc(CACHE:_MD_IRIS_REGRESSOR_PredictionsDF)\n",
      "Done Caching PredictionsDF\n",
      "Decompressing Analysis Dataset\n",
      "Finding ManifestKey(Accuracy) Records in RLoc(CACHE:_MD_IRIS_REGRESSOR_Accuracy)\n",
      "Decompressing Key(Accuracy)\n",
      "Done Decompressing Key(Accuracy)\n",
      "Finding ManifestKey(PredictionsDF) Records in RLoc(CACHE:_MD_IRIS_REGRESSOR_PredictionsDF)\n",
      "Decompressing Key(PredictionsDF)\n",
      "Done Decompressing Key(PredictionsDF)\n",
      "Done Decompressing Analysis Dataset\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_SepalLength)\n",
      "Found Model(_MD_IRIS_REGRESSOR_8c4c5b_0) Type(XGBRegressor) Target(SepalLength) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_PetalLength)\n",
      "Found Model(_MD_IRIS_REGRESSOR_8c4c5b_1) Type(XGBRegressor) Target(PetalLength) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_PetalWidth)\n",
      "Found Model(_MD_IRIS_REGRESSOR_8c4c5b_2) Type(XGBRegressor) Target(PetalWidth) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_SepalWidth)\n",
      "Found Model(_MD_IRIS_REGRESSOR_8c4c5b_3) Type(XGBRegressor) Target(SepalWidth) FeatureNames(4)\n",
      "Getting Single Model\n",
      "Getting Model RLoc(CACHE:_MD_IRIS_REGRESSOR_ResultTargetValue)\n",
      "Found Model(_MD_IRIS_REGRESSOR_8c4c5b_4) Type(XGBRegressor) Target(ResultTargetValue) FeatureNames(4)\n",
      "Sorting Predictions\n",
      "Done Decompressing Models(5)\n",
      "\n",
      "\u001b[32mDone Loading Model File for DSName(iris_regressor) S3Loc(unique-bucket-name-for-datasets:dataset_IRIS_REGRESSOR.cache.pickle.zlib)\u001b[0m\n",
      "\n",
      "\u001b[32mImporting and Caching Completed\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "s3_bucket      = \"unique-bucket-name-for-datasets\" # name this something under your AWS Account (This might be open to the public in the future...stay tuned)\n",
    "s3_key         = \"dataset_\" + core.to_upper(ds_name) + \".cache.pickle.zlib\"\n",
    "s3_loc         = str(s3_bucket) + \":\" + str(s3_key)\n",
    "ra_name        = \"CACHE\"\n",
    "ml_file        = data_dir + \"/\" + str(s3_key)\n",
    "\n",
    "lg(\"-------------------------------------------------\", 6)\n",
    "lg(\"Importing Models and Analysis from S3 into Caching Models from CACHE - S3Loc(\" + str(s3_loc) + \") File(\" + str(ml_file) + \")\", 6)\n",
    "lg(\"\", 6)\n",
    "\n",
    "if os.path.exists(ml_file) == False:\n",
    "\n",
    "    s3_loc              = str(s3_bucket) + \":\" + str(s3_key)\n",
    "    lg(\"Downloading ModelFile S3Loc(\" + str(s3_loc) + \")\", 6)\n",
    "    download_results    = core.s3_download_and_store_file(s3_loc, ml_file, core.get_rds(), core.get_dbs(), debug)\n",
    "\n",
    "    if download_results[\"Status\"] != \"SUCCESS\":\n",
    "        lg(\"ERROR: Stopping processing for errror: \" + str(download_results[\"Error\"]), 0)\n",
    "    else:\n",
    "        lg(\"\", 6)\n",
    "        lg(\"Done Downloading ModelFile S3Loc(\" + str(s3_loc) + \") File(\" + str(download_results[\"Record\"][\"File\"]) + \")\", 5)\n",
    "        ml_file         = download_results[\"Record\"][\"File\"]\n",
    "else:\n",
    "    lg(\"\", 6)\n",
    "    lg(\"Continuing with the existing file.\", 5)\n",
    "    lg(\"\", 6)\n",
    "# end of downloading from s3 if it's not locally available\n",
    "\n",
    "lg(\"Importing(\" + str(ml_file) + \") Models and Analysis into Redis(\" + str(ra_name) + \")\", 6)\n",
    "\n",
    "cache_req      = {\n",
    "                    \"RAName\"       : ra_name,\n",
    "                    \"DSName\"       : str(ds_name),\n",
    "                    \"TrackingID\"   : \"\",\n",
    "                    \"ModelFile\"    : ml_file,\n",
    "                    \"S3Loc\"        : s3_loc\n",
    "               }\n",
    "\n",
    "upload_results = core.ml_load_model_file_into_cache(cache_req, core.get_rds(), core.get_dbs(), debug)\n",
    "if upload_results[\"Status\"] == \"SUCCESS\":\n",
    "    lg(\"\", 6)\n",
    "    lg(\"Done Loading Model File for DSName(\" + str(ds_name) + \") S3Loc(\" + str(cache_req[\"S3Loc\"]) + \")\", 5)\n",
    "\n",
    "    lg(\"\", 6)\n",
    "    lg(\"Importing and Caching Completed\", 5)\n",
    "    lg(\"\", 6)\n",
    "else:\n",
    "    lg(\"\", 6)\n",
    "    lg(\"ERROR: Failed Loading Model File(\" + str(cache_req[\"ModelFile\"]) + \") into Cache for DSName(\" + str(ds_name) + \")\", 6)\n",
    "    lg(upload_results[\"Error\"], 6)\n",
    "    lg(\"\", 6)\n",
    "# end of if success"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Automation with Lambda - Coming Soon\n",
    "\n",
    "Native lambda uploading support will be added in the future. Packaging and functionality still need to be figured out. For now, you can extend the command line versions for the extractors below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Command-line Versions\n",
    "\n",
    "I built this notebook from the importer examples:\n",
    "\n",
    "https://github.com/jay-johnson/sci-pype/tree/master/bins/ml/importers"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}