notebooks/templates/ai_platform_notebooks_template_hybrid.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2020 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/templates/ai_platform_notebooks_template_hybrid.ipynb\"\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/templates/ai_platform_notebooks_template_hybrid.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "# Overview\n",
        "\n",
        "{Include a paragraph or two explaining what this example demonstrates, who should be interested in it, and what you need to know before you get started.}\n",
        "\n",
        "### Dataset\n",
        "\n",
        "{Include a paragraph with Dataset information and where to obtain it}\n",
        "\n",
        "### Objective\n",
        "\n",
        "In this notebook, you will learn how to {Complete the sentence explaining briefly what you will learn from the notebook, example \n",
        "ML Training, HP tuning, Serving} The steps performed include: \n",
        "\n",
        "    * { add high level bullets for the steps of what you will perform in the notebook }\n",
        "\n",
        "### Costs \n",
        "\n",
        "Example:\n",
        "\n",
        "This tutorial uses billable components of Google Cloud Platform (GCP):\n",
        "\n",
        "* Cloud AI Platform\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Cloud AI Platform\n",
        "pricing](https://cloud.google.com/ml-engine/docs/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ze4-nDLfK4pw"
      },
      "source": [
        "### Set up your local development environment\n",
        "\n",
        "**If you are using Colab or AI Platform Notebooks**, your environment already meets\n",
        "all the requirements to run this notebook. You can skip this step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gCuSR8GkAgzl"
      },
      "source": [
        "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
        "You need the following:\n",
        "\n",
        "* The Google Cloud SDK\n",
        "* Git\n",
        "* Python 3\n",
        "* virtualenv\n",
        "* Jupyter notebook running in a virtual environment with Python 3\n",
        "\n",
        "The Google Cloud guide to [Setting up a Python development\n",
        "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
        "installation guide](https://jupyter.org/install) provide detailed instructions\n",
        "for meeting these requirements. The following steps provide a condensed set of\n",
        "instructions:\n",
        "\n",
        "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
        "\n",
        "2. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
        "\n",
        "3. [Install\n",
        "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
        "   and create a virtual environment that uses Python 3.\n",
        "\n",
        "4. Activate that environment and run `pip install jupyter` in a shell to install\n",
        "   Jupyter.\n",
        "\n",
        "5. Run `jupyter notebook` in a shell to launch Jupyter.\n",
        "\n",
        "6. Open this notebook in the Jupyter Notebook Dashboard."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install additional dependencies not installed in Notebook environment\n",
        "(e.g. XGBoost, adanet, tf-hub)\n",
        "\n",
        "- Use the latest major GA version of the framework."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "wyy5Lbnzg5fi"
      },
      "outputs": [],
      "source": [
        "%pip install -U missing_or_updating_package --user"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Restart the Kernel\n",
        "\n",
        "Once you've installed the {packages}, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "EzrelQZ22IZj"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs\n",
        "import IPython\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Before you begin\n",
        "\n",
        "### GPU run-time\n",
        "\n",
        "**Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select Runtime --> Change runtime type**\n",
        "\n",
        "### Follow before you begin in Guide\n",
        "\n",
        "{ add link to any online before you begin tutorial on the product }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "### Set up your GCP project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "\n",
        "3. [Enable the AI Platform APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
        "\n",
        "4. If you are running this notebook locally, you will need to install [Google Cloud SDK](https://cloud.google.com/sdk).\n",
        "\n",
        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
        "Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Project ID\n",
        "\n",
        "**If you don't know your project ID**, you may be able to get your PROJECT_ID using gcloud."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "# Get your GCP project id from gcloud\n",
        "shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "PROJECT_ID=shell_output[0]\n",
        "print(\"Project ID: \", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Otherwise, set your project id here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
        "    PROJECT_ID = \"[your-project-id]\" #@param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dr--iN2kAylZ"
      },
      "source": [
        "### Authenticate your GCP account\n",
        "\n",
        "**If you are using AI Platform Notebooks**, your environment is already\n",
        "authenticated. Skip this step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "**If you are using Colab**, run the cell below and follow the instructions\n",
        "when prompted to authenticate your account via oAuth.\n",
        "\n",
        "**Otherwise**, follow these steps:\n",
        "\n",
        "1. In the GCP Console, go to the [**Create service account key**\n",
        "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
        "\n",
        "2. From the **Service account** drop-down list, select **New service account**.\n",
        "\n",
        "3. In the **Service account name** field, enter a name.\n",
        "\n",
        "4. From the **Role** drop-down list, select\n",
        "   **Machine Learning Engine > AI Platform Admin** and\n",
        "   **Storage > Storage Object Admin**.\n",
        "\n",
        "5. Click *Create*. A JSON file that contains your key downloads to your\n",
        "local environment.\n",
        "\n",
        "6. Enter the path to your service account key as the\n",
        "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import sys, os\n",
        "\n",
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your GCP account. This provides access to your\n",
        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "# If on AI Platform, then don't execute this code\n",
        "    if not os.path.exists('/opt/deeplearning/metadata/env_version'):\n",
        "    if 'google.colab' in sys.modules:\n",
        "      from google.colab import auth as google_auth\n",
        "      google_auth.authenticate_user()\n",
        "\n",
        "    # If you are running this notebook locally, replace the string below with the\n",
        "    # path to your service account key and run this cell to authenticate your GCP\n",
        "    # account.\n",
        "    else:\n",
        "      %env GOOGLE_APPLICATION_CREDENTIALS ''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "When you submit a training job using the Cloud SDK, you upload a Python package\n",
        "containing your training code to a Cloud Storage bucket. AI Platform runs\n",
        "the code from this package. In this tutorial, AI Platform also saves the\n",
        "trained model that results from your job in the same bucket. You can then\n",
        "create an AI Platform model version based on this output in order to serve\n",
        "online predictions.\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
        "Cloud Storage buckets. \n",
        "\n",
        "You may also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook. Make sure to [choose a region where Cloud\n",
        "AI Platform services are\n",
        "available](https://cloud.google.com/ml-engine/docs/tensorflow/regions). You may\n",
        "not use a Multi-Regional Storage bucket for training with AI Platform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"[your-bucket-name]\" #@param {type:\"string\"}\n",
        "REGION = \"us-central1\" #@param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION gs://$BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ucvCsknMCims"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "vhOb7YnwClBb"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al gs://$BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XoEqT2Y4DJmf"
      },
      "source": [
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Y9Uo3tifg1kx"
      },
      "source": [
        "{Put all your imports and installs up into a setup section.}\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "pRUOFELefqf1"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "import sys, os"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "E6ppE7imft-y"
      },
      "source": [
        "# Notes\n",
        "\n",
        "The tips below are specific to notebooks for Tensorflow/Scikit-Learn/PyTorch/XGBoost code. \n",
        "\n",
        "# General\n",
        "\n",
        "- Include the collapsed license at the top (this uses Colab's \"Form\" mode to hide the cells).\n",
        "- Only include a single H1 title.\n",
        "- Include the button-bar immediately under the H1.\n",
        "- Include an overview section before any code.\n",
        "- Put all your installs and imports in a setup section.\n",
        "- Always include the three __future__ imports.\n",
        "- Save the notebook with the Table of Contents open.\n",
        "- Write python3 compatible code.\n",
        "- Keep cells small (~max 20 lines)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6l6xDbaH2QYd"
      },
      "source": [
        "# Python Style guide\n",
        "\n",
        "- As Guido van Rossum said, “Code is read much more often than it is written”. Please make sure you are following\n",
        "the guidelines to write Python code from the Python style guide.\n",
        "- Writing readable code here is critical. Specially when working with Notebooks: This will help other people, to read and understand your code. Having guidelines that you follow and recognize will make it easier for others to read your code.\n",
        "\n",
        "- [Google Python Style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "euaAioez2WyE"
      },
      "source": [
        "# Code content\n",
        "\n",
        "Use the highest level API that gets the job done (unless the goal is to demonstrate the low level API). For example, when using Tensorflow:\n",
        "\n",
        "- Use TF.keras.Sequential > keras functional api > keras model subclassing > ...\n",
        "\n",
        "- Use  model.fit > model.train_on_batch > manual GradientTapes.\n",
        "\n",
        "- Use eager-style code.\n",
        "\n",
        "- Use tensorflow_datasets and tf.data where possible.\n",
        "\n",
        "\n",
        "# Text\n",
        "\n",
        "- Use an imperative style. \"Run a batch of images through the model.\"\n",
        "\n",
        "- Use sentence case in titles/headings.\n",
        "\n",
        "- Use short titles/headings: \"Download the data\", \"Build the Model\", \"Train the model\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "reBCSTKOg47l"
      },
      "source": [
        "# Code Style\n",
        "\n",
        " - Notebooks are for people. Write code optimized for clarity.\n",
        "\n",
        " - Demonstrate small parts before combining them into something more complex. Like below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "MjJTYC86hPOZ"
      },
      "outputs": [],
      "source": [
        "#Build the model\n",
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Dense(10, activation='relu', input_shape=(None, 5)),\n",
        "    tf.keras.layers.Dense(3)\n",
        "])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "LWymd3KPhP_S"
      },
      "outputs": [],
      "source": [
        "# Run the model on a single batch of data, and inspect the output.\n",
        "result = model(tf.constant(np.random.randn(10,5), dtype = tf.float32)).numpy()\n",
        "\n",
        "print(\"min:\", result.min())\n",
        "print(\"max:\", result.max())\n",
        "print(\"mean:\", result.mean())\n",
        "print(\"shape:\", result.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y-pQzaxnhR-0"
      },
      "outputs": [],
      "source": [
        "# Compile the model for training\n",
        "model.compile(optimizer=tf.keras.optimizers.Adam(),\n",
        "              loss=tf.keras.losses.categorical_crossentropy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g0d8u7FmhUNf"
      },
      "source": [
        "- Keep examples quick. Use small datasets, or small slices of datasets. You don't need to train to convergence, train until it's obvious it's making progress.\n",
        "\n",
        "- For a large example, don't try to fit all the code in the notebook. Add python files to tensorflow examples, and in the notebook run: \n",
        "! pip install git+https://github.com/tensorflow/examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all GCP resources used in this project, you can [delete the GCP\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "{Include commands to delete individual resources below}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "# Delete model version resource\n",
        "! gcloud ai-platform versions delete $MODEL_VERSION --quiet --model $MODEL_NAME \n",
        "\n",
        "# Delete model resource\n",
        "! gcloud ai-platform models delete $MODEL_NAME --quiet\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "! gsutil -m rm -r $JOB_DIR\n",
        "\n",
        "# If training job is still running, cancel it\n",
        "! gcloud ai-platform jobs cancel $JOB_NAME --quiet"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "ai_platform_notebooks_template_hybrid.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}