notebooks/02_tensorflow-basic-bert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow model serving with Konduit Serving\n",
    "\n",
    "This notebook illustrates a simple client-server interaction to perform inference on a TensorFlow model using the Python SDK for Konduit Serving.  \n",
    "\n",
    "This tutorial is split into two parts: \n",
    "\n",
    "1. Configuration \n",
    "2. Running the server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from konduit import ParallelInferenceConfig, ServingConfig, TensorFlowConfig, ModelConfigType\n",
    "from konduit import TensorDataTypesConfig, ModelStep, InferenceConfiguration\n",
    "from konduit.server import Server\n",
    "from konduit.client import Client\n",
    "\n",
    "import numpy as np\n",
    "import os "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "Konduit Serving works by defining a series of **steps**. These include operations such as \n",
    "1. Pre- or post-processing steps\n",
    "2. One or more machine learning models\n",
    "3. Transforming the output in a way that can be understood by humans\n",
    "\n",
    "If deploying your model does not require pre- nor post-processing, only one step - a machine learning model - is required. This configuration is defined using a single `ModelStep`. \n",
    "\n",
    "Before running this notebook, run the `build_jar.py` script and copy the JAR (`konduit.jar`) to this folder. Refer to the [Python SDK README](https://github.com/KonduitAI/konduit-serving/blob/master/python/README.md) for details. \n",
    "\n",
    "Start by downloading the model weights to the `data` folder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve \n",
    "from zipfile import ZipFile\n",
    "dl_path = \"../data/bert/bert.zip\"\n",
    "if not os.path.isfile(dl_path):\n",
    "    urlretrieve(\"https://deeplearning4jblob.blob.core.windows.net/testresources/bert_mrpc_frozen_v1.zip\", \n",
    "                dl_path)\n",
    "with ZipFile(dl_path, 'r') as zipObj:\n",
    "    zipObj.extractall()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure the step\n",
    "\n",
    "Define the TensorFlow configuration as a `TensorFlowConfig` object. \n",
    "\n",
    "- `tensor_data_types_config`: The TensorFlowConfig object requires a dictionary `input_data_types`. Its keys should represent column names, and the values should represent data types as strings, e.g. `\"INT32\"`. See [here](https://github.com/KonduitAI/konduit-serving/blob/master/konduit-serving-api/src/main/java/ai/konduit/serving/model/TensorDataType.java) for a list of supported data types. \n",
    "- `model_config_type`: This argument requires a `ModelConfigType` object. Specify `model_type` as `TENSORFLOW`, and `model_loading_path` to point to the location of TensorFlow weights saved in the PB file format.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data_types = {'IteratorGetNext:0': 'INT32',\n",
    "                    'IteratorGetNext:1': 'INT32',\n",
    "                    'IteratorGetNext:4': 'INT32'}\n",
    "\n",
    "tensorflow_config = TensorFlowConfig(\n",
    "    tensor_data_types_config = TensorDataTypesConfig(\n",
    "        input_data_types=input_data_types\n",
    "        ),\n",
    "    model_config_type = ModelConfigType(\n",
    "        model_type='TENSORFLOW',\n",
    "        model_loading_path=os.path.abspath('bert_mrpc_frozen.pb')\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a `TensorFlowConfig` defined, we can define a `ModelStep`. The following parameters are specified: \n",
    "- `model_config`: pass the TensorFlowConfig object here \n",
    "- `parallel_inference_config`: specify the number of workers to run in parallel. Here, we specify `workers=1`.\n",
    "- `input_names`:  names for the input data  \n",
    "- `output_names`: names for the output data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_names = list(input_data_types.keys())\n",
    "output_names = [\"loss/Softmax\"]\n",
    "\n",
    "tf_step = ModelStep(\n",
    "    model_config=tensorflow_config,\n",
    "    parallel_inference_config=ParallelInferenceConfig(workers=1),\n",
    "    input_names=input_names,\n",
    "    output_names=output_names\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure the server\n",
    "\n",
    "Specify the following:\n",
    "- `http_port`: select a random port.\n",
    "- `input_data_format`, `output_data_format`: Specify input and output data formats as strings. \n",
    "\n",
    "\n",
    "{% hint style=\"info\" %}\n",
    "Accepted input and output data formats are as follows:\n",
    "\n",
    "*  Input: JSON, ARROW, IMAGE, ND4J \\(not yet implemented\\) and NUMPY.\n",
    "*  Output: NUMPY, JSON, ND4J \\(not yet implemented\\) and ARROW.\n",
    "{% endhint %}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "port = np.random.randint(1000, 65535)\n",
    "serving_config = ServingConfig(\n",
    "    http_port=port,\n",
    "    input_data_format='NUMPY',\n",
    "    output_data_format='NUMPY'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ServingConfig` has to be passed to `Server` in addition to the steps as a Python list. In this case, there is a single step: `tf_step`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "server = Server(\n",
    "    serving_config=serving_config,\n",
    "    steps=[tf_step]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `Server()` looks for the Konduit Serving JAR `konduit.jar` in the directory the script is run in. To change this default, use the `jar_path` argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run the server \n",
    "\n",
    "Start the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting server.................\n",
      "\n",
      "Server has started successfully.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<subprocess.Popen at 0x28942780278>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "server.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure the client\n",
    "\n",
    "To configure the client, create a Client object specifying the port number:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(port=port)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load some sample data from NumPy files. Note that these are NumPy arrays, each with shape (4, 128): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_input = {\n",
    "    'IteratorGetNext:0': np.expand_dims(np.load('../data/bert/input-0.npy'), axis=0),\n",
    "    'IteratorGetNext:1': np.expand_dims(np.load('../data/bert/input-1.npy'), axis=0),\n",
    "    'IteratorGetNext:4': np.expand_dims(np.load('../data/bert/input-4.npy'), axis=0)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[9.9877256e-01 1.2274694e-03]\n",
      " [4.8793713e-04 9.9951208e-01]\n",
      " [9.9892980e-01 1.0701367e-03]\n",
      " [9.9384588e-01 6.1541530e-03]]\n"
     ]
    }
   ],
   "source": [
    "predicted = client.predict(data_input)\n",
    "print(predicted)\n",
    "\n",
    "server.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}