3_Tensorflow_programming_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cKf3AUdjAjms",
    "nbpresent": {
     "id": "6ab1abd5-af7f-4e9e-96f1-0cf6c9d1b158"
    }
   },
   "source": [
    "# Tensorflow\n",
    "\n",
    "When starting off with deep learning, one of the first questions to ask is, which framework to learn?\n",
    "\n",
    "Common choices include TensorFlow, PyTorch, and Keras. All of these choices have their own pros and cons and have their own way of doing things.\n",
    "\n",
    "> From [**The Anatomy of Deep Learning Frameworks**](https://medium.com/@gokul_uf/the-anatomy-of-deep-learning-frameworks-46e2a7af5e47#.3ywhrk1st)\n",
    "\n",
    "> The core components of a deep learning framework we must consider are:\n",
    "\n",
    "> + How **Tensor Objects** are defined. At the heart of the framework is the tensor object. A tensor is a generalization of a matrix to n-dimensions. We need a Tensor Object that supports storing the data in form of tensors. Not just that, we would like the object to be able to convert other data types (images, text, video) into tensors and back, supporting indexing, overloading operators, having a space efficient way to store the data and so on.\n",
    "+ How **Operations** on the Tensor Object are defined. A neural network can be considered as a series of Operations performed on an input tensor to give an output. \n",
    "+ The use of a **Computation Graph and its Optimizations**. Instead of implementing operations as functions, they are usually implemented as **classes**. This allows us to store more information about the operation like calculated shape of the output (useful for sanity checks), how to compute the gradient or the gradient itself (for the auto-differentiation), have ways to be able to decide whether to compute the op on GPU or CPU and so on. The power of neural networks lies in the ability to chain multiple operations to form a powerful approximator. Therefore, the standard use case is that you can initialize a tensor, perform actions after actions on them and finally interpret the resulting tensor as labels or real values. Unfortunately, as you chain more and more operations together, several issues arise that can drastically slow down your code and introduce bugs as well. There are more such issues and it becomes necessary to be able to get a bigger picture to even notice that these issues exist. We need a way to optimize the resultant chain of operations for both space and time. A Computation Graph which is basically an object that contains links to the instances of various Ops and the relations between which operation takes the output of which operation as well as additional information. \n",
    "+ The use of **Auto-differentiation** tools. Another benefit of having the computational graph is that calculating gradients used in the learning phase becomes modular and straightforward to compute. \n",
    "+ The use of **BLAS/cuBLAS and cuDNN** extensions for maximizing performance. BLAS or Basic Linear Algebra Subprograms are a collection of optimized matrix operations, initially written in Fortran. These can be leveraged to do very fast matrix (tensor) operations and can provide significant speedups. There are many other software packages like Intel MKL, ATLAS which also perform similar functions. BLAS packages are usually optimized assuming that the instructions will be run on a CPU. In the deep learning situation, this is not the case and BLAS may not be able to fully exploit the parallelism offered by GPUs. To solve this issue, NVIDIA has released cuBLAS which is optimized for GPUs. This is now included with the CUDA toolkit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The computational model for Tensorflow (`tf`) is a **directed graph**.\n",
    "\n",
    "**Nodes** are *functions* (*operations* in `tf` terminology) and **edges** are *tensors*. \n",
    "\n",
    "**Tensor** are multidimensional data arrays. \n",
    "\n",
    "$$f(a,b) = (a*b) + (a+b)$$\n",
    "\n",
    "![alt text](https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/t1.png?raw=1)\n",
    "\n",
    "There are several reasons for this design:\n",
    "+ The most important is that is a good way to split up computation into small, **easily differentiable** pieces. `tf` uses automatic differentiation to automatically compute the derivative of every node with respect any other node that can affect the first node's output.\n",
    "+ The graph is also a convenient way for distributing computation across multiple CPUs, GPUs, etc.\n",
    "\n",
    "The primary API of `tf` (written in C++) is accessed through Python.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3kCv6uNhAjmv",
    "nbpresent": {
     "id": "20019a20-ae7f-4beb-8cd4-d68150382b1e"
    }
   },
   "source": [
    "## Fundamentals\n",
    "\n",
    "Tensorflow approaches series of computations as a flow of data through a graph with nodes being computation units and edges being flow of Tensors (multidimensional arrays).\n",
    "\n",
    "Tensorflow builds the computation graph before it starts execution, so the computations are scheduled only when it is absolutely necessary (lazy programming).\n",
    "\n",
    "TensorFlow comes with a tool, TensorBoard, to visualize the computation graph.\n",
    "\n",
    "`tf` computation graphs are described in code with `tf` API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:08:04.523201Z",
     "start_time": "2019-03-05T09:08:01.208530Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "-L-ABfFnAjmv",
    "nbpresent": {
     "id": "6937c67a-43fa-4f1e-ab3d-89a199c83eb0"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.12.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Python `with` statement (context manager) is useful when you have two related operations which you’d like to execute as a pair, with a block of code in between. The classic example is opening a file, manipulating the file, then closing it:\n",
    "\n",
    ">```python\n",
    "with open('output.txt', 'w') as f:\n",
    "    f.write('Hi!')\n",
    ">```\n",
    "\n",
    "> The above `with` statement will automatically close the file after the nested block of code. The advantage of using a `with` statement is that it is guaranteed to close the file no matter how the nested block exits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:15:58.098283Z",
     "start_time": "2019-03-05T09:15:58.077783Z"
    },
    "code_folding": [],
    "colab": {},
    "colab_type": "code",
    "id": "e5RjdbW8Ajmy",
    "nbpresent": {
     "id": "aae85c45-c89c-401a-9131-01e07651e0af"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a= 2\n",
      "b= 3\n",
      "(a+b)+(a*b) = 11\n"
     ]
    }
   ],
   "source": [
    "# Basic constant operations = to assign a value to a tensor\n",
    "\n",
    "a = tf.constant(2)\n",
    "b = tf.constant(3)\n",
    "c = a+b \n",
    "d = a*b\n",
    "e = c+d\n",
    "\n",
    "# non interactive session\n",
    "# the context manager will automatically close the session\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(\"a= %i\" % sess.run(a))\n",
    "    print(\"b= %i\" % sess.run(b))\n",
    "    print(\"(a+b)+(a*b) = %i\" % sess.run(e))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sess.run(node)` executes the part of the computational graph that is needed to compute the value of `node` and only that part. While defining the graph, we are not manipulating any data, only building the nodes and symbols inside our graph.\n",
    "\n",
    "We can use `tf.get_default_graph().get_operations()` to see all the nodes in the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:14:33.706562Z",
     "start_time": "2019-03-05T09:14:33.699080Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'Const' type=Const>,\n",
       " <tf.Operation 'Const_1' type=Const>,\n",
       " <tf.Operation 'add' type=Add>,\n",
       " <tf.Operation 'mul' type=Mul>,\n",
       " <tf.Operation 'add_1' type=Add>]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.get_default_graph().get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "66Gd_bfpAjm0"
   },
   "source": [
    "You can create initialized tensors in many ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T15:47:44.792748Z",
     "start_time": "2019-03-04T15:47:44.757881Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "tdXwko77Ajm1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 0]\n",
      " [0 0 0]]\n",
      "[[1 1 1]\n",
      " [1 1 1]]\n",
      "[[23.9 23.9 23.9]\n",
      " [23.9 23.9 23.9]\n",
      " [23.9 23.9 23.9]]\n",
      "[0 1 2 3 4 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "a = tf.zeros([2,3], tf.int32)\n",
    "b = tf.ones([2,3], tf.int32)\n",
    "c = tf.fill([3,3], 23.9)\n",
    "d = tf.range(0,10,1)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(a))\n",
    "    print(sess.run(b))\n",
    "    print(sess.run(c))\n",
    "    print(sess.run(d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x5Z3H_tEAjm3"
   },
   "source": [
    "``tf`` sequences are not iterable!\n",
    "\n",
    "We can also generate random variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T15:47:46.294524Z",
     "start_time": "2019-03-04T15:47:46.265326Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "ic-M439_Ajm4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.83202136 -1.335937  ]\n",
      " [ 1.834337   -0.33183086]]\n",
      "[[0.3809768  0.12681663]\n",
      " [0.5305053  0.40903056]]\n"
     ]
    }
   ],
   "source": [
    "a = tf.random_normal([2,2], 0.0, 1.0)\n",
    "b = tf.random_uniform([2,2], 0.0, 1.0)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(a))\n",
    "    print(sess.run(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I_SM65uPAjm5"
   },
   "source": [
    "How to generate random shuffled number in tensorflow?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:20:36.032907Z",
     "start_time": "2019-03-05T09:20:36.007443Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "x5rWjqI-Ajm6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]\n",
      "[ 5  8 10 15  9 12 18  2  6 11 13  4  3 19  0 14 17  1  7 16]\n"
     ]
    }
   ],
   "source": [
    "idx = tf.constant(20)\n",
    "idx_list = tf.range(idx) # 0~19\n",
    "shuffle = tf.random_shuffle(idx_list)\n",
    "\n",
    "# in this case tf returns, in a list, two diferent results\n",
    "with tf.Session() as sess:\n",
    "    a, b = sess.run([idx_list, shuffle])\n",
    "    print(a)\n",
    "    print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Essentially, TensorFlow computation graph contains the following parts:\n",
    "\n",
    "+ **Placeholders**, variables used in place of inputs to feed to the graph\n",
    "+ **Variables**, model variables that are going to be optimized to make model perform better\n",
    "+ **Model**, a mathematical function that calculates output based on placeholder and model variables\n",
    "\n",
    "When used as an optimization engine, we will also have:\n",
    "+ **Loss Measure**, guide for optimization of model variables\n",
    "+ **Optimization Method**, update method for tuning model variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T15:47:49.671805Z",
     "start_time": "2019-03-04T15:47:49.639426Z"
    },
    "code_folding": [],
    "colab": {},
    "colab_type": "code",
    "id": "GfeJ94GWAjm7",
    "nbpresent": {
     "id": "3fd06df9-1eec-447c-bed7-d88be2d8d4d9"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a = 5\n",
      "b = 3\n",
      "(a+b)+(a*b) = 23\n"
     ]
    }
   ],
   "source": [
    "# Basic operations with variable graph input\n",
    "\n",
    "a = tf.placeholder(tf.int16)\n",
    "b = tf.placeholder(tf.int16)\n",
    "c = tf.add(a,b) \n",
    "d = tf.multiply(a,b)\n",
    "e = tf.add(c,d)\n",
    "\n",
    "values = feed_dict={a: 5, b: 3}\n",
    "\n",
    "# non interactive session\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print('a = %i' % sess.run(a, values))\n",
    "    print('b = %i' % sess.run(b, values))\n",
    "    print(\"(a+b)+(a*b) = %i\" % sess.run(e, values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LMSbJ3ydAjm9",
    "nbpresent": {
     "id": "92384d6d-bed6-4e4e-bfc1-ddf88de0963e"
    }
   },
   "source": [
    "A computational graph is a series of functions chained together, each passing its output to zero, one or more functions further along the chain.\n",
    "\n",
    "In this way we can construct very complex transformations on data by using a library of simple functions.\n",
    "\n",
    "Nodes represent some sort of computation being done in the graph context.\n",
    "\n",
    "Edges are the actual values (tensors) that get passed to and from nodes.\n",
    "\n",
    "![alt text](https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/t2.png?raw=1)\n",
    "\n",
    "+ The values flowing into the graph can come from different sources: from a different graph, from a file, entered by the client, etc. The *input* nodes simply pass on values given to them.\n",
    "+ The other nodes take values, apply an operation and output their result. \n",
    "\n",
    "Values running on edges are tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T15:47:51.354404Z",
     "start_time": "2019-03-04T15:47:51.327202Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "nnUc1We8Ajm-",
    "nbpresent": {
     "id": "a4071b69-ab10-4ccd-93ed-b445d8ba7299"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2]\n",
      "[3 3]\n",
      "[11 11]\n"
     ]
    }
   ],
   "source": [
    "# Basic operations with variable as graph input\n",
    "\n",
    "a = tf.placeholder(tf.int16,shape=[2])\n",
    "b = tf.placeholder(tf.int16,shape=[2])\n",
    "c = tf.add(a,b) \n",
    "d = tf.multiply(a,b)\n",
    "e = tf.add(c,d)\n",
    "\n",
    "variables = feed_dict={a: [2,2], b: [3,3]}\n",
    "\n",
    "# non interactive session\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(a, variables))\n",
    "    print(sess.run(b, variables))\n",
    "    print(sess.run(e, variables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LKKR-JUUAjnB",
    "nbpresent": {
     "id": "e8e3c449-56f2-4b1e-9112-6b013cfc6ec6"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Implement this computational graph:\n",
    "\n",
    "![alt text](https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/t3.png?raw=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T15:47:54.344869Z",
     "start_time": "2019-03-04T15:47:54.342769Z"
    },
    "code_folding": [],
    "colab": {},
    "colab_type": "code",
    "id": "FftsweNpAjnB",
    "nbpresent": {
     "id": "77252053-4889-4a38-a497-49465a2e9ac1"
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6gD413umAjnD",
    "nbpresent": {
     "id": "86c6424a-3e4a-4e6c-98f1-f9917cc659c8"
    }
   },
   "source": [
    "There are certein connections between nodes that are not allowed: you cannot create **circular dependencies**.\n",
    "\n",
    "> Dependency: Any node `A` that is required for the computation of a later node `B` is said to be a **dependency** of `B`.\n",
    "\n",
    "The main reason is that dependencies create endless feedback loops.\n",
    "\n",
    "There is one exception to this rule: *recurrent neural networks*. In this case `tf` simulate this kind of dependences by copying a **finite** number of versions of the graph, placing them side-by-side, and feeding them into another sequence. This process is referred as **unrolling** the graph.\n",
    "\n",
    "![alt text](https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/t4.png?raw=1)\n",
    "\n",
    "Keeping track of dependencies is a basic feature of `tf`. Let's suppose that we want to compute the output value of the `mul` node. We can see in the unrolled graph that is not necessary to compute the full graph to get the output of that node. But how to ensure that we only compute the necessary nodes?\n",
    "\n",
    "It's pretty easy:\n",
    "+ Build a list for each node with all nodes it directly depends on (not indirectly!). \n",
    "+ Initialize an empty stack, wich will eventually hold all the nodes we want to compute. \n",
    "+ Put the node you want to get the output from. \n",
    "+ Recursively, look at its dependency list and add to the stack the nodes it depends on, until there are no dependencies left to run and in this way we guarantee that we have all the nodes we need.\n",
    "\n",
    "The stack will be ordered in a way that we are guaranteed to be able to run each node in the stack as we iterate through it. \n",
    "\n",
    "The main thing to look out for is to keep track of nodes that were already computed and to store their value in memory.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yeHj0jDMAjnF",
    "nbpresent": {
     "id": "f3a49756-9e22-421c-a0d3-8bc670856ab0"
    }
   },
   "source": [
    "## `tf` workflows\n",
    "\n",
    "As we have seen in previous code, `tf` workflow is a two-step process:\n",
    "\n",
    "+ Define the computation graph.\n",
    "+ Run the graph with data.\n",
    "\n",
    "This can be done in a *non interactive* mode or in an *interactive* mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T15:47:56.899574Z",
     "start_time": "2019-03-04T15:47:56.873491Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "Uxfu2uWMAjnF",
    "nbpresent": {
     "id": "24af87db-9051-4768-9064-079dc5e9b965"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'input_a' type=Placeholder>,\n",
       " <tf.Operation 'input_b' type=Placeholder>,\n",
       " <tf.Operation 'add_1' type=Add>,\n",
       " <tf.Operation 'mul_1' type=Mul>,\n",
       " <tf.Operation 'add_2' type=Add>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new graph definition\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# we can assign a name to every node\n",
    "a = tf.placeholder(tf.int32, name='input_a')\n",
    "b = tf.placeholder(tf.int32, name='input_b')\n",
    "c = tf.add(a,b,name='add_1') \n",
    "d = tf.multiply(a,b,name='mul_1')\n",
    "e = tf.add(c,d,name='add_2')\n",
    "\n",
    "values = feed_dict={a: 5, b: 3}\n",
    "\n",
    "# now we can run the graph in an interactive session\n",
    "sess = tf.Session()\n",
    "print(sess.run(e, values))\n",
    "\n",
    "# it is our responsability to close the session\n",
    "sess.close()\n",
    "\n",
    "tf.get_default_graph().get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E-MSK9L7AjnH",
    "nbpresent": {
     "id": "efe7c304-c8de-4d24-8bd0-dbfa078133b5"
    }
   },
   "source": [
    "`tf` has a very useful tool: `tensor-board`. Let's see how to use it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:28:44.233732Z",
     "start_time": "2019-03-05T09:28:44.179164Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "tfkXlssXAjnI",
    "nbpresent": {
     "id": "c14452d1-3454-4704-b59a-e18b8251103b"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23\n"
     ]
    }
   ],
   "source": [
    "# cleaning the tf graph space\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.placeholder(tf.int16, name='input_a')\n",
    "b = tf.placeholder(tf.int16, name='input_b')\n",
    "c = tf.add(a,b,name='add_1') \n",
    "d = tf.multiply(a,b,name='mul_1')\n",
    "e = tf.add(c,d,name='add_2')\n",
    "\n",
    "values = feed_dict={a: 5, b: 3}\n",
    "\n",
    "# now we can run the graph\n",
    "\n",
    "# graphs are run by invoking Session objects\n",
    "session = tf.Session()\n",
    "\n",
    "# when you are passing an operation to 'run' you are \n",
    "# asking to run all operations necessary to compute that node\n",
    "\n",
    "# you can save the value of the node in a Python var\n",
    "output = session.run(e, values)\n",
    "print(output)\n",
    "\n",
    "# now let's visualize the graph\n",
    "\n",
    "# SummaryWriter is an object where we can save information\n",
    "# about the execution of the computational graph\n",
    "writer = tf.summary.FileWriter('my_graph', session.graph)\n",
    "writer.close()\n",
    "\n",
    "# closing interactive session\n",
    "session.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "EdGgjki5AjnK",
    "nbpresent": {
     "id": "33897391-05a6-4b4e-a235-9e162427f843"
    }
   },
   "source": [
    "Open a terminal, go to your working dir, and type in:\n",
    "\n",
    "`tensorboard --logdir=\"my_graph\"`\n",
    "\n",
    "This starts a `tensorboard` server on port 6006. There, click on the `Graphs` link. You can see that each of the nodes is labeled based on the `name` parameter you passed into each operation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "QPCXd0LcAjnL",
    "nbpresent": {
     "id": "757d1ac9-70da-430a-819a-903afd30e0bb"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Implement and visualize this graph for a constant tensor `[5,3]`:\n",
    "\n",
    "![alt text](https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/t5.png?raw=1)\n",
    "\n",
    "Check these functions in the `tf` official documentation (https://www.tensorflow.org/): `tf.math.reduce_prod`, `tf.math.reduce_sum`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "siL8qrPtAjnM",
    "nbpresent": {
     "id": "0cb9c1e0-21d6-4dfd-bfbe-582238143402"
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tf` statements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E152EKBoAjnN",
    "nbpresent": {
     "id": "1acdc391-57ed-4c09-854d-b7c8cb07701b"
    }
   },
   "source": [
    "### `tf` input data \n",
    "\n",
    "`tf` can take several Python var types that are automatically converted to tensors:\n",
    "\n",
    "`tf.constant([5,3], name='input_a')`\n",
    "\n",
    "But `tf` has a plethora of other data types: `tf_int16`, `tf_quint8`, etc.\n",
    "\n",
    "`tf` is tightly integrated with NumPy. In fact, `tf` data types are based on those from NumPy. Tensors returned from `Session.run` are NumPy arrays. NumPy arrays is the recommended way of specifying tensors.\n",
    "\n",
    "The `shape` of tensors describe both the number of dimensions in a tensor  as well as the length of each dimension. In addition to to being able to specify fixed lengths to each dimension, in some situations you can also assign a flexible length by passing in `None` as dimension's value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:30:58.338503Z",
     "start_time": "2019-03-05T09:30:58.327843Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "ZIes75o6AjnO",
    "nbpresent": {
     "id": "a637e0a1-a62b-49a1-99e2-d8eeb974fada"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.placeholder(tf.int16, shape=[2,2], name='input_a')\n",
    "shape = tf.shape(a)\n",
    "\n",
    "session = tf.Session()\n",
    "print(session.run(shape))\n",
    "session.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jGCVowRKAjnQ"
   },
   "source": [
    "We can feed data points to placeholder by iterating through the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T15:48:16.498111Z",
     "start_time": "2019-03-04T15:48:16.483258Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "Q9S_j5-LAjnQ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "4\n",
      "6\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "list_a_values = [1,2,3]\n",
    "\n",
    "a = tf.placeholder(tf.int16)\n",
    "b = a * 2\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    for a_value in list_a_values:\n",
    "        print(sess.run(b,{a: a_value}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y0OgxUQwAjnS",
    "nbpresent": {
     "id": "c4c8921f-ec0b-4302-bd4e-23251c7cd7de"
    }
   },
   "source": [
    "### `tf` operations\n",
    "\n",
    "`tf` overloads common mathematical operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:33:37.023890Z",
     "start_time": "2019-03-05T09:33:37.003499Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "2U9iE5MhAjnU",
    "nbpresent": {
     "id": "65a196d3-5416-4ccc-923c-29c06466427c"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(a+b)+(a*b) = 23\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.placeholder(tf.int16)\n",
    "b = tf.placeholder(tf.int16)\n",
    "c = a+b \n",
    "d = a*b\n",
    "e = tf.math.add(c,d) #equivalent to c+d\n",
    "\n",
    "variables = feed_dict={a: 5, b: 3}\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(\"(a+b)+(a*b) = %i\" % sess.run(e, variables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LeHoB5xDAjnX"
   },
   "source": [
    "There are more [Tensorflow Operations](https://www.tensorflow.org/api_guides/python/math_ops)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "THZN4O8xAjnX",
    "nbpresent": {
     "id": "6c4f248a-2039-4d50-946e-934257e12d29"
    }
   },
   "source": [
    "### `tf` graphs\n",
    "\n",
    "Creating a graph is simple:\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "g = tf.Graph()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "39TrGwdhAjnZ",
    "nbpresent": {
     "id": "3d475195-dd8a-4ce5-8e90-f7bcd42008f4"
    }
   },
   "source": [
    "Once the graph is initialized we can attach operation to it by using the `Graph.as_default()` method:\n",
    "\n",
    "```python\n",
    "with g.as_default():\n",
    "    a = tf.mul(2,3)\n",
    "    ...\n",
    "```\n",
    "\n",
    "`tf` automatically creates a graph at the beginning and assigns it to be the default. Thus, if not using `Graph.as_default()` any operation will be automatically placed in the default graph.\n",
    "\n",
    "Creating multiple graphs can be useful if you are defining multiple models that do not have interdependencies:\n",
    "\n",
    "```python\n",
    "g1 = tf.Graph()\n",
    "g2 = tf.Graph()\n",
    "\n",
    "with g1.as_default():\n",
    "    ...\n",
    "    \n",
    "with g2.as_default():\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "collapsed": true,
    "id": "kuW8d0KDAjna",
    "nbpresent": {
     "id": "e0b483f6-ccce-4f20-b082-0546e7f5f1e7"
    }
   },
   "source": [
    "### `tf` Variables\n",
    "\n",
    "`Tensor` and `Operation` objects are **immutable**, but we need a mechanism to save changing values over time (persisting during several `run`). \n",
    "\n",
    "This is accomplished with `Variable` objects, which contain mutable tensor values that persist accross multiple calls to `Session.run()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sh5DJMPTAjnb",
    "nbpresent": {
     "id": "8f062cb0-836d-47b9-a5fd-a9291431e3dc"
    }
   },
   "source": [
    "Variables can be used anywhere you might use a tensor.\n",
    "\n",
    "`tf` has a number of helper operations to initialize variables: `tf-zeros()`, `tf_ones()`, `tf.random_uniform()`, `tf.random_normal()`, etc.\n",
    "\n",
    "`Variable` objects live in a `Graph` but their state is managed by `Session`. \n",
    "\n",
    "Because of these they need an extra step for inicialization (`tf.global_variables_initializer`):\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "b = tf.add(5,a)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    ...\n",
    "\n",
    "```\n",
    "\n",
    "In order to chage the value of a `Variable` we can use the `Variable.assign()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:45:42.017470Z",
     "start_time": "2019-03-05T09:45:41.965606Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "qyaYZ4usAjnc",
    "nbpresent": {
     "id": "e46c73f6-5607-4af4-b22f-41f3f4dcec9e"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a: 3\n",
      "b: 6\n",
      "b: 12\n",
      "b: 24\n",
      "a: 24\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "b = a.assign(tf.multiply(2,a))  \n",
    "\n",
    "# The statement a.assign(...) does not actually assign any to a, \n",
    "# but rather creates a tf.Operation that you have to explicitly \n",
    "# run to update the variable.\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(\"a:\", a.eval())   # variables are objects, not ops.\n",
    "    print(\"b:\", sess.run(b))\n",
    "    print(\"b:\", sess.run(b))\n",
    "    print(\"b:\", sess.run(b))\n",
    "    print(\"a:\", a.eval())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:45:44.027383Z",
     "start_time": "2019-03-05T09:45:44.000929Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "SwMJKI2rAjne"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "6\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "b = a.assign(tf.multiply(2,a))\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(a.eval())\n",
    "    print(b.eval())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rkufls0yAjnh",
    "nbpresent": {
     "id": "defd975d-f087-4747-9b2f-06ccde7ee725"
    }
   },
   "source": [
    "We can increment and decrement variables: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:44:46.473565Z",
     "start_time": "2019-03-05T09:44:46.410678Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "StzrNzbRAjni",
    "nbpresent": {
     "id": "0eb291f0-5427-44ad-b55b-08fd1ce2ef01"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "3\n",
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "a = tf.Variable(3,name=\"my_var\")\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(a.assign_add(1)))\n",
    "    print(sess.run(a.assign_sub(1)))\n",
    "    print(sess.run(a.assign_sub(1)))\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vi6Rp1d1Ajnj",
    "nbpresent": {
     "id": "ca99d481-e62a-45c3-960c-ad87b458d8cb"
    }
   },
   "source": [
    "Some classes of `tf` (f.e. `Optimizer`) are able to automatically change variable values without explicitely asking to do so."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C1blMCC2Ajnk"
   },
   "source": [
    "Tensorflow sessions maintain values separately, each `Session` can have its own current value for a variable defined in the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T16:02:00.566725Z",
     "start_time": "2019-03-04T16:02:00.527187Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "xSJiRmn4Ajnl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "tf.global_variables_initializer()\n",
    "\n",
    "a = tf.Variable(10)\n",
    "\n",
    "sess1 = tf.Session()\n",
    "sess2 = tf.Session()\n",
    "\n",
    "sess1.run(tf.global_variables_initializer())\n",
    "sess2.run(tf.global_variables_initializer())\n",
    "\n",
    "print(sess1.run(a.assign_add(10)))\n",
    "print(sess2.run(a.assign_sub(2)))\n",
    "\n",
    "sess1.close()\n",
    "sess2.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6KNKLwV8Ajnp",
    "nbpresent": {
     "id": "ee6da6ee-dd82-49bf-8a6f-b92a8413b396"
    }
   },
   "source": [
    "### `tf` name scopes\n",
    "\n",
    "`tf` offers a tool to help organize your graphs: name scopes.\n",
    "\n",
    "Name scopes allows you to group operations into larger, named blocks. This is very usefu to visualize complex models with `tensorboard`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-04T16:04:47.994759Z",
     "start_time": "2019-03-04T16:04:47.965723Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "oBjYGJjSAjnq",
    "nbpresent": {
     "id": "ea824159-e736-4e2a-b024-524e337f6fac"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.name_scope(\"Scope_A\"):\n",
    "    a = tf.add(1, 2, name=\"A_add\")\n",
    "    b = tf.multiply(a, 3, name=\"A_mul\")\n",
    "\n",
    "with tf.name_scope(\"Scope_B\"):\n",
    "    c = tf.add(4, 5, name=\"B_add\")\n",
    "    d = tf.multiply(c, 6, name=\"B_mul\")\n",
    "\n",
    "e = tf.add(b, d, name=\"output\")\n",
    "\n",
    "writer = tf.summary.FileWriter('./name_scope_1', graph=tf.get_default_graph())\n",
    "writer.close()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(e))\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8-kivwa_Ajns",
    "nbpresent": {
     "id": "236459a0-db19-4ca3-ab64-d36697e6180f"
    }
   },
   "source": [
    "We can start `tensorboard` to see the graph: `tensorboard --logdir=\"./name_scope_1\"`.\n",
    "\n",
    "You can expand the name scope boxes by clicking `+`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nxmgzvqeAjns",
    "nbpresent": {
     "id": "58f4f8ea-e41a-469a-a078-bb1e1725a1d3"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Let's built and visualize and complex model:\n",
    "\n",
    "+ Our inputs will be placeholders.\n",
    "+ The model will take in a single vector of any lenght.\n",
    "+ The graph will be segmented in name scopes.\n",
    "+ We will accumulate the total value of all outputs over time.\n",
    "+ At each run, we are going to save the output of the graph, the accumulated total of all outputs, and the average value of all outputs to disk for use in `tensorboard`.\n",
    "\n",
    "\n",
    "![alt text](https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/t6.png?raw=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "colab": {},
    "colab_type": "code",
    "id": "Ds0WRSRuAjnt",
    "nbpresent": {
     "id": "a8218e4e-4edc-48b5-a6a3-3af3169dee3f"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Explicitly create a Graph object\n",
    "graph = tf.Graph()\n",
    "\n",
    "with graph.as_default():\n",
    "    \n",
    "    with tf.name_scope(\"variables\"):\n",
    "    # your code here\n",
    "\n",
    "    \n",
    "    # Primary transformation Operations\n",
    "    with tf.name_scope(\"transformation\"):\n",
    "        \n",
    "        # Separate input layer\n",
    "        with tf.name_scope(\"input\"):\n",
    "        # your code here\n",
    "\n",
    "    \n",
    "        # Separate middle layer\n",
    "        with tf.name_scope(\"intermediate_layer\"):\n",
    "        # your code here\n",
    "\n",
    "        \n",
    "        # Separate output layer\n",
    "        # your code here\n",
    "        \n",
    "    with tf.name_scope(\"update\"):\n",
    "        # Increments the total_output Variable by the latest output\n",
    "        # your code here\n",
    "\n",
    "    \n",
    "    # Summary Operations\n",
    "    with tf.name_scope(\"summaries\"):\n",
    "        # Calculating average (avg = total/steps)\n",
    "        avg = tf.div(update_total, tf.cast(increment_step, tf.float32), name=\"average\")\n",
    "\n",
    "        # Creates summaries for output node\n",
    "        tf.summary.scalar(\"output_summary\", output)\n",
    "        tf.summary.scalar(\"total_summary\", update_total)\n",
    "        tf.summary.scalar(\"average_summary\", avg)\n",
    "    \n",
    "    # Global Variables and Operations\n",
    "    with tf.name_scope(\"global_ops\"):\n",
    "        # Initialization Op\n",
    "        init = tf.global_variables_initializer()\n",
    "        # Merge all summaries[…]\n",
    "        merged_summaries = tf.summary.merge_all()\n",
    "\n",
    "# Start a Session, using the explicitly created Graph\n",
    "sess = tf.Session(graph=graph)\n",
    "\n",
    "# Open a SummaryWriter to save summaries\n",
    "writer = tf.summary.FileWriter('./improved_graph', graph)\n",
    "\n",
    "# Initialize Variables\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SenCvLOQAjnu",
    "nbpresent": {
     "id": "76d05e86-509b-494a-8364-953c74c1aadb"
    }
   },
   "source": [
    "Let's write a function to run the graph several times:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CZ6ovCjKAjnu",
    "nbpresent": {
     "id": "ac2057e4-ca62-4985-af03-04ecaf7c41b5"
    }
   },
   "outputs": [],
   "source": [
    "def run_graph(input_tensor):\n",
    "    \"\"\"\n",
    "    Helper function; runs the graph with given input tensor and saves summaries\n",
    "    \"\"\"\n",
    "    feed_dict = {a: input_tensor}\n",
    "    out, step, summary = sess.run([output, increment_step, merged_summaries], \n",
    "                                  feed_dict=feed_dict)\n",
    "    writer.add_summary(summary, global_step=step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vz7F313iAjnw",
    "nbpresent": {
     "id": "47fcb7a3-7d43-4dfc-9b03-2fc208b165e6"
    }
   },
   "outputs": [],
   "source": [
    "# Run the graph with various inputs\n",
    "run_graph([2,8])\n",
    "run_graph([3,1,3,3])\n",
    "run_graph([8])\n",
    "run_graph([1,2,3])\n",
    "run_graph([11,4])\n",
    "run_graph([4,1])\n",
    "run_graph([7,3,1])\n",
    "run_graph([6,3])\n",
    "run_graph([0,2])\n",
    "run_graph([4,5,6])\n",
    "\n",
    "# Write the summaries to disk\n",
    "writer.flush()\n",
    "\n",
    "# Close the SummaryWriter\n",
    "writer.close()\n",
    "\n",
    "# Close the session\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NA6nfTKlAjnx",
    "nbpresent": {
     "id": "4adbbc73-2fc9-4f83-8c54-081b32a3375e"
    }
   },
   "source": [
    "To start TensorBoard after running this code, run the following command:\n",
    "\n",
    "`tensorboard --logdir='./improved_graph'`   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2U662MyeAjnx"
   },
   "source": [
    "## ``tf`` eager execution\n",
    "\n",
    "``Eager`` execution is an interface to TensorFlow that provides an imperative programming style. When you enable eager execution, TensorFlow operations execute immediately; you do not execute a pre-constructed graph with ``Session.run()``.\n",
    "\n",
    "First, we must enable ``Eager`` execution. When we do this, operations will execute and return their values immediately. Some things to note:\n",
    "\n",
    "+ We will need to restart the Python kernel since we have already used TensorFlow in graph mode.\n",
    "+ We enable eager at program startup using: tfe.enable_eager_execution().\n",
    "+ Once we enable Eager with ``tfe.enable_eager_execution()``, it cannot be turned off. To get back to graph mode, start a new Python session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:47:32.971373Z",
     "start_time": "2019-03-05T09:47:26.887305Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "Y9_ArUMaAjny",
    "outputId": "2f1907ac-b41e-4aaa-bef4-3e741c9cbdc5"
   },
   "outputs": [],
   "source": [
    "import tensorflow.contrib.eager as tfe\n",
    "import tensorflow as tf\n",
    "\n",
    "tfe.enable_eager_execution()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:47:33.778679Z",
     "start_time": "2019-03-05T09:47:33.771286Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "WXYfy1U5Ajn1",
    "outputId": "8141d47b-bc55-4f75-9cdc-e850c1b46130"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(3, shape=(), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "print(tf.add(1, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mJFI_iR1Ajn4"
   },
   "source": [
    "TensorFlow Eager execution provides an autograd style API for automatic differentiation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-05T09:47:37.535183Z",
     "start_time": "2019-03-05T09:47:37.516692Z"
    },
    "colab": {},
    "colab_type": "code",
    "id": "QD0aJgYMAjn6",
    "outputId": "cccbcd0b-f112-483b-b438-a91e76461dfa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f(4) = 19.00\n",
      "df(4) = 8.00\n",
      "d2f(4) = 2.00\n"
     ]
    }
   ],
   "source": [
    "def f(x):\n",
    "    # f(x) = x^2 + 3\n",
    "    return tf.multiply(x, x) + 3\n",
    "\n",
    "print( \"f(4) = %.2f\" % f(4.) )\n",
    "\n",
    "# First order derivative\n",
    "df = tfe.gradients_function(f)\n",
    "print( \"df(4) = %.2f\" % df(4.)[0] )\n",
    "\n",
    "# Second order derivative\n",
    "d2f = tfe.gradients_function(lambda x: df(x)[0])\n",
    "print( \"d2f(4) = %.2f\" % d2f(4.)[0] )"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "3. Tensorflow programming model.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "nteract": {
   "version": "0.12.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "222.475px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}