17.06.05-Neural-networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my research, I have been using existing libraries to build neural network models. For example, [Keras](https://keras.io/) provides an extremely modular and clear Python API which sits on top of [Tensorflow](https://www.tensorflow.org/). It is very easy to train and experiment with neural networks, without needing to understand all the details under the hood. This was creating using the content of [Deep Learning Book](http://www.deeplearningbook.org/) chapter 6.\n",
    "\n",
    "Sometimes it's good to understand all of the details. I would like to better understand the backpropagation algorithm. As such, I am implementing a Multi Layer Perception from scratch using only Python and Numpy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural networks can be thought of as sequence of tranformations made to input data.\n",
    "\n",
    "$$y = f(x)$$\n",
    "\n",
    "Deep neural networks make very many such transformation, which creates complex representations of the data.\n",
    "\n",
    "$$y = f(f...f(f(x)))$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given examples of known $x$ and $y$, we want to define a mapping between them which reliably takes as input $x$ values and outputs values in $y$. Further, we want to capture some underlying structure that relates $x$ and $y$, meaning if another example $x^*$ are sampled in the same way as $x$, and the model has not seen it during training, we can predict ahead of time what the corresponding $y^*$ would be."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What form does this tranformation $f$ take?\n",
    "\n",
    "We take:\n",
    "\n",
    "$$ f = g(0, w^Tx + c) $$\n",
    "\n",
    "where $w$ and $c$ are vectors. $w^Tx + c$ constitutes what is known as an [affine map](https://en.wikipedia.org/wiki/Affine_transformation). Affine maps are transformations that preserve points, lines and planes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g(z) = $max(0,z)$ is a known as the [Relu](https://en.wikipedia.org/wiki/Rectifier_(neural_networks) operator. This simple non-linear operation allows the series of functions $y = f(f...f(f(x)))$ to become non-linear. Note that if each $f$ only comprised the affine map $w^Tx + c$, then the resulting composition $y = f(f...f(f(x)))$ would always be linear."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note this this is similar to the formulation of logistic regression. In logistic regression we find vectors $w$ and $c$ such that:\n",
    "\n",
    "$$y = \\sigma(w^T x + c)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "minimises the cross-entropy of the true and predicted distributions. Minimising this cross entropy defines the cost function. Cross entropy is defined as:\n",
    "\n",
    "$$H(p, q) = - \\sum_i p_i(x) \\log q_i(x) = \\mathop{{}\\mathbb{E}}[-\\log q] = H(p) + D_{KL}(p\\|q) $$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $H(p) = -\\sum_i p_i\\log p_i$ is the [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory) of $p$ and $D_{KL}(p\\|q) = \\sum_i p_i\\log \\frac{p_{i}}{q_{i}}$ is the [Kullback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) of $p$ given $q$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of our real values $y$ and our predicted values $y'$. Since we are predicting probabilities, $p \\in \\{y, 1-y\\}$ and $q \\in \\{y', 1-y'\\}$ the cross-entropy loss function is defined by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$L(y,y') = - \\sum_i p_i(x) \\log q_i(x) = -\\bigl(y \\log y' + (1-y) \\log (1-y')\\bigr)$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Chain Rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In calculus, the chain rule is used to define the derivate of composite functions. If $y = g(x)$ and $z = f(g(x)) = f(y)$, then the chain rule states that:\n",
    "\n",
    "$$ \\frac{dz}{dx} = \\frac{df(g(x))}{dx} = \\frac{df(y)}{dx} =  \\frac{df}{dy}\\frac{dy}{dx} = \\frac{dz}{dy}\\frac{dy}{dx} $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This generalizes beyond the scalar case. Suppose that $x \\in \\mathbb{R}^m$, $y \\in \\mathbb{R}^n$. Let $g$ be a mapping from $\\mathbb{R}^m$ to $\\mathbb{R}^n$, and let $f$ map from $\\mathbb{R}^n$ to $\\mathbb{R}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then,\n",
    "\n",
    "$$ \\frac{\\partial z}{\\partial x_i} = \\sum_{j}\\frac{\\partial z}{\\partial y_j}\\frac{\\partial y_j}{\\partial x_i}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In vector notation this is represented by:\n",
    "\n",
    "$$\\nabla_\\mathbf{x} z = \\Bigl( \\frac{\\partial \\mathbf{y}}{\\partial \\mathbf{x}} \\Bigr)^T \\nabla_{\\mathbf{x}} z$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $\\frac{\\partial \\mathbf{y}}{\\partial \\mathbf{x}}$ is the $n \\times m$ Jacobian matrix of $g$. We read this as the gradient of $z$ with respect to the vector $\\mathbf{x}$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see from this that we obtain the gradient of a variable $\\mathbf{x}$ by multiplying a Jacobian matrix $\\frac{\\partial \\mathbf{y}}{\\partial \\mathbf{x}}$ by a gradient $\\Delta_{\\mathbf{x}} z$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the context of deep learning we do not just operate on vectors, but also on tensors. We can compute the gradient of $z$ with respect to a tensor $X$ as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\nabla_\\mathbf{x} z = \\sum_j (\\nabla_{\\mathbf{X}} Y_j) \\frac{\\partial z}{\\partial Y_j}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, $(\\nabla_{\\mathbf{X}} z)_i$ corresponds to $\\frac{\\partial z}{\\partial X_i}$. For a vector, this enumerates all possible elements in the vector. For tensors, we just flatten across all the axis to achieve one long vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Forward propagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now describe the propagation algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Algorithm**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$l$ Network depth. <br>\n",
    "$\\mathbf{W}^{(i)}, i \\in \\{1,...l\\}$ weight parameters <br>\n",
    "$\\mathbf{b}^{(i)}, i \\in \\{1,...l\\}$ bias parameters <br>\n",
    "$\\mathbf{x}$ input\n",
    "$\\mathbf{y}$ target output <br>\n",
    "\n",
    "* $\\mathbf{h}^{(0)}$ = $x$\n",
    "* **for** $k = 1, ..., l$ **do**\n",
    "    * $\\mathbf{a}^{(k)} = \\mathbf{b}^{(k)} + \\mathbf{W}^{(k)}h^{(k)}$\n",
    "    * $\\mathbf{h}^{(k)} = f(\\mathbf{a}^{(k)})$\n",
    "* **end for**\n",
    "* $\\mathbf{y'} = \\mathbf{h}^{(l)}$\n",
    "* J = $L(y',y) + \\lambda \\Omega(\\theta)$\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will implement a Multi Layer Perception with a single hidden layer. As such, $\\mathbf{W}$ and $\\mathbf{b}$ have two columns. The number of rows is defined by the number of hidden units, $h$ of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* $a_1 = W_1X + b_1 $\n",
    "* $h_1 = relu(a_1)$\n",
    "* $a_2 = W_2h_1 + b_2$\n",
    "* $y' = \\sigma(a_2)$\n",
    "* $e = L(y',y)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A forward pass on our data will take a matrix $X$ and produce a vector of predictions $Y'$. We defined a cost function based on the cross-entropy of $Y'$ with respect to $Y$, defined above. This gives us $e$ = $L(Y,Y') = -(Y_i \\log Y_i' + (1-Y_i) \\log (1-Y_i'))$. Note that $e$ is a vector in its current form.\n",
    "\n",
    "We would like to know what the gradient of $e$ is with respect to all of our parameters, and then update them by a small amount to minimise the cost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The algorithm for backpropagation from the Deep Learning book is terse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* $\\mathbf{g}$ = $\\nabla_{y'}J$ Compute gradient on final layer.\n",
    "* **for** $k = l, ..., 1$\n",
    "    * $\\mathbf{g} = \\nabla_{\\mathbf{a}^{(k)}}J = \\mathbf{g} \\odot f^{'}(\\mathbf{a}^{(k)})$ Gradient on layer output. Element-wise multiplication if $f$ is element-wise.\n",
    "    * $\\nabla_{\\mathbf{b}^{(k)}}J = \\mathbf{g} + \\lambda \\nabla_{\\mathbf{b}^{(k)}}\\Omega(\\theta)$ Compute gradients on biases\n",
    "    * $\\nabla_{\\mathbf{W}^{(k)}}J = \\mathbf{g}\\ \\mathbf{h}^{(k-1)\\top} + \\lambda \\nabla_{\\mathbf{W}^{(k)}}\\Omega(\\theta)$ Compute gradients on weights\n",
    "    * $ \\mathbf{g} = \\nabla_{\\mathbf{h}^{(k-1)}}J = \\mathbf{W}^{(k)\\top}$ Propagate gradients w.r.t lower-level hidden layer activations.\n",
    "* **end for**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a function diagram displaying our operations and intermediate values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\require{AMScd}$\n",
    "\\begin{CD}\n",
    "    X @>W_1, b_1>> a_1 @>relu>> h_1 @>W_2, h_2>> a_2 @>\\sigma>> Y' @>L>> e\n",
    "\\end{CD}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is, we would like expressions for: $\\frac{\\partial e}{\\partial W_2}, \\frac{\\partial e}{\\partial b_2}, \\frac{\\partial e}{\\partial W_1}, \\frac{\\partial e}{\\partial b_1}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the chain rule, we can see that:\n",
    "$$ \\frac{\\partial e}{\\partial W_2} = \\frac{\\partial e}{\\partial a_2}\\frac{\\partial a_2}{\\partial W_2} , \\ \\ \\\n",
    "\\frac{\\partial e}{\\partial a_2} = \\frac{\\partial e}{\\partial Y'}\\frac{\\partial Y'}{\\partial a_2}  $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we see that,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial e}{\\partial W_2} = \\frac{\\partial e}{\\partial Y'}\\frac{\\partial Y'}{\\partial a_2}\\frac{\\partial a_2}{\\partial W_2}, \\ \\ \\ \\frac{\\partial e}{\\partial b_2} = \\frac{\\partial e}{\\partial Y'}\\frac{\\partial Y'}{\\partial a_2}\\frac{\\partial a_2}{\\partial b_2}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, we need to find expressions for:\n",
    "$$\\frac{\\partial a_2}{\\partial b_2}, \\frac{\\partial a_2}{\\partial W_2}, \\frac{\\partial Y'}{\\partial a_2},   \\frac{\\partial e}{\\partial Y'} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, $$e = -\\bigl(Y \\log Y' + (1-Y) \\log (1-Y')\\bigr) \\implies \\frac{\\partial e}{\\partial Y'} = -\\Bigl(\\frac{Y}{Y'} - \\frac{(1-Y)}{(1-Y')}\\Bigr)$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, $$ Y' = \\sigma (a_2) \\implies \\frac{\\partial y'}{\\partial a_2} = \\sigma(a_2)(1 - \\sigma(a_2))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Substituting $Y_{i}' = \\sigma(a_{i2})$\n",
    "\n",
    "$$ \\frac{\\partial e}{\\partial a_2} = \\Bigl(\\frac{Y}{\\sigma(a_{2})} - \\frac{(1-Y)}{(1-\\sigma(a_{2}))}\\Bigr) \\sigma(a_{2})(1 - \\sigma(a_{2})) \n",
    "= \\Bigl((1 - \\sigma(a_{2}))Y - \\sigma(a_{2})(1-Y)\\Bigr) = (Y' - Y)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\frac{\\partial e}{\\partial a_2} = (Y' - Y)$ is a remarkably simple expression, given the complexities of our loss function $L$, and final activation function $\\sigma$. In fact, it has been chosen to be so simple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $\\frac{\\partial e}{\\partial a_2} = \\delta_3$. Let us first calculate $\\frac{\\partial e}{\\partial a_1}$ and name it $\\delta_2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial e}{\\partial a_1} = \\frac{\\partial e}{\\partial a_2}\\frac{\\partial a_2}{\\partial a_1} = \\delta_3 \\frac{\\partial a_2}{\\partial h_1} \\frac{\\partial h_1}{\\partial a_1} = \\delta_3 W_{2}^T \\frac{\\partial h_1}{\\partial a_1}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since $h_1 = relu(a_1), \\frac{\\partial h_1}{\\partial a_1}$ is $1 $ for $a_1 > 0$ and $0$ otherwise. Call this function $drelu(a_1)$. Therefore: \n",
    "\n",
    "$$\\delta_2 = \\delta_3 W_{2}^T \\circ drelu(a_1)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now write down expressions for $\\frac{\\partial e}{\\partial W_2}, \\frac{\\partial e}{\\partial b_2}, \\frac{\\partial e}{\\partial W_1}, \\frac{\\partial e}{\\partial b_1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial e}{\\partial W_2}  = \\frac{\\partial e}{\\partial a_2} \\frac{\\partial a_2}{\\partial W_2} = h_1^T\\delta_3$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and,\n",
    "\n",
    "$$\\frac{\\partial e}{\\partial b_2}  = \\frac{\\partial e}{\\partial a_2} \\frac{\\partial a_2}{\\partial b_2} = \\delta_3$$\n",
    "\n",
    "Also,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\\partial e}{\\partial W_1} = \\frac{\\partial e}{\\partial a_2}\\frac{\\partial a_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial W_1} = X^T \\circ \\delta_2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and\n",
    "\n",
    "$$\\frac{\\partial e}{\\partial b_1} = \\frac{\\partial e}{\\partial a_2}\\frac{\\partial a_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial b_1} = \\delta_2$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the breast cancer data dataset. This defines a set of 30 predicters that are used to classify malignant vs benign breast cancer tumours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "((569, 30), (569, 1))\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import scipy\n",
    "import sklearn.datasets\n",
    "cancer_data = sklearn.datasets.load_breast_cancer()\n",
    "Y = cancer_data.target.reshape(569,1)\n",
    "X = cancer_data.data\n",
    "X = (X - np.mean(X,axis=0)) / np.std(X,axis=0)\n",
    "print (X.shape, Y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We randomly initialise the weight and bias parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def rand_init(s,h):\n",
    "    return (np.random.random(size=(s,h)) - 0.5)/10\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compute the forward pass where the network transforms the input $X$ input output $y$. We convert the final output of the neural network to a binary prediction using the $\\sigma$ function.\n",
    "\n",
    "$$ \\sigma(x) = \\frac{1}{1 + \\exp(-x)} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "code_folding": [],
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "W = {1: rand_init(30,5), 2: rand_init(5,1)}\n",
    "h = {}\n",
    "b = {1: rand_init(1,5), 2: rand_init(1,1)}\n",
    "a = {}\n",
    "def forward_pass(x):\n",
    "\n",
    "    h[0] = X\n",
    "    for k in [1,2]:\n",
    "        a[k] = b[k] + np.dot(h[k-1],W[k])\n",
    "        h[k] = a[k].copy()\n",
    "        h[k][h[k] < 0] = 0\n",
    "    y_pred = h[2]\n",
    "    y_pred = sigmoid(y_pred)\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backward pass "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the input $\\mathbf{x}$, we need a target $\\mathbf{y}$. This computation yields the gradients on the activations $\\mathbf{a} ^{(k)}$ for each layer $k$. From the gradients on each layer, we can find the gradients on the parameters in each layer and incorporate them into a stochastic gradient descent update."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to define the cost function, $J$, as the cross-entropy between $y$ and $y'$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Algorithm**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def cost(y_real,y_pred):\n",
    "    return - y_real*np.log(y_pred) - (1-y_real)*np.log(1-y_pred)\n",
    "#     return np.square(y_real-y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivate of the $\\sigma$ function is:\n",
    "\n",
    "$$ \\frac{d \\sigma}{dx} = \\sigma(x)(1 - \\sigma(x))$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dsigmoid(x):\n",
    "    return sigmoid(x)*(1 - sigmoid(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivative of the Relu function is: $0$ for $x <= 0$, and $1$ for $x > 0$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def drelu(x):\n",
    "    return 1 if x > 0 else 0\n",
    "drelu = np.vectorize(drelu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def backward_pass(Y_pred):   \n",
    "    delta_3 = (Y_pred-Y) # Compute delta_3 = de/da_2\n",
    "#     print (delta_3)\n",
    "    \n",
    "    #First layer updates\n",
    "    \n",
    "    db_2 = np.sum(delta_3, axis=0, keepdims=True)\n",
    "    dW_2 = h[1].T.dot(delta_3)\n",
    "    \n",
    "    delta_2 = np.multiply(delta_3.dot(W[2].T), drelu(a[1]))\n",
    "    dW_1 = X.T.dot(delta_2)\n",
    "    db_1 = np.sum(delta_2, axis=0, keepdims=True)\n",
    "\n",
    "    W[1] -= learning_rate * dW_1\n",
    "    b[1] -= learning_rate * db_1\n",
    "\n",
    "    W[2] -= learning_rate * dW_2\n",
    "    b[2] -= learning_rate * db_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "code_folding": [],
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "224414.024325\n",
      "223403.38667\n",
      "221132.091974\n",
      "219488.365817\n",
      "218368.97003\n"
     ]
    }
   ],
   "source": [
    "learning_rate = 0.000001\n",
    "W = {1: rand_init(30,5), 2: rand_init(5,1)}\n",
    "h = {}\n",
    "b = {1: rand_init(1,5), 2: rand_init(1,1)}\n",
    "a = {}\n",
    "\n",
    "def train():\n",
    "    for e in range(5000):\n",
    "        Y_pred = forward_pass(X)\n",
    "        backward_pass(Y_pred)\n",
    "        if e% 1000 == 0:\n",
    "            print (np.sum(cost(Y, predict(X))))        \n",
    "def predict(X):\n",
    "    predictions = []\n",
    "    for i in range(X.shape[0]):\n",
    "        predictions.append(forward_pass(X[i,:])[0][0])\n",
    "    return np.array(predictions)\n",
    "\n",
    "train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601,  0.55036601,\n",
       "        0.55036601,  0.55036601,  0.55036601,  0.55036601])"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict(X)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:deep]",
   "language": "python",
   "name": "conda-env-deep-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": false,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}