exercise_740.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backpropagation in a Multi-Layer Neural Network\n",
    "\n",
    "We explore backpropagation assuming the features are $\\begin{bmatrix}-10 & 20\\end{bmatrix}$ and the correct category is $1$ which is one-hot encoded as $\\begin{bmatrix}0 & 1\\end{bmatrix}$.\n",
    "\n",
    "The parameters of the multi-layer neural network (the weights and biases) which minimize the loss are discovered by descending the loss gradient.\n",
    "\n",
    "In exercises 710 through 730, we calculated the partial derivatives for a single-layer neural network.\n",
    "\n",
    "In this exercise, we compute partial derivatives for a multi-layer neural network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters (weights) 1st layer: \n",
      " 1  1\n",
      " 2  2\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "Parameters (weights) 2nd layer: \n",
      " 2  1\n",
      " 2  1\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "Parameters (bias) 1st layer: \n",
      "-1 -1\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "Parameters (bias) 2nd layer: \n",
      " 5  5\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "weights1 = torch.nn.Parameter(torch.Tensor([[1,1],[2,2]]))\n",
    "print(\"Parameters (weights) 1st layer: \"+str(weights1.data))\n",
    "\n",
    "weights2 = torch.nn.Parameter(torch.Tensor([[2, 1],[2,1]]))\n",
    "print(\"Parameters (weights) 2nd layer: \"+str(weights2.data))\n",
    "\n",
    "bias1 = torch.nn.Parameter(torch.Tensor([[-1,-1]]))\n",
    "print(\"Parameters (bias) 1st layer: \"+str(bias1.data))\n",
    "\n",
    "bias2 = torch.nn.Parameter(torch.Tensor([[5,5]]))\n",
    "print(\"Parameters (bias) 2nd layer: \"+str(bias2.data))\n",
    "\n",
    "inputs = torch.autograd.Variable(torch.Tensor([[-10, 20]]))\n",
    "\n",
    "target = torch.autograd.Variable(torch.LongTensor([1]))\n",
    "#print(target)\n",
    "\n",
    "one_hot_target = torch.autograd.Variable(torch.Tensor([[0, 1]]))\n",
    "#print(one_hot_target)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume the weights are $\\begin{bmatrix}1 & 1 \\\\ 2 & 2\\end{bmatrix}$ for the lower layer and $\\begin{bmatrix}2 & 1 \\\\ 2 & 1\\end{bmatrix}$ for the higher layer at the start.\n",
    "\n",
    "The training is assumed to proceed one data point at a time (batch size of 1).\n",
    "\n",
    "For this iteration of training, the features are $\\begin{bmatrix}-10 & 20\\end{bmatrix}$ and the correct category is $1$ which is one-hot encoded as $\\begin{bmatrix}0 & 1\\end{bmatrix}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a = Variable containing:\n",
      " 29  29\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "h = relu(a) = Variable containing:\n",
      " 29  29\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "c = Variable containing:\n",
      " 121   63\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "softmax(c) = Variable containing:\n",
      " 1.0000e+00  6.4702e-26\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "Cross entropy loss: Variable containing:\n",
      " 58\n",
      "[torch.FloatTensor of size 1]\n",
      "\n",
      "\tThe manually computed gradient of the loss with respect to the weights of the second layer is \n",
      " 29 -29\n",
      " 29 -29\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\tThe manually computed gradient of the loss with respect to the weights of the first layer is \n",
      "-10 -10\n",
      " 20  20\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\tThe manually computed gradient of the loss with respect to the biases of the second layer is \n",
      " 1 -1\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "\tThe manually computed gradient of the loss with respect to the biases of the first layer is \n",
      " 1  1\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "\tThe automatically computed gradient of the loss with respect to the weights of the second layer is \n",
      " 29 -29\n",
      " 29 -29\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\tThe automatically computed gradient of the loss with respect to the weights of the first layer is \n",
      "-10 -10\n",
      " 20  20\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\tThe automatically computed gradient of the loss with respect to the biases of the second layer is \n",
      " 1 -1\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n",
      "\tThe automatically computed gradient of the loss with respect to the biases of the first layer is \n",
      " 1  1\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass\n",
    "\n",
    "hidden = torch.mm(inputs, weights1) + bias1\n",
    "print(\"a = \"+str(hidden))\n",
    "\n",
    "hidden_relu = F.relu(hidden)\n",
    "print(\"h = relu(a) = \"+str(hidden_relu))\n",
    "\n",
    "result = torch.mm(hidden_relu, weights2) + bias2\n",
    "print(\"c = \"+str(result))\n",
    "\n",
    "softmax_result = F.softmax(result, dim=1)\n",
    "print(\"softmax(c) = \"+str(softmax_result))\n",
    "\n",
    "loss = F.cross_entropy(result, target)\n",
    "print(\"Cross entropy loss: \"+str(loss))\n",
    "\n",
    "# Backward Pass\n",
    "\n",
    "grad_softmax = (softmax_result.data - one_hot_target.data)\n",
    "grad_hidden_activation = grad_softmax.mm(weights2.data.t())\n",
    "grad_hidden_pre_activation = grad_hidden_activation.clone()\n",
    "grad_hidden_pre_activation[hidden.data < 0] = 0\n",
    "grad_weight2 = hidden.data.t().mm(grad_softmax)\n",
    "grad_bias2 = grad_softmax\n",
    "grad_weight1 = inputs.data.t().mm(grad_hidden_pre_activation)\n",
    "grad_bias1 = grad_hidden_pre_activation\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to the weights of the second layer is \"+str(grad_weight2))\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to the weights of the first layer is \"+str(grad_weight1))\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to the biases of the second layer is \"+str(grad_bias2))\n",
    "\n",
    "print(\"\\tThe manually computed gradient of the loss with respect to the biases of the first layer is \"+str(grad_bias1))\n",
    "\n",
    "# Autograd\n",
    "\n",
    "loss.backward()\n",
    "\n",
    "gradient = weights2.grad\n",
    "\n",
    "print(\"\\tThe automatically computed gradient of the loss with respect to the weights of the second layer is \"+str(gradient.data))\n",
    "\n",
    "gradient = weights1.grad\n",
    "\n",
    "print(\"\\tThe automatically computed gradient of the loss with respect to the weights of the first layer is \"+str(gradient.data))\n",
    "\n",
    "gradient = bias2.grad\n",
    "\n",
    "print(\"\\tThe automatically computed gradient of the loss with respect to the biases of the second layer is \"+str(gradient.data))\n",
    "\n",
    "gradient = bias1.grad\n",
    "\n",
    "print(\"\\tThe automatically computed gradient of the loss with respect to the biases of the first layer is \"+str(gradient.data))\n",
    "\n",
    "if weights1.grad is not None:\n",
    "    weights1.grad.data.zero_()\n",
    "if weights2.grad is not None:\n",
    "    weights2.grad.data.zero_()\n",
    "if bias1.grad is not None:\n",
    "    bias1.grad.data.zero_()\n",
    "if bias2.grad is not None:\n",
    "    bias2.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}