PyTorch_Vision_ChloeRobotics_Beginners.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## get PyTorch\n",
    "\n",
    "This is an adaptation of the Pytorch Tutorial\n",
    "\n",
    "Follow install instructions and see original tutorial here:\n",
    "\n",
    "http://pytorch.org/\n",
    "\n",
    "Why do we use PyTorch / Tensorflow for Deep learning?\n",
    "\n",
    "<img src=\"https://i.stack.imgur.com/Stmud.png\" >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.1.0\n",
      "('numpy: ', array([0., 1.]))\n",
      "('torch: ', tensor([0., 1.], dtype=torch.float64, requires_grad=True))\n",
      "('gradient', None)\n",
      "('d', tensor(2., dtype=torch.float64, grad_fn=<SumBackward0>))\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import numpy as np\n",
    "print(torch.__version__) # should be the latest version 0.4.0\n",
    "r = np.asarray([0.0,1.0])\n",
    "print(\"numpy: \",r)\n",
    "a = Variable(torch.from_numpy(r), requires_grad=True)\n",
    "print(\"torch: \",a)\n",
    "b = a**2\n",
    "c = a + 1\n",
    "d = torch.sum(c - b)\n",
    "e = c/2\n",
    "print(\"gradient\" ,a.grad) # should be none\n",
    "print(\"d\", d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$d = \\sum (a+1) - a^2$$\n",
    "\n",
    "$$\\frac{\\partial(d)}{\\partial(a_i)} = 1 - 2a_i$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2., 1.])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = np.asarray([2.0,1.0])\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 1.], dtype=torch.float64)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = torch.from_numpy(r)\n",
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1., dtype=torch.float64)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('gradient', tensor([ 1., -1.], dtype=torch.float64))\n"
     ]
    }
   ],
   "source": [
    "d.backward(retain_graph=True)\n",
    "print(\"gradient\" ,a.grad) # 1 - 2(0) = 1, 1 - 2(1) = -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "W = W - LR*Grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0., 1.], dtype=torch.float64, requires_grad=True)\n",
      "('gradient', tensor([ 1., -1.], dtype=torch.float64))\n",
      "tensor([-1.,  2.], dtype=torch.float64, requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "# create your optimizer\n",
    "optimizer = optim.SGD([a], lr=1.0)\n",
    "\n",
    "# in your training loop:\n",
    "print(a)\n",
    "optimizer.zero_grad()   # zero the gradient buffers\n",
    "d.backward()\n",
    "print(\"gradient\" ,a.grad)\n",
    "optimizer.step()    # Does the update\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The Data\n",
    "=====================\n",
    "\n",
    "For this tutorial, we will use the CIFAR10 dataset.\n",
    "It has the classes: ‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’,\n",
    "‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’. The images in CIFAR-10 are of\n",
    "size 3x32x32, i.e. 3-channel color images of 32x32 pixels in size.\n",
    "\n",
    ".. figure:: /_static/img/cifar10.png\n",
    "   :alt: cifar10\n",
    "\n",
    "   cifar10\n",
    "\n",
    "\n",
    "Training an image classifier\n",
    "----------------------------\n",
    "\n",
    "We will do the following steps in order:\n",
    "\n",
    "1. Load and normalizing the CIFAR10 training and test datasets using\n",
    "   ``torchvision``\n",
    "2. Define a Convolution Neural Network\n",
    "3. Define a loss function\n",
    "4. Train the network on the training data\n",
    "5. Test the network on the test data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Loading and normalizing CIFAR10\n",
    "----------------------------\n",
    "\n",
    "Using ``torchvision``, it’s extremely easy to load CIFAR10.\n",
    "\n",
    "The output of torchvision datasets are PILImage images of range [0, 1].\n",
    "We transform them to Tensors of normalized range [-1, 1].\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "transform = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
    "\n",
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n",
    "                                        download=True, transform=transform)\n",
    "\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,\n",
    "                                          shuffle=True, num_workers=2)\n",
    "\n",
    "testset = torchvision.datasets.CIFAR10(root='./data', train=False,\n",
    "                                       download=True, transform=transform)\n",
    "\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=4,\n",
    "                                         shuffle=False, num_workers=2)\n",
    "\n",
    "classes = ('plane', 'car', 'bird', 'cat',\n",
    "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us show some of the training images, for fun.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  dog truck   dog plane\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAB6CAYAAACvHqiXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztfWmUXVd15nfe/F7Nk2rSUCrNsmTJtjwhY8BgMIbYkBBCkpWGBNr5kV6dsMhKQ1jdCWt1usPqXiTprEDHAYKTEMAhEBw3s7FjAx6QJ9mSLGsulapUVaq56s3vnf6x97l7v6pXUmlAqqqcby27ns6979xzzzn3vr33twdjrYWHh4eHx/JH6FoPwMPDw8PjysC/0D08PDxWCPwL3cPDw2OFwL/QPTw8PFYI/Avdw8PDY4XAv9A9PDw8Vgj8C93Dw8NjheCyXujGmHuMMYeNMUeNMR+/UoPy8PDw8Lh4mEsNLDLGhAG8DuBuAP0AfgbgV621B6/c8Dw8PDw8FovIZXz3FgBHrbXHAcAY81UA9wNY8IWeSqVsY2PjZVzSw8PD498fBgcHz1lr2y503uW80LsBnFb/7gdw6/m+0NjYiAceeOAyLunh4eHx7w+f+tSnTi3mvJ87KWqMecAYs88Ysy+dTv+8L+fh4eHx7xaX80I/A2CN+vdqbquAtfZBa+0ea+2eVCp1GZfz8PDw8DgfLueF/jMAm4wx640xMQAfAPDIlRmWh4eHh8fF4pJt6NbaojHmPwH4HoAwgC9aaw9cbD9//Md/vKjzBgYGAAAPf/2rAIDe1auDY9s29gIAsrlM0HZ25BwAYHJ6Nmi76WYy8a/v3QQAKJVKwbFwOHyxQ2dU8xIyl9jX4jF33g7l1gafy2W6rwoPJlue12bntBl9K6X557tbLfP3qKlYta+Kz6qLchWvKvddlF0fJXVs3ulyrbKMw/Xrrqmv46SWUkVn9PmuzfO1xq88RHssr86PhGh/JNQ+iRoaZyQ7BQDYu3NLcOyGLbTHJseng7b9/WMAgBOTBRlbKQoAqAnTtUKJXHCsPknXyhVkHCPTeQCASTUHbTP5Mt8n7btoOBocy+WL7kpBm+HzIiHpd3aW7iEWor7Wtkr/UX6umhviQdvgBJlPy8m6oG3HHTugcRp3yH3yI2GMPBu27DaU3k98fojGW/Fccif6uXX9Vaxsif5VKNK9l1T/oQi98vS9R7iPaFTmzV3f7aNCSfooFPM8jmLQFo3SOGsTiaAtFqNrRWPUVywur1tTpmsWizKOTI76q5n4Pi4Vl0OKwlr7bQDfvpw+PDw8PDyuDC7rhX4lUVa/og4jQ8PB5z/5kz8BADz51BMAgE/8/u8Hx+rryRXy3PFzQdsXP/+3AIBXDogX5ZbtJEH8wcc/AQDYs2dPcMz96msJAnCSo/yKZrOZivPdLzMAxOMswYT0tF7YqlV5zUtDJCLXLJVYalESpglkGGkrl88joZsqErrrQUnoZZb2bPk8ErrCXElafxcsHcLKnAVnVem3YhzlyrZq1zZl3cf8/Sb3RJJamCUxAAhZkp6aWmqDtm097QCAWJn2RHdXV3Bsukjz8tTLh4K2PtYW02qtVteQhNtsWIpT2gnKpD2MT4nmaSJ0/Zq6mqBtZoQk/1g0SeMOyfyVw/zZqn3K169JyHnFPEnc2TxpD+lyLDjWUkvjKBQmgrZCnuYmUbfw/i6VRYIFj0mvS6nI96q1KdaE3DNRgpLeWfIuV0j01bRA95fOC4fk+YpGqP+QWn9jXL8y98VioWK8WsoHr1EqIXMUiYR4/HqPuXcKXbNckmP5DPWfzcoey7MWICt78fCh/x4eHh4rBP6F7uHh4bFCsCRNLo6Q+Md//ErQ9p1vk6m+volU1ExW1NBojEwdh4+dCNp+9vyL1K+R36wnn/oJAGBw6KMAgM9+9rPBsd27dwMAigUhrMKsPo2OiilnkMnZ5mYijfIZ8a1vbGoCADS1rZL7Yl3QqnGEfh6kqdEqJH9Wlwk+WmkMsVpbnnsOABOoyEq9DTRGOdPYgO1a1DDdLGiDSJm/68g6q/uvcj7mHRVV1zLZpG5TTEqLnPZa3k/NdWJeqY+TGaZnnZDxd73xJgBAUz2ZOrpXixfv0RP9AIDnDsmenBk9AgAolBUpGqN+r9+wHQCwur01OPaTV8hcmJkW00W8g/ZdOCyKeSqWBQBE2ZQSMmJeiTFBGg6LeSDBpsGQ2jM1tUxuZ4iUjcbl3tNsijg9IuOeKVK/0bFs0CaUMN+bIjTdPqogxXlBQmF5NiLhCB/ivalNY8F+Pb9ZD/ysRWN0z9oE5Z49U7E/6P402Tq3/6i6l3iC1tuZWXQfobB0HA+uT9/NZmSuClm6r0JePV9X4LXgJXQPDw+PFYIlI6FrUu/YsWMAgG9+85tBW55JmOHhEQDAc88+Gxx74969AIAfP/tM0JZmQiYSSwZt4SS5FL16gLwrP/3pTwfHPvOZzwAAOjrag7ZcegYAcOLYUemDf6mTCZJyoorEGj5DcVUlJVU0t3fQhyqSyZWEUWSM4WuFKsQQR16qL7HkErIBi6QOsTRUwR06SVfJAdad5yQfuWYgXV0g/5vTyNx5mpwV6Vr1W66mU3Afbh6quCgaU55/fhXs2rIBAHDz5k1B25tvvxkAcKrveNA2MTEOABg9S+ve3iZ7Z9eunTTWqOy//r98EADQNzwQtGXLJBE3JUir27F+d3CsYEkbPTP5UtBW07aejqnh55MkOUf5EXISIQBY67Qf5f7HEnzRyDMXryHHgjJoz0c0scrk5UxJ+ghHSfpMnEckNKH5+ySq2kw1zcm5Ic73aBR3VrW0Id6LehiG799J5hVPG+/Jiv0Epw3oYVBbLEZ91cSV1sNaVYX7Ln8MR7TWSH+zWZLMZ2fEquCelwotk7UTKC75YuEldA8PD48VAv9C9/Dw8FghWDImF43HHnsMAPD6iWNBmzMfGPbVzMyK+nLo9cMAgOdfejFoM2waKSr9rMSfa2qIUPreDyQi65FHKGvBb3/kPwZt4yNDAID89GTQ1tntfI1JL4omZQojHBF24tiRoC2RJJW7pk7SBs81I1T1mb5Is0woNN+8UtGHIwvVeXOjKg202Yb7VdeQcWqbSKVaW+n7HtLDqThePaLUHZtPutoKX+WQPr1qvxfyhz9fHYD2BjLNdbVJFGk4QmaNAoQYHB4l818qReefm5R9Eq2tBwDcukdMKPe8jUyDDz70j0HbLHsdj04SuZ4ZHguO3di7DQBwWrrFUJTMMNPKRz4zTeaPWHR+1KQzT+n9kS+QmSekzJyNtXQPUSZso1FZ+UzBRZZqIpbMDiYvvulzEVLmFcvEaliZ66IRF40pZksXkBlsCWVyKXEfsYhEdDoSVW//cGAupC+XShV2Gzqmng3D52t/dWcuivN9xhJyTddHsaBJVPZXV1G9xQytUTZP5xVK88n+UIWtCJcNL6F7eHh4rBAsGQndkZ4AsH//fgBAUf0859NELNzzpjcBALZt2Rocc9L1lk2bg7Z0+hX6qyKxsuySGHHSu5LSfvTEEwCAX/ml9wVtE+fIXbGYFdfEAqcALrGEpPnBmhqS6IZHxM3x2ad+DAC48ZbbgraGViLARHq6/N/VkB5IaD6RaPm321RIrvQ34ApVF+HyfEkm4HGUDBhisq3E19eEsIuoC1WK6NSXGoeQpxzFV1bjNpXSu/6uFrLnRhpXI1FDFdoJFsToFEmdB48LAXp2jCTnTF4Yq6kZJhA5YvDchIjSR05S+urdu3cFbW+/iyT0Hz7+bzKOPEuAq0iDOz1+Nji2pZ1cFLdt6Q3aBo5Qv+Njo0FbPkd70oSIqM8r11t3o3qPOZIurfb1+m4i77s5ElZLk6va6FjZCpm7fj3lDho8KZGwc6FdXp32F1WkoYvazOWUhF5yeYhYy9RRnuyCqR0oqpGiYXaDDL6r3XfNfGm82nPonB9cH1k1p4FLperYnV8syr1kea+4ubTKndTY+ZHpmsy+VHgJ3cPDw2OFwL/QPTw8PFYIlozJZWJCyJUTJyi6LpWUVJQhTlX5jrvfBgA4d04Sd02y+vmRB347aJue+hwAYP+rktHX3WyZSZKwilA7fJiI1ZdeeCFoK2RoTMWsRHi5BFyWVatIXMgSw6pge6uU/nvi1adovONTQdu73vuLdH9XsOBH6ALqmmGf43JI7iXK1oMyq4KFkKRuDVdRecsuYZciBiNlNrmEeV6MmCQsJ7QyyoQSLrI6rBI3FdlEFGKCNVTW/ssu2ZY2qdDcF7XD+pwQ1Ap1n9PcatLXSqJWzMUk748zKilWltPctjZLWtnJWfI/D8UoRW58eCg4lknP8t+ZoG39BvJv/8VfuDdoe+21kwCAqRDN6XitEI9DBdozBRXlGU/wfJdkHWfTU/yXSUY1VeUqScjCTCRGI9Lv1Dky9XT2dNL96jUo0jzs3rVBmjh51Zp1HfP6d4iqJHVhXoNETO9TR+JWSbwWRHRqppCJR0VyOpNchc87n+eeb21qc6YRbfoxVQIlXKyFELZyzSg7/IcTkk7YvVOsGpvzg3eJ4zQ3G5h+1DvoipheL7sHDw8PD48lgQtK6MaYLwJ4N4Bha+0ObmsG8DUAPQBOAni/tXb8cgZy9KhEY546RcSPJi4a64mscSkrZ6dEok+ya1FTveSfuO/ed1JfJySXxrkcfadUhRGbGKfhuyhVANhxHUkkw2ekSIEjmYwjUlR4mfvVrasRKau7kySenz73fNDWtIoiCu9++9u5y2oueRfnyljhIha4IeoTHEGkxutSlbKUbZW0UGCJPlRSUhZL1eUKlon6SzqxUJFpJUsSTCEkoonLH1JBSrk8GYGv2nx3MC12ikuljkqtPB0V0pBjf6WtXF6YFX3ttdcAAMdVlOfajm4AwL29IqUO7CPyO8+aSGd3Z3Csi0nG1hZxVw3zOH7hHW8L2kIlctF99P99DwDwnvvvD44dPUcS/6gqxVvgENEKAo33RS7HRH2FayAXdFBEYpRzkSTjImHWlUnj7ainvqbT0sfxs4MAgJ5eSQ+87wXKM9OjCs3MRaQiRwu7PkZk3IHbn3ZT5a8Ea6yIROd2HKogEukLETUfbr3dNXXhCoeQksrDVZ45cSukfjNZ0Sid08PGTXLveU47bEtau+TnCy6PjRxyGrPWgK+E3+JiJPQvAbhnTtvHATxmrd0E4DH+t4eHh4fHNcQFJXRr7ZPGmJ45zfcDeDN/fgjAEwD+y+UM5MUXJShonKVllWoCPevITWp6ilzDNvTKkDZtJreuA/tfDtp2brsOAPCRD30oaPuLv/wrAMAMSzK6vJVz5RpkaQQA3n4PuUjGVVY1515Z5wJeSto1iyUCJQ2t7aZf8deaRVP4zvdIGlu7ju5h61bJU+dcqC4+sEg+B+6IlSnl3JlyLR5nuMj5aXTxAdB9RpQkE3EFA5RrVg0XRkiWuUCCsveOxFx5MiUNsYSuBRknJ5bYll6K6AIGVXLQsE0+rOy8TuJy9thyhX394iyL0RJJW2Mq90YuR/zC08+LFhhje3ecNbKwWvee9T0AgFSN8EDFEs1pTVTa7t57OwBghgOLTvRJnfXWRrqWiYvGlx0l27yBzHM45GaQOQu17k4yTyRFGo8liLvReVUa2J5ezpD0WRuVvDS9a0g7OXr4taCtpYm4hHRa3IJr575OjNY8WQtU0njgLqieL8Mbw+2TqHoJmAjdQ1jtYZHQ9Rozv8X9hnVWRA5OKlWUUXTl7pQ7JP+Nxmi8rxwQF9Yzp+n9tGWrSOguEKtcVnPglGIuL1hQgUhuZkpKbM8X52d7vFhcqg293Vrr3nxnAbSf72QPDw8Pj58/LpsUtWR4WtAgaYx5wBizzxizL51OL3Sah4eHh8dl4lLdFoeMMZ3W2kFjTCeA4YVOtNY+COBBAOjq6pr34s9lSZV97ZCoc672XltTQ9D2gV+mCM4bdlBd0PFzI8GxDiYedZRWqo7U/Z07tgdtNTVEBk2xyUUTiUmu1j2m+i2xeaW1VYoOpKfIDc0l2Q9pzd65OikCr76exrFRmYhODVDxgy998fMAgI9+7GPBsVVMmFYQNIswv+hTqloYzPzxFpmQifK9tI/IvdeOsgueEdUe7DqaUMVFeiaJaE6GyRQ21lgfHDvS1AMAOJuSdYw1EUmYVSpvNkcmjjyPLa/UbBukxdU3w6q6kfUO6qPyv0MVBOj8OqPncxH7yK+9HwDwxYf/NWibmCGT3DM/E9Pgnuso10pdjEwjI4Myf9133cUXFxfPqRlyLzw1LOr71k20P2+78XoAwF9//h+CY6VeKpixvlfcYJvYNFPKyb1k+Bol/qtT5YaY8IYit12Uc1kVY4g00BoNjpNDwqpOuWZjPRVueU659N72llsBAC+/Iql9t629ARq6qIuL7i2oZ7Tk8jMp8jQRov3G2XkRiSkXWUPkpibDw0HkpzbD0HmuWE1JrYGrARxR5pWIu6ba6yXLBDPPkSo3ii2be2gcioh1aXZNRaEZV8DDrZUm8dlEqEy2OR3he4m4VAn9EQAf5M8fBPCtyx6Jh4eHh8dlYTFui18BEaCtxph+AH8E4E8BPGyM+TCAUwDef6kDmJkml8CcyrlSx5nq7r9XAjDeePstAIAY/7ImlftTnEuGNbZL0Ee2QP21NTcFbdu2Uq6Xviefpr6i0kcDuzxmMhIIkuaxtbWJhD5W5JwenLGuTgVnoEowRCxO493AuS8AYGSUSNB/4gIen//83wTH/vCT/5U+VHGhOl+GwEp3xPkSfZCupcJ3ilpjRZI+mw8dDA6tO0I5OmxUE0UkQbSGRVzZXKDvxgsU3JVOCGnYUUfBKt/pkHtfdfebAQAJNacjI0QyTU5TXxMZCXAqcrBRQRGg5VC1QgeVGfZ06UGRFBUpdZ6p3LqFJO8bd/UFbd/74eMAgIak7LFd26mIxeou0qr6TonrbYHzkyRUlr4wS87Ts7LHhsdIuXW5Ud5y5xuCY6PTpP3svU3ywdx8A2lJ+5SmMN5Nz0s8RRro2WFRmF3yv5DapxO8r2cVoTk+SSd2ttM46lfJ+pSLNKfX3yiZIwfO0tqu61mHhWCVvOikTwMtwca5f5mPFI8zFaN5NgkV1MdODBHlyuhITs2BO3fJUsS5Bsr5zm1SPyMlJvZNWcZR30j5lgaGnDYq67huvdOi5ZouL02FK23R5Wvh+1XzYfh8TYQWnbR+GYL6YrxcfnWBQ2+99Mt6eHh4eFxp+EhRDw8PjxWCa57LpVjkFJN5Idpu2E3E5/33vTtoa2oi00mZI7LCSt2JxYnQVClDUGLVW0eJrVtD6qQt/YQaFHlZw+qqTsA/PkamgPb2VUGby+UyO0tqWkONEH5Om6wkNKm/+rq6oG1jL9WF3LmDfOWfePyHwbE73vhGAMCb7nxz0HZeU4u7jvpplhS5Sq10LcqfO8ymi0SeCM3wjKT9NUmOAI2I/peapfnorZMIykSavhvPkBpfMy3n51mDtSqt7Lr3UYzanrv3Bm19feQBe/o0pWcdGBZycWSCOhmbVj7hHLRX0maYOcGgFS74VQpiiBFq/tx+7gt/DwC49Y43Bm3OB/uWG8X8EWHf43/7CaVIfv8vSZRngXOdIKPSrvLo1qxeE7S5VLd940SUr+4R89Td294CAEgkxFxy8CCNY02nkM937qUxtXW2AAAmpyRvULFI95fNyzhmp2nv9h+X3DOnhmn98iHyc9+0WWIjClkad6lW9vr063TvtUkxaZYxC41qdTu1L3bI0JiGTgvZmh07DQDYtYcivWsSkismZCvNagCC5auon8trm0i4KFll6uDTYlGZ05+9RNG6Z/rE5HjTzfcBAE7203ebm2S+E0lqK6pxhF0eoiopeF1shFUvLUcE54uFeedfDryE7uHh4bFCcM0l9FQtRa21tcovfV0tRaY1K7dFJ2kXy/MT9jsfNV1SK8aSdKkkORh61pBk5HJYJJMiadbVEJm3VRXJcL/6TosAgAS7N7rskJVZAOeTkU5CjyoSt50Jwa2bNgIAjqp8M1/+h78DAOzeJS5gdUq6XwghxQrNz68n0qGOsiuz3J7irJKb1kvkW08HXd+MitQeOUpEaUNaJEAzyy6EliQYqyJFnYtkZFYk9DpeojWdQrqtaqO1v34rF00YliIPx06R1N43KERf31m6/si4kKcul0Y5KLShSWWa+3JFlKKbh/kMVP8AaQyxF8Ul77d+8zcBANNjItU+8l2K+D3BuYeu3ylFV7rbSVqeUQSou3q+KERfxLnLcpnDJ598NjiWqqHJaqyXSNHn9xGhv2nzpqAtXyTpur+fCHudxbOlkccxJZJxWy3Nd3eDchjI0RyNFWi8pwflPk2IXE0TtbKHt22j9R48LWs1d/frPWmdC19ovktqOSdrcOIAScuhMt3TDXd+MDgWrSWitFxxITPv2nF2dojHaS8qAT3InhhRZewaWmgenvrBvqDt3EkixN/67g8BAHo27wiODU8xmWwk+la0EZ2dlIlPjjy26j1i3HtDSfl17FYNlTrqYuEldA8PD48VAv9C9/Dw8FghuOYmlxo2J9x115uDtv6TlMI2qnSlwL+Yaw7qGpqOTAgpn+mY80tWZFBHG/mPNrGqGYkrP/Q6UiE39Er9xlWriAx1dQ4BISgLTGbp6NQo+8hrMiiwDCl1v5arxNdzcqc1Ku1q3wCpukeOHAna9uzZUzEH1RBR6q2LuCzqSEA2VUUgiaEsp7WNjpIP+ewLonKO15EJyo6LuaSH/c/Lyl83xpGIoyVSb7MqAq/o+k8JoVSKiZrq4IaZ4vnYULs+OLZ2Lfk5j45LuuTT/aTmn+oXs8CZITINDQwTcTs1KyRqnv2odY3aollYlulqpyjJgf7+oG11D43puwf2B21HT5Na7tT4F14S3/Dd15H5xeZkHHVcc3ZKkZYF9gnvYzPPY09KvVEXYXjddiEo+7mIRn2rpOWdylEfhm1cqbiYEktccb6tUSI/u7q73AWCtpNHXgUA1DbT8ziaFpPEk/vI1Pbeu2+RfkNk3mneJaafg32SIhoASkUVmeuKTVj9LNHnrnXXBW1n+ykG4NWDPwUAlEMyjr13/ToAIJoUJwX3gIXDOdXGCe54icPK3BmFi+gUs9fW7UR+H9wq+//V574BADj0IsWKRENC1A9PU8fJWpnTBt67JizjzRY5BXWEnqVQhf88E7fKacPyeiz8lF8YXkL38PDwWCG45hK6i6y6fa9EyB3htKHaXc9JxE5arvjVZZJTl5pyckBcJ/HnNKdNDSSF1DUI2biJCxesWStuY6kUnW/Vb2aRI1AznGisrJM88C+sJmxdmk7NnUZYo2hrJcKquUFJsBwt1n/6VNC256ab6MMiC1wk3JCKsryFMI23JivSdWKSJMW6UZI+8rPCxgxNkcTbqqS4cohIHWskydosj/35CZJEp5UbYA+ndW1RbnopVcLNwZZdylu+F5WLxJHJnatEKmtvYVJ5Y0/Q1n+W7uH0EGkbp85IGuR+/jw6IZJxOrtwOF4za1BvuO32oG3/i+Rat7pdiONt64nULnKukFtuvik4VpOkuR+blPme4PS5E7Pi3pdh0n6S3T5LqhbDcy9SOujprJyfTLEmVJLxHz1IpHpHJ2mgrY2yT86cIi1j1a2qVBxrvsOjouHk2Md0/09Iy+jY8x4ZY5GI6ddeFa3x3e+4DQAwNCYa3Fzo1LARJirDSmss8YJHa0RD3bLzHQCAQppI8JOHngyO1dfT3rl57y9JvzHaf0a5BLpHwXA64bB6fpMxWpfaWtFinniSXCWHhuUZam6ldR6fJI3vp08/FhwrhknTStVqd2Z6lxgjhHSCJfjOtaTFJFQUdYSf5ZiyKuRLlyObE7yE7uHh4bFC4F/oHh4eHisE19zk4pCqE5/z1lXkh55TibKiUSZDXTSVriEYkKdV/MCVGcZVRmmoI5V6dZeooTu3UxrTVe2qVodzpFYV1gsZUn9PnqAUqBtZ7QYAE2UiVpkdXIpSa1W0GtcadFVf6hRR2Lq+BwAwOyH+35k0qeOJGjHNzLtPqxIQsakqpqxBaR5beHQgaMs+TSTW2TFSy0eUipxmsndNTNqSXPUmopJnJVht7mK/3mllCkix220yJURsJDp/yxXYjDXA1aLqa8UU5nzwYzFVoYf3QIOKXExtoPPWriNVeeuEJI0aGKS5PHlKSM6TfaRmF8ZlPhzueStFaKoMtXj2VSINt60XgvJdd1PUayLlYiTENHLsJCXqSioTQ6FI85ZRJHvBrRX7TKdqYuoYJ7RSa9DQzFWSknJeUyOR/I6Ud/7XALD7BkrL28Z+8QAwPUME8/S0mKDKnEJ2Mk2ms/wx8cFf307XrG+WcUxMnKm4ZjXowMcCOydoP/QSR0mqUsBoXkP3cvCnZH7ITot579nHHwZQmaL2jrf8Mo0jJHshxKaWWjbzNNWJHauzjZ6h1w5LCuOHH/pzAMDY4OtB2+4byDli581U/7VrzU51Yxx5HBHzSoY3S1bVfCgJK8v3rpL2OZOxssW6qluXU7fIS+geHh4eKwRLRkLXrn4zTBqZgqT3jHIFdle3UeduqF4Awqr/E1wRi2YustC5StyOetYRcacEKiliobSBs0NE1jz/IrmvrV0rVeC3cORnLCESqamS48Hyb3CCCdsOpRXUN5KkUdMoUXzjHK3ZVbNwxKiuD1mOsHuhkpAs17GcUimDM/X0nWNDpAkNTQkp6txDN9aLtJeMUB992fGg7U1cc3FbgV2u1FxNulwWmqyuslQz03T9Hz32I7oXFfHrXEddLh8AaGuhdWtSkY71TTQ3tQm6Zk2nSKRdbXT+lvU9QdvAIEmY33j4q/PGs3YjrenDj0qa/2gz7b+Xj70StL31DW+m8XRQ/9/414eDY1s2kIbQqnKAjDHRbKHd1+ivu+M7b705OOb2X0enkG9RJv+yGZEEA4KP51a7RbY0khY4rFLqTqdpvmeVS2+GpeVQmAY0cFzq827cQQ4LDa1y/hPPkTufy0tUDU3Nsl+zGXqWC3mRP8O8x0pZ0ZwOH3uO7mkVrYENixQ8O0bk71OPfV3GnSHt+d33/1bQ1t5Kc97V4faEPBv9fScBAH/2P/9O58dKAAAgAElEQVRb0Hb8IKVGbm3tCdp230qS+c6bqa5wPi/PQY610NmcKk7hXGNVkQznRupS9up3gPNA1rVNS4vI2XQheAndw8PDY4VgMQUu1gD4O1AhaAvgQWvtXxhjmgF8DUAPgJMA3m+tHV+onwtfJzTvc16VZKoPk6HNufxVy0CopXaX00PnWqmtpT66Okgirq+THBn1nFOmVBD7sFHSrMPAELnH7XuZAzHqRUrM5UhaWK+Ck+r4mto9yfAvtctP071aXOFcrphYStyqsulZHptoLHOh81XkWYrTtsbmMkkO50Jir09uJSkoN0a26/S0cAWxPNkhSyovyBnO0zKmkv2/gbWMYo4zNqoAp7ChewkpY2q5ioSeYWnzlVdI+k0qDWf/ftKEEqqtJkVz2rlK3N3ece/dAIDVayhopmyVdsfXb2kUibE2IXM+F2cnKWimpUs0p6E02Z1XdUsOGieMFTgfTLJW9tO5cXoUGuuVlMr7Y2paNCHHfTTWkdbYpa6Z5xJ7OS1JZ2lNcxlZq6kJulaeSwPGVZ6SJPMzNUlZx2ye9niyXgUnFdN8LZq3XddJwFCijsZ48PVDQdsMc0kvv3YgaGtokVw2ABCPKf6KZceiqgcTL9Ien55U7o2tFFj0huso2+KZQ08Fx574PgUcJqIi5T//9LcBAHU1sj8++ru/AwDoWkX3/vphGfd3v/MoAKBvQFwwd91MgUX33Cd5YzbtIJdVV1jFqKyjEdY4E4pzCnGem0JJZV/lbKZuHrQlIZem/goFeUatC77CpWMx3y0C+Ji1djuA2wD8jjFmO4CPA3jMWrsJwGP8bw8PDw+Pa4QLvtCttYPW2hf48zSAQwC6AdwP4CE+7SEA76neg4eHh4fH1cBFkaLGmB4ANwB4FkC7tdaF450FmWQuGZrX7OwiVfrsKVErnYlFksBrkwu7BlYpVqBNLvEEqWAbN5JJpKNNyKYku3q5iugAEIsluQ/pb2iYiC0XmXjoiNSR3Ni7jr8XU21sflGuao4kcVGsHR0qiT+rcwUVgVrkfCCl4sLRjZGQnG/4HpJpOb8hTWr2+Clx18qxC+P1aZrnXXG50SZ2mVulTBe1U2SKkCSgCFyybJhd7NS4Ixz2WFT3PjU7PzeoS33r5k3nznFpijWhFI870k/aZtMuZwrn6qi2tfX2sAs7h+1nM8KmreKSmhojlb5XmXlOsBtkMUod9/b0BMeO76c+mnsVqceE8agiNEuc48cVPCiqzRZlk2BTm1zz+BHKHzOg0taODhHBW5qluapX5rretTSmurXiG7iqlcxGDQ1iPjrbR2s7PUtrUaNcZJu4Lm9oUD2kbIorncdt0VrZKdEY5/VR7rvOullKSSRxOEnfiYSIuA3lxoJjLfy8zkydCdqSoL371Pe/FrR1NdE4162lfp98Ssw2Zwdpz7/3F6Wy5t43UWGSREqimNNcmMQaVxdU5wHiMaqI6TynoC7m5Z2V4xucYrNNPC5zGk/w3Kv8LsZePqW56B6MMbUA/hnA71lrp/QxS2/bqhStMeYBY8w+Y8y+tPLR9PDw8PC4sliUhG6MiYJe5l+21n6Dm4eMMZ3W2kFjTCeA4WrftdY+COBBAOjq6lqUX47LoZJUhJwai/ukruGSxes2dtJXGQoTLKFv2USEz+quruCYyxmSz8kvbJTPLyqpc5Jd7Aol6v+Uysg3NEJSe7uS/F1ARSwlUz03a6Im/FxBjpwiZ6dmiICamV44b0ZYSUN1Bfqc0nk2TlLQRO+ESHZNp7niPEu3zXG5z5YI59mwMo6xJI1zpEVcAmeZGGpisi6ixjEZZwJP5dLIMummN4K7/028LgNnJNjHzVVeEcJJ3h9FFYIRUVoRAFgVaOVKwivvSZjq8gcA4OjJkwCkLCEA3L6XSLKn/02yIb7OGRIHJ0k627lBiMQ37SV3t5aUEKUnBik/j1EqjssJdJbLHTauUXuSpcO1TSI5dnSQDPb6AdG00o5kZXe6KVUJro/nsrOrO2gb6Ke2nk7JWxTj9c4yGT4+KVkiO1fT3N6wXYq/vHqMNNPVvaLFDJ6qkPMQC4u8GGWngHRa1izH+XRsQZ4Np2kODFCpve075Zp777oDAPA3n/sfQds0B+Bl0vLcPvotekU5F8lJFZy0lkv8rVsrz+jhAyTBZ3Ky151rcZS1r1PHpQhNLEnEZygm++PcOGkSZaVdxpO09l3dtC9alEaUqKU9n1N7IV+6CoFFht6gXwBwyFr7GXXoEQCOFv4ggG/N/a6Hh4eHx9XDYiT0vQB+A8ArxhgXD/yHAP4UwMPGmA8DOAXg/T+fIXp4eHh4LAYXfKFba3+MaklSCG+9YiNRhIDlfA/aMBE7T+rYMqedjESU36vTuJUO4kjTBs4V0lgvPqOTXJl+YlIKKXQ48lKpza5jFwU2OiakzWtclEKTY6Uyp5xV5Kzzs3fkX0RFRjqfbaPUv9EhMpN0KLV5LuqMisCzpHMrF3kko0TYhuNCsHUXqVr95hNkKoorQjjOUXA5lTMksZmu39Ugqrp5mUw5mTqKlkzMSkrWIqg/7c0f5Wrr2uDR2Uljev+vfACARI4CwBTnG5mdFXX+TD+RYmGVF6RxTt3Vit3iLqZ85Cs+z0E9RxInE7LuYfa9z0fEPBZtoPNyrCqHlP/3xi3kT33i8OGgrRwhE15zo0q7GqW1mh5ni6Xyp87z7Q0NilmvIUFr0JiQ+3W5jDJRMlEWVGGJPjavrFsjuYEaOI10XKWgTiTpO7NZMt/MzMgajI3Q2NauFXNQz600jn61/wdRaXKpyAbL/Hwuo/IFcZxJOCKxEVEuaHHdTkpF3BwVEv3IQUph3NIq44hEiextWy3PciMX86jhwio6FfVAP+3Xv/6bB4O27deTH3pv766graaW9ulg/0EAwAsvSZ6Xe9/7YQBAQ5s8B+s5ZqU2qVLk8n5w/uXWyH510dC63EsmS++Kylm8OPhIUQ8PD48VgiWUy0XlOWBXLpc3BQDWcQSdI8kqcqO4IhJV3H5MWEvtdNwRlXkVgRdlUq1fkZxhZtF6NgrZVeDK3e6YdpUc4zJpY+Pj6vxCxRgByd/hJPSoKkMVYQ0grjIwOqI0nVZs1xzYnEgymUmSxobKqiwXF7tQQifWsitZPMqRq4qgKbKbltMwACB05CS1hZQMESeJcvYuyupnnvpJcKjMkad6VWKuXKAWlvmemziPjfsLANZ2uQ9B2w3X76Zx6PJ/bg6ZpNUSul1QwayOHTupJFpbk5C/B45QtOG26yQasnSUNLKGBhpvOCXazJHTRFpm8vPXrE0V6wizhNsVp1mqbZRrznBEZ2OdtI0N05zmlQNALOqq21MfdW0SATp+jgp+HH5NoiVvu5lKGh7jvCaAENM1vJ7TE0Kou+jsSZUjprWd7iGbXdhzLZvTe573k9K7EzGXOVVl9IzRPZwbpLE988qPgmPTM0TUruq5IWjb3EJScqpO5rS2nj43cAGbodOvBcceffizAIDGVsnjdMfdv0bXTkn08OQYSeTpY+R++oa7JMxm3TYq7lFW8nUpT2PLKq2kmKH9WWTN1xiViTRBa6ZzR9krIF97Cd3Dw8NjhcC/0D08PDxWCJaMyUUn23I+q9r/e5ZVu0C11pFbJRfNJfqLiygNq3SujqSIK79vhxT7vCeS4lt65Cj52q7boIpYsPruTD9RlXTL3cPQkBCDg4Pkq1yniFVHqLo+9LgDU1KVtqGzUidzLob7JNlQTYbNQip9aYFrXEaLSkVmEmZ2lgiwpPKPtzxXIaVCNk6S+WC2SeIDCpyAq7SBTGLhfXKfoSn+so4mZKbMFsSU42qK2sBcok0k/FmZ5CJMhoZjKsrOuEIii0xBWi1LGGMDp1J+/YjM6QjHGCQVSbxrE/lI59mMcFqRbyU2j9VH5Pxi3kVhyhzN8HktcbqndV1CtA0Mu7gGMQUcYJI1o/IQx9hEmeB6ma0twoaPnqO9OHRO4g/OTZFJcCYjZrpWTkn8zrdRTc9nnt0XHBsapj7WN0jRkJcPUNK07nU9QduRgcooYJ0wrsxzFFbBABE+oazMMK4O6ImTFBGbLsge3nMnRXemFNtf4leYi7gFgDLvhSz7t9c2yZy+7T4iNLvWyLjzlojM4UkhjotF2uObr78PAFDTIGTnBD8vEWVCsXm65mxe+bKbyuh2vTezXNs3on314/PfSxcLL6F7eHh4rBAsGQldw0nhraoAxdgYSSt1nDpVS94lV95KuWFpotEhzL+G9VXKmjnXwR4lcbz6CiX5L6g0vrV1dH0nXadUNKsrl6ZTHJw+TZJGQ638wtexu2Q+78gSo47F57UVmYidmlzYoclMiLtlnAnQSJ1cs8yidqgkknE+SteadAUUlBR8cobItyOjItm9gYuMJPOybdIshWQPkRSXgMzHBBcWKCmNqJglCab/mETeOQmmFOExqvuKcMreUHi+a6cO/XTz5dwiwyplr3Mn1CXACvmF8+LkxmgubV7cOJt5baeHRYpbU0/7c3CSiMdcRgjQCOeKiSsNrsiRx+eUq1+GrzHL300oJ8/re4mAPdYv6z4yTeeXwkLImaCQCK1FRpHhvBVQUIG0w5zzpVGl1I1ztOmmDaR1zExLpOiB18l1LwuVu8QVkDkzP8W0g04ZXWZyXRcvKbkKLEq6TsZprbZsIeIxpxwBVnf20LVVmcMCa+dFrV0al4aW90RSRdquvxEAMJuVtXXRsTlFsiNEe92whjWtCF6nNOZVPiBTdq7W6p6dRF52qbxV9/z+0lGh+SydcDlyupfQPTw8PFYI/Avdw8PDY4VgyZhcTJUaoTrhkiOPimwyiCnirJAj9SkUFzNLkJxLJctxn1M1pGrqCE13TU1Yuc+6RuOa1USYdXF04017bgqObd1A9RXHhsRMMc5Jl3REaZbH63x/cyoqtCGYB9HPnClnRqmJc5GoiMqj+0yr5FyRGjYVqeS3g0zC5DZT5aJZK8re918h393v5UT1BqcXva5efH4PFGhdYofJFHFAkTzjq4i8mlDRj6MvvQgAePoJSXIVdamLudJSTJlLerlmZTIpfYRdtGlFAi4mBtmsEdaqvZlPkCeilQnSNIrjNG9tCTFJuCpTRlWYOddH5rQymzo66iQ9apj3X31MRX5y1qyyIvsLbJ4zbNbLj4rpDFxlqqiSS8X4WqtVndFclswDZU6uFk/Ic2PZLLCqXRJD1TXSvp6dkn7DJTIbHTvCka1huc86JgSLadkLCa7ENDYmJiigBRp1yhxZKJIJLxbTrxznFCAtzi872e3iD9TpIdq7ibhaR15bfVqZCWPDpo6cWrM0m05UUxAXYks6Wp3r8hoat45xKTs7lh44n69jHsocsZsrzTfvGTZVaqK0zPK1N7l4eHh4eCwdCV3/UjlpOaRydcQ5VaUjBpMqpair5Rkpzf9t06lqC0x6uNwsVpGArvq2zsdR4hwdo8OjQVsdS23vufddAIB3vef+4FiKpcjvPvIvQVueCzroGpApJqOmJkkS1A507RxFqCXSeo5YnBysmqGYjinidpIJv7BKfbuFNYuSyt267yC7wI2wi11JJLYX2fUxrVy+nmkkSc28dU/Q1j9F7lelIZLexhIipZVqOCdKrbienTlJEZQHXpCq8hl29YryumRnhQjr7KR1bm6VPmY5YjCnPDBjHFk7OUVzqklUw3untVn2TM86jiZsm1+XJcTudEYRp9aRyUoqK7E46eTFGu0KydtZ1/c0JVcYQU6LcFpgw8R3UkUUFzgla53SejoTdO9r22Sej08083hovA0qyriBNZXdvRuCthYucJFrEk0hP07PVahI/ff1nwyONbNzQqsq0DCZoTULpeT+0nMEUaOiuWPR+cS0+xRR/o1h4+pvcoN6Lzg3wIjS4Jyjg36GXKnPfJ7mMqsKw4TMfNdYw++ZWExdC+48R2xqHYCld+0qWWR3XBVtXeI2t6K6C5d/qkIB0eHTlwgvoXt4eHisECwZCb0aOjslu2CR7c5FtjmWVekmV65NuxfqXC8OJZaCImyD1efPzlK/2t2tazUFdAwNjcg4OCDmvfeRZL75uuuCY2WWrto7JKPhmZMkmZ8bFSnflZwLsRgyqzLbFdiNTQcslfinffVaCex46ZCUvgOADXtEam6sJ0l6Rl0zFueiEAmRqLbcS5XVJ2dJ2ppQuXMKLEnt7d0StE0cp3wgYx0i1dp2snXOtlIf5bJoInHOhDd5VkqGzXJl+ptvvSVo23+A8mWs20Rueq+89FJwrMAyzNi0cBAjQ/R5qF84AicNugIDBeWCtmEzZT5ctW170FbULmpzMBCieyip5DZOUNSSYIxdQY0LiKoQ4jjwLCScTIntxzoHTSlG6zHFXMVESXiSSITOy6it3NZG0ncyJPbsxiTt2Qzv77RyYW1jraS5UbSTSJiumYyqzJEt9HmM93q3yinT0kzazNS4BA6VmPdpbxPb/OhEpYQ5pfa1y4Sqny/Dsmvcyp6M8xyFAy1d+gxzMGEsooMFQ3xMrmv5uxnOQJrNqvKFrJWHo9JHOTpfMnaafbk8n4szwTE532WW1LlqnJ08bFxZRCXRc3e2pG3oF5dzqBq8hO7h4eGxQuBf6B4eHh4rBBc0uRhjEgCeBOVijwD4urX2j4wx6wF8FeSr9DyA37DWLuxXdxFwBGkiJURY91pyXzt9gki1yVlR7ctMS6WVW5WLrtSuie5ztdwKs0zEaVNNB7smzqh6hfXNpPK69KGa5HHqkyOdAGCA0/GeGxc1eJKj8lzUXD6vo9ZIlS2qtLVuvL0bJacMvvcYKqDU57FpupcXX94ftLmiEVt3ioloNZuGonVEGkYKYh6ob6a2LTukpuPTw5RL5sArUs+yZhWZVXq5HmhS5aPI8rwdGOkL2lyK18ZWMQHUsYkolaLvtiqTzlvecicAIKFyqPz4KSrMcbrvyaDNRWTWN5CrZIXbIpttdJ3W9RwRmc3PT/86w5XbS8q9MDC56Bw7jpDjY6GiqNvO669duWxadsM1ETkvM83mGp6D0ZLKBcImvLIi7xM1dH5cja21hb47PUv7pKjMGjGOUJ5VZh7L5kuj9m6CZbscj1d7hJbZLGTUGkR4fyaMjhRVjCCAvLJJlEp8nqp64YjmdFZMn47wdHlp4srNMcFmm5DK4xR2thYlmroRRfn+Esq8UnK3oE2yLreSGluZ3RRdWu98QeYvm6PxFhXL6YjVQkTOy2RonsuOYFWuj67fkHIPDYUXjrpdLBYjoecA3GWt3QVgN4B7jDG3Afg0gD+z1m4EMA7gw5c9Gg8PDw+PS8ZiStBZAI7diPJ/FsBdAH6N2x8C8McAPnclBuUICV0UwgXhOJliYECItqhze1KpAVNcsV2TMDqT4ly4IKIxFQAU54CXplYhiGrqKA9Lll0lX3pBstJlMyTtFZWbVCO7HB45LAUGTvRRVj5HjuaUhO6kSe1umWAXxnhcF6yqRD4nksH4KAUztbYJOdvUzBkYVa6QUQ56cvlPMjnRRHKzTnJVJA+7bL50QAoGdIyTFpViKU7n0BkZoaCTvtMioY9x4IzO6TE1TeTmmTOn6TpKcozzNZMJWbv2diLLV68Vl0q3P3q4/F9jo2Tke/3IMQBAvjB/Xc4OzZfQC0z4aYktyOaoHM3KPE7DmllDXMbY3U5aRiokUq0jx6YyokmOciGJpnZyDZzKyV4osA+cVfl3GjgrY0qVT0yupnVezS6SqsobTjNBOqwKURQK1aRalmbZ3bOtW/bODI93QuWIiSbovKmC7Jm5rxNdsCGX55wrKqLHBhkYVaZVDi5zU6/L6eVZQo8Xpd8UvyNiUZWNk+c5cEVWpGeS3xUhVQ4u7Aj1spbQXeAPXTOdKapj/H5Sbq2hwBdVxlEsssbEgU5FpcAYfldFKvY6PzsL17G5IBZlQzfGhLlA9DCAHwA4BmDC2sDrsh9A1YKXxpgHjDH7jDH7dNIqDw8PD48ri0W90K21JWvtbgCrAdwCYOsFvqK/+6C1do+1do/OTOjh4eHhcWVxUX7o1toJY8zjAG4H0GiMibCUvhrAmfN/e/FwJGHfyWNBW4hzUkyNkq90ekpMI/WcV6Kg63uy6aRWpa11arlTm7VvqYs0rK+XaLgcR/Hp6MAZTm6fYjVtbFR81PPsY93QICaaciedNzUlPtM1nNY2wWYhoyIBXc1Sowhbp9lY7fg6D9LHoYNkEjmjTB01nL9Gk75xznvSwOr7zKzkrOk/Q/loXnhefMId6ZxTOWWOHyfz0bkRmu9yBRHm4gNE13T3ElW+xKGgjiqpnBs3SQ1XI3F2QdvOHeRPvnmTRD+64iXOdKbJcJdvp07lWnH5cc4OncZcnOO10iY/hwozDH92HGdzpxSiKDNx9rOXnwnanLWhb2AgaJvgOU810NjKan2KbAaxZZlvy+GxB89JjIHJktmjlgWmvOpjnOc+p1K9gucqofyeXcGRCJu4mmvFpDPE5HZGmX7iPJeZ8/B4OkNx3plcivPNWGr7w0adqYPOK6pCHnk2v+RVzpWSS5FbkOclFnafXS4X6d/VyNWcqPPL1xGoUTYDFYuu+IoaI5OhYdWJiykpKRNKmPe4M98YRea6MJOIChtOJK4CKWqMaTPGNPLnJIC7ARwC8DiA9/FpHwTwrcsejYeHh4fHJWMxEnongIeMMWHQD8DD1tpHjTEHAXzVGPPfAbwI4AtXalABAaV+zSeZ3Ely2fpwWNzBTIh+YcNKisty7pSWZpUBjgkOFxV6+rRIZ46IS6hiDC5Xw+SMRMhZJo0SjZSnZG2vuBLaootSlHHUNxM51rN+fdCW5/JT09MkncVVebpBJhLLKi9ISxu5SNrQwss1rtwiSywln+oTCb2dSbraGtFYptmVcYLzeOjkcY5kPa4KUaxfT/e6Z8+NQVuI3bRqa+ke9Pw5rUfn6nDupLq4iPtOLWtHLaoiuztPaxZO8te5McycLJU6j4iby7ISs0rFShc7jSmeF6vKD7hEjSEl1YY5m1+R98nxYZG8jw7S3pqdlmjJEMtPhbiMIxmjecsUmFBX5JuJOPdaOX+K17aoSp05jbbIGqIWP903w2q+41FHrst5DZzvppsjSofVs+FyoZw9I1lEz4I05XXreiGo3J+FfGneZ10qzuVJKRvtHsrnhVzeFlnkYthFaIbV+XxlRawG+419Lwt5rTXy/IblmuGwy+IohL7L+eI0C+W3AOOydyp3SBeZrJ9bE7ibskOCEsCdVqqdA5Rwf8lYjJfLfgA3VGk/DrKne3h4eHgsAfhIUQ8PD48VgiWTnEtH4Dl/681btgVtMxNkFsiwWtncLl6S01NkbpickGT7juDQNf5csNcwJ61KK9YmxT7qYVWQoKWZIj7blArr1OCIKyJQVqk/q/w+umhUrU4539NZTq07NS3mkmlOaFTfIGaHzk5KgBVPLpz6vqtboiubG+8AAFy3TchF52seV5GcmhQGKk0u6TSNo0mlKe7ooHFEFKEZi3OyI54jd51q/QPVk6a5NkcIa1/vIhPT2uzgkpWVdfLRwG+Y70Vfh2+sXJECdWE4NVgXKwix6h9SDF6ZyTkXMVhQUZ7BpWtVPVX2g4/ExNvLmYGyrJbnlbmuzGavitTS7r7UrTgirlyl2IMzLRWz4i/u5jSt5sMm6byhCXo2+jnCGQDCTJ7nFannIquTE+KckKyv9GIL6xTGrqatekZCcM+GmueQm2dOaKXMjC59rk5b6/zUSyozWsHFsbD5o6Sta+x/rulHw2NKq1gOZxp0JHg4rGoU85hy6v2RL7hx61rGroAHk+eKAI1EnclFRlJ0JDsuHV5C9/Dw8FghMPrX/+eNrq4u+8ADD1y163l4eHisBHzqU5963lq750LneQndw8PDY4XAv9A9PDw8Vgj8C93Dw8NjhcC/0D08PDxWCK4qKWqMGQElhzx3oXOXOFqxvO9huY8fWP73sNzHDyz/e1hO419nrW270ElX9YUOAMaYfYtha5cylvs9LPfxA8v/Hpb7+IHlfw/LffzV4E0uHh4eHisE/oXu4eHhsUJwLV7oD16Da15pLPd7WO7jB5b/PSz38QPL/x6W+/jn4arb0D08PDw8fj7wJhcPDw+PFYKr+kI3xtxjjDlsjDlqjPn41bz2pcAYs8YY87gx5qAx5oAx5ne5vdkY8wNjzBH+23Shvq4luMj3i8aYR/nf640xz/I6fM0YE7tQH9cSxphGY8zXjTGvGWMOGWNuX4Zr8FHeQ68aY75ijEks5XUwxnzRGDNsjHlVtVWdc0P4P3wf+40xNy7c89XDAvfwv3gf7TfGfNNVY+Njn+B7OGyMece1GfXl4aq90Lni0V8BeCeA7QB+1Riz/Wpd/xJRBPAxa+12ALcB+B0e88cBPGat3QTgMf73UsbvgsoGOnwawJ9ZazcCGAfw4WsyqsXjLwB811q7FcAu0L0smzUwxnQD+M8A9lhrd4Cyt34AS3sdvgTgnjltC835OwFs4v8eAPC5qzTGC+FLmH8PPwCww1p7PYDXAXwCAPi5/gCA6/g7nzWuNNEywtWU0G8BcNRae9xamwfwVQD3X8XrXzSstYPW2hf48zToRdINGvdDfNpDAN5zbUZ4YRhjVgN4F4DP878NgLsAfJ1PWerjbwBwJ7jEobU2b62dwDJaA0YEQNIYEwGQAjCIJbwO1tonAYzNaV5ozu8H8HeW8AyogHzn1Rnpwqh2D9ba73NhewB4BlTgHqB7+Kq1NmetPQHgKJZhRbar+ULvBqBLrPdz27KAMaYHVIrvWQDt1tpBPnQWQPsCX1sK+HMAfwApL9kCYEJt6qW+DusBjAD4WzYbfd4YU4NltAbW2jMA/jeAPtCLfBLA81he6wAsPOfL9dn+LQDf4c/L9R4q4EnRRcAYUwvgnwH8nrV2Sh+z5Ca0JF2FjDHvBjBsrX3+Wo/lMhABcCOAz1lrbwCljqgwr0dmPnsAAAIPSURBVCzlNQAAtjXfD/px6gJQg/mmgGWFpT7nF4Ix5pMgk+qXr/VYriSu5gv9DIA16t+ruW1JwxgTBb3Mv2yt/QY3DzmVkv8OX6vxXQB7AdxnjDkJMnHdBbJHN7LqDyz9degH0G+tfZb//XXQC365rAEAvA3ACWvtiLW2AOAboLVZTusALDzny+rZNsZ8CMC7Afy6Fb/tZXUPC+FqvtB/BmATM/sxEAHxyFW8/kWD7c1fAHDIWvsZdegRAB/kzx8E8K2rPbbFwFr7CWvtamttD2i+f2St/XUAjwN4H5+2ZMcPANbaswBOG2O2cNNbARzEMlkDRh+A24wxKd5T7h6WzTowFprzRwD8B/Z2uQ3ApDLNLCkYY+4BmSDvs9am1aFHAHzAGBM3xqwHEbzPXYsxXhastVftPwD3gpjlYwA+eTWvfYnjvQOkVu4H8BL/dy/IDv0YgCMAfgig+VqPdRH38mYAj/LnXtBmPQrgnwDEr/X4LjD23QD28Tr8C4Cm5bYGAD4F4DUArwL4ewDxpbwOAL4CsvcXQFrShxeacwAG5MF2DMArIG+epXoPR0G2cvc8/191/if5Hg4DeOe1Hv+l/OcjRT08PDxWCDwp6uHh4bFC4F/oHh4eHisE/oXu4eHhsULgX+geHh4eKwT+he7h4eGxQuBf6B4eHh4rBP6F7uHh4bFC4F/oHh4eHisE/x8dXIsrPBuD/wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "# functions to show an image\n",
    "\n",
    "def imshow(img):\n",
    "    img = img / 2 + 0.5     # unnormalize\n",
    "    npimg = img.numpy()\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "\n",
    "\n",
    "# get some random training images\n",
    "dataiter = iter(trainloader)\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "# how to convert back and forth from numpy arrays to torch Tensors \n",
    "images = images.numpy()\n",
    "labels = labels.numpy()\n",
    "\n",
    "images = torch.from_numpy(images)\n",
    "labels = torch.from_numpy(labels)\n",
    "\n",
    "# show images\n",
    "imshow(torchvision.utils.make_grid(images))\n",
    "# print labels\n",
    "print(' '.join('%5s' % classes[labels[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "images, labels = dataiter.next()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([5, 1, 8, 2])\n",
      "  dog   car  ship  bird\n"
     ]
    }
   ],
   "source": [
    "print(labels)\n",
    "print(' '.join('%5s' % classes[labels[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 3, 32, 32])\n",
      "torch.Size([4, 3, 32, 32])\n"
     ]
    }
   ],
   "source": [
    "print(images.shape)\n",
    "print(images.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(-1.0, 1.0)\n"
     ]
    }
   ],
   "source": [
    "print(min(images[1].numpy().flatten()), max(images[1].numpy().flatten()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(-1.0, 1.0)\n",
      "(4, 3, 32, 32)\n"
     ]
    }
   ],
   "source": [
    "print(min(images.numpy().flatten()), max(images.numpy().flatten()))\n",
    "print(images.numpy().shape)\n",
    "#print(images) # [torch.FloatTensor of size 4x3x32x32]\n",
    "#print(labels.numpy(), labels.numpy().shape)\n",
    "#print(labels) #[torch.LongTensor of size 4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Define a Convolution Neural Network\n",
    "\n",
    "-  see the CNN as compression of activations and go over the conversion formula\n",
    "\n",
    "<img src = \"http://deeplearning.net/software/theano/_images/numerical_padding_strides.gif\" >\n",
    "\n",
    "here is a 3 $\\times$ 3 kernel applied to a $5 \\times$ 5 input padded with a 1 $\\times$ 1 border of zeros using 2 $\\times$ 2 strides. \n",
    "\n",
    "kernel weights are [[2,0,1],[1,0,0],[0,1,1]]\n",
    "\n",
    "$$W2 = \\frac{(W1 - F + 2P)}{S} + 1$$\n",
    "\n",
    "$$\\frac{(5 - 3 + 2(1))}{2} + 1 = 2 + 1 = 3$$\n",
    "\n",
    "http://cs231n.github.io/convolutional-networks/\n",
    "\n",
    "https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md\n",
    "\n",
    "- activation volumes / compressed images \n",
    "\n",
    "<img src=\"http://cs231n.github.io/assets/cnn/cnn.jpeg\">\n",
    "\n",
    "- Affine Layers / Linear transformations / vanilla feed forward Neural network\n",
    "\n",
    "<img src=\"http://cs231n.github.io/assets/wb.jpeg\"> \n",
    "\n",
    "<img src=\"http://ufldl.stanford.edu/wiki/images/thumb/9/99/Network331.png/400px-Network331.png\">\n",
    "\n",
    "- Relu Activation\n",
    "\n",
    "<img src = \"http://laid.delanover.com/wp-content/uploads/2017/08/elu.png\">\n",
    "\n",
    "Leaky ReLUs allow a small, non-zero gradient when the unit is not active.\n",
    "\n",
    "Parametric ReLUs take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural network parameters\n",
    "\n",
    "$\n",
    "\\begin{split}\\text{PReLU}(x) =\n",
    "\\begin{cases}\n",
    "x, & \\text{ if } x \\geq 0 \\\\\n",
    "ax, & \\text{ otherwise }\n",
    "\\end{cases}\\end{split}\n",
    "$\n",
    "\n",
    "Without non-linearities, DNNs would just be a linear transformation\n",
    "\n",
    "W(Wx) = Wx\n",
    "\n",
    "## f(W(f(Wx)))\n",
    "\n",
    "- Max Pooling\n",
    "\n",
    "<img src = \"http://cs231n.github.io/assets/cnn/maxpool.jpeg\">\n",
    "\n",
    "## PyTorch nn.Conv2d\n",
    "\n",
    "torch.nn only supports mini-batches. The entire `torch.nn` package only supports inputs that are a mini-batch of samples, and not a single sample.\n",
    "\n",
    "For example, `nn.Conv2d` will take in a 4D Tensor of `nSamples x nChannels x Height x Width`.\n",
    "\n",
    "If you have a single sample, just use `input.unsqueeze(0)` to add a fake batch dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[[0., 1., 0.],\n",
      "          [0., 0., 0.]]]], grad_fn=<CopyBackwards>)\n",
      "tensor([[[[0.1717, 0.2969]]]], grad_fn=<MkldnnConvolutionBackward>)\n",
      "Net(\n",
      "  (conv): Conv2d(1, 1, kernel_size=(2, 2), stride=(1, 1))\n",
      ")\n",
      "('conv.weight', tensor([[[[-0.0775, -0.2027],\n",
      "          [-0.0470, -0.1212]]]]))\n",
      "('conv.bias', tensor([0.3744]))\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "x = np.asarray([[0.0,1.0,0.0],[0.0, 0.0, 0.0]])\n",
    "x = Variable(torch.from_numpy(x), requires_grad=True).unsqueeze(0).unsqueeze(0).float()\n",
    "print(x)\n",
    "\n",
    "class Net(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        \n",
    "        super(Net, self).__init__()\n",
    "        self.conv = nn.Conv2d(in_channels=1,out_channels=1,kernel_size=2,stride=1,padding=0)\n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        x = self.conv(x)\n",
    "        return x\n",
    "    \n",
    "net = Net()\n",
    "z = net(x)\n",
    "print(z)\n",
    "\n",
    "print(net)\n",
    "\n",
    "for name, param in net.named_parameters():\n",
    "    if param.requires_grad:\n",
    "        print(name, param.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model architecture as a class using the PyTorch F and nn.Module "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        \n",
    "        super(Net, self).__init__() # https://stackoverflow.com/questions/222877/what-does-super-do-in-python/33469090#33469090\n",
    "        self.conv1 = nn.Conv2d(in_channels=3,out_channels=8,kernel_size=5,stride=1,padding=0)\n",
    "        self.pool = nn.MaxPool2d(2, 2) \n",
    "        self.conv2 = nn.Conv2d(in_channels=8,out_channels=16,kernel_size=5,stride=1,padding=0)\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 64) # y = Wx+b\n",
    "        self.fc2 = nn.Linear(64, 32)\n",
    "        self.fc3 = nn.Linear(32, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # note that the expect shape of an input is (batch_size,channels, H,W)\n",
    "        x = self.pool(F.relu(self.conv1(x))) # (32 - 5)/1 + 1 = 28 , 28/2 = 14\n",
    "        x = self.pool(F.relu(self.conv2(x))) # (14 - 5)/1 + 1 = 10, 10/2 = 5\n",
    "        x = x.view(-1, 16 * 5 * 5) # flatten activation\n",
    "        x = F.relu(self.fc1(x)) # affine\n",
    "        x = F.relu(self.fc2(x)) # affine\n",
    "        x = self.fc3(x) # affine \n",
    "        return x # shape = (batch_size,output_dim), [torch.FloatTensor of size 4x10] \n",
    "\n",
    "####### instantiate Neural Network ############\n",
    "\n",
    "net = Net()\n",
    "\n",
    "##############################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Define a Loss function and optimizer\n",
    "\n",
    "Let's use a Classification Cross-Entropy loss and SGD with momentum.\n",
    "\n",
    "- What is a loss function?\n",
    "\n",
    "The Loss function is a scalar, calculated from the output of your neural network and the ground truth labels, that requires the neural network to do what you want it to do in order to be minimized. This means if you change your parameters in the opposite direction of the gradient with respect to the loss, the neural network gets better at classification.\n",
    "\n",
    "y = one hot label\n",
    "\n",
    "$\\hat{y}$ =  softmax outputs\n",
    "\n",
    "$$CE(y,\\hat{y}) = - \\sum_{i=1}^{C} y_i \\cdot log(\\hat{y}_i)$$\n",
    "\n",
    "Where C is the number of classes, ie length of the output vector and $\\hat{y}$ is the softmax probability. \n",
    "\n",
    " $$softmax(o + c)_i = \\frac{e^{o_i+c}}{\\sum_{j} e^{o_j +c}} = \\frac{e^{o_i}\\cdot e^{c}}{\\sum_{j} e^{o_j}\\cdot e^{c}} = \\frac{C \\cdot e^{o_i}}{C \\cdot \\sum_{j} e^{o_j}} = \\frac{e^{o_i}}{\\sum_{j} e^{o_j}} = softmax(o)_i$$\n",
    "\n",
    "y is a one hot encoded vector of the correct labels (\"frog\", 6 = [0,0,0,0,0,1,0,0,0]), lets assume that the k-th element is the only element = 1. The rest of the contributions to the cross entropy function are zeroed out from $y_i = 0$. \n",
    "\n",
    "We write the k-th and only contribution to the CE where $y_k = 1$ and $\\hat{y}$ has been replaced by the softmax function. $\\theta$, outputs, is the input to the softmax, raw class scores before they are softmax normalized. \n",
    "\n",
    "$\\theta$ = outputs\n",
    "\n",
    "$\\hat{y}$ = softmax($\\theta$). \n",
    "\n",
    "For one sample:\n",
    "\n",
    "$$ CE(y,\\hat{y}) = - log(\\frac{e^{\\theta_k}}{\\sum_{i=1}^{C} e^{\\theta_i} }) = log(\\frac{\\sum_{i=1}^{C} e^{\\theta_i}}{ e^{\\theta_k} }) $$\n",
    "    \n",
    "$$ = log(\\sum_{i=1}^{C} e^{\\theta_i} ) - log(e^{\\theta_k}) = log(\\sum_{i=1}^{C} e^{\\theta_i} ) - \\theta_k$$\n",
    "\n",
    "Now the cross entropy is expressed in a form in terms of the softmax input $\\theta$. We will take the derivative with respect to an arbitrary $\\theta_i$ not necessarily the correct score $\\theta_k$ and use the result for the individual element to see the result for the vector.\n",
    "\n",
    "$$\\frac{\\partial CE}{\\partial \\theta_i} = \\frac{\\partial}{\\partial \\theta_i} (log(\\sum_{i=1}^{C} e^{\\theta_i} ) - \\theta_k)  = \\frac{1}{\\sum_{i=1}^{C} e^{\\theta_i}} \\cdot e^{\\theta_i} - 1[i==k]$$\n",
    "\n",
    "The [i==k] term is a condition that means to subtract 1 only if the index of $\\theta$ is the index for the correct class in which $ y_i$ = 1, ie $ y_k$. This follows from the derivative $\\frac{\\partial \\theta_k}{\\partial \\theta_k} = 1$ and $\\frac{\\partial \\theta_k}{\\partial \\theta_i} = 0$ . In vector form: \n",
    "\n",
    "$$\\frac{\\partial CE}{\\partial \\theta}  = softmax(\\theta) - y $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[10., 10.]], grad_fn=<UnsqueezeBackward0>)\n",
      "tensor([0])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor(0.6931, grad_fn=<NllLossBackward>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)\n",
    "\n",
    "s = np.asarray([10,10]) # np.asarray([0.5,0.5]) # \n",
    "l = np.asarray([0])\n",
    "s = Variable(torch.from_numpy(s).float(), requires_grad=True).unsqueeze(0)\n",
    "l = Variable(torch.from_numpy(l)) # Variable(torch.from_numpy(l), requires_grad=False).unsqueeze(0) #\n",
    "print(s) # scores must be variables of shape NxC, a matrix \n",
    "print(l) # labels must be variables of shape N, not Nx1, a vector, not a matrix\n",
    "loss = criterion(s,l)\n",
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12500"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how big is our dataset? \n",
    "len(trainloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "loss = criterion(outputs, labels)\n",
    "```\n",
    "\n",
    "Input(outputs): (N,C) where C = number of classes\n",
    "\n",
    "```python\n",
    "Variable containing:\n",
    " 0.0999  0.1114 -0.1228 -0.1509 -0.1447  0.0225 -0.0651 -0.1255  0.1152 -0.0538\n",
    " 0.0997  0.1115 -0.1206 -0.1452 -0.1385  0.0162 -0.0626 -0.1249  0.1166 -0.0567\n",
    " 0.0999  0.0996 -0.1172 -0.1439 -0.1498  0.0165 -0.0616 -0.1226  0.1042 -0.0651\n",
    " 0.1010  0.1027 -0.1187 -0.1462 -0.1461  0.0162 -0.0634 -0.1284  0.1101 -0.0560\n",
    "[torch.FloatTensor of size 4x10]\n",
    "```\n",
    "\n",
    "Target(labels): (N) where each value is 0≤targets[i]≤C−1\n",
    "\n",
    "```python\n",
    "Variable containing:\n",
    " 6\n",
    " 2\n",
    " 9\n",
    " 8\n",
    "[torch.LongTensor of size 4]\n",
    "```\n",
    "\n",
    "Output: scalar = Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Train the network\n",
    "\n",
    "This is when things start to get interesting.\n",
    "We simply have to loop over our data iterator, and feed the inputs to the\n",
    "network and optimize.\n",
    "\n",
    "- Concept check: what do you expect the average loss to be at the start of training?\n",
    "\n",
    "-ln(0.1) = 2.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Conv2d(3, 8, kernel_size=(5, 5), stride=(1, 1))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer1 = [i for i in net.double().children()][0]\n",
    "layer1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAG3CAYAAAC30lEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADgZJREFUeJzt3PvPmHV9xvGr8MAW2UQZEJENjJsIgplaO0VNcLQqE2QjMgUZJxnIIWNCwKUIDCgLsCpOQciEqRwM6sgGswzsjDCWMBkpxsMGM0KRQzkMRIcxMnHP/oU+Sz7pD9fr9fM31/dOevd53r1/6LLFxcUAALTaaks/AADAliSGAIBqYggAqCaGAIBqYggAqCaGAIBqYggAqCaGAIBqYggAqLawlMMrH99h9L+r3v2BiyfnkyRH7rfv6P4Dq08b3U+SY37w3dH9heueWDa1/dABZ46+Q68447nJ+STJ8pWHj+7f+4X5vwdXb/0Po/vHHb7N2DuUJN/a+TOj79FbH14xOZ8ked1BPxjdf+BFW4/uJ8ntF20Y3d9z7/PG3qNrDjpg9B06eu3uk/NJksdOvnJ0/yPffdPofpIc+N8Hj+5/4PlzNusd8mUIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKi2sJTD1755tp2ueOPZo/tJ8vSZD4/uv+us60f3k+T+73x6dH+fwe3jf/37g+vJdcf+0uh+kmz79K2j+/e9bNvR/STZ5aQPDt9w3ej6skv/ZXR/uzXvHt1Pkqt//Mjo/qGXfn50P0leceE9sxfccN7Y9Bv2PGxsO0mOWv2m0f0k+c6G/Uf311yydnQ/Sb5x9sXjd2wOX4YAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACotrCUw1c+v+PUcyRJHs1bR/eT5KJD9x/dv+ZzHxndT5LXXjR8waNz01//nRvnxpO8dtUvRveT5KUrvj26f/jr7hvdT5Kt9vnm6P6G0fVkq6N/OLq/1+/uNrqfJCu+Oftn8Nwth43uJ8n777pydP9Lg9tfP3bd4HpyyYffPbqfJEe8b/CHdZJ1/3bx6H6SXP/op0b3L9zMc74MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVFpZy+NBPfH/qOZIkzx5z5uh+klz1rlNG9x95y5+M7ifJHkf8eHT/F1kztn3Gzj8f206Su285f3Q/STYdsOPo/itv/tnofpLstvFH43dMevu2T47un37eA6P7SbJp63NH9x//4bdG95Nkhzt/bfiG/ceWLz93r7HtJFn5yWNH95Nkw9eOGt1/bMPbRveTZNWpR85ecNXdm3XMlyEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqLSzl8N8+e+bUcyRJ7nj+j0f3k2TTMyeN7p9z212j+0ly5JN7jt8x5azfOmN0/9b9Pz66nyTbnXTV6P7f7f3Xo/tJ8tN7XjJ7wXGz83/xqfeM7r//xA+O7ifJHRuXje5fcMJXR/eTZMNBd85esDg3veaG7efGk+y78UOj+0my+8N7j+6vu3af0f0kOeThh8bv2By+DAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1ZYtLi5u6WcAANhifBkCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCg2sJSDu+x++2LUw+SJKf/442T80mSG6/fb3R/7UXnje4nyV077ja6f8rTty2b2n7Ln60afYc2HfC9yfkkyV9u9ZPR/XOP/8TofpLc9Pqvje7v+aXrxt6hJLnhidNH36PzV28/OZ8kee9NPx3df/CmJf14/3+5bo93jO4v7PL2sffo0JMvGX2HNj7zyOR8kuSfH75gdP/ly395dD9Jrnjja0b3/+iYhzbrHfJlCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGoLSzn86gNunnqOJMlXDnnf6H6SHH35M6P7Fy87e3Q/Sb78+odG908Z3P7zXS8fXE92/Px/jO4nycuP+9no/h/8yvy/UVavf8fo/t+Prie/t+ons/sn7Da6nyRHPPvI6P5Vr95+dD9Jnnxh1ej+rnlhbPt/rn/J2HaSbDho/nfBhXttM7r/4PrnRveT5LZTbh2/Y3P4MgQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEC1haUcfuqiX516jiTJ25afPLqfJMtXvHN0/7YXfXp0P0l+f5fTx++Y8sw3/mp0/7Tdvzi6nyQrd/7e6P5lK5aP7ifJR+/+8vgdk9b/xj2j+w8ee97ofpI8/9nHRvcfO2L16H6SvHDQ7M/TXU+b2/7itZfOjSf56ok3j+4nyYv3XDe6f+xvbxjdT5JVrzpi+IZ7N+uUL0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQLWFpRw+5NtXTz1HkuSc9T8a3U+Sr2y1/ej+WYetGd1Pkvvv32H8jinX7LXz6P59d20a3U+Sdbu9bHT/qf/de3Q/Sd581M9nL7hhdv7AG/5pdH+ndz47up8k+66+ZnT/kLu2Ht1PkstOO3/8jinbHf6a0f2PbTP/u2DF/qeP7n/oAzuN7ifJx9Yun71g9eYd82UIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAamIIAKgmhgCAagtLOfyHB58/9RxJksve+4bR/SR5/LQ1o/srH/js6H6S3Lt+7fgdU67Y7onR/Vv3Pn50P0l+c6enRvc/+pkLRveTZJtT/2t0/z2j68keLz5xdH/lUX86up8k69bfObq/3YdfObqfJCfd8q+j+4ccuN/Y9uc++e9j20ly6vP/ObqfJLff8YXR/b/ZOPv7Mkke+vgJwzccvFmnfBkCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKotW1xc3NLPAACwxfgyBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQLX/A1XIViQrG649AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x131eb4a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dont run this cell \n",
    "# first layer filters before training , they should look random before training\n",
    "\n",
    "mm = net.double()\n",
    "filters = mm.modules\n",
    "layer1 = [i for i in mm.children()][0]\n",
    "tensor = layer1.weight.data.numpy()\n",
    "\n",
    "fig = plt.figure(figsize=(10,10))\n",
    "for i in range(8):\n",
    "    ax1 = fig.add_subplot(2,4,i+1)\n",
    "    tensor[i] = tensor[i] - min(tensor[i].flatten())\n",
    "    tensor[i] = tensor[i] / max(tensor[i].flatten())\n",
    "    ax1.imshow(np.transpose(tensor[i], (1, 2, 0)))\n",
    "    ax1.axis('off')\n",
    "    ax1.set_xticklabels([])\n",
    "    ax1.set_yticklabels([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Loop\n",
    "\n",
    "Show the random filters and the evaluation results (after the training cell) first before training, then show again after training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,  2000] loss: 2.233\n",
      "[1,  4000] loss: 1.828\n",
      "[1,  6000] loss: 1.662\n",
      "[1,  8000] loss: 1.555\n",
      "[1, 10000] loss: 1.495\n",
      "[1, 12000] loss: 1.457\n",
      "[2,  2000] loss: 1.371\n",
      "[2,  4000] loss: 1.368\n",
      "[2,  6000] loss: 1.313\n",
      "[2,  8000] loss: 1.316\n",
      "[2, 10000] loss: 1.293\n",
      "[2, 12000] loss: 1.256\n",
      "[3,  2000] loss: 1.208\n",
      "[3,  4000] loss: 1.190\n",
      "[3,  6000] loss: 1.200\n",
      "[3,  8000] loss: 1.184\n",
      "[3, 10000] loss: 1.186\n",
      "[3, 12000] loss: 1.187\n",
      "[4,  2000] loss: 1.101\n",
      "[4,  4000] loss: 1.144\n",
      "[4,  6000] loss: 1.135\n",
      "[4,  8000] loss: 1.136\n",
      "[4, 10000] loss: 1.113\n",
      "[4, 12000] loss: 1.101\n",
      "[5,  2000] loss: 1.036\n",
      "[5,  4000] loss: 1.046\n",
      "[5,  6000] loss: 1.067\n",
      "[5,  8000] loss: 1.056\n",
      "[5, 10000] loss: 1.090\n",
      "[5, 12000] loss: 1.059\n",
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "# Training Cell\n",
    "\n",
    "for epoch in range(5):  # loop over the dataset multiple times\n",
    "\n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0): # for each epoch, sample from the dataset \n",
    "        \n",
    "        # get the input normalized images and labels\n",
    "        inputs, labels = data\n",
    "\n",
    "        # wrap them in Variable so they are connected to the computational graph\n",
    "        inputs, labels = Variable(inputs), Variable(labels)\n",
    "\n",
    "        # zero the parameter gradients so that this batch's gradients are not added to previous batch's\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # forward + backward + optimize: ALOT happens here in just 4 short lines of code \n",
    "        outputs = net(inputs) # forward pass\n",
    "        #print(outputs)\n",
    "        #print(labels)\n",
    "        #break\n",
    "        loss = criterion(outputs, labels) # calculate loss\n",
    "        loss.backward() # backward pass to calculate gradients\n",
    "        optimizer.step() # apply gradients to all parameters\n",
    "\n",
    "        \n",
    "        running_loss += loss.item() #.data[0] # add all losses together so they can be averaged at print time\n",
    "        \n",
    "        # print statistics\n",
    "        if i % 2000 == 1999:    # print every 2000 mini-batches\n",
    "            print('[%d, %5d] loss: %.3f' %\n",
    "                  (epoch + 1, i + 1, running_loss / 2000))\n",
    "            running_loss = 0.0\n",
    "\n",
    "print('Finished Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Test the network on the test data\n",
    "\n",
    "We have trained the network for 5 passes over the training dataset.\n",
    "But we need to check if the network has learnt anything at all.\n",
    "\n",
    "We will check this by predicting the class label that the neural network\n",
    "outputs, and checking it against the ground-truth. If the prediction is\n",
    "correct, we add the sample to the list of correct predictions.\n",
    "\n",
    "Okay, first step. Let us display an image from the test set to get familiar.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('GroundTruth: ', '  cat  ship  ship plane')\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAB6CAYAAACvHqiXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztfWmQJVl13ncz8+2vXu1dXdXd08t09+wwA8MAEkIIJHtAEihsAiMrpLGNYyIcIiw5FGEj64dMhH9IYYdkOULGMSEQSFYIYUACIywDA2KRNDA9K8z09DK9Vnd1Vdde9faXef3jnJvnvFp6qruarq7ifhEdlX0zX+a9N29mnnO+sxhrLTw8PDw8tj+Cre6Ah4eHh8fNgX+he3h4eOwQ+Be6h4eHxw6Bf6F7eHh47BD4F7qHh4fHDoF/oXt4eHjsEPgXuoeHh8cOwaZe6MaYR40xJ4wxp40xH7lZnfLw8PDwuH6YGw0sMsaEAE4C+BkA4wCeBvCL1tqXb173PDw8PDw2imgTv30EwGlr7RkAMMZ8GsD7AKz7Qi8Wi7avr28Tl/Tw8PD40cPExMS0tXb4tY7bzAt9D4CL6v/jAN58rR/09fXh8ccf38QlPTw8PH708NGPfvT8Ro77oZOixpjHjTHHjDHHarXaD/tyHh4eHj+y2MwL/RKAfer/e7mtC9baJ6y1D1trHy4Wi5u4nIeHh4fHtbCZF/rTAI4YYw4aY7IAPgjgizenWx4eHh4e14sbtqFbazvGmA8D+H8AQgCfsNa+dL3n2b/wBQCAsUnals1Qt0wg35tWqwkA6MRtOiabTffFCf3WJuKxY4IYABCEqs/tEu0D7ctkG+m+EO6aco446QAA2h3pW5IYvkDE/THpvibvkxYg4XEZI62tFo0hjqNVYw+4b61E2qrUDdRacdpWuvcxaHz4wx9Otzudzqpr3gxc9/nsir+6KdBt1Bq4Ru14Zdz8Jep4N89ykmt5a63Vb3f8xz72sVX79v8kz23cSdtmrl4BADQbsmYO3XkYANDXWwEAZELpTzZDCy+r23g9R0atsU4dAFAuZfgc0teIt0O1iOfmZgEAPT09aVsmk+Hz0nEmkHN0khYAIFhDdAuMNNaqZA6NIlqT+Xw+3ddq0Tk6/AwCQCFf4GtJ3/7g93636/x79+1Kt8tDR+l3oTy3lZ4yAGCpKeu6ujjD/aX7najFEPEgClEubcuH/ApTz236AHJTnMj5XVui2tw13Njp+jyXa6wdw/fPBPq9EK9xHP02l6P+ZgPpNyxtm6zMX23mOADgG0/9YNW5NorNkKKw1n4ZwJc3cw4PDw8Pj5uDTb3QbwZaLGVZW5dGlk5zKKVNAehLFkUseWuJg7+6JiONTSdVJPIFjFgCDLkpUucwCUnN6IgU4qTlRJ2jZUhyiUP6wrb0vjjgc8nX2rCUn1d9i1gyCiLqeNxuq450eEhyDieRhuH6FrIwDNfdd7NwoxK/no9UjlJSZOJEKstjsLLPaUwGIg3JWTYvoa+FcpHubWDl8WhWqS1pCbGfz9J5SwU6LlKXcWsnpxZZIcv3XY2lGbvjaF1l1TpxUxRFcm+d5B8oKd/NTY61Vr1MqrU2X1PgtFsLOW/AF8uwlOqkfgBoN5s8PjUWljpxjTWRWJHyO2E/nSsjz3QckoQeZJSEXl+mvsVV7oecr2npuLaSjBs8v0poR6tNWlTAz0S9Ju8W95zo8TmNOQjkObROs+HJ1BaBTifmY+Saxrj3k6yZ/n4ac67Qw+eXe5a4dZ2TfsTLZWwWPvTfw8PDY4fAv9A9PDw8dgi23ORi2SQBK6YOy2SUiUUlTNqkAoUFNmsotdVZGzQxkWWVqmNFpUnaYddxTnUCAGNXEHMADBM4NhTVsR6TbndlhtSzakvUqOVlagutnLcnz+SYIvUqRSKUCjkaZxK00n1Bal6RsbsRtJP1zQTahPDDqhO7kfN2mTfc8V26qdulTUQ05802zUek9eyYfhuata6drNG2MVxrLBGbvQJl9sqGdK1MIG25gM1pbp8iNJt1Ms2EoSLwIrrv7aYQqwHYxNahNmvkkYzZtJTNFOR4Nw9qjTlyOGazoY73mLl6FQAwMtQvx7N5JczKtUK+lptnZflBxMc3FUnsCNt2W9pWIrCyL+b+xuo5iA2NOd8j/RjcP0K/XZgDAJRry+m+VoPeEXFZnseklyLPe7Iy9+66AdtlW015vpwDRT4v9yWdUrUm3Dp2fwNl4+3wmBO9/Pjy2UjWbqHAxDGc2VBMOokz52qZ+iY4MXgJ3cPDw2OHYMsl9ChmyTyUr2PAkkYuVF9/xzjxlzLQzA//tKMlWEfyZEW62X3gLgDA4vw0AGB6RiSZTETSeAD5crc6ND11KwFRx8+TxGNzgwCAdigkT4slh+WF2bTt0iRLGnkleU3MAwDu2E3XHOzRUpxzZZSxO+Ejtqtdoxy0ZHwz3BVvipSf9ltpD+za2VHiTZs1pVNnzgAARnaLu1vC5PbwgEiYeSaSkk308VpzlGUpPOmIZBeydJVRhFyG24KY1lE2o6S+kF1jlfaVCejeJkZpZAm74zaYHFXrqcFjLxZlDYeOKdXiIc9DlV0qn3nm2XRXmzWF/sqb0rZcjp0D1BSkrrOsvQbKXdBY5xwga9ImjhhcX0LvQFwrA9BaT0JFCLOWFiptrcTsZqXI9/jZp9N9rWmS1kfvv0v6dpWeuaaReSvzwJbqRKzm1VhyrLEHg0JABkyK6ldKs0jnjdqsubRlspZKdF9yCwtpW7TvXgBAra83bUtY64r5nuUTIVZTi0AsbWG8efnaS+geHh4eOwT+he7h4eGxQ7DlJhenl5tI0uo6dbijIyiZgGqxGpxVZFMcO/VPmST4HNqv980//TMAgGf+/h8AAJfZ9AIA1Y6L/BRV7Pz4FADg7LikqMn1jwIA9o4cpGvmRK1ssbqYKUuWy06D1MSZqctpW7GfzDXjyxR92FDq80gPqYTFjKihcZvUZh0Mt5IOXIsUvRWRotc2zTD5llFRvexjXl8WEnx+gVTjyWkyVRV6RH0e5IhIHdXoSEAdPbpGZ1f0YuPIsnnPqnNk3OTH0u8Qjryntozy6247dTuRc4QVmgdjVdwB+zsnLho5lnW9vEimuXJRSMCA51tHbUYcWT3PZOjsopgSC+yn3VKWkVabrhVl9ZqhtpgjsTvK3OSitLPKx9rymk3i9c2AeuadCTFQY487PFZl6zBsEmkYuu+ZRNaCGSJTXG1J+tY+e5L6a8QslfB0VZ1/u3q+sm2OH7moSHmeD+1o0WDzadjguZJLormb+li/IqbVHkPPvOkdkvHxdduBI5pV7AXPd6hI9ijYvJnTS+geHh4eOwRbLqE3A/oSL9RUBBlLN/1lESsqTDJFLKFowip1O1IEjSNNa7W5tO3rX6K8MZPzJHFMLsv37PwlOu78ZUnxHuZJWo/DStpWqtCXOFOkfVFeJIMcS5H5QMYy3aIotdG9d6RtDSZrzpwhCX12XuWU2UPnPTAsmkKGXfeMchsT+YzHq77+Nrk+mTQNzFxDQNBSebCGhB6zFJawNKKjWV0E3tWZxbRtsUpjrev8HTUaTZAj8rlal3tbLrJEqvrm5P2NKiDXq6nkjHOxk/l2ZOiaLocJRyYql8OINcpIMY+hofmwsb57PD52BIiVa9vyEs3bBX3NyEVWizS5r0Lz5lwUX3jxxXTf6+67DwCQaJfKmOY3r116WVOo11gDjuT8HdYQw0icA9qcL6jZXD8ldqyk94TXsNUyJDsxtLR7I1+3d4nnangk3VfYtZ/6Y4WMBLte2qHdaVM9w7lZrlBeGCgX4Co/r3ZkMG3LJNSnhtLwS6wltpZofE2dY6fAEblVuS/RIGkPJqPcMjlfSw//NFQaQMfQ3JtAuehi89HeXkL38PDw2CHwL3QPDw+PHYItN7lcrZOaMdsWUvSbf/e3AIB7j4rp4qfuI7Khn/3VNRnjkvAESn2JmXxRXBrOnic/59k6qUK2OJDuC8tMvg2IeaDA9U9bKmVqi4m4Sj/1rVKWPk5dIRPK4pwiS1glzBfENHNhjsjYTIXUyakJqS5VvrIEANhdkeMLLlVvosi0FajWdHIzVjmVqulSC4cq0ZPbdulAVU4sBMnqb72LYtW2jmU2BzhytKCIswZH1E0ok8vUHG0nijBrsz2ltkQE8tS0zN/4pQkAwL1HDqVtdx7YS/1XfvkpOesifbWVxXVbhylcgyoN2eSXtMWcELCJr74gYwGbGywndQoLMvYs36usmm/TJlNbrM0UHA1tUiJWzE3VKpkWJifl+FKlzNdUicl4zlvLdFxe+cNfnSdi9dkfiBmmlKNrHj4kcxqx6adZo/VXiFQiqSatrVilkY7do9ZQ87ESaopdCtukK1aE96lnOcPmrtzpU3T6Z76d7uu8iU1VKg2t5RiR7JI8Gw3QPJQ53iPMyfFJic5vrCLqOTlez6C8gzKX2FyzTGsyMyLOD7hI+6KKmEUbV2l+w6K0JUfJN73Bib0CReJnOzQ5kbIl2mtw/BuFl9A9PDw8dgheU0I3xnwCwM8BmLLW3s9tAwD+AsABAOcAfMBaO7feOa7ZgV6SEmoz8m1pZ4l4nK2p5O8tciOqZNnNSxEpTiINQyFtGi2ScK8q/ml6ib7OxT4iRPqHhaisJiRpDEFF5TGB0sqI1NSokgTTWKbj9ytypcbS+FRLpGXD0tLCrJLKWFqp89c/zEq/JxdpGicWRCvYP8QayDW+4PN1GWi5SFpDoPJKuGIdXYK3I2tcEG5X2to1vvVruENemSCXzoEB0nYKeZF8mg0aczEnbbuHSdOySnyr1misJZZkWg2V7pQHvdyU8XXSPBvKjS51n3T7Vg2zS2K8lrdl3hUwUAc5CT2ntIIyk8+9TGYF7H4JADm+x3ktkLIWFTRkLaRFD7hQSmtR1lpPifb1D4gmeXactMAzF6+kbSdPPwkAmJsmiXS5IeeotanmTATlhsiS/wN3HU3b3vuzjwIA9vB6buZlnI1qlX8n16xwAXpTX8J6yISy/lz6a0eOApJCNlJyZXmOrtUZJzffitI2li7T9Vt5ica0oPeCuTKVtpXGmNCssOYJeZYK7C6bnZd+N5iI7kxPpG1ZnsPOIs1VblYcI9p11qYKouHMnyVnimxBJPSeUSJxXSooq1wUm44MV2u4lWxeRN+IhP5JAI+uaPsIgCettUcAPMn/9/Dw8PDYQrymhG6t/ZYx5sCK5vcBeAdvfwrA3wL4DzfSgbte9wgAYPypE2lbuZe+/o+89c1pWzEkO3OLJWQtfRrORhdbyffRs4vqVz//4ik5bx9Jh3v2kyuXVba4DEvhSXMmbWu1klXXCvmL+tILLwAAKipBfbFEkkFJ2dEuX5kE0J1nJmSpY4DdzebnxP43N0vbZyfENWtshFyyoqyKbliBqCKaQszSdVvX32PbZPoXYtd0wSpaIrVr+DA6AV55SKYBLi7fB5TraB+7frXb6lwstRXLYpN0ErrhYDGjXMRyBefepcqqMTHSZXNc1Te5Zqb7EN69voh+8dw57rfM99Iirbu4LZrCpUuknczxGqguiz151yBJ1eWSBAWFXJylpTIURpxrKOBcQlUlvTfcYFShjQuXiX85Oy48Q7VFv833sutcSSbGrcRSVmS3ifMUjHP58mTa9u1v/x0A4B7mKob7RCKtL5Pk78rDAUD7HsqnsrywvmKey8rYrZPWE6Uys4YTKDfbZQ4EXH749QCASvTGdF9tie5BW+V9MjmeG1WeMVOg61bZPVO727Y5X0pGPRt1nhvtNFhnu35tma5ZKshYGnx8rizP+UAPvXti9a5Y5rULdqMstFXGRu6T9jBu34T8STdqQx+x1jr95AqAkWsd7OHh4eHxw8emSVFLxst1Py3GmMeNMceMMcd0nmYPDw8Pj5uLG3VbnDTGjFprJ4wxowCm1jvQWvsEgCcAYGxsbNWLv9hLpoL9h4SgqbMF4o6Dh9O2IVbb58+eAwC0dXRZh0wXj7z9F9K2Ow49DAA4+MC5tO2Z58hM0l8mE8blKcnlErEbU04XV+DeLleF7JqfJbVzoJzRh1A/2KwyNCy5XFzRhuk5MaEYjqbsYZfHKFTECKvcr14cT9uG+0ktP7JXuU6twCf+5H/J+bkfGaX+lXtIZTx8UIjgN72O3Kpc2UurzEKOZLTavuJy7CiziiPssjk6vyY7s1kyoQz2K/dJVxtW1WhMc4Rk6ByNjpx/nknieZWqdGmBTABt7arJROYgu54dOSyEVcZFE+rC8EGXAaYL3/77p3i4qsCKI7LrshbOXSHiLq39qcSj/l4yWZQUSZzj4zLKlTFil7qAa4rWFKEZ8Tmsylt0ZZaI9LZit4s9zt2O8x0tK3dLvh+NhvS70kPnfcsbH0jbqpzyucEuuhcuiCnl1VdfpbErF7vzMzT39ZqcN8oJuQ8ApZI4GHR4HtqxvmdcaEaRgYZNUIURIj4XqzKWqws0dqPccVtcMzWrycV5+o3LBZXLynOwyGs8n1GvPpfWWEWKNjl6GVwzeKEua9Kl0SmqaNqevWTiDbUZMK2Hy/dK17Jwbw61KJOb4Ld4oxL6FwE8xtuPAfjCpnvi4eHh4bEpbMRt8c9BBOiQMWYcwG8D+B0AnzHGfAjAeQAfuNEOhDkiFi5PHk/bHnwjJeMv9coXP1wiAipmKSFS5bPOXCTi4m39B+XERQo+6SmpKu0RXavAboL5rCoVzl/nPWOjadPLLJlkFbmzyMTMwX2kURy9+9503+wsF7OoSIDCZXanMoqE6esnqXaBpU+d/6RQpN/Wl6Tfpy5wsIcitkYkdQUdX1PBT3XazqggnyUWcIuqLb7nbgBAwzJ5pCT0HEtKWqp1hSp0FsLeAdJGUuJJuTs6N6xQSeMu0kvLIglLK+c48OvSlCh8szOkEdXrItnFTZZEVc4Xl1Nk7z6ic+7YtzfdV0rXiiZ915fQnz9F/SgWRCOyrBE2O3JfejlrpiP/WkoKvrpM9yBUc9WTJ42sEwsJbpgEDNm3zUQSqJarkmTZagvZOjvryFBdLo3+tjhHzFJV5qrF7qz7hsX1cbCfFo8LXAKA2TnKAzPYR/14+PX3pfvG2TV1oS5r+JVxui+BWtcHVzBpkcp0WuihZ25ZlZSLWKWJVZbBiINvAl6TiXK3NFzwJlLXdFvtlsowyVp2xJK31ogcGRorLdCVtuuoVZkpMGkZr87a6nK/ZDpKU2CPAZ2xMR+7DJ18LbXkXGBdtxfx5rOjbsTL5RfX2fWuTV/dw8PDw+OmwUeKenh4eOwQbHkul0yeCJpGQ6vPXL9RRVAWS45kIlOArjdajkhl+uQTH0/bfv6ffZjOoaLbslxL0RXLOHhoT7pvapYIrsayqM27d5Hfui4Y0OQ6j4cOE2F752Ehcxeeo1qO1SVRKx2p01ERcnU2ifRx/cHYStRabz+pix1VkSAMaHzjl8UUMfI6dOED/+SfSh+ZLCyp/DGOhCkoU5VLLbG4yPlVOmIKyDBJFyn/W8uqa135Z9uEzueqomsiNuLjMxkdgbrabOP8bxuc/6SkcmT0cz6duCV9y4c0rvkZMRmMXzoHADjMRHoYKNOSdRXtVYrha7j8LrJZz2rikWMLCqHMx959d1L/XZrgK7LWptlUNDIi9VFzQ2QGqs6LP3fCkbC9/WSvyOUklqLBQ651xOSS5+cgbssaC5lcdEVfMllVaCNP24+8QUwoR/eP0flbstbPvkrjevXEywCAt75JCNN9++j4Cy9KzqF27HIqrV9TNKv6keWauokVM2eBSfCOSlO8xJGyMROf+V4xFY2U2ASmyEO3rrW5IoSrmUp/dWGOtWD52dQml5h93V2a4kBdM+sMPSpRVJPfKTp3VMQmxxicP0YXXeHnRtd11abXG4WX0D08PDx2CLZcQjccQVZTknGDJcyMzuMwwy5FnK8lg/l032gffTFPHZeo0Mvjp2mjJqXfzo+fAwA8tJuiU/fsF2ZxbIokpOppkUIGciQd9vRJWalXXz1L1xwj6X5+UaSnNn/pJ68qCcyRJco1scYSuuHcDpoKKbnsjYlEfmYNzUdr+grWQ9IWCSKVUNT+cpbOW8jLnNY5U16tTf04d+acXJNJ0TsO7k/bzl6kufzS3zyZtrU5w2We87UU1flddF1vRaIO+3pJynroIVExhodIKr1zL81poNwFnZTliCtAyK76LpHexkbpXo3tIVJbZ/CrsWtbl8ZyDVEmw0T98K6xtC3PhPT0tLiTVjlq2YX7NVQEaO8wra09yvW2p5fGWRkSqX2GifSYJba2qujmXCRrikhstR3hKRpL1mX0zNE9zljRoHbx3A/3yz3IM8E33C8sZoVd+2YuXAAAnH/1XLpv9wCt/4XJp9K2DJPhrXD9V0ikcpeEnEUyr/K7zE8RwTu7LDlUrk7Q/Pb30Pq//17RFDKsnTcVIdxmDUET+m79u6IvgSLqnZSsSyfGKRGrWcvu3EA6kyvSc8gzF/Hxeu2632Sc5qQfdD59oFww42u40m4UXkL38PDw2CHwL3QPDw+PHYItN7mkqW+V+jI6ROqWVt+//iL5hPdzkv0jA6IC5XNMCkXii3116hydvikRb3fcSX7qIZ+3WBECamiECKuZWVFvF5gM1YXNd+0idTlic1BDkZcu6VJdmQc6/OOOOkmjyak5O/Q9HVQquOFag1kjY8kxaRTb7kg8jb/6P19JtxNO2B8oH94yE8w9yvxx4AiNeXiQTAyDoxJFOsB9yqvkUvPHyRz1/eNSd7VuXTEN+n+k1OEK//bwHWK2eesjb6BrlcTHu8Rqu9N4W2pOO+xbXVsQE1ub/bgLRelbXx+ZGyY5Gdq0KpJR4IjFkd0yz8WiikFYgX42sYXKnNDkQh5GyUCzM9SnxUVOg6xMhCFHGJ6/JAmwKotkLuntlTgF53/eZKcAowjCnItmLMl9L1gXWapzAdMzUSqwOdKKOWbvIM1LURGU1UXqd0eZclzxj4NsIjr+ypl039GjlIgLigC9fJl80/P9YvYC9HY3CeiKrSTK/LHEMR1Xr4opcX6Oznvyxe8BAF554R/SfYcPU8zHgcP3pG39Q2w2UuYKlyraFTvRhoww9WFXfUsLvUibq5ErhXQU6crHa149jaxeg21PSdeu5Hd8VnW/9bvkRuEldA8PD48dgi2X0F0UV29ZCKu+Hto2KmfIoiVJY3qOvpRDPdL1EhM6cSCSybnL5wAAI/2SDH8/f+GdO9j3npHo1EsTJMn3lEVqz7Bb1UunL6geu0hH+ttUX9VljtDrUwUJOix2TkyqBPw91KeIXaOKRZHAXP4TtIVYjavUt5Fd6+dyefq5H6TbhQwRlM2mELZZJvXe/JY3pW3nL5GkPcOc1P33iWtblgnNWlOk/AxrNm94gxCaDY5EzLI0eeSQROvexylWx4ZEIq0U6d4myk314hWKUpya4+Ie01fTfVUmy+fnRUJvcQrbjHLBdLlkXCRxWxGUxT6at/sh4+vtXX8unaRdU5GooXEl/EQriDkVa8QRyIkV+Sibo/MPDUnkcZnXeF65gvZyvyO+Z9qd07JrYEe5k/ayS2egoisTThMbuejKpkjevZxAxnZEa4xZ62mpSMc6348ir83zV2T9vfwqaX/NpkSgths0vzbU1Pv6cFJtPi9jv/suilQ+fI+4D9eWSFp/6VlyAX7umBCx3/4WaYjHX5a1fvSeBwEAR+4Sqb2vn9abI4vDrj66+V0j97ImW13JvM7qso8uejRWJGqSuk+uj6701MaVzZQ1rFNs3yi8hO7h4eGxQ+Bf6B4eHh47BFtucnHRe7t3iU+4qzGYKHJxdC+p8sfYlDJvJEWtDUkt7x0S4rG3wj6geVGtD7DJpcwpe//4E3+a7qvxtRbrQqbV2A9YZ9rczZGcjVlS/6o5fU0yC71yQvzhJyfJfLCookf7+uiElRKpz6EisTIcvRfWLqVtwyXa35sXhU4lIQUAXL2o/OcHyGy0d6+QgPe+7gidPyfneOl5Ip5GWA0uq2pGU1xfsVQRk9VghY5776NvT9sCduju7aXjhgbFf36WUw2fPS/zsTBPZqDFBYmOXWLyeZ7TFM8uSgRohwnejEprnOUKQYGKrOut0Lj6OLK0X5mncmzSyhbEtLVcF9J5JQbZh1z79pe5+kyi0r9mApqPXeyvblSUbJZ9pp0pCADyHC0Zqjy7zsSSVmlSJhfng1+rytpxEYs5tSgtm19qCzTfl87JfM+y83NfQY4f4RTD+byuwcsmlIjMTVFRyPOrXN9z36g8cz1czWuxuT6Rl6i0uC6Jlw10G/UtVL7pfYOUhvZt76C1e/iwmPC+882/BQCcPSvPRvU5fm4XxST3wOuo2tG+fXQunZ467tAaj1XfEjbtdlXpSuvnur+yy9Xb1QS5s5Zon3dHkKbX6iJF+R2nzDbahHOj8BK6h4eHxw7BlkvojgSs9IuE3ompW7lI3MCOcmGGY8+Q5LWYkQi8xJC0N7JHvvQvHyd3px/7yX+Ztv0DFy6oVklKbLekwMXUFeeKJ9+4Za4BGKmovP6AJPg9BTrHwlWRhjohScYju4RYjdnVq64kwkadJNIqk2+dRCSwdoMi5XZlRBIcK5Mk1exI20oJ/dLJl9LtRSbOfv4f/Zu07dFHKTnm174u7o27mCzcVeQoUuUKl+fouZFekdR6eDuv3AU7LNU4SVTnrLlygiSpC1PiutfiQiVRXtLE9vQQibyLJcZ2azURlVFFClzOC537oqeHxlKp9PA+VaeS8+lMTsr9bjTWr55VZOm0rYjbArtg9lVE60nSVM5EaBZUndSU9FLSYWK5TctRrriI+6vIug7f704sfV2coTHoBzfDEvryAmmDE5clOnpkgMbSV5Jo5xpL14nSFDp8RkfE7uGCDQBwF9cZffBeKRpy8gw9L899XxwLVkKnjA64AEUQidadYaeAWEVXuvSzAZPER44KAZ+wm+/ExOfStrlpGuuppmh1k5eoPvGdR4h0vec+OceuESKpI/Vu6bS5+IZKqRtzjVx3H9csiNKVU2b1/jRFM8+DPkVaTEaJ/l3RqDcIL6F7eHh47BBspMDFPgB/AioEbQE8Ya39A2PMAIC/AHAAwDkAH7Av2Ih9AAAgAElEQVTWrl8CfB243CX9QyJBdPhr3gikMEK+zJIGZyi8cFGCEd72JnJHayzLF7PYQ26CE5ck98bpk1TtvOOqgStvpirbbXsGxc1sYYEko96ySKR3HaXcEk+/8AoA4NnjZ6UfP/UeAN1ZIs+cJgl+XmVsdC6PjTpJ5vtHRLIrcBDJwIBIxjYiyaHTWt+tqaFKgT3weurjO9/1zrRtsI9s2z/+ZmX/ZsmuhzWFSlmk5pCLNriq9IDYanXRgYU5sttWWOJJVAaZQ3fdDwDYtVcyUs7OkWbT0yeujC5zn7GrK7I7O6wrjQYAy2xTtqpkmCuccHGCbP9OCwKANhf/0PldiqX1A4uqrE31qAIXLshoSuXpWeRgp4SzMh52ATgA+jj/SZjR0idtay2mxfXMasydNJrS706L5sqoghi2SceXlMbS10caTiFLNu7IyDrpY+2ut0fWZIvPUVPZJFuc4TTgQJd+pZkVOUvpuOJpWLjGfXcdSduuKndTOpfmA9hervqW5d2JfhBZcnU25pbS1vbuOwAAOHDgQNr29CTd744qj3d1ap77Q9L78eMvpvtc4NSdd0q/R0bIbbKnR/gicIBfo8U2d/XsZVgj00FEzm1RxxVZo10jaVTp6dOCGILwJhS42IiE3gHwG9baewG8BcCvGmPuBfARAE9aa48AeJL/7+Hh4eGxRXjNF7q1dsJa+yxvLwE4DmAPgPcB+BQf9ikAv7D2GTw8PDw8bgWuixQ1xhwA8BCA7wIYsda6vJdXQCaZ60bCNRp7B6SoQbVOak4tFhXFEWCuVuTJl5QrXI1Um3JJcpFw7QGcPylq4iUmi976Vkqfq9OS9nA63IExcZO6MEtmlXpTJbcvkXpbGSbS6KEeqV15ldXxc+efl7HUyDwxvyDX2jVMqnGvpf7sL4ur364KF4UwYkJxKVNLSoUVpz/CobsfTLc/+Cv/msYXi1p+4jQRk4lROXCYPG2z+jc7r5LWJC6PjdCvrrB6AiG2lhapJ+EkqcaXVT1QV6gkaQjZVGIC9swpMYWd5ZStzu1vYEjmw5kHFhaE9JqZJmLQKhNKwO5wJnB5TVTkMROweZ06eHklrSzIsYvkzLSM5dU5uqaLsgSAvn4iv0dHaem3VFRhu0Vmm8RKHxfZLFZX5qCYIzhDNmfp2pXOrJIvyVgK7K7YUGs3YSKxVGY3WLVOshwlqQlkRzA3FAlo+DhHSrZVEZPxGbKk1lQNUkcq7h6V9b8SoTI5pNvqmjA8X13ufO43ZtU+F2Xa0yPmoJSs7Cpe4kx4dK2lObmPz3EK6pdeeDptGxik+7h7txDBu0cP8DXJDDOoTLHDXNDXKOLd3eeOMgN2mDRN3Ra16yObu6wyv9lkpYnm+rFhUtQYUwbwOQC/bq1d1PsszeCaBl5jzOPGmGPGmGO12vqeBR4eHh4em8OGJHRDKQA/B+DPrLWf5+ZJY8yotXbCGDMKYGqt31prnwDwBACMjY2teukvcSKRgspUl2aeS1S5NCZThgZIejsZSDa4qVmSfGZC+cL1lukrevf9QnScOUeSoCsioInKI0eIJDly8M607fwESSQvvfT9tG1mmoNUuAhCv3JVG3+JJPqJafneGSZ2QxXgNLqP3L/28xf7jh6RwPJcyqrZ0IEPJFFpt6qVeP8v/fN0u383SU0v/ECkYEcutZQUEDNJ50qtaVLGlfaKtQTBbUGXGMC5UzgL5vSMuCg6tzsVS4K+Sh/3RyTd2RnWRlhKnJ4WArTJ2klHuX3GXAYwVLlcinma55xzadQV2V3yHoj0VFBZJFdinoney5fE/a/EZPXdquCCy0hZ5Pw0jbpoVXNz5N7abss4a5xrpajcPnsrtO5LOfpbUGRnxFJnrEjRTqfF51XZO135s7QYgyqawFpuWz15UcikXqJcaTmb5MxV0kSmZ8TF02VFnFP5dJymlesRbWoljNUSOv3VRKFhqVbnOEklbf7rCEgAqC9TP65ckYIYly/T9kJRjsvwOnIkf0nljylGdJwmyC9xUY1T5+SdUq9TEZdOTOcaGpZiJw88QAGKRw6LRD88TGuh0ivOHbkCaRIWfH317HXSJI6KmL4VpKihnJIfB3DcWvt7atcXATzG248B+MKme+Ph4eHhccPYiIT+4wB+GcD3jTHOOPwfAfwOgM8YYz4E4DyAD/xwuujh4eHhsRG85gvdWvsdrJ8V8l2b7cCZ06Tm3HFE0l/mA04D2hLiKmK1SYgRIVHLXLTh7rvFD/hrX/kyAKC2IP7qxUEir06Pk3Vo314hUQ/eRYUXckqNP3QH7Z+fFff6l7luacKEy/ickEeLTOY2YjEfLc6TWWeXIlzOz1DbwD4yP8zklE90wiSqMq/YiGspJqK+r/Sifu75Y+n2i9+n766BmHJcvoxIF2FIU8Fm+BhR1SNOt6vTnbp8KlnV34D91ENL+ypZiZIN2CzVDpV5gCNnldswspxrpV1j/+iqmKxaTBqatooeZZtPS5HmMUeDVpfo+KK6j8O91I9ImTqcZWMtanRgmNZJvyo84go0RGo+lpaJmFxepv7mcmIucaSiTr86NkJkeC4v5gFHhlrOJ1JtSI8aTDjPz0l+oZlZ8vWuK/POPZymOMO+/d0FHbjeqVpPTa6FOp5GR4sPeYvNWbWqnH9hnkyPWRX16sb+5Ne/nra9/c0PoQuqeEPi/Ms7KkKTTTLKHR4mNQfRvlBFzr7w7DMAgOU58XcfZP/6ixPSVmEf+iw/N4mKsK6U2R9exQdkIy4MklNxGAGbcefIzHTurERiz8/RvD17TOXu4biNffskmnaMC8aMjtGzPzYi75sSp+k2BVXvNFg/NmKj8JGiHh4eHjsEW57L5fnTJC3fcf8jaVsC+joaTQLyF36RCZr5eSFtBgfIZe89j/5U2vbg6ymPw2c+/5dpm+G8DL1cfX3PmLhclZmsCzsimQzspukZPShS1gIXJ3j2eZKCJ5aVu1SGCNjeUSGKhg5TW1dhBHYTPMFFO05fEQk2y+xRXUVGVnkaOolIFe8W4REA8O1vfjXdrnHmuWxGlS4rOlJWbnloOX+Hq5Ke0RI69SOfU4Qtu/1lVZa+qERjzWdpnDmVj8KlCjEqS6Qjt9uqcEaDCc9UqtURdny8Lm2XhvgqibivRNu9JRpTuSBScC5D58sYuY9GuR+uRJtJOu3mGLFLZdxF9Lnyezx/SjTOsxRer8o465xhsq58Tp0mFGScG5us+RPHXwYAnD93Lm1zUc5WuUOOjZIDwABnvKwrbzK3PT8nhOYMk751pQG7nEPOE21+UbSkgOe+GMnacflirlwRDXilhN5WRTUcKW86cg4Xlaqd9SyozZGoy8syWa6Yyl1HRZt/w4MPAwCeeVGKXjz1NGURnefiKHFH7sGuUSI33/a2t6VtEd/nc+fFxfmppygX1P33UhR6pVecKyZ5zJOT4gDg1u7uEXFvPHjwAF2fHQuqS+L26RwMMpFoBY01chhdL7yE7uHh4bFD4F/oHh4eHjsEW25yOblAKv10rFKPZkgFD1pKRUlcDT76OzYqNoef+DEiNPMZUUMP7qfIz599/wfTts/+5V/Tta7QeScWRNlrNE4DALIQlXe2Ttunz4taCVaL7DCZdPpHxPyQ1hVU0ZgJmycSIyYAl4xqgSM58xmVhIxT2FaNSi7FZKRNtErWrZ6NDEv03ESdCKI4FjW7wnVOI9W3xWkie5cWq9wvUU0Tpy6vFb2mzCqZAt0Hm6Hru8RqABCwzaWokpW5yvRxe7U5DZwEymTFdpFncrOgzB8DPaSm7lMxAHtHyf/X8Z7NhqjqgaX1FKnIvr4Krbua5NpKcfIkpYS9775707YCm1D0dARMPyYcHTipomRdsrdmXZk12IQYK7PKocMHAADDu6j/uvBChs08fSpRliNUdZlM50P+yglKG7usCmK4fTqGIWGTUnVJ5qjG/axxNGtLmcRcMY0Lk0I8uhqv8TXqYNquCFDrNlK4KE8VxIrEEal8qwqq3u5PvONdvEt+4IpXHH1QTLb3v5Hq5rqyq4GiiV0BlkOHJN4k4jk9cETS7I7dQURzgSOOe5XJxY3LFXABxKyya1jSgLtkXyGbqgLF/sbs4NBWdrrErD+XG4WX0D08PDx2CLZcQj8xT9+UL3xHojEf3E/Syu6sEAZFlhJGd9MXcHRIpJY7DzG5aUWqmOC8Kp/49F+nbc88TySTi0TtCry0jpSSc8Q5ukasiT52BewwwdoJFGnoZlOVkmq0+LzqSxwxQRqyNGZVrpMOU0QZ9TV3pcha7fUjyWxbJPreEkkcS4pYbccktd19z/3ymzGSVqY4OnBKRQcuc14Xna7BSZY2lvOWIpJC7n49pSW9rErLXV0kDaDeEomxzoUldFRqjl0pS6yJ9KncJcNcwX10TCSfw3vIrXBXTsTUZXZ1nGW3vjAr81csEQleVhG5g5y/4/JZIcIc2izdN5ZFwwkcGalETFe8ImbXxFOnTqb7lhYcMS2PmCsCEinxOuGQwYAjbaFcMQdZq9Jka41TLtfrMqcXL453HaeCD2HZxbPWknvmpOvqtGjAGe6nK/nXUZGUVXZb7ChXSYm0XF+qrCvtJGQXzMiqCF5+XjsqgrfD8+DOr8vYOYG/ozQcVw6upXKojN3B+ZgSTlGbqCIS/JyfvSCuoPWWywOkCqb0Huy6/tyCXDNiibtUOSCDdfmQFmTMlydn+RzU8ZxKB+4CYE1Z1kdjbv2yiBuFl9A9PDw8dgj8C93Dw8Njh2DLTS7LrIZ87VlRV0++StGj736jkFJ3jpFqf/YMRWq+/U1iOsizqr7UEnXuM39D6TGffVkSLNVclBqbPAKVqtSpRYGKbnNmklipc002hbRZJTTKt7nJEZeaDIqi1fUvi5xIKAtXgTzdhZhJRZ0Uq8MEYrZHqvyszIU2c1kSccVtUt3qSh2uXaTEZAOqwvowp5XNcJWcgsqiVQ9dBRZtl1qtZtfqZKZ5O1eNuu8eSV514QKZM2bmJdK26cg2RaZFTHQXmMUaUgRoX6nEV5Z7cGWaxnJiWpI0GSa2KrvIjFSoCGFaZBJVp+UtK5JrJQp8z1rKrOHI6q46mc7/nM0VlYpEL+fZp79cElIv5HEVVbSpM3GceoUSuy3MiilggSM6Y+VznslyxKpaTznW3w3PX01Fm04xcVdrijof8hj6e2U9tdg8V2Mn+Y5K/pWk5hWd/5Xnw6wvE37rW9+QsXSoalApkvmIed21lVnFEfMuIZl+ltps2tLPoyMcG01pi9MKWJyKWtUPHegjc265rCtm0Rg0v2vS8bmEZyqik8ccKBNKxEm/ArP6ODeErvAKw++PohwfNNhcqAjv64WX0D08PDx2CLZcQh8covwWs3PyeZzgqLa/57qdABC39/MWfQmHd0uUpwnpC/y9YxIt9tdfp0ivZiISAfhLHQSrv2MxS45WfaadO5qWElyUZ4YlA6M/p5yHQpNerhalzj0T8vVDyxKHVZoCS/labB/dTdJkT0VJlbVuCX336EC6PX5hnMekiwnQ9tmTJ9KmBXYndFevKrfIKktDSdzFHNPxqphAq0kS3bPf+QoA4B0lGef9PM56r0jLjgTUUcANJuwWOHpTk7PnX6FovOm6RC42MnT9wi4Zc/9ukrhyFRpTqCJFi+z2lysKyW7C9Ze+c42NO3IPXJRx0lHaGo/dkaIFFUkZsNZYVzlRmrOkLV7QxSl4HlwKWZcvBxDyPJNXWgFfotWS+VuaI4m80Vjmv0JkuzuVV2u+XecUvKr+qyMw3V9NRjr3wo7STixLtdnM+kR9XkUqt0O+Lyoldo6dDhLl6urcNgO+piahE853o7UCFzGbWBUFzKO2rm6nUSQ0375A1cWNQk5Z3ZTI1pQg5eHpmqVt1pi11u3WjFHPxsr3TEtFvVo+R0O9PnIhaVNjY/txo/ASuoeHh8cOwZZL6E6azagsgJ0GSVdnJ0Uqa1Yp2OPtb6AK8oU+yZmwwMUgvvldyThYZ9tvW2W7y7HbmJM+1qqgFCppIf3YKttajiU740SlQB2fIymkoMqfORentgqkWWKpzQVlNJUk2NvPLpujkii/zP6QdRUIsvJTfMdRyeS2yC581fFpdQRn3VPuaLN83SyPuaXs5WK3Xe2W1lWQgHHqRcqfcXFJJJ/hgOajS8NhqWVZ2euvWJIKT7NNdVzlAKkVWcO5QwoMjBwkCSbfJ66r6X1gqalcFk2hyPb0QK0xew3b7yLnCaotidvi1GVak42G9M2Vj3N5PPQ9dppeoIKZMhz45ngVQDJcRmxz1y6KbbYj63wwzSatnSXlHuduW6nC7rBKMrRtmufmsqx1VyRjQUmkTjJ39mmj7OWJXR1c5nLbmGT9oiuJuo/LVeJRiqG+B/Q3VovZBUC12A2301GufFzIwyppXLJaynPYYRt67LRBda9dUJUWnq2lfjYbOrdN3HW81txtyufEqs0FFeoiMd3XDFu635w7p18XvqHtMXgJ3cPDw+NHHv6F7uHh4bFD8JomF2NMHsC3QDUVIgCftdb+tjHmIIBPAxgE8AyAX7ZWhWpuECnJpInBkFTHliJtJpdJLXr2BBFL76mJCrRkyRRxaU5MEnlWuTs1OUeDVUxXAzJSUXxuX5dbmnFuT3KcDbpTzmZy4oK2zK5eLZWC15lftNnBmViqHLFa7hPzSj/ngmiplJ+vsEtbRrlrvXGFVlbpF4JweITyq0wok0uq/qnfNNms4upNatfA+BoRgF17+MRtVtmr05LvI8hxSmLlMneZr/E8RB0/HfF8lEmNL+2TIhnDY5STZ5CLTgBAjl0BW6onls0CuYir3EeamHZtirS8hm/YlXPkQqursDsV3OiIX07f66q/a3U7y+YdncfG7deEY4dNDMvLXPO1qXOusMuc0S6EtC6yqhjDyJ4xPgdFdC7OiZtohwtWWEVCO3NKraXNMM6c4XzssOr4jBq7KzxRqykz4ApcvChOCqcmqB8lVSM0YltR3FWSg+bURYMmiqjPcq4f3eZMNLFObcTz7EhLo3KkOLJV27ZcPhh9X5x7bRK7KFJFdrKJsitnkyvgYVdHtrpftlWeqHiA1sWeB8Q1u9fd0k2kdNmIhN4E8E5r7esBPAjgUWPMWwD8LoDft9YeBjAH4EM33g0PDw8Pj81iIyXoLADnZ5XhfxbAOwG4UvOfAvCfAHzsunvgyAZdOICDXxKV98HlUzk7RRLBJz7z5XTfO99BSe7PXhbpsOqCBdQ3K+My1bGUUFRuR1kuXFFfEunaERdWkZYZJiidBKiJMCcJJopAqbOLmm5zx/WxVD2okuJfnaHAkvlpyfA4f56CqQ4fOoj1UMiLxJbjAJaMymcSMzmmP/6dVHLh8emd15ASuigyloaWeXyvKKmvl8vTvdKQQgAvsfYyUxHJdXAfjWv0IEnjfcoFM8dukIHKx9HmtRJGqpQbS8RRGmQjx6fStXYpuwYpGibsuqdcR1P3Qn1e1tYC6yQ2OUeTXTA7bVlPTuLWFecdHHmeyeoSgVw2UJPKvBbzOeX+V6DfzM7QNXUWxQxrnKGuLs/aaEdLkytIva5AGlfwQ2k9y1xEpVaVfDArEVhVvtBJq7FItU4b6ApOCtlt0TrXQKVpsWSs4qzSubfKNdHdCCs+iimcFK5dizt8/bZyCkj4HWRdiUD1PKR5mVRHDFaPxTL53eEAxorKR7T3AXLuiIzc7/mTnM9qr2ij14sN2dCNMSEXiJ4C8FUArwKYtxJGOA5gzzq/fdwYc8wYc2wtrxIPDw8Pj5uDDb3QrbWxtfZBAHsBPALg7o1ewFr7hLX2YWvtw0WV29jDw8PD4+biuvzQrbXzxphvAHgrgD5jTMRS+l4Al26kA4NcqbyhChJUOZItG4o/t0ur6XyJv/m9F9N9Z7m+4XxVmJHZZVKbFbeIEqvvHVa7cqp6vVPV8wWVJyJwPsKi2juf2Q6bGIz2T2UVLFYV6lvsJ1tQ+Ttckv2BITK1tBQh3OSCDvWcXDPh6EFdEX4l2iqis8r5OHr65JqNKqnZuoBCzOphmrFVpW41q60CKaxKD2yZUKqyj/C3VVGS8zVqm1H5KqIRqoA+unc4bTs4TNuDvTQvgYo2rbIq21DEVsSqv675meco0Iirr+cLIjzkeO51FOa1kKyRR8Qpo1aZfiyzyalJR53DRRrG2mTA60ivO7fGHEnbZfVK3HoSUjlm8rmVkXtb57S2ztSSaAKUc780lHbsxmW1L7Y73pkrVD8iHottCZE9N0NmtHZr/TXZUX7oMR/XCjQh7PL66KIo3MTPUqDugUuRm2jTCJvFEpVu2hHSzvqhj3cmM23lSZx/uDKxOTNTaprR/uVsFoImbJ3ZRr0P2pzGeuAuKqax58C+dF+D65G++orEzhTabNmWIPjrxmtK6MaYYWNMH28XAPwMgOMAvgHg/XzYYwC+cOPd8PDw8PDYLDYioY8C+JShhAgBgM9Ya79kjHkZwKeNMf8ZwHMAPn4jHWiw1JlTn5YmS0iZUKTUDn8oXcL+oCBS3DkmQwNF2nRYeuooQrPBGeWqHKmpiR8nNZWyIsUVmCgNlFThCMdCka6vc2pc5Ux5iXJPipgQ6a8Iabl7gLSS3buJ/JuviiSzyJkJlxckSrGPCx1MX9WRn0PQaKsq9mGWxt4/LNdsl2kuO22V2S5xf5kwVRK6G7KOGEylN83+OeKOsxG2VQ6VZi/1+84+IXn6Byi6s1yRpVcu0n3LMeHcUPlSWuzmaJV0HTp3U90P3s6wpqXdFl3xBk2w2Wuwvg129Yu0u6pzhdOujzx2V+hCr6eVkjd3gLqqIzl57p3bYKwiL9s8D6HSzNqcDyRW7rWlJmk2TjLXuXaadZbu1ygVl6wR8ev6Een55n7PTkr+oDZHrOpbsAp66JzzJcjKNTMu22ncVZGDf8pzpU5nXYZCpSHmWQPprwiR7krOuYIsek5DdjHNKQ3Y5Wnpio7l++IiZ5cWVR4WXp5JJHO0wKkUoyHpx/6jRHz2c/T3pVdOp/umT1NG2Uj1LX+NvDgbxUa8XF4E8NAa7WdA9nQPDw8Pj9sAPlLUw8PDY4dgy5NzOZUwp5IYFR0x0hZV07mZJuwFrRMGJayedVqKxIpdCk1NbNF2kqbolO/Z3CyZOmbVNStcGKFXRWFW2Hc9DzLHuOrdABCxShiqWpdNTubkCiTo4zo1rtVYU0mM5md47MLm5jkisXGN6MZQqWt9g2QOKpeUH3qTTVDK5NKJnW+68z1Wicb4Wx90pQNlM4JKLhWxCl1kE0dPj4pg5CIC5ZyQ2yX2Tc/mRF1t8eYy+83XFcHriNu8Um+zofPZFrU5WGHO0Pe9xaRXNqtIrMz6c+mifwNl1sg4U582l3Df3Ax1FW1PIwdV8qp4NTHtIqVdoYtWS+57nU0tcV1FdDIpWlJmqUIvqfQdHme7IecI1rCJpP74miB34SBsiiqpGI0q14ZdXBQzoLNY6TWzEmFHzTHX7UxUhLAF9TeEShnM2xJVqwhNY7v+AkDCyfdqkSTyk2hvl/5azTdHczfa0je31k2XL3vaST6TCkXl62vCu8KpnIePSqxIwO+qE09/l645JSbTkO+fLlSylgnseuEldA8PD48dAmNvwldhoxgbG7OPP/74Lbueh4eHx07ARz/60WestQ+/1nFeQvfw8PDYIfAvdA8PD48dAv9C9/Dw8Ngh8C90Dw8Pjx2CW0qKGmOuAqgCmH6tY29zDGF7j2G79x/Y/mPY7v0Htv8YtlP/91trh1/roFv6QgcAY8yxjbC1tzO2+xi2e/+B7T+G7d5/YPuPYbv3fy14k4uHh4fHDoF/oXt4eHjsEGzFC/2JLbjmzcZ2H8N27z+w/cew3fsPbP8xbPf+r8Itt6F7eHh4ePxw4E0uHh4eHjsEt/SFbox51Bhzwhhz2hjzkVt57RuBMWafMeYbxpiXjTEvGWN+jdsHjDFfNcac4r/9W93Xa4GLfD9njPkS//+gMea7fB/+whiTfa1zbCWMMX3GmM8aY14xxhw3xrx1G96Df8dr6AfGmD83xuRv5/tgjPmEMWbKGPMD1bbmnBvCf+dxvGiMecPW9Vywzhj+C6+jF40xf+mqsfG+3+QxnDDG/OOt6fXmcMte6Fzx6A8BvBvAvQB+0Rhz7626/g2iA+A3rLX3AngLgF/lPn8EwJPW2iMAnuT/3874NVDZQIffBfD71trDAOYAfGhLerVx/AGAv7HW3g3g9aCxbJt7YIzZA+DfAnjYWns/qJbPB3F734dPAnh0Rdt6c/5uAEf43+MAPnaL+vha+CRWj+GrAO631r4OwEkAvwkA/Fx/EMB9/Jv/Ybry6W4P3EoJ/REAp621Z6y1LQCfBvC+W3j964a1dsJa+yxvL4FeJHtA/f4UH/YpAL+wNT18bRhj9gL4WQB/xP83AN4J4LN8yO3e/14AbweXOLTWtqy189hG94ARASgYYyIARQATuI3vg7X2WwBmVzSvN+fvA/AnlvAUqID86K3p6fpYawzW2q9YSVL/FKQk8/sAfNpa27TWngVwGtuwItutfKHvAXBR/X+c27YFjDEHQKX4vgtgxFo7wbuuABhZ52e3A/4bgH8PwGX5HwQwrxb17X4fDgK4CuCP2Wz0R8aYErbRPbDWXgLwXwFcAL3IFwA8g+11H4D153y7Ptv/CsD/5e3tOoYueFJ0AzDGlAF8DsCvW2sX9T5LbkK3pauQMebnAExZa5/Z6r5sAhGANwD4mLX2IVDqiC7zyu18DwCAbc3vA32cxgCUsNoUsK1wu8/5a8EY81sgk+qfbXVfbiZu5Qv9EoB96v97ue22hjEmA3qZ/5m19vPcPOlUSv47td7vtxg/DuC9xphzIBPXO0H26D5W/YHb/z6MAxi31n6X//9Z0At+u9wDAPhpAGettVettW0Anwfdm+10H4D153xbPdvGmH8B4OcA/JIVv+1tNYb1cCtf6F5pGKAAAAF3SURBVE8DOMLMfhZEQHzxFl7/usH25o8DOG6t/T2164sAHuPtxwB84Vb3bSOw1v6mtXavtfYAaL6/bq39JQDfAPB+Puy27T8AWGuvALhojLmLm94F4GVsk3vAuADgLcaYIq8pN4Ztcx8Y6835FwH8Cnu7vAXAgjLN3FYwxjwKMkG+11pbU7u+COCDxpicMeYgiOD93lb0cVOw1t6yfwDeA2KWXwXwW7fy2jfY37eB1MoXATzP/94DskM/CeAUgK8BGNjqvm5gLO8A8CXePgRarKcB/G8Aua3u32v0/UEAx/g+/BWA/u12DwB8FMArAH4A4E8B5G7n+wDgz0H2/jZIS/rQenMOKqn8h/xcfx/kzXO7juE0yFbunuf/qY7/LR7DCQDv3ur+38g/Hynq4eHhsUPgSVEPDw+PHQL/Qvfw8PDYIfAvdA8PD48dAv9C9/Dw8Ngh8C90Dw8Pjx0C/0L38PDw2CHwL3QPDw+PHQL/Qvfw8PDYIfj/168Wu0MvjO8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# the right answers\n",
    "dataiter = iter(testloader)\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "# print images\n",
    "imshow(torchvision.utils.make_grid(images))\n",
    "print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now let us see what the neural network thinks these examples above are:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.6886, -1.6748,  0.1477,  3.6640,  0.4096,  1.8184,  2.2522, -1.5627,\n",
       "          0.7418, -1.9075],\n",
       "        [ 3.8578,  5.6504, -0.5568, -4.2354, -2.6664, -3.5884, -4.9890, -2.2154,\n",
       "          5.9530,  3.8955],\n",
       "        [ 1.9718,  1.9960,  0.1403, -0.9796, -1.9188, -0.8466, -3.2666, -0.3644,\n",
       "          2.5302,  1.8721],\n",
       "        [ 4.1118,  1.7155,  1.3498, -1.8422, -0.6677, -2.4545, -2.7999, -1.6346,\n",
       "          5.0910,  1.2039]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# what our network thinks\n",
    "outputs = net(Variable(images))\n",
    "outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The outputs are energies for the 10 classes.\n",
    "Higher the energy for a class, the more the network\n",
    "thinks that the image is of the particular class.\n",
    "So, let's get the index of the highest energy:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Predicted: ', '  cat  ship  ship  ship')\n"
     ]
    }
   ],
   "source": [
    "_, predicted = torch.max(outputs.data, 1)\n",
    "\n",
    "print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]\n",
    "                              for j in range(4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results seem pretty good.\n",
    "\n",
    "Let us look at how the network performs on the whole dataset.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the network on the 10000 test images: 60 %\n"
     ]
    }
   ],
   "source": [
    "correct = 0\n",
    "total = 0\n",
    "for data in testloader:\n",
    "    images, labels = data\n",
    "    outputs = net(Variable(images))\n",
    "    _, predicted = torch.max(outputs.data, 1)\n",
    "    total += labels.size(0)\n",
    "    correct += (predicted == labels).sum()\n",
    "\n",
    "print('Accuracy of the network on the 10000 test images: %d %%' % (\n",
    "    100 * correct / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks waaay better than chance, which is 10% accuracy (randomly picking\n",
    "a class out of 10 classes).\n",
    "Seems like the network learnt something.\n",
    "\n",
    "Hmmm, what are the classes that performed well, and the classes that did\n",
    "not perform well:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_correct = list(0. for i in range(10))\n",
    "class_total = list(0. for i in range(10))\n",
    "\n",
    "for data in testloader:\n",
    "    images, labels = data\n",
    "    #outputs = net(Variable(images).double())\n",
    "    outputs = net(Variable(images))\n",
    "    _, predicted = torch.max(outputs.data, 1)\n",
    "    c = (predicted == labels).squeeze().int()\n",
    "    for i in range(4):\n",
    "        label = labels[i]\n",
    "        class_correct[label] += c[i]\n",
    "        class_total[label] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of plane : 55 %\n",
      "Accuracy of   car : 80 %\n",
      "Accuracy of  bird : 40 %\n",
      "Accuracy of   cat : 57 %\n",
      "Accuracy of  deer : 63 %\n",
      "Accuracy of   dog : 36 %\n",
      "Accuracy of  frog : 68 %\n",
      "Accuracy of horse : 63 %\n",
      "Accuracy of  ship : 78 %\n",
      "Accuracy of truck : 66 %\n"
     ]
    }
   ],
   "source": [
    "class_total[0]\n",
    "\n",
    "for i in range(10):\n",
    "    \n",
    "    print('Accuracy of %5s : %2d %%' % (classes[i], 100 * class_correct[i] / class_total[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAG3CAYAAAC30lEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAADc9JREFUeJzt3OvP2PVdxvGrcLe0UCwtBYSKQNtlHFJ0QFExwiBR3GZwi4oJPhMPyyJLNMaozAyzaDQmM0aMM85DFlngyfCB2BgnJCQwZG6l0o4O6GgLrAJtB5T1MA63/0Jvk09qcr1ej3+5vr+237bv/h502eLiYgAAWp12ql8AAOBUEkMAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUW1jKw7/w6S2j/13129/4gcn5JMlZmzaN7q96ZdfofpKsXH3V6P69f3nvsqntv/vNT4/eoVWrZn99k+TY6oOj++d+44zR/STZdeHs/zx/95/9xtgdSpLPfeae0R/AJU8fn5xPkuzZ/djo/neuvXl0P0l+YvP60f0P3v3JsXv00c/eNnqHrv/VOyfnkyRHH3t8dP+LBw6M7ifJT15yxej+39zyeyd1h3wZAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqLSzl4aO3fnHqPZIkV9xy+eh+kmw/bfno/ta9z47uJ8nnj+0Y3b93cHvdjj2D68my9V8f3U+SNw69Obr/0vr5f6Ocs/vK8TMmXXjbTaP7f7XjT0b3k+SFp58a3b/77j8e3U+SM9+3YfyMKYv71o7uH3ho/+h+kqy46oLR/dP2Pz+6nySH9qyaPeCWk3vMlyEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqLSzl4ce3Pz/1HkmSb33/4dH9JFn56qHR/b/YePHofpK8/diR8TOmLF7zQ6P7N95x8+h+kpx77IzR/e/mwOh+kjy6e//4GZOW735tdH/7kwdH95Nk/YVbRvfvuOny0f0kefClncMnbBxbvv3nPzy2nSSf/MTdo/tJ8qO/NnuH/vSuXxndT5IXn5z9O/lk+TIEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFQTQwBANTEEAFRbWMrDm7/3ran3SJI8/+Llo/tJsuad80f3z3vve6P7SfLiBQfGz5jy4IHDo/v/9sAzo/tJcv55r4/uv5sfHN1Pku++u3Z0/0Oj68mXd7w1un/V1atG95PkIxuvH91/ednpo/tJsmv7v47uf+y6m8e21794fGw7ST718Q+P7ifJWR84Z3T/hiNvj+4nyT88uWP2gJP8bebLEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANUWlvLwW/+xbOo9kiRrNxwd3U+SM65/dHT/f750/uh+kpz+/ivHz5jyzLU3j+6/tnff6H6S7N/x6uj+lmPPje4nyeatW8fPmHRk8yuj+5uuuGt0P0nOWDw4un/flz83up8krx/aNH7GlG8+8PnR/V/63U+M7ifJl3b+y+j+bT/3mdH9JLnqZ39k/IyT4csQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBNDAEA1cQQAFBt2eLi4ql+BwCAU8aXIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKqJIQCgmhgCAKotLOXhe/8xi1MvkiS79u2cnE+SHHzwudH9F9ZuG91Pkg+cOfvz9LcPPb5savvP//m3R+/Q/v96a3I+SXLuoUtH9488/+3R/ST59ZuuHN3f+KmPj92hJPnrJ/aO3qOdB+6fnE+SfOVrK0f3rzj7gtH9JLl0/TWj+3905/vH7tEdv/j3o3do72t7JueTJOvPeWd0/2c2Xj26nyQvrzk8uv+Hf3DXSd0hX4YAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACoJoYAgGpiCACotrCUh//9P/9p6j2SJGvOWjW6nyQHtr4xur/miXNH95Pk8Mol/bL9v/LSV1eP7r+5dv7n/9Zf/sjo/v23/87ofpJ84fETo/v3jK4nj3z14dH9h/c9O7qfJIceeGR0f/uG9aP7SfKhGz82fMLvjy1f9M6lY9tJ8vIb838WPbL6qdH9FVveN7qfJHe+cnD8jJPhyxAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVxBAAUE0MAQDVFpby8NHdT0y9R5Lk8I9tHd1PktO/b+3o/rrLjozuJ8nZeW/8jCl73jw4uv/Tt2wZ3U+SD27cPLp/38p1o/tJsm3b06P794yuJxvO/c7o/mc33jC6nyRX/9ato/sHX579sy5Jnt42fsSY/Qc3jO7ve+++0f0kuX7b8tH9iz56/eh+kpx/3rfHzzgZvgwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANXEEABQTQwBANUWlvLwDSd2Tr1HkmTvq0t6nf+T9e+uGt0/Z/WJ0f0kOX5sw/gZU67bdN7o/hnHjozuJ8met94a3T9x8ZrR/SRZffXK8TMmXfLU4dH9H7/uitH9JNm0cMHo/nsnXh3dT5K3V7wwfMJPjS3feOmZY9tJsusLl43uJ8nyXDS6v2rb8dH9JDl65YrxM06GL0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUE0MAQDUxBABUW1jKw8e/8rWp90iSXHvevtH9JFmz4abR/eObnx3dT5LVe18bP2PK8muWdOWW7JmHnx/dT5Lndj80un/mmgtG95Pkh2+/dfyMSZedffHo/onnjo7uJ8mj37x/dP+/V5w/up8kW04sHz9jyrrjr4/un7n2ktH9JLlsxZ7R/QtXfn10P0kOrV43fML6k3rKlyEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoJoYAgCqiSEAoNqyxcXFU/0OAACnjC9DAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVBNDAEA1MQQAVPtfjKtL3YYXhNAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x720 with 8 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mm = net.double()\n",
    "filters = mm.modules\n",
    "layer1 = [i for i in mm.children()][0]\n",
    "tensor = layer1.weight.data.numpy()\n",
    "\n",
    "fig = plt.figure(figsize=(10,10))\n",
    "for i in range(8):\n",
    "    ax1 = fig.add_subplot(2,4,i+1)\n",
    "    tensor[i] = tensor[i] - min(tensor[i].flatten())\n",
    "    tensor[i] = tensor[i] / max(tensor[i].flatten())\n",
    "    ax1.imshow(np.transpose(tensor[i], (1, 2, 0)))\n",
    "    ax1.axis('off')\n",
    "    ax1.set_xticklabels([])\n",
    "    ax1.set_yticklabels([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so what next?\n",
    "\n",
    "How do we run these neural networks on the GPU?\n",
    "\n",
    "Training on GPU\n",
    "----------------\n",
    "Just like how you transfer a Tensor on to the GPU, you transfer the neural\n",
    "net onto the GPU.\n",
    "This will recursively go over all modules and convert their parameters and\n",
    "buffers to CUDA tensors:\n",
    "\n",
    "\n",
    "        net.cuda()\n",
    "\n",
    "\n",
    "Remember that you will have to send the inputs and targets at every step\n",
    "to the GPU too:\n",
    "\n",
    "\n",
    "        inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())\n",
    "\n",
    "Why dont I notice MASSIVE speedup compared to CPU? Because your network\n",
    "is realllly small.\n",
    "\n",
    "\n",
    "**Goals achieved**:\n",
    "\n",
    "- Understanding PyTorch's Tensor library and neural networks at a high level.\n",
    "- Train a small neural network to classify images\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def im2col(x,hh,ww,stride):\n",
    "\n",
    "    \"\"\"\n",
    "    Args:\n",
    "      x: image matrix to be translated into columns, (C,H,W)\n",
    "      hh: filter height\n",
    "      ww: filter width\n",
    "      stride: stride\n",
    "    Returns:\n",
    "      col: (new_h*new_w,hh*ww*C) matrix, each column is a cube that will convolve with a filter\n",
    "            new_h = (H-hh) // stride + 1, new_w = (W-ww) // stride + 1\n",
    "    \"\"\"\n",
    "\n",
    "    c,h,w = x.shape\n",
    "    new_h = (h-hh) // stride + 1\n",
    "    new_w = (w-ww) // stride + 1\n",
    "    col = np.zeros([new_h*new_w,c*hh*ww])\n",
    "\n",
    "    for i in range(new_h):\n",
    "       for j in range(new_w):\n",
    "           patch = x[...,i*stride:i*stride+hh,j*stride:j*stride+ww]\n",
    "           col[i*new_w+j,:] = np.reshape(patch,-1)\n",
    "    return col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 0, 0],\n",
       "       [0, 1, 0, 0],\n",
       "       [1, 1, 0, 0]])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "x = np.random.randint(2, size=12).reshape(1,3,4)\n",
    "xcol = im2col(x,2,2,1)\n",
    "x = np.squeeze(x)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "#im2col(x,3,3,1)\n",
    "# filter size = 3 x 3, stride = 1\n",
    "ker = np.array([\n",
    "                 [1., -1.],\n",
    "                 [1., -1.],\n",
    "                ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2., -2.,  0.],\n",
       "       [ 1., -2.,  0.]])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.signal import convolve2d\n",
    "\n",
    "### WRITE CODE IN HERE , you can have up to 2 cells for this question but only one is required ######\n",
    "\n",
    "convolve2d(x, ker, 'valid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 1., 0., 1.],\n",
       "       [1., 0., 1., 0.],\n",
       "       [0., 0., 0., 0.],\n",
       "       [0., 1., 1., 1.],\n",
       "       [1., 0., 1., 0.],\n",
       "       [0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xcol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1., -1.,  1., -1.])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ker_flat = np.flip(ker.flatten(),axis=0)\n",
    "np.flip(ker_flat,axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2., -2.,  0.,  1., -2.,  0.])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = xcol.dot(ker_flat)\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def col2im(mul,h_prime,w_prime,C):\n",
    "    \"\"\"\n",
    "      Args:\n",
    "      mul: (h_prime*w_prime*w,F) matrix, each col should be reshaped to C*h_prime*w_prime when C>0, or h_prime*w_prime when C = 0\n",
    "      h_prime: reshaped filter height\n",
    "      w_prime: reshaped filter width\n",
    "      C: reshaped filter channel, if 0, reshape the filter to 2D, Otherwise reshape it to 3D\n",
    "    Returns:\n",
    "      if C == 0: (F,h_prime,w_prime) matrix\n",
    "      Otherwise: (F,C,h_prime,w_prime) matrix\n",
    "    \"\"\"\n",
    "    F = mul.shape[1]\n",
    "    if(C == 1):\n",
    "        out = np.zeros([F,h_prime,w_prime])\n",
    "        for i in range(F):\n",
    "            col = mul[:,i]\n",
    "            out[i,:,:] = np.reshape(col,(h_prime,w_prime))\n",
    "    else:\n",
    "        out = np.zeros([F,C,h_prime,w_prime])\n",
    "        for i in range(F):\n",
    "            col = mul[:,i]\n",
    "            out[i,:,:] = np.reshape(col,(C,h_prime,w_prime))\n",
    "\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "tuple index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-50-5853175c5793>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcol2im\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-47-e099fd9d0867>\u001b[0m in \u001b[0;36mcol2im\u001b[0;34m(mul, h_prime, w_prime, C)\u001b[0m\n\u001b[1;32m     10\u001b[0m       \u001b[0mOtherwise\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mF\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mC\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mh_prime\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mw_prime\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0mmatrix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m     \"\"\"\n\u001b[0;32m---> 12\u001b[0;31m     \u001b[0mF\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmul\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     13\u001b[0m     \u001b[0;32mif\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mC\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m         \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mF\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mh_prime\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mw_prime\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: tuple index out of range"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}