exercise_430.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss Function\n",
    "\n",
    "We can create a probability distrbution over the outputs by using a softmax function which is defined as:\n",
    "\n",
    "$$\\frac{e^{x}}{\\sum_{x'}e^{x'}}$$\n",
    "\n",
    "We can then use that probability distribution and knowledge of the correct categories to evaluate the classifier.\n",
    "\n",
    "A classifier with high confidence in an incorrect category is a poor classifier.  A classifier with low confidence in an incorrect category and high confidence in the correct category is a good classifier.\n",
    "\n",
    "We show below how the softmax function is applied to the output values to turn them into probabilities, and how a measure of the incorrectness, the cross-entropy loss, is computed from them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c1 and c2: tensor([[ 6.0000,  3.9000]])\n",
      "e to the power of c: tensor([[ 403.4288,   49.4025]])\n",
      "e to the power of c normalized: tensor([[ 0.8909,  0.1091]])\n",
      "softmax: tensor([[ 0.8909,  0.1091]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "data = torch.Tensor([[6, 2, 1.9]])\n",
    "\n",
    "weights = torch.Tensor([[1, 0],[0, 1],[0, 1]])\n",
    "\n",
    "c = torch.mm(data, weights)\n",
    "print(\"c1 and c2: \" + str(c))\n",
    "\n",
    "exp = torch.exp(c)\n",
    "print(\"e to the power of c: \"+str(exp))\n",
    "\n",
    "soft = exp / torch.sum(exp)\n",
    "print(\"e to the power of c normalized: \"+str(soft))\n",
    "\n",
    "result = F.softmax(torch.autograd.Variable(c), dim=1)\n",
    "print(\"softmax: \" + str(result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convince yourself that the output of the 'softmax' function is the same as 'e to the power of c normalized'. nll_loss() is a function that takes in the output of the model (log of softmax), and looks for the target class, or the correct answer (identified by its position) and returns the negative of the calcualtion. For example, if the second class was the right answer, target[0] would be set to 1, and this would return the second item in the result array (after negating it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "log(softmax): tensor([[nan, nan]])\n",
      "Loss: nan\n"
     ]
    }
   ],
   "source": [
    "result = torch.log(result)\n",
    "print(\"log(softmax): \" + str(result))\n",
    "\n",
    "# The correct category\n",
    "target = torch.LongTensor(1)\n",
    "target[0] = 0 \n",
    "\n",
    "loss = F.nll_loss(result, target)\n",
    "print(\"Loss: \"+str(loss.data.item()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pytorch also has a ready-made function to compute cross entropy directly from the neural network's outputs.\n",
    "Convince yourself that the loss calculated by either method is the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 0.11551953107118607\n"
     ]
    }
   ],
   "source": [
    "loss = F.cross_entropy(c, target)\n",
    "print(\"Loss: \"+str(loss.data.item()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try this for different values of input data.\n",
    "\n",
    "You could try them in the following order - in order of increasing difficulty\n",
    "\n",
    "$\\begin{bmatrix}6 & 2 & 1.9\\end{bmatrix}$\n",
    "$\\begin{bmatrix}5 & 2 & 1.9\\end{bmatrix}$\n",
    "$\\begin{bmatrix}4 & 2 & 1.9\\end{bmatrix}$\n",
    "\n",
    "You should see the loss increase as the input data points become more difficult to decide about (as $f_{1}$ and $f_{2} + f_{3}$ get increasingly closer.  Remember, we're still using the weight matrix for Toy Problem 2 and the decision that classifier has to take is whether $f_{1} < f_{2} + f_{3}$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}