gradient_check.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Checking\n",
    "\n",
    "To determine whether the gradient calculation is performed correctly, I've implemented a simple gradient checker using information from this [online article](https://towardsdatascience.com/how-to-debug-a-neural-network-with-gradient-checking-41deec0357a9).<br>\n",
    "\n",
    "We start by approximating the gradient of the model using the limit definition of a derivative:\n",
    "$$\\frac{d}{d \\theta}J(\\theta) = \\lim\\limits_{\\epsilon \\rightarrow 0} \\frac{J(\\theta + \\epsilon) - J(\\theta - \\epsilon)}{2\\epsilon}$$\n",
    "Where $J$ represents our cost function, and $\\theta$ represents our model's parameters. We can use an arbitrarily small value for $\\epsilon$ (1e-4 by default) to approximate the gradient.<br>\n",
    "Then, we calculate the normalized distance between the actual and approximated gradient vectors, as follows:\n",
    "$$\\text{distance} = \\frac{\\| d\\theta_{approx} - d\\theta \\|_2}{\\| d\\theta_{approx} \\|_2 + \\| d\\theta \\|_2}$$\n",
    "If the distance is sufficiently small (less than  $\\epsilon^2$), the gradient is *probably* correct.<br>\n",
    "The source code for the `gradient_check()` function can be found in `/src/sandbox/utils.py`.<br>\n",
    "\n",
    "Last update: 12/30/23"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from importlib import reload\n",
    "import numpy as np\n",
    "from sandbox import activations, costs, initializers, layers, model, optimizers, utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking activation functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.045160875240258e-09\n"
     ]
    }
   ],
   "source": [
    "reload(activations)\n",
    "\n",
    "# Create dummy data\n",
    "X = np.random.randn(100, 3)\n",
    "Y = np.random.randint(0, 2, (100, 1))\n",
    "\n",
    "# Create model\n",
    "activation = model.Model()\n",
    "activation.add(layers.Dense(units=10, activation=activations.ReLU()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.Linear()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.Arctan()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.BentIdentity()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.LeakyReLU(alpha=0.05)))\n",
    "activation.add(layers.Dense(units=10, activation=activations.Tanh()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.ELU()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.SELU()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.SLU()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.Softplus()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.Softsign()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.Gaussian()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.PiecewiseLinear()))\n",
    "activation.add(layers.Dense(units=10, activation=activations.Softmax()))\n",
    "activation.add(layers.Dense(units=1, activation=activations.Sigmoid()))\n",
    "\n",
    "activation.configure(\n",
    "    input_size=3,\n",
    "    cost_type=costs.BinaryCrossentropy(),\n",
    "    optimizer=optimizers.SGD(),\n",
    ")\n",
    "\n",
    "# Check gradient\n",
    "diff = utils.gradient_check(activation, X, Y)\n",
    "print(diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking cost functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.541350906223112e-10\n",
      "1.2971624135369784e-10\n",
      "1.0305088231610491e-09\n",
      "1.6677526619094892e-10\n"
     ]
    }
   ],
   "source": [
    "reload(costs)\n",
    "\n",
    "# Create dummy data\n",
    "X = np.random.randn(100, 3)\n",
    "Y = np.random.randint(0, 2, (100, 1))\n",
    "\n",
    "# Create model\n",
    "cost = model.Model()\n",
    "cost.add(layers.Dense(units=4, activation=activations.ReLU()))\n",
    "cost.add(layers.Dense(units=2, activation=activations.ReLU()))\n",
    "cost.add(layers.Dense(units=1, activation=activations.Sigmoid()))\n",
    "\n",
    "# Check gradients for each cost type\n",
    "\n",
    "# Binary Cross-Entropy\n",
    "cost.configure(\n",
    "    input_size=3,\n",
    "    cost_type=costs.BinaryCrossentropy(),\n",
    "    optimizer=optimizers.SGD()\n",
    ")\n",
    "diff = utils.gradient_check(cost, X, Y)\n",
    "print(diff)\n",
    "\n",
    "# Categorical Cross-Entropy\n",
    "cost.configure(\n",
    "    input_size=3,\n",
    "    cost_type=costs.CategoricalCrossentropy(),\n",
    "    optimizer=optimizers.SGD()\n",
    ")\n",
    "diff = utils.gradient_check(cost, X, Y)\n",
    "print(diff)\n",
    "\n",
    "# Mean Squared Error\n",
    "cost.configure(\n",
    "    input_size=3,\n",
    "    cost_type=costs.MSE(),\n",
    "    optimizer=optimizers.SGD()\n",
    ")\n",
    "diff = utils.gradient_check(cost, X, Y)\n",
    "print(diff)\n",
    "\n",
    "# Mean Absolute Error\n",
    "cost.configure(\n",
    "    input_size=3,\n",
    "    cost_type=costs.MAE(),\n",
    "    optimizer=optimizers.SGD()\n",
    ")\n",
    "diff = utils.gradient_check(cost, X, Y)\n",
    "print(diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking layer types - excluding dropout, which does not work with gradient checking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.81837716670426e-10\n"
     ]
    }
   ],
   "source": [
    "reload(layers)\n",
    "\n",
    "# Create dummy data\n",
    "X = np.random.randn(100, 3)\n",
    "Y = np.random.randint(0, 2, (100, 1))\n",
    "\n",
    "# Create model\n",
    "layer = model.Model()\n",
    "layer.add(layers.Dense(units=4, activation=activations.ReLU()))\n",
    "layer.add(layers.Dense(units=2, activation=activations.ReLU()))\n",
    "layer.add(layers.Dense(units=1, activation=activations.Sigmoid()))\n",
    "\n",
    "layer.configure(\n",
    "    input_size=3,\n",
    "    cost_type=costs.BinaryCrossentropy(),\n",
    "    optimizer=optimizers.SGD()\n",
    ")\n",
    "\n",
    "# Check gradient\n",
    "diff = utils.gradient_check(layer, X, Y)\n",
    "print(diff)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}