Copy_of_Assignment_1.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ssLI7a7GKzgB"
      },
      "source": [
        "# CS4240 Deep Learning - Assignment 1\n",
        "\n",
        "*These lab assignments are new in the CS4240 Deep Learning course. We'd like to hear what you think!*\n",
        "\n",
        "*Please post any feedback you have on Brightspace. Thanks!*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7rSnXZSgqpfk"
      },
      "source": [
        "To start working on the assignment in Colab, save a copy on your Google Drive (`File` $\\rightarrow$ `Save a copy in Drive`).\n",
        "\n",
        "To work on the assignments locally, configure your conda environment (see instructions on Brightspace) and download this assignment as an IPython Notebook (`File` $\\rightarrow$ `Download .ipynb`)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdx7o5N2S2Tn"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "In this assignment you will be introduced to some basic functionality in PyTorch, you will write your own fully connected neural network layers and non-linear activation functions and will use those to solve a simple XOR classification problem.\n",
        "\n",
        "**Prerequisites:**\n",
        "* Basic knowledge of Python and Numpy. Recommended tutorial for Python and Numpy [here](https://cs231n.github.io/python-numpy-tutorial/).\n",
        "\n",
        "**Learning objectives:**\n",
        "* Using basic tensor operations in PyTorch;\n",
        "* Understanding and implementing the forward pass of a fully connected layer and the ReLU and Sigmoid non-linear activation functions in PyTorch;\n",
        "* Using basic PyTorch layers to build a simple neural network;\n",
        "* Understanding and calculating the MSE loss;\n",
        "* Understanding the training loop.\n",
        "\n",
        "We will share the solutions one week after the assignments are published. Throughout the assignment you will validate your code by comparing the outputs of your own implementations to the equivalent PyTorch implementations.\n",
        "\n",
        "**For your own implementations you may only use basic tensor operations from the `torch` module: no `torch.nn`, `torch.nn.F` or others. There is a small introduction into pytorch tensors in A1.0**\n",
        "\n",
        "\n",
        "When answering coding questions make sure to write your own code within the designated part of the code block as illustrated here:\n",
        "```python \n",
        "#############################################################################\n",
        "#                       TODO: Implement function x                          #\n",
        "#############################################################################\n",
        "# Replace \"pass\" statement with your code\n",
        "pass\n",
        "#############################################################################\n",
        "#                            END OF YOUR CODE                               #\n",
        "#############################################################################\n",
        "```\n",
        "\n",
        "Please pay attention to the question boxes and try to think about them. The boxes are indicated as follows:\n",
        "\n",
        "****\n",
        "**Questions?**\n",
        "****"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "gIvAOzFjhtw6"
      },
      "outputs": [],
      "source": [
        "# Setup\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OeLiiyq-Hnw1"
      },
      "source": [
        "##A1.0 Tensors in Pytorch\n",
        "\n",
        "Throughout the course, you will use the Pytorch library to design and evaluate your models. In those implementations you will (and must) often use Pytorch tensors (ranging from 1D to high-D). You have probably worked with `numpy` arrays quite much. Fortunately, creating and manipulating Pytorch tensors is highly similar. A detailed description of *how* to deal with them is given [here](https://colab.research.google.com/drive/1L9aMXms7ZXqjOktbwtH8H1VY8ixah6yU) (which is in *view* mode; for editing you must copy paste it).\n",
        "\n",
        "It is encouraged to have a look as it will make solving the lab assignments easier. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IuIHuF2dMzSV"
      },
      "source": [
        "## A1.1 Linear (fully connected) layer\n",
        "\n",
        "In this first exercise, you will 1. implement the linear layer, 2. perform a forward pass on some dummy input, and 3. compare the output tensor to the *Pytorch* implementation of a linear layer.\n",
        "\n",
        "A linear layer is defined as a fully-connected map from some input neurons to some output neurons. Say you have $N_{\\text{in}}$ and $N_{\\text{out}}$ neurons, there are a total of $N_{\\text{in}} \\cdot N_{\\text{out}}$ connections (weigths) and $N_{\\text{out}}$ biases. The fully-connected layer for some input vector $\\mathbf{x} \\in \\mathbb{R}^{N_{\\text{in}}}$, ouput vector $\\mathbf{y}  \\in \\mathbb{R}^{N_{\\text{out}}}$, weighting matrix $\\mathbf{W}  \\in \\mathbb{R}^{N_{\\text{in}} \\times N_{\\text{out}}} $, and bias vector $\\mathbf{b}  \\in \\mathbb{R}^{N_{\\text{out}}}$ is given by:\n",
        "\n",
        "$$ \\mathbf{y} = \\mathbf{x}^T \\mathbf{W} + \\mathbf{b}$$\n",
        "\n",
        "****\n",
        "**What does $\\mathbf{W}_{ij}$ represent exactly? What is the size of $\\mathbf{W}$? And why is $\\mathbf{x}$ tranposed?** \n",
        "****\n",
        "\n",
        "We will implement the linear layer as a Python *object*. Keep in mind that we want to be able to process multiple samples in one forward pass, so $\\mathbf{x}$ will include a batch dimension such that $\\mathbf{x} \\in \\mathbb{R}^{\\text{batch},N_{\\text{in}}}$ and $\\mathbf{x}\\mathbf{W}$ will become a matrix multiplication.\n",
        "\n",
        "****\n",
        "**Why would you want to do that? Hint: think bigger, literally ;)!** \n",
        "****"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([1.3483e-05, 1.3086e-11, 1.0860e-05, 1.0722e-08, 2.1349e+20])\n"
          ]
        }
      ],
      "source": [
        "t = torch.Tensor(5)\n",
        "print(t)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "oWcqUhVgfR5A"
      },
      "outputs": [],
      "source": [
        "class Linear(object):\n",
        "    \"\"\"\n",
        "    Fully connected layer.\n",
        "    \n",
        "    Args:\n",
        "        in_features: number of input features\n",
        "        out_features: number of output features\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, in_features, out_features):\n",
        "        super(Linear, self).__init__()\n",
        "        \n",
        "        ########################################################################\n",
        "        #      TODO: Define placeholder tensors for layer weight and bias.     #\n",
        "        #       The placeholder tensors should have the correct dimension      #\n",
        "        #        according to the in_features and out_features variables.      #\n",
        "        #                    Note: no for loops are needed!                    #\n",
        "        ########################################################################\n",
        "        # placeholder tensors\n",
        "        self.weight = torch.Tensor(in_features, out_features)\n",
        "        self.bias = torch.Tensor(out_features)\n",
        "        \n",
        "        ########################################################################\n",
        "        #                         END OF YOUR CODE                             #\n",
        "        ########################################################################\n",
        "\n",
        "        # Initialize parameters\n",
        "        self.init_params()\n",
        "\n",
        "    def init_params(self, std=0.1):\n",
        "        \"\"\"\n",
        "        Initialize layer parameters. Sample weight from Gaussian distribution\n",
        "        and bias uniform distribution.\n",
        "        \n",
        "        Args:\n",
        "            std: Standard deviation of Gaussian distribution (default: 0.1)\n",
        "        \"\"\"\n",
        "        self.weight = std*torch.randn_like(self.weight)\n",
        "        self.bias = torch.rand_like(self.bias)\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Forward pass of Linear layer: multiply input tensor by weights and add\n",
        "        bias.\n",
        "        \n",
        "        Args:\n",
        "            x: input tensor\n",
        "\n",
        "        Returns:\n",
        "            y: output tensor\n",
        "        \"\"\"\n",
        "        ########################################################################\n",
        "        #                  TODO: Implement this function                       #\n",
        "        ########################################################################\n",
        "\n",
        "        y = torch.matmul(x, self.weight) + self.bias\n",
        "\n",
        "        ########################################################################\n",
        "        #                         END OF YOUR CODE                             #\n",
        "        ########################################################################\n",
        "\n",
        "        return y\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TtE7ytKSFBst"
      },
      "source": [
        "Now test the forward pass of the layer on some dummy input $\\mathbf{x}$.\n",
        "\n",
        "****\n",
        "**What will be the shape of output tensor $\\mathbf{y}$?** \n",
        "****"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "bSALY1wPFGXZ"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of ouput tensor y: torch.Size([2, 4])\n",
            "Shape of output tensor y correct:  True\n"
          ]
        }
      ],
      "source": [
        "# Define layer dimensions and dummy input\n",
        "n_samples, in_features, out_features = 2, 3, 4\n",
        "# Make random input tensor of dimensions [n_samples, in_features]\n",
        "x = torch.randn((n_samples, in_features))\n",
        "\n",
        "########################################################################\n",
        "#          TODO: Create a layer from the Linear object class           #\n",
        "#                     above and do a forward pass                      #\n",
        "########################################################################\n",
        "\n",
        "layer = Linear(in_features, out_features)   \n",
        "y = layer.forward(x)\n",
        " \n",
        "########################################################################\n",
        "#                         END OF YOUR CODE                             #\n",
        "########################################################################\n",
        "\n",
        "# What will be the shape of output tensor y?\n",
        "print('Shape of ouput tensor y:', y.shape)\n",
        "print('Shape of output tensor y correct: ', y.shape == torch.Size([n_samples, out_features]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8L5vbHyFIE9m"
      },
      "source": [
        "If the forward pass has not returned any errors we can compare our implementation to the PyTorch linear layer [[docs](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear)] from `torch.nn`. We do so by initializing a `nn.Linear` layer and setting the `weight` and `bias` to the same values as in our own linear layer.\n",
        "\n",
        "PyTorch layers store their parameters as a `Parameter`, which is a `Tensor` subclass with some special properties [[docs](https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html)]. We therefore need to wrap our `layer.weight` and `layer.bias` in a `nn.Parameter` when using in `nn.Linear`. Moreover, the `weight` tensor is transposed in `nn.Linear`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "xa-_W-MgFJzN"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of ouput tensor torch_y: torch.Size([2, 4])\n",
            "Shape of output tensor y correct:  True\n"
          ]
        }
      ],
      "source": [
        "# Create Linear layer from torch.nn module\n",
        "torch_layer = nn.Linear(in_features, out_features)\n",
        "\n",
        "# Load the parameters from our layer into the Pytorch layer\n",
        "torch_layer.weight = nn.Parameter(layer.weight.T) # transpose weight by .T\n",
        "torch_layer.bias = nn.Parameter(layer.bias)\n",
        "\n",
        "# Perform forward pass\n",
        "torch_y = torch_layer(x)\n",
        "\n",
        "# What will be the shape of output tensor torch_y?\n",
        "print('Shape of ouput tensor torch_y:', torch_y.shape)\n",
        "print('Shape of output tensor y correct: ', y.shape == torch.Size([n_samples, out_features]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjE7DIp9Ltut"
      },
      "source": [
        "We now compare the outputs of both implementations using `torch.allclose` [[docs](https://pytorch.org/docs/stable/generated/torch.allclose.html)], which returns `True` if all elements in both tensors are sufficiently \"close\" to each other (see documentation for what that means exactly).\n",
        "\n",
        "Your forward implementation of the linear layer is *correct* if `True` is returned."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "3DMC0nrYNkIf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Outputs identical:  True\n"
          ]
        }
      ],
      "source": [
        "# Compare outputs using torch.allclose\n",
        "outputs_same = torch.allclose(y, torch_y)\n",
        "print('Outputs identical: ', outputs_same)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N41y5ntsMw40"
      },
      "source": [
        "## A1.2 Non-linear activation functions\n",
        "\n",
        "As the XOR problem cannot be solved by a simple linear classifier we require our neural network to be able to learn non-linear functions. This is where non-linear activation functions come into play. In this assignment we will implement the forward passes of two popular activation functions, namely the Rectified Linear Unit\n",
        "\n",
        "$$\n",
        "\\text{ReLU}(x)=\\max(0,x)\n",
        "$$\n",
        "\n",
        "and the Sigmoid function\n",
        "\n",
        "$$\n",
        "\\sigma(x) = \\frac{1}{1+\\exp(-x)}\n",
        "$$ \n",
        "\n",
        "To verify your implementation you will again compare against their PyTorch counterparts.\n",
        "\n",
        "****\n",
        "**Can you still remember from the lectures what the (dis)advantages are for each activation function? Think in terms of efficiency and effectiveness.** \n",
        "****"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "rOphZSwhMqn6"
      },
      "outputs": [],
      "source": [
        "class ReLU(object):\n",
        "    \"\"\"\n",
        "    ReLU non-linear activation function.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        super(ReLU, self).__init__()\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Forward pass of ReLU non-linear activation function: y=max(0,x).\n",
        "        \n",
        "        Args:\n",
        "            x: input tensor\n",
        "\n",
        "        Returns:\n",
        "            y: output tensor\n",
        "        \"\"\"\n",
        "\n",
        "        ########################################################################\n",
        "        #                  TODO: Implement this function                       #\n",
        "        ########################################################################\n",
        "\n",
        "        y = torch.max(torch.zeros_like(x), x)\n",
        "\n",
        "        ########################################################################\n",
        "        #                         END OF YOUR CODE                             #\n",
        "        ########################################################################\n",
        "\n",
        "        return y\n",
        "\n",
        "class Sigmoid(object):\n",
        "    \"\"\"\n",
        "    Sigmoid non-linear activation function.\n",
        "    \"\"\"\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Forward pass of Sigmoid non-linear activation function: y=1/(1+exp(-x)).\n",
        "        \n",
        "        Args:\n",
        "            x: input tensor\n",
        "\n",
        "        Returns:\n",
        "            y: output tensor\n",
        "        \"\"\"\n",
        "        \n",
        "        ########################################################################\n",
        "        #                  TODO: Implement this function                       #\n",
        "        ########################################################################\n",
        "\n",
        "        y = 1 / (1 + torch.exp(-x))\n",
        "\n",
        "        ########################################################################\n",
        "        #                         END OF YOUR CODE                             #\n",
        "        ########################################################################\n",
        "\n",
        "        return y"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FtOu5BCKQvky"
      },
      "source": [
        "Test the forward pass of both non-linearities using the dummy input from before.\n",
        "\n",
        "****\n",
        "**Again: what will be shapes of the output tensors?** \n",
        "****"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "i-sCWIhNOetw"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of ouput tensors y_relu and y_sigmoid: torch.Size([2, 3]) torch.Size([2, 3])\n"
          ]
        }
      ],
      "source": [
        "########################################################################\n",
        "#    TODO: Perform a forward pass for the ReLU and Sigmoid layers.     #\n",
        "########################################################################\n",
        "\n",
        "y_relu = ReLU().forward(x)\n",
        "y_sigmoid = Sigmoid().forward(x)\n",
        "\n",
        "########################################################################\n",
        "#                         END OF YOUR CODE                             #\n",
        "########################################################################\n",
        "\n",
        "# What will be the shapes of output tensors y_relu and y_sigmoid?\n",
        "print('Shape of ouput tensors y_relu and y_sigmoid:', y_relu.shape, y_sigmoid.shape)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7hiL-1zwRh_G"
      },
      "source": [
        "Now perform a forward pass with the ReLU and Sigmoid activation functions from `torch.nn` and compare the outputs to your implementation.\n",
        "\n",
        "A list of all available non-linearities in PyTorch can be found [[here](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity)]."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "rCpv6e_bR6Dg"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "ReLU outputs identical:  True\n",
            "Sigmoid outputs identical:  True\n"
          ]
        }
      ],
      "source": [
        "# ReLU forward pass\n",
        "torch_relu = nn.ReLU()\n",
        "torch_y_relu = torch_relu(x)\n",
        "\n",
        "# Sigmoid forward pass\n",
        "torch_sigmoid = nn.Sigmoid()\n",
        "torch_y_sigmoid = torch_sigmoid(x)\n",
        "\n",
        "# Compare outputs using torch.allclose\n",
        "outputs_same = torch.allclose(y_relu, torch_y_relu)\n",
        "print('ReLU outputs identical: ', outputs_same)\n",
        "outputs_same = torch.allclose(y_sigmoid, torch_y_sigmoid)\n",
        "print('Sigmoid outputs identical: ', outputs_same)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X7bn6PyoNF8r"
      },
      "source": [
        "## A1.3 Network class\n",
        "\n",
        "We will now create a base class for our neural network that will make it possible to stack individual layers on each other. For now we will only implement the forward pass."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "vi8luwCMgobr"
      },
      "outputs": [],
      "source": [
        "class Net(object):\n",
        "    \"\"\"\n",
        "    Neural network object containing layers.\n",
        "    \n",
        "    Args:\n",
        "        layers: list of layers in neural network\n",
        "    \"\"\"\n",
        "    \n",
        "    def __init__(self, layers):\n",
        "        self.layers = layers\n",
        "\n",
        "    def reset_params(self, std=0.1):\n",
        "        \"\"\"\n",
        "        Reset network parameters. Applies `init_params` to all layers with\n",
        "        learnable parameters.\n",
        "        \n",
        "        Args:\n",
        "            std: Standard deviation of Gaussian distribution (default: 0.1)\n",
        "        \"\"\"\n",
        "\n",
        "        for layer in self.layers:\n",
        "            if hasattr(layer, 'init_params'):\n",
        "                layer.init_params(std=std)\n",
        "    \n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Performs forward pass through all layers of the network.\n",
        "        \n",
        "        Args:\n",
        "            x: input tensor\n",
        "\n",
        "        Returns:\n",
        "            x: output tensor\n",
        "        \"\"\"\n",
        "\n",
        "        ########################################################################\n",
        "        #                  TODO: Implement this function                       #\n",
        "        ########################################################################\n",
        "\n",
        "        for layer in self.layers:\n",
        "            x = layer.forward(x)\n",
        "        \n",
        "\n",
        "        ########################################################################\n",
        "        #                         END OF YOUR CODE                             #\n",
        "        ########################################################################\n",
        "\n",
        "        return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7xLhaGmqNdAw"
      },
      "source": [
        "We will create a simple 2-layer network with ReLU non-linearity and test the forward pass using the same dummy input as before.\n",
        "\n",
        "****\n",
        "**What will be the shape of the output tensor?** \n",
        "****"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "3 4\n"
          ]
        }
      ],
      "source": [
        "print(in_features, out_features)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "5k4uPheAjLsY"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of ouput tensor y: torch.Size([2, 4])\n"
          ]
        }
      ],
      "source": [
        "hidden_dim = 5\n",
        "\n",
        "# Define and initialize layers\n",
        "layers = [Linear(in_features, hidden_dim),\n",
        "          ReLU(),\n",
        "          Linear(hidden_dim, out_features)]\n",
        "\n",
        "# Initialize network\n",
        "net = Net(layers)\n",
        "\n",
        "# Do forward pass\n",
        "y = net.forward(x)\n",
        "\n",
        "# What will be the shape of output tensor y?\n",
        "print('Shape of ouput tensor y:', y.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_LMryIa6V7Zd"
      },
      "source": [
        "We will now create the same neural network in PyTorch. PyTorch uses the `nn.Module` base class for neural network architectures, which is similar to the `Net` object we have just defined. However, other than in the `Net` class, you have to define all layers inside the network definition.\n",
        "\n",
        "This is an important exercise, as this is how you will define all your future models in PyTorch.\n",
        "\n",
        "You can print a PyTorch `module` to see all sub-modules (i.e. layers) in the module."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "Mln6iWyrP1NF"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TorchNet(\n",
            "  (layer1): Linear(in_features=3, out_features=5, bias=True)\n",
            "  (relu): ReLU()\n",
            "  (layer2): Linear(in_features=5, out_features=4, bias=True)\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "# We will call the network TorchNet\n",
        "class TorchNet(nn.Module):\n",
        "    \"\"\"\n",
        "    PyTorch neural network. Network layers are defined in __init__ and forward\n",
        "    pass implemented in forward.\n",
        "    \n",
        "    Args:\n",
        "        in_features: number of features in input layer\n",
        "        hidden_dim: number of features in hidden dimension\n",
        "        out_features: number of features in output layer\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, in_features, hidden_dim, out_features):\n",
        "        super(TorchNet, self).__init__()\n",
        "        \n",
        "        ########################################################################\n",
        "        #      TODO: Define all layers that will be used in this network.      #\n",
        "        ########################################################################\n",
        "\n",
        "        self.layer1 = nn.Linear(in_features, hidden_dim)\n",
        "        self.relu = nn.ReLU()\n",
        "        self.layer2 = nn.Linear(hidden_dim, out_features)\n",
        "\n",
        "\n",
        "        ########################################################################\n",
        "        #                         END OF YOUR CODE                             #\n",
        "        ########################################################################\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Performs forward pass through all layers of the network.\n",
        "        \n",
        "        Args:\n",
        "            x: input tensor\n",
        "\n",
        "        Returns:\n",
        "            x: output tensor\n",
        "        \"\"\"\n",
        "        \n",
        "        ########################################################################\n",
        "        #                  TODO: Implement this function                       #\n",
        "        ########################################################################\n",
        "\n",
        "        x = self.layer1(x)\n",
        "        x = self.relu(x)\n",
        "        x = self.layer2(x)\n",
        "\n",
        "        ########################################################################\n",
        "        #                         END OF YOUR CODE                             #\n",
        "        ########################################################################\n",
        "\n",
        "        return x\n",
        "\n",
        "# Initialize Pytorch network\n",
        "torch_net = TorchNet(in_features, hidden_dim, out_features)\n",
        "print(torch_net)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQAFYgOtXTFX"
      },
      "source": [
        "We will now again compare the outputs of the two networks. Therefore we again need to load the weights from our network into the PyTorch network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "dBYiJnRwQXqB"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of ouput tensor y: torch.Size([2, 4])\n",
            "Network outputs identical:  True\n"
          ]
        }
      ],
      "source": [
        "# Load the parameters from our model into the Pytorch model\n",
        "torch_net.layer1.weight = nn.Parameter(net.layers[0].weight.t()) # transpose weight by .t()\n",
        "torch_net.layer1.bias = nn.Parameter(net.layers[0].bias)\n",
        "torch_net.layer2.weight = nn.Parameter(net.layers[2].weight.t()) # transpose weight by .t()\n",
        "torch_net.layer2.bias = nn.Parameter(net.layers[2].bias)\n",
        "\n",
        "# Perform forward pass\n",
        "torch_y = torch_net(x)\n",
        "\n",
        "# What will be the shape of output tensor torch_y?\n",
        "print('Shape of ouput tensor y:', torch_y.shape)\n",
        "\n",
        "# Compare outputs using torch.allclose\n",
        "outputs_same = torch.allclose(y, torch_y)\n",
        "print('Network outputs identical: ', outputs_same)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eKrDW462NvNQ"
      },
      "source": [
        "## A1.4: Introducing the XOR problem\n",
        "\n",
        "To introduce the XOR classification problem that we will aim to solve in this assignment using a neural network we'll first generate and visualize the data points. \n",
        "\n",
        "The `plot` function will also be used to visualize the decision boundaries of your neural network.\n",
        "\n",
        "The XOR problem consists of 4 data points belonging to 2 classes which cannot be separated by a linear decision boundary. \n",
        "\n",
        "| x0   | x1   | y    |\n",
        "| ---- | ---- | ---- |\n",
        "| 0    | 0    | 0    |\n",
        "| 0    | 1    | 1    |\n",
        "| 1    | 0    | 1    |\n",
        "| 1    | 1    | 0    |\n",
        "\n",
        "The class labels `y` are [one-hot encoded](https://en.wikipedia.org/wiki/One-hot#Machine_learning_and_statistics), i.e. for each class a binary value indicates whether the sample belongs to it or not. For instance, given classes `1,2,3,4,5` the one-hot encoding of class `4` is given by `[0,0,0,1,0]`. One-hot encoding is a natural way to represent class labels in a classification task since a neural network outputs a class probability vector. `[0,0,0,1,0]` then simply corresponds to a 0% chance of the sample belonging the classes `1,2,3,5` and a 100% chance of it belonging to class `4`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "lR9D37H6JytZ"
      },
      "outputs": [],
      "source": [
        "x_xor = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=torch.float32)\n",
        "y_xor = torch.tensor([[1, 0], [0, 1], [0, 1], [1, 0]]) # one-hot encoded"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "Ll4xMtgPKpSc"
      },
      "outputs": [],
      "source": [
        "def plot(x, y, net=None):\n",
        "    \"\"\"\n",
        "    Plotter function for XOR dataset and classifier boundaries (optional).\n",
        "\n",
        "    Args:\n",
        "        x: Nx2 dimensional data\n",
        "        y: N dimensional labels\n",
        "        net: Model which has a forward function\n",
        "    \"\"\"\n",
        "    # Convert one-hot to class id\n",
        "    y = torch.argmax(y, dim=1)\n",
        "\n",
        "    # Plot decision boundary if net is given\n",
        "    if net:\n",
        "        h = 0.005\n",
        "        x_min, x_max = x[:, 0].min() - 1, x[:, 0].max() + 1\n",
        "        y_min, y_max = x[:, 1].min() - 1, x[:, 1].max() + 1\n",
        "\n",
        "        xx, yy = torch.meshgrid(torch.arange(x_min, x_max, h),\n",
        "                                torch.arange(y_min, y_max, h))\n",
        "        \n",
        "        in_tensor = torch.cat((xx.reshape((-1,1)), yy.reshape((-1,1))), dim=1)\n",
        "\n",
        "        z = net.forward(in_tensor)\n",
        "        z = torch.argmax(z, dim=1)\n",
        "        z = z.reshape(xx.shape)\n",
        "        plt.contourf(xx, yy, z, cmap=plt.cm.coolwarm)\n",
        "\n",
        "    # Plot data points\n",
        "    plt.scatter(x[:, 0], x[:, 1], c=y, cmap=plt.cm.coolwarm, edgecolors='k')\n",
        "    plt.title('XOR problem')\n",
        "    plt.xlabel('x0')\n",
        "    plt.ylabel('x1')\n",
        "    \n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "e4yWIa8TjIBX",
        "outputId": "407c83c9-5b3d-49e2-b481-47c73b58d643"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXCElEQVR4nO3de5RdZZ3m8e+TcAkB5JYgyEVQQMU1olgGl4piM7QBVHQ1YwMK4qhINzg92qM4jpdptWfUtXq8AXKXYVAYR2nFFkWxEVRAU1EEgg2dBiURkATwwp0kv/njHFyHopJU4tlVVN7vZ61anL3ft/b+vaTWec6+nP2mqpAktWvGVBcgSZpaBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAmmSJTkgydI1tJ+b5GOTWZPaZhBoWkmyRZJfJjlqYN2WSW5Lcnh/eeckX0xyd5L7k/wkyavHbKf6bfcl+XWS/5Vk5mSPR3oyMAg0rVTVfcBxwGeSzO2v/iQwWlVfSbIt8EPgEeC5wBzgU8CXHguKAftU1RbAK4C/BP7jMGpMstEwtiNNFoNA005VfQf4JvDZJAcAbwBO6De/C7gPeGtV3VlVD1bVBcDfA/+QJONsbzHwI+D5q9tn/yjkvya5Mcm9Sb6QZFa/7YAkS5OclORO4AtJNk3y6SS3938+nWTTMdt8f5Ll/W2/cQ37fnWSa5P8NslVSZ43pq73JLmuf4RzdpKnJvlWkj8kuSzJNhP436qGGQSart4FHAB8BfgvVXVHf/1BwFeratWY/l8GdgX2GruhJM8G9gcWr2WfbwReBTyzv50PDLTtAGwLPJ3eEct/A15ML1z2AeaN038OsBPwZuCMJM8ap7Z9gXOAdwDbAacDF48Jlb/oj3sv4DXAt4D397c/A/hPaxmXGmcQaFqqqnuBRcBs4KKBpjnAHeP8yh0D7Y/5aZL7gV8A3wdOXctuT66qJVV1D70jjCMH2lYBH66qh6vqQXqh8ZGququqlgF/Bxw9Znsf7Pe/gt4RzhvG2efbgdOr6sdVtbKq/jfwML2Qecznquo3VfVr4AfAj6vqZ1X1MPCPwAvWMi41ziDQtJTkTcBuwGXAJwaalgM7jvMrOw60P2ZfYAt61wf2AzZfy26XDLz+FfC0geVlVfXQwPLT+n1W1//eqrp/De2PeTrwt/3TQr9N8ltglzF9fzPw+sFxlrcYfzhSj0GgaSfJ9vQuAL+d3imTNyR5eb/5MuAvkoz9234DvTfymwdXVs+XgauBD61l17sMvN4VuH1wU2P63k7vTXx1/bdJsvka2h+zBPj7qtp64Gd2/7qHNBQGgaajk4GvVdXl/WsD7wXO7J83/xTwFODsJDskmZXkSHrn7N9Tq3/u+seB45LssIb9ntC/NXVbeufg/+8a+l4AfCDJ3CRz6IXM+WP6/F2STZLsD7wa+H/jbOdM4Pgk+6Vn8ySHJtlyDfuW1olBoGklyeuAlwHveWxdVZ0FLAU+VFV399tnATcCdwPvBo6uqtW+cVfV9cAVg9sdx5eA7wC39H/W9KWvjwGjwHXA9cBPx/S/E7iX3lHAF4Hjq+pfxqlrlN6Rz8n9/ouBY9ewX2mdxYlppLVL8kvgbVV12VTXIg2bRwSS1DiDQJIa56khSWqcRwSS1Lhp93CsOXPm1G677TbVZUjStLJw4cLlVTV3vLZpFwS77bYbo6OjU12GJE0rSX61ujZPDUlS4wwCSWqcQSBJjTMIJKlxzQTBbbfdxpVXXsmyZcumuhRJWif3338/P/zhD7nxxhs72X5nQZDknCR3JblhNe1J8tkki/vT7O3bRR0PPPAAh7/6NezzrOfwzsMO55m77Mq7T3wnq1aNncBKkp58Tj/jTHbYcWcO/8u/4iUvPZAX7LsfS5YsWfsvroMujwjOBeavof1gYM/+z3HA57so4l0nnMjd/3wNZz/6NP7Hfdtw+oqd+M65X+SUk0/uYneSNDQ/+tGPeO97P8ieL/wMe+x7Ms996Rf53UPP49BXv55hPhWisyCoqiuBe9bQ5TDgvP7EINcAWycZb2ap9fbwww/zxQsu4G2PPIVN+/OUPCUzOfqhzfn8pz4zzF1J0tCdfMrpzNn5cGZv0ZvjKJnJjs84iiVLf8N11103tP1M5TWCnXj81H9L++ueIMlxSUaTjK7LOf4HH3yQWrWKpzDzceu3Z2Puvvfe9ShZkibPb+5cxiaznvq4dckMZm++PcuXL1/Nb627qQyCjLNu3GOdqjqjqkaqamTu3HG/IT2urbbaimfs+nQWPG5qWLiC+3jlKw+YeKWSNAUOOeTf8/vllz/uNNBDD9zBb+9ZzMjIyND2M5WPmFjK4+eA3Znx52xdb0n49Omf5w2vfR2/emQFz1y1MT/f+BF+MOtRfviJjw9zV5I0dO94x3GceeYX+OUNH2Or7Q/k4YeWcffSL/ORj/wdW2211dD2M5VHBBcDx/TvHnox8Lv+/LNDdeCBB/L9a65i4yPnc/kLd2G3dxzJwuuvY6+99hr2riRpqLbccksWLLiKdx5/ENtsehn/7plL+cqXz+Vv3/2fh7qfzuYjSHIBcAAwB/gN8GFgY4CqOi1J6M3DOh94AHhLf37WNRoZGSkfOidJ6ybJwqoa93xSZ6eGqurItbQXcEJX+5ckTUwz3yyWJI3PIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmN6zQIksxPclOSxUneN077Vkm+keTnSRYleUuX9UiSnqizIEgyEzgFOBjYGzgyyd5jup0A3FhV+wAHAP+QZJOuapIkPVGXRwTzgMVVdUtVPQJcCBw2pk8BWyYJsAVwD7Ciw5okSWN0GQQ7AUsGlpf21w06GXgOcDtwPfA3VbVq7IaSHJdkNMnosmXLuqpXkprUZRBknHU1ZvlVwLXA04DnAycnecoTfqnqjKoaqaqRuXPnDrtOSWpal0GwFNhlYHlnep/8B70FuKh6FgO3As/usCZJ0hhdBsECYM8ku/cvAB8BXDymz23AgQBJngo8C7ilw5okSWNs1NWGq2pFkhOBS4GZwDlVtSjJ8f3204CPAucmuZ7eqaSTqmp5VzVJkp6osyAAqKpLgEvGrDtt4PXtwJ93WYMkac38ZrEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqXKdBkGR+kpuSLE7yvtX0OSDJtUkWJbmiy3okSU+0UVcbTjITOAU4CFgKLEhycVXdONBna+BUYH5V3ZZk+67qkSSNr8sjgnnA4qq6paoeAS4EDhvT5yjgoqq6DaCq7uqwHknSOLoMgp2AJQPLS/vrBu0FbJPk+0kWJjlmvA0lOS7JaJLRZcuWdVSuJLWpyyDIOOtqzPJGwAuBQ4FXAR9MstcTfqnqjKoaqaqRuXPnDr9SSWpYZ9cI6B0B7DKwvDNw+zh9llfV/cD9Sa4E9gFu7rAuSdKALo8IFgB7Jtk9ySbAEcDFY/p8Hdg/yUZJZgP7Ab/osCZJ0hidHRFU1YokJwKXAjOBc6pqUZLj++2nVdUvknwbuA5YBZxVVTd0VZMk6YlSNfa0/ZPbyMhIjY6OTnUZkjStJFlYVSPjtfnNYklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXHrFQRJDhp2IZKkqbG+RwRnD7UKSdKUWe1jqJOMnTvgj03Adt2UI0mabGuaj2B/4E3AfWPWh97E9JKkDcCaguAa4IGqumJsQ5KbuitJkjSZVhsEVXUwQJK9q+rGMc0f6rQqSdKkmcjF4i8nOSk9myX5HPA/uy5MkjQ5JhIE+wG7AFfRm5D+duClXRYlSZo8EwmCR4EHgc2AWcCtVbWq06okSZNmIkGwgF4QvAh4GXBkkq90WpUkadKs6a6hx7y1qkb7r+8EDktydIc1SZIm0VqPCAZCYHDd/+mmHEnSZPOhc5LUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGdRoESeYnuSnJ4iTvW0O/FyVZmeTwLuuRJD1RZ0GQZCZwCnAwsDe9ZxTtvZp+nwAu7aoWSdLqdXlEMA9YXFW3VNUjwIXAYeP0eyfwVeCuDmuRJK1Gl0GwE7BkYHlpf90fJdkJeD1w2po2lOS4JKNJRpctWzb0QiWpZV0GQcZZV2OWPw2cVFUr17ShqjqjqkaqamTu3LnDqk+SxMQeQ72+ltKb2ewxO9Ob3WzQCHBhEoA5wCFJVlTV1zqsS5I0oMsgWADsmWR34NfAEcBRgx2qavfHXic5F/gnQ0CSJldnQVBVK5KcSO9uoJnAOVW1KMnx/fY1XheQJE2OLo8IqKpLgEvGrBs3AKrq2C5rkSSNz28WS1LjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMZ1GgRJ5ie5KcniJO8bp/2NSa7r/1yVZJ8u65EkPVFnQZBkJnAKcDCwN3Bkkr3HdLsVeEVVPQ/4KHBGV/VIksbX5RHBPGBxVd1SVY8AFwKHDXaoqquq6t7+4jXAzh3WI0kaR5dBsBOwZGB5aX/d6rwV+NZ4DUmOSzKaZHTZsmVDLFGS1GUQZJx1NW7H5JX0guCk8dqr6oyqGqmqkblz5w6xREnSRh1ueymwy8DyzsDtYzsleR5wFnBwVd3dYT2SpHF0eUSwANgzye5JNgGOAC4e7JBkV+Ai4OiqurnDWiRJq9HZEUFVrUhyInApMBM4p6oWJTm+334a8CFgO+DUJAArqmqkq5okSU+UqnFP2z9pjYyM1Ojo6FSXIUnTSpKFq/ug7TeLJalxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuMMAklqnEEgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLjDAJJapxBIEmNMwgkqXEGgSQ1ziCQpMYZBJLUOINAkhpnEEhS4wwCSWqcQSBJjTMIJKlxBoEkNc4gkKTGGQSS1DiDQJIaZxBIUuM2muoCJsPdd9/Neeedx003/xvzXrQvRxxxBLNnz57qsiRprVauXMk3v/lNLvv2pWw3dw7HHHssu++++1D3kaoa6gYft/FkPvAZYCZwVlV9fEx7+u2HAA8Ax1bVT9e0zZGRkRodHZ1wDTfccAMvf8WBbL71C9h41h48fN/P2HSj5Vxz9Q/Yfvvt13VIkjRpHn30UV77qvncuuBaXvLADO7dOHx/xgN84Uvn87rXvW6dtpVkYVWNjNfW2RFBkpnAKcBBwFJgQZKLq+rGgW4HA3v2f/YDPt//79C87e0nsO1Ob2KHp7+2v+Y/cNu/fI4PfOC/c8YZpw5zV5I0VOeffz53/ORaPvnQdsycEVgJ+6+YxduOeTMHL7uLTTfddCj76fIawTxgcVXdUlWPABcCh43pcxhwXvVcA2ydZMdhFfCHP/yBn/10AdvvcvDj1s/d+fX849e+PqzdSFInLjr/S/z5g5swM/njumdlM3aYsQlXX3310PbTZRDsBCwZWF7aX7eufUhyXJLRJKPLli2bcAEzZ86EwKpVjz5u/cqVDw4tSSWpK5tuthkP8/jT91XFQ7WKWbNmDW0/XQZBxlk39oLERPpQVWdU1UhVjcydO3fCBcyePZuDDprPnbecz2PXQmrVCu761fm8+Zg3Tng7kjQVjjnubXxj9sPcVyv/uO5HdR8rZs9i3rx5Q9tPl3cNLQV2GVjeGbh9Pfr8Sc4681QOOOAg/nXhX7PZFnvw+3t+xr7Pfy4f+MD7h7kbSRq617zmNVxx7NH81dln86IZW3LPzFUsmbmSb/7Td5gxY3if4zu7ayjJRsDNwIHAr4EFwFFVtWigz6HAifTuGtoP+GxVrTHm1vWuIYBVq1bxve99j1tvvZV99tmHefPmkYx3MCJJTz6LFy/m8ssvZ9ttt+XQQw9dr9NCU3LXUFWtSHIicCm920fPqapFSY7vt58GXEIvBBbTu330LV3UMmPGDA466KAuNi1Jndtjjz3YY489Ott+p18oq6pL6L3ZD647beB1ASd0WYMkac18xIQkNc4gkKTGGQSS1DiDQJIa1+lD57qQZBnwq/X89TnA8iGWMx045jY45jb8KWN+elWN+43caRcEf4oko6u7j3ZD5Zjb4Jjb0NWYPTUkSY0zCCSpca0FwRlTXcAUcMxtcMxt6GTMTV0jkCQ9UWtHBJKkMQwCSWrcBhkESeYnuSnJ4iTvG6c9ST7bb78uyb5TUecwTWDMb+yP9bokVyXZZyrqHKa1jXmg34uSrExy+GTW14WJjDnJAUmuTbIoyRWTXeOwTeBve6sk30jy8/6YO3mK8WRJck6Su5LcsJr24b9/VdUG9UPvkdf/BjwD2AT4ObD3mD6HAN+iN0Pai4EfT3XdkzDmlwDb9F8f3MKYB/r9M72n4B4+1XVPwr/z1sCNwK795e2nuu5JGPP7gU/0X88F7gE2mera/4QxvxzYF7hhNe1Df//aEI8I5gGLq+qWqnoEuBA4bEyfw4DzqucaYOskO052oUO01jFX1VVVdW9/8Rp6s8FNZxP5dwZ4J/BV4K7JLK4jExnzUcBFVXUbQFVN93FPZMwFbJnebFNb0AuCFZNb5vBU1ZX0xrA6Q3//2hCDYCdgycDy0v66de0znazreN5K7xPFdLbWMSfZCXg9cBobhon8O+8FbJPk+0kWJjlm0qrrxkTGfDLwHHrT3F4P/E1VrZqc8qbE0N+/Op2YZoqMNwfl2HtkJ9JnOpnweJK8kl4QvKzTiro3kTF/GjipqlZuIFOTTmTMGwEvpDdF7GbA1Umuqaqbuy6uIxMZ86uAa4E/A54JfDfJD6rq9x3XNlWG/v61IQbBUmCXgeWd6X1SWNc+08mExpPkecBZwMFVdfck1daViYx5BLiwHwJzgEOSrKiqr01KhcM30b/t5VV1P3B/kiuBfejNHz4dTWTMbwE+Xr0T6IuT3Ao8G/jJ5JQ46Yb+/rUhnhpaAOyZZPckmwBHABeP6XMxcEz/6vuLgd9V1R2TXegQrXXMSXYFLgKOnsafDgetdcxVtXtV7VZVuwFfAf56GocATOxv++vA/kk2SjIb2A/4xSTXOUwTGfNt9I6ASPJU4FnALZNa5eQa+vvXBndEUFUrkpwIXErvjoNzqmpRkuP77afRu4PkEGAx8AC9TxTT1gTH/CFgO+DU/ifkFTWNn9w4wTFvUCYy5qr6RZJvA9cBq4Czqmrc2xCngwn+O38UODfJ9fROm5xUVdP28dRJLgAOAOYkWQp8GNgYunv/8hETktS4DfHUkCRpHRgEktQ4g0CSGmcQSFLjDAJJapxBIA1Rkjcn+df+z5unuh5pIrx9VBqSJNsCo/S+0VzAQuCFAw/7k56UPCKQ1kN/joPrksxKsnmSRcAJwHer6p7+m/93gflTW6m0dhvcN4ulyVBVC5JcDHyM3sPdzgceZcN6qq0a4RGBtP4+AhxE71TQJ9nwnmqrRhgE0vrblt5EKFsCs9jwnmqrRnixWFpP/VNDFwK7AzvSe7DfQnrTDAL8lN7F4jXNNiVNOa8RSOuhP/PXiqr6UpKZwFXA8+k9CXNBv9tHDAFNBx4RSFLjvEYgSY0zCCSpcQaBJDXOIJCkxhkEktQ4g0CSGmcQSFLj/j9LhMY02TQ2/QAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Visualize\n",
        "plot(x_xor, y_xor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9psFGX5JQtq4"
      },
      "source": [
        "We will now use our non-linear neural network to solve the XOR problem.\n",
        "\n",
        "To gain intuition with the transformations in a linear layer you will manually set the correct network parameters. Keep in mind that the network predicts class probabilities, i.e. for each sample it outputs the probability of belonging to class 0 and class 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "_5cj9RQk_pmv"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[1.0000, 0.5000],\n",
            "        [0.0000, 0.5000],\n",
            "        [0.0000, 0.5000],\n",
            "        [1.0000, 0.5000]])\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAcWklEQVR4nO3df5TcdX3v8ecrIb9MCEkIIT/AwOUQWRBBuga4VhuPpQ0UxV6RirbCak25Vw/3KFWpGjx1pVc5R1EwraaaBa+ipYgRroCCBdEqlgBBgRUIJEASTCAkxJCYH+z7/jHfxc0yMzu7OzPf+X7m9ThnT2bn+83Me/Ld7Gs/35l5rSICMzOzSsbkPYCZmbU2B4WZmVXloDAzs6ocFGZmVpWDwszMqnJQmJlZVQ4KsxYkaZGk9VW2XyXpM82cydqXg8KSI2mKpHWS3jXgugMlPSnp7OzzwyR9S9IWSS9I+i9JZw66nci27ZC0QdIXJI1t9uMxy5uDwpITETuAJcCXJB2SXX0ZsCoirpM0A/gZsAc4DpgJXA5c0x8kA5wQEVOAPwH+CnhvPWaUdEA9bsesGRwUlqSI+BHwA+AKSYuAc4APZJs/BOwA3hcRv42IXRHxbeBS4POSVOb21gD/CZxY6T6zVcw/SHpI0lZJPZImZtsWSVov6WOSfgv0SJog6YuSNmYfX5Q0YdBtflzSs9ltv7vKfZ8pabWkbZJ+Luk1g+b6iKRfZSukr0s6VNLNkn4n6TZJ02v4Z7U25aCwlH0IWARcB/x9RDydXX8a8N2I6Bu0/7XAK4EFg29I0jHAG4A1Q9znu4E/B47KbueTA7bNBmYA8ymteD4BnEIpfE4AFpbZfyYwDzgPWC7pVWVmOwlYAfwdcDDwVeCGQaHz9uxxLwDeAtwMfDy7/THAhUM8LmtjDgpLVkRsBR4EXgFcP2DTTODpMn/l6QHb+90r6QWgF7gD+Och7vbLEfFURDxHaYVy7oBtfcCnImJ3ROyiFCqfjojNEfEM8I/A3wy6vaXZ/j+htEI6p8x9vh/4akT8MiJejIirgd2UQqjflRGxKSI2AD8FfhkR90XEbuB7wGuHeFzWxhwUlixJfw0cAdwGfG7ApmeBOWX+ypwB2/udBEyh9PzEycDkIe72qQGXnwDmDvj8mYj4/YDP52b7VNp/a0S8UGV7v/nARdlpp22StgGHD9p304DLu8p8PqX8wzFzUFiiJM2i9AT1+ymdkjlH0huzzbcBb5c0+Ov/HErf6B8ZeGWUXAv8ArhkiLs+fMDlVwIbB97UoH03UvomX2n/6ZImV9ne7yng0oiYNuDjFdnzLmaj5qCwVH0ZWBkRt2fPTXwU+NfsvP3lwFTg65JmS5oo6VxKzxl8JCp3738WWCJpdpX7/UD20tsZlJ4D+Lcq+34b+KSkQyTNpBRC3xy0zz9KGi/pDcCZwL+XuZ1/BS6QdLJKJkv6C0kHVrlvs5o5KCw5kt4G/DHwkf7rIuJrwHrgkojYkm2fCDwEbAE+DPxNRFT8xh4RvwZ+MvB2y7gG+BHwePZR7U1xnwFWAb8Cfg3cO2j/3wJbKa0ivgVcEBG/KTPXKkorpy9n+68Bzq9yv2bDIv/iIrP6kLQO+NuIuC3vWczqySsKMzOrykFhZmZV+dSTmZlV5RWFmZlVlWQx2bTx42LuxAlD71hgT088Mu8RzCwh25554NmIOKTctiSDYu7ECXzzdSfmPUZTdHf05D2CmSVg5bIFT1Ta5lNPBbe0t4sTTzki7zHMLGEOigS8pedNLO3tynsMM0uUgyIhS3u7eG+Xn7sws/pyUCRm3kcXeXVhZnXloEiUVxdmVi8OioR5dWFm9eCgaANLe7v4/Id8qM1sZPzdo01sW3KeVxdmNiIOijaztLeLa85elfcYZlYgSb4z26rr7V7G0uyy39ltZkPxiqLN+XSUmQ3FQWEOCzOrykFhQCksHBhmVo6DwvbjkkEzG8xBYS/jkkEzG8hBYRW5BsTMwEFhQ3ANiJk5KKwmrgExa1/+n281668B8Tu7zdqLg8KGrbd7mU9HmbURB4WNmMPCrD04KGxUlvZ2cdX8K/Iew8wayEFho/bYLfd5dWGWMAeF1Y1XF2ZpclBYXXl1YZYeB4U1hEsGzdLhoLCGcsmgWfE5KKzh+ksG3RtlVkwOCmsa90aZFZODwprOYWFWLA4Ky4VLBs2KI9f/qZJWSNos6YEK2xdJel7S6uzjkmbPaI3TXzJoZq0t7x/prgIWD7HPTyPixOzj002YyZrMjbRmrS3XoIiIO4Hn8pzBWoMbac1aV94rilqcKul+STdLOq7STpKWSFoladXWPfuaOZ/VkWtAzFpPqwfFvcD8iDgBuBJYWWnHiFgeEZ0R0Tln/txmzWcN4BoQs9bS0kEREdsjYkd2+SZgnKSZQ/29tVsn093R0/D5rLFcA2LWGlo6KCTNlqTs8kJK826p9e87LNLgsDDLV94vj/028AvgVZLWS3qfpAskXZDtcjbwgKT7gSuAd0ZEDOc+ujt6mLb86voObk3n1YVZfjTM77uFMH3W8bHonOtfdv17u45k3kcXNX8gq6sbu25n9V3r8h7DLCkrly24JyI6y21r6VNP9baiZ61PRyWgv2TQzJqjrYKiX3dHDzd23Z73GDZKbqQ1a462DAqA1Xet8+oiAW6kNWu8tg2Kft0dPQ6MBLhk0Kxx/D8r093Rw1GLX5v3GDYKLhk0awwHxQDnP3GhVxcJcMmgWX05KMpwWBSfSwbN6sdBUYFPRaXBJYNmo+egqKL/VJTf2V1sLhk0Gx0HRQ0uurzPp6MS4BoQs5FxUAxDd0cPGy67I+8xbJQcFmbD46AYJteApMGrC7PaOShGyDUgaVja28WJpxyR9xhmLc1BMQquAUmDSwbNqnNQ1IGfu0iDSwbNynNQ1Imfu0iDSwbNXs5BUWcuGUyDSwbN/sD/ExrE7+wuPpcMmpU4KBrIJYNpcMmgtTsHRRN0d/TQsfQDeY9ho+CSQWtnDoomedd1nV5dJMAlg9aOHBRN5pLB4nPJoLUbB0UOXDKYBteAWLtwUOTIYZEG14BY6hwUOfP7LtLgGhBLmYOiRbhkMA2uAbEUOShaiEsG0+AaEEvNAXneuaQVwJnA5oh4dZntAr4EnAHsBM6PiHubO2Xz9YdFO32z+dGmZ/jWug1s2r2H46ZO4f1HvZJjDpyS91ijsrS3iw2X3cGKnrV5j9JQfS/uZc39V/Fk7/Xs27uTQ+e/kWMWfpBJkw/NezSrk7xXFFcBi6tsPx04OvtYAvxLE2ZqGe2yuvi3pzay7DfrePuug/hcHMYx28bxv+55kEd+90Leo41aO6wu7v3xP7Bxze0cedyHOfZ1l7F3t7jzu+eyZ/f2vEezOsk1KCLiTuC5KrucBXwjSu4Cpkma05zpWkPqT3bv7evja48/xceZw+vGTGGWxnHmmGm8g+msWPtU3uPVTaolgzu2rWXTkz/jmM5LmTr9OCZNOZwjOy5gytRX8WTvd/Mez+qk1b9y5wEDv1usz65rO6mWDD6zew8HIF6pCftdf5JewcPbd+Q0VWOkWDK47ZmHOOjg1zB27P7H76CZnWzb/GBOU1m9tXpQqMx1UXZHaYmkVZJW7d5VbZFSXCmWDE4fP46d0cfW2Lff9Y/HbuZOmpjTVI2VUsng5KmH8cL2NUT07Xf9C9vXMHnq4TlNZfXW6kGxHhj41XYYsLHcjhGxPCI6I6JzwqQZTRkuLymVDE4aO5a3zpnFF9jEM7GXiKA3dnE1W3j3EekuHlMpGZw26zVMnHwIax9axr69O4h4kWc2/JhnN/4H8487J+/xrE5aPShuAN6jklOA5yPi6byHagUplQxeePQRHD/nIC6MJ3l3PM4Xxmzifx9zBP/94Ol5j9ZwRS8ZlMSpZ34FjdnFqh+fwy9/+FY2r1/JqWd+lVcc2FZPJyZNEWXP5DTnzqVvA4uAmcAm4FPAOICI+Er28tgvU3pl1E6gKyKGXLNPn3V8LDrn+kaN3XKuOXsVvd3L8h5j1Pb09fHCvhc5aNwBjFG5s45pK3rw79u7k74X9zB+4rS8R7ERWLlswT0R0VluW65B0SjtFhT9UjiVYcUPDCumakHR6qeebBi6O3rYcNkdeY9ho+SSQWs1DorErOhZ659IE+CSQWslDopEuWQwDS4ZtFbgoEiYSwbT0A41INbaHBRtIPUakHaRag2ItT5/1bURh0XxpVgDYq3PQdFmvLpIQ0o1INb6HBRtKtWSwXaSSg2ItT4HRRtLsWSwHRW9BsRan4PCkioZbFeP3XKfVxfWMA4KA9IqGWxnS3u7HBhWdw4K2093Rw/Tll+d9xg2Sg4LqycHhb3MRZf3eXWRAK8urF4cFFaRSwbT4JJBGy0HhVXlksE0uGTQRsNBYTXx6iINLhm0kXBQWM28ukiDSwZtuBwUNmyuAUmDSwatVv4qsRFzDUjxuWTQauGgsFFxDUgaXDJo1TgorC5cA1J8Lhm0ShwUVjfvuq7T7+pOgEsGbTAHhdVV/7u6vbooNpcM2kAOCmsIlwymwTUgBg4KazCXDKbBYdHeHBTWcC4ZTINXF+3LQWFN4xqQNLhksP04KKypXAOSBpcMtpcRBYWk0+px55IWS3pY0hpJF5fZvkjS85JWZx+X1ON+LX/dHT3c2HV73mPYKLlksD2MdEXx9dHesaSxwDLgdOBY4FxJx5bZ9acRcWL28enR3q+1jtV3rfPqIgEuGUzfAZU2SLqh0ibg4Drc90JgTUQ8nt3fd4CzgIfqcNtWIP1h4W82xba0t4tpy6/mosv78h7F6qxiUABvAP4a2DHoelH6Jj9a84CnBny+Hji5zH6nSrof2Aj8fUQ8WO7GJC0BlgBMmjK3DuNZs3V39HDV/Ct47Jb78h7FRmjbkvNYCl4pJqbaqae7gJ0R8ZNBH3cAD9fhvlXmuhj0+b3A/Ig4AbgSWFnpxiJieUR0RkTnhEkz6jCe5cElg2lwyWBaKgZFRJweEbdXeN6gHk8qrwcOH/D5YZRWDQNn2B4RO7LLNwHjJM2sw31bi3MNSPG5ZDAdtTyZfa2kj6lkkqQrgf9Th/u+Gzha0pGSxgPvBPZ7XkTSbEnKLi/M5t1Sh/u2AnANSBpcMlh8tQTFyZR+8v85pW/uG4HXj/aOI2If8EHgh0AvcG1EPCjpAkkXZLudDTyQPUdxBfDOiBh8esoS5xqQ4nPJYLFpqO+72U/7lwKnAVOAT0bEd5ow24hNn3V8LDrn+rzHsAbwN5s0eKXYelYuW3BPRHSW21bLiuJuYBfwOuCPKb3f4bo6zmdWM9eApMGBXyy1BMX7IuKSiNgbEb+NiLOA7zd6MLNKXAOSBpcMFseQQRERL3uNW0T838aMY1Y714CkwSWDrc+lgFZorgFJg0sGW5uDwpLQ3dHjwEiASwZbk4PCkuKwKD6XDLYeB4Ulx6uLNCzt7eLzH/K3qFbgo2DJ6u7o4ajFr817DBuFbUvO8+qiBTgoLGkuGUyDSwbz5aCwtuCSweJzyWB+HBTWNlwymAaXDDafg8LajksGi88lg83loLC2dNHlfV5dJMA1IM3hoLC25pLBNLgGpLEcFNb2XDKYBteANI6DwizjksE0uAak/hwUZgO4ZDANrgGpLweFWRmuAUmDa0Dqw/+CZlU4LIrPNSCj56AwG4JXF2nw6mLk/K9mViOXDBafVxcj46AwGwaXDKbBJYPD46AwGwGXDBafSwZr56AwGyGXDKbBJYNDc1CYjZJLBovPJYPVOSjM6sAlg2lwyWB5DgprGdu3PMJv193B71/YnPcoI9auJYN9Edz//HZ+sWUrO/e9mPc4o+aSwf0pIvK7c2kx8CVgLPC1iPjsoO3Ktp8B7ATOj4h7h7rd6bOOj0XnXN+Aia0R9vx+K7+8+UJ2bFvHK6bM53dbe5l/7Nm8+vUXU/oSKKZ2+cn00R0v8JH7exm7D6ZqLGv7dvPhBUfy1rmH5j1aXbTLSnHlsgX3RERnuW25rSgkjQWWAacDxwLnSjp20G6nA0dnH0uAf2nqkNYU9/3HUsZPmMcfvekajl14GSe96VtsfuIXPPHQv+c92qi0Q8ngvr7gQ6sf4uy907mSV/JPHMZlOowrHlnHw7/bkfd4deGSwXxPPS0E1kTE4xGxB/gOcNagfc4CvhEldwHTJM1p9qDWOHt+/zyb1/8n81/1t5R+doBx46dy+IIu1hU8KCD9ksF7tj3P1L6xvHnM1JdWf4drAqdzEDduLO4pxMHavWQwz6CYBzw14PP12XXD3QcASUskrZK0aveu5+o6qDXOvr0vMHbsBMaMnbTf9eMnHsze3dtzmqr+Uq0B2b5vH9MZ+7LrZzCW7Xv25jBRY7VrDUiej7jcyefBT5jUsk/pyojlEdEZEZ0TJs0Y9XDWHJOmzGbc+APZ9uz+75LdvP6HzDr89TlN1TiphcVJ06byQN8utsS+l657MYI7tINTDpme42SN0441IHkGxXrg8AGfHwZsHME+VmDSGF7zJ5fw6Op/4slHruLZp3/Co/d/jq2bf86Czr/Le7yGSGl1cfD48Zw/fx4f4ylu7NvKHX3b+RQbmDj5AE6bNTPv8RqqnWpAcnvVk6QDgEeANwMbgLuBd0XEgwP2+Qvgg5Re9XQycEVELBzqtv2qp+LZ/twa1v76Gnb+7mlmzD6BI1/9V4yfmOZPpANdNf8KHrvlvrzHGLW7t27jxg2b2LnvRd4w62BOn30I48e0zymaFIK/2que8n557BnAFym9PHZFRFwq6QKAiPhK9vLYLwOLKb08tisihoxwB4UVTbudykjRUYtfy/lPXJj3GCPWskHRKA4KK6Jrzl5Fb/eyvMewOijiCqMl30dhZvtzyWA6UlshOijMWoxLBtOQUlg4KMxakEsG05BKyaCDwqyFtWvJYGqKXjLooDBrcSt61josEvCWnjcVdnXhoDArgBU9a9uiZLAdFLFk0EFhViCplwy2i6KVDDoozAoopRqQdlaUksHWn9DMKuru6OGoxa/Newwbhf6SwVbujXJQmBXc+U9c6NVFAnq7l7Xs6SgHhVkiujt66Fj6gbzHsFFqxbBwUJglxDUgaVja28VV86/Ie4yXOCjMEuQakOJ77Jb7WmZ14aAwS9RFl/f5fRcJaIUaEAeFWcL633fhd3YXX55h4aAwawP97+y2YstrdeGgMGsjXl2kodklgw4Kszbj1UUa+ksGm9Eb5aAwa1MuGUzDvI8uanhYOCjM2phLBtPQ6JJBB4WZuWQwEY0qGXRQmNlLXDJYfP0lg/XkoDCz/bhkMA31bKR1UJhZWS4ZLL56NdI6KMysIpcMpmG0JYMOCjMbkksGi280JYMOCjOryUWX93l1kYCR1IDkEhSSZki6VdKj2Z/TK+y3TtKvJa2W1Lq/J9CsjfiVUWkYTljktaK4GPhxRBwN/Dj7vJI3RcSJEdHZnNHMbCh+ZVQaal1d5BUUZwH9JzyvBt6W0xxmNgquAUnDUGGRV1AcGhFPA2R/zqqwXwA/knSPpCXVblDSEkmrJK3aveu5Oo9rZpW4BiR9BzTqhiXdBswus+kTw7iZ10fERkmzgFsl/SYi7iy3Y0QsB5YDTJ91fAx7YDMblf6wyPu3sVn9NWxFERF/GhGvLvPxfWCTpDkA2Z+bK9zGxuzPzcD3gIWNmtfM6sOri/TkderpBuC87PJ5wPcH7yBpsqQD+y8DfwY80LQJzWzEXDKYlryC4rPAaZIeBU7LPkfSXEk3ZfscCvxM0v3AfwE/iIhbcpnWzEbEL6VNQ8Oeo6gmIrYAby5z/UbgjOzy48AJTR7NzOrs/CcuhA4/d1Fkfme2mTWFSwaLy0FhZk3jksFiclCYWdM5LIrFQWFmufAro4rDQWFmueru6GHDZXfkPYZV4aAws9yt6Fnr1UULc1CYWctwyWBrclCYWUtxyWDrcVCYWUvyk92tw0FhZi3NYZE/B4WZtTyvLvLloDCzwnDJYD4cFGZWKP593c3noDCzQnLJYPM4KMyssFwy2BwOCjMrvO6OHqYtvzrvMZLloDCzJFx0eZ9XFw3ioDCzpLhksP4cFGaWnBU9ax0WdeSgMLMkuZG2fhwUZpY0v6t79BwUZtYWHBgj56Aws7bisBg+B4WZtR2vLobHQWFmbcslg7VxUJhZW3PJ4NAcFGZmuGSwGgeFmVnmXdd1OizKyCUoJL1D0oOS+iR1VtlvsaSHJa2RdHEzZzSz9uRG2pfLa0XxAPA/gDsr7SBpLLAMOB04FjhX0rHNGc/M2p1fGfUHuQRFRPRGxMND7LYQWBMRj0fEHuA7wFmNn87M7A9cMggH5D1AFfOApwZ8vh44udLOkpYAS7JPd69ctuCBBs6Wp5nAs3kP0UB+fMWW5ONbueyli0k+vsz8ShsaFhSSbgNml9n0iYj4fi03Uea6qLRzRCwHlmf3vSoiKj73UWQpPzbw4ys6P740NSwoIuJPR3kT64HDB3x+GLBxlLdpZmbD1Movj70bOFrSkZLGA+8Ebsh5JjOztpPXy2P/UtJ64FTgB5J+mF0/V9JNABGxD/gg8EOgF7g2Ih6s8S6WN2DsVpHyYwM/vqLz40uQIiqe9jczM2vpU09mZtYCHBRmZlZV4YMi9ToQSTMk3Srp0ezP6RX2Wyfp15JWS1rV7DmHa6jjoZIrsu2/knRSHnOOVA2Pb5Gk57PjtVrSJXnMORKSVkjaLKnse5USOHZDPb7CHrsRi4hCfwAdwKuAO4DOCvuMBR4D/hswHrgfODbv2Wt8fJcBF2eXLwY+V2G/dcDMvOet8TENeTyAM4CbKb2f5hTgl3nPXefHtwj4f3nPOsLH90bgJOCBCtsLe+xqfHyFPXYj/Sj8iiLSrwM5C7g6u3w18Lb8RqmbWo7HWcA3ouQuYJqkOc0edISK/PU2pIi4E3iuyi5FPna1PL62U/igqFG5OpB5Oc0yXIdGxNMA2Z+zKuwXwI8k3ZPVmbSyWo5HkY9ZrbOfKul+STdLOq45ozVFkY9drVI9dmW1ctfTS5pdB9Js1R7fMG7m9RGxUdIs4FZJv8l+MmpFtRyPlj5mQ6hl9nuB+RGxQ9IZwErg6EYP1iRFPna1SPnYlVWIoIjE60CqPT5JmyTNiYins+X75gq3sTH7c7Ok71E6/dGqQVHL8WjpYzaEIWePiO0DLt8k6Z8lzYyIFArninzshpT4sSurXU49FbkO5AbgvOzyecDLVlCSJks6sP8y8GeUfudHq6rleNwAvCd7Bc0pwPP9p+AKYMjHJ2m2JGWXF1L6v7il6ZM2RpGP3ZASP3ZlFWJFUY2kvwSuBA6hVAeyOiL+XNJc4GsRcUZE7JPUXwcyFlgRtdeB5O2zwLWS3gc8CbwDSnUnZI8POBT4Xva1ewBwTUTcktO8Q6p0PCRdkG3/CnATpVfPrAF2Al15zTtcNT6+s4H/KWkfsAt4Z2QvqWl1kr5N6ZU/M1Wq4vkUMA6Kf+ygpsdX2GM3Uq7wMDOzqtrl1JOZmY2Qg8LMzKpyUJiZWVUOCjMzq8pBYWZmVTkozJpI0nlZE/Cjks4b+m+Y5c8vjzVrEkkzgFVAJ6VKi3uAP4qIrbkOZjYEryjMGkDS67LfxTAxe+f8g8AHgFsj4rksHG4FFuc7qdnQCv/ObLNWFBF3S7oB+AwwCfgmsJf0W1UtQV5RmDXOp4HTKJ1quoz0W1UtUQ4Ks8aZAUwBDgQmknirqqXLT2abNUh26uk7wJHAHOASSk9g9/8O6XspPZnt36ZmLc3PUZg1gKT3APsi4hpJY4GfAycC3ZRqyAE+7ZCwIvCKwszMqvJzFGZmVpWDwszMqnJQmJlZVQ4KMzOrykFhZmZVOSjMzKwqB4WZmVX1/wHt2eNOc44KIQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "in_features, hidden_dim, out_features = 2, 2, 2\n",
        "\n",
        "# Define and initialize layers\n",
        "layers = [Linear(in_features, hidden_dim),\n",
        "          ReLU(),\n",
        "          Linear(hidden_dim, out_features)]\n",
        "\n",
        "# Initialize network\n",
        "net = Net(layers)\n",
        "\n",
        "########################################################################\n",
        "#   TODO: Manually set network parameters to solve the XOR problem.    #\n",
        "########################################################################\n",
        "\n",
        "net.layers[0].weight = torch.tensor(([1., 1.], [1., 1.]))\n",
        "net.layers[0].bias = torch.tensor([[-1., 0.]])\n",
        "\n",
        "net.layers[2].weight = torch.tensor(([2., 0.], [-1., 0.]))\n",
        "net.layers[2].bias = torch.tensor([[1., 0.5]])\n",
        "\n",
        "\n",
        "########################################################################\n",
        "#                         END OF YOUR CODE                             #\n",
        "########################################################################\n",
        "\n",
        "# Forward pass\n",
        "y_pred = net.forward(x_xor)\n",
        "print(y_pred)\n",
        "\n",
        "# Show decision boundary\n",
        "plot(x_xor, y_xor, net)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "83GFFkfiHq1O"
      },
      "source": [
        "We will now increase the size of the hidden representation, i.e. the number of features between the first and second layer, from 2 to 5. Adjust the weights and biases in such a way that this new network yields exactly the same solution as the smaller network. It should be a fairly simple and straightforward modification.\n",
        "\n",
        "*Hint: use the previous solution as a starting point.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "id": "iu5xCxbSFOiz",
        "outputId": "c36ec0c0-6f78-49bd-c157-7171df3feb8e"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAcWklEQVR4nO3df5TcdX3v8ecrIb9MCEkIIT/AwOUQWRBBuga4VhuPpQ0UxV6RirbCak25Vw/3KFWpGjx1pVc5R1EwraaaBa+ipYgRroCCBdEqlgBBgRUIJEASTCAkxJCYH+z7/jHfxc0yMzu7OzPf+X7m9ThnT2bn+83Me/Ld7Gs/35l5rSICMzOzSsbkPYCZmbU2B4WZmVXloDAzs6ocFGZmVpWDwszMqnJQmJlZVQ4KsxYkaZGk9VW2XyXpM82cydqXg8KSI2mKpHWS3jXgugMlPSnp7OzzwyR9S9IWSS9I+i9JZw66nci27ZC0QdIXJI1t9uMxy5uDwpITETuAJcCXJB2SXX0ZsCoirpM0A/gZsAc4DpgJXA5c0x8kA5wQEVOAPwH+CnhvPWaUdEA9bsesGRwUlqSI+BHwA+AKSYuAc4APZJs/BOwA3hcRv42IXRHxbeBS4POSVOb21gD/CZxY6T6zVcw/SHpI0lZJPZImZtsWSVov6WOSfgv0SJog6YuSNmYfX5Q0YdBtflzSs9ltv7vKfZ8pabWkbZJ+Luk1g+b6iKRfZSukr0s6VNLNkn4n6TZJ02v4Z7U25aCwlH0IWARcB/x9RDydXX8a8N2I6Bu0/7XAK4EFg29I0jHAG4A1Q9znu4E/B47KbueTA7bNBmYA8ymteD4BnEIpfE4AFpbZfyYwDzgPWC7pVWVmOwlYAfwdcDDwVeCGQaHz9uxxLwDeAtwMfDy7/THAhUM8LmtjDgpLVkRsBR4EXgFcP2DTTODpMn/l6QHb+90r6QWgF7gD+Och7vbLEfFURDxHaYVy7oBtfcCnImJ3ROyiFCqfjojNEfEM8I/A3wy6vaXZ/j+htEI6p8x9vh/4akT8MiJejIirgd2UQqjflRGxKSI2AD8FfhkR90XEbuB7wGuHeFzWxhwUlixJfw0cAdwGfG7ApmeBOWX+ypwB2/udBEyh9PzEycDkIe72qQGXnwDmDvj8mYj4/YDP52b7VNp/a0S8UGV7v/nARdlpp22StgGHD9p304DLu8p8PqX8wzFzUFiiJM2i9AT1+ymdkjlH0huzzbcBb5c0+Ov/HErf6B8ZeGWUXAv8ArhkiLs+fMDlVwIbB97UoH03UvomX2n/6ZImV9ne7yng0oiYNuDjFdnzLmaj5qCwVH0ZWBkRt2fPTXwU+NfsvP3lwFTg65JmS5oo6VxKzxl8JCp3738WWCJpdpX7/UD20tsZlJ4D+Lcq+34b+KSkQyTNpBRC3xy0zz9KGi/pDcCZwL+XuZ1/BS6QdLJKJkv6C0kHVrlvs5o5KCw5kt4G/DHwkf7rIuJrwHrgkojYkm2fCDwEbAE+DPxNRFT8xh4RvwZ+MvB2y7gG+BHwePZR7U1xnwFWAb8Cfg3cO2j/3wJbKa0ivgVcEBG/KTPXKkorpy9n+68Bzq9yv2bDIv/iIrP6kLQO+NuIuC3vWczqySsKMzOrykFhZmZV+dSTmZlV5RWFmZlVlWQx2bTx42LuxAlD71hgT088Mu8RzCwh25554NmIOKTctiSDYu7ECXzzdSfmPUZTdHf05D2CmSVg5bIFT1Ta5lNPBbe0t4sTTzki7zHMLGEOigS8pedNLO3tynsMM0uUgyIhS3u7eG+Xn7sws/pyUCRm3kcXeXVhZnXloEiUVxdmVi8OioR5dWFm9eCgaANLe7v4/Id8qM1sZPzdo01sW3KeVxdmNiIOijaztLeLa85elfcYZlYgSb4z26rr7V7G0uyy39ltZkPxiqLN+XSUmQ3FQWEOCzOrykFhQCksHBhmVo6DwvbjkkEzG8xBYS/jkkEzG8hBYRW5BsTMwEFhQ3ANiJk5KKwmrgExa1/+n281668B8Tu7zdqLg8KGrbd7mU9HmbURB4WNmMPCrD04KGxUlvZ2cdX8K/Iew8wayEFho/bYLfd5dWGWMAeF1Y1XF2ZpclBYXXl1YZYeB4U1hEsGzdLhoLCGcsmgWfE5KKzh+ksG3RtlVkwOCmsa90aZFZODwprOYWFWLA4Ky4VLBs2KI9f/qZJWSNos6YEK2xdJel7S6uzjkmbPaI3TXzJoZq0t7x/prgIWD7HPTyPixOzj002YyZrMjbRmrS3XoIiIO4Hn8pzBWoMbac1aV94rilqcKul+STdLOq7STpKWSFoladXWPfuaOZ/VkWtAzFpPqwfFvcD8iDgBuBJYWWnHiFgeEZ0R0Tln/txmzWcN4BoQs9bS0kEREdsjYkd2+SZgnKSZQ/29tVsn093R0/D5rLFcA2LWGlo6KCTNlqTs8kJK826p9e87LNLgsDDLV94vj/028AvgVZLWS3qfpAskXZDtcjbwgKT7gSuAd0ZEDOc+ujt6mLb86voObk3n1YVZfjTM77uFMH3W8bHonOtfdv17u45k3kcXNX8gq6sbu25n9V3r8h7DLCkrly24JyI6y21r6VNP9baiZ61PRyWgv2TQzJqjrYKiX3dHDzd23Z73GDZKbqQ1a462DAqA1Xet8+oiAW6kNWu8tg2Kft0dPQ6MBLhk0Kxx/D8r093Rw1GLX5v3GDYKLhk0awwHxQDnP3GhVxcJcMmgWX05KMpwWBSfSwbN6sdBUYFPRaXBJYNmo+egqKL/VJTf2V1sLhk0Gx0HRQ0uurzPp6MS4BoQs5FxUAxDd0cPGy67I+8xbJQcFmbD46AYJteApMGrC7PaOShGyDUgaVja28WJpxyR9xhmLc1BMQquAUmDSwbNqnNQ1IGfu0iDSwbNynNQ1Imfu0iDSwbNXs5BUWcuGUyDSwbN/sD/ExrE7+wuPpcMmpU4KBrIJYNpcMmgtTsHRRN0d/TQsfQDeY9ho+CSQWtnDoomedd1nV5dJMAlg9aOHBRN5pLB4nPJoLUbB0UOXDKYBteAWLtwUOTIYZEG14BY6hwUOfP7LtLgGhBLmYOiRbhkMA2uAbEUOShaiEsG0+AaEEvNAXneuaQVwJnA5oh4dZntAr4EnAHsBM6PiHubO2Xz9YdFO32z+dGmZ/jWug1s2r2H46ZO4f1HvZJjDpyS91ijsrS3iw2X3cGKnrV5j9JQfS/uZc39V/Fk7/Xs27uTQ+e/kWMWfpBJkw/NezSrk7xXFFcBi6tsPx04OvtYAvxLE2ZqGe2yuvi3pzay7DfrePuug/hcHMYx28bxv+55kEd+90Leo41aO6wu7v3xP7Bxze0cedyHOfZ1l7F3t7jzu+eyZ/f2vEezOsk1KCLiTuC5KrucBXwjSu4Cpkma05zpWkPqT3bv7evja48/xceZw+vGTGGWxnHmmGm8g+msWPtU3uPVTaolgzu2rWXTkz/jmM5LmTr9OCZNOZwjOy5gytRX8WTvd/Mez+qk1b9y5wEDv1usz65rO6mWDD6zew8HIF6pCftdf5JewcPbd+Q0VWOkWDK47ZmHOOjg1zB27P7H76CZnWzb/GBOU1m9tXpQqMx1UXZHaYmkVZJW7d5VbZFSXCmWDE4fP46d0cfW2Lff9Y/HbuZOmpjTVI2VUsng5KmH8cL2NUT07Xf9C9vXMHnq4TlNZfXW6kGxHhj41XYYsLHcjhGxPCI6I6JzwqQZTRkuLymVDE4aO5a3zpnFF9jEM7GXiKA3dnE1W3j3EekuHlMpGZw26zVMnHwIax9axr69O4h4kWc2/JhnN/4H8487J+/xrE5aPShuAN6jklOA5yPi6byHagUplQxeePQRHD/nIC6MJ3l3PM4Xxmzifx9zBP/94Ol5j9ZwRS8ZlMSpZ34FjdnFqh+fwy9/+FY2r1/JqWd+lVcc2FZPJyZNEWXP5DTnzqVvA4uAmcAm4FPAOICI+Er28tgvU3pl1E6gKyKGXLNPn3V8LDrn+kaN3XKuOXsVvd3L8h5j1Pb09fHCvhc5aNwBjFG5s45pK3rw79u7k74X9zB+4rS8R7ERWLlswT0R0VluW65B0SjtFhT9UjiVYcUPDCumakHR6qeebBi6O3rYcNkdeY9ho+SSQWs1DorErOhZ659IE+CSQWslDopEuWQwDS4ZtFbgoEiYSwbT0A41INbaHBRtIPUakHaRag2ItT5/1bURh0XxpVgDYq3PQdFmvLpIQ0o1INb6HBRtKtWSwXaSSg2ItT4HRRtLsWSwHRW9BsRan4PCkioZbFeP3XKfVxfWMA4KA9IqGWxnS3u7HBhWdw4K2093Rw/Tll+d9xg2Sg4LqycHhb3MRZf3eXWRAK8urF4cFFaRSwbT4JJBGy0HhVXlksE0uGTQRsNBYTXx6iINLhm0kXBQWM28ukiDSwZtuBwUNmyuAUmDSwatVv4qsRFzDUjxuWTQauGgsFFxDUgaXDJo1TgorC5cA1J8Lhm0ShwUVjfvuq7T7+pOgEsGbTAHhdVV/7u6vbooNpcM2kAOCmsIlwymwTUgBg4KazCXDKbBYdHeHBTWcC4ZTINXF+3LQWFN4xqQNLhksP04KKypXAOSBpcMtpcRBYWk0+px55IWS3pY0hpJF5fZvkjS85JWZx+X1ON+LX/dHT3c2HV73mPYKLlksD2MdEXx9dHesaSxwDLgdOBY4FxJx5bZ9acRcWL28enR3q+1jtV3rfPqIgEuGUzfAZU2SLqh0ibg4Drc90JgTUQ8nt3fd4CzgIfqcNtWIP1h4W82xba0t4tpy6/mosv78h7F6qxiUABvAP4a2DHoelH6Jj9a84CnBny+Hji5zH6nSrof2Aj8fUQ8WO7GJC0BlgBMmjK3DuNZs3V39HDV/Ct47Jb78h7FRmjbkvNYCl4pJqbaqae7gJ0R8ZNBH3cAD9fhvlXmuhj0+b3A/Ig4AbgSWFnpxiJieUR0RkTnhEkz6jCe5cElg2lwyWBaKgZFRJweEbdXeN6gHk8qrwcOH/D5YZRWDQNn2B4RO7LLNwHjJM2sw31bi3MNSPG5ZDAdtTyZfa2kj6lkkqQrgf9Th/u+Gzha0pGSxgPvBPZ7XkTSbEnKLi/M5t1Sh/u2AnANSBpcMlh8tQTFyZR+8v85pW/uG4HXj/aOI2If8EHgh0AvcG1EPCjpAkkXZLudDTyQPUdxBfDOiBh8esoS5xqQ4nPJYLFpqO+72U/7lwKnAVOAT0bEd5ow24hNn3V8LDrn+rzHsAbwN5s0eKXYelYuW3BPRHSW21bLiuJuYBfwOuCPKb3f4bo6zmdWM9eApMGBXyy1BMX7IuKSiNgbEb+NiLOA7zd6MLNKXAOSBpcMFseQQRERL3uNW0T838aMY1Y714CkwSWDrc+lgFZorgFJg0sGW5uDwpLQ3dHjwEiASwZbk4PCkuKwKD6XDLYeB4Ulx6uLNCzt7eLzH/K3qFbgo2DJ6u7o4ajFr817DBuFbUvO8+qiBTgoLGkuGUyDSwbz5aCwtuCSweJzyWB+HBTWNlwymAaXDDafg8LajksGi88lg83loLC2dNHlfV5dJMA1IM3hoLC25pLBNLgGpLEcFNb2XDKYBteANI6DwizjksE0uAak/hwUZgO4ZDANrgGpLweFWRmuAUmDa0Dqw/+CZlU4LIrPNSCj56AwG4JXF2nw6mLk/K9mViOXDBafVxcj46AwGwaXDKbBJYPD46AwGwGXDBafSwZr56AwGyGXDKbBJYNDc1CYjZJLBovPJYPVOSjM6sAlg2lwyWB5DgprGdu3PMJv193B71/YnPcoI9auJYN9Edz//HZ+sWUrO/e9mPc4o+aSwf0pIvK7c2kx8CVgLPC1iPjsoO3Ktp8B7ATOj4h7h7rd6bOOj0XnXN+Aia0R9vx+K7+8+UJ2bFvHK6bM53dbe5l/7Nm8+vUXU/oSKKZ2+cn00R0v8JH7exm7D6ZqLGv7dvPhBUfy1rmH5j1aXbTLSnHlsgX3RERnuW25rSgkjQWWAacDxwLnSjp20G6nA0dnH0uAf2nqkNYU9/3HUsZPmMcfvekajl14GSe96VtsfuIXPPHQv+c92qi0Q8ngvr7gQ6sf4uy907mSV/JPHMZlOowrHlnHw7/bkfd4deGSwXxPPS0E1kTE4xGxB/gOcNagfc4CvhEldwHTJM1p9qDWOHt+/zyb1/8n81/1t5R+doBx46dy+IIu1hU8KCD9ksF7tj3P1L6xvHnM1JdWf4drAqdzEDduLO4pxMHavWQwz6CYBzw14PP12XXD3QcASUskrZK0aveu5+o6qDXOvr0vMHbsBMaMnbTf9eMnHsze3dtzmqr+Uq0B2b5vH9MZ+7LrZzCW7Xv25jBRY7VrDUiej7jcyefBT5jUsk/pyojlEdEZEZ0TJs0Y9XDWHJOmzGbc+APZ9uz+75LdvP6HzDr89TlN1TiphcVJ06byQN8utsS+l657MYI7tINTDpme42SN0441IHkGxXrg8AGfHwZsHME+VmDSGF7zJ5fw6Op/4slHruLZp3/Co/d/jq2bf86Czr/Le7yGSGl1cfD48Zw/fx4f4ylu7NvKHX3b+RQbmDj5AE6bNTPv8RqqnWpAcnvVk6QDgEeANwMbgLuBd0XEgwP2+Qvgg5Re9XQycEVELBzqtv2qp+LZ/twa1v76Gnb+7mlmzD6BI1/9V4yfmOZPpANdNf8KHrvlvrzHGLW7t27jxg2b2LnvRd4w62BOn30I48e0zymaFIK/2que8n557BnAFym9PHZFRFwq6QKAiPhK9vLYLwOLKb08tisihoxwB4UVTbudykjRUYtfy/lPXJj3GCPWskHRKA4KK6Jrzl5Fb/eyvMewOijiCqMl30dhZvtzyWA6UlshOijMWoxLBtOQUlg4KMxakEsG05BKyaCDwqyFtWvJYGqKXjLooDBrcSt61josEvCWnjcVdnXhoDArgBU9a9uiZLAdFLFk0EFhViCplwy2i6KVDDoozAoopRqQdlaUksHWn9DMKuru6OGoxa/Newwbhf6SwVbujXJQmBXc+U9c6NVFAnq7l7Xs6SgHhVkiujt66Fj6gbzHsFFqxbBwUJglxDUgaVja28VV86/Ie4yXOCjMEuQakOJ77Jb7WmZ14aAwS9RFl/f5fRcJaIUaEAeFWcL633fhd3YXX55h4aAwawP97+y2YstrdeGgMGsjXl2kodklgw4Kszbj1UUa+ksGm9Eb5aAwa1MuGUzDvI8uanhYOCjM2phLBtPQ6JJBB4WZuWQwEY0qGXRQmNlLXDJYfP0lg/XkoDCz/bhkMA31bKR1UJhZWS4ZLL56NdI6KMysIpcMpmG0JYMOCjMbkksGi280JYMOCjOryUWX93l1kYCR1IDkEhSSZki6VdKj2Z/TK+y3TtKvJa2W1Lq/J9CsjfiVUWkYTljktaK4GPhxRBwN/Dj7vJI3RcSJEdHZnNHMbCh+ZVQaal1d5BUUZwH9JzyvBt6W0xxmNgquAUnDUGGRV1AcGhFPA2R/zqqwXwA/knSPpCXVblDSEkmrJK3aveu5Oo9rZpW4BiR9BzTqhiXdBswus+kTw7iZ10fERkmzgFsl/SYi7iy3Y0QsB5YDTJ91fAx7YDMblf6wyPu3sVn9NWxFERF/GhGvLvPxfWCTpDkA2Z+bK9zGxuzPzcD3gIWNmtfM6sOri/TkderpBuC87PJ5wPcH7yBpsqQD+y8DfwY80LQJzWzEXDKYlryC4rPAaZIeBU7LPkfSXEk3ZfscCvxM0v3AfwE/iIhbcpnWzEbEL6VNQ8Oeo6gmIrYAby5z/UbgjOzy48AJTR7NzOrs/CcuhA4/d1Fkfme2mTWFSwaLy0FhZk3jksFiclCYWdM5LIrFQWFmufAro4rDQWFmueru6GHDZXfkPYZV4aAws9yt6Fnr1UULc1CYWctwyWBrclCYWUtxyWDrcVCYWUvyk92tw0FhZi3NYZE/B4WZtTyvLvLloDCzwnDJYD4cFGZWKP593c3noDCzQnLJYPM4KMyssFwy2BwOCjMrvO6OHqYtvzrvMZLloDCzJFx0eZ9XFw3ioDCzpLhksP4cFGaWnBU9ax0WdeSgMLMkuZG2fhwUZpY0v6t79BwUZtYWHBgj56Aws7bisBg+B4WZtR2vLobHQWFmbcslg7VxUJhZW3PJ4NAcFGZmuGSwGgeFmVnmXdd1OizKyCUoJL1D0oOS+iR1VtlvsaSHJa2RdHEzZzSz9uRG2pfLa0XxAPA/gDsr7SBpLLAMOB04FjhX0rHNGc/M2p1fGfUHuQRFRPRGxMND7LYQWBMRj0fEHuA7wFmNn87M7A9cMggH5D1AFfOApwZ8vh44udLOkpYAS7JPd69ctuCBBs6Wp5nAs3kP0UB+fMWW5ONbueyli0k+vsz8ShsaFhSSbgNml9n0iYj4fi03Uea6qLRzRCwHlmf3vSoiKj73UWQpPzbw4ys6P740NSwoIuJPR3kT64HDB3x+GLBxlLdpZmbD1Movj70bOFrSkZLGA+8Ebsh5JjOztpPXy2P/UtJ64FTgB5J+mF0/V9JNABGxD/gg8EOgF7g2Ih6s8S6WN2DsVpHyYwM/vqLz40uQIiqe9jczM2vpU09mZtYCHBRmZlZV4YMi9ToQSTMk3Srp0ezP6RX2Wyfp15JWS1rV7DmHa6jjoZIrsu2/knRSHnOOVA2Pb5Gk57PjtVrSJXnMORKSVkjaLKnse5USOHZDPb7CHrsRi4hCfwAdwKuAO4DOCvuMBR4D/hswHrgfODbv2Wt8fJcBF2eXLwY+V2G/dcDMvOet8TENeTyAM4CbKb2f5hTgl3nPXefHtwj4f3nPOsLH90bgJOCBCtsLe+xqfHyFPXYj/Sj8iiLSrwM5C7g6u3w18Lb8RqmbWo7HWcA3ouQuYJqkOc0edISK/PU2pIi4E3iuyi5FPna1PL62U/igqFG5OpB5Oc0yXIdGxNMA2Z+zKuwXwI8k3ZPVmbSyWo5HkY9ZrbOfKul+STdLOq45ozVFkY9drVI9dmW1ctfTS5pdB9Js1R7fMG7m9RGxUdIs4FZJv8l+MmpFtRyPlj5mQ6hl9nuB+RGxQ9IZwErg6EYP1iRFPna1SPnYlVWIoIjE60CqPT5JmyTNiYins+X75gq3sTH7c7Ok71E6/dGqQVHL8WjpYzaEIWePiO0DLt8k6Z8lzYyIFArninzshpT4sSurXU49FbkO5AbgvOzyecDLVlCSJks6sP8y8GeUfudHq6rleNwAvCd7Bc0pwPP9p+AKYMjHJ2m2JGWXF1L6v7il6ZM2RpGP3ZASP3ZlFWJFUY2kvwSuBA6hVAeyOiL+XNJc4GsRcUZE7JPUXwcyFlgRtdeB5O2zwLWS3gc8CbwDSnUnZI8POBT4Xva1ewBwTUTcktO8Q6p0PCRdkG3/CnATpVfPrAF2Al15zTtcNT6+s4H/KWkfsAt4Z2QvqWl1kr5N6ZU/M1Wq4vkUMA6Kf+ygpsdX2GM3Uq7wMDOzqtrl1JOZmY2Qg8LMzKpyUJiZWVUOCjMzq8pBYWZmVTkozJpI0nlZE/Cjks4b+m+Y5c8vjzVrEkkzgFVAJ6VKi3uAP4qIrbkOZjYEryjMGkDS67LfxTAxe+f8g8AHgFsj4rksHG4FFuc7qdnQCv/ObLNWFBF3S7oB+AwwCfgmsJf0W1UtQV5RmDXOp4HTKJ1quoz0W1UtUQ4Ks8aZAUwBDgQmknirqqXLT2abNUh26uk7wJHAHOASSk9g9/8O6XspPZnt36ZmLc3PUZg1gKT3APsi4hpJY4GfAycC3ZRqyAE+7ZCwIvCKwszMqvJzFGZmVpWDwszMqnJQmJlZVQ4KMzOrykFhZmZVOSjMzKwqB4WZmVX1/wHt2eNOc44KIQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "in_features, hidden_dim, out_features = 2, 5, 2\n",
        "\n",
        "# Define and initialize layers\n",
        "layers = [Linear(in_features, hidden_dim),\n",
        "          ReLU(),\n",
        "          Linear(hidden_dim, out_features)]\n",
        "\n",
        "# Initialize network\n",
        "net = Net(layers)\n",
        "\n",
        "########################################################################\n",
        "#   TODO: Manually set network parameters to solve the XOR problem.    #\n",
        "########################################################################\n",
        "\n",
        "net.layers[0].weight = torch.tensor(([1., 1., 0., 0., 0.],\n",
        "                                     [1., 1., 0., 0., 0.]))\n",
        "net.layers[0].bias = torch.tensor([[-1., 0., 0., 0., 0.]])\n",
        "\n",
        "net.layers[2].weight = torch.tensor(([2., 0.],\n",
        "                                     [-1., 0.],\n",
        "                                     [0., 0.],\n",
        "                                     [0., 0.],\n",
        "                                     [0., 0.]))\n",
        "net.layers[2].bias = torch.tensor([[1., 0.5]])\n",
        "\n",
        "########################################################################\n",
        "#                         END OF YOUR CODE                             #\n",
        "########################################################################\n",
        "\n",
        "# Forward pass\n",
        "y_pred = net.forward(x_xor)\n",
        "\n",
        "# Show decision boundary\n",
        "plot(x_xor, y_xor, net)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g1PAklq65pew"
      },
      "source": [
        "## A1.5 A first introduction to the training loop\n",
        "\n",
        "Of course the power of a neural network lies in the fact that it is able to learn from data. We therefore now introduce the training loop. Roughly speaking the training loop consists of the following steps:\n",
        "```\n",
        "while not converged:\n",
        " 1. feed training sample to network to predict output (forward step)\n",
        " 2. compare prediction to label (compute loss)\n",
        " 3. use comparison to update network parameters (backward step)\n",
        "```\n",
        "Since we have not yet implemented the backpropagation algorithm we will attempt to optimize our neural network in a rather primitive fashion by randomly generating weights until the problem is solved, i.e. all data points are classified correctly.\n",
        "\n",
        "Even though a loss function is not required in our case, we will start by implementing the mean squared error loss to act as a heuristic. Given prediction $\\hat{y}$ and groud-truth $y$ the MSE loss is defined as:\n",
        "$$\\text{MSE}(\\hat{y},y)= \\frac{1}{n} \\sum_{i=1}^n (\\hat{y}-y)^2 $$\n",
        "\n",
        "Again keep in mind that $\\hat{y}$ is a class probability vector and $y$ is a one-hot encoded representation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "gLZE2-Pe0zkq"
      },
      "outputs": [],
      "source": [
        "def MSELoss(y_true, y_pred):\n",
        "    \"\"\"\n",
        "    Computes mean squared error loss between y_true and y_pred.\n",
        "\n",
        "    Args:\n",
        "      y_true: Tensor containing true labels.\n",
        "      y_pred: Tensor containing predictions.\n",
        "\n",
        "    return:\n",
        "      loss: Mean squared error loss\n",
        "    \"\"\"\n",
        "\n",
        "    ########################################################################\n",
        "    #                  TODO: Implement this function                       #\n",
        "    ########################################################################\n",
        "\n",
        "    loss = torch.mean((y_true - y_pred)**2)\n",
        "\n",
        "    ########################################################################\n",
        "    #                         END OF YOUR CODE                             #\n",
        "    ########################################################################\n",
        "\n",
        "    return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FZ8jint2Kbqk"
      },
      "source": [
        "You will now implement our primitive training loop consisting of the steps:\n",
        "```\n",
        "while acc < 1:\n",
        " 1. randomize network weights\n",
        " 2. forward pass\n",
        " 3. compute loss\n",
        " 4. compute accuracy\n",
        " ```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "--1sxOkIQzA4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tensor([[0.9014, 0.7639],\n",
            "        [0.6561, 0.6647],\n",
            "        [1.1831, 1.2388],\n",
            "        [1.1052, 1.0903]])\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAscAAAEWCAYAAACKfDo5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABH2ElEQVR4nO3dd5xcdb3/8dcnm0oSUggEUiCUUEILEHpxUUHAgl6Ri14V0WtE8aoIKqjYAH/ovYAiCEYFRClKUyQhIYEsgRDSO+l9s+nJZlu2f39/zNnN7O70dubMvJ+Pxz525swpnzPn7M5nvudzvl9zziEiIiIiItDN7wBERERERPKFkmMREREREY+SYxERERERj5JjERERERGPkmMREREREY+SYxERERERj5JjERERCSwzKzWz8hivP2lm9+QyJgk2JcciIiJFysz6mdlGM/tc2LT+ZrbZzK7zno8ws6fNbI+Z1ZrZHDP7WKf1OO+1GjPbamYPmFlJrvdHJBOUHIuIiBQp51wNMB74rZkd7k3+NTDPOfeCmQ0G3gEagVOBIcCDwDNtyXOYM51z/YAPAP8JfDkTMZpZ90ysRyRRSo5FRESKmHPudWAi8JCZlQLXA7d4L98K1ABfcc5td84dcM49C9wL3G9mFmF9a4GZwNho2/Raq+80s/fNbJ+ZPWFmvb3XSs2s3Mx+YGbbgSfMrJeZ/cbMKryf35hZr07r/KGZ7fbW/V8xtv0xM1tkZpVm9q6ZndEpru+Z2RKvJfzPZjbUzF4zs2ozm2ZmgxJ4WyXAlByLiIjIrUAp8AJwu3Numzf9CuBF51xrp/n/ARwNnNh5RWZ2MnApsDbONv8L+AhwvLeeH4e9diQwGDiGUMv2j4ALCCXcZwLnRZh/CDAcuBGYYGYnRYjtbOBx4GvAYcAfgFc6Jdqf9vb7RODjwGvAD731dwO+FWe/JOCUHIuIiBQ559w+YDlwCPBS2EtDgG0RFtkW9nqbBWZWC6wAyoDfx9nsw865Lc65vYRaoj8b9lor8FPnXINz7gChRPoXzrmdzrldwM+BL3Ra313e/G8Ragm/PsI2vwr8wTk32znX4pz7C9BAKPFu8zvn3A7n3FbgbWC2c26hc64BeBk4K85+ScApORYRESlyZvZ5YBQwDfhV2Eu7gaMiLHJU2Ottzgb6Eao3Ph/oG2ezW8IebwKGhT3f5ZyrD3s+zJsn2vz7nHO1MV5vcwxwm1dSUWlmlcDITvPuCHt8IMLzfpF3RwqFkmMREZEiZmZHELrJ7quEyg2uN7PLvJenAZ82s875wvWEktvV4RNdyD+AWcBP4mx6ZNjjo4GK8FV1mreCUGIbbf5BZtY3xutttgD3OucGhv0c4tVRiwBKjkVERIrdw8A/nXPTvVrj7wN/9OpwHwQOBf5sZkeaWW8z+yyhGuDvOec6J7Ft7gPGm9mRMbZ7i9dN3GBCNb1/jzHvs8CPzexwMxtCKPH+W6d5fm5mPc3sUuBjwPMR1vNH4GYzO99C+prZR82sf4xtS5FRciwiIlKkzOyTwCXA99qmOef+BJQDP3HO7fFe7w28D+wBvgt8wTkXNZl1zi0F3gpfbwTPAK8D672fWAN13APMA5YAS4EFnebfDuwj1Fr8NHCzc25lhLjmEWohf9ibfy3wpRjblSJk0b/0iYiIiGSemW0E/ts5N83vWEQ6U8uxiIiIiIhHybGIiIiIiEdlFSIiIiIiHrUci4iIiIh4uvsdQLghQ4a4UaNGJbVMbW0tffvG62c8v2kf/Bf0+CH4+xD0+OfPn7/bOXe433Hk0sCePdyw3r3izygikmdWVNdG/Z+dV8nxqFGjmDdvXlLLlJWVUVpamp2AckT74L+gxw/B34egx29mm+LPVViG9e7F384d63cYIiJJO+fNmVH/Z6usQkRERETEo+RYRERERMSj5FhERERExKPkWERERETEo+RYRERERMSj5FhERERExKPkWERERETEo+RYAmHtzhreW7/H7zBERESkwOXVICAi0Xz4gbcA2HjfR32ORERERAqZWo5FRERERDxKjkVEREREPEqORUREREQ8So5FxDe7axqorGv0OwwREZF2uiFPRHwz7p5pgG60FBGR/KGWYxERERERj5JjERERERGPkmMREREREY+SYxERERERj5JjERERERGPkmMREREREY+SYxERERERj5JjERERERFPVgcBMbONQDXQAjQ758Zlc3siIiIiIunIxQh5lzvndudgOyIiIiIiaVFZhYiIiIiIJ9stxw543cwc8Afn3ITOM5jZeGA8wNChQykrK0tqAzU1NUkvk2+0D4nL1jZ0DPxVVlYW6PhFRKRwZDs5vtg5V2FmRwBTzWylc25G+AxewjwBYNy4ca60tDSpDZSVlZHsMvlG+5CAyRMBsrYNHQOfhB3XQMYvIiIFJ6tlFc65Cu/3TuBl4Lxsbk9EREREJB1ZS47NrK+Z9W97DFwJLMvW9kRERERE0pXNsoqhwMtm1radZ5xzk7O4PRERERGRtGQtOXbOrQfOzNb6RUREREQyTV25iYgUODN73Mx2mlnE0jYzKzWz/Wa2yPv5Sa5jFBHJF7kYBERERPz1JPAw8FSMed52zn0sN+GIiOQvtRyLiBQ4rwvNvX7HISISBEqORUQE4EIzW2xmr5nZqdFmMrPxZjbPzObta2zOZXwiIjmh5FhERBYAxzjnzgR+B/wz2ozOuQnOuXHOuXGDeqoyT0QKj5JjEZEi55yrcs7VeI8nAT3MbIjPYYmI+ELJsYhIkTOzI83rlN7MziP02bDH36hERPyha2IiIgXOzJ4FSoEhZlYO/BToAeCcewy4Dvi6mTUDB4AbnHPOp3BFRHyl5FhEpMA55z4b5/WHCXX1JiJS9FRWISIiIiLiUXIsIiIiIuJRciwiIiIi4lFyLCIiIiLiUXIsIiIiIuJRciwiIiIi4lFyLCIiIiLiUXIsIiIiIuJRciwiIiIi4lFyLCIiIiLiUXIsIiIiIuJRciwiIiIi4lFyLCIiIiLiUXIsIiIiIuJRciy+21p5gFF3TGTuxr1+hyIiIiJFTslxDGWrdrK3ttHvMArerHV7AHh2zmafIxEREZFip+Q4igONLXzpibnc+Pgcv0MRERERkRxRchxFi3MArN9V43MkIiIiIpIrSo5FRERERDxKjkVEREREPFlPjs2sxMwWmtmr2d6WiIiIiEg6ctFy/G1gRQ62IyIiIlLQtv66zO8QCl73bK7czEYAHwXuBb6bzW2JiIiIBNUpd93S/vjX5dex6L2NEecbu8IxPEcxFausJsfAb4DvA/2jzWBm44HxAEOHDqWsrCypDdTU1CS9TCIONId6q2hpacnK+sNlax9yKZ19WLm1CYAd23fEXUe23qdiPwZ+KysrC3T8IiKJuvuUJyK/8EL4k41Rl//4E5dnMhyJIGvJsZl9DNjpnJtvZqXR5nPOTQAmAIwbN86VlkadNaKysjKSXSYRNQ3NMG0KJSUlWVl/uGztQy6lsw+755fD0sUMPXIopaVjI880eSJA1t6nYj8Gvgk7roGMX0TE8++bprc/jtbqK8GQzZbji4FPmNk1QG/gUDP7m3Pu81ncpgSZ8zsAERGRrsLrfB9/YkPkmXKQED9z3TxW3J31zRS9rCXHzrk7gTsBvJbj25UYi4iISD5pq/X93Avjos8ULSHOsRV3P+J3CEUh2zXHIokzvwMQKUxm9jjQVup2WoTXDfgtcA1QB3zJObcgt1FKOmbs3stT68spr69ndL++/PfxIzlzwKF+h5U3otb5Qqda38xpaa5n9fwJlK/+N62tzRx13Ic5+dxb6Nl7YErr+/JNx4bu4pKsy0ly7JwrA8pysS0REeniSeBh4Kkor18NjPZ+zgce9X5LALy2fSe/XbmR/2YIo20Ii6rq+O7CFfzf2JM5a+AAv8PLurZa33yq83XO8d7Eb9DaYpxwxg/pVtKTig0v8s7LX+AD179ISUnPpNc5/PulmQ9UIlLLsYhIgXPOzTCzUTFmuRZ4yjnngPfMbKCZHeWc25abCCVVzjkeW7uZ2+1ITrU+AFxpAyhpNSas3cyj4073OcLUJVTnCzmp9U3W3m3zqanczFmXPY51C6Vax59+G+/PuY2KdVMYeeLHk1qfWo1zS8mx5A/dkCfil+HAlrDn5d40Jcd5rq6lhd1NTYzp1rvD9LPtEB6v3e1TVLEdf9VZ9Dz3ovbnUWt986TONxWVu95n4JCz2xNjADNjwGHj2L9zedLJsVqNc0vJsfhOpcYivov0Zxjx62p43/RH9uqVzZgkAb1LSujdrRsVNDGcg5fq17sGjuqd/KX7dB1/1Vntj7+06VuRZ9rk/RSwQw4dQW31v7pMr6tex1FHXJzUusZeMErjDOeYkmPxnRqMRXxXDowMez4CqIg0Y3jf9GMO7ac/X5+VmPG5kcP4zZYd3OaGcqT1ZL2rZ4Lt4uujjsnKNo+/6qzYia8w9JjLWP7ur9m8+kmGH38DZiXs2DKJqr1LOPfqXyW1Lg36kXtKjuPQf/70rNxexXFD+tGze7f4M6sJWcQvrwDfNLPnCN2It1/1xsFx07EjaMFx25ZyXKujV7du/PdxI7n6yCOSXtcpd93CtP6fAWLU+SoBjqtbt+5cfO2TLJz+E+ZO/RRgDBgyhouvfZKevdSLSL5TcixZs7XyAFf95m0+f8HR3PPJ4N4UIhJ0ZvYsUAoMMbNy4KdADwDn3GPAJELduK0l1JXbTf5EKqnoZsb4447mplEjqG5uZkCPHpRY19aGtv58IUad7wsAwa31zSd9+h3JRR+fQFNjDc61ppQU60Y8fyg5jkONmanbV9sIwIJNlf4GIlLknHOfjfO6A26JNY/kr4TqfCFr/flKbD169kt5Wd2I5w8lx5I/VMMS1Q9eWMLumgb+/KVz/Q5FRHwSdSALlTkUJLUa+0fJsWSdct70/X3elvgziUggDZzwF5ZXhm6ei9mfrxQVtRr7R8mxZE2Ekrc4C2QlDBER3yRU5/tgK6rzFckfSo5FRERS0Fbru+c/vsNtD7ZGnkl1vpKC+2/tRuV4v6MoXkqO81xdYzPdzOjdo8TvUEREik7UOl84WOsbLTEWSVHl+Bv9DqGoKTnOc2N+MoVBh/Rg4U+uzNo21u+qoaXVMXpo/6xtIyEqThafzNu4l/e3VfHFC0f5HYrk0NZfl7U/Vq2v5Au1GvtPyXEA7Ktryur6P3j/WwBsvO+jWVl/qJeo6FRqLH677rFZAEqOC8TACX9pfxy13AFACbHkIbUa+0/JsfhODcYikqi2Ot8ZH3wwemuvyhwkoNRqnB+UHEv+UBOyiBAqd4g7dLFafaUAqdU4Pyg5lqwxZbsi0klCdb5KfEXER0qOJX+ovkIksMLrfJdXHqPEVyRJKqnIH0qOxXdqXxYJpm29jz3Y1VmHOl8lwCLJUklF/ujmdwAiIiIixez+W5WO5RMdDfGdqilERKSYqdU4vyg5lqyxZOslVF8hIiJF5pnr5vkdgnSi5LiIOOd4ds5m6pta/A5FREREJC8pOS4i01ft5M6XlnLfayv9DiUy1VeIiEiRWXH3I36HIJ0oOS4i1fXNAOyuafA5ko5UTSEiIsVIJRX5SclxETGvCFgNtCIiIv5Tq3F+UnJcRNpbaHOcHbs421OyLiIixUatxvlLyXERSbr3iFxvT/UVIiJSJNRqnL+UHEfh4jV3BpjL17baPA1LREQkk8ZeMMrvECSGrCXHZtbbzOaY2WIzW25mP8/WtiQx5jXN5lverwZjEREpJh9/4nK/Q5AYEkqOzayvmXXzHp9oZp8wsx5xFmsAPuicOxMYC1xlZhekFa2kpa3MId+SYxEREZF8kWjL8Qygt5kNB94AbgKejLWAC6nxnvbwfpSW+aithTbXZRXxtqeTQkREisWTxzzkdwgSR/cE5zPnXJ2ZfQX4nXPu12a2MO5CZiXAfOAE4BHn3OwI84wHxgMMHTqUsrKyhIMHqKmpSXqZRBxoDqVsLS0tWVl/uET2IRMxLN/u9XO8e3fE9aWzjUj7UF7dCkBtbW3Mda/c2gTAjh07cvI+RJKt8yjTYsUYlH2IpKysLC/i93v7IlL41k2Omz6JzxJOjs3sQuC/gK8kuqxzrgUYa2YDgZfN7DTn3LJO80wAJgCMGzfOlZaWJhhSSFlZGckuk4jq+iaY9jolJSVZWX+4mPsweSJARmKoX7YNFi3gsMOGUFo6LqPbiLQPq7ZXw8wZ9O3bl9LSD0Rc7tUlFfSorwLWMXToUEpLx0beQAbfh0iydR5lTAL7n/f7EEnYfvkaf5bPLxERCLUar1vhdxQST6LJ8XeAO4GXnXPLzew4YHqiG3HOVZpZGXAVsCzO7HmhMC/1598gIN98Rt+gRUSkOKjVOBgSqjl2zr3lnPuEc+5X3o15u51z34q1jJkd7rUYY2Z9gA8DK9MNWFKnG/IS09rqmLR0G62txftG7alpoKq+ye8wBDCzKzKwjqvMbJWZrTWzOyK8Xmpm+81skffzk3S3KSId3X+res8NikR7q3jGzA41s77A+8AqM/tenMWOAqab2RJgLjDVOfdqeuFKOvzqMi1oyfgL88v5xtML+NvsTX6H4ptz7pnGBb98w+8wJOTP6Szs3fvxCHA1MAb4rJmNiTDr2865sd7PL9LZpoh0ddhLv/E7BElQomUVY5xzVWb2X8Ak4AeEbrT732gLOOeWAGelH6JkXsCy1RzbWV0PwI6qep8j8VddY4vfIRQNM3sl2kvAYWmu/jxgrXNuvbet54BrCTV0iEiOqKQiOBJNjnt4/Rp/EnjYOddkZsqwAsbMn0FAEh5GOotN262tjlbn6F6iy1qSly4FPg/UdJpuhJLbdAwHtoQ9LwfOjzDfhWa2GKgAbnfOLY+0svAehvr0G5ZmaCLFQTfiBUuiyfEfgI3AYmCGmR0DVGUrKMmOg/0c51bCyXgWA7vxiTm8vWY3G+/7aPY2IpK694A659xbnV8ws1VprjvS187Of20LgGOcczVmdg3wT2B0pJWF9zA06IjT1UgikgC1GgdLojfkPeScG+6cu8Yb3GMTUNBjHwatTjYRCbfgBnR7sby9ZrffIUT00BtreHbOZr/DEJ855652zk2PUguc7s1x5cDIsOcjCLUOh2+/qm3QJufcJEJXC4ekuV0RQYN+BFGiN+QNMLMHzGye93M/0DfLsUmWuBxn/gX4PSNjHpi6mjtfWup3GJI//mFmP7CQPmb2O+D/pbnOucBoMzvWzHoCNwAdapzN7Ejz6q7M7DxCnw170tyuiKBW4yBKtADzcaAauN77qQKeyFZQkh3tXbn5G4aIRHc+oVbedwkltRXAxems0DnXDHwTmAKsAP7h9Vd/s5nd7M12HbDMqzl+CLjB5fpbtIhInki05vh459ynw57/3MwWZSEeySLDpxvycrs5kSBrAg4AfYDewAbnXGu6K/VKJSZ1mvZY2OOHgYfT3Y6IdHTXipv8DkFSkGjL8QEzu6TtiZldTOgfuASJTy3HQW1+2rinjlF3TGTOhr1+hyLFYy6h/63nApcQ6pP4BX9DEhEpLom2HN8MPGVmA7zn+4AbsxOSZEuuW3CD3mL87trQTXwvL9zKeccO9jkaKRJfcc7N8x5vB641sy/4GZCIpEatxsGVUHLsnFsMnGlmh3rPq8zsO8CSLMaWVXtrG+ndoxuH9IzyFgS1uTMBOb8hL/Cli0GPX4IiLDEOn/ZXP2IRESlWSY2I4HX309a/8XezEE/OnH33VK54YIbfYeSU5VPfajn0XIpdpRXr+yUiIulR923Bls5wYYHPHLZWFlfZdPsgIDkfIc/fU+X5+eVpLT9nw95AtH5X1rfywOurAhGriEghU/dtwZZOcpxXn8C/mbaat1bv8jsMiSDoydq6XbU8O2dL/Bl9NmFpAw+9uZYFmyv9DiXvPD9vCzPX5udAMCJSWL5807F+hyBpillzbGbVRE6CjVBXQ3njN9PWAGh44ATkqiE3X6oSMpGcb9xTm4FIsquxJfS7NeBfRrLhey+Ebo/Q/wcRybbh3y/1OwRJU8yWY+dcf+fcoRF++jvnEu3pItCUZqQu6DlaeG4fhNbveN9F9h9o4n+nrKS5Je1uc0VERApWOmUVBc0VcFocgDwvo/yuec4X9722gkemr2Pi0m1+hxLV5GXb2La/uO4FEJHCoe7bCoOS4zgKKa3KdY6YLzlpEFp9Myna7tY3hVqMm1vy8/1wznHz3xbwmcdm+R2KiIgUMSXHSXphfjnrd9X4HUZK2pKm7VX1vLlyR+62m8Iye2oa8mpkOudgX20jVfVNfocSVSpfRrbtP8DHf/cOu6obYs7X0NySYlTJK9+nlmMRCR61GhcOJcdJuv35xVz927fTWsf2/fVMXOLfpe21O2v48pNdxhrIgtSbjq97bBbX/yEzLYiZKqs46+6pnPWLqRlZV7548t2NLN26nxdidHc3f9NeTvrxZGZkoDeYslU7o24rP9uzRUSk2Cg5TkFDcyuj7pjI5j11Cc3f2NzKjqr69uc3TJjFLc8soLE5tzdG5b7MIfV0Z8PuzPUOkWpZRaT3q6U1uClcez/X4RMT2J05G/YBMHNd+l2hfemJudz+/OK01yMiIpItSo7jiJU7LNi8L6F13PHSEs7/5RvUN4UuTVdU1nvrDm6iVWyCdKSifRl4aeHWqMsk9MUpSG+CiEgOqaSisCg5jiKRxsZEk9upy0P1vQ05bin2X37ckZd6WUV+xJ+sTOew+XJjpYiISC4oOY4jI3lBsScXPrc4FltvFcmI98787JXlOYlDRCSo1GpceJQcZ5Bzrr10QoLb4rhye3WXaYWeX0c7VE++u7H9cbbfgky/xxt218bthUNERKQzJcdp6Pxh/ud3NnDyXZPZWV0fZYG2XwWeaXn8SCj//M4Gzrt3WlrreNXrSSRoyX14uC2tLmr3a8m2pOf6bTCDn/xrGaPumJjQ/FOWb+feie93mX75/5Vx/i/TOxdERGK5/1alUYVIRzUJ8ZKKVxZXALB13wEemb6WZ2ZvBgqvqqKqvol7J76f8942EnH3q++zM49aCw80tkS9mvDywujdp6XrG0/P56QfT447Xz6VnIRH8tSsTQkv97W/zuePb2+I+FqAOxcRkQCoHH+j3yFIFnT3O4B8F/7ZGi+PaHv9vtdWMtsbwOJz5x9NVX1zh/kMw/dC3DTcP2UVf5m1ieMP78cN5x0ddb6gtbzG0tiSWrnMKT+ZTN+eJSz/xVVdXrv179np0sw5mLI8uUFeEjlW6STS++uaGHBIj5SXFxHJN2MvGAUr/I5CskEtx1G0pQGtzvHHGeupbWjuks5GyxVmh43stqfmYCtmWzlF0MsqmrzmuOYEm+XS2dt8adn823ubU162trFrYv3e+j3phJOWXL+jry6p4MxfvM6iLZU53rKISPZ8/InL/Q5BskTJcRz1Ta3cO2kF/++1FSklak0t+ZHcZVLEwSQyINL6/MyNs9nwfcOE97K49sTl4v2duTb0RWDZ1v3Z35iISA6MvWCU3yFIFmUtOTazkWY23cxWmNlyM/t2traVC1UHurYcdxapRTjS5WrzqQo549tNMLMKUnVFtJvYCl2sc6PtHI52uGsbmlkco1W4kMprRERArcaFLpstx83Abc65U4ALgFvMbEwWt5d1nZODZBvd2pb3q6wime02tbTy0oLyiK3l7clSwttNTKQcKtfvVJ5UcSSsqaWVW55ZwOodB7ufy/T51ZY4N7VEvgHzf55dyLWPzKS6vimt7QTsrReRIqVW48KXtRvynHPbgG3e42ozWwEMB7r2uRQQmU468jkRe7RsHQ9MXU1JN+PascM7vJZoC3QmGgxDyblPLe0BaPF8v6KKiUu2sWVv3cF441/iSMlfZm1i7NED+dRZIzpMb6slbmhupX8GNpvp21UnLd2WwbWJSLFTq3Hhy0nNsZmNAs4CZmdj/d97Pjt3/YdzpJbMRsqvglBW0dZX8/4D0VsDc5Xch9/UKJGleiySXWzysu1dpsU7q5I92+PFtGZHNfM37Ut4fd94ekGSEYiISDHLelduZtYPeBH4jnOuKsLr44HxAEOHDqWsrCyp9dfU1PD8/Nr258kuH23+qsaOH9E7d+xgxowZHaatWLGCsuq17c+rqw90Wc+7777b/vidmTPp39NoaQ1dnp4xYwY9S4yampq4cb/y+nTufLuO743rzagBJTHnjWb57o71tJ23Gf68YmsoIV29eg1lDRs7zFdR4b22Zg1ljaHXIu3DtprQftbV1SV0XLbv2NE1prfe4olljVFjjiZ8vqqqAxGnh2uLvzHsBsqGhsaI8yZ7jiWyTCr7BbBhf+iYVldX08NaAGPR4sVR5wdYuWolZbXrACjfEjqW69ato6w1co8c69ata3+8e/fuLutsbAq9TzNnvsuAXl1T4bbzZc3q1VHjKisro6amFrAO2XGk+L80OfT3/uRVfbusI55480yfPh0LwiUDkQiq962ndv9mDj3sRA7pP8zvcApSst23Nbc6Fu+votk5xg44lF4l6gchCLKaHJtZD0KJ8dPOuZcizeOcmwBMABg3bpwrLS1NahuhD7uDyXHCy0+eGHP+PTUN8ObB0bWOGDqUSy89A6YeHFjh5JNPpnTcyPbn/Ze+DVUd8/+LLr4Iyt4A4OKLL2Zw3550mzqJlhbHZZddRu8eJZSVlUWP24vTHXEitU2LmF83mC9de1Zi+9hJyZpdMG9O+/P2bUZ4L6ZVLoUtmznxxNGUXjiqw3qm718Gmzcx+oQTKL34WICI+7BuVw288xY76hyPre7Fc+MvjLp/EPpyVFo6tsP0yy77ABN3LYGKrV1ijCjCvjz0/kyorIy5/P1/n8buxsH89OOnth/jXr16QkPXVuukztFo59nkjqO/pbJfAIPLK2HWTPr170fLgRqglTPPPBPmzu46v7eOxdV9uf2i8zikZ3fernkfNm3ghBOOp/TS4yLGd8IJx8Oq0KfBkCFDKC0dx9/nbqakWzeuO2cEvd6ZSnVjIxdddBGH9+/VJfSp+0Ln0ujRo+H95R3jCtuv19+YDoTKQ9pawjvvb1NLK0x+Leo6Ir1n4eL9nZWWlhZkcmxmVwG/BUqAPznn7uv0unmvXwPUAV9yzqnJPSCaG2uZ+/p32bdjKX0PPZ6aypUMO/5Kziz9Od26aTiDTEqmpGLx/iruWLKKAa4bPTAqXBN3jTmB0sMPy2KEkgnZ7K3CgD8DK5xzD2RrO/G8t34PLRkYJss5F7fmONFL2+mWVTS1tCZ089P8TXsZdcdEFmxO/BJ0Z7EiTeZdfW/93vgzeZ6bk3qfwqn63cIGnp2zhV1ho+v5Vf6SjFRinL9pH3e/Gir9T7Yco23+H7y4lNsTLGfKZK65oyrK0OwSlZmVAI8AVwNjgM9GuDn6amC09zMeeDSnQUpals68j9aWnpzzwWcZc96vOOeDz7J/1zrWLvyz36EVlC/fdGzC89Y1t3DbohV8vWUID3I0v2Ykd3EUP1++hm31KhXMd9ls378Y+ALwQTNb5P1ck8XtdfHu2t3cMOE9fj99bfyZ45i2YkcGeqvIzCAgX/vrfE7/2evtzxubW/nrrI1dvgSUrdoFwDtrdgNdE6lUB9jIVsta21qfnn0wOc51zx6Z+CIVBLuyMMR2Lo5Vtuvc8/km2TScB6x1zq13zjUCzwHXdprnWuApF/IeMNDMjsp1oJK81pYmylf/m1Gn3Ey3bqFRKEu6H8LRJ49n0/sv+BxdYbnszVsTnrds9x5OtN6c261f+7STrA+XWn9e274zG+FJBmWzt4p38LmL2+1eK9O6XTVpr6u+qTWlj/7OCWl48pXqB/GbKzv+YT32VqhniZ7du/Gf50YfzjmeHVX1DD20d8KxJTqcdjpSvtHMuby6PN7a6qhpbObQ3tkZQjlXSV3kzSTWt19h5p2BMBzYEva8HDg/gXmG4/U4FC78PpE+/VTX6rfW1iaca6FHj459xfTsNZimxi63+Uga1k1emPC8Nc0tDHIlXbKgQa6EqqbmDEcmmVbQleGZzo3itbTGS1AmLt3G8T+clPFR8yrrQiUW1fXp/cFFbDWN8SbmW7Lz2FsHbxzLRO8NmTx/jvvhJM742evUNGT2n2K6MWaitTdeDImWfuTD+ZQPMWRBIl2IJ9zNuHNugnNunHNuXK8+g9MOTtLTvcchHDr4JHZve6vD9J3lUzh8xMU+RVV4kimpADh30ABmu1pq3MEb4ZtcK+9YDRceNijT4UmGFUWlfiofeBGHMk5zxdnqbzVagpNIy24qyVW2GmQTfTunr9zJK4srePA/x3aYft9rKzMeU6bV1DfTr1fm/+zCj3Uqrf6ZaGWPt9nWNEpWWlpd1EFIJK5yYGTY8xFARQrzSJ4647IfMevVr1FXvZ5+A06ics989u18l0s/9YzfoRWM4d8vTWr+Y/sewtVHHc73t5fzidYB9MB4zao4bmBfzh00IDtBSsYUdstxhqs6UurnOEYIGR/JLM0E55Eka7NTrVmOu96w9yXSJm56ci4vL9waZx3BtKu6gVF3TOTtNbuSXjZbX1rCz6tIxyNuP8feDD/7d5zxf2IctC89MYeT75ocfYY0pDuyXwDMBUab2bFm1hO4AXil0zyvAF+0kAuA/d5AThIAg488i9LPvECvvj3Zt7uMAUNGcvn1/6TfwGP8Dq0gJNtq3Oa7Jx7Lracex9rBrSwZ1MQXThrJfWecTLc8KvmTyIqj5ThTmVKn9eysrufVJRV87IzCqLt7evZm7v3U6R2m7a3p2tdvtntxyEytsn8j66VjsTfa3JMzN3Lp6MNTWkcyX7qSf6+jL5DNuue3vZtKsyG8hj+o500szrlmM/smMIVQV26PO+eWm9nN3uuPAZMIdeO2llBXbjf5Fa+kpu+Aozn9kjv9DqMgJdtq3MbMuGzIYC4bovKjoCmK5DgZe2oaGNy3Z8TXOicd//d6aFCDD558BIf0jPxWTu/wwZuhIFPU+ctqrHDaXntw2mq+/eHRMdcTfy2JiVz02HEdibZWp/pWh68/GylStnp0SDelS2vZDL1RQW3tDwLn3CRCCXD4tMfCHjvgllzHJZLvvnzTsfB9v6OQXCvosopkbdpTyzn3TOPP72xIarlY3X9974Ul7Y/9/vBPJjnPRG8VyXpp4VYONLYkXT8bj9/veyJemF+e0nJtiWmyJS7JzB2e+2ZqCPWY8/t4yTEI54qI5E6qrcYSbAWdHH/n74uSmn/z3joApq/aGTEJSDtR69xPcl5/EkcPztrnyPwObNhdG3+mTpxzzNmwt9O01Lafzh6NumMi//NsqKufttrhRN3+/GL21jamtf324+LTeRV+PuysrmdfbeTht0VERPJZQSfHbRLNFeLV0rZGyTp2ZmFAhWSk0xdwKrLZsNd5HMLOEZbvO9BlmZcXbuX6P8zqMG3z3rr23g1y2Q7578WhG/zf35Z8/6LNrbnrjaHzsU/nmEb6uznv3jc46+6pycWUeggZk99fWEUkl+6/tShSJIlARz4J0aonPnR/qH9Jvz9XE+01IJ47X1rKs3O2RH297bJ3LhKJzknc8oquSefGPXVdpn34gbf4hdc7QjJhhm8uncv70ZbM9M2MuR7iuvN7GX58op0P+TQYS7jFWyoZdcdEtu/XkNQi0lXl+Bv9DkF8UpA35P3838t5ffmO9uedE6x5Gztegi/fV8eIQYeEzR95vel2Xdbl5rK01pbC9hPc4LNzNsd8/WBZRWa213W5RG4VjG/m2uz1cJCqn76yjJOG9o8/YxqS+zKQ3ln4yuKKg3XPaa0p9/763iYA3snD80RE/HX/rd2oHO93FOKXgmw5fmLmRrZWdr383ua6xw5egn9z5Q4u+dV0pizfHrdlNY0xDIDEksV/LdrKpj3J192mIu0S6jgr+N8pq9LcQqSWyuTXEeuwOueyUhsb7VyasnwHD72ZXH/SqWwn6fUkuI3OyfTOqobAd3zW4QbQwKX4IpINajUubgWZHHcW6+Puy0/OA2BJeWXMXicges1x+3bivR7z1ZBvP7eIa377dgJzdhV3GN9MZTHtLYWx9+j193fEfD3S+7WzqqHTPJ2WiR9d1+2EPZ6xuuPgGr8vW9epNrY4kiMHvLSgnBkJ9B8c67TO04oJEZGUqdZYdAZ4Hpm+ji8+Pqf9eaTEL15yHE/nZLDt+SOL6jv0bFDb2EI+y2ad623PL+44IY0ePjbvreOVxR1HwP3i43OYG1ZWM21F9AQ+2tWH1Tuq2V8Xe1S1dN+jN8L6x06Ec3CgObltfPcfixPqHSTRtzztsiOfvpd0bBn3JwYREckfSo6TkK0Pzrnb4yfDM1Znvi4y2f1pamll7C9eb++RwTlYt6smo8PvNjRl7otBc6vjW88u7JKm7qmJ3rtIIu/JlQ/O4D8enZlecBnSlthV1Texfn+ot4vOSWpTSyufePid1LcR9jjS2xPvhrtMti5/9al5mVuZiEgEKqmQokqO/zprI6u2V8edL1qClG7LcTpeXJDaIBHhGprTSzwr65qorGvq0KL6ofvf4r/+NDul9SU00EiXmxiTPwbZOGrrdsVucU0pIewU6HNzNsdtjd3r1Uvvq4teN11ReYAl5fs7bqrTamMluLEiuGfiCvYfaIq4zmS1LR6rvGllp7/fdFurRUTCqaRCoEB7q+jC+/y861/LE5p91vo9PFq2rsv0tG/Ii/M827793KIOz/cfaIo6VHYi2hKzJeX74bS+6YTWQawR8jIyYl6MdWTimKzeEf8LWCLueGkp/Xp352NnDIs6z63eQDf1TdH7SE47aY3TXVtNQ8d6jlwO/vF8iiMLhlN+LSJt1GosUGQtx8l4YubGLtPSrzlOa/G0Vdd3TGLOvXcav568Mu5yU5ZvB2DLvo79CSc7zHaiwluHZ67b3WH0u0y8hWt31rCkvDIDa4rsygdnRDx/UtH5mHVWGaH2Odun2eItlVFfa25p7XCD49Ly/VmtUV+7syZr6xaR4qJWY2lTcGdCfYSa1Ux1z5R+P8cdRYo1G2KlJq8u2RZ3+a/9dT4AX3lyboYiCknk3fzmMwuj3hj3XJz+mNu302lD909dzScejlwznKkvMMu27o8/UwZ0y1BBbzKrmb4q8s2CznUdLfLjadQ6Z1vbsdYNeSICajWWgwouOT75rslZW3e6ZRWdPfD66oysJ5GkPRO1mQcymMy3tDoqoiS9Mcsewl6846WlCW3Lj7rUbHVx9vlO9d2JbCfS3nee9nCCfS/Heye7d8tt327NLQcjam11OesjXEQKy9gLRvkdguSRgkuOI8lUbpT2DXmdlo90STyl1Xq/Y6Ulu2L00JDwdjKYYz44dTWX/np6l+m1jS2syfCl8kx/qcmmeIl8QqO5pbC/O6sbaGyOXrecqG4RkuNEvyikcn5t2H3wXPl92Vo+8L9ladV8axAQkeL08Scu9zsEySNFkRxnSmuc3CHeh3u2P3ZjdqmVpSGxU/V2ikP2phLu0iRKHDKVHG3bX5+R9cSTajttpOMe7fTpeIOkiz68Oo6SLDSZz+003Hu48HN+tlebHmt0TBERkXiKIjnORctx+CAeqerc4pWJlrxkXZvHNaK5ELSa03h9DFfXN3H5/5UltK509z2ZLyHJ+EzYcO8JCdgxFBF/PXOd+k+XjooiOc6U2oYkhyDrpEtfs50+xa98cEaH5+t2Zaa84Nk5mznvl28kNO/i8ugJTiZyjgPe6H8pt3h2imJp+f60srp8T4bjxRfvfVy4uTJTocT1zWcWRh4kJGcRhKTT+p/v54OIZN6Kux/xOwTJM0qOk3DPxBUZXV+8Lq7iXaFOtCSiIsYl/mRygXRbsl9buo1TfjKZ15ZuS7mc44GpHW9izHRvCBsTGE65s/mb9mbkRrBo78jkZdujLxThHAlPDjOd662PM/hJJPla89329/X2msyPPikiwaBWY4mkOAYByZBd1enf1BYuXgtXIv3Dlq3ayVOzNqUeQw4Tl7YuwL7+9IKU17Flb3brSVOJ7dOPJnnZPwnbq+r54cvRe+XIZKtstPMxfPrWygPMjFEvnk4Ne6ZOxXTO6TzN40UkS9RqLJEURctxru5AT3Y78T7E4/WK5RzcmWB3Zvkg3y5Zb9pTy6IYA1rkg3it9fFqjpNNVuubWrq0nv9yUseBYtbGKPeJ3G1c9g58pP3Lt/NMRPKTBv2QaIqi5XjK8h0ZuWEu0x/ysfKabfsPpN2lWSLxFnPXVdm6gSxVzRHqD+JeXYhUVhFj+O1YnINvP7eQKct3JL5QBuX6TFQSLVLcjtoyh0q/g5C8pK9NScjlh+mF/+9NvhHnEr+ja0zV9U388OWl1DWmd/Og5F7EQWGydM5FO5dnrI5ffxurf+5I683m8NHhLefJ/n1W1UcYelsZs0jRUEmFRKPkOAmZ/tjMxufw78vW8czszfzl3cTqkBON4bMT3ksqjkemJzbimp/yLQ9avzuxKwXJ1L4ne2Ug3W6KmyN0Bp7O1Ylkrvi0DZASa2utrY5n52xm8ZZKpr7vTwu5iPhPN+JJLFlLjs3scTPbaWbLsrWNXIvXqlTflNt+iTvHY3awL+ZEE5JE05ZZ6/ckExr/O2VVUvMHVSZbGhPtdu3ce6cxa13oeETKZdMpq0hX5+4Ko613wox1XQf3yNBbGeuYvDC/nDtfWsotz0S+KpNn35dEJEvUaiyxZLPl+EngqiyuP+fiJQ+b99blJhBP53CM5C9hZ7oHjiDJRCL0p7c3ZGAt0UWLcXlFYvXSSSXHuLQLICKWVURY6S8nrUx+cI8IauqTKx/afyBUSlG+T6PoiRSrJ495yO8QJM9lLTl2zs0Aoo/7KilJNHnJt5KBQvVOisNgZ1NdU0v742inQfZuxEy994hUIpoTYWjpWOuJt9/6uxEpfOsmL/Q7BMlzvvdWYWbjgfEAQ4cOpaysLKnla2pqyNUYXHtqGzO6vl27dzN9+vSkltmx42Cd5FtvvUVDw8GW39Vr1rD3QOjTfcP69dS3dFk8a2pqauIeu23bYwxmkYaNm1Lr5/lbz6b3D/JLD09h94HMltIsWNKxCmnLli0R51u7dh1lLZtpaup6U1n4fi1dGrmrv3379nWZNmPG27S0pHfSLF22vMu08vLyuMuVlZVRW1tLMn/L0c63ZcuW0WvXyoivrVu3PuY633nnHfr2yPWYftllZoOBvwOjgI3A9c65LieAmW0EqoEWoNk5Ny53UYqI5A/fk2Pn3ARgAsC4ceNcaWlpUsuHPiDTH53MDwt3tsCwU4G5CS9zxNChsL0CgMsu+wA933sTvAT5xNGjQ6PhbVjHsccdF+qxYv26bITeRb9+/ehw7CZ3vZHqyCOPhK3xE6VkjTrmGFiX+xsAy8qbOaxvTyBzX5pOOeUUWLyo/fnIkSNhQ9eE7oQTjqf00uPoMeN1iJAgtznttNNg4fwu0wcOHAR7OtaRX3LppXR/+w1oSb2nkykVPYCOpTojRoyAzRtjLldaWsq/pkwHEi9NKi0tjXienXbaaZSeemTE1w4fdjSsin6uXHLJJQzo0yPhGALiDuAN59x9ZnaH9/wHUea93DmXf5dDRDLkyWMeYl1mB7uVAqTeKny2L43WaF0iLjxbOtWtv7q4Iub8+2J0qwbJlyqk22a6NsW+uUfdMZHapmQH0Ykyol86531h/s1cC/zFe/wX4JP+hSLiL5VUSCKUHAfY6xEGa3i07GBLsR+f8+X76tgbJeFXsh7f/3Xq67hif33Ued+vqIq7vgONiZdJZKuP30TXu2x3ciUdz8+PdhUi+vbS7aouoIY657YBeL+PiDKfA143s/leuVtEZjbezOaZ2byGA7qtRIJDN+JJorLZlduzwCzgJDMrN7OvZGtbQTZ/U9faz0T9T5ya2Vwmo08tb6ChuYVLfjWds++emrsN+yzTdeiJumfiipjDOLf5zt8XJbzOfy2qoLoh84PH/GVWYjXhyeatO2J8cUh1G0EdMdLMppnZsgg/1yaxmoudc2cDVwO3mNllkWZyzk1wzo1zzo3r1WdwRuIXyQW1GkuislZz7Jz7bLbWXUienr05Y+vq/LH+XpJ9E6fjzS3NvLp4W862J+nfUNjZj//pb5fk71Qkl5hHawUuxu4JnXMfjvaame0ws6Occ9vM7ChgZ5R1VHi/d5rZy8B5wIysBCySY6o1lmSorCJgYrV8/X56x5vvFm2pzGosncVrc5u9ITvJeuWB2HW30tW763L3xSlRm6qS6/kj2jDWd/1rOc0tkdcVzHbhtL0C3Og9vhH4V+cZzKyvmfVvewxcCRTMAE4iajWWZCg5DpjOH+5NLQenbK86eJk5W/WjscS7ZJ2tgReeSvCyvRSWWGUlJ/zotYjT//h27K7cCrQu/j7gCjNbA1zhPcfMhpnZJG+eocA7ZrYYmANMdM5N9iVakQz78k3H+h2CBIzvXblJehqbcztkdSxFerOT+OTtNcn3OBZviPfHZ27gtitPSjWkvOSc2wN8KML0CuAa7/F64MwchyaSE8O/X+p3CBIwajkOmH936tqrNY+auv65KHa3YyKZ1NKa+XP/d2/mvr9sERHJL0qOAy6fkuMZq3f5HYJI2h56Y43fIYhIhty14ia/Q5AAUnIccNloPRMpZg9MXR1/JhERKViBT47Lq/On5tYP4TfkhcujBmUREZGcU6uxpCrwyfF9c7LTA4KIiIiIFJ/AJ8d51FlDXlm/u9bvEEREREQCJ/DJsXLjyF5euNXvEERERHyhkgpJR+CTY9XWioiIiEimBD45jnI/moiIiBQhtRpLugKfHKsnMxERERHJlMAnxyIiIiIA99+qtEbSp7NIRERECkLl+Bv9DkEKgJJjERERERGPkmMREREJPN2IJ5kS6OR4+qqdfocgIiIiIgUk0MlxfWOL3yGIiIiIz9RqLJkU6OS4R0mgwxcREZE0jb1glN8hSIEJdHZZUmJ+h5CQ4w/vm/Ky3/vISfzxi+MyGE1IN4ORg/tkfL1BdtPFo2K+fnj/XhnZzlcuOZZThx1K354lGVlfIv72lfNztq1MG3/ZcXyj9Hi/wxARkSIR6OQ4PDU+akDvlNfz8jcuipso/u6zZ6W8/uGDDuGi4w+L+NqVY4ZyZN/ISf4TN53LLZefwBVjhsZc/4tfv5AXv35Rl+kXHDe4y7Q7rj4ZgDNHDuSVWy6JF3pEk79zaULzvfj1C7tM+/TZIzo8H31Ev6jLd37PXv5G130Md+qwQxOKq82ZIwZ0eP71TgnYjz96SvvjZ796AZO+dSl3X3tq+7RUzrmhh/biro+NYeK3LuW3N0Q/p+70jlOywvfp2a9ewGOfP5s1917NJaOHsPG+jzLwkB4JreeB689MafuxfOtDo1Na7ofXnMLwQfG/yJ05cmDM1x/8zzM5I+z9+dz5R6cUj4jkl48/cbnfIUiB6e53AOlw3uh4l590OIf168UL88sB+L/PnMkzszexYHMlf/ziOCrrGvneC0val/vIqUO591On89KCcq4YcyTHDunLH784jqt+8zYA0757GYf36815v5zG4186lwuOO4ySbsbRgw/hnonvM3fjPgAuHT2E73z4RAA+/ei7AAw8pAfHDenLgs2VQKjV66aLR9G9Wzf+uXAryyr2M/qIfiyvqOKi4w/jCxeO4h+T3uT7Mw4w/rLjmDBjPcMG9ObvX7uQkYMPaY/59OEDWLp1Pxvv+ygPTl3Nb99YA8C/v3kJp3sf+M+Nv4AbJrzXvszT/30Byyv2s7yiis+cM4KSbsaOqgbue20lHzr5CAb17cm6X17Dj15eynNzt0R8j3993RlceNxh9O5RQr9e3WlsaWVAnx6s/+U1rN5ZzfpdtXzj6QXt8w/p15PdNY28e8cHGTawD3dfeyrduhkPTl3NH75wDmcfPYjbrjyRt9fs4gcvLuXiE4bw/M0X8slHZrJxT137er566bH86KNj2LSnls1766htaOasowfxs4+P4Wf/fh+AHiXGZ8aN5JnZm/nhNScz/rLjWbW9mhcXlDNhxvr2dd144TH8ZdYmAO755GlcdPxh9OvVnUP79KCmoZlnZ2/m0NrNHNG/NzO+dzmzN+xhR1U9X7nkWD52xjBWbKviQi9R//wFx3D6iIH07VnCoX168MqiCu6dtIK3vlfK8IF9OOFHrwGhVvktew/wg6tO5sSh/Rg5+BCmr9zJf11wTHtcHx4zlKm3Xsa8TfsYdEhPzhgxgM//aTbXjh3O1z5wPF/7wPGMumMiAI99/hy27T/Az719f+zz53Dz3+YDcN6owWzaW8uOqgb+eOM4zv/lGzhHe8zhym4vZfPeOj7x8Mz2aZeOHsKYow7ljZU7WbuzhivGDOU/zh7Bd/+xuH2eK8cM5TPjRvLVp+aFYj9lKNX1TczesJflP/8IK7dX8Yt/v8+YYYdy2ejD+e0bazisX0++eOEovvbXUJzfveJEHvLOW4A5P/oQ1z82i4176nj+5gv5zGOz2l+7+rQjeW3Zdl64OfQF6xNnDuOvszaxcnu1d571YndNAxBqia+oPMCjnz8HgLkb9/La0u1cffqR7es8vH8vSk88gk+OHc7Jd03mP84ezi8/dTrvrdvD+t213HbFifTv3Z2f/ft95v7ow13eNxHJT2MvGAUr/I5CCo05lz/jL48bN87Nmzcv4fkPNLbwn7+bym++eAmH9e3FhLfX8d0rTqKkW9eW2J3V9fTqXkKPEuOQnpG/E7w4v5wPnzKUAXFa15ZX7GfznjquPv2oqPN8+7mFrNpezeTvXBZ3P8rKyigtLQ3FWVVPn54l9O/dMYbahmb2H2hi2MA+7K9r4sf/WsY9154WMdYte+uoa2zhpCP7R9xeZV0jh/buQTfvfWo7B8wOvm/7DzRR0s3o1yv+96fxT82jqXoPf/r6lTS1tNLc6uIu19zSykNvrmX8ZcfRr1d36hqb2b6/nl49Shg+MHYr4bpdNQwb0Ic+PUtobXW8sriCj51xFN3DatCbW1rZtLeO4w8PtUzvrKqnsaWVEYMOibjO8GOQjobmFkrM+PbfFzFxyTbm/OhDHNE/9asabcnxxvs+2uX5dY++y5kjB3LXx8bgnKOsrIzLL7+c9btqWFK+n0+eNTzqeudt3Mvh/XsxbGCfqLX7K7ZVcf/rq5m2Ygfrf3lN+/nSprq+iS17DzAmTov9V56cyxsrd7Lxvo+ytfIAfbwvWj27d8M5R01DM/179+Da+yfz1SvO4JrTjqK51bG7poFhYedCXWMzY34yhQ+efAS3XXki97++mkc/fza9uqdenvLqkgq++cxCym4vZdSQ1MufAMxsvnMu8zVQeWzQEae70utf8jsMKWK6EU9Sdc6bM6P+zw50cgyZS2r8pH3wX6bjP9DYwvvbqjjnmEFpref15dtxwEdOPRKAF+aXM3JQH84/LkKrcJ4eg5ZWR0uro2f32FVcicS/vGI/xw7pG/ULbiqccx2+GKZKybFIbo29YJRKKiRlsZLjQJdViOSrPj1L0k6MAa70kuI2150zIsqc+aukm0W8mpOKU4cNiD9TkjKRGItI7n1/xAuqqJCsCPQNeSIiIlKcVtz9iN8hSIFSciwiIiKBor6NJZuUHIuIiEigqNZYsimrybGZXWVmq8xsrZndkc1tiYiISOFTq7FkW9aSYzMrAR4BrgbGAJ81szHZ2p6IiIgUPrUaS7Zls+X4PGCtc269c64ReA64NovbExGRTszsM2a23MxazSxqV3O60idB8OWbjvU7BCkC2ezKbTgQPuxaOXB+55nMbDwwHmDo0KGUlZUltZGampqkl8k32gf/BT1+CP4+BD3+PLYM+A/gD9FmCLvSdwWh/9VzzewV59z7uQlRJDHDv1/qdwhSBLKZHEfqPLTLiCPOuQnABAgNApLsIAb5OvBBMrQP/gt6/BD8fQh6/PnKObcC4vbn3H6lz5u37UqfkmMRKTrZTI7LgZFhz0cAFbEWmD9//m4z25TkdoYAu5NcJt9oH/wX9Pgh+PsQ9PiP8TuANCR0pQ86Xu0DGv75yInLshxbrgT9/GtTKPsBEfbln/7EkQkFfVwCKur/7Gwmx3OB0WZ2LLAVuAH4XKwFnHOHJ7sRM5sX9CFbtQ/+C3r8EPx9CHr8fjKzacCREV76kXPuX4msIsK0Llf6oOPVvkI6ZoWyL4WyH6B9yVeFtC/RZC05ds41m9k3gSlACfC4c255trYnIlKsnHMfTnMVSV/pExEpVNlsOcY5NwmYlM1tiIhI2pK+0iciUqgKYYS8CX4HkAHaB/8FPX4I/j4EPf68ZGafMrNy4EJgoplN8aYPM7NJELrSB7Rd6VsB/CPBK32FdMwKZV8KZT9A+5KvCmlfIjLnIpaViYiIiIgUnUJoORYRERERyQglxyIiIiIinkAnx0EZ7tTMNprZUjNbZGbzvGmDzWyqma3xfg8Km/9Ob59WmdlHfIr5cTPbaWbLwqYlHbOZnePt+1oze8jijESQg334mZlt9Y7FIjO7Jl/3wcxGmtl0M1vhDf/7bW96II5DjPgDcwykIyugoahj/R11mq/L/+98EO89tpCHvNeXmNnZfsSZiAT2pdTM9of9z/iJH3HGE+kzp9PrQTom8fYlEMckZc65QP4Q6h5uHXAc0BNYDIzxO64osW4EhnSa9mvgDu/xHcCvvMdjvH3pBRzr7WOJDzFfBpwNLEsnZmAOoRuBDHgNuNrnffgZcHuEefNuH4CjgLO9x/2B1V6cgTgOMeIPzDHQT5djdApwElAGjIsyTyD+N0f7O4owX5f/337/JPIeA9d4fysGXADM9jvuNPalFHjV71gT2JcunzlBPCYJ7ksgjkmqP0FuOW4f7tQ51wi0DXcaFNcCf/Ee/wX4ZNj055xzDc65DcBaQvuaU865GcDeTpOTitnMjgIOdc7NcqG/pqfClsm6KPsQTd7tg3Num3Nugfe4mlAvAsMJyHGIEX80eRW/dOWcW+GcWxVntqD8b472dxQEibzH1wJPuZD3gIHe31K+Ccr5ElcCnzlBOSbJfn4WnCAnx5GGO431wesnB7xuZvMtNPQqwFDn3DYIJRHAEd70fN6vZGMe7j3uPN1v3/QuaT0edik1r/fBzEYBZwGzCeBx6BQ/BPAYSMLy+X9YuGh/R51F+v/tt0Te46Ach0TjvNDMFpvZa2Z2am5Cy7igHJNEFcIxiSirg4BkWcLDneaBi51zFWZ2BDDVzFbGmDdI+9UmWsz5uC+PAnd7cdwN3A98mTzeBzPrB7wIfMc5VxWj3DYv9yFC/IE7BsXEcjgUdbbF2pckVtPl/7fXquanRN7jvDkOcSQS5wLgGOdcjXePwj+B0dkOLAuCckwSUSjHJKIgJ8eBGe7UOVfh/d5pZi8Tuoy0w8yOcs5t8y6r7PRmz+f9Sjbmcu9x5+m+cc7taHtsZn8EXvWe5uU+mFkPQonl0865l7zJgTkOkeIP2jEoNq6AhqKOtS9mFu3vqPM6Iv3/9js5TuQ9zpvjEEfcOJ1zVWGPJ5nZ781siHNud45izJSgHJO4CuiYRBTksor24U7NrCeh4U5f8TmmLsysr5n1b3sMXAksIxTrjd5sNwJtLTKvADeYWS8LDeU6mtDNSPkgqZi9S5XVZnaBhZo7vxi2jC861Xd9itCxgDzcB297fwZWOOceCHspEMchWvxBOgaSkkD8byb631G7GP+//ZbIe/wK8EWvh4QLgP1tZSR5Ju6+mNmR3t8+ZnYeodxlT84jTV9QjklcBXRMIvPrTsBM/BC683M1oTtdf+R3PFFiPI7Q3beLgeVtcQKHAW8Aa7zfg8OW+ZG3T6vw6a584FlgG9BE6NvuV1KJGRhH6MNkHfAw3qiMPu7DX4GlwBJC/6iOytd9AC4hdMltCbDI+7kmKMchRvyBOQb66XJMP+X9LTUAO4Ap3vRhwKSw+YLwvzni31H4vhDl/3c+/ER6j4GbgZu9xwY84r2+lCi9i+TDTwL78k3v/V8MvAdc5HfMUfYj0mdOUI9JvH0JxDFJ9UfDR4uIiIiIeIJcViEiIiIiklFKjkVEREREPEqORUREREQ8So5FRERERDxKjkVEREREPEqOpSCYWY33e5SZfc7veEREJD1mdqOZrfF+boy/hEhmKDmWQjMKSCo5NrOS7IQiIiKpMLPBwE+B8wmNSvhTMxvkb1RSLJQcS6G5D7jUzBaZ2a1mVmJm/2tmc81siZl9DcDMSs1supk9Q6gzdhER8YGZnev9f+7tjUq4HLgFmOqc2+uc2wdMBa7yN1IpFt39DkAkw+4AbnfOfQzAzMYTGqLzXDPrBcw0s9e9ec8DTnPObfApVhGRouecm2tmrwD3AH2AvxEamW1L2GzlwHAfwpMipORYCt2VwBlmdp33fAAwGmgE5igxFhHJC78A5gL1wLeA70aYR0P6Sk4oOZZCZ8D/OOemdJhoVgrU+hGQiIh0MRjoB/QAehNqKS4Ne30EUJbzqKQoqeZYCk010D/s+RTg62bWA8DMTjSzvr5EJiIi0UwA7gKeBn5F6H/3lWY2yLsR70pvmkjWqeVYCs0SoNnMFgNPAr8l1IPFAjMzYBfwyc4LmdmfgMecc/NyFqmIiGBmXwSanXPPeL0HvQuMBe4mVGoB8Avn3F6fQpQiY86phEdEREREBFRWISIiIiLSTsmxiIiIiIhHybGIiIiIiEfJsYiIiIiIR8mxiIiIiIhHybGIiIiIiEfJsYiIiIiI5/8Dlh6tq9wvRPoAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 864x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Define network dimensions\n",
        "in_features, hidden_dim, out_features = 2, 2, 2\n",
        "\n",
        "# Define and initialize layers\n",
        "layers = [Linear(in_features, hidden_dim),\n",
        "          ReLU(),\n",
        "          Linear(hidden_dim, out_features)]\n",
        "\n",
        "# Initialize network\n",
        "net = Net(layers)\n",
        "\n",
        "# Define list to store loss of each iteration\n",
        "losses = []\n",
        "# Initial accuracy is zero\n",
        "acc = 0\n",
        "\n",
        "while acc < 1:\n",
        "    # Randomize network weights\n",
        "    net.reset_params(std=0.5)\n",
        "\n",
        "    ########################################################################\n",
        "    #     TODO: Perform forward pass with x_xor as input and y_pred as     #\n",
        "    #                          output variables.                           #\n",
        "    ########################################################################\n",
        "\n",
        "    y_pred = net.forward(x_xor)\n",
        "\n",
        "    ########################################################################\n",
        "    #                         END OF YOUR CODE                             #\n",
        "    ########################################################################\n",
        "    \n",
        "\n",
        "    ########################################################################\n",
        "    #    TODO: Calculate MSE loss between prediction and labels (y_xor)    #\n",
        "    #                         and append to list.                          #\n",
        "    ########################################################################\n",
        "\n",
        "    loss = MSELoss(y_xor, y_pred)\n",
        "    losses.append(loss)\n",
        "\n",
        "    ########################################################################\n",
        "    #                         END OF YOUR CODE                             #\n",
        "    ########################################################################\n",
        "    \n",
        "\n",
        "    ########################################################################\n",
        "    #               TODO: Calculate accuracy of prediction.                #\n",
        "    ########################################################################\n",
        "\n",
        "    correct = torch.argmax(y_pred, axis=1) == torch.argmax(y_xor, axis=1)\n",
        "    acc = torch.sum(correct)/len(y_pred)\n",
        "\n",
        "    ########################################################################\n",
        "    #                         END OF YOUR CODE                             #\n",
        "    ########################################################################\n",
        "\n",
        "# Print output tensor\n",
        "print(y_pred)\n",
        "\n",
        "# Plot loss\n",
        "plt.figure(figsize=(12,4))\n",
        "plt.subplot(1,2,1)\n",
        "plt.plot(losses)\n",
        "plt.grid()\n",
        "plt.xlabel('Iter.')\n",
        "plt.ylabel('Loss')\n",
        "\n",
        "# Show decision boundary\n",
        "plt.subplot(1,2,2)\n",
        "plot(x_xor, y_xor, net)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "48obmfkSS956"
      },
      "source": [
        "This concludes the first assignment. You can experiment a bit with the network architecture and see how the various settings affect the decision boundary. For example:\n",
        "\n",
        "****\n",
        "- **What happens if the you increase/decrease the hidden dimension?**\n",
        "- **What happens if you increase the number of network layers?**\n",
        "- **What happens if you replace the ReLU non-linearity by a Sigmoid? Does the network still converge? If not, why? What setting(s) should you adjust to make the network converge again? (Hint: check the magnitude of the network predictions.**)\n",
        "****"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "tf_cpu",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "vscode": {
      "interpreter": {
        "hash": "c787a08491dcdbfffe458afac131a89c5dc65d7591e57c155dcecfdeb67d0cb5"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}