13. SVM part 1.ipynb

{
  "cells": [
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ea9ab27b57fbc7617ef9a747c0bec9b083adb44b"
      },
      "cell_type": "markdown",
      "source": "# Support vector machine, part 1: linearly separable data\n\n## Introduction \n\nIn this notebook we explore the support vector machine (SVM) algorithm. This is a supervised learning classification algorithm. For the sake of simplicity, we focus on the binary classification task; for multiple classes classification, one may perform a one-vs-rest or one-vs-one reduction. We start with the case of linearly separable datasets, in which case the main idea is to find the hyperplane that maximizes the margin (distance) to both classes. For non-linearly sepparable datasets, we allow points to cross the decision boundary up to a certain margin (in this case we say that the classificator has a *soft margin*). Finally, we can also use non-linear decision boundaries by making use of the kernel trick, which relies on the idea of implicit feature augmentation.\n\n## The model: linearly separable case\n\nSuppose we have a dataset $S = \\{(x^1,y^1),\\dots,(x^m,y^m) \\}\\subset \\mathbb{R}^n\\times\\{-1,1\\}$ consisting of pairs (features,class). Denote by $w$ the normal vector to the separating hyperplane $H$ and $a$ a position vector of such hyperplane. Then the points $z$ belonging to $H$ satisfy the equation\n\n$$\nw\\cdot(z-a) = w\\cdot z  - w\\cdot a = 0 .\n$$\n\nHere $\\cdot$ denotes the (Euclidian) inner product and we write $b = -w\\cdot a$. Suppose that $w$ is a unit vector. The hyperplane separates $\\mathbb{R}^n$ in three regions: $R_1 = \\{ x : w\\cdot x + b >0 \\}$, $R_2 = \\{ x : w\\cdot x + b < 0 \\}$ and $H$. In order for $H$ to be a separating hyperplane for our dataset, it is necessary that $x^j\\in R_1$ for all $j$ such that $y^j =1$ and $x^k \\in R_2$ for all $k$ such that $y^k = -1$ (or vice versa). This condition can be summarized by\n\n$$\ny^j(w\\cdot x^j + b) > 0\n$$\n\nfor all $j$. Assume now that $H$ is a separating plane for our dataset. Then the distance between a point $x^j$ and the hyperplane $H$ is given by \n\n$$\nd(x^j,H) = |(x^j-a)\\cdot w| = |x^j\\cdot w + b| = y^j(w\\cdot x^j + b).\n$$\n\nIf we want to maximize the size of the margin, then we want to find the parameters $w,b$ for the hyperplane $H$ such that $\\min_j d(x^j,H)= \\min_j  y^j(w\\cdot x^j + b)$ is maximal. In other words, we want to solve the optimization problem\n\n$$\n\\begin{align*}\n\\max_{w,b} &\\ M\\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) \\geq M \\text{ for all } j, \\\\\n& \\| w \\| = 1.\n\\end{align*}\n$$\n\nWe can remove the constraint $\\|w\\| = 1$ by normalizing the distance by the norm of $w$. That is, the above problem is equivalent to \n\n$$\n\\begin{align*}\n\\max_{w,b} &\\ M/\\| w\\| \\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) > M \\text{ for all } j. \\\\\n\\end{align*}\n$$\nIf we perform the coordinates transformation $w,b \\mapsto w/M ,b /M$, then we get the equivalent optimization problem:\n\n$$\n\\begin{align*}\n\\max_{w,b}\\ &\\|w\\|^{-1}\\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) > 1 \\text{ for all } j. \\\\\n\\end{align*}\n$$\n\nThis is now equivalent to the convex minimization problem\n\n$$\n\\begin{align*}\n\\min_{w,b}\\ &\\dfrac{1}{2}\\|w\\|^2\\\\\n\\text{subject to } &y^j(w\\cdot x^j + b) > 1 \\text{ for all } j. \\\\\n\\end{align*}\n$$\n\nwith linear constraints."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "887f6e532cd105f54410f9eea7cd83ddcdb8c779"
      },
      "cell_type": "markdown",
      "source": "## Lagrangian form\n\nThe previous minimization problem can be formulated in its dual form by considering the lagrangian\n\n$$\n\\mathcal L (w,b,\\alpha) = -(\\dfrac{1}{2}\\|w\\|^2 - \\sum_{i=1}^m \\alpha_i (y^i(w\\cdot x^i + b) - 1 ))\n$$\n\nfor $\\alpha_i \\geq 0$. Solving the original (primal) problem, is equivalent to the minimization of the Lagrangiagian subject to the [Karush-Kuhn-Tucker conditions](https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions). These conditions can be written as \n\n$$\n\\begin{align*}\n\\nabla_w \\mathcal{L} &= 0 , \\\\\n\\nabla_b \\mathcal{L} &= 0, \\\\\n\\alpha_i & \\geq 0, \\\\\n\\alpha_i (y^i(w\\cdot x^i + b) - 1 ) & = 0.\n\\end{align*}\n$$\n\nfor all $i$. The first two conditions can be written as\n\n$$\n\\begin{align*}\n\\nabla_w \\mathcal{L} =  -(w - \\sum_{i=1}^m \\alpha_i y^i x^i &)=   0, \\\\\n\\nabla_b \\mathcal{L} =   -\\sum_{i=1}^m \\alpha_i y^i &=   0 .\n\\end{align*}\n$$\n\nWith this conditions, the lagrangian can be reduced to\n\n$$\n\\begin{align*}\n\\mathcal L (w,b,\\alpha) &= -(\\dfrac{1}{2}\\|w\\|^2 - \\sum_{i=1}^m \\alpha_i (y^i(w\\cdot x^i + b) - 1 )) \\\\\n&= -(\\dfrac{1}{2}w\\cdot w - \\sum_{i=1}^m  (\\alpha_i y^i w\\cdot x^i + \\alpha_i y^i b - \\alpha_i )) \\\\\n&= -(\\dfrac{1}{2} \\sum_{i,j=1}^m \\alpha_i \\alpha_ j y^i  y^j x^i\\cdot x^j - \\sum_{i,j=1}^m \\alpha_i \\alpha_ j y^i  y^j x^i\\cdot x^j  + \\sum_{i=1}^m \\alpha_i )\\\\\n&=-\\sum_{i=1}^m \\alpha_i + \\dfrac{1}{2} \\sum_{i,j=1}^m \\alpha_i \\alpha_ j y^i  y^j x^i\\cdot x^j\n\\end{align*}\n$$\n\nThis is a quadatic problem that can be solved using the usual optimization tools. We proceed to implement the algorithm. Once the optimization is solved, for a new observation $x$, we can predict its class by\n\n$$\n\\widehat y = \\mathrm{sign}(w\\cdot x + b).\n$$\n\nFor this, we need the values of $w$ and $b$ as a function of the training data and the parameters $\\alpha_i$. An expression for $w$ follows directly from the KKT conditions. For $b$, we can get a condition by subtracting the equation $(y^i(w\\cdot x^i + b) - 1 )  = 0$ for two indices $i,j$  such that $\\alpha_s,\\alpha_t >0$ and $y^s = 1$, $y^t=-1$, and then solving for $b$. Thus we obtain\n\n$$\n\\begin{align*}\nw &= \\sum_{i=1}^m \\alpha_i y^i x^i  ,\\\\\nb &= -\\dfrac{1}{2}w\\cdot (x^s+x^t).\n\\end{align*}\n$$\n\nThe vectors $x^i$ so that $\\alpha_i > 0$ are called the **support vectors**, and the separating hyperplane is completely defined by them."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "bad345e3bbb070073830b35876e66e8606756d9f"
      },
      "cell_type": "markdown",
      "source": "## Implementation\n\nIn this section we implement the algorithm described above. We start by implementing the usual scientific libraries. This time, we will use `scipy` minimizing tool:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f8a0d13fd5d6c5ad71612970ce8490fa47f02e30"
      },
      "cell_type": "code",
      "source": "import numpy as np\nimport matplotlib.pyplot as plt\nfrom scipy.optimize import minimize",
      "execution_count": 1,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "617875a12555609e9edfe89819d68a503fc87247"
      },
      "cell_type": "markdown",
      "source": "We generate two classes of points which are linearly separable. We fix a seed of random numbers so we always get the same dataset:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "1527939ac76db262eeebe28cf3092d647075ed7e"
      },
      "cell_type": "code",
      "source": "np.random.seed(6)\nclass_0 = 25 \nclass_1 = 25\n\nx0 = np.random.multivariate_normal([-2, 0], [[1, .15],[.15, 1]], class_0)\nx1 = np.random.multivariate_normal([3, 2], [[1, .25],[.25, 1]], class_1)\n\nplt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 1: two Gaussian clouds of points. Red points represent class 1 and blue class 0', \n            wrap=True, horizontalalignment='center', fontsize=12)\nplt.axis('equal')\nplt.show()\n\ny0 = -np.ones(len(x0))\ny1 = np.ones(len(x1))\n\nX_train =  np.vstack((x0,x1))\ny_train = np.append(y0,y1)",
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhUAAAEPCAYAAAD1QTYeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XucHGWd7/HvjyRclJuS8QIBoqLuxgtymCAedXVQVq6CrCIqcFz1Fc3qEVZwJOrkhKCLiYqXXRdl1cMB1DhHBBVFlJMBdQWciQK74baIQLjJAEYSRSHyO388TzM1PdW3SXU/1d2f9+vVr5nqrq76dV1/9TxP1WPuLgAAgK21TeoAAABAbyCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhSCpAAAAhZibOgA0Nn/+fF+4cGHqMACga6xbt+4Bdx9IHUe/IanoAgsXLtTExETqMACga5jZHalj6EdUfwAAgEKQVAAAgEKQVAAA2mb1amlsbPp7Y2PhffQekgoAQNssXiwde+xUYjE2FoYXL04bF9qDhpoAgLYZGpJGR0MisXSpdPbZYXhoKHVkaAdKKgAAbTU0FBKKM84If0koehdJBQCgrcbGQgnFyEj4W93GAr2DpCIRM5tjZr8ys0tSxwIA7VJpQzE6Kq1cOVUVQmLRm0gq0jlJ0o2pgwCAdhofn96GotLGYnw8bVxoDxpqJmBmCyQdLunjkj6QOBwAaJvh4ZnvDQ3RrqJXUVKRxmclDUt6vNYIZrbEzCbMbGJycrJzkQEAMEskFR1mZkdIut/d19Ubz93PcfdBdx8cGKBPHABA+ZFUdN7LJb3ezG6XtEbSQWZ2QdqQAADYeiQVHebuy9x9gbsvlHScpLXufnzisAAA2GokFQDQJPqxaD+WcXcjqUjI3a9w9yNSxwGgOfRj0X4s4+7GLaUA0CT6sWg/lnF3o6QCAFpAPxbtxzLuXiQVANAC+rFoP5Zx9yKpAIAm0Y9F+7GMuxtJBQA0iX4s2o9l3N3M3VPHgAYGBwd9YmIidRgA0DXMbJ27D6aOo99QUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEALVq9Whobm/7e2Fh4H+hnJBUdZmbbm9kvzOw6M1tvZqenjglAaxYvlo49diqxGBsLw4sXp40LSG1u6gD60J8lHeTum81snqSfmdml7n516sAANGdoSBodDYnE0qXS2WeH4aGh1JEBaVFS0WEebI6D8+LLE4YEYBaGhkJCccYZ4S8JBUBSkYSZzTGzayXdL+nH7n5NzjhLzGzCzCYmJyc7HySAusbGQgnFyEj4W93Gop1o04GyIqlIwN3/4u4vkbRA0gFm9sKccc5x90F3HxwYGOh8kABqqrShGB2VVq6cqgrpVGJBmw6UFUlFQu6+UdKYpENSxwKgeePj09tQVNpYjI93Zv7ZNh3Ll08lOFTBIDUaanaYmQ1IeszdN5rZDpIOlrQqcVgAWjA8PPO9oaHOntSzbTpGRkgoUA6UVHTeMyWNmdn1ksYV2lRckjgmAF0mZZsOoBZKKjrM3a+XtF/qOAB0r2ybjkoJCVUgKANKKgCgy6Ru0wHUYu48IqHsBgcHfWJiInUYANA1zGyduw+mjqPfUFIBAAAKQVIBAAAKQVIBAAAKQVIBAAAKQVIBACVCvx7oZiQVAFCQIhKCVvv1IAlBmZBUAEBBiujoq9V+PehcDGVCUgH0Ai5XS6Gojr6y/XosXVr/+3QuhjIhqQB6AZerpdFKQlBLq/16NJonOSc6xt15lfy1//77O9DQ2rXu8+e7j4yEv2vXpo6oL23taqh8v/K96uHZzHM20+x2kia8BMfvfnslD4BX4xdJBZo2MhJ265GR1JH0hVWrpp+Y165132UX9yVLpoZbPXlXT7MynVWr8sdvNmHot5yTpCLNi+oPoFfQF3bHVdc6rVkjuUvHHReGZ9PR1/DwzOqLoaHwfp5mOxcroloGaIQOxboAHYqhoeq+sKuH0TaVRb10acjlyrrIuyXOotChWBqUVAC9gL6wk+mGEoBsjrly5dTdIhRmoWiUVHQBSiqA8uqGEoDVq0NVTTausbGQc9aqVul2lFSkQVLRBUgqgHKi1qm8SCrSoPoDAGaJWidgOkoqugAlFQDQGkoq0qCkAgBaxBMqgXwkFQDQIp6KDuSbmzoAAOg22U68ynzXB9BplFQAwCx0w/MpgE4jqQCAWeCp6MBMJBUdZmZ7mtmYmd1gZuvN7KTUMQFoDU+oBPKRVHTeFkmnuPsiSQdKeq+ZLUocE4AWlO35FNyNgrIgqegwd7/X3X8Z/98k6UZJe6SNCkArWu1JtN24GwVlQVKRkJktlLSfpGtyPltiZhNmNjE5Odnp0IAkuOKenezdKMuX86hwpENSkYiZ7SjpQkknu/vD1Z+7+znuPujugwMDA50PEEiAK+7Z424UlAFJRQJmNk8hofiau387dTxAWXDFPXvcjYIyIKnoMDMzSV+RdKO7n5U6HqBsuOJuHXejoCxIKjrv5ZJOkHSQmV0bX4elDgooC664W5e9G6XS/iR7NwrtUtApPKa7w9z9Z5IsdRxAGWWvuIeGwosqkMayd51U2qWMjob3s8sUaDdKKgCURtme/9CNaJeClMzdU8eABgYHB31iYiJ1GAC6yPLloV3KyEhoZ9FvzGyduw+mjqPfUFIBAD2GdilIhaQCAHoId4IgJZIKAOghtEtBSrSp6AK0qQCA1tCmIg1KKgAAQCFIKgCgBXR6BtRGUgEALaDTM6A2nqgJAC3IPlxq6dJwyyYPlwICSioAoEV0egbkI6kAgBbxcCkgH0kFALSAh0sBtZFUAEALeLgUUBsPv+oCPPwKQCOrV4c7ULLtO8bGQrKT7Rq9X/DwqzQoqQCAHsCtrigDbikFgB7Ara4oA0oqAKDL1Hqq5/h4a7e68nRQFI2kAqjGkRYlV6uq46c/lT7/+em3utbbdKkyQdFIKoBqHGlRctmqjuXLw99ly0JS4T71+RveIB19dO1NN286VJlga5BUANU40qILVD/Vc8sW6aKLpIsvnsqJ3aXjjqu/6c726aAU6CEPSQWQh+cwo+Sqn+pZuZ10aEjad9+w6Z50kvSlL02Nn3fCn+3TQSnQQy5351Xy1/777+/osLVr3efPdx8ZCX/Xrk0dEfCEyuZZ2Syzw2vXuu+8s/uTnhT+Vt7L24zrTaeVOMq4m0ia8BIcv/vtRUkFekdR5bE8h7k5lH8nU+upnmvWhE314oulSy6RzKTDDw9tK/Jq8Lb26aAU6GGG1FkNr8YvSiqatLWXXRWrVuVf0q1aVUycvaKo5Y3CVG+6IyPukvt++00fr6jNmZIKXtWv5AH020vSVyXdL+k/m/0OSUULynyU60Us79KqrJoTTnA3c//0p6e/v7Wrquw5JUlFmhfVH513rqRDUgfRsyiP7SyWdylla/DOO0/61KekU0+VTjyxuJuZ6FgNeehQLAEzWyjpEnd/YTPj06FYCypHU55T3Bks71LK61zsxBOl888Pd3msXJkutk6hQ7E0KKkoKTNbYmYTZjYxOTmZOpzWpGrARwPLzkq4vGkjWt/w8MzeSi+9tPXbRoFWkVSUlLuf4+6D7j44MDCQOpzWpLqBnfLYzkq4vHlGQvPItdFJVH8k0BfVHxSLo836YRPLq8aodBw2PNy5aXQjqj/SoKQC7UEDPrRZP2xiRZTIVFeFSGG4lxMKpENS0WFm9g1JV0l6vpndZWbvTB1TW8z22b9Ak/phE6MbGnSbuakD6Dfu/pbUMbRdthK30hkBR0MUqJ82sWyJzMhI7/0+9BZKKlA8Gkym0Ue3RPTTJtYPJTLoHTTU7AJd2VCz7Hqx9Vr15Xv1MLoOq3T2aKiZBiUV6E+9eE8iFfA9p59KZNAbKKnoApRUtEmv3pO4fPlUBXw/PDoRyEFJRRqUVKB/9eI9iVTAA0iIpAL9q9dOwDw6EUBiJBXoT714AqYCHkBitKnoArSpaINevPsDwBNoU5EGSUUXIKkoGRISoPRIKtKg+gP9a7YPi+rF21EBoAAkFehfs00OeB4Eauijh5oCuUgq0L+2JjnoxdtRsdUoxEK/I6lAf5ttctBrt6OiEBRiod+RVKC/zSY56MXbUVEYCrHQz0gq0L9mmxzwPAjUQSEW+hm3lHYBbiltE24NRcHoVbQ8uKU0DZKKLkBSAXQH8tTyIKlIg6SiC5BUAEBrSCrSoE0FAAAoBEkFAAAoBEkFAAAoBEkFAAAoBEkFAAAoBEkFyoUemQCga5FUJGBmh5jZzWZ2q5mdljqeUqFHJgDoWiQVHWZmcyR9QdKhkhZJeouZLUobVYl0ukcmSkYAoDAkFZ13gKRb3f02d39U0hpJRyWOqTOaPYF3skcmSkYAoDAkFZ23h6QNmeG74nu9r9kTeCd7ZKKvagAoDElFSZnZEjObMLOJycnJ1OEUo5kTeIpuxbMlI4ceOjMeqkIAoCkkFZ13t6Q9M8ML4nvTuPs57j7o7oMDAwMdC67tGlVtpOhWvFIycsIJ0gUXSGedNfU+VSEA0LS5qQPoQ+OSnmtmz1JIJo6T9Na0IXVQddXG0ND0xCKvK8fqcYqOJ1ti8pKXSKeeKl17rfTtb4fSErqcBICmUFLRYe6+RdL7JF0m6UZJo+6+Pm1UHZKiaqOR6pKRD3xAOv546fzzpWOOkc48k0acANAkSioScPcfSPpB6jg6rl7VRqqGkdUlDmNj0qWXTpWkLFsWEomlS8MwjTgBoCZz99QxoIHBwUGfmJhIHUbvq64KqQwfemgouRgZCSUsqaxeHUpJqI4BGjKzde4+mDqOfkP1B1CRV5KybJl04YWdub21EZ6pAaDkKKnoApRUJFKr5CJlFUglBqpjgLooqUiDkgqglhS3tzbSyaeNAkCLKKnoApRU4AmUVABNoaQiDUoqgG5RxltyASCDpALoFmWsjgGADKo/ugDVHwDQGqo/0qCkAgAAFIKkAgAAFIKkAuhFq1fPbMBJN+4A2oykAuhFPH0TQAJ0KAb0osqdITzTAkAHUVIB9Cqevgmgw0gqgF41NhZKKMrQGRqAvkBSAfQinr4JIAGSCswOdxeUG0/fBJAASQVmh7sL8pUl2RoentmGYmgovA8AbUJSgdnJ3l2wfPlUUXu/NwYk2QLQx0gqMHvcXTBTkclWWUo9AKBJJBWYPe4uyFdUskWpB4AuQ1KB2eHugtqKSraoYgLQZUgqMDvcXZCv6GSLKiYAXcTcPXUMaGBwcNAnJiZSh4FmrF4dqieyJ/+xsZBszebOi0qSwqO2gZaY2Tp3H0wdR78hqegCJBV9KlvqMTQ0cxhATSQVaVD90UFm9iYzW29mj5sZGzvqo4oJQJehl9LO+k9Jx0j6UupA0AXyqkuGhiilAFBaJBUd5O43SpKZpQ4FAIDCUf0BAAAKQUlFwczscknPyPnoI+7+nRams0TSEknaa6+9CooOAID2IakomLu/tqDpnCPpHCnc/VHENAEAaCeqPwAAQCFIKjrIzN5gZndJepmk75vZZaljAgCgKFR/dJC7XyTpotRxAADQDpRUAACAQpBUAACAQpBUAACAQpBUAACAQpBUAACAQpBUoDutXh26As8aGwvv99I8AaCLkFSgOy1eLB177NRJfmwsDC9e3FvzBIAuwnMq0J2GhqTR0XBSX7pUOvvsMNzObsFTzBMAugglFeheQ0Ph5H7GGeFvvZN7UVUXrcwTAPoMSQW619hYKC0YGQl/q5OGrKKqLlqZJwD0GZIKlE8zpQqVpGB0VFq5cqpaotZJPlt1sXz51HdbKWlodZ4A0GdIKlA+zZQqjI9PTwoqScP4eO3pbm3VxWzmCQB9xNw9dQxoYHBw0CcmJpr/wurV4QScPWmOjYWT3/Bw8QG2QyWRKLJBZDumCaCUzGyduw+mjqPfUFLRi3rh1seiG0RSdQEAbUdS0YuKaD+QWtENIqm6AIC2o/qjC7Rc/VGxfHm40h8ZCVfn3SJbqjA0NHMYABqg+iMNSip6VTff+kipAgB0JUoqukDLJRVc6QPoc5RUpEFJRS/iSh8AkAAlFV1g1m0qAKBPUVKRBiUVAACgECQVQDOK6pAMAHoYSQXQjF54oBgAtNnc1AEAXSH7QDEe8w0AuSipAJpV9KPDAaDHkFR0kJl90sxuMrPrzewiM9s1dUxoQTc/UAwAOoCkorN+LOmF7v5iSbdIWpY4HjSLDskAoCGSig5y9x+5+5Y4eLWkBSnjQQt4oBgANERDzXTeIembqYNAk4aHZ743NES7CgDIIKkomJldLukZOR99xN2/E8f5iKQtkr5WZzpLJC2RpL322qsNkQIAUCySioK5+2vrfW5mb5d0hKTXeJ1npLv7OZLOkcJjuouMEQCAdqDvjw4ys0MknSXpVe4+2cL3JiXd0bbAZpov6YEOzq/sWB7TsTymY3lMV5blsbe7D6QOot+QVHSQmd0qaTtJD8a3rnb39yQMKZeZTdARzxSWx3Qsj+lYHtOxPPob1R8d5O77pI4BAIB24ZZSAABQCJIK5DkndQAlw/KYjuUxHctjOpZHH6NNBQAAKAQlFQAAoBAkFajLzE4xMzez+aljSYnO4MIt0WZ2s5ndamanpY4nJTPb08zGzOwGM1tvZieljqkMzGyOmf3KzC5JHQvSIKlATWa2p6S/lXRn6lhKoK87gzOzOZK+IOlQSYskvcXMFqWNKqktkk5x90WSDpT03j5fHhUnSboxdRBIh6QC9XxG0rCkvm94Q2dwOkDSre5+m7s/KmmNpKMSx5SMu9/r7r+M/29SOJHukTaqtMxsgaTDJX05dSxIh6QCuczsKEl3u/t1qWMpoXdIujR1EB22h6QNmeG71Ocn0QozWyhpP0nXpI0kuc8qXIQ8njoQpMPDr/pYvc7PJH1YoeqjbxTVGRz6h5ntKOlCSSe7+8Op40nFzI6QdL+7rzOzV6eOB+mQVPSxWp2fmdmLJD1L0nVmJoWi/l+a2QHufl8HQ+yoojqD61F3S9ozM7wgvte3zGyeQkLxNXf/dup4Enu5pNeb2WGStpe0s5ld4O7HJ44LHcZzKtCQmd0uadDdy9BJUBKz7QyuV5jZXIUGqq9RSCbGJb3V3dcnDSwRC9n2/5H0kLufnDqeMoklFae6+xGpY0Hn0aYCaM6/SNpJ0o/N7Foz+2LqgDopNlJ9n6TLFBoljvZrQhG9XNIJkg6K28O18Sod6GuUVAAAgEJQUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAApBUgEAAAox66TCzPYys82xS2R0KTNb3+ln9ZuZm9k+bZjuCjO7oOBp7mBm3zOz35vZ/y142l80s5Eip5lCG9dnx7dNBO1ap2VjZuea2cdqfPZ2M/tZne9eYWbval90ufMs/XppmFSY2e1m9khMICqv3d39Tnff0d3/0olAG8S4rZl9K8bqrR6IOrVxWPA+M7vezP5oZvfFeR/X7nnX4u4vcPcrUs2/C7xR0tMl7ebubypywu7+Hnc/o5lx6x38WlW1T98Xp71jEdMuUivbZvxNdftu6SdmtjAeC0vfv5OZvdDMLjOzB8yMpzGWiJn9YzxGPGxmXzWz7Rp9p9mSiiNjAlF53bOVsdY1yx3hZ5KOl1TmDq8+L+lkSadI2k2h6+iPSjokZVCoa29Jt8THVPeSI919R0kvUei2e1nieEqh6JNwN5zUS+AxSaOS3pk6EEwxs9dJOk2hv5+9JT1b0ukNv+judV+Sbpf02pz3F0pySXPj8LMk/UTSJkmXS/qCpAviZ6+WdFet6UpaIelbki6Q9LCkdykkPKdJ+rWkBxU2uqc2Ee9dkl7daLzM+B+X9BdJf5K0WaGPh9Ml/XP8fJ6kP0j6ZBzeIY771Dj8eknrJW2UdIWkv64xn+fF+Qw2iOfvFfpW2CTpNknvznz2dkk/qxrfJe0T/z9M0g3xu3crdOojSfMlXRJjfEjSTyVtk7MeDpB0VRzv3rgstq2a13sk/Vcc5wuKj3rP+R1zFLpP/3WMZ52kPXNi3kXSeZImJd2hkGRVYltR2YbqbHNXxun/OMZb2ea2j9vTgzHWcUlPrxHrX8d1tzGuy9fH90+X9KjCQW+zpHfmfHeFwrb7zRjHLyXt22ja8bNzJX0su48oJJz3x+X/9/GzJTGGR2Mc34vvfyiu502SblboPbXlfVrSaknfzwxvJ+lTku6U9FtJX5S0Q+bzD8b47pH0juz6zJnXFZLOlPQLhX37O8rsx6qz/2jmMWI0biub4ncG42fnS3pc0iNx+Qy3uP5vj8vyekl/Vui9eXeFHkgnJf1G0vtbWOetTu8ASRNx+fxW0lmZzw6U9PP4G65T5tgWl9cZkv49xvEjSfPjZ3fG9bI5vl62Ffvo4ZJ+FePbIGlFZho1l7PC8eq2OO3fSHpbg+1yH0nexPb7uRjHwzHmV1atm9ztJH6+X1xfm+L6W6O4D+bM5+1x2f6LpN9LukmZfSwu/3c1eazaRdJXFPabuyV9TNKcrTx2tnW9SPq6pH/KDL9G0n0N10+rB6A6C+0qhQPRtpJeEX9oK0nFY5KOVkgmdpB0kqSrFbpY3k7SlyR9o4l4ZyQVkt4q6fo633li44jDB0n6j/j/f48r95rMZ9fF/5+nkHAcrJB8DEu6VZkTcWaa75F0exPxHy7pOZJM0qsk/VHSf8tsDPWSinsVdzBJT8l870yFE8O8+Hqlpvp9ya6H/RUOYnPj+r1R0slV87pE0q6S9lI4QB5S43d8UNJ/SHp+/C37KlQhVMd8nsKJZqc4z1sUT95qvKNepdBz6HaS/kZhJ6lsc++W9D1JT1LYSfeXtHNOnPPiOvuwwrZ7UJzO8/NiyPn+CoVt941xWqcq7Kjzmpj2uZqeVGyRtDJ+77C47p9SPW4cfr7CgWT3zLJ5TqPtK2edL4jr6XOZzz8j6buSnhrXy/cknRk/O0ThxPdCSU9WOPA0Siruzox/YWYd1d1/NPMY8ae4XOYobNNX1zpONbv+M9+9VqFr9x0UjkHrJC2P6+3ZCgfh1zVa57Oc3lWSToj/7yjpwPj/HgonhcPiNA6OwwOZZfvruBx3iMOfyNtXtnIffbWkF8UYXhzX/9H1lnNc1w9ralt/pqQXNNgum00qjlco6Z2rkITfJ2n7RttJXPZ3SPrHuN7eGNdjvaRiS2b8NyskF5ULyivUfFJxkcI57MmSnqaQZL+7xnxLsV4Uktg3Z4bnx/nvVnf9NHkA2qyQ7WyUdHH1QlM4wWyR9KTM9y5Qa0nFT6o+v1HTs8Jnxg2g5k4Sx2uppKJ644jDldKI3RRKSz4cp7ujwtXr5+N4Iwq9NVa+t43CAXTG/BWuwK/OiXVjnNfeNWK7WNJJmY28XlJxZ9yYdq4aZ6XCiXvGgV81ksb42cmSLqqa1ysyw6OSTqvx3ZslHVXjM1c4gMxRuPpelPns3ZKuaLSjZra5J2c+/3pmm3uHwhXeixus+1cqHJS2ybz3DcWsvzqGnO+v0PST2zaKyV0T0z5X05OKR7Lbt0KJxYHV48bhfeLnr1U8mbWwvd+usE9visvz/0naNX5mCif652TGf5mk38T/v6p44orDz1PjpCI7/qK4zueowf6jmceIy6um80it7bjZ9Z/57jsywy+VdGfVOMsk/e9G63yW0/uJwnFlftU4H5J0ftV7l0n6H5ll+9HMZ/8g6YfV+0qd391wH63x2WclfabeclY4eW2U9HfKlHI1WA9NJRU53/udYklRve1E4cLjHmVKV2Ps9ZKK6vF/oakE8Ao1kVQotMn6s6aX9r1F0liZ14tCwnpIZnhenP/Cet9rtk3F0e6+a3wdnfP57pIecvc/Zt7b0OS0a42/t6SLzGyjmW1USDL+orCC2srdH1EojnyVwoZ4pcIKenl878o46u4KmW/le48r/I49cib7oEJilJ3PAoXsbzuFg7nM7FAzu9rMHoq/+7A4TjP+Lo5/h5ldaWYvi+9/UuEK8EdmdpuZnZb3ZTN7npldUmmYI+mfcuadbbPyR4VEK8+eChtlPfMVNtQ7Mu/dofzlV213Sb9z9z9UfbfifIUD8Bozu8fMVpvZvBrT2RDXXasxVDyx7cbp3BWn2+q0H/TpbTdqLl93v1Uh6Vsh6X4zW2Nmu7cQ89HuvpNCMvNXmlrPAwpXN+sy+94P4/uq/Kaq39NI9fjz4vxa2X+kmdve9nXaLDS7/vNi3FvS7pXfH5fBhzX92FNrnc9meu9USM5uMrNxMzsi8703VX3vFZp+HGl2f8zTzD4qM3upmY2Z2aSZ/V6h1LWyveQu57hfvjmOe6+Zfd/M/qqF2OrFc6qZ3RjvyNqoULWQPU7V2k52l3S3xzNk1Gj7zRu/lf1MCutxnsJyqKzHLymUWOQpy3rZrFC6UVH5f1O9uIp6TsW9kp5qZk/KvLdn5v8/KByoJEkWbkMd0HReNbxB0qGZZGZXd9/e3e8uKOZ685ZC4nCQQh3ceBx+nUL950/iOPcobDCSwt0dCr87L8a1khaY2WCtIGLL2gsVqpGe7u67SvqBYsKhmcvxGdN+hPu4ux+lsLFerFCSIHff5O6nuPuzFeqwP2Bmr8kJ4WyFesPnuvvOCgc+yxmvGRsUqnHqeUCh9GnvzHt7aWr5Tfu9krK/915JTzGzJ1d9V5Lk7o+5++nuvkihCusISSfmxHCPpD3NLLsvZGNoxhPbepzOgjjdIqZdMWMbdfevu/srFJafS1rV8kTdr1QoBflUfOsBhRKTF2T2u108NOqUwnLP7tt7qbHq8R+L82ll/2n4U6YNNL/+876/QaFkJnvs2cndD8v7TVXrvOXpuft/uftbFPbbVZK+FbfrDQolFdnvPdndP9Hq8qihmX1UCiWA31Wo199FoSrVYuw1l7O7X+buByskQTdJ+rcm5lWXmb1SoZrsWIWqwV0VqiSaOU7dK2mPuJ1VNNp+88bPu1Gh3rFqg0JJxfzMetzZ3V9QY56Fk4C0AAAE6ElEQVRlWS/rFapeKvaV9Ft3f7BeUIUkFe5+h8KV/Yp4e+fLJB2ZGeUWhWzx8Hi18FGFq/N6vijp42a2tySZ2YCZHVVrZDPbzsy2j4Pbmtn2VRtDPb9VqOfMulJhJdzg7o8qFnUpHBwm4zijkg43s9fE33WKwsbz8+oZuPvNCtnpGjM72MLzD+YorPCKbRWWy6SkLWZ2qKS/zXx+naQXmNlL4m9dkfn925rZ28xsF3d/TKHe7PH42RFmtk9cHr9XKPHJXj1X7BS/tzlmr0trL7KGvizpDDN7rgUvNrPdsiN4uB15VGE97xTX9QcUqs6kUC/9NxaeibKLMncoZLa50+Nvf4Uy25yZDZnZi+IyfljhRJb3m69RuJoZNrN5Fm5HPlKhAVez9jezY+LV0MkK28DVBU27Yto2ambPN7ODYiL6J4VEIO/3NeOzkg42s33jVfe/SfqMmT0tzmsPCy3BpbC+3m5mi+JFxP9qYvrHZ8ZfKelbmXXf1P7ThOrl0+z6z/MLSZvM7EOV/dTCbY+LM+PUWuctT8/MjjezgbjsN8bvPK6wHxxpZq+L39nezF5tZgua+A2TcRrVx7WshvtotJNCSfSfzOwAhTZqirHnLmcze7qZHRWToz8rXPXmLv847+0Vjn+Kv7PW+WEnhWrPSUlzzWy5pl9N13NV/O774/54jMJFYj1Py4z/JoWG1z/IGa/esepehUa0nzaznc1sGzN7jpm9qsY8S7FeFNq7vTPuu7sqnLfPrTHuE4p8oubbFOpeH1Ro2fpNhaDl7r9XqO/7ssJVyB8Uigvr+ZxCFvYjM9uksMO+tM74NyscWPdQKPZ5RPEqKJ5s1zeY1xvN7Hdm9vn43s8V2lZUSiVuUDh4V4YricLxkv5Z4crrSIVb9R6tMZ/3KtxWepbCXRh3KbTefrNCnesmSe9XONj+TmEj+W5mfrcoHJQvV7gDo/rBLCdIut1C1cV7FNaJJD03fmezwo71r+4+lhPfqXGemxROLN+s8TuacVb8HT9S2Ki/orA8q/1Phe3htvh7vq5Qby93/3GM4XqFhm6XVH33rQrbxEMKJ7fzMp89Q6GF/sMKVWdXKhQJThPX1ZGSDlVYh/8q6UR3v6mF3/odhXX4O4V1cEy8Uihi2hVfkbTIQvHpxQrJ5yfidO9TOPgtk5ra3qeJSfJ5Cg0JpVCXf6ukq+O2dLlCozG5+6UKScjaOM7aJmZxvsLB6D6FVunvj9Nqdf+p50xJH43L51Q1uf7zxITnCIXbbX8TY/uyQjF7Re46n+X0DpG03sw2KxyLjnP3R9x9g6SjFEoMJxWuYD+oJo7bsSr645L+PS6TA3NGa3Yf/QdJK+NxeHn8TkWt5byNwgXCPQr756tU+yJlb4XjdWWbfUTheJ7nMoXquFsUqiL+pCar2uN2dYxCW4mHFNbftxt87RqF4+cDCsvzjXlX6k0cq05USJpuUNhmvqWq6vCMUqwXd/+hwp1hYwrt9e5QExcRlTsACmdm35R0k7s3cyUDdCUzW6HQcOr41LGUkZldodCA7cupYykK6xyorbCSCjNbHIt0tjGzQxQy7IuLmj4AACi3Ip/29gyFoqTdFIr1l7r7rwqcPgAAKLG2VX8AAID+QtfnAACgECQVAACgECQVAACgECQVAACgECQVAACgECQVAACgEP8fmw9apJVZZWUAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "96404aa419f0afcb1b393dcd175c9ffabdc243c1"
      },
      "cell_type": "markdown",
      "source": "We compute the matrix containing the inner products of the points of the dataset."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a5165bdaa07a0f94d387c2d3b6fb2f5754ca7510"
      },
      "cell_type": "code",
      "source": "def inner_prods(X_train):\n    return np.dot(X_train,X_train.T)",
      "execution_count": 3,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "15af46d41d392d59fc5988e6213a6dc93034a8e7"
      },
      "cell_type": "markdown",
      "source": "With this, we can write the cost function as:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6a83a7499ebe162cc407b85b26493c0d1ef9abbd"
      },
      "cell_type": "code",
      "source": "def cost(alpha): \n    return -(np.sum(alpha) - (1/2)*np.dot(np.multiply(alpha,y_train),\n                                        np.dot(inner_prods(X_train),np.multiply(alpha,y_train))))",
      "execution_count": 4,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "f8c74947cc7f78563b2a6cfd2ac24b10cad9b3ac"
      },
      "cell_type": "markdown",
      "source": "We define the constraint given by the KKT conditions. This is of the form $\\sum_{i=1}^m \\alpha_i y^i = 0$."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "565e648ae4ec0623dfa1c79171af121bc753e0aa"
      },
      "cell_type": "code",
      "source": "def cons1(alpha):\n    return np.dot(alpha,y_train)\n\ncons = ({'type': 'eq', 'fun': cons1})",
      "execution_count": 5,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "c329f3b3b2067cc037b09d059618db2a5831cbed"
      },
      "cell_type": "markdown",
      "source": "We also set the conditions for the Lagrange multipliers: $\\alpha_i \\geq 0$:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ec4c8ee6382e84eb3d572a1302eb925d04d23407"
      },
      "cell_type": "code",
      "source": "bds = [(0,None) for _ in range(len(X_train))]",
      "execution_count": 6,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0fee47d9ba66f8bc42dec14848557508a9435ae5"
      },
      "cell_type": "markdown",
      "source": "Now, we need an initial guess for the value of $\\alpha_i$ for the optimizer:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "23cb29a47e86344313477bf65fd09dcd746791a5"
      },
      "cell_type": "code",
      "source": "alpha = np.random.random(len(X_train))",
      "execution_count": 7,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "196bbeb5cd0b41db7f4f6059040056ae9a2ac6a4"
      },
      "cell_type": "markdown",
      "source": "Finally, we optimize the cost function "
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "58420d8706d87d2695faac8d82f59da1f6352099"
      },
      "cell_type": "code",
      "source": "res = minimize(cost, alpha , bounds=bds, constraints=cons)",
      "execution_count": 8,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "692ac353c202e4de221de9388e6e9261a492ba48"
      },
      "cell_type": "markdown",
      "source": "We recover the values of $\\alpha_i$ that optimize the Lagrangian:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "5c6343f0598677e5eaccaa893d203cb309117a9a"
      },
      "cell_type": "code",
      "source": "alpha = res.x",
      "execution_count": 9,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "25f31cfc13796b8a5006daa8531c1d93868a79e7"
      },
      "cell_type": "markdown",
      "source": "We will set to zero the values $\\alpha_i$ which are smaller than $10^{-7}$:"
    },
    {
      "metadata": {
        "trusted": true,
        "scrolled": true,
        "_uuid": "5001faa00b97641f2bd3c05792d5bd06cac7a4b1"
      },
      "cell_type": "code",
      "source": "alpha = alpha*(1*(alpha > 10**(-7)))",
      "execution_count": 10,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "073f23d46de099555abd7e1d6a2a652dcfe25e2f"
      },
      "cell_type": "markdown",
      "source": "We can now construct the parameters $w,b$:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ed71dd719b6a20e6eb2cae497d2e2d5e17cb8e81"
      },
      "cell_type": "code",
      "source": "w = 0\nfor i in range(len(X_train)):\n    w += y_train[i]*alpha[i]*X_train[i,:]\n\nb = (-1/2)* np.dot(w,X_train[((alpha > 0) & (y_train == 1))][0] + \n                   X_train[((alpha > 0) & (y_train == -1))][0] )",
      "execution_count": 11,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0d1d4676674ebce1c8abd1da0ebc30b5e9dc7f34"
      },
      "cell_type": "markdown",
      "source": "With this, we can build the prediction function:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "1440c5e3027132f81c60c51c40a92db7a6a5da22"
      },
      "cell_type": "code",
      "source": "def predict(x,w,b):\n    return np.sign(np.dot(x,w)+b)",
      "execution_count": 12,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4642ebb91cbb509cc2077fe26ad2c86968e4ef5c"
      },
      "cell_type": "markdown",
      "source": "Finally, we can plot the decision boundary given by the SVM as well as the regions predicted for each class. We can see how the separating hyperplane maximizes the margin to both classes."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "98a44a029aeef73ae3b5434bcde4338df1f66e23"
      },
      "cell_type": "code",
      "source": "plt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 2: SVM decision boundary', \n            wrap=True, horizontalalignment='center', fontsize=12)\n\nx = np.linspace(-5, 5, 200)\ny = np.linspace(-5, 5, 200)\nX, Y = np.meshgrid(x, y)\nz = np.zeros(X.shape)\nZ = np.array(z)\nfor i in range(Z.shape[0]):\n    for j in range(Z.shape[1]):\n        Z[i,j] = predict(np.array([x[j],y[i]]).reshape(1,2),w,b)\nplt.contourf(X, Y, Z, alpha=.5, cmap='jet_r')\nC = plt.contour(X, Y, Z,  colors='black',zorder=4)\nplt.axis('equal')\nplt.show()",
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEPCAYAAABWc+9sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt8FOW9P/DPN5ALBELAoBBAs4paEaNoVDhgFOVgrVeUVmnVQ61arG3t+dlabc/poa/Tyzmtp1Zfaj3US221CMZCwXoB2wiC4DEiBRHQSLgGSrjmBpvb9/fHTJbNspvdTWb3mZn9vF+vvJzd2cx8dw2fefaZZ54RVQUREflHlukCiIjIWQx2IiKfYbATEfkMg52IyGcY7EREPsNgJyLyGQY7EZHPMNiJiHyGwU5E5DN9Tey0qKhIS0pKTOw6rVpqa02XkLHW794NAMjPH4qCgmGGq6FUKy7OMV1CWnzwwQf7VHVovNcZCfaSkhJUVVWZ2HVa7Zw923QJGWvUj38MAGhqqsPdd7/KcPe52bNPNl1CWojItkRex64Y8qXf3XJLaHnRou8ZrIQo/Rjs5EtXnHlmaDkYbDJYCVH6MdiJiHyGwU6+dWZhHgBg796NUO0wXA1R+jDYybd+P3MWAKClpRlz595puBqi9GGwk28VDxqEwrzOVvtmw9UQpQ+DnXytqaUFAHD06CHDlRClD4OdfO284mIAQDDYiIULv2u4GqL0YLCTr71yxx2hP/KNG183WgtRujDYyddEBAV2PztHxlCmYLCT751WVAQAaG1txjvvPGG4GqLUY7CT7y382tdw8xknAgDefvtRw9UQpR6DnTLCAR0EAFBtN1wJUeox2CkjbNm/33QJRGnDYKeMkNPXmqFatY0XKxm2cuVA1NTkdnmupiYXK1cONFSR/zDYKSM8csMNoeWnnrraYCVUXNyCioqiULjX1OSioqIIxcUthivzD8eCXUT6iMiHIvKqU9skcsrZw4djgn3XLtVWs8VkuEAgiOnT96GiogiVlYNQUVGE6dP3IRAImi7NN5xssd8HYKOD2yNy1A1jx4aWW1qaDVZCgUAQZWWNWL58EMrKGhnqDnMk2EVkJICrATztxPaIUmHG+ecju7/V1/6LX15kuJrMVlOTi6qqASgvP4yqqgHH9blT7zjVYv81gAcA8NI+ci0RwY/KpwAA2tsaDFeTuTr71KdP34fJkw+HumUY7s7pdbCLyDUA9qrqB3Fed7eIVIlIVV1dXW93S9QjNQcPhpZV1WAlmau2NqdLn3pnn3ttbY7hyvzDiRb7RADXichWAC8BuFxEXoh8karOUdUyVS0bOnSoA7slSt7FJx+7m/0jj4w3WEnP+GGo4MSJDcf1qQcCQUycyG9RTul1sKvqQ6o6UlVLANwC4G+qemuvKyNKgS+MGYNx9lS+DQ170Np6xHBFycmEoYJ+OHiZxnHslHFuvfDC0LLXZnzMhKGCmXDwSjVHg11V31bVa5zcJpHTivLzQ8svv3yvwUp6xu9DBTPh4JVqbLFTxrn89NPx8HXXAQCqqyvR0LDXcEXJyYShgn4/eKUag50y0vhTTgktt7Q0GawkOZkyVDATDl6pxGCnjLS/2ZtXnmbCUMFMOXilEoOdMpKELVdXv22qjKRlwlDBTDh4pRqDnTLSOcXFoXB/443ZqK/fY7QeOiYTDl6pxmCnjNQ3Kwt/veee0OPdu9cbrIbIWQx2yljFhYWhVrvXRsYQdYfBThkrPycHpedaY9r/8pcforn5gOGKiJzBYKeMtuj6++0lxZo1LxmthcgpDHbKaFlybHxMezvvrET+wGAnsr297FHPTQpGFA2DnTLe9ydPtha0HW+99XOzxRA5gMFOGe+b5eWh5ebmQwYrIXIGg50IQD8PXdTI+copHgY7EYDzT7GSffv29w1XEh/nK6d4GOxEAB4ovw0AUF9fiyVLfmq4mu5xvnKKh8FOBOD8kSNDy9u2rTZYSWKcnK+cXTv+w2AnsnWOaG9q2m+0jkQ4OV85u3b8h8FOZBuRb/WzHz68C+++O8dwNbE5PV85u3b8h8FOZFv+rw+EllevftpgJd1LxXzlvBWdvzDYiWzZffrglMGDAQCqaria2FIxXzlvRecvDHaiMLOvvBIA0Ni4Fxs3vmG4mvTgrej8h8FOFGbKmWfipALrNOqiRd8zXE168FZ0/sNgJ4rQ1tYPANDe3ma4EudFG9pYXNxyXIjzVnTexmAnipDdpw8AoLW1Gfv3bzVbTAKSGYceb2gjx7T7A4OdfGHAypXIranp8lxuTQ0GrFyZ9LaeuPHG0PKTv7mq17WlWjLj0OMNbeSYdn9gsJMvtBYXY0hFRSjcc2tqMKSiAq3FxUlv66KSEpxeVAQA6Ohw/7C/ZMehdze0Mdq2Pve55uO2wVa8uzHYyReCgQAOTJ+OIRUVKKisxJCKChyYPh3BQKBH2zvrpJMcrtBZkV0mgUAQo0cfSWgceryhjZHBP3ZsM1vxHsNgJ98IBgJoKitDwfLlaCor63GoA8BDU6ZYC9qBjo52hyp0TmSXyapVA7BuXT5KSxu7HYeeyNDGyOAHwCtTPYbBTr6RW1OD/Koq1JeXI7+q6rg+92QMKygAsgSA4tFHJzpXpEPCu0wWLBiCJUsGY+rUg5g27UC349DjDW2MFfwAeGWqhzDYyRc6+9QPTJ+O+smTQ90yPQ33vllZKBtu9c/X1+92slTHdHaZrFs3AKWlTZgwoTH0fKxx6PGuWo0V/B991J9XpnoIg518Ibu2tkufemefe3ZtbY+3OSQ/36nyUiK8y6S6ut9xfe49GYceLfgBYNOm/rwy1UMY7OQLjRMnHtenHgwE0Dix590opSNGhJaffnpaj7fjtJUrB2LVqgFdukwmTTqMuXOHpiRseWWq9zDYiWK4r7wcJ/bvDwDYtWuN4WqOKS5uQWVlISZNOoxAIIiamlysWDEIkycfSknYpmLSMUqtvqYLIHKz+y+/HN9/9VXTZXQRCAQxY0YdKiqKcPRoH1RVDeAoFeqi1y12ERklIpUi8rGIbBCR+5wojMgNvnzBBaHlpUt/ZrCSrjh/OnXHia6YNgD3q+oYAOMB3CsiYxzYLpEr3H6WdbHSu+/+L9rajhquxpKq+dM5V4w/9DrYVXW3qq6xlxsAbAQwovvfIvKOi886dgK2tTV6yzidgZjK+dM5V4w/OHryVERKAIwD8F6UdXeLSJWIVNXV1Tm5WyLj0hmIqRylwvuf+oNjJ09FZACAVwB8R1XrI9er6hwAcwCgrKzMvfcdI+rGP/7xMUpKJhz3fHgglpU1pvSEZrTRKIFA0LF9hfffl5cfZqh7kCMtdhHJhhXqL6rqn5zYJpFbXHr66aHl55+/BS0tR6K+zi8nNGtqcrFq1cDj5p1hX7t3ODEqRgA8A2Cjqv6q9yURucugvDy88JWvhB4fPBh9mgI/3BC6swtp8uRDqK7uh0mTDqOioih0QRT72r3Bia6YiQBuA7BeRNbaz/1AVV9zYNtErnDp6NHIyRa0tCqCwabj1oef0AwEgigpOerJ/unw/vthw1pRUVGE0aOPoLKyEDNm1HnqvWQyJ0bFrFBVUdVSVT3P/mGok++cUGS1g5577ma0tXUNOL9cdh9+lWn4JGMTJkSfQ4bciVMKECXohRvutJfa8emnlV3W+fGyez90LWUqBjtRgoYOPHbisL291WAlqZfKsfKUegx2ogTl9T12Smrx4geh2pH2GtJ1IZRfupYyFYOdKEH9srNx/RhrtoyWlka8//4f0l5Dui6E8mPXUiZhsBMl4fEvfjG03NCwN+3755WhlAgGO1GSzhhmdcls2vSmkf375UIoSh0GO1GS/u3yLwEA9u37FKtXP5P2/ad6tApnePQ+BjtRkiaHTTGwdetx892lVDpGq3CGR+9jsBN5SCpGq0S20AOBYOgeqtH68dmidz8GOzluwMqVyK3pOp9Kbk0NBqxcaaii1Glu3p/W/aVitEpkC33x4sFYtqwQZ53VHOrHBxAKbrbo3Y/BTo5rLS7GkIqKULjn1tRgSEUFWouLDVfmnH59rP/u2FGFTz75q9lieilypM2GDfno6FBs3twf5eWH8d57AzFv3rHgTnRkDlv25jDYyXHBQAAHpk/HkIoKFFRWYkhFBQ5Mn45gIGC6NMe8f/8DoeXKyv8xWIkzwkfanHlmM1QFHR2dt01QAII9e7JDoZzIyBy27M1hsFNCku1eCQYCaCorQ8Hy5WgqK/NUqCfyXgf164evjR0OAOjoaEtrfakQPtJm48b+OO+8RgCC5csH4eKLG3HppYdQWVkYCuVERuZwzL05DHZKSLLdK7k1NcivqkJ9eTnyq6qOC0o3S/S9BgblAQD27t2MPXs+TnudTokcaTNjRh3Wrx8AQJGV1YEVKwZi2bJj0/auWjUAc+cOTWhkDsfcm8Fgp4Qk073SGYQHpk9H/eTJod/zSrgn+l5vv+K20PJLL92V7jIdEznSxqI455xmjB3bjI6OLLTac57V1OSisrIQkycfSmhkDmeINIPB7lOpGJmSaPdKdm1tlyDsDMrs2toe7zvdEnmvIhKaGKyt7WjMbbn9JGLkSJva2hzcfPM+jB3bjOrqfigtbURHh2DJkkJUVBRhxow6TJjQ2GUb0UbmcIZIcxjsPpWKkSmJdq80Tpx4XBAGAwE0TpzY432nW6LvNUsEANDUtB9Hj0Yfcui2k4jxDjSdAd0ZytOmHUBpaRP27MnF6NFHEu5O4QyR5jDYfcrpkSle715JRjLvdfZVV9lLiiefnBJ1e247iZjIgSY8lGtqckMt940b+yfc4uYMkeYw2H3MyZEpfuheSVQy73XGuHEYUVAAADhy5FDMbbrpJGIiB5rOUA7vTpk27QBmzKhjd4oHMNh9zMmRKd11r7jlSlOn6ki2K+nXN94IwLqrUqybb7jtJGKiBxp2p3gTg92n0tl14pYrTU3VUTp8OACBajuefPKfj1vvxpOIiR5o2J3iTQx2n0pn14lbrjQ1VUf/nByUjbAOHvv2bTluvdtavW480JCz+sZ/CXlRtG6DYCCQspAL78+vLy83dqWpqTqCHbHvfxqtdRsIBI31s3d3oOEFRP7AFjs5wi1XmpqvQ+O/xDB2r/gfg516rXDxYgyZN69rf/68eShcvDj0mnScYDU5JPPUIUPsJcWf//xAt68lSjUGOzlC4jxOx4lNk0MyH7vpptBVqGvXzkv5/sK5/cpWSj8GO/XaoWuvxf6bb+5y0nL/zTfj0LXXhl6TjhObJq94zRLBpoceSvl+onHbla1kHoOdHJHIxVBenso3ESLHvqd8+OHLaduv265sJfMY7OSIRE5amj+xmVpZIjixwAr3RYu+m9Z9u+nKVjKPwU69lshJy0yZa2bGeZcY2a/brmwlsxjs1GuJnLTMlLlm+mYd+ye1d+/mtOyTFxxRJAY79VoiJy39MJVvIm4ZNy60/JvfXJmW2+a57cpWMo/BTuSgYQUF+PGVV9qPtNsZH53CC44oEoOdyGF3jB8PyYocyU+UPo4Eu4h8XkQ2i0i1iDzoxDYpPdwy5a5ftbenviuGKFKvg11E+gB4AsBVAMYAmCEiY3q7XUoPp64I5QGiK1VrzphHH5scWiZKFyda7BcBqFbVLaraAuAlANc7sF1yUKzg7Ryt0tsrQt0yJ7tbfKn0XABAR3sz9uz52HA1lGmcCPYRAHaEPd5pP0cu0l3wOnFFaOfwxRPmzsXgBQu6HCAyseV+7yXHxrO3t7carIQyUdpOnorI3SJSJSJVdXV16dot2bqbq8WpK0KDgQCOnHUW8tetw9HRo5FdW4sBq1Z1ablnSsj3z84OLS9Y8B2DlVAmciLYdwEYFfZ4pP1cF6o6R1XLVLVs6NChDuyWkhWtZe7kFaG5NTXIq65GU2kp+q9bh7wtWzBoyRI0TJrUZV+Z0D0zrKAAZ514IgDgwIEa1NT4/2BG7uFEsL8P4HQRCYhIDoBbACxyYLvksGgtc6euCA0/QBycNg3NpaXI27IFwVNPxcAVKxyf0dELJ2vfnDULhYP6AADq6/cYroYySa+DXVXbAHwTwJsANgKYr6obertdclaslnlnH3u4nlwRGn6ACG+55+zYgaOjRzs+o6MXTtaKCDpnGNiwYXH3LyZykCN97Kr6mqqeoaqnqepPndgmOSvVc7V0ThkQ2XKvnzwZ/detQ1NpqaMzOrrlBtrx3Hj6BQCATz+txKefVhquhjIFrzzNEOmaqyWy5T5wxQocnjoVrSee6PiMjl6Y3/2HU6eGlj/77B2DlVAm6Wu6APKX8ANF5LcEAKFvCU6EcOQ5g2BJievCvW+fPqHlhobdBiuhTMJgp5SJ9m0gGAg4Furh3S/BkhLXdsdkZwGtHcDHH7+G7dvfx8knX2i6JPI5dsWQJ3V3zsBtI2be/3/H7qb0zjtPGKmBMguDnTypu3MGbhsxc0J+PiYNLwAAqHYYqYEyC7tiyHfCR8w0lZUhv6rKeBdNXnZnXzsnBKPUY4udfMltI2aaW9sBWCNjGhv3G62F/I/BTr7k1Pw3TvnKhCn2kmLevLuM1kL+x2D3CbedMExEqmp2cv4bp1x3zjmhf2xHj9Ybq4MyA4PdJ9x2wjARydac6IEg1VfZ9lTnP7YDB2p4ZyVKKQa7T3jlEvtwydac6IEgXVfZJuur48cDADo62vDcc9ON1kL+xlExPhJ+wrC+vNzVod4pmZrdONolGT+68kqoKp5+7z0cPLjNdDnkY2yx+4jbThgmItma3TbaJVlnDxsGAGhpaTZcCfkZg90n3HjCMJ6e1OzFg1e48SUlAIC2tqP44x+/arYY8i0Gu0+49YRhd5Kt2YsHr0gjCwuRa08MtmXLCsPVkF+xj90nUjnhVqokW3N3BwI3v89IJ+Tno7aeQx4pdRjs5BlePHh1j9MLUGqwK4YozYbm5wMA2ttbsXr1M4arIT9isBOl2byZM9HPnhNsyZKfmS2GfInBTpRm+Tk5WPWv1hztnMaXUoHBTmTAax9/bC91YMeONUZrIf9hsBMZcMlpp4WWn332JoOVkB8x2CkqJ2de9OLMk6lWMmQIrjv7bPsRu2PIWQx2isrJ2SK9OPNkOuT1PTbauLn5oMFKyG8Y7BRVvJkXk2mFe3HmyXS49cILQ8u/fPhCqHJcOzmDwU4xQzq7tjbmhFvJtsK9PnlXKowbMQJPTben79VWjpAhxzDYKWZIIysr5oRbybbCvT55V6pcHepnB1TbDVZCfsJgdwmTJxijhXTDpEkYuGJFtxNuJdoK98PkXakkWQIAePjhC+O8kigxDHaXMH2CMTKk0dERd+bFRFvhXpx5Mp2uONUa+nj06CE0NtYZrob8gJOAuYTpuwNFhnS0fYdPuBXeCg8GAgiWlMTsjvHf5F3OmnnxxXiruhoA0NHB7hjqPbbYXcTUCcaedJWwFU7kXgx2FzF1grEnIe3WG0Z7UX5ubmj59df/w2Al5BcMdpcweYIx1SHNK0+7d8HIkTihf38AwKZNb2D/fp5Upt5hsLuEn7s2TJ8YdjsRwdrvfQ/98qzRMQ0N/zBcEXkdT566hJ9PMJo+MewFzS0tCLZYV55WV7+NkpLxhisiL+tVi11Efikim0RknYgsEJFCpwojf+GVp93Ly85Gln3h6cqVv0FdXbXZgsjTetsVsxTAWFUtBfAJgId6XxL5Ea887V6WCN7/7ndDj7duXWWwGvK6XgW7qi5R1Tb74WoAI3tfEvkNrzxNzICw0TFNTfsNVkJe5+TJ0zsAvB5rpYjcLSJVIlJVV8er6zKJn08MOymnT5/Q8rJlj+DQoZ0GqyEvixvsIvKWiHwU5ef6sNf8EEAbgBdjbUdV56hqmaqWDR061JnqyRM45j0xWSJ48+67kWX/q3z33TlmCyLPijsqRlWndLdeRGYCuAbAFcoJpYl6Zczw4Th7RF+s39HG2R6px3o13FFEPg/gAQCXqmqzMyURZbadB6xAZzuJeqq3feyPAxgIYKmIrBWRpxyoiSijHWyyAn3Nmvloawsaroa8qLejYkar6ihVPc/+meVUYUSZ6l773INqKxYtesBwNeRFnFKAyGUenHLstFZ9PacXoOQx2IlcbM+eDexrp6Qx2Ilc6NLTrLsqBYP1WLjwfsPVkNcw2Ilc6IVbb8XtF1wAANi58wPD1ZDXMNiJXOqz/da0AkeP1huuhLyGwU7kUnUNDQCA5uYDWLr054arIS9hsBO51NJ774XYy2vWzDVaC3kLg53IpbJEMNCe8ZHTC1AyGOxELlZo3ws1GGzEhx/ON1wNeQWDncjF/nLXXehj98e8/vqPzBZDnsFgJ3Kxwn798MbXrZk62to7DFdDXsFgJ3K597dvt5d4BSolhsFO5HLPrF4NANCOFtTX7zZcDXkBg53I5X529dWh5UcfvcxcIeQZDHYil/unU0/FucOHAwA6Ojg/O8XHYCfygPycHHtJ0dbWYrQWcj8GO5EHzBg3LrT8i4f/yWAl5AUMdiIPuOHcc/H2N74BAGgN1hmuhtyOwU7kEX2y+M+VEsO/FCKPWLVtW2j5ySf/2WAl5HYMdiKPmHH++QgMHgwAqKv7BO3tPIlK0THYiTzkzgkTQsuqnGKAomOwExH5DIOdyEOOjWcHVqx4ymAl5GYMdiIPuW7sWFwxqhAAsGzZIwgGGwxXRG7EYCfykOw+ffDErfdA7H+5R44cMlsQuRKDnchj1u/ejc7zprt2rTNbDLkSg53IY84tLg4tV1R8A83NBw1WQ27EYCfymH7Z2Vh4xx2hx7W1bLVTVwx2Ig8aUVgYWuYJVIrEYCfyoMJ+/ULLFa98G8Fgo8FqyG0Y7EQelNe3L/5y553IyhZA27F27XzTJZGLMNiJPOqc4mKckGstt7W1mi2GXIXBTuRRCqCuUQEAf/3b/6C9neFOFkeCXUTuFxEVkSIntkdE8WWJ4F/KygAA2hFEZeWvDFdEbtHrYBeRUQCmAtje+3KIKBk/ufrq0HJjI++sRBYnWuyPAHgA1jdDIiIyrFfBLiLXA9ilqn93qB4i6qFduz40XQK5RNxgF5G3ROSjKD/XA/gBgB8lsiMRuVtEqkSkqq6OXxmJnHJTaSkAYN++aqxY8aThasgN4ga7qk5R1bGRPwC2AAgA+LuIbAUwEsAaERkWYztzVLVMVcuGDh3q5Hsgymi/njYNl4woAABs3rzUcDXkBn17+ouquh7AiZ2P7XAvU9V9DtRFREl4Z1c9AODo0cOGKyE34Dh2Ih8YnGP9U9637zOsXVthuBoyzbFgV9USttaJzFj93QdDy8uW/dpgJeQGbLET+UD/7GxkZ1n/nDs62gxXQ6Yx2Il8Iq+vdcqsvn43tm9/33A1ZBKDncgnFt91F4YWWeE+f/4sw9WQSQx2Ip84ragIP5k8DQDQ1hY0XA2ZxGAn8pEHFy8GAKhyho9MxmAn8pEsEQBAS0sjWluPGK6GTGGwE/nI/ZdfHlp+7LFyg5WQSQx2Ih+5rawMxQMGAAAam/YbroZMYbAT+Ux+rn2/PO1gX3uGYrAT+cyMCy6wlxRz5lzd7WvJnxjsRD5z14QJeGXmTADAnj0fmy2GjGCwE/nQk+++ay+xKyYTMdiJfOjUIUNCyy+9dJfBSsgEBjuRD/371KkYkJMDANi8eYnhaijdGOxEPiQiuGXcONNlkCEMdiKfKsjLCy2vXv2swUoo3RjsRD41a+JEjB5khXvl6r8YrobSSUxcwCAidQC2pX3H3SsC4JU7QHmpVsBb9XqpVsBb9XqpVsCd9Z6iqkPjvchIsLuRiFSpapnpOhLhpVoBb9XrpVoBb9XrpVoB79Ubjl0xREQ+w2AnIvIZBvsxc0wXkAQv1Qp4q14v1Qp4q14v1Qp4r94Q9rETEfkMW+xERD7DYI8gIt8SkU0iskFEfmG6nnhE5H4RUREpMl1Ld0Tkl/bnuk5EFohIoemaIonI50Vks4hUi8iDpuuJRURGiUiliHxs/53eZ7qmRIhIHxH5UEReNV1Ld0SkUEQq7L/XjSIywXRNyWKwhxGRyQCuB3Cuqp4N4GHDJXVLREYBmApgu+laErAUwFhVLQXwCYCHDNfThYj0AfAEgKsAjAEwQ0TGmK0qpjYA96vqGADjAdzr4lrD3Qdgo+kiEvAogDdU9XMAzoU3au6Cwd7VPQD+S1WDAKCqew3XE88jAB6AB+ZmVdUlqtpmP1wNYKTJeqK4CEC1qm5R1RYAL8E6yLuOqu5W1TX2cgOs4BlhtqruichIAFcDeNp0Ld0RkUEAygE8AwCq2qKqh8xWlTwGe1dnALhERN4TkWUicqHpgmIRkesB7FLVv5uupQfuAPC66SIijACwI+zxTrg8LAFAREoAjAPwntlK4vo1rEZIh+lC4ggAqAPwnN1t9LSI5JsuKll9TReQbiLyFoBhUVb9ENbnMQTW19sLAcwXkVPV0NChOLX+AFY3jGt0V6+q/tl+zQ9hdSW8mM7a/EhEBgB4BcB3VLXedD2xiMg1APaq6gcicpnpeuLoC+B8AN9S1fdE5FEADwL4d7NlJSfjgl1Vp8RaJyL3APiTHeT/JyIdsOaLqEtXfeFi1Soi58BqWfxdRACrW2ONiFykqnvSWGIX3X22ACAiMwFcA+AKUwfLbuwCMCrs8Uj7OVcSkWxYof6iqv7JdD1xTARwnYh8AUAegAIReUFVbzVcVzQ7AexU1c5vQBWwgt1T2BXT1UIAkwFARM4AkAP3TQIEVV2vqieqaomqlsD6YzzfZKjHIyKfh/VV/DpVbTZdTxTvAzhdRAIikgPgFgCLDNcUlVhH82cAbFTVX5muJx5VfUhVR9p/q7cA+JtLQx32v6EdInKm/dQVADx349iMa7HH8SyAZ0XkIwAtAP7FhS1Lr3ocQC6Apfa3jNWqOstsSceoapuIfBPAmwD6AHhWVTcYLiuWiQBuA7BeRNbaz/1AVV8zWJOffAvAi/YBfguArxquJ2m88pSIyGfYFUNE5DMMdiIin2GwExH5DIOdiMhnGOxERD7DYCci8hkGOxGRzzDYXUREThaRRnsKWXKAiPxORH7Sy218RUSWJPC6p0TE8TlFRGS2iLzg9HZ7UEeJPfc/L2x0OQZvM/OWAAAIMUlEQVS7ASKyVUSO2CHe+VOsqttVdYCqtrugxvEislREDohInYi8LCLDk/j9s0Vkif37h0TkAxH5goiMEJE2ETktyu8sEJGH7WUVkb3hISIi2fZzab2qTlVfVNW4E66p6ixV/c901ETUHQa7OdfaId75U5vKnfWglTUY1s18SwCcAqABwHNJ/P5iWDfXGAbgRADfBlCvqrsA/BXWJfHh9Q0B8AUAz4c9fRDWjS86XWU/Rx7DVn56MdhdJPKrrj0h1XIRaRCRt0Tkic6v5CJymYjsjPj9rSIyxV6ebd/e6wURqQcwU0SyRORBEflMRPaLyHw7UI+jqq+r6suqWm9P2vU4rDlKEnkfRbBmn/ytfaOCFlVdqaor7Jc8j4hghzU51Mequj7suT8AuD3s8e0Afh9n3+NEZI39mc2DNZtg+PprRGSt/S3iXREpDVs3SkT+ZH9D2S8ij9vPzxSRFfayiMgj9jeHehFZLyJj7XVdun1E5C6xbrN3QEQWiUhx2DoVkVki8qldyxP25F6x5InIPPt9rRGRc8O2dZaIvG1vZ4OIXBe27m0RuTPscei9xKtDrFvZPSwi+0RkC6wbZYR/ll8V69ZxDSKyRUS+HrbuMhHZKSLfF5E9sOY3/0hErg17Tba97XHdvG/qAQa7u/0RwP8BOAHAbBwfhvFcD2va0UJY859/C8ANAC4FUAyr9ftEgtsqBxCaFEtEviwi62K8dj+AagAviMgNInJSxPoFAIpEZFLYc7eha2sdsGbbLBfrHpSDAVwC4M+xChRr0qaFsA4IQwC8DOCmsPXjYE309nVYn+n/AlgkIrlindd4FcA2WN9SRsC6i1KkqbA+izMADALwJfv9RtZyOYCf2+uH29uN3N41sOb9L7Vfd2Ws9wbr/+XL9vv6I4CFdjBmw/p2tATWN6POCazOjLml48Wq4y573TgAZQCmR/zeXnt9AayJsh4RkfPD1g+z6z0FwN2wDsrhszp+AcBuVf0wiVopEarKnzT/ANgKoBHAIftnof18Cazb3PUFcDKsG1L0D/u9FwC8YC9fBmve6MjtTrGXZwNYHrF+I6y50DsfDwfQCqBvnHpLARwAcEkS73EkrFb+Z7DumrMcwOlh658GMMdePh3WbJonhq1XAKPt130dwCwAv7Wf0xj7LAdQC3tyO/u5dwH8xF7+DYD/jPidzbAOdBNgzbt/3GcBYCaAFfby5bDu2ToeQFbE634Xtq9nAPwibN0A+7MuCXt/k8LWzwfwYIz3NRvWbJidj7MA7IZ1oLsEwJ7wWgDMBTDbXn4bwJ3R3ku8OgD8DcCssHVT7ddH/XuBdVC9L+zvswVAXtj6YlhdegX24woAD5j+9+jHH7bYzblBVQvtnxuirC8GcEC7zl2+I8rruhP5+lMALLC/ch+CFfTtACJb1CEiMhrWbezuU9V3Et2xqu5U1W+q6mn2fpvQtRvleQBfFJE8WK31NzX6PWZ/D6sLJm43DKzPbJfaqWHbFrZ8CoD7O9+//RmMsn9vFIBteuy+rLHe199gHbCeALBXROaISEGMWraF/V4jrJZ9+O32wufPb4YV/rGE/l+qagesOfiL7Z8d9nOdtiG52/rFqqMYXf+Gwj9LiMhVIrLa7mo6BKsFXhT2kjpVPRpWdy2AlQBuEpFCWOdMeCetFGCwu9duAENEpH/Yc+F3+GkCEFpndyUMjdhG5OiRHQCuCjugFKpqnlonNI8jIqcAeAtWK/cPPX0jqroDVhCODXt6BaxvAdfD+noe2Q3T6R1Y3yxOsn+nO7sBjIjoqz45bHkHgJ9GvP/+qjrXXneyJHCST1UfU9ULAIyB1SXzvSgvq4V1IAEAiHXfzBPQ87syhf7fi0gWrG9EtfbPKPu5TieH7afL3wmi37owlt3o+jcX+ixFJBfWHZweBnCSqhYCeA1A+GcfbfTS87D+f38RwKpYf3vUOwx2l1LVbQCqAMwWkRwRmQDg2rCXfALrhNrVdj/rv8G6kUV3ngLwUzuwISJDxbop9nFEZASsr+KPq+pTydQuIoNF5MciMlqsE7ZFsG5gvTrs/SmsFvh/wzoHsDjatuzXXQvrzkvxhjmugtV99W27//lGABeFrf8tgFkicrF9EjTf/vwGwjqXsRvAf9nP54nIcSeLReRC+/ezYYXmUUS/QfNcAF8VkfPsEPwZgPdUdWuc9xDLBSJyo33g+Q6AIKzP8z1YrewH7Pd8GazPq7M/fy2AG0Wkv/3t62tJ7HM+rM9ypH2OI/wWcTmw/t7qALSJyFVI7B68C2HdU/Q+xP8GRj3EYHe3r8Dq+90P4CcA5sH6Bw1VPQzgG7D6oHfBCpmd0TcT8iis270tEZEGWMFwcYzX3gngVFgHltB4+86VYl20E+sOQy2wzhe8BaAewEd23TMjXvd7WK3AeaoajFW0qm7QBO5mpKotAG6093MAwM0A/hS2vgrWCcHHYZ04ru6sSa1rB66F1Ye/HdZneXOU3RTAOkAchNU1sR/AL6PU8hasGyC/AuuAcRqskT899We7noOwuq5uVNVW+z1fC6tbYx+AJwHcrqqb7N97BNb/j3/Aai0n0/XxW1h3lPo7gDXo+lk2wBrCOt+u6ctI4FaCqnoE1mcSCN8eOYt3UPIQsYbvbVLV/zBdC1FPiciPAJyhLr3vqR+wxe5i9tf+0+zujM/D6o9eaLouop4S67qJr8G6+I1ShMHubsNgDVdrBPAYgHuUY37Jo0TkLlgnqV9X1eWm6/EzdsUQEfkMW+xERD7DYCci8hkGOxGRzzDYiYh8hsFOROQzDHYiIp/5/w+zZkLAXuH2AAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "_uuid": "835d5be8da0bf460e03d0b95a95c6b06fa6727d0"
      },
      "cell_type": "markdown",
      "source": "We can check the support vectors and plot them together with the separating hyperplane"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "282b83ac5feca00c626be63acdb19d2735fbf967"
      },
      "cell_type": "code",
      "source": "sup_vect = X_train[alpha > 0]\nsup_vect",
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 14,
          "data": {
            "text/plain": "array([[0.15695433, 1.0690201 ],\n       [0.76728419, 1.90999537],\n       [1.06908872, 0.91924923]])"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "dc5a969d0d5e50c1b52abd8c6acbf06eb9e83bbd"
      },
      "cell_type": "code",
      "source": "plt.plot(x0[:,0],x0[:,1],'x',color='red')\nplt.plot(x1[:,0],x1[:,1],'x',color='blue')\nplt.figtext(0.5, 0.01, 'Figure 3: support vectors defining the separating hyperplane', \n            wrap=True, horizontalalignment='center', fontsize=12)\n\nx = np.linspace(-5, 5, 200)\ny = np.linspace(-5, 5, 200)\nX, Y = np.meshgrid(x, y)\nplt.contourf(X, Y, Z, alpha=.5, cmap='jet_r')\nC = plt.contour(X, Y, Z,  colors='black',zorder=4)\nplt.plot(sup_vect[:,0],sup_vect[:,1],'o',color='green')\n\n\nplt.axis('equal')\nplt.show()",
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEPCAYAAACqZsSmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmYFNW5+PHvOysMAwMIBAeEaYUYF4iaiUpADHGJJrhFvErMokmuV5Oo+V2zuSQxud7c3JjF3CdeE41JjAvRTFxuNIkaRc0gLuMSUFBRBoMMyLANDDDDLO/vj1Pd1PR093QP1VO9vJ/n4aG6q6f67erqt06dc+ocUVWMMcYUl5KwAzDGGDP0LPkbY0wRsuRvjDFFyJK/McYUIUv+xhhThCz5G2NMEbLkb4wxRciSvzHGFCFL/sYYU4TKwnrjcePGaV1dXVhvP2T2tLSEHUJR2rprF++0tQEwceLhiEjIEZlsqq2tCDuEIfHCCy9sUtXxQWwrtORfV1dHU1NTWG8/ZN659tqwQyhKi1et4jN33QVAV9cuvvzlxSFHZLLp2munhB3CkBCRt4PallX7mII0b/p0Dh7vCkibN69m585NIUdkTG6x5G8K1ndPPTW23N3dGWIkxuQeS/7GGFOELPmbgjV9/Hjw2nmfe+72cIMxJsdY8jcFa0J1NZfMqAXg6advYtu2d0KOyJjcYcnfFLR/O/mTseUtW9aEF4gxOcaSvylond3dseWNG98IMRJjcoslf1PQJowcGVt++OHvsn37+hCjMSZ3WPI3Ba2spIQnvvSl2OMVK/4SYjTG5A5L/qbgTRkzJrbc3d0RYiTG5A5L/qbglZaUUO4d6Y89dj27d28NNyBjcoAlf1PwSkR47cqrKassBXp54okbwg7JmNBZ8jdFoaKsDO3qAaCry6p+jLHkb4pGT2/YERiTOyz5m6KzYsWDqGrYYRS1JUtG0txc2ee55uZKliwZmeQvTNAs+ZuiccrBBwPQ2dnOQw9dE3I0xa22dg8NDeNiJ4Dm5koaGsZRW7sn5MiKR6DJX0RKReQlEXkwyO0aE4Rbzjsvttzaanf7hikS6WTBgk00NIxj8eIaGhrGsWDBJiIRG3p7qARd8r8cWBnwNo0JTG1VOQDbtq0NORITiXRSX9/OU0/VUF/fbol/iAWW/EVkMvBx4FdBbdOYoP383E8BsH37eh599L9Cjqa4NTdX0tRUzdy5bTQ1VfdrAzDZFWTJ/wbg64D1qTA564NTpjB2+HAAVqx4KORoile0jn/Bgk3Mm9cWqwKyE8DQCST5i8h8YKOqvjDA6y4SkSYRaWptbQ3irY3J2PZOV71gUzuGp6Wlok8df7QNoKWlIuTIikdQJf/ZwOkisgb4PfAREbkj/kWqerOq1qtq/Xhvcm1jhtq4qioA2ts38txzt4UcTWYKpYvk7Nk7+tXxRyKdzJ69I6SIik8gyV9Vr1TVyapaB5wHPK6qnwpi28YE7cnLLostL1784xAjyVyxdJEslJNcLrN+/qboVJWXUzNsGACqPSFHk5li6SJZLCe5MAWe/FX1CVWdH/R2jQlStNG3s7Odt99+LuRoMlMMXSSL5SQXJiv5m6L05KWXEqlzpf/f3X5BqLFkqli6SBbDSS5MlvxNURIR3l89DQDtzZ+qhGLqIlksJ7mwWPI3RWvZunXeUv4M8lYsXSSL6SQXlrKwAzAmLCICgGo3XV27KS8fHnJEA0vUFTIS6Sy4KpFUJ7lC+6xhsZK/KVqXH398bPmH1x8bYiQmnt0HkH2W/E3ROmvmTA7xbjbs7moLORpjhpYlf1PUzpgxw1tSm+DFFBVL/qaoXTx7Nnh1/zfddFLI0RgzdCz5m6JWWlLCN7y6/9bWVSFHY8zQseRvit7GXbvCDsGYIWfJ3xS9KaNHx5bvuutzIUZizNCx5G+K3uePPTY20NuqVY/ZOP+mKFjyN0VPRLj6pL2Nvb29+TXSpzGDYcnfGGDauHFhh5A2G+veBMGSvzG4uX29Hp+8/vqj4QYzABvr3gTBkr8xnhMOdcn03nsvY8+e3O0BZGPdmyBY8jfG868fODe2vHPnphAjGVjQY91bVVLxseRvjKeybO8gt21tLSFGMrCgx7q3qqTiY8nfGM90b5A3gNtuOy9nq36yMda9VSUVH0v+xnhGDRvGLeec4z1SmpufDjWeZLI1oYtNm1hcbDIXY3w+dOCBsWXV7hAjSS5bE7rEVyXV1XXYCaCAWcnfGJ9Rw4YxqbYcgD80XF40d/vatInFx5K/MXH+cv5XAOjt6eCll+4OOZqhUSxzA5u9LPkbE2dMVVVsuaOjsKYNTNalE7BpE4uMJX9jUnjmmVvzYoavdPvpD9Sl0/r7Fw9L/qZgVC9ZQmVzc5/nKpubqV6yJONtzZo6FYBduzbz97/fGEh82ZRuP/2BunRaf//iYcnfFIyu2lrGNjTETgCVzc2MbWigq7Y2423dc8EFseVNm3J/hq9M+umn6tKZbDstLRV2RVBgLPmbgtEZibBlwQLGNjQwavFixjY0sGXBAjojkUFtrzzHfx3xVTSRSCfTpu0esJ/+QHcHJzo52BVB4cnxw9uYzHRGIuysr2fUU0+xs75+0Ikf4KiI6/K5ZcvbQYUXqPiEvHRpNcuWjWDmzPakQz6k06Uz0cnB7gAuPJb8TUGpbG5mRFMT2+fOZURTU782gEx8ZmodAOvWvcTLLzcEFGFw/An5vvvG8sgjYzj55K2cddaWpP30B+rSmerkYHcAFxZL/qZgROv4tyxYwPZ582JVQIM9AZw2Z2Fsuanp9qDCDFQ0IS9bVs3MmTuZNas99nyifvqzZ+9I2aUz1ckh6MHkTLhseAdTMMpbWvrU8UfbAMpbWgZV/SMiCKCQs3f6JquigcEN+ZBs6AigT1VPXV2HVf3kOUv+pmC0z57d77nOSGSf6v2Hlwq7epR3313JW28t4aCD+r9HGJYsGUlJidLYWBNLwMOG9bBo0XgWLmwNPCGnuiKw5J+frNrHmBSeuOwrseUHH/xGiJH0VVu7h8WLRzNnThuRSCfNzZU0NtYwb962rAzJMFB1kck/VvI3JoX9R43iPdXVvNveTk9PV9jhxEQinSxc2EpDwzg6Okppaqq2KhiTkUBK/iJygIgsFpEVIvKqiFwexHaNyQVfnTcPgB07NuTUDF/W+8bsi6CqfbqBK1T1UOBY4EsicmhA2zYmVOcccQTDhrnlW245PdxgfLLV+8bG9ykOgSR/VV2vqi96yzuAlcCkILZtTNhKS0qokuEA7NmzM+nrliwZyWMbH+QGPsR3qeMGPsRjGx/MStLM5vj7djdvcQi8zl9E6oAjgWcTrLsIuAhgypQpQb+1MVkjabymY/oiGsd+DcTN/dvGOhrHfo050wHmBxpPNnvf+G8eq69vt/aEAhVobx8RqQb+CHxFVbfHr1fVm1W1XlXrx/smyzYmX3R3d6Dam3Dd8gnfh/K4Sd/Ld7nnA5bt3jfWnlD4Akv+IlKOS/x3quq9QW3XmFyw8KijAFDt5cYbT0r4mjYSNwYnez6XNTdXsnTpyH7jBFndf+EIqrePALcCK1X1J0Fs05hc8o0TT2T6uHEAbN6SeLiIGhIPHZ3s+VwVreOfN28bb745nDlz2mhoGMfSpdVW919Agir5zwY+DXxERF72/n0soG0bkxNuO//8lOtnbLwKuqr6PtlV5Z7PI9H2hFmz2lmwYBONjTVMm7abxYtHW91/AQmqt0+jqoqqzlTVI7x/fw5i28bkiv2ic/tqD7ff/ul+64etWsicLddTo5NAhRqdxJwt1zNs1cJ+r81l/vYE/8Bxs2b1b2cw+cvu8DUmTVUVFRw4diyrt2xh9eqn+q13ja3zOcHfs2cCMCF/h0CIv5egrq7DTgAFwsb2MSYDB+63X9ghDJls3ktgwmfJ35gMTBi5t6fLY49dH0oMQ3UH7kATv5j8ZsnfmAx879RTY3WljY0/DyWGoboD10byLGyW/I3JQGVZGVeefHKoMdh8uiYIlvyNydBFs2bFlt9884lQYrA7cM2+suRvzCAcc1A5AHfe+dlQxvnP9ny6NrJn4bPkb8wgXHjUmbHlrq5dKV4ZvKHohWMjexY+S/7GDEJXT09o752NXjjxJf1IpJM5c9pYtGh8wnYFuzLIf5b8TVZUL1lCZXPfMXAqm5upXrIkpIiyp7Mz+Rj/2ZCNXjjxJf0//WkMTz45mkMO2RVrVwBiyd2uDPKfJX+TFV21tYxtaIidACqbmxnb0EBXbX4NcpbMEZP2zlX0s58dl3SY53wR34Po1VdH0NurvP56FXPntvHssyO5++69yT2dHkd2dZDbLPmbrOiMRNiyYAFjGxoYtXgxYxsa2LJgAZ2RSNihBaJu7Fi++uEPA6DazaZNb4UbUAD8PYgOPngXqkJvr3prFRA2bCiPJe+BehzZ1UFus+Rv0jKYapzOSISd9fWMeuopdtbX503ir16yhAc3Psax3MAUvsux3MCDGx/r91kvP/54SitKAejtDa8NICj+HkQrV1ZxxBHtgPDUUzUcc0w7xx+/jcWLR8eS90A9jux+hNxmyd+kZTDVOJXNzYxoamL73LmMaGrqd/LIVb+f3sEVYxtZJ22owDpp44qxjfx+eke/11ZVupLxbbedh6r2W58v4nsQLVzYyvLl1YBSUtJLY+NInnxyNAsXthKJdLJ0aTWLFo0fsMeR3Y+Quyz5m7RkWo0TPTlsWbCA7fPmxf42H04A101Yzq7yvs/tKnfPx/vvk1yXz927t9LcnL+N2fE9iBxlxoxdHH74Lnp7S+jybmdobq5k8eLRzJu3bcAeR9m+H8EMng3pXMCqlyyhq7a2T4KubG6mvKWF9tmzM96evxpn+9y5Katxylta+pwcoieP8paWnK/+aaEt7efnHHRQbLmjI/HfLVkyko7pi1g+4fu00UINtczYeBXDVi3MmXFy4uNoaang3HM3AdDQMI6ZM9tZtmwEjzwymu3by2JXAH6RSGef5/xXE5FIJ3V1HVb1k0Os5F/Agu5xk0k1Tvvs2f2SfGckMqiTzlCrpSbt50tL9v6EGhtvSvh3HdMX0Tj2a7TJOhClTdbROPZrdExfFEzAgzBQT5zoySCarM86awszZ+5kw4ZKpk3bnVbytlFBc5sl/wIWZI+bfK7GydQ1G2dQFTdiQ1WXez7eqGHDmFzjTgrr1y9nxYq/9HvN8gnfh/K4u4DLd7nnQ5JOTxx/8m5uruTNN4czc2Y7K1dWpVV9Y6OC5jZL/gUuqB43qapxCs15q4bx4y1zmKQ1iMIkreHHW+Zw3qphCV//9OWXx5Y3bFjRb30bifdRsueHQjo9caLJ2199c9ZZW1i4sNUmdSkAVudf4OKrajrr6gZ1AkhUXdMZidAZiQTetjBYQcXRPns283ETMsZMgPYJiV8vIhw4vpTVrT1s2PBqv/U11NLGuoTPh8nfE2fu3LakVTmpqm+s7j5/Wcm/gA1VVU2u3M0bZhyfmXYAAKtWPcYbbzzeZ92MjVdBV1VcsFXu+RCl2xPHqm8KkyX/AjZUVTW5cjdvmHF8/uTPxpZffvnuPuuGrVrInC3XU6OTQIUancScLdczbNXCrMeVjM3Pa6zap4ClqqoJWibdQLMpV+LwcyXk+ZzA/L1PTgAmhFdytqocYyV/E4hcuZs3F+Lo7s79sWusKsdY8jf7rLK5mf0WLWLHnDl92haqly7tMx5Otod5Drs7qnj/r1r1OJs2FV4XWFNYLPmbfVbe0sL2efMY2dhIZXMznZEIO+bMYdTixX0aW7PdIBt2d9T7Pv/52PIDD1wxJO8JNnSyGRyr8zf7LNq20DVxImMbGthZX8+IpiY2L1zYp87d3yAbfU2QDbJD2caRyAcmT+bCwybym1c30NXVfxC4bInesOW/ISv62JhkLPmbwKTT2JqLDbJBGlXhflLvvruSPXt2UVFRNcBf7Dv/DVv19e00NVXb+DlmQFbtYwKTTmNrLjTIZtMlH/2Ut9TLr3999pC9rw2dbDJlyd8EIp3G1rAbZIfCiMpKKkrdBC/t7a1D9r42dLLJlCV/E4h0GlvDbpAdKiKu38/u3VuHZIIXu2HLDIYlfxOIdIZwzudhnjPxiRlu9M/e3m5uvfUTWX8/GzrZDIYlf2MC9sPTT2dEuZsK7N13+4/yGTS7YcsMhiV/Y7Lg1wvduD0i9hMzuSmwI1NEThGR10XkTRH5ZlDbNdmX7Ttvi5lqT9ghGJNQIMlfREqBG4FTgUOBhSJyaBDbNtkX1J23dhLZa7hX7dPd3clf//q9kKMxpr+gSv5HA2+q6mpV3QP8HjgjoG2bgCRLztFeOPs6FHKujOufC2bsvz+lXq+fZ5/7bbjBGJNAUMl/ErDW9/gd7zmTQ1Il5yCme4x23dxv0SLG3Hdfn5NIsV0BlJWWcsjEie7BEHT3NCZTQ9oaJSIXiUiTiDS1tg7dDTDGSTXZSVB33nZGIuw+5BBGLFtGz6hRQN+TTDGdBGoqo/3se1m+/E+hxmJMvKCS/zrgAN/jyd5zfajqzapar6r148ePD+itTSYSlfCDvPO2srmZYW++yc6ZMynfsIFxd97JfnffzZYFCwCKqhro52fvHd7h3nsvDTESY/oLKvk/D0wXkYiIVADnAf8X0LZNgBKV8IO689Z/Etl61lnsmjkT6elBurupXLMm0GkV86FxeVx1Na9feWXYYRiTUCDJX1W7gS8DDwMrgXtU9dUgtm2Ck6yEH63z9xvMnbf+k4j/CgDVfWpLSCRfGpcryqID5yqbNq0ONRZj/AKr81fVP6vqe1X1IFX9z6C2a4KT7bF1osM3+E8yu444gt7ycnrLyxnx7LOBDeKWK5PGD6SspIQKb5SFG288IdxgjPGx2w+LyFCNrRM9yYCr499y7rlsXriQ3YcdFugonkH0UBoK5874gLfUG2ocxvjZZC4mcNGTSfWSJf2uNHYffjjlLS2BJOr49ovOurqcPAGMq66OLXd17aa8fHiI0RjjWMnfZE02rzTyaW6Aj0yfHlv+wQ/ePyTDPBszEEv+Ji+lar/ItZ5AR0yaxLlHHAFAb28nu3dvDSUOY/ws+Zu8lOqqIhd7An1//nyiA3xayd/kAqvzNwXH3xNoZ309I5qaQu8J5KZ2FMASv8kNVvI3BSnXegKpKtrrEv/tt58faizGgCV/U6CCGqsoKCJCZMwYAN59dyVbtrwdajzGWPIvILnW0JmObMScqz2BfvPJT8aWd+9uCzESYyz5F5RcbOgcSKYxp3OyyPadzINVWrL35/bss78JMRJjLPkXlHwZ8sAv05jTOVkM1Z3MmZo6ZgxVpW6Cl+XL7+Xdd1eGGo8pbpb8C0yuNXSmI5OY8/EEFyUiLP/mVQwf7k4AGzZY8jfhseRfYHKtoTMdmcacjye4qIqyMior3c9u48bXQ47GFDNL/gUkVxs6UxlMzPl4gvM7srYUgKef/gUbN74RcjSmWFnyLyC52tCZSqYx5+MJLt71H/1SbPnll+8JMRJTzOwO3wKSqEGzMxLJ6WqRTGNOdbLI5c/p9x5vbmNjwmTJ3+SVfDzBpdLT0x12CKZIWbWPMSF67rnb6OiwG77M0LPkb0wIfnT66d5SLw899O1QYzHFyZK/MSE498gjOWGKG+vHSv4mDJb8jQnJq607AFiz5mlUbX5fM7Qs+RsTkg+/9zAAurs7ueeeS0KOxhQbS/4mqSBH3MzHEUez7fozz4z9ADdvXh1qLKb4WPI3SQU5Smg+jjg6lHbs2BB2CKbIWPI3SQ00iFompfl8HpAtmz54wAEAdHRs5/77rwg5GlNMLPkbIHkiL29pSTqIWqal+XwekC1b/nDhhRwzZQoAq1YtDjkaU0ws+RsgeSKnpCTpIGqZlubzfUC2bBAR/uXIIwHo7e0KORpTTCz555AwG0UTJfIdc+YwsrEx5SBq6ZbmC2FAtmyZPm4c4Kp+nnzyf0KOxhQLS/45JOxG0fhETm/vgCNupluaz8cRR4fKYRMnIt5yY+ONocZiiocN7JZD/KXvnfX1jGhqGtJG0fhEnui9/YOo+UvznZEInXV1Sat+Cm1AtiBVlJUxceRI1u/YYTd7mSFjJf8cE1aj6GCqZaw0b0z+suSfY8JqFB1MIs/VidLzUVV5OQA9PXtoaXkl5GhMMbDkn0PCbBTNZiK3u3sHdtM558SWb7317BAjMcXCkn8OKdRqlLAbsvPBIRMn8uIV7iavXrUunyb7rME3hxRqo2jYDdn54uHXXnML2sPu3dsZPtymezTZs88lfxG5XkReE5FlInKfiIwOIjBTWOzu3oFNGr33p/OjHx8dYiSmGARR7fMocLiqzgTeAK4MYJumwNjdvQObN20ac6ZOBaC3Z3fI0ZhCt8/JX1UfUdXoLNTPAJP3dZumsNjdvembWFMTW7Y+/yabgm7w/Rzwl2QrReQiEWkSkabW1taA39rkqkJtyM6G46dNiy3/5CdW9WOyJ60GXxH5GzAxwaqrVfUB7zVXA93Ancm2o6o3AzcD1NfXa8bRmrxUqA3Z2XDmjBksbW7mrpdeor29FdVeRKxTngleWslfVU9MtV5ELgDmAyeoqiV1Y/bBf59+One99FLYYZgCt89dPUXkFODrwPGqumvfQzKmuL2+cWPYIZgiEMT15M+BkcCjIvKyiPwigG0aU7SW+dpC7r33KyFGYgpZEL19pqnqAap6hPfv4iACM6ZYnf3+91NdUQHAK688QEfH9pAjMoXIWpKMyTElItx49t7xfbq6rM+/CZ4lf2NyUGlpaWz51VcfCjESU6gs+RuTgz5UV0c0/T/88Hdpb98Uajym8FjyNyYHlZeW8tY111BR4SZ43Lx5dcgRmUJjyd+YHLWnt5fubnfbzLZt74QcjSk0lvyNyVElIvR6w/vcf///Y9eureEGZAqKJX9jclRlWRn3Xnhh7PHKlUmHzTImY5b8jclhh++/f2y5u3tPiJGYQmPJ35gcVu7r8vnXv36PPXtsBBUTDEv+xuSwspISvnvKKd6jHv7+95+HGo8pHJb8jclxnzvmGA6f7MZg7OjYEXI0plBY8jcmD2zc3hN2CKbAWPI3Jg9s3O76+1uPHxMUS/7G5IH6yW5q7J07W3nssR+GHI0pBJb8jckD933+87Hldetsli+z7yz5G5Nntm2zie/NvrPkb0ye2H/kSAC2bl3DkiW/DDkak+8s+RuTJ5Z+5SucMnUMAC++uCjkaEy+s+RvTJ4oLSmhrNQN8dzb2xVyNCbfWfI3Jo80rtsJuCGeV6ywGb7M4FnyNyaPPHHppbHlhx66JsRITL6z5G9MHtlvxAhGVlYC0Ntrd/2awbPkb4wxRciSvzF5pqzE/Ww7Otpob28NORqTryz5G5NnfnHOOVSMKAfgpps+GnI0Jl9Z8jcmz3woEuHHJ88HoLPThng2g2PJ35g8dP3ixQD09Oyhp8f6/JvMWfI3Jg9dePTRseUf/ui4ECMx+cqSvzF56AuzZjF19GgA9nRsCDkak48s+RuTp/avqfGWNNQ4TH6y5G9Mnjp6ypTY8m9+syDESEw+suRvTJ762kc+wvdOOQWAf/7z+ZCjMfnGkr8xeeyMww8POwSTpyz5G5PHfvv83hL/gw9eHWIkJt8ElvxF5AoRUREZF9Q2jTGpXXbccVR6v+IXXriD3t7ucAMyeSOQ5C8iBwAnA/8MYnvGmPSUlZZy9cmnxB7bSJ8mXUGV/H8KfB3rc2bMkBORsEMweWifk7+InAGsU9V/BBCPMSZDJb7k/847L4UYicknaSV/EfmbiLyS4N8ZwFXAt9PczkUi0iQiTa2tNhStMUE47bDDeM973Cift912Ht3dHSFHZPJBWslfVU9U1cPj/wGrgQjwDxFZA0wGXhSRiUm2c7Oq1qtq/fjx44P6DMYUtTFVVSz9wteREgClvX1T2CGZPFC2L3+sqsuBCdHH3gmgXlXt6DNmCD2/di3a65Z3725j9OjJ4QZkcp718zemALx3QqwMxs03z6enZ0+I0Zh8EGjyV9U6K/UbM/TGjRjBdR/7mPeol7ffbgo1HpP7rORvTIE4+eCDY8u9vVbyN6lZ8jemQIyoqIgt//GPl9kNXyYlS/7GFIhRw4bx7ZNOAqCjo43ly+8POSKTyyz5G1NAvjBrFvuPdj/rXbu2hhyNyWWW/I0pID2qrN/WG3YYJg9Y8jemQL3yyv+FHYLJYZb8jSkgZSUlvKe6GoCWln9Yvb9JypK/MQXm+X//99jy6tVLQozE5DJL/sYUGP8Qz21tLSFGYnKZJX9jCtBR46sAaG5u5I03Hg85GpOLLPkbU4Ae+OLXOGw/dwJobPzfkKMxuciSvzEF6tXNuwBsXl+TkCV/YwpUuVf1v27dS7S0vBJuMCbnWPI3pkD9+eJLYsv33ntpiJGYXGTJ35gC9b4JEygvcT/x7m4b5dP0ZcnfmAJW4iX/trZ17N69LeRoTC6x5G9MAbv+tNMorSgBlHXrloUdjskhoqrhvLFIK/B2KG+e3DggX2Yis1izJ5/izadYIb/izcVYp6rq+CA2FFryz0Ui0qSq9WHHkQ6LNXvyKd58ihXyK958inUwrNrHGGOKkCV/Y4wpQpb8+7o57AAyYLFmTz7Fm0+xQn7Fm0+xZszq/I0xpghZyd8YY4qQJf84InKpiLwmIq+KyA/DjicdInKFiKiIjAs7lmRE5Hpvvy4TkftEZHTYMcUTkVNE5HUReVNEvhl2PKmIyAEislhEVnjH6uVhxzQQESkVkZdE5MGwYxmIiIwWkQbvmF0pIrPCjilolvx9RGQecAbwflU9DPhRyCENSEQOAE4G/hl2LAN4FDhcVWcCbwBXhhxPHyJSCtwInAocCiwUkUPDjSqlbuAKVT0UOBb4Uo7HC3A5sDLsINL0M+Cvqvo+4P3kT9xps+Tf1yXAD1S1E0BVN4YcTzp+CnwdyOnGG1V9RFWjYws/A0wOM54EjgbeVNXVqroH+D2uIJCTVHW9qr7oLe/AJadJ4UaVnIhMBj4O/CrsWAYiIjXAXOBWAFXdo6oFNzaGJf++3gscJyLPisiTIvLBsANKRUTOANap6j/CjiVDnwP+EnYQcSYBa32P3yGHk6mfiNT2sk8XAAAOK0lEQVQBRwLPhhtJSjfgCim9YQeShgjQCvzGq6b6lYiMCDuooJWFHcBQE5G/ARMTrLoatz/G4i6jPwjcIyIHaohdogaI9ypclU9OSBWrqj7gveZqXJXFnUMZW6ESkWrgj8BXVHV72PEkIiLzgY2q+oKIfDjseNJQBhwFXKqqz4rIz4BvAt8KN6xgFV3yV9UTk60TkUuAe71k/5yI9OLG92gdqvjiJYtXRGbgSij/8Cbsngy8KCJHq+qGIQwxJtW+BRCRC4D5wAlhnlCTWAcc4Hs82XsuZ4lIOS7x36mq94YdTwqzgdNF5GPAMGCUiNyhqp8KOa5k3gHeUdXolVQDLvkXFKv26et+YB6AiLwXqCD3BnYCQFWXq+oEVa1T1TrcAXtUWIl/ICJyCu6y/3RV3RV2PAk8D0wXkYiIVADnAf8XckxJiTvj3wqsVNWfhB1PKqp6papO9o7T84DHczjx4/2G1orIwd5TJwArQgwpK4qu5D+AXwO/FpFXgD3AZ3OwhJqvfg5UAo96VyrPqOrF4Ya0l6p2i8iXgYeBUuDXqvpqyGGlMhv4NLBcRF72nrtKVf8cYkyF5FLgTq8gsBq4MOR4Amd3+BpjTBGyah9jjClClvyNMaYIWfI3xpgiZMnfGGOKkCV/Y4wpQpb8jTGmCFnyN8aYIjTo5C8iU0Sk3RsK15jAiUidN09BWjcjish1IrJJRDZkcnwO9bEsImtEJOVQGIXOm4Pgw1nY7gUi0hj0drPBO7anhfX+AyZ/70Dd7f04ov9qVfWfqlqtqj1DEegAMR4qIk0istX797c8GNs8IyLyYRF5J4T3/a2IXDfU75spEZkCXAEcqqoTMzk+s3ks58v+y6ZE+0BVD1PVJ0IKyZB+yf8078cR/deSzaDSLen5tAALcCNyjsONyfL7oOMKyyD2R84YwivDKcDmPJmDoWDk87GZTXlRI6KqKf8Ba4ATEzxfh5tApMx7HAGeAnYAf8PNinSHt+7DuFHyEm4XuBY3ct4dwHbgC7gT0zeBt4DNwD3A2DTiLQO+BOwa6LW+v7kAN37HDqAZON8X1x0pPvMTwH8Bz3lxPxCN0ffai3Anp/XAV33bqsSNcd7i/bsBqPTvL+AbwAbgD8Bu3Fjo7d6/2rjPcIz32lLfc2cBy7zllPsTmAM8DWzDjWt/gRd7F26co3bgT95rD/E++zbgVdxgbdHt/Ba4CfgzsBM4EfgYbmCsHbiRMr+a5Hsoxc2etsn7Pr4Ut79rcIOZrfe2c533NyfG7Z/fJvmu/gNY4sXxCDAuxfea8LXe+s8Ab3v78Vsk/40k239rgK8Cy4A24G5gmO/v5gMve/v3aWBmkv0luMl8NuKOv+W42dLAHV8/ws3w9i7wC2B43PF1lbev1+Ad8976jwMvedtcC1yb4DfweW/bT3nP/wF3/LXh8sBhaewD/+//HuB33v5+Faj3vedRXjw7vPe5G7guxW+50fvsW3G/51O9decAL8S9/t+BB3zH7i9ws87tAJ4Epvpe+z5v3RbgdeBfBjjuB9qeAtMy2Oef9fb5Jtww6dH1g8uVASb/pd4Or8Alku1klvy7gDO9DzIcN+VbdManSuCXwKIBYt2GGyu+F7jG9/wn8ZJggr8Z4cV6sPd4f/YeuNcycPJfBxzubeePvs8cfe0ib90M3NDQ0c/8Pe/zTQDG437k/+HbX93Af3uffXiifZjgs7wFnOR7/Afgm95y0v0JTMUdnAuBcmA/4AjfQX2db5vlwJu4xFEBfMT724N9r2/DDTxWghvCdz1wnLd+DG700UTxXwy8hhtaeSywOG5/3+fFPcLbb88B/5boGEvyXb2Fm7BnuPf4B4N47aG4JDbH+/w/wh27/X4jifaf79h/Dqj1PudK4GJv3ZG4ZH4M7sT2We/1lQm2/VHgBWA07kRwCLC/t+6nuCvgscBI4E/Af8UdXz/BHQvH4xLWwb71M7zvbybu5HFm3L76nfc9RE8on/PeJ1qoeTmNfeD//XfgCgmluALVM966CtyJ9nLcsfcJ3IkkVfLvAv7V29YluMKVeLFtAQ7xvf4l4GxfnDtws3hV4qZybPTlibW4Ad7KvO9pE66aMfq38cd90u0lSP7p7PNbcMfj+4HO6OdgELkyk+Tfjkus24D7438wuEvubqDK93d3kFnyfypu/UrcuO/Rx/t7X2rZAPGOAL4IfHygz+Z7/TbgbLwD2bfuWgZO/j/wrT8Ud2CW+l77Pt/6HwK3+hL1x+J+yGt8+2sPfUuD/fZhgs9yHW40SnA/xJ14JY1U+xM3n+596SQv4DhcCa/E99wivJKK9/rfxW3jn8C/AaMGiP9xvCToPT7Zd4y9B3fAD/etXwgsTrR/knxX/gLBF3FztGb62m/j+2EBVd53lWny/1TccfELb/kmvEKAb/3rwPEJtv0R3HzIx8Z9H+J99wf5npsFNPv2VTcwwrf+HuBbST7DDcBP4/bVgSm+x9Hea2oG2Af+3//f4n5Hu73lubgClvjWN8Zvz7fuAtx0nP7vR4GJvv37n97yYbirg0pfnL/3/W010IMrjJwL/D3uvX4JfCfFcZ90e97jWPJPc59P9q1/DjhvoN92qt9bunX+Z6rqaO/fmQnW1wJbtO847WsTvC6V+NdPBe4TkW0isg33AXtwSSApVd2Ju9T6nYhMGOhNvdefiyt1rheRh0TkfYOM+21c6WRcivW13nKt9zjROoBWVe3IIA6Au4BPiEglroT0oqpG3yPV/jwAdzJKRy2wVlX90/G9Td8pD+O/y7Nxpbq3vekxZ6Xadtx2o6bi9u1632f4Je4KIF3+uQ524X6Mmb62T4zeMb85gxgG2v5U4IroZ/Q+5wH0PTai7/04bqjsG4GNInKziIzCXUlWAS/4tvFX7/mord6xHxU7/kTkGBFZLCKtItKG+234j2nw7QMRKRWRH4jIWyKyHZfYSfA3qcTvj2Fee0ItbqpSTfTeA23Ll5Oi+/c24JPefAifBu5Rb87u+G2rajvuSqEW970cE/e9nE/fmesSxZVse32kuc9THTMZ58qg+vmvB8aKSJXvOf+sSDtxByMQawzxH4jQfwLytbi6utG+f8NUNZ3ZlUq890trDlZVfVhVT8KdMV/DXV71i5vEUxT6P+cU3Bl3U4r10cbyFtyXlmgd9N8f8Y/7UdUVuB/xqbiqrrt8q1Ptz7XAQck2G/e4BThARPzHzhT6znrV529U9XlVPQOXqO/HlTITWU///eWPvxNX9x6Nf5SqHpZkW9myHt/k8yIyHFdNlsyA31uctbiSqf97qlLVRQk3rvo/qvoBXGn5vcDXcMffblz1ZXQbNarqP9mNiZuX1n/83YWrMjpAVWtwhSlJ8bk+iZvs/kRcu0yd97wkeG2m1gOTvGQddUCyFw9EVZ/BXakdh4v79riXxLbtTZE5Frdf1gJPxn0v1ap6iX/zCd4y2fbipbPPkxlUrgwk+XulyybgWhGp8Ep2p/le8gbuTP5xb+q5a3B1U6n8AvhPEZkKICLjxU1Y3o+InCQiR3olkFG4usytuDNgSiLyHhE5w/shdOKquKKl2peBuV4/8Bpc9Ui8T3ldTatw9fgN2rfL4LdEpEpEDsPVF97tPb8IuMb7XONw1Ql3pAj1XWA/L45U7sLVAc7F1flHpdqfdwInisi/iEiZiOwnIkf43vdA33aexZU6vi4i5V5f7dNI0rvKOx7OF5EaVe3Cta8km8T7HuAyEZksImPwTZ2nqutxDa8/FpFRIlIiIgeJyPED7I+gNQCniciHxE30cS2pf6Tx+28gtwAXeyVBEZER3u9mZPwLReSD3uvKcQWVDqDXuyq7Bfhp9OpXRCaJyEfjNvFd7/s5DtfIHD1eRuKu5DtE5GhckkxlJO63sxlXWPp+3PpM94HfUlwp9svesXkGcPQgtxX1O9wVU5eqxt8T8DERmeN9t/+Ba3tYCzwIvFdEPu0d9+Xe/j9kgPdKtr14me5zv7RzpV+Qd/iej6tX3Iyre74bd0Cgqm24etNf4UqIO3G9DVL5Ge5M+IiI7MA1aByT5LWjccm0DVd9cRBwSrTaxEs+yWZlKsG1+LfgLsmOxzUSoaqPep9jGa5h7cEEf387rm5vA66R57K49U/iGkgfA36kqo94z1+HO2Euw/XSeNF7LiFVfc37jKu9y7t+l46eRd5neFxV/VcgSfenqv4TVy1zhbcPXsY1KoHrXXOo9573q+oeXLI/FVfC/F/gM158yXwaWONVCVyMO1YSuQU3k9Y/cPsjfl7az+AaAFfgTu4NuKu1IaNudq9LcSe79bjCwka8Yz2BPvsvje034Rorf477jG/i6rETGYXbZ1vZ2/voem/dN7y/fcbb738DDvb97Qbv71pwJ/+Lfd/hF4HvecfJt0l+pRb1O+/91+G+m2fi1me0D/y84+0TuN5F24BP4X6HyfZ3Om7HddJIVNi6C/gO7nfwAe/9UNUduDao83D7bAN7O2SkknB7CWS6z/0yyZUxWZvJS0TuBl5T1e9k5Q1ygIg8gWsQ/lWCdXW4bmblqto9tJGZoeJdym8Dpqtqc9jxpMO7WrtDVScP9NpcJCLP4hrIfzPIvx+OO2EfpaqrfM//Ftdp4JqA4gx0e0ELrOTvXQId5F2On4KrA8zoLG9MPhCR07yqvBG4rp7L2dvIaQImIseLyESv2uezuK6Qf92HTV4CPO9P/MUoyLvzJuIu0/fDVelcoqovBbh9Y3LFGbiqA8FV3Z2n2bqENuCqq+7BdcteDSzw2oAyJiJrcN9bol6LRcUmcDfGmCJkQzobY0wRsuRvjDFFyJK/McYUIUv+xhhThCz5G2NMEbLkb4wxRej/AzQOGmuWa9vcAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "3794d4d5b722d19532d5039af6861a077e3a4552"
      },
      "cell_type": "markdown",
      "source": "## Final remarks\n\nIn this notebook we explored the most basic setup for a support vector machine. In the next notebooks, we will explore the non-linearly separable case as well as the usa of kernels for introducing non-linear decision boundaries."
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}