Transformer Language Models-Complete.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "welcome-housing",
   "metadata": {},
   "source": [
    "# Transformer Language Models\n",
    "### For sentiment analysis\n",
    "Author: [Camilo](https://github.com/camilocarvajalreyes/)\n",
    "\n",
    "**Objectives**: You will get familiarised with the use of pre-trained models, how to interpret them and implement them in a simple NLP pipeline.\n",
    "\n",
    "**To do**: Trying a [pre-trained](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english) sentiment classifier based on Transformers in an industry case! \n",
    "\n",
    "### Instructions\n",
    "1. Download the vectors, install packages and test if everything is okay (if you haven't done so yet)\n",
    "2. Load the packages and import vectors by running cells in section 0. We will get familiarised with a pre-trained model that uses Transformers from HuggingFace\n",
    "3. Read the instructions in Section 1 and implement the model in a sentiment classifiction task.\n",
    "4. In section 2 we will see how to visualise the attention weights of the model for some examples\n",
    "5. Don't forget to ask if you have any question\n",
    "\n",
    "## Section 0 - Importing a pre-trained model\n",
    "In this tutorial we will use a library called **transformers** from the French company **HuggingFace**. [Transformers](https://github.com/huggingface/transformers) offers more than 40 architectures for NLP based on the [the original transformer module](https://arxiv.org/abs/1706.03762)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "moved-gasoline",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertTokenizer, DistilBertForSequenceClassification\n",
    "import torch\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viral-contributor",
   "metadata": {},
   "source": [
    "We'll use [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english)\n",
    "\n",
    "This model is a based on the [distilBERT](https://arxiv.org/abs/1910.01108), a lighter version of [BERT](https://arxiv.org/abs/1810.04805) and it has been trained for the [Stanford Sentiment Analysis dataset](https://paperswithcode.com/dataset/sst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "mobile-worship",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
    "\n",
    "tokenizer = DistilBertTokenizer.from_pretrained(MODEL_NAME)\n",
    "model = DistilBertForSequenceClassification.from_pretrained(MODEL_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "undefined-assessment",
   "metadata": {},
   "source": [
    "First of all we need to do some pre-processing on the input text. Fortunately, HuggingFace includes **Tokenizers** from pre-trained models.\n",
    "\n",
    "Applying this particular pre-trained model to a tokenised input returns a [SequenceClassifierOutput](https://huggingface.co/transformers/main_classes/output.html#transformers.modeling_outputs.SequenceClassifierOutput) object. More precisely, we can get the **logit** values, which reflect the probability of a Class to be chosen over the other ones.\n",
    "\n",
    "In this case, we have a **binary sentiment classification**, thus the the logits vector show the odds of the sample being **negative** or **positive** respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "funded-might",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "SequenceClassifierOutput(loss=None, logits=tensor([[-3.9886,  4.2962]], grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)\n",
      "Logits:\n",
      "tensor([[-3.9886,  4.2962]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "input_example = tokenizer(\"The cutest dog I've ever seen\", return_tensors=\"pt\")\n",
    "output = model(**input_example)\n",
    "#logits = outputs.logits\n",
    "#print(logits)\n",
    "print('Output:')\n",
    "print(output)\n",
    "print('Logits:')\n",
    "print(output.logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "friendly-underwear",
   "metadata": {},
   "source": [
    "Furthermore, if we happen to have labels for our samples, we can add them when calling the model and it will output the corresponding **loss**. In this case, we know that our phrase is **positive**, the label should be **1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "accepted-insured",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0003, grad_fn=<NllLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "label = torch.tensor([1]).unsqueeze(0)  # Batch size 1, postive sentiment (label 1)\n",
    "output = model(**input_example, labels=label) \n",
    "print(output.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "joint-bedroom",
   "metadata": {},
   "source": [
    "Both outputs are PyTorch tensors, but we can transform it to a **numpy array**.\n",
    "\n",
    "Remark: the first dimension is 1 because it corresponds to the batch size, which is one since we are only inputing one example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "superior-inventory",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2)\n",
      "[[-3.9885974  4.2962294]]\n"
     ]
    }
   ],
   "source": [
    "tensor = output.logits\n",
    "array = tensor.detach().numpy()\n",
    "print(array.shape)\n",
    "print(array)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opponent-brunswick",
   "metadata": {},
   "source": [
    "## Section 1 - Clasifying tweets\n",
    "Let's consider a real case scenario: a major **US Airline** needs you to classify tweets from its costumers. They provide the following [dataset](https://www.kaggle.com/crowdflower/twitter-airline-sentiment):\n",
    "\n",
    "Since we won't fine tune the mode, we will only take 2000 out of the ~14000 tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cordless-afghanistan",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica why are your first fares in May over three times more than other carriers when all seats are available to select???\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>0.3486</td>\n",
       "      <td>@VirginAmerica plus you've added commercials to the experience... tacky.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica it's really aggressive to blast obnoxious \"entertainment\" in your guests' faces &amp;amp; they have littl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica and it's a really big bad thing about it</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica seriously would pay $30 a flight for seats that didn't have this playing.\\nit's really the only bad t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>positive</td>\n",
       "      <td>0.6745</td>\n",
       "      <td>@VirginAmerica yes, nearly every time I fly VX this “ear worm” won’t go away :)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  airline_sentiment  airline_sentiment_confidence  \\\n",
       "1          positive                        0.3486   \n",
       "3          negative                        1.0000   \n",
       "4          negative                        1.0000   \n",
       "5          negative                        1.0000   \n",
       "6          positive                        0.6745   \n",
       "\n",
       "                                                                                                                      text  \n",
       "1                                                 @VirginAmerica plus you've added commercials to the experience... tacky.  \n",
       "3  @VirginAmerica it's really aggressive to blast obnoxious \"entertainment\" in your guests' faces &amp; they have littl...  \n",
       "4                                                                  @VirginAmerica and it's a really big bad thing about it  \n",
       "5  @VirginAmerica seriously would pay $30 a flight for seats that didn't have this playing.\\nit's really the only bad t...  \n",
       "6                                          @VirginAmerica yes, nearly every time I fly VX this “ear worm” won’t go away :)  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_dataset = pd.read_csv(\"data/Tweets-subset.csv\")\n",
    "pd.set_option('max_colwidth', 120)\n",
    "full_dataset = full_dataset.drop(['tweet_id','negativereason','negativereason_confidence','retweet_count',\n",
    "             'airline','airline_sentiment_gold','name','negativereason_gold',\n",
    "             'tweet_coord','tweet_created','tweet_location','user_timezone'], axis=1)\n",
    "dataset = full_dataset[(full_dataset['airline_sentiment']!='neutral')]\n",
    "\n",
    "# example sentence \n",
    "index = 16\n",
    "print(dataset.iloc[index].text)\n",
    "\n",
    "dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "approved-diameter",
   "metadata": {},
   "source": [
    "**1. Using our pre-trained Model, code a function that takes a string of text and outputs 1 if it's positive and 0 if negative**\n",
    "\n",
    "**Solution**: the easiest way is to use [numpy's argmax](https://numpy.org/doc/stable/reference/generated/numpy.argmax.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bridal-service",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "def positive_or_negative(text):\n",
    "    input_tokens = tokenizer(text, return_tensors=\"pt\")\n",
    "    output = model(**input_tokens)\n",
    "    tensor = output.logits\n",
    "    array = tensor.detach().numpy()\n",
    "    return np.argmax(array)\n",
    "positive_or_negative(\"The cutest dog I've ever seen\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "architectural-foster",
   "metadata": {},
   "source": [
    "**2. Iterate over the dataset and compute the average loss and the accuracy of the classifier**\n",
    "\n",
    "(This cell should take about 1 minute to run)\n",
    "\n",
    "For this part we will use the loss integrated in Transformers' output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "clean-defendant",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average loss = 0.44881485334042215\n",
      "accuracy = 89.26045016077171%\n"
     ]
    }
   ],
   "source": [
    "true_positive, true_negative, total, sum_loss = 0, 0, 0, 0\n",
    "for index, row in dataset.iterrows():\n",
    "    text = row.text\n",
    "    input_tokens = tokenizer(text, return_tensors=\"pt\")\n",
    "    if row.airline_sentiment=='positive':\n",
    "        label = torch.tensor([1]).unsqueeze(0)\n",
    "        if positive_or_negative(text)==1:\n",
    "            true_positive += 1\n",
    "    else:\n",
    "        label = torch.tensor([0]).unsqueeze(0)\n",
    "        if positive_or_negative(text)==0:\n",
    "            true_negative += 1\n",
    "    total += 1\n",
    "    output = model(**input_tokens, labels=label)\n",
    "    sum_loss += output.loss.item()\n",
    "\n",
    "average_loss = sum_loss/total\n",
    "accuracy = 100*(true_positive+true_negative)/total\n",
    "print('average loss = {}'.format(average_loss))\n",
    "print('accuracy = {}%'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "natural-civilian",
   "metadata": {},
   "source": [
    "### Visualising Attention\n",
    "\n",
    "The airline wishes to understand how the model works. Therefore, you decide to show a visualisation of the attention weights from the first and last layer of transformers. Thankfully, Transformers provides the attention weights when output_attentions=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "thermal-asian",
   "metadata": {},
   "outputs": [],
   "source": [
    "def attention(list_text,layer=0):\n",
    "    ''' parameters:\n",
    "    text: input string for the classifier\n",
    "    layer: layer to show the attention weights from\n",
    "    '''\n",
    "    input_sentence = tokenizer(list_text, return_tensors=\"pt\",is_split_into_words=True)\n",
    "    output = model(**input_sentence,output_attentions=True)\n",
    "    print('text: '+list_text)\n",
    "    array = output.attentions[layer].detach().numpy()\n",
    "    array = array.mean(axis=1) # taking average over all attention heads\n",
    "    return array[0,:,:]\n",
    "\n",
    "#example\n",
    "#attention(\"@VirginAmerica and it's a really big bad thing about it\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hawaiian-receiver",
   "metadata": {},
   "source": [
    "**3. Using the attention function, code a heatmap to visualise the scoresfrom the model's self attention**\n",
    "\n",
    "You can take inspiration from [this example](https://matplotlib.org/stable/gallery/images_contours_and_fields/image_annotated_heatmap.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "genuine-flavor",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[\"figure.figsize\"] = (10,6) # setting figure size\n",
    "\n",
    "def heatmap_attention(text,layer=0):\n",
    "    tokenized_sequence = tokenizer.tokenize(text)\n",
    "    labels = ['[CLS]']+tokenized_sequence+['[SEP]']\n",
    "\n",
    "    matrix = attention(text,layer=layer)\n",
    "\n",
    "\n",
    "    fig, ax = plt.subplots()\n",
    "    im = ax.imshow(matrix)\n",
    "\n",
    "    # We want to show all ticks...\n",
    "    ax.set_xticks(np.arange(len(labels)))\n",
    "    ax.set_yticks(np.arange(len(labels)))\n",
    "    # ... and label them with the respective list entries\n",
    "    ax.set_xticklabels(labels)\n",
    "    ax.set_yticklabels(labels)\n",
    "\n",
    "    # Rotate the tick labels and set their alignment.\n",
    "    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\",\n",
    "             rotation_mode=\"anchor\")\n",
    "\n",
    "    ax.set_title(\"Attention visualisation for our Transformer based classifier\")\n",
    "    fig.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "original-world",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text: @VirginAmerica and it's a really big bad thing about it\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ0AAAGoCAYAAACdaDi8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABCHElEQVR4nO3deZwcVbn/8c83kwSykQBh36Lse4RBFgUioogCoiIgiyBiEFEEwateUVFRUFTg6gWMG4qoKP5ARS+gYkBA0CBbAEFBFNkkkAAJIcvM8/vjnDaVZpaemerq6fB9v17zmuqq6qdPV5+qp86pTRGBmZlZFUa0ugBmZvbS4aRjZmaVcdIxM7PKOOmYmVllnHTMzKwyTjpmZlYZJ50KSbpQ0icq/szDJV3T5M84XdL38/CGkuZL6ij5M5oSN8d+laS/5vgHlh1/RSNpLUnXS3pO0pdbXZ7+FOtnC8twtKQbBvG+iySd0Ywy5fjzJb08D4+R9AtJz0j6SbO2HSt80pE0U9JcSSvVjX9I0t6F11MkhaSRJX3uiypZRLw3Ij5bRvxGRcQlEfH6Cj/vnxExPiK6hhKn/vcpK24vPgN8Lce/ognxS5M3ErW/bkkLC68Pr6gY04E5wCoRcUpFn2lNkOv8g/nlQcBawOoR8fZmbTtW6KQjaQqwOxDAAa0tjQ1jGwF3D+aNZe2k9BJbkpZbR/NGYnxEjAf+CexfGHdJFeUiLa97YhBXlje5XE2Pv4LbCLg/IpYONVCfPRIRscL+AZ8EbgS+AlxZGH8x0A0sBOYD/0VagSO/ng/smuc9BrgXmAtcDWxUiBPAe4G/AvOA/wUEbAm8AHTlWPPy/BcBZxTe/x7gb8DTwM+BdfuL3cN3XDd/j9UK415B2hMdBRwN3JDHCzgH+DfwLHAXsE2eNhM4thDjP+/Lr88DHs7vuxXYvTDtdOD7eXhKLvvIQpwHgeeAvwOH5/EbA9cCT+WyXgJM6uP3qY+7bl5mT+dl+J668vwY+F7+3LuBzl7qyAN1n7VSA7EvA76fl8WxPcScmD/7SeAfwGnAiPpl1cvymgl8jlRvFwKb9FG/HwL2zsPTgH8BHwEez8twVeDKXI65eXj9wvtnAp/Nn/UccA0wOU9bOX/Hp0j170+kveCLgCXA4ry89s7L7Fzg0fx3LrBSH+U6HfhJjv8cqR5uBnyMVDcfBl5ftzy/BTwGPAKcAXQU6teNpHr9FIX1q4ff7NL8eX8Gti9M/yipHjwH3AO8pTBtE+A64BlSPb20MG0L4NekenIfcHBh2uqkOvQs8Me8nG/o47d8NXBTXtYPA0fXbzMa+D2Ppud1ra/vEHn6p/NvuiT/ru/mxduAvr7vRcAFwK+ABeR62eN3bWVSaPYfaYPxPmDHvDDX6mmF7Wnlz+PenGNsCYwkbTxuqvvBrgQmARvmyvCGQgW4oa48xQq0V64AO5BW2q8C1zcSu4fveS3LbxjPBi6sLwewDylhTGJZclynsAHqK+kcQVqRRgKnkDYgKxdW6hclHWAcaaXbPE9bB9i6sCK8Ln/3NYDrgXMb/X3y/OeTNo5T8/LZq1CeF4A3Ah3AmcDNfdST+s/qL/YS4EBST8GYHuJ9D/gZMCGX+37g3fXLqpfvNZO0A7R1XoajGik3aeO+FPhCXqZj8u/1NmBsLstPgCsK759J2thuluefCZyVpx0H/CK/t4O0Dq1SX4/z688ANwNr5t/yJuCzfZSr9vvsk7/j90gbyY+TdpTeA/y9EP9y4Ouk+rQmaSN+XKGeLgU+kGP19HvUfrODcvxT8+eNytPfTtrRGAEcQtpo1taLH+ZyjSDVh1fn8eNIyeFd+XNrO3pb5ek/Iu34jAO2ISXLHpMOqYXxHPCOXL7Vgak9bDN6/T3pe13r8TsUtjOb9FI3j2bZtqO/73sRKam9qvY5vdbbZm70W/lH2nNYwrI9t78AJze6Ucvj/o+8scivRwDPk1s7ef7iD/hj4KP1P1hherECfQv4YmHa+FzeKf3F7uG7Hgtcm4eVK8cePVScvUgbwF3Ie951G6Bek04PnzmXvLdI30lnHmlFedHGoC7egcBtjfw+wAakVuSEwvQzgYsK5flNYdpWwMI+Pvs/n9Vg7Ov7iNVB2mPcqjDuOGBm/bLqqd7l3+EzDdbxYrmn5c/tfWVPCXRu3W9+WuH1+4Cr8vAxpOSxXQ9xLmL5pPMA8MbC632Ah3orV14Gvy683p+0d11rvUzIy2QSqXW1qFh/SBvn3xXq6T/7WU6nU9jpIK3Hj1FordfNfzvw5jz8PWAGhRZFHn8I8Pu6cV8HPpXrwBJgi8K0z9N70vkYcHkv05Zb1r39nvSxrvX2HfK0RpNOr9+3UM7vNVJvV+RjOkcB10TEnPz6B3ncQGwEnCdpnqR5pGalgPUK8zxeGH6elDwasS6p6wWAiJhP6h4YTOyfArtKWgfYg9Rd9Pv6mSLiWuBrpK66f0uaIWmVRgor6VRJ9+YzW+aRujwm9/WeiFhAqqzvBR6T9EtJW+R4a0n6kaRHJD1L6mrpM17BusDTEfFcYdw/6HvZrdxgf38jsR/u4/2TSXur/yiMq39/f/qK35cnI+KF2gtJYyV9XdI/8jK+HphU19/eWx27mNSd/CNJj0r6oqRRvXzucnU5D6/bW7myJwrDC4E5sewkkYX5/3jSOjiKVH9q6+HXSS2emkaW13/miYhuUpffugCS3inp9kL8bVhWF/+LtM7/UdLdko7J4zcCdq69J7/vcGBtUmtvZF25isun3gakxN2nvn7Pvta1Pr7DQPT1fWsaqrcrZNKRNAY4GNhT0uOSHgdOBraXtH2eLereVv8a0kI8LiImFf7GRMRNDRSjp3hFj5J+yFqZx5Gaz480EHv5D4qYS+qPPwQ4DPhR5N2PHub9n4jYkbT3vxnw4TxpAanZXvOfyiRpd1LFPRhYNSImkZrSaqBsV0fE60jN/b8A38iTPk9aRttGxCqk7rtivL6W36PAapImFMZtyCCW3SBj91W2OaS93I0K44rv73U5Nxi/L/XvOwXYHNg5L+M98vhGfrclEfHpiNgK2A3YD3hnL7MvV5dJ3/fRPso1EA+TWjqTC+vgKhGx9QDjb1AbyCdnrA88KmkjUp18P+msrUnAbPIyiojHI+I9EbEuqcV6vqRNcrmuq9s2jI+I40ndsUuLn0laJn19x40b+A59/p69rWt9fIeB6Ov71jT0O6+QSYfUVdNF2rBOzX9bkvb+ayvOE8DLC+95ktRCKI67EPiYpK0BJE2U9PYGy/AEsL6k0b1M/yHwLklT8+ncnwduiYiHGoxf7wek73ZQHn4RSTtJ2jnvsS4g9at358m3A2/Ne1ObkA4k1kwgrURPAiMlfRLot4WUWzNvzgl1EakLpfZ5E/LrZyStx7LkV1P/+/xHRDxM6vo5U9LKkrbL5R3ytRhDjZ331n8MfE7ShLxR+1Dh/bcDeyhddzSR1LXSLBNIrYZ5klYjdf00RNJrJG2bW0XPkhJpdy+z/xA4TdIakiaTTuAp5bqYiHiMtEP1ZUmrSBohaWNJew4w1I6S3ppbuyeR6uPNpG6pINVtJL2L1NIhv367pPXzy7l53m7S8dbNJB0paVT+20nSlrkO/D/g9Lw+bUXfvSyXAHtLOljSSEmrS5raw3y9/p59rWt9fIeB6PX7DjDOCpt0jgK+E+najsdrf6SupcNzxTuTtKLMk3RqRDxPPmsoj9slIi4nHQD9UW7Ozgb2bbAM15LOmnpc0pz6iRHxG+ATpK6xx0h7OocO4Tv/HNgUeDwi7uhlnlVIez9zSc39p0gnHUA6+2cxaWP/XdKKUHM1cBXpeNA/SMmqkab0CNIG91FS1+SeQG3P6NOkkyieAX5JWkmLlvt9eoj9DtLxkEdJB5o/lZdpGYYa+wOkpP4gcANpJ+DbABHxa9JZVHeSTuq4sqQy9+Rc0oH7OaQN7FUDeO/apDO+niWdvXkdqcutJ2cAs0jf6S7S2WFlXtD4TmA06cyyublc6wwwxs9IPQFzgSOBt+bW3D3Al4E/kOr+tqSz4Wp2Am6RNJ+0jn0wIh7M3a+vJ62zj5K6KWsnS0BqOY3P4y8CvtNbwSLin6STXk4hrSe3A9v3MOu59P579rWu9fgdeitPL2Xs7/s2TL30wpiZmZVuRW3pmJnZMOSkY2ZmlXHSMTOzyjjpmJlZZVbIm+NNXq0jpmzQ23Vsg/PX2Y1e89k6zTgpROr3ko7BBC09ZHQ34+bT5Zez7OXpE4HK1ozl2YR1qGTdE8eUGm/R83NZsmhBj198hUw6UzYYxR+v3qD/GQdg3012KzUeUPrGN5YO+eawL6KR5VcRjSo/Zvf8BaXHROV3BJT93aOrCcm2u00S2YgmbMybsTw7Sn4EVBN+n+enTS013u0zz+t1mrvXzMysMk46ZmZWGScdMzOrjJOOmZlVxknHzMwq46RjZmaVKT3pSJoiaaGk2/PrtZUe1vWApFsl/UrSZnm+2T28fxdJt+SHKt0r6fQ8/hBJf5PUzLvymplZEzXrOp0HImKq0pVwlwPfjYhDAfJD1Nai91vjfxc4OCLuyM/y2BwgIi6V9ATp+eZmZtaGmn1x6GuAJRFxYW1E7Vkvkqb08p41Sc+XqT0Q654ml9HMzCrS7GM625AeVDUQ5wD3Sbpc0nGSVm7kTZKmS5oladaTTzXjlihmZjZUw+5Egoj4DNBJekTtYTT4tMOImBERnRHRucbqJd92wszMStHspHM3sONA3xQRD0TEBcBrge0lrV56yczMrHLNTjrXAitJml4bIWk7Sbv39gZJb9KyW/FuCnQB85paSjMzq0RTTySIiJD0FuBcSR8BXgAeAk7Ks2wu6V+Ft5wMvA04R9LzwFLg8HxCgZmZtbmmP9ogIh4FDu5lck8PvflJE4tjZmYt1IzutS5gYu3i0LJIOgQ4H5hbZlwzM6tO6S2diHgYKPcJainupcClZcc1M7PqDLtTps3MbMW1Qj6u+i8LJ/GqO99aasyJ65f/aNw5u65ZarzJP2vCzRua8cjmcWNLjxnPzi895ojRTfjuo3s6jDl4saD8R5Q35RHYTaBowuOqmyCWlPsbqezHXwNjHltYarwRS3p/pLZbOmZmVhknHTMzq4yTjpmZVcZJx8zMKuOkY2ZmlXHSMTOzyjjpmJlZZYZt0pG0lqTzJN0p6c+Svimp9DsdmJlZdYZl0pG0MenhbTcCnRGxA/BD4PI8zczM2tCwTDrABcBREfHjiFgMEBG/BY4AvtzSkpmZ2aANu6QjaTPgyYi4U9J+uWvtMkk/jYi/AN2SJre6nGZmNnDD8d5r2wM3S+oAPgXsBUwEZufpfwVeBswpvik/nXQ6wOg1V6mssGZm1rhh19LJuoDJwAMRMS8i/gHU7ma5JvDv+jdExIyI6IyIzlETx1RYVDMza9RwTDqzgZ1JLZmNJU2UtCGwpaRtgTVzEjIzszYz7LrXIuLenGQ2B84Afgc8CPwcOBU4poXFMzOzIRh2SSd7H3AJ8BFgxzxuB2DdiHiiZaUyM7MhGY7da0TEvcABwNuAPwN3AMcDd7ayXGZmNjTDtaVDRPwLeG+ry2FmZuUZli0dMzNbMTnpmJlZZYZt99pQLFk4in/fuVapMVd5bHb/Mw3Q5D+Wm/Mff8dWpcYDWOvGuaXH1L/Ljzli9KjSY3YvXlJ6zLL38kaMK/+atO4FC0uPqVFN2NR0d5cfshm/+ZiVSw7YhLZCV8nLMqLXSW7pmJlZZZx0zMysMk46ZmZWGScdMzOrjJOOmZlVxknHzMwq46RjZmaVaWrSkbSupMsG8b5fSZrUhCKZmVkLNfXi0Ih4FDiofrykkRGxtI/3vbGZ5TIzs9YoraUj6SxJJxReny7pVEmz8+ujJf1c0rXAbyWNlfRjSfdIulzSLZI687wPSZosaYqkeyV9Q9Ldkq6R5MeCmpm1qTK71y4FDi68Phi4pW6eHYCDImJP0jNz5kbEVsAnWPbcnHqbAv8bEVsD80iPO3gRSdMlzZI0q3v+gsF/CzMza5rSkk5E3AasmY/jbA/MBR6um+3XEfF0Hn418KP83tn0/qycv0fE7Xn4VmBKL58/IyI6I6JzxPhxg/8iZmbWNGUf0/kJ6RjO2qSWT73BNEEWFYa7AHevmZm1qbKTzqXAN4DJwJ7ASn3MeyOpC+53krYCti25LGZmNsyUesp0RNwNTAAeiYjH+pn9fGANSfcAZwB3A8+UWR4zMxteSj9lOiK2LQw/BGyThy8CLirM+gJwRES8IGlj4DfAP/K8U/I8c2rvz+O/VHZ5zcysOq18iNtYUtfaKEDA+yJicQvLY2ZmTdaypBMRzwGdrfp8MzOrnu+9ZmZmlWll91rTrD1xHqfs9/NSY17x8fVKjQfA3+svYxqatZd2lRoP4MEj1yo95pTPPlh6zFha/rPt+3rO+2B1Lyr5WfSLm/C9u8uvR9FVfkyi5GUJzfnNF5R8sbpUbjxg5NPzS42npb3/Nm7pmJlZZZx0zMysMk46ZmZWGScdMzOrjJOOmZlVxknHzMwq46RjZmaVcdIxM7PKDCjpSDpT0mskHSjpY3XTNpf0XUkjJP2h3GKamdmKYKAtnZ2Bm0nPyrm+btruedy2wOyhF83MzFY0Dd0GR9LZwD7Ay4A/ABsDr5V0GfA74KvAhsATpOfpdEuaFRGdkqYAFwO1Z0i/PyJukjQN+DQwj5SofgzcBXyQ9HTQAyPiAUlrABfm+AAnRcSNQ/jOZmbWIg0lnYj4sKQfA+8EPgTMjIhXFWaZmrvUdgO+DXwpP9AN4N/A6/JzczYFfsiyu0tvD2wJPA08CHwzIl4p6YPAB4CTgPOAcyLiBkkbAlfn9yxH0nRgOsBq6/b1wFIzM2uVgdzwcwfgDmAL4N7iBEljgUURETmx3FeYPAr4mqSpQBewWWHan2pPGJX0AHBNHn8X8Jo8vDewlZbd5G4VSeMjYrk71EXEDGAGwEbbTCj/rn1mZjZk/SadnCwuAtYnPclzbBqt24FdgUtJiWiSpDuBKcAsSWdGxKXAyaRut+1Jx5BeKIRfVBjuLrzuLpRtBLBLRBTfZ2ZmbajfEwki4vaImArcD2wFXAvsExFTI2JhRBwAfAM4HjgRuDBPuzSHmAg8FhHdwJFAxwDLeA2pqw34TxI0M7M21NDZa/lg/tycOLaIiHvqZtkDuIF0Btt1ddPOB46SVOuaG+jDJU4EOiXdKeke4L0DfL+ZmQ0TjZ5I8CTwpjy8Sw/T98+Dn+1h2l+B7QqjPpLHzwRmFuabVhj+z7SImAMc0kg5zcxsePMdCczMrDJOOmZmVhknHTMzq8xArtNpG0/MncQ5/++AUmNusvqDpcYD6H5ufv8zDcCCzSeXGg9gvesWlx7zyXftWHrMNb5za+kxm2KE+p9nILqbcEla2WUE6BjoSav90+jRpcfsXvB86TFV9neP7nLjASxeUm686L1euqVjZmaVcdIxM7PKOOmYmVllnHTMzKwyTjpmZlYZJx0zM6uMk46ZmVXGScfMzCoz6KQj6UxJr5F0oKSP1U3bXNJ3JY3ITxTtLca6+ZHXZmb2EjCUls7OwM3AnsD1ddN2z+O2BWb39GZJIyPi0Yg4aAhlMDOzNjLg2+BIOhvYB3gZ8AdgY+C1ucXyO+CrwIakp4VOALolzYqITklHA28FxgMdko4CroyIbSR1AF8A3kB6cug3IuKrkj4J7A+MAW4Cjovo4x4LZmY2bA24pRMRHwbeTXqE9U7AnRGxXUR8JiJ+n58yeh/pKaO/BvaNiM5CiB2AgyJiz7rQ00mPup4aEdsBl+TxX4uInSJiG1Li2a+nckmaLmmWpFldCwb6nDgzM6vCYLvXdgBqTwK9tzhB0lhgUW6NbEpKQEW/joine4i5N/D1iFgKUJjnNZJukXQXsBewdU8FiogZEdEZEZ0d48YN8muZmVkzDah7TdJUUgtnfWAOMDaN1u3ArsClpEQ0SdKdpJbLLElnRsSlOUzDzRBJK5Med90ZEQ9LOh1YeSBlNjOz4WNALZ2IuD13n91P6j67FtgnIqZGxMKIOAD4BnA8cCJwYZ52aa9Bl/k1cJykkQCSVmNZgpkjaTzgkw7MzNrYgLvXJK0BzI2IbmCLiLinbpY9gBtIZ7BdN4DQ3wT+Cdwp6Q7gsIiYR0pis4GrgT8NtLxmZjZ8DPjstYh4EnhTHt6lh+n758HP9jDtIlL3XO31Q8A2eXgp8KH8V3zPacBpAy2nmZkNP74jgZmZVcZJx8zMKuOkY2ZmlRnwMZ22IIiyv1lHR8kBQWPHlhpv7G/uLDUe0JTvvdZj65Ue88kjdiw95uTLeryD05Bo/bVLjRf/erzUeACxeHHpMeku/yYisXBh6THVjPW8o+x9+/LbCkumrFlqvJjX+wbYLR0zM6uMk46ZmVXGScfMzCrjpGNmZpVx0jEzs8o46ZiZWWWcdMzMrDJtl3QkzW91GczMbHDaLumYmVn7aknSkXSFpFsl3S1peh43X9LnJN0h6WZJa+XxL5P0B0l3STqjFeU1M7NytKqlc0xE7Ah0AidKWh0YB9wcEdsD1wPvyfOeB1wQEdsCj7WktGZmVopWJZ0T84PabgY2ADYFFgNX5um3kh51DfAq4Id5+OLeAkqaLmmWpFldCxp+IraZmVWo8qQjaRqwN7BrbtXcRnos9ZKIqN0VsIvlb0ba790CI2JGRHRGRGfHuHHlFtrMzErRipbORNLjrp+XtAXwoqeP1rkRODQPH97UkpmZWVO1IulcBYyUdC9wFqmLrS8fBE6QdBdQ/j3xzcysMpU/TyciFgH79jBpfGGey4DL8vDfgV0L853W1AKamVnT+DodMzOrjJOOmZlVxknHzMwq46RjZmaVqfxEgiqMfB7W+HO/l/YMSIxZqdR4ADF3XqnxuhctKjUewIgxY0qPqbnPlh5zjd8vKT3mk2/fpvSYq977fKnxRs0r/5q0ricXlh6Tjo7SQ0or5OarX0s7tyg95qi55f7mWtrd6zS3dMzMrDJOOmZmVhknHTMzq4yTjpmZVcZJx8zMKuOkY2ZmlXHSMTOzygz7pCPppvx/iqTDWl0eMzMbvGGfdCJitzw4BXDSMTNrY8M+6UianwfPAnaXdLukk1tZJjMzG5x2uo/ER4FTI2K/niZKmg5MBxg9dtUqy2VmZg0a9i2dRkXEjIjojIjOUSuVfz8qMzMbuhUm6ZiZ2fDXTknnOWBCqwthZmaD105J506gS9IdPpHAzKw9DfsTCSJifP6/BNirxcUxM7MhaKeWjpmZtTknHTMzq4yTjpmZVcZJx8zMKjPsTyQYjCXj4PFdyo058aqnyg0IxOLFpcdsBzFuTOkxtbSr9Jhr/OJvpcf85zGblhpvo4c7So0HQEcTYjZBdHWXHlMdw38/vOOWe0qP2b39ZqXGixHqddrwX8JmZrbCcNIxM7PKOOmYmVllnHTMzKwyTjpmZlYZJx0zM6uMk46ZmVXGScfMzCrTdklH0kOtLoOZmQ1O2yUdMzNrX+2YdJ7saaSk6ZJmSZrVtWBB1WUyM7MGtF3SiYidehk/IyI6I6KzY9y4qotlZmYNaLukY2Zm7ctJx8zMKuOkY2ZmlXHSMTOzyjjpmJlZZZx0zMysMk46ZmZWGScdMzOrzMhWF6AZRi2AdW6KcoOOHlVuPCDml1zGKDke0L1wYekxR3Y3oZyTxpceU/OeKT3mqn/tKjXeE2/YsNR4AGt8/6nSY0ZXd+kxm2HE6quVHjOem19uwK5y6xBA10olp4IR6n1SuZ9kZmbWOycdMzOrjJOOmZlVxknHzMwq46RjZmaVcdIxM7PKOOmYmVllnHTMzKwyTjpmZlaZtkg6ksZJ+qWkOyTNlnRIq8tkZmYD1y63wXkD8GhEvAlA0sT6GSRNB6YDjB47qdLCmZlZY9qipQPcBbxO0hck7R4RL7opVkTMiIjOiOgctVL59+EyM7Oha4ukExH3AzuQks8Zkj7Z4iKZmdkgtEX3mqR1gacj4vuS5gHHtrhIZmY2CG2RdIBtgbMldQNLgONbXB4zMxuEtkg6EXE1cHWry2FmZkPTFsd0zMxsxeCkY2ZmlXHSMTOzyrTFMZ2BWjIOHt+l92d0D8aEXy0sNR7AiNGjSo3X/UL5z07XyHLLCBDPLSg95ojFS0qPGd1ReswJdz1ZbrxSoyVPHzS19JiTLv1z6TFHjBtTekxKXicBYvHicgOOKL+tMHLu86XG09LuXqe5pWNmZpVx0jEzs8o46ZiZWWWcdMzMrDJOOmZmVhknHTMzq4yTjpmZVcZJx8zMKuOkY2ZmlWmLpCPpCkm3Sro7P5bazMzaULvcBueYiHha0hjgT5J+GhFPtbpQZmY2MG3R0gFOlHQHcDOwAbBp/QySpkuaJWlW14Ly7+1lZmZDN+yTjqRpwN7ArhGxPXAbsHL9fBExIyI6I6KzY9y4agtpZmYNGfZJB5gIzI2I5yVtAezS6gKZmdngtEPSuQoYKele4CxSF5uZmbWhYX8iQUQsAvZtdTnMzGzo2qGlY2ZmKwgnHTMzq4yTjpmZVcZJx8zMKjPsTyQYlJFB12pLSw0ZXV2lxgPQ6NHlBly8pNx4wIiJE0qPGU24eLfrqadLj6lR5a8eHfOfLzVeLC2/Xq46+0WXwQ3ZnCN3LD3mmv/399Jj8sKi8mOOKHffPhYvLjUewIinnyk3YB/bS7d0zMysMk46ZmZWGScdMzOrjJOOmZlVxknHzMwq46RjZmaVcdIxM7PKDKukI+khSZPz8PxWl8fMzMrVtKSjZFglNTMza61Sk4KkKZLuk/Q9YDbwCUl/knSnpE8X5rtC0q2S7pY0vZ+Y35N0YOH1JZLeXGa5zcysGs1oiWwKnA+cDKwHvBKYCuwoaY88zzERsSPQCZwoafU+4n0LOBpA0kRgN+CX9TNJmi5plqRZXc+Vf5sVMzMbumYknX9ExM3A6/PfbcCfgS1ICQlSormD9BTQDQrjXyQirgM2lbQG8A7gpxHxohurRcSMiOiMiM6OCeNK/UJmZlaOZtzws9bMEHBmRHy9OFHSNGBvYNeIeF7STKC/Owx+DzgCOBR4V5mFNTOz6jTzQP/VwDGSxgNIWk/SmsBEYG5OOFsAuzQQ6yLgJICIuKc5xTUzs2Zr2qMNIuIaSVsCf5AEMJ/UWrkKeK+ke4H7SF1s/cV6Is9/RbPKa2ZmzVdq0omIh4BtCq/PA87rYdZ9e3n/lMLw+NqwpLGk4z4/LKmoZmbWAsP+OhpJewP3Al+NiJKfNGRmZlUa9k8OjYjfABu1uhxmZjZ0w76lY2ZmKw4nHTMzq8yw714blKWi46lR5cbs6io3XhOoo6P0mPH8wtJjNsOIlVcqPWYsXlx6zO755d4tY8Qafd3MY5Aem1N6yDVnPl96zIcPe3npMTe49B+lx9S4NrhYfWTZqUC9TnFLx8zMKuOkY2ZmlXHSMTOzyjjpmJlZZZx0zMysMk46ZmZWGScdMzOrTEuTTn689ewexn9T0latKJOZmTXPsLw4NCKObXUZzMysfMOhe22kpEsk3SvpMkljJc2U1Akg6d2S7pf0R0nfkPS1VhfYzMwGZzgknc2B8yNiS+BZ4H21CZLWBT5Berroq4AtegsiabqkWZJmdS0o91YjZmZWjuGQdB6OiBvz8PeBVxemvRK4LiKejoglwE96CxIRMyKiMyI6O9rhXkdmZi9BwyHpRD+vzcxsBTEcks6GknbNw4cBNxSm/QnYU9KqkkYCb6u8dGZmVprhkHTuA06QdC+wKnBBbUJEPAJ8HvgjcCPwEOBHVpuZtamWnjIdEQ/R88kB0wrDP4iIGbmlczlwRfNLZmZmzTAcWjr9OV3S7cBs4O846ZiZta1heXFoUUSc2uoymJlZOdqhpWNmZisIJx0zM6vMsO9eG5SOoGuVpSXH7Cg3HhCLF5caT6Oa8HN2d5ceUqNHlx6za36b3IXihUWlhosRKjUeAIvKLSM05+K7DX78z9JjPrb/hqXHXOfyB0uN14xl2T1pQrkB5/S+vXRLx8zMKuOkY2ZmlXHSMTOzyjjpmJlZZZx0zMysMk46ZmZWGScdMzOrTOVJR9IUSbOrfq+ZmbWeWzpmZlaZViWdkZIukXSvpMskjZX0SUl/kjRb0gxJApC0o6Q7JN0BnNCi8pqZWQlalXQ2B86PiC2BZ4H3AV+LiJ0iYhtgDLBfnvc7wAciYvu+AkqaLmmWpFldz7XJLVHMzF5iWpV0Ho6IG/Pw94FXA6+RdIuku4C9gK0lTQImRcT1ed6LewsYETMiojMiOjsmjGtm2c3MbJBadcPP+nvWBXA+0BkRD0s6HVi58lKZmVlTtaqls6GkXfPwYcANeXiOpPHAQQARMQ+YJ+nVefrhlZbSzMxK1aqWzn3ACZK+DdwDXACsSnok9ePAnwrzvgv4tqQArqm6oGZmVp7Kk05EPARs0cOk0/Jf/fy3AsWTCP6rOSUzM7Nm83U6ZmZWGScdMzOrjJOOmZlVxknHzMwq06qz15pLoJW7Wl2KfsXixeXGKzVa84wYUf6+TsfEVUqP2fXMs6XH1MorlRtv0ZJS4wGE2mNftPupp0uPufb/K3edBPjXYZuUGm+9i+8rNR5AVPiTt0ftMjOzFYKTjpmZVcZJx8zMKuOkY2ZmlXHSMTOzyjjpmJlZZZx0zMysMqUnHUmTJL0vD0+TdGUv831T0lZlf76ZmQ1fzWjpTCI9frpPEXFsRNzThM83M7NhqhlJ5yxgY0m3A2cD4yVdJukvki6RJABJMyV15uH5kj4n6Q5JN0taK4/fOL++S9IZkuY3obxmZlaRZiSdjwIPRMRU4MPAK4CTgK2AlwOv6uE944CbI2J74HrgPXn8ecB5EbEt8K++PlTSdEmzJM3qem5BGd/DzMxKVsWJBH+MiH9FRDdwOzClh3kWA7VjP7cW5tkV+Eke/kFfHxIRMyKiMyI6OyaMG2qZzcysCapIOosKw130fJPRJRER/cxjZmZtrhlJ5zlgQkmxbgbelocPLSmmmZm1SOlJJyKeAm6UNJt0IsFQnAR8SNKdwCbAM0OMZ2ZmLdSUbqyIOKyX8e8vDE8rDI8vDF8GXJZfPgLsEhEh6VBg82aU18zMqjHcj53sCHwtn2Y9DzimtcUxM7OhGNZJJyJ+D2zf6nKYmVk5fO81MzOrzLBu6QzaEtHxeMnPoh9Z/qLS6NGlxuue34QbNqj8/ZJmLMvu554rPWYzjFilrBM7k+55TTi3ZsmS0kOqq6v0mNGMmBusWXrM9X75RKnxntx/s1LjAax6/8JS40WHep3mlo6ZmVXGScfMzCrjpGNmZpVx0jEzs8o46ZiZWWWcdMzMrDJOOmZmVpmmJ52ynvYpaaqkN5YRy8zMWqOdWjpTAScdM7M2VmrSkXSFpFsl3S1pemH8OXncbyWtkcdNlXSzpDslXS5p1Tx+pqTOPDxZ0kOSRgOfAQ6RdLukQ8ost5mZVaPsls4xEbEj0AmcKGl1YBwwKyK2Bq4DPpXn/R7wkYjYDrirMP5FImIx8Eng0oiYGhGXllxuMzOrQNlJ50RJd5Ce+LkBsCnQDdSSxPeBV0uaCEyKiOvy+O8CewzlgyVNlzRL0qzuBQuGEsrMzJqktKQjaRqwN7BrRGwP3Aas3MOs0U+opYVy9fT+HkXEjIjojIjOEePGNfo2MzOrUJktnYnA3Ih4XtIWwC6FzzgoDx8G3BARzwBzJe2exx9J6noDeIj08DYK7wN4Dij3Fr1mZlapMpPOVcBISfcCZ5G62AAWAK+UNBvYi3RCAMBRwNmS7iSdmVYb/yXgeEm3AZML8X8HbOUTCczM2ldpDzaJiEXAvj1MGt/L/LezrDVUHP8XYLvCqNPy+KeBnYZcUDMza5l2uk7HzMzanJOOmZlVxknHzMwq46RjZmaVKe1EguFE3TDyeZUbs6Oj1HgA3YsWlRtQ5e9DNON7R/R3qdYgYnZuVXpM3XZf6TFj8ZJyA3Z3lxuvSZrxmzeD7v17+TFXW7XUeGv+7pFS4wH89bj1So23+O+9b4vc0jEzs8o46ZiZWWWcdMzMrDJOOmZmVhknHTMzq4yTjpmZVcZJx8zMKjPsk46km/L/KZIOa3V5zMxs8IZ90omI3fLgFNLzeMzMrE0N+6QjaX4ePAvYPT9P5+RWlsnMzAannW6D81Hg1IjYr6eJkqYD0wFGTiz3thNmZlaOYd/SaVREzIiIzojo7Bg7rtXFMTOzHqwwScfMzIa/dko6zwETWl0IMzMbvHZKOncCXZLu8IkEZmbtadifSBAR4/P/JcBeLS6OmZkNQTu1dMzMrM056ZiZWWWcdMzMrDJOOmZmVplhfyLBYGy5xhNc+54vlxrz4M/s1v9MA6SOjmEdr1k0elT5QW+5q/SQofL3ybrnzi01XnRHqfEA6O4qP+aiRaWHbEZ97168uPyYC18oN2B0lxsP6Fi4frkB+yiiWzpmZlYZJx0zM6uMk46ZmVXGScfMzCrjpGNmZpVx0jEzs8o46ZiZWWWcdMzMrDKlJx1JUyQtlHR7fv1xSXdLulPS7ZJ2zuNnSrovj7td0mV5/OmSHsnjZks6II8/WdI/JX2t7DKbmVk1mnVHggciYqqkXYH9gB0iYpGkycDownyHR8SsHt5/TkR8SdKWwO8lrRkR50iaC3Q2qcxmZtZkze5eWweYExGLACJiTkQ82uibI+JeYCkwub95JU2XNEvSrDlPlX+bCDMzG7pmJ51rgA0k3S/pfEl71k2/pNC9dnb9m3NXXDfwZH8fFBEzIqIzIjonr+5DVWZmw1FTb/gZEfMl7QjsDrwGuFTSRyPiojxLb91rJ0s6AngOOCQimnBXQzMzq1rT7zIdEV3ATGCmpLuAo4CL+nnbORHxpSYXzczMKtbUfihJm0vatDBqKvCPZn6mmZkNX81u6YwHvippEumEgL8B0wvTL5G0MA/PiYi9m1weMzNroWYf07kV6PHpZxExrZfxpzexSGZm1kLN6F7rAibWLg4ti6STgY8Bz5YZ18zMqlN6SyciHgY2aELcc4Bzyo5rZmbV8QUtZmZWGa2Il8BIepLGzpKbDMwp+eNfqjHboYztErMdytguMduhjO0ScyDxNoqINXqasEImnUZJmhURpd7L7aUasx3K2C4x26GM7RKzHcrYLjHLiufuNTMzq4yTjpmZVealnnRmOOawjfdSjtkOZWyXmO1QxnaJWUq8l/QxHTMzq9ZLvaVjZmYVctIxM7PKOOnYsCNJxf9mg+H6Mzw56dhwtCVARIQ3HDYEL291AezFXrJJR9KBkkq595ykUZJeLeljkg6QtEsJMV+0sR3qBriXmKXUgTKSg5KRwM8kXQzlJR5JowvDpd1zUNJXJK1aVrxC3IklxxtTZrwcc5UmxJxQUpzxwMWSvlBGvELc0lrhkiZJWjMPbyZp5aHGzLGaVdf3LGN78ZJMOpLOA74GnDfUHyX/wD8FdgFWyn+nS3r7UFagvLF9paTdJa1fGDeoyi5Jtcd+S3qFpO1zzO6hxCy8HDWYGHVGRMTSiNgU2FXSl3IZh5R48sbxbZJWk/SmPFxWC+o3wDVlJglJ7wPeW9ZGXdL7gS9KOrOsckraHPhgHi5lZ0jSZsCXJG07xHgjImI+cATwakkfGUq8OlOglDo5EtgOOE7SF4H/BjqGWrhm1XVJ5wNfBU6RNLRyRsRL6g9YGTiU1IVzLin5jBxkrJHAFcC7CuPWBFYHPge8bRAxa6ex7wE8CPwa+B/ggPp5BlnmDwA3ABcCdwHjBhlHheH3AN8GjgW2K+E3ekP+bZ4H/mco37v22wJHAn8B7gZWKrlOvRbYpqRYxwG3ABsUyz+EeO8DrgPWA54AvgdsOpTfO7/eMtfPaUMsX62+vxH4Sf59LgB2KGFZviXHegj47xLKuDmwEDitt+UywHhTgOuBx4A35HEjhlDOptR1YG3gJGA/4CvAh4GOQccbaoHa6Q94OzC58Hq7XCn/dzArd66Ep9d+8PyDPJ43wquS9oK3bjBWcSO+G3AxsCkwFjgxb4QPGGgZ6z5jD+BqYHSuONcVK/kgV6D3Ar/Pse8AvgHsPcTf6AFgM2DPvPJcOJgyAmsAP8vDe+eN7tXA2nncoFfwJtXPMaSdmH1IOy7Hk3YO3jfIeKsA3wTWyXXoKlLS+T8GkXhyzHHAynn4XcAnc90f0LKsq3db5t98+7xh+wRwPkNI5MA7gftJPRBH5zp6+hDi7Q98N28v5gCfHkydrIs5Mpftf4AvATsWpw0wVlPqOukxNcVt5oGkHfVBJ56XTPeapG+RNmirF5qHd5ESTgeD62qbBqyfh78GzAPeTKpIB5Cu4J3XQNnWAS4slGsP4HBgYkQ8T+q+ewB4o6S3Nlq4HprVjwM/Aj4KvA54XaTutbdA6jIYSMzcH705aUOxLekBe48BR0raq9Fy1ukGLoqI+yPiOlLiOVDSBY2WsSYingTeIWkacBMpkV0JfF3SNvm7bylppUGWtVQRsRD4FXAW8B1gQ+BOYOtiP/0A4j0LnEBqfb8lIt4AHAXsRPqN+o0paV1J1+ThtUgbyM9ImgLcQ9pxGxcD6KaVtAGpW6m2vq0G/CUi7oiIK0m/0RTgRElbNvyFlzcG+EJE3EzagfsQ8HZJnxxooNwleRpwaUQcD+wMvEfSx2FgdbIQc2dSUvw1aX18HjhG0nqSdgdeP5BusWbUdUnfIe1E/kXS4Xn01flvI+BDg+pqG2zmb6c/0l7TZXXjak3cDmAbBtHiAV4PnJSH/wfYNQ9vBjxK2ms9pMFYW5LOtlk1v/4SMAuYkl+vD5wMbNVgvGLLaSwg4GWkin5LYdqRpIq/+gBjbkfaU5sAbAVcncdvSmqdfI5+uu7oYQ+R1M1yFzC6MO4c4O/AWj29p4FyvwX4JzAx/97/DfwS+DSpS2e1VtfRQllXJiWF1fLrQ4HfAWOHEHNT0p7+tqQdhEuBDQfw/muB6/LwDsDHSRu2I4EngTMGWJ41SK2atYFJpNbTb4H3FOY5g9RleyLp2HOvv3sv9eg4UvfS6Px6FPB94MZG6npdrFHARcAmhXGHAQtq638DMXoq4xnAbaSuz9WBTwEzSd2WGw/yty6lrgPfAn6ah/cj7Tzvnl+PzuO+yiBaPC1fyZr9lyvsBYWVeGfS3t4lwP553Mi8EpxDSlANLURScvkVsDEpAV2aV+ytgB/kz9qX3B3RS4yOwvC3gD+zLPGcnleSTfLrAffPAqfkFfqLwNbAK0ndAx/K426jgW4Mlk84J5MS6oaFZXpfoYL+FFhzAPGOz8v+CFJyPBO4l9TKOSkvyzWGWA/eAPyV1HoEmJ5Xxoa6P1tUb99NSsBDOl5EOrnlI6Sdi7tpYMeF1K1ybuH1z4EbCq+nkboBr8v1fnSDZSnuTPyC1KpbC3hrXk+/nGPfAZyaf/uGEg5wEOm44vb59ZdIx8c2IXUF/rCResSyHdINgFF5+Ezg5sI8e5J6N24FOgfwW+xULAOpBXUnsE5+/UrgZa2s68ArgD8ARxbGnUXhGDUp8bw5/17/xQC671q+cjX7L6+8vyX1bZ+ah39KytLPA28qzLtL/rHWH0D8g0kH5zcg9aX+ktT83Iq0Af4+MKGfGJvUKiKpD/86liWeM3PFHtPID1u3Em4C/JjUlXZCLttWpFbKiaS9oAH17ZOS6x+p21sk7ZXOAmYD2w4g3jTSXvNHSS3NL+ff7ATgvFzmIZ+ckD/rjaRkVtsBKfWEgpLr7di8odyypHijch1dr8H5x5AS1NcL4y4Hbq2bb3VSK+qd/cSbUBjeHXgTaedvJmmnY/O8sfsB6bjTDsCrSMehVmmgvCflcnyK1DJ8b16Gn83rwHXkZNRPnFrC2ZfU4vg2cGoe903SxvgM0kMid8z1dWoD8TpIx9iuJ/UCFI+T/CR/1kYl1p9B1XXy9pCUvL9MSuQvB24G9u1h/kPztD53Mpd7T1lfcjj+sexsjnGkls3PSAcEX57Hv5/lz0Q5mtxFNoDPmEzak/g80Jlfb0faC7yLXvYuCpVxF+B20l7oKnncN0jJsVZhGkoMLJ9w9iEllQ/n1xOBY0h7lw0f6Ce1Yj5UeH04MKO2fFm+pbYZsNYAYh9Jatltn193klpfX2JZ0m1oD3oAn/nmvLxHMMxOJOjr96z4czvy/7Gklva3CtMuB26qm/800gkFPZY3x7keeBuwRd4YXpJ/5+tIOzEnkRMTaQP9elJrZ/teYhZPRNiR1NoaQdqx/DPwdVIXW20967W3oYfYO+R6+Jq8Hp1LPnEgv35H/h6vznVpSn+/H7mrmdSq+ympu6u2o3k8qRX28lbWdVIvz4/z8EjSTs83SF3bH6qbt7ZcPwbsNqBytaJSN/uPfPphHl6ptpCoO15D6jL4UP28g/i8cbmC/iZXnotJe+19dmPkCnwDqRvlNtLB41qiuSRPG8xZdUeRWhxX5BVw0zx+Qq7glwLje9tI1MVal9TnPDW/3o20J7pOYZ5DgcMbiFV/2u0GwFyWPy16R1Ir9Oxc8Uvf8ALjW11Hh+tfYWNSOxYyjpR4vl2Y52rgz3l41bxR7rMLkHSs4RbS8aHd8rhNSNf6fIfUSvkCy3YUj6CBna1cN1fLsfYgJbGRpK7p2aQWc3/HhDYC9iIluwm5TtaOUY7Kdf4c0tmptcS4XV63+m2Fk85m/VneJrwjL9Of5uV2Xl4uazfp92yorpO6Nn9ceL1x/n84aZt2AD3sADKISy5aXsmbsJCnkJqqpxbGjSwMjyUd2/gpeY+9xM9eubDS9ntciHRw8oQ8vBJwGak7qVaxpw6iDK8jJataq+mrpC6LzWuVkNzX20+c4l7amLwyfzm//hZpL/W9pIR5H4WDrA3Eez+pG/EEUkvvaeAjhelTGeIxHP8Nqv7W6u5rSMngoMLvf31xfWH503tHNRh/b+AZcu9C3qAfQOpCXo8Grs3JCeDQPPwB0p78d3J9PJ18PQ6pVT+DBrp9SNdZ7cCyYyC7AM8B786vR5C6BM8jd3eSuhUbOT50GGmHchrpdOMbSDt+40ktiTMp6RqvIfzup5MS6Ij8+kRSl19H/juWlCBPYIjXjUWsYEmH1Cc8Pm+0/gCcUphW23MT6aDYuYVppXSz1G1Y+9qz2i9XwBNJB3hrSWYy8AhwdiNxevjM0aRWx1PA+wvjzyOdPtlnYugl/omkfvKXkbr8Tssr4Xtz3G8zgIPxLLtYcX1Ssvk8aS/4EeAzra5DL/U/0kHoe0kH9p8mdQONIiWePwPfrdW7waw3+bf+G/CO/HrPHHdSg+9/E6m757OknamNSUnjI6TWUjfpepq/AlsMoFwT8/uPzK93JZ2xVUw8DZWxLu6xwDGF11uRTj5q+LhxBb/5AaQW6C6kE4xuonB2I6nleBxwVCmf1+ovXOKC+yqpS2oaKTt3kvqKT6mb74vAqwuvK+3Xz+W6jrRntQ+pxTWN1KzfktRnfg9wbAOx6k+LriXWd+Qkc0Td9+7zIDIv7v56A+lA7ur59RTSgd/TC/M03CXJiy9WvDpvOGrXUD1A2oNsybGMl/Jf3qiuCVyTN4x7kU4k+C3wlTzPGGCXEj5rf1JC+xmpq3f/Ab7/daSus0vy65XyuvM50s7cUTSwg8WLu9sPzd/3kPz6lUAXML2BWKr91Y0/mpTEa+vmiPydNx8Gv/nHgVfm4YNI3aj31C87csuytM9t9RcvaeF9O/+QK9eN3xH4E3B8fn0lcGWxolRQtpeRLswjb2wvAf6vMP39pGNAl5O6qbYg7VX0eTZQ3Wd8iHTR51XAHnncwXmlfs8A4qxbGF6btMe43K00SP3ftwJnDmYZ5g3E9sDvau8ndbkcTz9n+fmv+X+ka2i2Ie2wKW/MF5MOGJe2g0ZqSc1m2bVtA61HbyYdezmkMO7nwH4NvHdSYXha/m5b59dvIu0UHpxf7wq8voGYLy8MH0fqNjuRdB3SJ/N2aDPSsapbaPAswib+zl8n7WCsxbLTwt9A6kYtHhP/Aeli7fI+u5VfvIQFJ9LxmStZdsbNiLp5diA1F+eT94xq762ojDuSzs2vdaEdkyvduwvzbErau9yA1PqZTS+nytaXm9TPei2pFfFL0mng++ZpR5KS8Sr9fV/S8ajf54SwSx63e650n6ZwQJJ0pXzDFxf28Fk9Xay4Uavr00vtj2XHcHYiHTDeimU7Bb/J07bM9arha1EG8PlDuig3150HScckDiSdLdrnRZWkg/jfIx2z2oF0jcz383d8Z55nX9Lp/++oX1Y9LUNSC3AO6bT/Wg/LB0jHQW4hdSV/PK9LV9H6YzinAlfUfYfarY1el7cnbyEd1xnULaj6/PxWfvmSFuDm5DNN6hcM6fjOhqTTfr9SGF91l1rtTgC1/uEjSacivrNuvrVI3Ve9nhFDbo0UNhjvJ3WLnJIr9RHAC+S9MxpoPZCOhe2Zk8C1pG6u8aSugNfmlecTjcRqcHkM+GJF/zWtbu5P2sn5FOmA9ztyPbyAdOzhr6TbJbW8rL2U/0BSF9jP6OeUY5adGbcbqWfhCmCnPG65dZKU0PrtSmTZwfctSceabiVfuZ/Hfwz4Th4exTC4NiwnnU/k4dfnMt5COrnnZXmdf5DlTxwpr5Xb6gUwhAX3C9LZHyuRzggpXj1by9pbUHcbmhYknDVJeznvIrW4js7jDyd1tR1VN/+YPmIVWyO7FcZtQLogrnbe/+9JF6/1Gqsu7s6kLrrPkw7ovzmPr11b8BrSmUAfoay9nQFerOi/ptTNLUlX1a9JOoZzB/miRdIFgfsCr2p1ORv4HnvSy7UyhXlGknbIOkldZreQ7j59bmGe2jp5TIOfW9/rsC7p3oPfLIzbnNSyGvRdmUtcTr/M26FppK61n5Naep8j7bR+HLggz7tt4X2lbjNLe8BPlSRtTep//EFELJL0DdJzM+ZGxJUR8UKe9fOkhfofEdFdcXGfIiWJVUlnfH1H0pKIuCTfLO+2uvIt7CmIpFeQusneRzprbENJO0XEXEnzSStQZ372zi2kroEeY/Xgz6Rb22xC6iLYR9JCYE9Jnyf1cXeTbsoYA/juvYqIJcDDZcSyxtWeqyRpN1K//vWk4w9bkXY25kjaB3ggIv6vlWVtVKQbw/Y3z1JJs0l32F5EOnYzDjhN0gcj4rze1sme1D2f6gOkbv67Sd1Sv5H0EOm6o51I6/8qpGNQLZG3ma8nJcSZkpaSTt75POkWVnMlvZaUmImIu/L7RpS+zWx19h1C1n4jaWP5NtLZah8mNZm/QtpD+wVwcQvLty7L7pk2OZdnD9JVzH9jACcK5Bi9tUZWJfXJ1m4Hfx+N3UtttcLw/qQbN/4hvz6bdJ3CW1r9O/uv/L9cl36b15ODSNe67J2n7Zrr0M6tLmcTvnftQte/AnvlcW8g3SbnlEHG7OkSgLeSjvHMJt/zsNXfPZf1TXmb2ePZaKSLQL/c7HK0ZUsHICJ+lW/9fRawKCLOlvRKUjNxF2BWRHwampSt+yBpHKmfdENJV7Ds7gLrR8QPJB1HOiNoIHprjUwj9cd/jtSFMC4i+tyjyo8kmC7pbNIZNXNIZ9btLukPpAOjlwAjJHVERNcAy2rD20RSvbmcdObnXsCxkg4jrTunRMQtrStec0TEAkl7k04guEDSZyPiJ/kpqEdI+lFEPNJovPyUzh1Ip1q/nXSG2stIx8Smk64l+t+I+EfZ32UwIuKX+WkJZ0oiIn6UH3XwMtJx20ci4hRYviXXjIK09R8pe99JPsWxh+ktub8WqTm9O+lMlo+Rzlp5gML5+fR/RlnDrZH+YhXirE+6/uCTuaJ9k2XX4byCdCzoM7nsH6Tke5/5b3j8kQ7A30/qchGpG+itwCsGUp/a9Y90osADpDMzb2OQDx6k90sATmCY3m6J1Ev0n20m6caqHytMb+o2s21bOjWxfPZeKSIurk2ruoVTV65ngd9LOpDUX74OqWttXGGeXvckmtEakTSedCrnt0h3LziWtLI9lad9gnTN0HOkroF7I2KgLTJrAxFxRe7XP4t08sAPSHvqtenN2csdJiLiSklPkdaB/4qI3wwyziJJzwMjJW1Luo7tKtL1gPPLK3F5IvUSAZyVWzSXkrodqaJno+2TDvwn8QRwhqQxpD33M1uVcOrK9ijpgW6/kfS1iLi/v/fkkwHWJZ2y/AXS6csfiXQwdF6e9l3SKcc7kI5p9VtRImK+pEtJp8I+TbqlzcGS1iY9+/2QiFgi6cuk00ufHPAXtraRN7wdpB22mcDjw2GdqUpE/EHSnyJi6RBD/ZN0reBXSOvm22OYdKn1JpYdnvhsfjJqbZvZ9K70FSLpwHIL8Zuk22AMG7UWVy3h9NVfWkFrZCNSMrsnIs7NLakLSXdtWAIQuV/XVnwR8TNJN71UdzBKSDi11s5XSNfJdccAjgu1UqGXqNJtpnrZ9rUtSRMj4plWl2MoJO1AOqPmaVJF3pB0autC4NpCa+SsgWws8gkOHyTtlW1MulvCIaQnIh6Z52neAUQzG3aq3maOqOqDqtLuCSdbrjVC6jq7EBhbbI0MdO80IhaQEtWdpLPgjiK1qv4l6ZN5Hiccs5eQqreZK1xLp901uzWSuyBXJV2NvIB05s5/k+4M4Is1zaypVriWTrtrdmskv/dZ4MCI2Id0EsKqTjhmVgW3dIahKlojhduh+OJPM6uMWzrDUBWtkVpryQnHzKrkls4w5taIma1onHTMzKwy7l4zM7PKOOmYmVllnHTMzKwyTjpmZlYZJx0zM6uMk46ZmVXGScfMzCrz/wH9Uf8B8Im7fQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "heatmap_attention(\"@VirginAmerica and it's a really big bad thing about it\",layer=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "refined-horizontal",
   "metadata": {},
   "source": [
    "### Bonus task\n",
    "In order to make the task simpler, we have so far ignored neutral tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "comprehensive-chamber",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica What @dhepburn said.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6837</td>\n",
       "      <td>@VirginAmerica I didn't today... Must mean I need to take another trip!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6340</td>\n",
       "      <td>@VirginAmerica Really missed a prime opportunity for Men Without Hats parody, there. https://t.co/mWpG7grEZP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6769</td>\n",
       "      <td>@VirginAmerica did you know that suicide is the second leading cause of death among teens 10-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica will you be making BOS&amp;gt;LAS non stop permanently anytime soon?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   airline_sentiment  airline_sentiment_confidence  \\\n",
       "0            neutral                        1.0000   \n",
       "2            neutral                        0.6837   \n",
       "7            neutral                        0.6340   \n",
       "10           neutral                        0.6769   \n",
       "23           neutral                        1.0000   \n",
       "\n",
       "                                                                                                            text  \n",
       "0                                                                            @VirginAmerica What @dhepburn said.  \n",
       "2                                        @VirginAmerica I didn't today... Must mean I need to take another trip!  \n",
       "7   @VirginAmerica Really missed a prime opportunity for Men Without Hats parody, there. https://t.co/mWpG7grEZP  \n",
       "10               @VirginAmerica did you know that suicide is the second leading cause of death among teens 10-24  \n",
       "23                               @VirginAmerica will you be making BOS&gt;LAS non stop permanently anytime soon?  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_dataset[(full_dataset['airline_sentiment']=='neutral')].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "through-possible",
   "metadata": {},
   "source": [
    "**Design a way of predicting that the tweet is neutral**\n",
    "\n",
    "You may use the attribute **logits** from the model's output\n",
    "\n",
    "**Solution:**\n",
    " There are many possible ones of addressing the problem. We'll make a geometric exploration. Since logits is a 2D vector we can easily plot it. Thus we can see in which geometric region we have more neutral samples. We will only visualise it, but with such information, some machine learning algorithm as K-means or SVM can be applied on data to predict for the three classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "synthetic-moment",
   "metadata": {},
   "outputs": [],
   "source": [
    "def logits(text):\n",
    "    # outputs logits\n",
    "    input_tokens = tokenizer(text, return_tensors=\"pt\")\n",
    "    output = model(**input_tokens)\n",
    "    tensor = output.logits\n",
    "    return tensor.detach().numpy()\n",
    "\n",
    "logits_set, colors = [], []\n",
    "for index, row in full_dataset.iterrows():\n",
    "    text = row.text\n",
    "    logits_text = logits(text)\n",
    "    logits_set.append(logits_text)\n",
    "    \n",
    "    if row.airline_sentiment=='positive':\n",
    "        #label = 1\n",
    "        colors.append('tab:red')\n",
    "        \n",
    "    elif row.airline_sentiment=='negative':\n",
    "        #label = 0\n",
    "        colors.append('tab:blue')\n",
    "    else:\n",
    "        #label = 2\n",
    "        colors.append('tab:green')\n",
    "        \n",
    "    if index > 200: # we don't need the whole thing\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "oriented-singer",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Logits from Twitter sentiment dataset')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlIAAAF1CAYAAAAjngRgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABMEklEQVR4nO3dd3hcxdXH8e+5W6RVde/d2IArBmGbbkrooQdCCIHQ0yEkJC8E0iCEJBACJPRAEkpC7z20gCku2Ma4gXtvKlbdduf9Y9dGsiRbtiWtJP8+z6MH7cy9c8/dXbTHM7Mz5pxDRERERHacl+kARERERNorJVIiIiIiO0mJlIiIiMhOUiIlIiIispOUSImIiIjsJCVSIiIiIjtJiZRICzOzu8zs2mZq6ztmttbMKsysa3O02RrMbEA65kCmY2kJZvaZmU3KdBxNYWZLzOyoTMch0lEokRKppSU+ZJxzlznnfptuf5KZrdjJ2ELALcDRzrk859zG5oxzO9e+Op0IVZhZjZklaz3+bHvnO+eWpWNOptt728wu2uoazsz2aKl7aC5m9qCZXV+7zDk30jn3dgZiqfc8NnP7rfKatJfXXqQhSqRE2o+eQDbQYOJiZsGWurBz7nfpRCgPuAz4YPNj59zIlrpuU7XkvYuIbIsSKZEmMLMsM7vVzFalf241s6xa9VeZ2ep03UW1/4W9uQfDzHKBl4E+tXpz+pjZeDObamab0sN2tzRw/eHA/PTDUjN7M13uzOx7ZvY58Hm67GIz+8LMis3sOTPrU6sdZ2bfNbPPzazczH5rZkPNbHL6+o+ZWXgHnpdfm9nt6d9DZlZpZn9MP46ke6+6mNmg9LWDZnYDcAhwR/o5uMPM3k03OTNddla6jRPNbIaZlaZjHFPr2kvM7GdmNguo3DqZspQ/m9m69L19amajar2efzKzZenn/C4zi6TrJpnZCjO7Mn3uajP7drruEuAc4Kp0nM/XiuWo9O+/MrPHzeyh9HP8qZkNN7P/S7e33MyOrhVnoZndn77OyvR7JZCuO9/M3kvHWmJmi83suHRdveexkdfoXDNbamYbzeyarerGm9kH6ed3dfq1CKfr6r0mZtbZzF4ws/XpeF4ws3612jvfzBal73uxmZ1Tq+4CM5ubPu9VMxvY2HUafreJtFHOOf3oRz/pH2AJcFQD5b8BPgR6AN2BycBv03XHAmuAkUAO8BDggD3S9Q8C16d/nwSs2KrtD4Bz07/nARMbiW1Qut1grTIHvA50ASLAEcAGYF8gC7gdeHer458FCtLxRoH/AkOAQmAOcN52nqPzgffSvx8BfJr+/UBgIfBRrbqZDcUOvA1ctFW7W56z9ONxwDpgAhAAzku/Plm1XqsZQH8g0kCcxwDTgE6AAXsDvdN1fwaeSz9v+cDzwI21XqNE+jUPAccDVUDnrV/Pht43wK+AmvT1g8A/gcXANen2LgYW1zr3aeBuIJfU++tj4NJaz3U8fU4A+A6wCrDGnset4hoBVACHpt8Pt6TvbXOs+wET03EOAuYCl2/jNekKnE7qfZ4PPA48k67LBTYBe6Yf9wZGpn8/Gfgi/RoEgV8Akxu7jn70055+1CMl0jTnAL9xzq1zzq0Hfg2cm647E3jAOfeZc66K1AfpjogDe5hZN+dchXPuwx08/0bnXLFzrjod59+dc9Odc1Hg/4ADzGxQreP/4Jzb5Jz7DJgNvOacW+ScKyPVYzZuB679ATDMUhPfDwXuB/qaWR5wGPDODt5LbZcAdzvnPnLOJZ1z/yCV+E2sdcxtzrnl6XvfWpzUh/1epBKPuc651WZm6bavSD9v5cDvgK9vde5vnHNx59xLpJKRPXcg9v855151ziVIJRvdgd875+LAv4FBZtbJzHqSStQud85VOufWkUryasey1Dl3r0vNL/sHqQSlZxPjOAN4wTn3bvr9cC3gb650zk1zzn3onEs455aQSugOa6wx59xG59yTzrmq9PN2w1bH+8AoM4s451an32OQGg6+Mf0aJEg93/ts7pUSac+USIk0TR9gaa3HS9Nlm+uW16qr/XtTXAgMB+aZ2RQzO3EHz699vTpxOucqgI1A31rHrK31e3UDj/OaeuF0AjOV1IfpoaQSp8nAQex6IjUQuDI97FRqZqWkep/61Dqm0efaOfcmcAfwV2Cdmd1jZgWkkpocYFqtdl9Jl2+2Mf2Bv1kVO/C8UP853ZBOhDY/Jt3eQFK9VKtrxXI3qZ6pzdbUuqeqWuc2RZ33pnOuktT7AUgNGaeH59aY2SZSCU63xhozsxwzuzs9VLgJeBfoZGaBdNtnkUqaVpvZi2a2V/rUgcBfat1jMalewr4NXEakXVEiJdI0q0h9GGw2IF0GsBroV6uu/zbacfUKnPvcOXc2qQ/Pm4AnLDWfqqlqt1knznQ7XYGVO9DejnqH1DDeOGBK+vExwHhSH7QNqfc8NGA5cINzrlOtnxzn3KNNbcc5d5tzbj9SQ1zDgZ+SGvqsJjXstLndQpeaSN8UTYm9qZaT6mXrViuWAtf0Cfzbi2U1td6PZpZD6v2w2Z3APGCYc64AuJpUgtOYK0n1zE1IH3/o5qYB0r1wXyHVazYPuDddv5zUcGXt1zLinJvclJsUacuUSInUFzKz7Fo/QeBR4Bdm1t3MugHXkZoLBfAY8G0z2zv9QbWtNaPWAl3NrHBzgZl908y6O+d8oDRd7Dd0chM8mo5lH0tNhv8dqTlLS3ayvaZ4B/gWMMc5FyM9b4fUPKD1jZyzltS8rG2V3QtcZmYTLCXXzE4ws/ymBGVm+6fPDQGVpOYt+enn+V7gz2bWI31sXzM7pkl323DsO8U5txp4DbjZzArMzLPU5P9Gh9d2MJYngBPN7OD0JPLfUPfvfj6peU0V6d6j72yn/XxSSWipmXUBfrm5wsx6mtnJ6eQ9Smo4dPP7+C7g/8xsZPrYQjP72g7ch0ibpURKpL6XSH1YbP75FXA9qSGsWcCnwPR0Gc65l4HbgLdITajdPMcpunXDzrl5pJKdRelhjj6kJqt/ZmYVwF+Arzcy52e7nHNvkErkniTVGzGUuvNtWsJkUhPdN/c+zSGVtDTWGwWp+zwj/Q2u29JlvwL+kX5eznTOTSU1yfoOoITUc3v+DsRVQCphKiE13LkR+GO67mfp9j5MD1G9QdPnQN0PjEjH+cwOxNOYbwFhUs9bCankp3cTz23oedwiPUfpe8AjpN4PJUDtdcx+AnwDKCf1XP1nqyZ+Ra3XBLiV1Gu9gdT7/JVax3rAj0n1ihaTGtr9TjqOp0n1tv47/XzPBo7bxnVE2o3N3/wQkWZiZnuT+qDI2mqejYiIdDDqkRJpBmZ2qqXWJupM6l/ezyuJEhHp+JRIiTSPS0mtebQQSFJ/romIiHRAGtoTERER2UnqkRIRERHZSUqkRERERHZSRnZM79atmxs0aFAmLi0iIiKyQ6ZNm7bBOde9obqMJFKDBg1i6tSpmbi0iIiIyA4xs6WN1WloT0RERGQnKZESERER2UlKpERERER2khIpERERkZ2kREpERERkJymREhEREdlJSqREREREdpISKREREZGd1GETKReLoQ2ZRUREpCV1uERqw333M2/f/Zg3ZiwLivZn4z//qYRKREREWkRGtohpKWtvvpnie+/b8tivrGTdjb8H36fr+ednLjARERHpkDpMj5TzfYrv/3sDFY71f7mt9QMSERGRDq/D9EjFliwB32+wzlVX45JJaubNY8Pf7iQ6bx5Zw4bR7XvfJTJ6dOsGKiIiIh1Gh0mkvOxsMIOG5kMFg1TPmMGyCy/CRaPgHPFVq6j88EP633UnuRMntn7AIiIi0u51mKG9UJ8+hPr1a7Cu8JRTWHX1Nbiami8TLedwNTWsuf6GVoxSREREOpIOk0gBDHzwAQJduqR6ptJyJkwgd+JE4kuXNnhO7IsvcI0MCYqIiIhsS4cZ2gMI9e3LsPf+R9XHU0isX0dkn30I9e3L/P2Ktnle1ccfa3hPREREdliHSqQAzPPInThhy+PoosWpeVHbsPyy7zDoP/8he8/hLR2eiIiIdCAdamivIV5WuM5QX0NcLMbGv98PQHzVKtbcdBNLL7qY9X/9G4ni4tYIU0RERNqhDtcjtbVQ376EBw4ktmhR4wf5PrGFi6j48COWX3ghJJMAVL3/PsUPPsjgp58i3MhEdhEREdl9dfgeKYD+d92ZmoTuNXK7wSDZY8aw4rLLtiRRADiHX17Omt9e3zqBioiISLuyWyRS4QEDGPbuO/S/604i48dDVtaXlWZ42dnkFO2XWh6hAZXvvotLJFopWhEREWkvdotECsCCQfIOPZSBDz5AjyuuINi7N15eHnmHHcagx/6Dq9nGhHTntN6UiIiI1NPh50htzTyPruefR9fzz6tT7mIxCATqDu3VUvb00/T4yU8I5OWSLCsjsWEDoX798Gr3bomIiMhuZbdLpBqTvddeRPbbj+qPP26w3nkeZcs+p/qBRyh/9VUsGMQB3b//fbpe8O3WDVZERETahN1maK8pBtx7D6EBAxqsq0lU8/h151D8you4WAy/qgpXVcW6v/yFTS+/3MqRioiISFugRKoWLyuLfrffhkUidcqjQfhkCBwwxycY32o7mWiUVdf9EtfIkKCIiIh0XEqktpK9554M/Ne/yJk4kWhWgKowBJKwz0IINrIlnysvZ9W117VuoCIiIpJxSqQaEBk1koEPPsDzx3bC8yHoIJKAba2Pvunppyl/++3WClFERETaACVS23DUlDjZTV0+yjlWXPYdFhxyKBXvvtuicYmIiEjboERqGzqTs8PnJNevZ/kll7L+rrtaICIRERFpS5RIbUPnY46F0M6tELHh1r9ow2MREZEOTonUNnS79FJCPXp++S2+YBCLRCg87TQIbjvBcsDDv7uXc+/7iHveXcimmnjLBywiIiKtqtkW5DSzADAVWOmcO7G52s2kQKdODHn+OUqffZaqDz8i1L8fnb/+dcL9+pF38EGs+8ttxJcubfT8Lz5fyf+CG5iypJgH/reQF394CF3yI40eLyIiIu2LOeeapyGzHwNFQMH2EqmioiI3derUZrlupn1++BEkVq+uVx6zANccdAmzuw0FIJhMcOrKj7nma0V0OvXU1g5TREREdpKZTXPOFTVU1yxDe2bWDzgBuK852mtPBj/7DJadXacsbh7Te+7J7K5DtpQlAkEmd96D1Vdfw8qrfkayorK1QxUREZFm1lxzpG4FrgIaWbISzOwSM5tqZlPXr1/fTJfNvGBBAcM+mJza8BiI4XH72NP57YTzwequPJUbrwbn2PT88yw56yz8aDQDEYuIiEhz2eVEysxOBNY556Zt6zjn3D3OuSLnXFH37t139bJtiquqwoKpRCqMz8r8HthWQ6bZiSinLHovfYIjvmolm17SHn0iIiLtWXP0SB0EnGRmS4B/A0eY2UPN0G67EejcGS8vb8vjaz7+J/3L15GdiJITryaUjHP84g+YtOKTLce46hrW3XILyy77Dit/9jMqP/44E6GLiIjILmi2yeYAZjYJ+MnuNNl8s7IXXmD1L67F1dQA4DyPhZ36sjGUx7DSFXSJlm/zfMvKosv559HjiitaI1wRERFpom1NNm+25Q92d4Unnkiwa1fW//VvxFesIDJ2LIceeyyrrroK4ttfQ8pFo2y8+x7CgwfT6ZRTWj5gERER2WXN2iPVVB2xR6oxS775TapnzmpSMgVAMMger71KqE+flg1MREREmqTFlz+Qxg245x46n3lmvSUSGuUcpU8/XbfIb/TLkCIiIpJBSqRamJeTQ69rf8Ge06cRGjhwyzIJjUomSWzYQHFljNcefIaZRxzNvBEjmT9+AuvvuktJlYiISBuiRKqVmOcx8F//JKeoCAuFGk2oLCeHlwZM4NtX3kf3P15HeNVyAPxNm9h49z2su+XPrRm2iIiIbIMSqVYU6tGDgf94kGHvv8ewd9+h4KSTIBzeUm/Z2awdWcQfFzrOmv0y2cm686pcdTUlDz2EX13d2qGLiIhIA/StvQwIFBQA0Oem35N32KGU/uc/+NEohV/9Ks912ZfEe0voX7Gu4ZPNSGzYQLh//1aMWERERBqiRCqDzIzCE06g8IQTtpRFX5hD0ncsKehNl5pNDXYZBnv0aL0gRUREpFEa2mtjjh7Rk+xwgIf2OppYIFS3Mjubrt/+Nl5WVmaCExERkTqUSLUx4wd34fhRvVjeawjXTbyQLwr7kDCPaEFnev74Crr94PsAOOfwYzEysQ6YiIiIpGhBzjbIOcf7X2zkhVmryAp6nLZvP8b277SlvuTf/2H9bbeRLCkh0KUz3X/4IzqfdWbmAhYREenAtEVMO2NmHDysGwcP61avruSJJ1h700249Df3khuLWXvjjRAI0PmM01s7VBERkd2ahvbamQ2337ElidrM1dSw4fbbMxSRiIjI7kuJVDuTWNfwsgiNlYuIiEjLUSLVzoT69m24vF+/Vo5ERERElEi1M91/+pN6GyBbdjY9fvKTDEUkIiKy+9Jk83am8Jhj8AIB1t3yZ+IrVhDq358eP76C/COPrHes73yWbVpGdjCbXrm9MhCtiIhIx6ZEqh3KP+oo8o86apvHTFkzhZ//7+eUx8rxnc+QwiHcMukW+uVrCFBERKS5aGivA1pdsZrv/fd7rKtaR3WimmgyyvyS+Xz71W+T9JOZDk9ERKTDUCLVAT35+ZMk/ESdMt/5lEfL+WjNRxmKSkREpONRItUBrapYRdyPb3kcSDq+/naS2/5URuExl7DswguJLlyYwQhFREQ6BiVSHdCE3hOIBCNbHv/wWZ8Tpjjyq8GLJ6mc/AFLzvo68bVrMxiliIhI+6dEqgM6dvCx9MrpRdgL073Usd9CR1btkb70hscl/3oIgGWblnHnjDu5ZeotTFs7TRshi4iINJG+tdcBZQWyeOSER3jwswdZ8sqTJIPrIOHXPSgWo/rTT3nm82e4/qPrSfpJki7Jv+c9ylH9j+SGQ2/EzDJzAyIiIu2EeqQ6qLxwHt8f931+d/YD5LhQ/QNCIWz4YK7/6HqiySgJl8DhqE7W8NqCF3nhhkvwt9rTT0REROpSItXBhQcNImf8eCwrq065hULMP3IYAQvUOycacrxe9iErf3xla4UpIiLSLimR2g30u+0vFJ56aiqZMiN7zGgG/vOfWO8eDQ7fmYNgzKdy8mTiq1dnIGIREZH2QXOkdgNeJELvX/2SXr+8DnwfC6R6oQ6MD8Z3fr3jwwk4bLZPcecg93/0SyZXf0Z2IJsz9zyTC0ZfQMhrYKhQRERkN6Qeqd2ImW1JogByQjncMukWsvwA2TEIxxyhuOPEj336boCrzozyetlHbIptYmPFWmY/eievnvMVVl93HdWzP8vgnYiIiLQNlomvuhcVFbmpU6e2+nWlYcVLF/D4NV+n2o+yzyKfnqXw/EFhHjvYiHpJAknHdY8kGbwWsuOA52HhMD1/dhWdzz470+GLiIi0KDOb5pwraqhOPVJCl4HDOf83T3J6p0n0TuYT6tuXJYcOJeql9uU7cI77MokC8H1cTQ1rf38TyfLyzAUuIiKSYUqkBICsIYPp/7e/sueUj9njv2+w1+hJhL0wAAfOc18mUbW4YICqKepZFBGR3ZcSKWnQmXueSdBLfRehIgsaGgD2q6qoCiQaqBEREdk9KJGSBvXK7cX9x9zPsE7D+HRI/bWmILVMwqvvPQhAcU0xLyx6gdeWvEZVvKoVIxUREckcLX8gjRrVbRRPnfwUczf+hcSLdxHcaqUEA2o+mcGjcx/l5mk3E7AAZobvfG49/FYO7HNgRuIWERFpLeqRku0q6NmfWAMpdywAZdmOm6feTDQZpSpRRWW8kupENZe/dTmV8crWD1ZERKQVKZGS7er9lRNxHmy9dKczWNjHI+HXnydlGO8sf6d1AhQREckQJVKyXV44TO45Z7GxAGqCUB2CTRF4ucijU7e+JEnWO8fhiCajGYhWRESk9SiRkibZ68fXMvjwk1jZK8i8oVlUFYQ5OzqGk067ikgwUu/4pEtyUN+DMhCpiIhI69Fkc2kSCwTof9NN9Fr1I2rmzyfUpy/Zew5noHMcvupw3lr+FtWJajzzCHthLht7GT1yegAQX7mS0meeJVlSTN4hh5B7yCGYpxxeRETaP20RI7vMOccHqz7g1aWvkh3I5qShJzGy20gAyt98k5U/vhKXTEI8DqEQgfx8Op97Ll3O/jqBTp0yG7yIiMh2bGuLGCVS0mL8WIzPDzgQv7KBb+95HoGCAgY98Tjhfv1aPzgREZEm0l57khHVM2aAWcOVvk+ytJS1v7+pVWMSERFpTkqkpMVYKATb6fGseOstMtErKiIi0hyUSEmLiYwZg2Vnb/ugZJJ1f/hD6wQkIiLSzJRISYuxQID+d90JodA2jyt55FESJSWtFJWIiEjzUSIlLSoyejSD/v3oNpMpC4eJzl+w5bFzjicWPMHxTx3PAY8cwKWvX8r84vmtEa6IiMgOUSIlLS4yciR9bvwdBAIN1rtYjFCf3lse/23G3/jDlD+wvHw5FfEKJq+azLde/haLyha1VsgiIiJNokRKWkXhiScy8JGH6/VMWShEZJ99CA8YAEBVvIrHpz7AHp9X0m/9l5PQa5I13D3z7laNWUREZHu0srm0mpyxY+l3+22sue6XJMvKwDnyDjuM3jf+DkgN6S275Sb+8q9K4gEI+LCyK9x4ZoBNuT6zN8zO8B2IiIjUpURKWlX+pEnkvfM2iXXr8HJzCeTlbakrf+113L+fI5yAcCJVNnAdnPShz0NHBlhXtY7ffPAbLh59Mb3zejdyBRERkdajoT1pdWZGqGfPOkkUQPEDD0B1TZ2yt0Ybjx2SepvWJGt4+vOnOeP5M1hatpTpa6czv3i+1qESEZGMUY+UtBnJ0tI6jxMePHSkRyz85eroCZegPFbOKc+eQnYwm6RL0j3Snb8e+VcGFQ5q3YBFRGS3px4paTPyDp9UZzL6ms7QUF+Tw5FwCSriFVQnqllevpyLXruIpJ9srVBFREQAJVLShnS9+GKCnTtjWVkAFNR4JAON7NVXi8NREa9g2tppLR2iiIhIHRrakzYj2KULQ557lpJHH6Xivfco6NOHiV3X81H5p8T82DbPTfpJVk99l+UvPoBLJCg8+SQKjj8ea2TtKhERkeZgmZioW1RU5KZOndrq15X2pzJeyc/f/TmTV08m5IWIJ+M4HHE/XvdAB/svNK58PI4HWCRC7vjx9LvrTsy236slIiLSGDOb5pwraqhOPVLSpuWGcrn9yNvZUL2B4ppieuX24tsvf5sFpQvqHmgwa4DP5BHGwXMcrrqayilTqPrwQ3IPOCAzwYuISIenOVLSLnSLdGN45+EUhAu4euLVhL1wvWOiYeNvJ3jccKbH533AVVVR8b//ZSBaERHZXSiRknYn6AUJBRreBDkRNGYOMf59iEdVllHy8CMsOu10Sh57jOpPZ+PHtj3XSkREZEdoaE/anZFdRxLyGk6kAEYudVz1pE9WAhxRonPmsOa6X0I4jBcK0fOX19HppJNaMWIREemo1CMl7U7QC/Knw/5EJBghy8uqV3/um6kkqp5YDL+ykjXX/ZLq2Z+1fKAiItLhKZGSdmlC7wm8fNrLXL7f5fXmS/XbsO1zXSxG8b/+1YLRiYjI7mKXEykz629mb5nZHDP7zMx+1ByBiWxP10hXvjnim3xn7HeIBCJbykvyt3Oi75NYs4bkpk1Uf/YZiZKSlg1UREQ6rOaYI5UArnTOTTezfGCamb3unJvTDG2LbNcFoy+gIl7Bw3MfxjOPpw+t4eJXfAKxhsb3gKwsXDzO54ccioVCuFiMgpNOovevfokFNW1QRESartkX5DSzZ4E7nHOvN3aMFuSUllCdqGZD1Qa653Sn6tEnWH/H7fgVlZD8cg8+C4ex7GxcLIarqfmyPDubLueeS48rf5yJ0EVEpA3b1oKczZpImdkg4F1glHNu01Z1lwCXAAwYMGC/pUuXNtt1RRrikkmSmzZRNXUaJQ8/TLKkhPyvHEXJI4+SLC6ud7yXm8vwqVO0ErqIiNTRKomUmeUB7wA3OOee2tax6pGSTJo7egzE4/UrzNjr01ka3hMRkTq2lUg1y7f2zCwEPAk8vL0kSiTTIiNHNlietccedZKo6pkzWXHFj1nyjXPYcNfdJDdtavA8ERHZfTXHt/YMuB+Y65y7ZddDEmlZPa+5GotEwEu//c2w7Gx6/uIXW44pffpplp53PuWvvEL19OlsuPNOFp18CsnS0swELSIibVJz9EgdBJwLHGFmM9I/xzdDuyItIjJ6NIMff4yC444jPGQI+ccczaBHHyF3wngA/FiMtdffkJqMnh76dtEoyY0b2fjggxmMXERE2ppdngzinHsP0OxcaVey9tiDvjf/qcG66ILPGyx3sRgVb75Fj8svb8HIRESkPdHK5iJbCXQqxCUaXoMq0KVzK0cjIiJtmRIpka2E+/Uja889IRCoU26RCF3OOy9DUYmISFukREqkAf3uuJ2s4cOxSAQvLw/LyqLbZZeSf/jhmQ5NRETaEC2YI9KAUI8eDHn6KWoWLCC5cSPZI0cSKCjIdFgiItLGKJES2Ybs4cMzHYKIiLRhGtoTaUbNvXeliIi0bUqkRJpBbPlyll10MfNGjWbe2H1YdfU1JCsqMh2WiIi0MA3tieyi5KZNLDnzTJJlm6gJ+Mwc6OMveI79L5rLqEef3LIJcnFNMQ/NeYgPVn9A79zenDfyPMZ2H5vh6EVEZFcokRLZRaXPPItfU8P0wY4/nxLA8wFz/NX7nP978w7OOPIHbKjewBnPnUF5rJyYH+OzDZ/xvxX/41cH/ooThpyQ6VsQEZGdpKE9kV0UnTuXclfDLad6RMNGdbZRnWXEQvD7FX9nRfkK7vv0PspiZcT8GAAOR02yhhs+uoG4H8/wHYiIyM5SIiWyi7L23puPR4axBuaZJ3G8vPhl/rfifyT8+qulJ/0kS8uWtkKUIiLSEpRIieyiTqecTDwnhN/AjpNJfCrjlXTJ7tLguQk/QWFWYQtHKCIiLUWJlMguChQU8NWf/g3z6v/vlB3MZlL/SZw38jwiwUiduqAXZFyPceTMWsSKK37MsksupfSZZ3BxDfWJiLQXmmwu0gyGDh/PeRUX8a85/yKajOJwRIIRjhl4DGO7j8XMWFi6kHs/vZeQFyLux9m7y978cu7eLP/Hd3DV1QBUTZlC6RNPMvDBB7Cg/vcUEWnrLBMLCBYVFbmpU6e2+nVFWton6z7h+YXPk/ATHDv4WA7ofcCW5Q8AymPlzC+eT/ec7vSpjrDwK1/BxVIT0Bf0gccO9Vje3WNo52H86Kjr2KfHPhm6ExER2czMpjnnihqq0z95RZrRuB7jGNdjXKP1+eF8inql/l8se/dFCIUgFmPWQI/fHjuW6opxUOqz3k3l01cv5vYjb+OAPge0VvgiIrKDlEiJZEggPw8zwweuH/8tKtYNA5cFQHXFMBIV07np45t45pRnMhqniIg0TpPNRTIk94ADsGCQWd2GUp74MokCwGURL9uPz9eVa/8+EZE2TImUSIZYOMyAv9/P9IH7gAs1dASh6Og6c6xERKRtUSIlkkHZI0Yw6IJvEWgwV/KZNGBCa4ckIiI7QImUSIadWtSfUANLHYQCIX5zzOlNbqcqXsXC0oWUx8qbMzwREdkGTTYXybDehRFuP3tfLv/3J5iB7xzBgMd939qfguzwds93znH7J7fzrzn/ImAB4n6cU4edys/H/5ygp//FRURakv7KirQBXxnRk2nXfoWPFxcTDBj7D+pCKNC0DuNH5z3KQ3MfoiZZs6Xs2S+eJS+Ux+X7Xd5CEYuICGhoT6TNyA4FOHR4dw4c2q3JSRTA32f/nepEdZ2ymmQNj857FN/5zR2miIjUoh4pkXauNFraYHlNsoZYdSWx9z4gWVpKzv77kzV4cOsGJyLSwSmREmnnRnQdwSfrPqlX3iVQyHf+cAg+PofO9Zj4O0eXk0+h169+pSUVRESaiYb2RNq5nxb9lOxANsaXyZGHR1lNCVMHJ5k+2HHnUQluPj5B6XPPUf7a6xmMVkSkY1EiJdLOje4+moeOf4gjBxxJv7x+jOwyEg8jXqu/ORo2Zg025nSroeQ//wEgWV5OsqwsQ1GLiHQMSqREOoA9u+zJzZNupqhXEfNK5pF0yXrHREMwc7BHsqyMJd/4BgsOPIgFBx3M4jPOILpoUQaiFhFp/5RIiXQQj859lFcWv0LSJXHU358vlIS8RJD40qVUz5gJ8TgkEtR8Noel3zgHv7IyA1GLiLRvSqREOohH5j3y5VpSDUwmN2ccXtoLfD/1s5lz+LEo6194iYpoopWiFRHpGJRIiXQQlfH6PUrmO4IJiCSD3NDrIgYfdwZ+LFb3vGAWN+59Ggd9EmGfX7/GMX9+h0+WlbRW2CIi7ZoSKZEO4pC+hxCwQJ0y5xmd8rrxv299yLEn/oj4iCFs7BKqM/B33cQLmdxnNAmMhO+Yv7aCc+77iOXFVa17AyIi7ZASKZEO4gf7/oDCrEKyAlkABC1IJBjhD4f9gepENZe9fhknLPopPzo/wXe/F2TWIGNxQS8WdupHPFB3Sbl40ucfHyzJwF2IiLQvWpBTpIPokdOD5055jscWPMbUNVMZXDiYs/c6m4EFAzn7xbOZVzyPhJ+AAGwsgD+eEeDsNwYSDAaIbtVWPOn4fG1FRu5DRKQ9USIl0oEUZhVy8eiLuXj0xVvK5hfP54uSL1JJVC2JcIA1F+9N4q0QJOruyZcV9Nh3QKfWCFlEpF3T0J5IB7emcg1Br/6/mZIuSWnyC74yoifZoS//FHgGkXCAb04c2Jphioi0S0qkRDq4vbvuTSwZq1eeFchi/1778+ez9uH7h+9Bz4Is8rOCHD0wl4f6rCf45qtaW0pEZDvMufoL97W0oqIiN3Xq1Fa/rsju6rcf/JbnFj63ZZ2pgAXolNWJZ095lsKsQgCc77P6mmvY9PIrqXWmgkHM8xjw9/uJjBmTyfBFRDLKzKY554oaqlOPlMhu4JqJ13DV/lcxpHAI3SPdOXWPU3nsq49tSaIAyl97jU2vvIqrqcHFYriqKvyKCpZ/93s4399G6yIiuy9NNhfZDXjm8bU9v8bX9vxao8eUPPY4rrq6Xrmrrqbm00+JjB3bkiGKiLRLSqREJCXRyPYwZrhk/U2QGzJn4xxum34bczbOoXduby4bexmHDzi8GYMUEWlbNLQnIgAUnnwyFonUr/A8IqNHb/f8ORvncN4r5/H+qvcpiZYwp3gOV717FU99/lQLRCsi0jYokRIRAApPPomcfffFcnIAsHAYy86m7803Y6HQds//y/S/UJOoqVNWk6zhlmm3kPSb1qMlItLeaGhPRACwYJD+995D5QcfUDl5MoEuXSg88auEevZo0vmfbfyswfKaRA3FNcV0z+nenOGKiLQJSqREZAvzPPIOOoi8gw7a4XN75fSiLFpWv02MgqyC5ghPRKTN0dCeiDSL7+zzHbID2XXKsgPZnDH8jC0bKYuIdDRKpESkWRw54Eiu2v8qCsIFZAWyyA5kc/rw07my6MpMhyYi0mI0tCcizeZre36NU4edSnFNMYVZheqJEpEOT4mUiDSroBekR07TJqiLiLR3SqREJCOSFRVUTZmCF4mQU1SEBfXnSETaH/3lEpFWV/LEk6y9/vpU8uQcFg7T/567m7Twp4hIW6LJ5iLSqmrmL2Dt9dfjamrwKyrwKytJlpSw7KKLcbFYpsMTEdkhSqREpFWVPvlEwwlTIkHF+++3fkAiIrtAiZSItKpkWRn4fr1y5xx+RUUGIhIR2XlKpESkVeUfeeSW/fzqSCTImTCh9QMSEdkFSqREpFXlH3kkkTFjsEgkVWCGRbLpetmlhHpo2QQRaV/0rT0RaVUWCDDgvnvZ9MqrbHrpJbzcXDqf+TVy9t8/06GJiOwwJVIi0uosGKTwxBMoPPGETIciIrJLNLQnIu2O7zucc5kOQ0REPVIi0n4s2VDJ1U9/yoeLNhLwjONG9ea3J4+iMCeU6dBEZDfVLImUmR0L/AUIAPc5537fHO2KiGxWVh3nlL+9T1l1HOfATzpenr2az9eV89IPD2FByQKeX/g8cT/OUQOPoqhnEWaW6bBFpIPb5UTKzALAX4GvACuAKWb2nHNuzq62LSKy2VPTVhCN+9Qe0YsnHcs2VvGbtx7hhVW3EkvGcDie/uJpjh10LL8+8NdKpkSkRTXHHKnxwBfOuUXOuRjwb+DkZmhXRGSLeWvKqY4n65Unnc9js9+hJlmDj4/DUZ2o5pUlrzBt7bQMRCoiu5PmSKT6AstrPV6RLqvDzC4xs6lmNnX9+vXNcFkR2Z2M7ldAJBSoV+47n3D2hnrlNYkaXl/6emuEJiK7sVb71p5z7h7nXJFzrqh79+6tdVkR6SBOGdeP3KwAXq2RunDAo3dnCOWsrHe8Zx6RYKQVIxSR3VFzJFIrgf61HvdLl4mINJu8rCDPff9gvjKiJ1lBj7ysIGfu34//XHIQjvp79wW9ICcM0TpVItKymuNbe1OAYWY2mFQC9XXgG83QrohIHX06Rbj73KJ65TdPupkr374SzzwcjqSf5PJ9L2dY52EZiFJEdie7nEg55xJm9n3gVVLLH/zdOffZLkcmItJEh/Y7lDfPfJN3V7xLLBnjkH6H0C3SLdNhichuoFnWkXLOvQS81BxtiYjsjPxwfrMN5W2s3siby98k6Sc5rN9h9M7r3SztikjHo5XNRURqeXnxy1z7/rUYhsPxp6l/4vvjvs/5I8/PdGgi0gZprz0RkbTimmKuff9aoskoNckaosko0WSUOz65g4WlCzMdnoi0QUqkRETS3l7+Np7V/7MYTUZ59otnWz8gEWnzlEiJiKQl/ARJv/7q6QD/nPNP1laubeWIRKStUyIlIpJ2aL9DifvxBuuSLskVb1/RyhGJSFunREpEJK1Xbi86Z3VutH5u8Vw2Vm9sxYhEpK1TIiUiUsvhAw5vtC5gAaLJKAB+NErlhx9SNX06LtnwcKCIdHxa/kBEpJaLx1zMc188R9zVH+KLBCLcOeNOeqyNUvTnN+hSEwDnsOxs+t91F5HRozIQsYhkknqkRERq6ZvXl4eOf6jOhsceHoZRnazmmYXP8EDJy/zwvBjzCirwKytJbtzIsgsvxI9GMxi5iGSCEikRka2M6DaCt898m+smXsdxg45jVLdRBL3glmG9eAhqwsbtJwVwm09KJql4+52MxSwimaFESkSkATmhHL6259f4w2F/YEP1hga/zVecBxvzU7873ye5qayVoxSRTFMiJSKyHeFAuMFyZxDaPM/c98mdOLH1ghKRNkGJlIjIdpwx/AyyA9l1yrwkDFnjKKwCi0TodNZZhPv3z1CEIpIp+taeiMh2nLP3OUxfO53JqyZjZngYeRbm5ysGkHd4IZ3OOpO8ww7DOcc7K97hhYUv4JnHyXuczITCsWz869/Y9OKLYEbhqafQ7dJL8SKR7V9YRNo8c85t/6hmVlRU5KZOndrq1xUR2RULShYwe8NseuX0YkLvCQS8wJY65xzXvHcNbyx7g+pENQC5XjZ//meQTmsqIZ6eY2VGaMAAhrz8Ep6nQQGR9sDMpjnnihqqU4+UiEgTDe88nOGdhzdYN2vDrDpJFMDe86rIWuND7XnqzhFfupQVl1zCgPvua+GIRaSl6Z9DIiLN4P2V71OTqKlTNmS1I9Lw1n1UfvgRNfPnt0JkItKSlEiJiDSD3FAuIS9Up2x9JyNhjZzgHJWTP2j5wESkRSmREhFpBscNPg6zulnT5L2NZKDh4y0cJlBY2AqRiUhLUiIlItIMeuT04A+H/IFIMEJuKJe8UB7BnDxq/vBTsAa6pczIP/orrR+oiDQrTTYXEWkmRww8grf7vM2UNVMwM8b3Gk92MJuK/GGsvPyKLcdZKES/v/2VQF5eBqMVkeag5Q9ERFqBi8WomjEDCwSIjB2LBfXvWJH2QssfiIhkmIXD5I4fn+kwRKSZKZESEWmnZqybwT9m/4P5G1dg1Xuxd+7xnDdhBKP7aRK7SGtRIiUi0g49+8WzXP/h9am1qwyc/wVLSt7khXuv4P+OKeK8AwdlOkSR3YK+tSci0oatL4/y7IyVvDFnLdFEEoBYMsaNH99ITTKVRAGYl8AClfgFb/O7l+ZSVtXISqAi0qzUIyUi0kbd8dY8bn9nOkGXi5GF5xn/+PZ4snPXNHi8eUmCeXPxNp3Ih4s3cszIXq0cscjuR4mUiEgb9KfJ/+DBxXcQGpgEHPGycUTXnMx5f/+Yk4oiVMdjDZ7nknmAkZelP+8irUFDeyIibcxby97ioc9vwwI1mBfHvAShwhlk9XqOTTUJHp1cTqyqD87V/RPu/BCx4kMIBz0mDO6SoehFdi9KpERE2pi7Z91Nkro9TubFCRVOB4uR9KFmxbn4NX1wfgiXzMb5QSg5igJ/DP+8YDzBgP68i7QG9f2KiLQxayobngMFhgWqcIkwLplH1ZLv44XX0Tk/ytVHHUW33EIOHNqVkJIokVajREpEpI0Z030Mby9/G8dWO0+4IC6RX6fIj/Wg0HI5fdwerRegiGyhf7aIiLQxPxz3QyLBCMaXmx0HCNM1eioBC9Q5NhLyOHfiwNYOUUTSlEiJiLQxe3Teg4ePf5gjBhxBt0g3Rncbza1H3Mzj51zBwK655IYD5IYDZAU9jhrRk29p8U2RjNGmxSIi7Yhzjo8XF7O6rIYx/QoZ0j0v0yGJdHjatFhEpIMwMyYM6ZrpMEQkTUN7IiIiIjtJiZSIiIjITlIiJSIiIrKTlEiJiEiTlVbFKK5seJ8/kd2RJpuLiMg2xVasYMm1v6Zi6gc8O8F4dd8wNdlZHDP4GK4+8HIKswozHaJIxiiREhGRRiU3bWLxGV8jVlbGn84y5veDWCgGxHhpydPMKv6IZ055mqxAVqZDFckIDe2JiEijSp95lkR1NUt6ORb0NWKhL1dbx0uytnI9ry15LXMBimSYEikREWlUdM4cvGiURb0M3+rXx10NM9bNaPW4RNoKJVIiItKorL33wmVl0W0TBP0GDvBDDCzQXn+y+1IiJSIijep0yikEIhFGLYLcGvCSX24r5hw45zGp73EZjFAks5RIiYhIowKFhQx+7D/M77Unv3rIZ/hKw0sazg/gR3sSW3EZ+aFC4sk4j89/nG+9/C0ufPVCXl78Mr5rqAtLpGPRt/ZERGSbwgMGsODKG/jF5MXEKoEvqjFLQjKPEX0KKMwJctGrF/Hphk+pSdYA8OmGT3lv5XvccPANmQ1epIWpR0pERLbrh0cNY2C3PHLDAfAjZHkFhEIJ1ub9gQkPT2Dq2qlbkiiA6kQ1ry55lc9LPs9g1CItTz1SIiKyXQXZIV760SH8d+5api0t5oXlD1Cd/R5Rq4QkJKM9iJeOAxcimD+HQM4iAKasmcKwzsMyHL1Iy1EiJSIiTRIKeBw7qjeWN4vnSv9HMlEFQKx4ItF1x4MLAEa8dH+C+bOJ9H+Bztmdt5y/uGwx7yx/h/dWvse84nlkBbI4ddipXDzmYi3oKe2WEikREdkh84vnU5VOovxEHtF1J4ALfXmAyyJRPopE9WdM6j8J5xzXf3g9zy58lmgyWqetBz97kJnrZnLvMfe25i2INBvNkRIRkR0ysGAgOcEcAJIVw4EGvp3nQuybcyGRYIQ3l73J84uer5dEAUSTUWZumMlnGz5r4ahFWoYSKRER2SFHDzqaSDCCZx5YAsw1eNygTv0AeHzB41QnquvVO2dEN06kZMlp/PmNz1m0vgKATbFNLNu0jHgy3nI3IdJMNLQnIiI7JBKM8PAJD/PL93/Jx4mZ1Lj6/ybPCgY5bd++AMT8WMMN+WGS5aNJVg/ljRmO/858h97dN1LT9Q6CQfDM4/J9L+fre329JW9HZJeoR0pERHZY37y+3HfMfUw7bzL3nzeR7JBHVgiyQkZW0OPHXxnOyD6FAJw45ESyA9kNtpOs3ry9jOEcrFpfQPmaw6lOVFMZr+RPU/7E28vfbpV7EtkZ5lzDXbItqaioyE2dOrXVrysiIi2jvCbOf+euI5pIctjwHvQq/DJxivtxLn3tUmZvnE11ohrnABekeuU3SFaMqN+YV0P+nr/a8nDPznvyxElPtPxNiDTCzKY554oaqtPQnoiI7LL87BCnjOvbYF3IC3Hv0ffy7op3ufLtK6mJ5VG9/DxcrFfDjfnhOg8XlS5q7nBFmo2G9kREpMUFvACHDzicn47/KTnZUVwyv5EjfbycJXVKEi7R4vGJ7Cz1SImISKs5e6+z6ZLdhesTj7F8wfHgNi/EaUACvATZPZ+rc45zAVaVVjN302Tu+OQOVlasZHDBYH603484sM+BrX4PIrVpjpSIiLQ65xxXPzWbx6ctI+En8CLLCESWE+4yGS9UVus4SJTtSxdvNPHO/6mzn192IJtbD7+Vg/oelIlbkN3ItuZI7dLQnpn90czmmdksM3vazDrtSnsiIrJ7MDNuPH00D104gUDAJ2fgfWT3fLlOErVZzZpT2JTzTJ0kCqAmWcMt025prZBFGrSrc6ReB0Y558YAC4D/2/WQRERkdxEIeOSEckhW7oFzVqfO+UZswxHgPAhsavD8xWWLWyNMkUbtUiLlnHvNuS2zAD8E+u16SCIisrsY1DWXaMKnZtXX8GPdcMkwLhnC+SFiG48gtuEoIEB0zUn4ibx65/fKaeSbfyKtpDknm18A/KexSjO7BLgEYMCAAc14WRERaa+652dx8tg+PD8LqhZdQSBnMRYqIVExDJKFW45LlE0gUTGKnMG34gUrgdQcqe/u810gtWffeyvfoyJWwYTeE+iVqwRLWsd2EykzewNo6B15jXPu2fQx1wAJ4OHG2nHO3QPcA6nJ5jsVrYiIdDg3njaavp0j/GPyEiqiezCiSwGzN5WRrHWMcx4BPw8rm0Sw26vkh/P5wbgfcOLQE5m9YTaXvn4pSZfEOUfCJThvxHn8cN8fZuqWZDeyy9/aM7PzgUuBI51zVU05R9/aExGRxkxZUswFD0yhPFp//agxfQt59LJx5ARzMDMSfoIjHjuCkmhJneMMo09eH04fdjrn7H0OOaGc1gpfOqCW/NbescBVwElNTaJERES2pVteFnHfr1duQJ9OEXJDuZilJqZ/su6TBjdFdjhWVqzk7ll3882Xvkks2cjGyUDCT1AV10eY7JxdnSN1B5AFvJ5+U3/onLtsl6MSEZHd1uBuuYzoXcCsFWUk/C9HTbJDAS4+dHCdY6PJKIZt3USd+hUVK3ht6WucOOREnHPMWF7K7JVl9CgI8VHZAzy36BkSfoK+eX259oBrmdh7Yovdm3Q8u5RIOef2aK5ARERENrvvvP357sPT+GRZKcGA4Znx66+OZL+BXeoct2+PfUn4295CpjpRzeOf/Zej+h/HBQ9O4ZNlpem5VDGS3iByBmbjhcpYVr6MH/z3Bzx0/EPs2WXPlrw96UC0srmIiLRZa8pqKKmKMbR7HuFgw7NRXlj4Ar/+4NfEkjF86g8JOj+AK53E/p2+wYcLN1KTqH1MEossJbf/v4hvGoX5ORy8Rw/uO+m6FrojaY+2NUdKiZSIiLR7i8oW8eSCJ3ls/mPUJKJgX362OT9E5aIrIN6Fhj/xHJAA88EF8DzHSWMGcsuZ++B5jQ8byu6jxSabi4iItAVDCofw0/1/yhMnPYEleuL8EC4Zxk/kUr3iXFyjSRSkprGH0hsoB/H9EK/NWcsrn61ptfil/WrOBTlFREQyamDBQIZGf8WMNV9gXgI/2oPNfQaegd/EQZiqWJLHpizn+NG9ASiLlvH28reJ+TEm9jqIyfMTPD9zNblZAc6ZMJBDh3dvmRuSNk+JlIiIdCg/PHIY3324nOrol0t6Zoc8vjqmN49PW9nkdvz01Je3l7/NT9/5KZ55JB2ULlqPRfsTT6YStHcXbODCgwfxk2P2atb7kPZBQ3siItKhHL5XD64/ZRRdcsNkBT2ygx5nFfXnd6eNoUd+VpPaMOfY8NFUfvbbe/nJ2z+hJllDVaKKipKhJKp7bUmiAKrjSe7932LWlNW01C1JG6YeKRER6XBO368fp4zrS3FljPzsINmhAAA/P3YvrnpyVp31qdgyeyoOhAgmkxStncuPp/+Hj/ZOQi8H4dQRifK903Op6gp6xoeLNnLKuL4tel/S9iiREhGRDingGd236oE6bb9+mMH1L81lY0UUSBLI/5Ts7m+QLD2Evov7ccGnLzBuwxcYEMBIDd6kvr1nwUogCQTqtFuVqGJx+WxAidTuRomUiIjsVk7dtx+n7tuPWDLGi4te5LWlKynMmkin6DGMePrXDCtZseXYcYscfq1JMKFOU4iXTABXO5HycRbnn0uuoSTwPhN7T2RI4RCeX/Q8JTUlTOo/iSMGHEHQ00duR6R1pERERICPFm2k/KzT6Fu+rk75y/saDx0RIB4wnEG8dF+ia08GSwKGeVEi/R8kkL0agLAXJubH8PDw8ckJ5rBXl7247+j7CAVCGbgz2VXbWkdK6bGIiAgwfnAX7hhWRPcZrxGute3McdMdA1eFuPrEA3FeEhftQe7QP+DHemEWw4ssx2otALp5E+XNq6xXJaqYWzyXFxa9wKnDTm3dm5IWpx4pERGRtNL1xcw9+TSyyoqJJOMkMDwcN+/7dd7uNw7fC4BXQ96w6zFv23v8ba13Tm965vZkj857cN6I8xhUOKhlbkKanVY2FxERaYJO3bsw4b8vM+fUC1ha0AsD4hbg23NeYv+18/B8Hy8RJrR+BNvrh3B+kOj6I6lcchEumcXqyrXMWD+Dpz9/mjNfOJNP1n3SKvckLUs9UiIiIltxzvH2/PU8++Ysei6czb5De7D/wE7gHIuGDObiqecQ95P4sW54oRKwBGa1z4fqZReTrB5Adt+HCObNBz+CH+uGBUvxQuUM7TSUZ05+JlO3KDtAc6RERER2gJlx+F49OHyvo4Cj6tTl+3HCn4SJVnWhatlFgE/OoDvxgmXgxcEFSVb3IVndD1yIQM5iouuOJ15yAFgCXJBA3jwWuyeoTlQTCUYyco/SPJRIiYiI7ICQF+Kcvc/hgWmvUeUMXA5Viy4nkDefQNZa/Fg3EuV7sXmtqVjxAeklE0KpHyBZvhdVq0+itCJJpFOqB2z6uuksKlvE0MKhjOsxDqvdxSVtlob2REREdpDvfG6ffgd/eaYrfryAbU459qrBb6jXyeFhjOmfR3mXP7MpuSpVZh6DCwdz39H3kRfOa6E7kB2hyeYiIiLNyDOPH+33Q5697DgKI2FyswIEGutB8rMbacXwgRnLS1m+uic1yWpqkjVUxqv5bEWcS558gE9XlLXULUgzUY+UiIjILogmkrw5dx2TF27g8WkrqIn7O9yGhTaQt8ef8BO5VC+9FD9eCEAkmMO+Aztx/3n7b9kvUFqfeqRERERaSFYwwHGje/Prk0YxfnAXcsI7nvAE8uYBULPqDPxYl9TGyC6L6niS9xeu4eC//Y4nFjyBcw7nHDPWzeCx+Y/x0eqP8N2OJ27SfDTZXEREpBl4nvH38/bnpdlrePD9xcxYXorflEEfqya7x8s4P0iycjhbb4iMC7Fx3R7c+NFNzN4wm4VlC5lfPB/nUvOpeuX24oFjH6BLdpeWuC3ZDvVIiYiINJNgwOOksX146rsH8dNj9kgtd8C2silHqNPHqX37XOMfyc4FiSaSPP7pB8xcvYjqRGo+VVWiimWblvHryb9u9nuRplGPlIiISAv4zqQ9CRRO4Y8vrCdeOThVaIALEAkFMTNi2R8R7v7f1GKegRhe1mr8aB/q9nMk8ELFVCy4FvCBAMH82WT3fhIsSU35UF79xHiq8zKOH9VXc6lamSabi4iItKDJqybzt4+fYdnaCCN7DObaI79Kn4KuALy1cA4/m3wZ1X4JAMmaXlQtvRRcAFwYLJrqrfKDQPjLRi1GIH8WLtoXP94Z/DA54RC5WUGe+s6B9O+Sk4E77bi2NdlciZSIiEgGOef42bs/441lbxD34/iJXOKl++HHuhOILCe28SBcvGcDZybB/C2LfEKqw2vPXvncdPoYRvctxPO0qGdzUCIlIiLSxs1cN5NLXr+EqkRVnfKKBb/AJRtamNORSp3qC3tQmBPmXxdNYK9eBc0f7G5Gyx+IiIi0cWN7jOVvR/2NSDBC2PtyGM+LLCU1N6rpYj6sr4hy3K3/Y0NFTTNHKrUpkRIREWkj9uu5Hy+c+gKXjr2Ug/seTNgLk9XjFfBiQDJ91OaRJGPb3wg0HI7j//Jui8a8u1MiJSIi0ob0yOnBJWMu4a9H/pXxvceTl1tB7uDbCRbMwAsX06trJd8/YigTB3cGi6d+Gk2ojHXlMZYXVzVSL7tKyx+IiIi0QZ553H7E7by65FVeXPQi4T2KOW3YJA7pewhmxudryznpvulUV3WF5LbnQa0oqdY3+VqIEikREZE2KugFOWHICZww5IR6dcN65nP/10/nG/du/8tbw3p+OVk94SeYtX4WDseY7mMIeaFtnCnbo0RKRESknTpwaG865XqUVjZ2hOPYUV3plpcFwNQ1U7n87ctJ+AkAAhbg5kk3M7H3RADKquLc9Mo8Xvx0NZ7BKeP6cuXRe5KXpXShMVr+QEREpB175KNl/OaF2dTEa3+eO8wSnHNgN35zwkF4nlEWLePoJ46ut7xCJBjh9oMf57GPSnhh1mpiCX/LjKugZwzpnssrPzp0t16TalvLHyjFFBERacfOHt+fWCLJrW98TmUsQSQc4NLD+vPdw/bG7Mvk5/Wlr+MamJReE0/wzX/fQ7zkwHp1Cd+xYG0Fp985mUcunkgkrO1ntqZESkREpB0zM84/aDDfOmAQFbEEeeFgg71HZdEyYslYvXKfOBbY9rf6Zq0s49sPvcKB+yxkcOFgjh50NFmBrGa7h/ZMyx+IiIh0AJ5nFGSHGh2Cm9B7QsMTy12IZOUe22w76Ts+/DzBPbP+zvUfXs/xTx3P2sq1zRF2u6dESkREZDcwqtsoDh9wOJFgZEuZ88MkKoaTrB64/QZcAL+mB1WJKjZWb+SGj25owWjbD002FxER2U34zue1pa/x9OdPUx1LMOXToVSXjqZp/SoOvEryht2IealV1vftsS/fGvktjhxwJAAbqjcwc/1MumZ3ZWz3sXXmaLVn2rRYRERE6vnBI9N58dPV+E1OBZKEu79CuOt7mKVOigQjnD/yfBJ+gn989g9CgRDOObpGunLv0ffSN69vi8XfWpRIiYiISD3xpM+NL83l7+8vafpJFiOYP5tI38e2FAW9IEELUpP8coNkzzyGFg7lqZOfasaIM2NbiZTmSImIiOymQgGP6746kn9dOJ5ueWFywgGygttJDVyYRPlo/FjnLUW+8+skUZvLlpcvZ3HZ4pYIvc3Q8gciIiK7uUOGdefjq4/ii/UVREIBjr31XSpjycZPsCTJmj544RIA/EZGtwJegMp4o8uudwjqkRIRERE8zxjeM5/+XXI4cWwfttkx5QwvVJr61XmQyG9wXamqWBIv3qdlAm4j1CMlIiIidVx9/N7MWFbKko2VRBP+VrU+XngjFtqA84Mka/oS3PgNeo98lCWlK8GL43wPCFCz6lRO/etH7NU7n2g8ydpNUYKexwFDuzCgSy49C7I4cUwfOueGM3GbzUKTzUVERKQe33e8+/l6Xpuzlpc/XU1FNEEiuXmTmSRe1lpcMgfP78TJY/tyxIhO/OzVv5PMnoNLFBIvOQA/2nub18gKGgHP4/7z9ueAoV1b47Z2ir61JyIiIjvNOcf6iihLNlRywYNTiSd9ogmfSMgjPzvE8z84mBdnreZ3L80l0fS1FLbwDP70tbGcOq5vm1x7SomUiIiINIt15TX8Z8pyvlhXwbj+nTh9v37kZ4f4eHExZ939QQPbIjddl5wwd527H+MHd2m2eJvDthIpzZESERGRJuuRn80PjhhWr3z/QZ13KYkCKK6K8fV7PuC3p4zi7P0HNLpvYFuiREpERER2mZnRqyCLNZuiu9SO7+BXz33Gx4uLOf/AQTzzyUrivuPEMb05YEjXNjf0p6E9ERERaRaPfryM656ZTXwn5kltLegZAc+IJ32cg0g4wMn79OHG08Y0Q6Q7RkN7IiIi0uK+vn9/1pVH+fPrC3a5rYTv6kxcr4olefqTleRnBZm1soxehREuOGgQY/p12uVr7Qr1SImIiEizen3OGn746AyiieQObIi84wIG15ywNxccPKTlLoL22hMREZFW9JURvZh+7Ve44dRRdM0NkRMOkJuV2sfvoD26YpbY6oydy7aSDm54cS6lVbFdD3onaWhPREREml0kHODs8QM5q2gAHy0uZkNFlP0GdqZPpwg3v/kBf31zCX4iHwuWES6cSax0PJFALp55+L5PVXzrFdUblnTw5rx1nLZvvxa+o4ZpaE9ERERa3fJNy3nws3/weekCRnUdxdl7fpPVxdmYwfMzV/HPD5Y2ua2QB6/9eBKDu+W2SKyabC4iIiJtSv+C/lx7wC/qlhWm/vvJstIdaivuw08em8mT3z2wmaJrOs2REhERkTbl+NG9COzgYpwzV5RSEd167lXLUyIlIiIibUq/zjl8b9JQwsEdS1MysVSnhvZERESkzfnx0Xty0B7deGzqcqrjSYb3zOfZT1ayeGNVvWM9oGhQZ3KzWj+tUSIlIiIibdKEIV2ZMKTrlseXHzWcuas2cfa9H1Bek8B3EAl75GeHuPnMfTISoxIpERERaTf27lPA9GuP5v2FG5izahP9u+Rw1N49d3gYsLk0SyJlZlcCfwK6O+c2NEebIiIiIg3xPOOQYd05ZFj3TIey65PNzaw/cDSwbNfDEREREWk/mqMf7M/AVezs+u4iIiIi7dQuJVJmdjKw0jk3swnHXmJmU81s6vr163flsiIiIiJtwnbnSJnZG0CvBqquAa4mNay3Xc65e4B7ILVFzA7EKCIiItImbTeRcs4d1VC5mY0GBgMzzQygHzDdzMY759Y0a5QiIiIibdBOf2vPOfcp0GPzYzNbAhTpW3siIiKyu9AWMSIiIiI7qdkW5HTODWqutkRERETaA/VIiYiIiOwkJVIiIiIiO8mca/2VCMxsPbC01S+8c7oBmkDfvuk1bN/0+rV/eg3bN71+MNA51+B+NBlJpNoTM5vqnCvKdByy8/Qatm96/do/vYbtm16/bdPQnoiIiMhOUiIlIiIispOUSG3fPZkOQHaZXsP2Ta9f+6fXsH3T67cNmiMlIiIispPUIyUiIiKyk5RI7QAzu9LMnJl1y3Qs0nRm9kczm2dms8zsaTPrlOmYpGnM7Fgzm29mX5jZzzMdjzSdmfU3s7fMbI6ZfWZmP8p0TLLjzCxgZp+Y2QuZjqWtUiLVRGbWHzgaWJbpWGSHvQ6Mcs6NARYA/5fheKQJzCwA/BU4DhgBnG1mIzIbleyABHClc24EMBH4nl6/dulHwNxMB9GWKZFquj8DVwGaVNbOOOdec84l0g8/BPplMh5psvHAF865Rc65GPBv4OQMxyRN5Jxb7Zybnv69nNSHcd/MRiU7wsz6AScA92U6lrZMiVQTmNnJwErn3MxMxyK77ALg5UwHIU3SF1he6/EK9EHcLpnZIGAc8FGGQ5EdcyupDgQ/w3G0acFMB9BWmNkbQK8Gqq4BriY1rCdt1LZeP+fcs+ljriE13PBwa8YmsjszszzgSeBy59ymTMcjTWNmJwLrnHPTzGxShsNp05RIpTnnjmqo3MxGA4OBmWYGqWGh6WY23jm3phVDlG1o7PXbzMzOB04EjnRa86O9WAn0r/W4X7pM2gkzC5FKoh52zj2V6XhkhxwEnGRmxwPZQIGZPeSc+2aG42pztI7UDjKzJUCRc25338Cx3TCzY4FbgMOcc+szHY80jZkFSX054EhSCdQU4BvOuc8yGpg0iaX+5fkPoNg5d3mGw5FdkO6R+olz7sQMh9ImaY6U7A7uAPKB181shpndlemAZPvSXxD4PvAqqYnKjymJalcOAs4Fjkj/fzcj3bsh0qGoR0pERERkJ6lHSkRERGQnKZESERER2UlKpERERER2khIpERERkZ2kREpERERkJymREhEREdlJSqREREREdpISKREREZGd9P9Nd1C6DBFW6AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_vecs, y_vecs = [v[0][0] for v in logits_set], [v[0][1] for v in logits_set]\n",
    "plt.scatter(x_vecs, y_vecs, c=colors)\n",
    "plt.title('Logits from Twitter sentiment dataset')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lesbian-alabama",
   "metadata": {},
   "source": [
    "**We can see that such an algorithm would maybe not be accurate, since there's a tendency for neutral tweets to be classified as positive. A better solution would be to actually fine-tune the model on the dataset, because this one is specially trained for binary classification**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fluid-demographic",
   "metadata": {},
   "source": [
    "### Section 2: More on HuggingFace's transformers\n",
    "\n",
    "We have used a pre-trained model, but we can fine-tune it for our specific task or data-set. You can find the instructions for doing so in [this tutorial](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb)\n",
    "\n",
    "Furthermore, there's an even simpler way of using HuggingFace's pre-trained models: [pipelines](https://github.com/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "german-lawyer",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vulnerable-trunk",
   "metadata": {},
   "source": [
    "**Example using a sentiment classification task:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "quiet-reverse",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'NEGATIVE', 'score': 0.9995822906494141}]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp_sentence_classif = pipeline('sentiment-analysis')\n",
    "nlp_sentence_classif(\"@VirginAmerica and it's a really big bad thing about it\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hourly-superior",
   "metadata": {},
   "source": [
    "**Example of machine translation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "extra-trauma",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d39d9397f6334ccba3cb024256ce875a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.20k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0ee27889a204c39af0b5023f2737347",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/892M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d67dd0ebe014446951aaea132bb51a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/792k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d4b3ac1d0a64b76bb4438285a54794d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.39M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'translation_text': 'HuggingFace est une entreprise française basée à New York.'}]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# English to French\n",
    "translator = pipeline('translation_en_to_fr')\n",
    "translator(\"HuggingFace is a French company that is based in New York City. HuggingFace's mission is to solve NLP one commit at a time\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "intro_NLP",
   "language": "python",
   "name": "intro_nlp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}