Transformer Language Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "welcome-housing",
   "metadata": {},
   "source": [
    "# Transformer Language Models\n",
    "### For sentiment analysis\n",
    "Author: [Camilo](https://github.com/camilocarvajalreyes/)\n",
    "\n",
    "**Objectives**: You will get familiarised with the use of pre-trained models, how to interpret them and implement them in a simple NLP pipeline.\n",
    "\n",
    "**To do**: Trying a [pre-trained](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english) sentiment classifier based on Transformers in an industry case! \n",
    "\n",
    "### Instructions\n",
    "1. Download the vectors, install packages and test if everything is okay (if you haven't done so yet)\n",
    "2. Load the packages and import vectors by running cells in section 0. We will get familiarised with a pre-trained model that uses Transformers from HuggingFace\n",
    "3. Read the instructions in Section 1 and implement the model in a sentiment classifiction task.\n",
    "4. In section 2 we will see how to visualise the attention weights of the model for some examples\n",
    "5. Don't forget to ask if you have any question\n",
    "\n",
    "## Section 0 - Importing a pre-trained model\n",
    "In this tutorial we will use a library called **transformers** from the French company **HuggingFace**. [Transformers](https://github.com/huggingface/transformers) offers more than 40 architectures for NLP based on the [the original transformer module](https://arxiv.org/abs/1706.03762)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "moved-gasoline",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import DistilBertTokenizer, DistilBertForSequenceClassification\n",
    "import torch\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "viral-contributor",
   "metadata": {},
   "source": [
    "We'll use [distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english)\n",
    "\n",
    "This model is a based on the [distilBERT](https://arxiv.org/abs/1910.01108), a lighter version of [BERT](https://arxiv.org/abs/1810.04805) and it has been trained for the [Stanford Sentiment Analysis dataset](https://paperswithcode.com/dataset/sst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "mobile-worship",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME = \"distilbert-base-uncased-finetuned-sst-2-english\"\n",
    "\n",
    "tokenizer = DistilBertTokenizer.from_pretrained(MODEL_NAME)\n",
    "model = DistilBertForSequenceClassification.from_pretrained(MODEL_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "undefined-assessment",
   "metadata": {},
   "source": [
    "First of all we need to do some pre-processing on the input text. Fortunately, HuggingFace includes **Tokenizers** from pre-trained models.\n",
    "\n",
    "Applying this particular pre-trained model to a tokenised input returns a [SequenceClassifierOutput](https://huggingface.co/transformers/main_classes/output.html#transformers.modeling_outputs.SequenceClassifierOutput) object. More precisely, we can get the **logit** values, which reflect the probability of a Class to be chosen over the other ones.\n",
    "\n",
    "In this case, we have a **binary sentiment classification**, thus the the logits vector show the odds of the sample being **negative** or **positive** respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "funded-might",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "SequenceClassifierOutput(loss=None, logits=tensor([[-3.9886,  4.2962]], grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)\n",
      "Logits:\n",
      "tensor([[-3.9886,  4.2962]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "input_example = tokenizer(\"The cutest dog I've ever seen\", return_tensors=\"pt\")\n",
    "output = model(**input_example)\n",
    "#logits = outputs.logits\n",
    "#print(logits)\n",
    "print('Output:')\n",
    "print(output)\n",
    "print('Logits:')\n",
    "print(output.logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "friendly-underwear",
   "metadata": {},
   "source": [
    "Furthermore, if we happen to have labels for our samples, we can add them when calling the model and it will output the corresponding **loss**. In this case, we know that our phrase is **positive**, the label should be **1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "accepted-insured",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.0003, grad_fn=<NllLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "label = torch.tensor([1]).unsqueeze(0)  # Batch size 1, postive sentiment (label 1)\n",
    "output = model(**input_example, labels=label) \n",
    "print(output.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "joint-bedroom",
   "metadata": {},
   "source": [
    "Both outputs are PyTorch tensors, but we can transform it to a **numpy array**.\n",
    "\n",
    "Remark: the first dimension is 1 because it corresponds to the batch size, which is one since we are only inputing one example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "superior-inventory",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2)\n",
      "[[-3.9885974  4.2962294]]\n"
     ]
    }
   ],
   "source": [
    "tensor = output.logits\n",
    "array = tensor.detach().numpy()\n",
    "print(array.shape)\n",
    "print(array)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opponent-brunswick",
   "metadata": {},
   "source": [
    "## Section 1 - Clasifying tweets\n",
    "Let's consider a real case scenario: a major **US Airline** needs you to classify tweets from its costumers. They provide the following [dataset](https://www.kaggle.com/crowdflower/twitter-airline-sentiment):\n",
    "\n",
    "Since we won't fine tune the mode, we will only take 2000 out of the ~14000 tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cordless-afghanistan",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@VirginAmerica why are your first fares in May over three times more than other carriers when all seats are available to select???\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>0.3486</td>\n",
       "      <td>@VirginAmerica plus you've added commercials to the experience... tacky.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica it's really aggressive to blast obnoxious \"entertainment\" in your guests' faces &amp;amp; they have littl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica and it's a really big bad thing about it</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>negative</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica seriously would pay $30 a flight for seats that didn't have this playing.\\nit's really the only bad t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>positive</td>\n",
       "      <td>0.6745</td>\n",
       "      <td>@VirginAmerica yes, nearly every time I fly VX this “ear worm” won’t go away :)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  airline_sentiment  airline_sentiment_confidence  \\\n",
       "1          positive                        0.3486   \n",
       "3          negative                        1.0000   \n",
       "4          negative                        1.0000   \n",
       "5          negative                        1.0000   \n",
       "6          positive                        0.6745   \n",
       "\n",
       "                                                                                                                      text  \n",
       "1                                                 @VirginAmerica plus you've added commercials to the experience... tacky.  \n",
       "3  @VirginAmerica it's really aggressive to blast obnoxious \"entertainment\" in your guests' faces &amp; they have littl...  \n",
       "4                                                                  @VirginAmerica and it's a really big bad thing about it  \n",
       "5  @VirginAmerica seriously would pay $30 a flight for seats that didn't have this playing.\\nit's really the only bad t...  \n",
       "6                                          @VirginAmerica yes, nearly every time I fly VX this “ear worm” won’t go away :)  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_dataset = pd.read_csv(\"data/Tweets-subset.csv\")\n",
    "pd.set_option('max_colwidth', 120)\n",
    "full_dataset = full_dataset.drop(['tweet_id','negativereason','negativereason_confidence','retweet_count',\n",
    "             'airline','airline_sentiment_gold','name','negativereason_gold',\n",
    "             'tweet_coord','tweet_created','tweet_location','user_timezone'], axis=1)\n",
    "dataset = full_dataset[(full_dataset['airline_sentiment']!='neutral')]\n",
    "\n",
    "# example sentence \n",
    "index = 16\n",
    "print(dataset.iloc[index].text)\n",
    "\n",
    "dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "approved-diameter",
   "metadata": {},
   "source": [
    "**1. Using our pre-trained Model, code a function that takes a string of text and outputs 1 if it's positive and 0 if negative**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bridal-service",
   "metadata": {},
   "outputs": [],
   "source": [
    "### to do ###"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "architectural-foster",
   "metadata": {},
   "source": [
    "**2. Iterate over the dataset and compute the average loss and the accuracy of the classifier**\n",
    "\n",
    "(This cell should take about 1 minute to run)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "clean-defendant",
   "metadata": {},
   "outputs": [],
   "source": [
    "for index, row in dataset.iterrows():\n",
    "    text = row.text\n",
    "    ### to do ###\n",
    "    \n",
    "print('average loss = {}'.format(average_loss))\n",
    "print('accuracy = {}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "natural-civilian",
   "metadata": {},
   "source": [
    "### Visualising Attention\n",
    "\n",
    "The airline wishes to understand how the model works. Therefore, you decide to show a visualisation of the attention weights from the first and last layer of transformers. Thankfully, Transformers provides the attention weights when output_attentions=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "thermal-asian",
   "metadata": {},
   "outputs": [],
   "source": [
    "def attention(list_text,layer=0):\n",
    "    ''' parameters:\n",
    "    text: input string for the classifier\n",
    "    layer: layer to show the attention weights from\n",
    "    '''\n",
    "    input_sentence = tokenizer(list_text, return_tensors=\"pt\",is_split_into_words=True)\n",
    "    output = model(**input_sentence,output_attentions=True)\n",
    "    print('text: '+list_text)\n",
    "    array = output.attentions[layer].detach().numpy()\n",
    "    array = array.mean(axis=1) # taking average over all attention heads\n",
    "    return array[0,:,:]\n",
    "\n",
    "#example\n",
    "#attention(\"@VirginAmerica and it's a really big bad thing about it\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hawaiian-receiver",
   "metadata": {},
   "source": [
    "**3. Using the attention function, code a heatmap to visualise the scoresfrom the model's self attention**\n",
    "\n",
    "You can take inspiration from [this example](https://matplotlib.org/stable/gallery/images_contours_and_fields/image_annotated_heatmap.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "genuine-flavor",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[\"figure.figsize\"] = (10,6) # setting figure size\n",
    "\n",
    "def heatmap_attention(text,layer=0):\n",
    "    tokenized_sequence = tokenizer.tokenize(text)\n",
    "    labels = ['[CLS]']+tokenized_sequence+['[SEP]']\n",
    "\n",
    "    matrix = ### to do ###\n",
    "\n",
    "\n",
    "    fig, ax = plt.subplots()\n",
    "    im = ax.imshow(matrix)\n",
    "\n",
    "    # We want to show all ticks...\n",
    "    \n",
    "    ### to do ###\n",
    "    \n",
    "    # ... and label them with the respective list entries\n",
    "    \n",
    "    ### to do ###\n",
    "    \n",
    "    # Rotate the tick labels and set their alignment.\n",
    "    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\",\n",
    "             rotation_mode=\"anchor\")\n",
    "\n",
    "    ax.set_title(\"Attention visualisation for our Transformer based classifier\")\n",
    "    fig.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "union-eugene",
   "metadata": {},
   "outputs": [],
   "source": [
    "heatmap_attention(\"@VirginAmerica and it's a really big bad thing about it\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "refined-horizontal",
   "metadata": {},
   "source": [
    "### Bonus task\n",
    "In order to make the task simpler, we have so far ignored neutral tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "comprehensive-chamber",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>airline_sentiment</th>\n",
       "      <th>airline_sentiment_confidence</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica What @dhepburn said.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6837</td>\n",
       "      <td>@VirginAmerica I didn't today... Must mean I need to take another trip!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6340</td>\n",
       "      <td>@VirginAmerica Really missed a prime opportunity for Men Without Hats parody, there. https://t.co/mWpG7grEZP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>neutral</td>\n",
       "      <td>0.6769</td>\n",
       "      <td>@VirginAmerica did you know that suicide is the second leading cause of death among teens 10-24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>neutral</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>@VirginAmerica will you be making BOS&amp;gt;LAS non stop permanently anytime soon?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   airline_sentiment  airline_sentiment_confidence  \\\n",
       "0            neutral                        1.0000   \n",
       "2            neutral                        0.6837   \n",
       "7            neutral                        0.6340   \n",
       "10           neutral                        0.6769   \n",
       "23           neutral                        1.0000   \n",
       "\n",
       "                                                                                                            text  \n",
       "0                                                                            @VirginAmerica What @dhepburn said.  \n",
       "2                                        @VirginAmerica I didn't today... Must mean I need to take another trip!  \n",
       "7   @VirginAmerica Really missed a prime opportunity for Men Without Hats parody, there. https://t.co/mWpG7grEZP  \n",
       "10               @VirginAmerica did you know that suicide is the second leading cause of death among teens 10-24  \n",
       "23                               @VirginAmerica will you be making BOS&gt;LAS non stop permanently anytime soon?  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_dataset[(full_dataset['airline_sentiment']=='neutral')].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "through-possible",
   "metadata": {},
   "source": [
    "**Design a way of predicting that the tweet is neutral**\n",
    "\n",
    "You may use the attribute **logits** from the model's output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "mechanical-publicity",
   "metadata": {},
   "outputs": [],
   "source": [
    "### to do ###"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fluid-demographic",
   "metadata": {},
   "source": [
    "### Section 2: More on HuggingFace's transformers\n",
    "\n",
    "We have used a pre-trained model, but we can fine-tune it for our specific task or data-set. You can find the instructions for doing so in [this tutorial](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb)\n",
    "\n",
    "Furthermore, there's an even simpler way of using HuggingFace's pre-trained models: [pipelines](https://github.com/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "german-lawyer",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vulnerable-trunk",
   "metadata": {},
   "source": [
    "**Example using a sentiment classification task:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "quiet-reverse",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'NEGATIVE', 'score': 0.9995822906494141}]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp_sentence_classif = pipeline('sentiment-analysis')\n",
    "nlp_sentence_classif(\"@VirginAmerica and it's a really big bad thing about it\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hourly-superior",
   "metadata": {},
   "source": [
    "**Example of machine translation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "extra-trauma",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d39d9397f6334ccba3cb024256ce875a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.20k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0ee27889a204c39af0b5023f2737347",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/892M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d67dd0ebe014446951aaea132bb51a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/792k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d4b3ac1d0a64b76bb4438285a54794d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/1.39M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'translation_text': 'HuggingFace est une entreprise française basée à New York.'}]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# English to French\n",
    "translator = pipeline('translation_en_to_fr')\n",
    "translator(\"HuggingFace is a French company that is based in New York City. HuggingFace's mission is to solve NLP one commit at a time\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "intro_NLP",
   "language": "python",
   "name": "intro_nlp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}