fine_tuning_bert_sentence_classification.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "baXW9uwiS2Mt"
      },
      "source": [
        "Adapted from: https://mccormickml.com/2019/07/22/BERT-fine-tuning/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DerwenAI/spaCy_tuTorial/blob/master/BERT_Fine_Tuning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jNKaJz5j_ylj"
      },
      "source": [
        "# Notes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BJR6t_gCQe_x"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "### History\n",
        "\n",
        "2018 was a breakthrough year in NLP. Transfer learning, particularly models like Allen AI's ELMO, OpenAI's Open-GPT, and Google's BERT allowed researchers to smash multiple benchmarks with minimal task-specific fine-tuning and provided the rest of the NLP community with pretrained models that could easily (with less data and less compute time) be fine-tuned and implemented to produce state of the art results. Unfortunately, for many starting out in NLP and even for some experienced practicioners, the theory and practical application of these powerful models is still not well understood.\n",
        "\n",
        "### What is BERT?\n",
        "\n",
        "BERT (Bidirectional Encoder Representations from Transformers), released in late 2018, is the model we will use in this tutorial to provide readers with a better understanding of and practical guidance for using transfer learning models in NLP. BERT is a method of pretraining language representations that was used to create models that NLP practicioners can then download and use for free. You can either use these models to extract high quality language features from your text data, or you can fine-tune these models on a specific task (classification, entity recognition, question answering, etc.) with your own data to produce state of the art predictions.\n",
        "\n",
        "### What Can BERT Do For Me?\n",
        "\n",
        "This post will explain how you can modify and fine-tune BERT to create a powerful NLP model that quickly gives you state of the art results.\n",
        "\n",
        "### Why Fine-Tuning Rather Than Building My Own Model ?\n",
        "\n",
        "In this tutorial, we will use BERT to train a text classifier. Specifically, we will take the pre-trained BERT model, add an untrained layer of neurons on the end, and train the new model for our classification task. Why do this rather than train a train a specific deep learning model (a CNN, BiLSTM, etc.) that is well suited for the specific NLP task you need?\n",
        "\n",
        "#### Easy Training\n",
        "\n",
        "First, the pre-trained BERT model weights already encode a lot of information about our language. As a result, it takes much less time to train our fine-tuned model - it is as if we have already trained the bottom layers of our network extensively and only need to gently tune them while using their output as features for our classification task. In fact, the authors recommend only 2-4 epochs of training for fine-tuning BERT on a specific NLP task (compared to the hundreds of GPU hours needed to train the original BERT model or a LSTM from scratch!).\n",
        "\n",
        "#### Less Data\n",
        "\n",
        "In addition and perhaps just as important, because of the pre-trained weights this method allows us to fine-tune our task on a much smaller dataset than would be required in a model that is built from scratch. A major drawback of NLP models built from scratch is that we often need a prohibitively large dataset in order to train our network to reasonable accuracy, meaning a lot of time and energy had to be put into dataset creation. By fine-tuning BERT, we are now able to get away with training a model to good performance on a much smaller amount of training data.\n",
        "\n",
        "#### Good Results\n",
        "\n",
        "Second, this simple fine-tuning procedure (typically adding one fully-connected layer on top of BERT and training for a few epochs) was shown to achieve state of the art results with minimal task-specific adjustments for a wide variety of tasks: classification, language inference, semantic similarity, question answering, etc. Rather than implementing custom and sometimes-obscure architetures shown to work well on a specific task, simply fine-tuning BERT is shown to be a better (or at least equal) alternative.\n",
        "\n",
        "#### A Shift in NLP\n",
        "\n",
        "This shift to transfer learning parallels the same shift that took place in computer vision a few years ago. Creating a good deep learning network for computer vision tasks can take millions of parameters and be very expensive to train. Researchers discovered that deep networks learn hierarchical feature representations (simple features like edges at the lowest layers with gradually more complex features at higher layers). Rather than training a new network from scratch each time, the lower layers of a trained network with generalized image features could be copied and transfered for use in another network with a different task. It soon became common practice to download a pre-trained deep network and quickly retrain it for the new task or add additional layers on top - vastly preferable to the expensive process of training a network from scratch. For many, the introduction of deep pre-trained language models in 2018 (ELMO, BERT, ULMFIT, Open-GPT, etc.) signals the same shift to transfer learning in NLP that computer vision saw.\n",
        "\n",
        "Let's get started!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RX_ZDhicpHkV"
      },
      "source": [
        "## Install and Import"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nSU7yERLP_66"
      },
      "source": [
        "Google Colab offers free GPUs and TPUs! Since we'll be training a large neural network it's best to take advantage of this (in this case we'll attach a GPU), otherwise training will take a very long time.\n",
        "\n",
        "A GPU can be added by going to the menu and selecting:\n",
        "\n",
        "Edit -> Notebook Settings -> Add accelerator (GPU)\n",
        "\n",
        "Then run the following cell to confirm that the GPU is detected."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DEfSbAA4QHas"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "device_name = tf.test.gpu_device_name()\n",
        "if device_name != '/device:GPU:0':\n",
        "  raise SystemError('GPU device not found')\n",
        "print('Found GPU at: {}'.format(device_name))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ElsnSNUridI"
      },
      "source": [
        "Next, let's install the pytorch interface for BERT by Hugging Face. (This library contains interfaces for other pretrained language models like OpenAI's GPT and GPT-2.) We've selected the pytorch interface because it strikes a nice balance between the high-level APIs (which are easy to use but don't provide insight into how things work) and tensorflow code (which contains lots of details but often sidetracks us into lessons about tensorflow, when the purpose here is BERT!).\n",
        "\n",
        "At the moment, the Hugging Face library seems to be the most widely accepted and powerful pytorch interface for working with BERT. In addition to supporting a variety of different pre-trained language models (and future models to come - BERT will not be state of the art forever), the library also includes pre-built modifications of BERT suited to your specific task. For example, in this tutorial we will use BertForSequenceClassification, but the library also includes BERT modifications designed for token classification, question answering, next sentence prediciton, etc. Using these pre-built classes simplifies the process of modifying BERT for your purposes.\n",
        "\n",
        "**Note**: as of July 2019, the pytorch library called \"pytorch-pretrained-bert pytorch-nlp\" has been updated and renamed as \"pytorch-transformers.\" Though the old name still works, there is a short summary [here](https://github.com/huggingface/pytorch-transformers#Migrating-from-pytorch-pretrained-bert-to-pytorch-transformers) about migrating to the new library. (This mostly involves a few name changes.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0NmMdkZO8R6q"
      },
      "outputs": [],
      "source": [
        "!pip install pytorch-pretrained-bert pytorch-nlp"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ok002ceNB8E7"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from sklearn.model_selection import train_test_split\n",
        "from pytorch_pretrained_bert import BertTokenizer, BertConfig\n",
        "from pytorch_pretrained_bert import BertAdam, BertForSequenceClassification\n",
        "from tqdm import tqdm, trange\n",
        "import pandas as pd\n",
        "import io\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqG7FzRVFEIv"
      },
      "source": [
        "In order for torch to use the GPU, we need to identify and specify the GPU as the device. Later, in our training loop, we will load data onto the device."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oYsV4H8fCpZ-"
      },
      "outputs": [],
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "n_gpu = torch.cuda.device_count()\n",
        "torch.cuda.get_device_name(0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "guw6ZNtaswKc"
      },
      "source": [
        "## Load Dataset\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_9ZKxKc04Btk"
      },
      "source": [
        "We'll use The Corpus of Linguistic Acceptability (CoLA) dataset for single sentence classification. It's a set of sentences labeled as grammatically correct or incorrect. The data is as follows:\n",
        "\n",
        "**Column 1**:\tthe code representing the source of the sentence.\n",
        "\n",
        "**Column 2**:\tthe acceptability judgment label (0=unacceptable, 1=acceptable).\n",
        "\n",
        "**Column 3**:\tthe acceptability judgment as originally notated by the author.\n",
        "\n",
        "**Column 4**:\tthe sentence."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NTU8Awqv4Eln"
      },
      "source": [
        "Download the dataset from this link, extract, and move them to your local drive.\n",
        "\n",
        "https://nyu-mll.github.io/CoLA/\n",
        "\n",
        "Both tokenized and raw versions of the data are available. We will use the **raw** version because we need to use the BERT tokenizer to break the text down into tokens and chunks that the model will recognize."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DOpikSXbsvn-"
      },
      "outputs": [],
      "source": [
        "# Upload the train file from your local drive\n",
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_UkeC7SG2krJ"
      },
      "outputs": [],
      "source": [
        "df = pd.read_csv(\"in_domain_train.tsv\", delimiter='\\t', header=None, names=['sentence_source', 'label', 'label_notes', 'sentence'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3UfxtwQy3axu"
      },
      "outputs": [],
      "source": [
        "df.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AQfTaYDo42zu"
      },
      "outputs": [],
      "source": [
        "df.sample(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GuE5BqICAne2"
      },
      "outputs": [],
      "source": [
        "# Create sentence and label lists\n",
        "sentences = df.sentence.values\n",
        "\n",
        "# We need to add special tokens at the beginning and end of each sentence for BERT to work properly\n",
        "sentences = [\"[CLS] \" + sentence + \" [SEP]\" for sentence in sentences]\n",
        "labels = df.label.values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ex5O1eV-Pfct"
      },
      "source": [
        "## Inputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTREubVNFiz4"
      },
      "source": [
        "Next, import the BERT tokenizer, used to convert our text into tokens that correspond to BERT's vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z474sSC6oe7A"
      },
      "outputs": [],
      "source": [
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\n",
        "\n",
        "tokenized_texts = [tokenizer.tokenize(sent) for sent in sentences]\n",
        "print (\"Tokenize the first sentence:\")\n",
        "print (tokenized_texts[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87_kXUeT2-br"
      },
      "source": [
        "BERT requires specifically formatted inputs. For each tokenized input sentence, we need to create:\n",
        "\n",
        "- **input ids**: a sequence of integers identifying each input token to its index number in the BERT tokenizer vocabulary\n",
        "- **segment mask**: (optional) a sequence of 1s and 0s used to identify whether the input is one sentence or two sentences long. For one sentence inputs, this is simply a sequence of 0s. For two sentence inputs, there is a 0 for each token of the first sentence, followed by a 1 for each token of the second sentence\n",
        "- **attention mask**: (optional) a sequence of 1s and 0s, with 1s for all input tokens and 0s for all padding tokens (we'll detail this in the next paragraph)\n",
        "- **labels**: a single value of 1 or 0. In our task 1 means \"grammatical\" and 0 means \"ungrammatical\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xytsw1oIfnX0"
      },
      "source": [
        "Although we can have variable length input sentences, BERT does requires our input arrays to be the same size. We address this by first choosing a maximum sentence length, and then padding and truncating our inputs until every input sequence is of the same length.\n",
        "\n",
        "To \"pad\" our inputs in this context means that if a sentence is shorter than the maximum sentence length, we simply add 0s to the end of the sequence until it is the maximum sentence length.\n",
        "\n",
        "If a sentence is longer than the maximum sentence length, then we simply truncate the end of the sequence, discarding anything that does not fit into our maximum sentence length.\n",
        "\n",
        "We pad and truncate our sequences so that they all become of length MAX_LEN (\"post\" indicates that we want to pad and truncate at the end of the sequence, as opposed to the beginning) `pad_sequences` is a utility function that we're borrowing from Keras. It simply handles the truncating and padding of Python lists."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cp9BPRd1tMIo"
      },
      "outputs": [],
      "source": [
        "# Set the maximum sequence length. The longest sequence in our training set is 47, but we'll leave room on the end anyway.\n",
        "# In the original paper, the authors used a length of 512.\n",
        "MAX_LEN = 128"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mFhowDMohU4H"
      },
      "outputs": [],
      "source": [
        "# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n",
        "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kDs-MYtYH8sL"
      },
      "outputs": [],
      "source": [
        "# Pad our input tokens\n",
        "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KhGulL1pExCT"
      },
      "source": [
        "Create the attention masks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cDoC24LeEv3N"
      },
      "outputs": [],
      "source": [
        "# Create attention masks\n",
        "attention_masks = []\n",
        "\n",
        "# Create a mask of 1s for each token followed by 0s for padding\n",
        "for seq in input_ids:\n",
        "  seq_mask = [float(i>0) for i in seq]\n",
        "  attention_masks.append(seq_mask)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aFbE-UHvsb7-"
      },
      "outputs": [],
      "source": [
        "# Use train_test_split to split our data into train and validation sets for training\n",
        "\n",
        "train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, labels,\n",
        "                                                            random_state=2018, test_size=0.1)\n",
        "train_masks, validation_masks, _, _ = train_test_split(attention_masks, input_ids,\n",
        "                                             random_state=2018, test_size=0.1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jw5K2A5Ko1RF"
      },
      "outputs": [],
      "source": [
        "# Convert all of our data into torch tensors, the required datatype for our model\n",
        "\n",
        "train_inputs = torch.tensor(train_inputs)\n",
        "validation_inputs = torch.tensor(validation_inputs)\n",
        "train_labels = torch.tensor(train_labels)\n",
        "validation_labels = torch.tensor(validation_labels)\n",
        "train_masks = torch.tensor(train_masks)\n",
        "validation_masks = torch.tensor(validation_masks)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GEgLpFVlo1Z-"
      },
      "outputs": [],
      "source": [
        "# Select a batch size for training. For fine-tuning BERT on a specific task, the authors recommend a batch size of 16 or 32\n",
        "batch_size = 32\n",
        "\n",
        "# Create an iterator of our data with torch DataLoader. This helps save on memory during training because, unlike a for loop,\n",
        "# with an iterator the entire dataset does not need to be loaded into memory\n",
        "\n",
        "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n",
        "train_sampler = RandomSampler(train_data)\n",
        "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n",
        "\n",
        "validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n",
        "validation_sampler = SequentialSampler(validation_data)\n",
        "validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pNl8khAhPYju"
      },
      "source": [
        "## Train Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwQ7JcuJQZ0o"
      },
      "source": [
        "Now that our input data is properly formatted, it's time to fine tune the BERT model.\n",
        "\n",
        "For this task, we first want to modify the pre-trained BERT model to give outputs for classification, and then we want to continue training the model on our dataset until that the entire model, end-to-end, is well-suited for our task. Thankfully, the huggingface pytorch implementation includes a set of interfaces designed for a variety of NLP tasks. Though these interfaces are all built on top of a trained BERT model, each has different top layers and output types designed to accomodate their specific NLP task.  \n",
        "\n",
        "We'll load [BertForSequenceClassification](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L1129). This is the normal BERT model with an added single linear layer on top for classification that we will use as a sentence classifier. As we feed input data, the entire pre-trained BERT model and the additional untrained classification layer is trained on our specific task.\n",
        "\n",
        "### Structure of Fine-Tuning Model\n",
        "\n",
        "As we've showed beforehand, the first token of every sequence is the special classification token ([CLS]). Unlike the hidden state vector corresponding to a normal word token, the hidden state corresponding to this special token is designated by the authors of BERT as an aggregate representation of the whole sentence used for classification tasks. As such, when we feed in an input sentence to our model during training, the output is the length 768 hidden state vector corresponding to this token. The additional layer that we've added on top consists of untrained linear neurons of size [hidden_state, number_of_labels], so [768,2], meaning that the output of BERT plus our classification layer is a vector of two numbers representing the \"score\" for \"grammatical/non-grammatical\" that are then fed into cross-entropy loss.\n",
        "\n",
        "\n",
        "\n",
        "### The Fine-Tuning Process\n",
        "\n",
        "Because the pre-trained BERT layers already encode a lot of information about the language, training the classifier is relatively inexpensive. Rather than training every layer in a large model from scratch, it's as if we have already trained the bottom layers 95% of where they need to be, and only really need to train the top layer, with a bit of tweaking going on in the lower levels to accomodate our task.\n",
        "\n",
        "Sometimes practicioners will opt to \"freeze\" certain layers when fine-tuning, or to apply different learning rates, apply diminishing learning rates, etc. all in an effort to preserve the good quality weights in the network and speed up training (often considerably). In fact, recent research on BERT specifically has demonstrated that freezing the majority of the weights results in only minimal accuracy declines, but there are exceptions and broader rules of transfer learning that should also be considered. For example, if your task and fine-tuning dataset is very different from the dataset used to train the transfer learning model, freezing the weights may not be a good idea. We'll cover the broader scope of transfer learning in NLP in a future post.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WnQW9E-bBCRt"
      },
      "source": [
        "OK, let's load BERT! There are a few different pre-trained BERT models available. \"bert-base-uncased\" means the version that has only lowercase letters (\"uncased\") and is the smaller version of the two (\"base\" vs \"large\")."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gFsCTp_mporB"
      },
      "outputs": [],
      "source": [
        "# Load BertForSequenceClassification, the pretrained BERT model with a single linear classification layer on top.\n",
        "\n",
        "model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=2)\n",
        "model.cuda()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8o-VEBobKwHk"
      },
      "source": [
        "Now that we have our model loaded we need to grab the training hyperparameters from within the stored model.\n",
        "\n",
        "For the purposes of fine-tuning, the authors recommend the following hyperparameter ranges:\n",
        "- Batch size: 16, 32\n",
        "- Learning rate (Adam): 5e-5, 3e-5, 2e-5\n",
        "- Number of epochs: 2, 3, 4"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QxSMw0FrptiL"
      },
      "outputs": [],
      "source": [
        "\n",
        "param_optimizer = list(model.named_parameters())\n",
        "no_decay = ['bias', 'gamma', 'beta']\n",
        "optimizer_grouped_parameters = [\n",
        "    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
        "     'weight_decay_rate': 0.01},\n",
        "    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
        "     'weight_decay_rate': 0.0}\n",
        "]\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GLs72DuMODJO"
      },
      "outputs": [],
      "source": [
        "# This variable contains all of the hyperparemeter information our training loop needs\n",
        "optimizer = BertAdam(optimizer_grouped_parameters,\n",
        "                     lr=2e-5,\n",
        "                     warmup=.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QXZhFb4LnV5"
      },
      "source": [
        "Below is our training loop. There's a lot going on, but fundamentally for each pass in our loop we have a trianing phase and a validation phase. At each pass we need to:\n",
        "\n",
        "Training loop:\n",
        "- Tell the model to compute gradients by setting the model in train mode\n",
        "- Unpack our data inputs and labels\n",
        "- Load data onto the GPU for acceleration\n",
        "- Clear out the gradients calculated in the previous pass. In pytorch the gradients accumulate by default (useful for things like RNNs) unless you explicitly clear them out\n",
        "- Forward pass (feed input data through the network)\n",
        "- Backward pass (backpropagation)\n",
        "- Tell the network to update parameters with optimizer.step()\n",
        "- Track variables for monitoring progress\n",
        "\n",
        "Evalution loop:\n",
        "- Tell the model not to compute gradients by setting th emodel in evaluation mode\n",
        "- Unpack our data inputs and labels\n",
        "- Load data onto the GPU for acceleration\n",
        "- Forward pass (feed input data through the network)\n",
        "- Compute loss on our validation data and track variables for monitoring progress\n",
        "\n",
        "So please read carefully through the comments to get an understanding of what's happening. If you're unfamiliar with pytorch a quick look at some of their [beginner tutorials](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) will help show you that training loops really involve only a few simple steps; the rest is usually just decoration and logging.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9cQNvaZ9bnyy"
      },
      "outputs": [],
      "source": [
        "# Function to calculate the accuracy of our predictions vs labels\n",
        "def flat_accuracy(preds, labels):\n",
        "    pred_flat = np.argmax(preds, axis=1).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6J-FYdx6nFE_"
      },
      "outputs": [],
      "source": [
        "# Store our loss and accuracy for plotting\n",
        "train_loss_set = []\n",
        "\n",
        "# Number of training epochs (authors recommend between 2 and 4)\n",
        "epochs = 4\n",
        "\n",
        "# trange is a tqdm wrapper around the normal python range\n",
        "for _ in trange(epochs, desc=\"Epoch\"):\n",
        "\n",
        "\n",
        "  # Training\n",
        "\n",
        "  # Set our model to training mode (as opposed to evaluation mode)\n",
        "  model.train()\n",
        "\n",
        "  # Tracking variables\n",
        "  tr_loss = 0\n",
        "  nb_tr_examples, nb_tr_steps = 0, 0\n",
        "\n",
        "  # Train the data for one epoch\n",
        "  for step, batch in enumerate(train_dataloader):\n",
        "    # Add batch to GPU\n",
        "    batch = tuple(t.to(device) for t in batch)\n",
        "    # Unpack the inputs from our dataloader\n",
        "    b_input_ids, b_input_mask, b_labels = batch\n",
        "    # Clear out the gradients (by default they accumulate)\n",
        "    optimizer.zero_grad()\n",
        "    # Forward pass\n",
        "    loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n",
        "    train_loss_set.append(loss.item())\n",
        "    # Backward pass\n",
        "    loss.backward()\n",
        "    # Update parameters and take a step using the computed gradient\n",
        "    optimizer.step()\n",
        "\n",
        "\n",
        "    # Update tracking variables\n",
        "    tr_loss += loss.item()\n",
        "    nb_tr_examples += b_input_ids.size(0)\n",
        "    nb_tr_steps += 1\n",
        "\n",
        "  print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n",
        "\n",
        "\n",
        "  # Validation\n",
        "\n",
        "  # Put model in evaluation mode to evaluate loss on the validation set\n",
        "  model.eval()\n",
        "\n",
        "  # Tracking variables\n",
        "  eval_loss, eval_accuracy = 0, 0\n",
        "  nb_eval_steps, nb_eval_examples = 0, 0\n",
        "\n",
        "  # Evaluate data for one epoch\n",
        "  for batch in validation_dataloader:\n",
        "    # Add batch to GPU\n",
        "    batch = tuple(t.to(device) for t in batch)\n",
        "    # Unpack the inputs from our dataloader\n",
        "    b_input_ids, b_input_mask, b_labels = batch\n",
        "    # Telling the model not to compute or store gradients, saving memory and speeding up validation\n",
        "    with torch.no_grad():\n",
        "      # Forward pass, calculate logit predictions\n",
        "      logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)\n",
        "\n",
        "    # Move logits and labels to CPU\n",
        "    logits = logits.detach().cpu().numpy()\n",
        "    label_ids = b_labels.to('cpu').numpy()\n",
        "\n",
        "    tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n",
        "\n",
        "    eval_accuracy += tmp_eval_accuracy\n",
        "    nb_eval_steps += 1\n",
        "\n",
        "  print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iyRa-5CcHv_g"
      },
      "source": [
        "## Training Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-G03mmwH3aI"
      },
      "source": [
        "Let's take a look at our training loss over all batches:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "68xreA9JAmG5"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(15,8))\n",
        "plt.title(\"Training loss\")\n",
        "plt.xlabel(\"Batch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.plot(train_loss_set)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkyubuJSOzg3"
      },
      "source": [
        "##Predict and Evaluate on Holdout Set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DosV94BYIYxg"
      },
      "source": [
        "Now we'll load the holdout dataset and prepare inputs just as we did with the training set. Then we'll evaluate predictions using [Matthew's correlation coefficient](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html) because this is the metric used by the wider NLP community to evaluate performance on CoLA. With this metric, +1 is the best score, and -1 is the worst score. This way, we can see how well we perform against the state of the art models for this specific task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyBuXVGoOLQf"
      },
      "outputs": [],
      "source": [
        "# Upload the test file from your local drive\n",
        "from google.colab import files\n",
        "uploaded = files.upload()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mAN0LZBOOPVh"
      },
      "outputs": [],
      "source": [
        "df = pd.read_csv(\"out_of_domain_dev.tsv\", delimiter='\\t', header=None, names=['sentence_source', 'label', 'label_notes', 'sentence'])\n",
        "\n",
        "# Create sentence and label lists\n",
        "sentences = df.sentence.values\n",
        "\n",
        "# We need to add special tokens at the beginning and end of each sentence for BERT to work properly\n",
        "sentences = [\"[CLS] \" + sentence + \" [SEP]\" for sentence in sentences]\n",
        "labels = df.label.values\n",
        "\n",
        "tokenized_texts = [tokenizer.tokenize(sent) for sent in sentences]\n",
        "\n",
        "\n",
        "MAX_LEN = 128\n",
        "\n",
        "# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n",
        "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n",
        "# Pad our input tokens\n",
        "input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")\n",
        "# Create attention masks\n",
        "attention_masks = []\n",
        "\n",
        "# Create a mask of 1s for each token followed by 0s for padding\n",
        "for seq in input_ids:\n",
        "  seq_mask = [float(i>0) for i in seq]\n",
        "  attention_masks.append(seq_mask)\n",
        "\n",
        "prediction_inputs = torch.tensor(input_ids)\n",
        "prediction_masks = torch.tensor(attention_masks)\n",
        "prediction_labels = torch.tensor(labels)\n",
        "\n",
        "batch_size = 32\n",
        "\n",
        "\n",
        "prediction_data = TensorDataset(prediction_inputs, prediction_masks, prediction_labels)\n",
        "prediction_sampler = SequentialSampler(prediction_data)\n",
        "prediction_dataloader = DataLoader(prediction_data, sampler=prediction_sampler, batch_size=batch_size)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hba10sXR7Xi6"
      },
      "outputs": [],
      "source": [
        "# Prediction on test set\n",
        "\n",
        "# Put model in evaluation mode\n",
        "model.eval()\n",
        "\n",
        "# Tracking variables\n",
        "predictions , true_labels = [], []\n",
        "\n",
        "# Predict\n",
        "for batch in prediction_dataloader:\n",
        "  # Add batch to GPU\n",
        "  batch = tuple(t.to(device) for t in batch)\n",
        "  # Unpack the inputs from our dataloader\n",
        "  b_input_ids, b_input_mask, b_labels = batch\n",
        "  # Telling the model not to compute or store gradients, saving memory and speeding up prediction\n",
        "  with torch.no_grad():\n",
        "    # Forward pass, calculate logit predictions\n",
        "    logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)\n",
        "\n",
        "  # Move logits and labels to CPU\n",
        "  logits = logits.detach().cpu().numpy()\n",
        "  label_ids = b_labels.to('cpu').numpy()\n",
        "\n",
        "  # Store predictions and true labels\n",
        "  predictions.append(logits)\n",
        "  true_labels.append(label_ids)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cRaZQ4XC7kLs"
      },
      "outputs": [],
      "source": [
        "# Import and evaluate each test batch using Matthew's correlation coefficient\n",
        "from sklearn.metrics import matthews_corrcoef\n",
        "matthews_set = []\n",
        "\n",
        "for i in range(len(true_labels)):\n",
        "  matthews = matthews_corrcoef(true_labels[i],\n",
        "                 np.argmax(predictions[i], axis=1).flatten())\n",
        "  matthews_set.append(matthews)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IUM0UA1qJaVB"
      },
      "source": [
        "The final score will be based on the entire test set, but let's take a look at the scores on the individual batches to get a sense of the variability in the metric between batches.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xytAr_C48wnu"
      },
      "outputs": [],
      "source": [
        "matthews_set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oCYZa1lQ8Jn8"
      },
      "outputs": [],
      "source": [
        "# Flatten the predictions and true values for aggregate Matthew's evaluation on the whole dataset\n",
        "flat_predictions = [item for sublist in predictions for item in sublist]\n",
        "flat_predictions = np.argmax(flat_predictions, axis=1).flatten()\n",
        "flat_true_labels = [item for sublist in true_labels for item in sublist]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p-4EBZ8D8VsZ"
      },
      "outputs": [],
      "source": [
        "matthews_corrcoef(flat_true_labels, flat_predictions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jXx0jPc4HUfZ"
      },
      "source": [
        "Cool! In about half an hour and without doing any hyperparameter tuning (adjusting the learning rate, epochs, batch size, ADAM properties, etc.) we are able to get a good score. I should also mention we didn't train on the entire training dataset, but set aside a portion of it as our validation set for legibililty of code.\n",
        "\n",
        "If you look at the leaderboard [here](https://gluebenchmark.com/leaderboard/submission/zlssuBTm5XRs0aSKbFYGVIVdvbj1/-LhijX9VVmvJcvzKymxy), you'll see that our score (54) using the small version of BERT actually slightly outperforms the BERT authors' CoLA score (52) with the same model. If you wanted to tune the hyperparameters, use the entire train set, or use the larger BERT model instead you're likely able to push that score up higher into the mid- 60s, which looks like the state of the art as of July 2019!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GfjYoa6WmkN6"
      },
      "source": [
        "## Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xlQG7qgkmf4n"
      },
      "source": [
        "This post demonstrate that with a pre-trained BERT model you can quickly and effectively create a high quality model with minimal effort and training time using the pytorch interface, regardless of the specific NLP task you are interested in."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "BERT Fine-Tuning Sentence Classification.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}