notebooks/W3E_BPE_Transduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/gsarti/ik-nlp-tutorials/blob/main/notebooks/W3E_BPE_Transduction.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run in Colab to install local packages\n",
    "!pip install spacy transformers sentencepiece tokenizers datasets simplet5\n",
    "!python -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a BPE tokenizer and a Lexicon-based Transduction Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*This exercise follows the explanation of using BPE tokenization as explained on Huggingface [Build a Tokenizer from Scratch](https://huggingface.co/docs/tokenizers/python/latest/quicktour.html#build-a-tokenizer-from-scratch). Adapted from a notebook by Wietse de Vries*\n",
    "\n",
    "The [Tokenizers](https://huggingface.co/docs/tokenizers/python/latest/quicktour.html) library by Huggingface provides implementations of today’s most used tokenizers (especially subword-based ones) that is both easy to use and blazing fast (Rust-compiled code!).\n",
    "\n",
    "You will start by exploring the impact of different vocabulary sizes on a subword tokenizer using the Tokenizers library, and how these can be imported and used with spaCy. Finally, you will be asked to train a small transformer model to perform transduction from feminine to masculine words.\n",
    "\n",
    "Exercise 1 is mandatory and will be part of your graded midterm portfolio. Exercise 2 is optional, but we highly recommend you to complete it, especially if you're interested in the \"Modern Neural Networks meet Linguistic Theory\" final project.\n",
    "\n",
    "## Exercise 1: Byte Pair Encoding with Huggingface Tokenizers\n",
    "\n",
    "In the following exercise, we will use a byte-pair encoding (BPE) tokenizer (see Jurafsky & Martin Sec. 2.4.3 and [Sennich et al, 2015](https://aclanthology.org/P16-1162/) to create a vocabulary of frequent words and subwords, allowing us to handle less frequent words.\n",
    "\n",
    "### Setup\n",
    "\n",
    "The following code loads a BPE tokenizer and trainer, tells the system to use whitespace as a separator and defines `[UNK]` as a special token intended to handle unknown words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tokenizers import Tokenizer\n",
    "from tokenizers.models import BPE\n",
    "from tokenizers.trainers import BpeTrainer\n",
    "from tokenizers.pre_tokenizers import Whitespace\n",
    "\n",
    "tokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\n",
    "tokenizer.pre_tokenizer = Whitespace()\n",
    "trainer = BpeTrainer(special_tokens=[\"[UNK]\"], vocab_size=20000) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corpus\n",
    "\n",
    "The tokenizer creates a dictionary by concatenating characters and substrings into longer strings (possibly full words) based on frequency. So we need a corpus to learn what the most frequent words and substrings are. \n",
    "\n",
    "[Wikitext-103](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/) is a dump of the (English) Wikipedia. You can use the `train_from_iterator` method to train from the data in memory, which can be done using the `wikitext` corpus in the Huggingface Datasets library. Alternatively, you can download using wget, or directly from the webpage:\n",
    "\n",
    "```shell\n",
    "wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip\n",
    "unzip wikitext-103-raw-v1.zip\n",
    "```\n",
    "\n",
    "The unzipped data is 500 MB. Note that the file extension for the data-files is .raw but the data is just a (unicode) text file. Because this confuses (Ubuntu) Linux, files were renamed to .raw.txt. If you maintain the original .raw filenames, adapt the path below accordingly.\n",
    "\n",
    "### Run the trainer\n",
    "\n",
    "The command below trains the tokenizer on the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset wikitext (/home/gsarti/.cache/huggingface/datasets/wikitext/wikitext-103-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)\n"
     ]
    }
   ],
   "source": [
    "# UNCOMMENT AND ADAPT PATH TO TRAIN ON MANUALLY DOWNLOADED DATASETS\n",
    "# data = [f'wikitext-103-raw/wiki.{split}.raw.txt' for split in ['train','test','valid']]\n",
    "# tokenizer.train(trainer, data)\n",
    "\n",
    "import datasets\n",
    "dataset = datasets.load_dataset(\n",
    "    \"wikitext\", \"wikitext-103-raw-v1\", split=\"train+test+validation\"\n",
    ")\n",
    "\n",
    "# Build a generator to iterate over the dataset\n",
    "def batch_iterator(batch_size=1000):\n",
    "    for i in range(0, len(dataset), batch_size):\n",
    "        yield dataset[i : i + batch_size][\"text\"]\n",
    "\n",
    "tokenizer.train_from_iterator(batch_iterator(), trainer=trainer, length=len(dataset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the tokenizer\n",
    "\n",
    "Now that we have created a vocabulary, we can use it to tokenize a string into words and subtokens (for infrequent words).\n",
    "\n",
    "The example shows that most of the words are included in the vocabulary created by training on Wikipedia text, but that the acronym *UG*, the name *Hanze*, and the word *Applied*, *jointly* and *initiating* are segmented into subword strings. This suggests that these words were not seen during training, or very infrequently. (*UG* occurs 5 times in the training data and *Applied* over 200 times,  also note that the encoding is case-sensitive.). \n",
    "\n",
    "Try a few other examples to get a feeling for the lexical coverage of the tokenizer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['The', 'U', 'G', 'and', 'the', 'Han', 'ze', 'University', 'of', 'Ap', 'pl', 'ied', 'Sciences', 'are', 'joint', 'ly', 'initi', 'ating', 'a', 'pilot', 'rapid', 'testing', 'centre', ',', 'which', 'will', 'start', 'on', '18', 'January', '.']\n",
      "25 words and 31 segments\n"
     ]
    }
   ],
   "source": [
    "def show_tokens(text):\n",
    "    output = tokenizer.encode(text)\n",
    "    print(f\"Tokens: {output.tokens}\")\n",
    "    number_of_words = len(tokenizer.pre_tokenizer.pre_tokenize_str(text))\n",
    "    number_of_segments = len(output.tokens)\n",
    "    print(f\"{number_of_words} words and {number_of_segments} segments\")\n",
    "\n",
    "example = \"The UG and the Hanze University of Applied Sciences are jointly initiating a pilot rapid testing centre, which will start on 18 January.\"\n",
    "show_tokens(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your Turn: Experiment with Vocabulary Size\n",
    "\n",
    "The training data contains 103 M tokens and has a vocabulary size of 267,000 unique types. The default setting for the trainer is to create a dictionary of max 30,000 words. This means that a fair amount of compression takes place. Even more compression can be achieved by setting the vocab_size to a smaller value. \n",
    "\n",
    "1. Choose an example text consisting of at least 100 words. You may want to ensure that it contains some rare words or tokens. \n",
    "\n",
    "2. Experiment with various settings for vocab_size.\n",
    "\n",
    "3. Count the number of words in the example, and the number of segments created by the BPE-tokenizer. Note that if the number segments goes up, more words are segmented into subwords. \n",
    "\n",
    "4. What is the vocabulary size where the number of segments is approx. 150% of the number of words? \n",
    "\n",
    "5. For this setting, what was the longest word in your example text that was not segmented? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['Enter', 'some', 'English', 'text', 'containing', 'at', 'least', '100', 'words']\n",
      "25 words and 9 segments\n"
     ]
    }
   ],
   "source": [
    "# TODO: Try with various vocab_sizes\n",
    "# Important: You will need to redefine the tokenizer for every new vocab size,\n",
    "# otherwise you might incur in an \"PanicError: no entry found for key\" exception\n",
    "tokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\n",
    "tokenizer.pre_tokenizer = Whitespace()\n",
    "trainer = BpeTrainer(special_tokens=[\"[UNK]\"],vocab_size=30000)\n",
    "\n",
    "tokenizer.train_from_iterator(batch_iterator(), trainer=trainer, length=len(dataset))\n",
    "\n",
    "test_text = \"Enter some English text containing at least 100 words\"\n",
    "\n",
    "show_tokens(test_text)\n",
    "\n",
    "# Answer question 5 by going over the output, or write a \n",
    "# few lines of code to provide the answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the BPE Tokenizer into spaCy\n",
    "\n",
    "Now that you experimented with the creation of many tokenizers using Huggingface Tokenizers, you might want to move them to a more familiar environment. The following class lets you load a Huggingface Tokenizer into spaCy: the `get_words_spaces` function is used to preserve the whitespaces before tokens that are not word pieces.\n",
    "\n",
    "### Your Turn: Fill in the missing code\n",
    "\n",
    "Your task is to complete the `__call__` method of the `BPETokenizer` class to go from text to spaCy `Docs`, and finally to print the tokenized text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.tokens import Doc\n",
    "from spacy.vocab import Vocab\n",
    "import spacy\n",
    "\n",
    "class BPETokenizer:\n",
    "    def __init__(self, tokenizer, vocab):\n",
    "        self.tokenizer = tokenizer\n",
    "        self.vocab = vocab\n",
    "    \n",
    "    def get_words_spaces(self, tokens):\n",
    "        words = []\n",
    "        spaces = []\n",
    "        for i, (text, (_, end)) in enumerate(\n",
    "            zip(tokens.tokens, tokens.offsets)\n",
    "        ):\n",
    "            words.append(text)\n",
    "            if i < len(tokens.tokens) - 1:\n",
    "                # If next start != current end we assume a \n",
    "                # space in between\n",
    "                next_start, _ = tokens.offsets[i + 1]\n",
    "                spaces.append(next_start > end)\n",
    "            else:\n",
    "                spaces.append(True)\n",
    "        return words, spaces\n",
    "\n",
    "    def __call__(self, text):\n",
    "        # TODO: Encode the texts to obtain tokens\n",
    "        tokens = None\n",
    "        # TODO: Use get_words_spaces to obtain the words and spaces\n",
    "        words, spaces = None, None\n",
    "        return Doc(self.vocab, words=words, spaces=spaces)\n",
    "\n",
    "nlp = spacy.blank(\"en\")\n",
    "nlp.vocab = Vocab(strings=[\n",
    "    tok for tok in tokenizer.get_vocab().keys()\n",
    "])\n",
    "nlp.tokenizer = BPETokenizer(tokenizer, nlp.vocab)\n",
    "\n",
    "text = \"Jeff Bezos is a billionaire who became famous after the Dutch bridge controversy.\"\n",
    "# TODO: Convert the text in a list of tokens and print them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Exercise 2: Lexicon-based Transduction System\n",
    "\n",
    "In this exercise you will build a rule-based tool to transduce a given input text **from masculine to feminine**. You are provided with a list of pairs including feminine words and their masculine counterparts. To create a rule based transducer, the following components will be needed:\n",
    "\n",
    "1. Extract a subset of sentences from the `wikitext-103-raw-v1` containing masculine words (words from the list, gendered pronouns (e.g. he/his/him)). **Tip**: you can try to use the spaCy lemmas annotations to avoid removing inflected forms of words.\n",
    "\n",
    "Fill the `is_masculine` function so that only sentences containing masculine words are preserved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import datasets\n",
    "\n",
    "gender_lexicon = [\n",
    "    (\"Brother\", \"Sister\"),\n",
    "    (\"Drake\", \"Duck\"),\n",
    "    (\"Father\", \"Mother\"),\n",
    "    (\"Gentleman\", \"Lady\"),\n",
    "    (\"Husband\", \"Wife\"),\n",
    "    (\"Man\", \"Woman\"),\n",
    "    (\"Nephew\", \"Niece\"),\n",
    "    (\"Son\", \"Daughter\"),\n",
    "    (\"Wizard\", \"Witch\"),\n",
    "    (\"Boy\", \"Girl\"),\n",
    "    (\"Bull\", \"Cow\"),\n",
    "    (\"Cock\", \"Hen\"),\n",
    "    (\"Dog\", \"Bitch\"),\n",
    "    (\"Drone\", \"Bee\"),\n",
    "    (\"Gander\", \"Goose\"),\n",
    "    (\"Horse\", \"Mare\"),\n",
    "    (\"King\", \"Queen\"),\n",
    "    (\"Monk\", \"Nun\"),\n",
    "    (\"Sir\", \"Madam\"),\n",
    "    (\"Stag\", \"Hind\"),\n",
    "    (\"Stallion\", \"Mare\"),\n",
    "    (\"Tutor\", \"Governess\"),\n",
    "    (\"Drone\", \"Bee\"),\n",
    "    (\"Brother-in-law\", \"Sister-in-law\"),\n",
    "    (\"Son-in-law\", \"Daughter-in-law\"),\n",
    "    (\"Maternal-uncle\", \"Maternal-aunt\"),\n",
    "    (\"Step-son\", \"Step-daughter\"),\n",
    "    (\"Hostess\", \"Steward\"),\n",
    "    (\"Widow\", \"Widower\"),\n",
    "    (\"author\", \"authoress\"),\n",
    "    (\"count\", \"countess\"),\n",
    "    (\"heir\", \"heiress\"),\n",
    "    (\"manager\", \"manageress\"),\n",
    "    (\"patron\", \"patroness\"),\n",
    "    (\"priest\", \"priestess\"),\n",
    "    (\"baron\", \"baroness\"),\n",
    "    (\"giant\", \"giantess\"),\n",
    "    (\"host\", \"hostess\"),\n",
    "    (\"lion\", \"lioness\"),\n",
    "    (\"mayor\", \"mayoress\"),\n",
    "    (\"poet\", \"poetess\"),\n",
    "    (\"shepherd\", \"shepherdess\"),\n",
    "    (\"actor\", \"actress\"),\n",
    "    (\"conductor\", \"conductress\"),\n",
    "    (\"hunter\", \"huntress\"),\n",
    "    (\"prince\", \"princess\"),\n",
    "    (\"traitor\", \"traitress\"),\n",
    "    (\"master\", \"mistress\"),\n",
    "    (\"benefactor\", \"benefactress\"),\n",
    "    (\"founder\", \"foundress\"),\n",
    "    (\"instructor\", \"instructress\"),\n",
    "    (\"emperor\", \"empress\"),\n",
    "    (\"tiger\", \"tigress\"),\n",
    "    (\"waiter\", \"waitress\"),\n",
    "    (\"murderer\", \"murderess\"),\n",
    "    (\"hero\", \"heroine\"),\n",
    "    (\"fox\", \"vixen\"),\n",
    "    (\"sultan\", \"sultana\"),\n",
    "    (\"grandfather\", \"grandmother\"),\n",
    "    (\"manservant\", \"maidservant\"),\n",
    "    (\"milkman\", \"milkwoman\"),\n",
    "    (\"salesman\", \"saleswoman\"),\n",
    "    (\"great-uncle\", \"great-aunt\"),\n",
    "    (\"landlord\", \"landlady\"),\n",
    "    (\"he\", \"she\"),\n",
    "    (\"him\", \"her\"),\n",
    "    (\"his\", \"her\")\n",
    "]\n",
    "\n",
    "def is_masculine(text):\n",
    "    # TODO: Fill your regex with words from the wordlist\n",
    "    # (use '|'.join(...) to join them in the regex)\n",
    "    regex = None\n",
    "    return bool(re.search(regex, text, re.IGNORECASE))\n",
    "\n",
    "\n",
    "dataset = datasets.load_dataset(\n",
    "    \"wikitext\", \"wikitext-103-raw-v1\", split=\"train+test+validation\"\n",
    ")\n",
    "\n",
    "# We consider only the first 200 characters to avoid long paragraphs\n",
    "filtered_dataset = dataset.filter(lambda x: is_masculine(x[\"text\"][:200]))\n",
    "filtered_dataset = filtered_dataset.map(lambda x: {\"text\": x[\"text\"][:200]})\n",
    "filtered_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Create a `feminize` function that takes a sententence from the the filtered dataset and returns a feminized version of it, based on lexical pairs. Use it to create a new field \"feminine_text\" in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def feminize(text):\n",
    "    \"\"\"Returns a feminized version of text\"\"\"\n",
    "    feminized_text = text\n",
    "    for m, f in gender_lexicon:\n",
    "        # TODO: fill in your regex to select word m (adapted from is_masculine)\n",
    "        match_regex = None \n",
    "        # TODO: fill in your regex to replace m by f\n",
    "        substitute_regex = None\n",
    "        feminized_text = re.sub(match_regex, substitute_regex, feminized_text, re.IGNORECASE)\n",
    "    return feminized_text\n",
    "\n",
    "# TODO: Use filtered_dataset.map to add a feminized version of the text column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Rename the `text` field to `source_text` and the `feminine_text` field to `target_text` (this is needed for `SimpleT5` to work properly). Transform the dataset to Pandas DataFrame format and use the following code to train a simple neural transduction model.\n",
    "\n",
    "*(More info on the [T5 model](https://huggingface.co/t5-small) and the [SimpleT5](https://github.com/Shivanandroy/simpleT5) library)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from simplet5 import SimpleT5\n",
    "\n",
    "# TODO: Convert the Huggingface Dataset in a Pandas dataframe and split it in training\n",
    "# and evaluation sets (you decide the sizes based on your computational resources)\n",
    "train_df, eval_df = None, None\n",
    "\n",
    "model = SimpleT5()\n",
    "model.from_pretrained(model_type=\"t5\", model_name=\"t5-small\")\n",
    "model.train(\n",
    "    train_df=train_df,\n",
    "    eval_df=eval_df, \n",
    "    source_max_token_len=128, \n",
    "    target_max_token_len=128, \n",
    "    batch_size=8, max_epochs=3, use_gpu=torch.cuda.is_available()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Conclude by testing the model on a few examples of your choice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_model(\"t5\", \"<YOUR SAVED MODEL PATH>\", use_gpu=torch.cuda.is_available())\n",
    "\n",
    "text_to_feminize = \"my brother thought that his uncle was a duke\"\n",
    "model.predict(text_to_feminize)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "f9f85d98b63f393548f3009c8d52d8286e609a1467b1184fe464fb700873fbd3"
  },
  "kernelspec": {
   "display_name": "Python 3.8.10 64-bit ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}