train-your-own-ner-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train your own NER system using Conditional Random Fields\n",
    "\n",
    "#### Based on Barbara Plank's exercises"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-05T10:28:21.336962Z",
     "start_time": "2019-05-05T10:28:21.034459Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn_crfsuite import CRF,metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "Today, we are going to build a named entity recognizer by using an off-the-shelf toolkit called crfsuite.\n",
    "\n",
    "If you have an error in the previous line, you will need to install a package that is called `sklearn-crfsuite`.\n",
    "\n",
    "[Original CRFsuite tutorial](http://www.chokkan.org/software/crfsuite/tutorial.html)\n",
    "\n",
    "[sklearn-crfsuite documentation](https://sklearn-crfsuite.readthedocs.io/en/latest/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity Recognition\n",
    "\n",
    "Named Entity Recognition (NER) is the task to identify named entities, e.g., people, locations, organizations, in input text. It is usually treated as a sequence labeling task. Tokens that are not NEs (verbs, adjectives, etc.) are tagged as ”O” (other or not a named entity), so that every token is assigned a label. There are different ways on how to present NE tags, the most common is the BIO scheme (also known as IOB2). Here is an example of two sentences annotated for named entities in the BIO scheme:\n",
    "\n",
    "```\n",
    "John  B-PER\n",
    "lives   O\n",
    "in   O\n",
    "Copenhagen B-LOC\n",
    "\n",
    "Dirk B-PER\n",
    "flew O\n",
    "to O\n",
    "New B-LOC\n",
    "York I-LOC\n",
    "```\n",
    "\n",
    "  * What do you think B, I, and O stand for, respectively?\n",
    "  * Alternatively, we can drop the ”B-” and ”I-” prefix and just have the label by itself. Can you identify disadvantages and advantages of this approach? Discuss with your neighbor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to create a NER tagger yourself. For this exercise, we will provide you the required corpus, divided in the following files:\n",
    "\n",
    "  * `train.ner`\n",
    "  * `test.ner`\n",
    "  * pos/train.noun\n",
    "  * pos/test.noun.\n",
    "  \n",
    "Additionally, you could perform extra experiments with corpora from [this website](https://github.com/davidsbatista/NER-datasets). Or you could also perform experiments using the data from CONLL 2003, available on NLTK, by using the following command:\n",
    "\n",
    "```\n",
    "from nltk.corpus import conll2002\n",
    "\n",
    "train_sents = list(conll2002.iob_sents('esp.train'))\n",
    "test_sents = list(conll2002.iob_sents('esp.testa'))\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1: Create data for every token\n",
    "\n",
    "Acquaint yourself with the functions below.\n",
    "  * What input does it expect?\n",
    "  * What output does it produce?\n",
    "  * Run the script on the train file and make sure you understand what it produces. Explain to your neighbor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import conll2002\n",
    "\n",
    "train_sents = list(conll2002.iob_sents('esp.train'))\n",
    "test_sents = list(conll2002.iob_sents('esp.testa'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<_sre.SRE_Match object; span=(1, 2), match='.'>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'[^\\w\\d\\s]', 'a.b')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# createdata.py\n",
    "def extract_data(file_ner,file_pos,separator=\" \"):\n",
    "    \"\"\"\n",
    "    extract token-level data (features) from file that is NER tagged and the corresponding POS file \n",
    "    this method should output the format expected by the next function word2features\n",
    "    that then looks at a tokens context and produces the respective crfsuite format\n",
    "    \n",
    "    # Field names ==> this is what this script produces, needs to be aligned with nerfeats.py\n",
    "    fields = 'w pos y'\n",
    "    # alternative (add whatever you wanna add and make sure its aligned with nerfeats.py)\n",
    "    fields = 'w pos cap y'\n",
    "    \"\"\"\n",
    "\n",
    "    # read NER and POS from the two files\n",
    "    words_tags=read_conll_file(file_ner)\n",
    "    words_pos=read_conll_file(file_pos)\n",
    "    \n",
    "    # result variable\n",
    "    # It will be a list of sentences\n",
    "    # Each sentence will have a list of words\n",
    "    # Each word will be represented by a three element tuple: WORD, POS, BIO-TAG\n",
    "    result = []\n",
    "    \n",
    "    ## some checks, e.g., that both files have same length, same tokens\n",
    "    assert(len(words_tags)==len(words_pos))\n",
    "    \n",
    "    for (words,tags),(_,pos) in zip(words_tags,words_pos):\n",
    "        \n",
    "        result.append([]) # We append one list more.\n",
    "        \n",
    "        for word,pos,tag in zip(words,pos,tags):\n",
    "            # first letter is capitalized\n",
    "            cap=\"+\" if word[0].isupper() else \"-\"\n",
    "            all_caps = \"+\" if word.isupper() else \"-\"\n",
    "            word_punct = \"+\" if re.search(r'[^\\w\\d\\s]', word) else \"-\"\n",
    "            #################################\n",
    "            ###### YOUR FEATURES HERE #######  \n",
    "            #################################\n",
    "            \n",
    "            ## todo: output the cap feature and more \n",
    "            \n",
    "            # In the last list we created before (result[-1]),\n",
    "            # We will add each word, represented by the three element tuple\n",
    "            result[-1].append((word,pos,tag,cap,all_caps,word_punct))\n",
    "\n",
    "    return result\n",
    "        \n",
    "        \n",
    "def read_conll_file(file_name):\n",
    "    \"\"\"\n",
    "    read in a file with format:\n",
    "    word1    tag1\n",
    "    ...      ...\n",
    "    wordN    tagN\n",
    "    \n",
    "    returns a list of list with (words,tags) tuples for every sentence\n",
    "    \"\"\"\n",
    "    content=[]\n",
    "\n",
    "    current_words = []\n",
    "    current_tags = []\n",
    "    \n",
    "    for line in open(file_name, encoding='utf-8'):\n",
    "        line = line.strip()\n",
    "        \n",
    "        if line:\n",
    "            word, tag = line.split('\\t')\n",
    "            current_words.append(word)\n",
    "            current_tags.append(tag)\n",
    "\n",
    "        else:\n",
    "            content.append((current_words, current_tags))\n",
    "            current_words = []\n",
    "            current_tags = []\n",
    "\n",
    "    # if file does not end in newline (it should...), check whether there is an instance in the buffer\n",
    "    if current_tags != []:\n",
    "        content.append((current_words, current_tags))\n",
    "    return content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('We', 'PRON', 'O', '+', '-', '-'),\n",
       "  ('stay', 'VERB', 'O', '-', '-', '-'),\n",
       "  ('in', 'ADP', 'O', '-', '-', '-'),\n",
       "  ('Manhattan', 'NOUN', 'B-LOC', '+', '-', '-'),\n",
       "  ('and', 'CONJ', 'O', '-', '-', '-'),\n",
       "  ('it', 'PRON', 'O', '-', '-', '-'),\n",
       "  (\"'s\", 'PRT', 'O', '-', '-', '+'),\n",
       "  ('a', 'DET', 'O', '-', '-', '-'),\n",
       "  ('long', 'ADV', 'O', '-', '-', '-'),\n",
       "  ('way', 'NOUN', 'O', '-', '-', '-'),\n",
       "  ('to', 'PRT', 'O', '-', '-', '-'),\n",
       "  ('come', 'VERB', 'O', '-', '-', '-'),\n",
       "  ('.', '.', 'O', '-', '-', '+')]]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#train_sents = extract_data(\"data/train.ner\",\"data/pos/train.noun\")\n",
    "#test_sents = extract_data(\"data/test.ner\",\"data/pos/test.noun\")\n",
    "\n",
    "\n",
    "train_sents = extract_data(\"data/train.ner\",\"data/upos/train.lap-upos\")\n",
    "test_sents = extract_data(\"data/test.ner\",\"data/upos/test.lap-upos\")\n",
    "\n",
    "train_sents[0:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2: Create features for crfsuite\n",
    "\n",
    "Given the output of the previous `extract_data` function, the next step is to produce features for a given token. In NER it is common to use the surrounding POS information as well as shape features. The function `word2features` gets a sentence and a word index, and generates a feature representation for each of the words. Below this function, there is also a function `sent2features()`, together with `sent2labels` and `sent2tokens`. \n",
    "\n",
    "It would be helpful for you to check how these functions work. Why don't you call each of those functions with their respective input variables?\n",
    "\n",
    "After that, you should extend the function to add more features.\n",
    "\n",
    "##### Note: whenever you add new information to `extract_data()` (e.g. start by outputting the cap information) you need to adjust the `word2features()` function accordingly to take these new features as input. `extract_data()` just looks at one token at a time, `word2features()` looks at the context and produces the features that crfsuite expects.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nerfeats.py\n",
    "\n",
    "def word2features(sent, i):\n",
    "    word = sent[i][0]\n",
    "    postag = sent[i][1]\n",
    "    caps = sent[i][3]\n",
    "    all_caps = sent[i][4]\n",
    "    word_punct = sent[i][5]\n",
    "\n",
    "    features = {\n",
    "        'bias': 1.0,\n",
    "        'word.lower()': word.lower(),\n",
    "        'word[-3:]': word[-3:],\n",
    "        'word[-2:]': word[-2:],\n",
    "        'postag': postag,\n",
    "        'postag[:2]': postag[:2],\n",
    "        'caps': caps,\n",
    "        'allcaps': all_caps,\n",
    "        #'wordpunct': word_punct,\n",
    "    }\n",
    "    if i > 0:\n",
    "        word1 = sent[i-1][0]\n",
    "        postag1 = sent[i-1][1]\n",
    "        caps1 = sent[i-1][3]\n",
    "        all_caps1 = sent[i-1][4]\n",
    "        word_punct1 = sent[i-1][5]\n",
    "        features.update({\n",
    "            '-1:word.lower()': word1.lower(),\n",
    "            '-1:postag': postag1,\n",
    "            '-1:postag[:2]': postag1[:2],\n",
    "            '-1:caps': caps1,\n",
    "            '-1:allcaps': all_caps1,\n",
    "            #'-1:wordpunct': word_punct1,\n",
    "        })\n",
    "    else:\n",
    "        features['BOS'] = True\n",
    "\n",
    "    if i < len(sent)-1:\n",
    "        word1 = sent[i+1][0]\n",
    "        postag1 = sent[i+1][1]\n",
    "        caps1 = sent[i+1][3]\n",
    "        all_caps1 = sent[i+1][4]\n",
    "        word_punct1 = sent[i+1][5]\n",
    "        features.update({\n",
    "            '+1:word.lower()': word1.lower(),\n",
    "            '+1:postag': postag1,\n",
    "            '+1:postag[:2]': postag1[:2],\n",
    "            '+1:caps': caps1,\n",
    "            '+1:allcaps': all_caps1,\n",
    "            #'+1:wordpunct': word_punct1,\n",
    "\n",
    "        })\n",
    "    else:\n",
    "        features['EOS'] = True\n",
    "\n",
    "    return features\n",
    "\n",
    "\n",
    "def sent2features(sent):\n",
    "    return [word2features(sent, i) for i in range(len(sent))]\n",
    "\n",
    "def sent2labels(sent):\n",
    "    return [label for token, postag, label, caps, all_caps, word_punct in sent]\n",
    "\n",
    "def sent2tokens(sent):\n",
    "    return [token for token, postag, label, caps, all_caps, word_punct in sent]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('He', 'PRON', 'O', '+', '-', '-'), ('would', 'VERB', 'O', '-', '-', '-'), ('give', 'VERB', 'O', '-', '-', '-'), ('no', 'DET', 'O', '-', '-', '-'), ('forecasts', 'NOUN', 'O', '-', '-', '-'), ('on', 'ADP', 'O', '-', '-', '-'), ('advertising', 'NOUN', 'O', '-', '-', '-'), ('revenue', 'NOUN', 'O', '-', '-', '-'), ('.', '.', 'O', '-', '-', '+')]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'bias': 1.0,\n",
       " 'word.lower()': 'would',\n",
       " 'word[-3:]': 'uld',\n",
       " 'word[-2:]': 'ld',\n",
       " 'postag': 'VERB',\n",
       " 'postag[:2]': 'VE',\n",
       " 'caps': '-',\n",
       " 'allcaps': '-',\n",
       " '-1:word.lower()': 'he',\n",
       " '-1:postag': 'PRON',\n",
       " '-1:postag[:2]': 'PR',\n",
       " '-1:caps': '+',\n",
       " '-1:allcaps': '-',\n",
       " '+1:word.lower()': 'give',\n",
       " '+1:postag': 'VERB',\n",
       " '+1:postag[:2]': 'VE',\n",
       " '+1:caps': '-',\n",
       " '+1:allcaps': '-'}"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#YOUR CODE HERE\n",
    "print(train_sents[5])\n",
    "word2features(train_sents[5],1)\n",
    "#sent2features(train_sents[5])\n",
    "#sent2labels(train_sents[5]),sent2tokens(train_sents[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create training data\n",
    "\n",
    "You have done all the required previous steps to generate training data that will be used to train a CRF model.\n",
    "\n",
    "In the following four lines we just create training data and save them in a similar format to the one used by `scikit-learn`. There is a difference, though. Can you think about a difference from regular classification tasks that you have done using `scikit-learn`?\n",
    "\n",
    "##### Hint: You can check how the variable `X_train`  or `y_train` looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.53 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "X_train = [sent2features(s) for s in train_sents]\n",
    "y_train = [sent2labels(s) for s in train_sents]\n",
    "\n",
    "X_test = [sent2features(s) for s in test_sents]\n",
    "y_test = [sent2labels(s) for s in test_sents]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 3: Train your NER system using crfsuite\n",
    "\n",
    "Train your system using `sklearn-crfsuite`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading training data to CRFsuite: 100%|██████████████| 9034/9034 [00:04<00:00, 1859.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Feature generation\n",
      "type: CRF1d\n",
      "feature.minfreq: 0.000000\n",
      "feature.possible_states: 1\n",
      "feature.possible_transitions: 1\n",
      "0....1....2....3....4....5....6....7....8....9....10\n",
      "Number of features: 343308\n",
      "Seconds required: 9.401\n",
      "\n",
      "L-BFGS optimization\n",
      "c1: 0.000000\n",
      "c2: 1.000000\n",
      "num_memories: 6\n",
      "max_iterations: 100\n",
      "epsilon: 0.000010\n",
      "stop: 10\n",
      "delta: 0.000010\n",
      "linesearch: MoreThuente\n",
      "linesearch.max_iterations: 20\n",
      "\n",
      "Iter 1   time=0.66  loss=92395.76 active=343308 feature_norm=1.00\n",
      "Iter 2   time=0.32  loss=85403.92 active=343308 feature_norm=1.18\n",
      "Iter 3   time=0.33  loss=79392.87 active=343308 feature_norm=1.35\n",
      "Iter 4   time=0.34  loss=63782.86 active=343308 feature_norm=2.02\n",
      "Iter 5   time=0.33  loss=43881.53 active=343308 feature_norm=3.83\n",
      "Iter 6   time=0.33  loss=39055.84 active=343308 feature_norm=4.81\n",
      "Iter 7   time=0.34  loss=35664.45 active=343308 feature_norm=6.56\n",
      "Iter 8   time=0.33  loss=30998.36 active=343308 feature_norm=7.74\n",
      "Iter 9   time=0.32  loss=30064.46 active=343308 feature_norm=8.56\n",
      "Iter 10  time=0.34  loss=29156.31 active=343308 feature_norm=9.36\n",
      "Iter 11  time=0.33  loss=27255.97 active=343308 feature_norm=10.94\n",
      "Iter 12  time=0.36  loss=26815.50 active=343308 feature_norm=13.25\n",
      "Iter 13  time=0.32  loss=24282.21 active=343308 feature_norm=14.09\n",
      "Iter 14  time=0.34  loss=23686.29 active=343308 feature_norm=14.35\n",
      "Iter 15  time=0.36  loss=22873.14 active=343308 feature_norm=14.83\n",
      "Iter 16  time=0.33  loss=22394.20 active=343308 feature_norm=15.61\n",
      "Iter 17  time=0.35  loss=21655.21 active=343308 feature_norm=15.59\n",
      "Iter 18  time=0.33  loss=21089.34 active=343308 feature_norm=15.57\n",
      "Iter 19  time=0.38  loss=20422.50 active=343308 feature_norm=15.94\n",
      "Iter 20  time=0.36  loss=19828.39 active=343308 feature_norm=16.35\n",
      "Iter 21  time=0.33  loss=19240.12 active=343308 feature_norm=17.07\n",
      "Iter 22  time=0.35  loss=19011.02 active=343308 feature_norm=17.36\n",
      "Iter 23  time=0.34  loss=18776.73 active=343308 feature_norm=17.22\n",
      "Iter 24  time=0.34  loss=18440.60 active=343308 feature_norm=17.27\n",
      "Iter 25  time=0.36  loss=18150.85 active=343308 feature_norm=17.51\n",
      "Iter 26  time=0.36  loss=17727.65 active=343308 feature_norm=18.10\n",
      "Iter 27  time=0.35  loss=17488.09 active=343308 feature_norm=19.68\n",
      "Iter 28  time=0.32  loss=16982.20 active=343308 feature_norm=19.80\n",
      "Iter 29  time=0.33  loss=16741.11 active=343308 feature_norm=19.95\n",
      "Iter 30  time=0.34  loss=16408.77 active=343308 feature_norm=20.77\n",
      "Iter 31  time=0.64  loss=16188.11 active=343308 feature_norm=21.38\n",
      "Iter 32  time=0.33  loss=15910.35 active=343308 feature_norm=22.02\n",
      "Iter 33  time=0.33  loss=15438.93 active=343308 feature_norm=23.26\n",
      "Iter 34  time=0.36  loss=15133.64 active=343308 feature_norm=24.08\n",
      "Iter 35  time=0.33  loss=14694.46 active=343308 feature_norm=25.28\n",
      "Iter 36  time=0.64  loss=14440.81 active=343308 feature_norm=26.52\n",
      "Iter 37  time=0.33  loss=14239.02 active=343308 feature_norm=26.96\n",
      "Iter 38  time=0.47  loss=13974.10 active=343308 feature_norm=27.60\n",
      "Iter 39  time=0.42  loss=13893.56 active=343308 feature_norm=29.26\n",
      "Iter 40  time=0.35  loss=13601.76 active=343308 feature_norm=29.74\n",
      "Iter 41  time=0.39  loss=13497.91 active=343308 feature_norm=29.99\n",
      "Iter 42  time=0.40  loss=13319.37 active=343308 feature_norm=30.72\n",
      "Iter 43  time=0.36  loss=13009.74 active=343308 feature_norm=32.40\n",
      "Iter 44  time=0.72  loss=12853.73 active=343308 feature_norm=33.54\n",
      "Iter 45  time=0.33  loss=12620.31 active=343308 feature_norm=35.07\n",
      "Iter 46  time=0.34  loss=12474.89 active=343308 feature_norm=35.93\n",
      "Iter 47  time=0.34  loss=12290.38 active=343308 feature_norm=37.17\n",
      "Iter 48  time=0.35  loss=12151.76 active=343308 feature_norm=39.07\n",
      "Iter 49  time=0.34  loss=11960.38 active=343308 feature_norm=40.25\n",
      "Iter 50  time=0.36  loss=11864.86 active=343308 feature_norm=40.31\n",
      "Iter 51  time=0.52  loss=11769.11 active=343308 feature_norm=41.09\n",
      "Iter 52  time=0.40  loss=11668.37 active=343308 feature_norm=41.79\n",
      "Iter 53  time=0.35  loss=11550.04 active=343308 feature_norm=42.84\n",
      "Iter 54  time=0.35  loss=11432.11 active=343308 feature_norm=43.77\n",
      "Iter 55  time=0.37  loss=11332.14 active=343308 feature_norm=44.20\n",
      "Iter 56  time=0.35  loss=11131.52 active=343308 feature_norm=45.22\n",
      "Iter 57  time=0.81  loss=11058.44 active=343308 feature_norm=45.93\n",
      "Iter 58  time=0.35  loss=10982.84 active=343308 feature_norm=46.37\n",
      "Iter 59  time=0.33  loss=10935.94 active=343308 feature_norm=46.49\n",
      "Iter 60  time=0.35  loss=10842.27 active=343308 feature_norm=47.11\n",
      "Iter 61  time=0.36  loss=10716.94 active=343308 feature_norm=47.94\n",
      "Iter 62  time=0.41  loss=10592.66 active=343308 feature_norm=48.86\n",
      "Iter 63  time=0.87  loss=10542.65 active=343308 feature_norm=49.45\n",
      "Iter 64  time=0.37  loss=10468.96 active=343308 feature_norm=49.81\n",
      "Iter 65  time=0.39  loss=10397.33 active=343308 feature_norm=50.21\n",
      "Iter 66  time=0.42  loss=10347.68 active=343308 feature_norm=50.89\n",
      "Iter 67  time=0.32  loss=10276.54 active=343308 feature_norm=51.48\n",
      "Iter 68  time=0.40  loss=10244.04 active=343308 feature_norm=51.50\n",
      "Iter 69  time=0.35  loss=10193.82 active=343308 feature_norm=51.65\n",
      "Iter 70  time=0.38  loss=10084.15 active=343308 feature_norm=52.25\n",
      "Iter 71  time=0.70  loss=10024.71 active=343308 feature_norm=52.64\n",
      "Iter 72  time=0.39  loss=9948.42  active=343308 feature_norm=53.31\n",
      "Iter 73  time=0.37  loss=9906.47  active=343308 feature_norm=53.69\n",
      "Iter 74  time=0.35  loss=9865.51  active=343308 feature_norm=54.36\n",
      "Iter 75  time=0.39  loss=9808.01  active=343308 feature_norm=54.59\n",
      "Iter 76  time=0.48  loss=9755.86  active=343308 feature_norm=54.80\n",
      "Iter 77  time=0.37  loss=9675.26  active=343308 feature_norm=55.48\n",
      "Iter 78  time=0.72  loss=9644.12  active=343308 feature_norm=55.87\n",
      "Iter 79  time=0.35  loss=9604.65  active=343308 feature_norm=56.09\n",
      "Iter 80  time=0.51  loss=9541.49  active=343308 feature_norm=56.54\n",
      "Iter 81  time=0.35  loss=9519.10  active=343308 feature_norm=56.75\n",
      "Iter 82  time=0.36  loss=9488.52  active=343308 feature_norm=56.67\n",
      "Iter 83  time=0.37  loss=9453.45  active=343308 feature_norm=56.64\n",
      "Iter 84  time=0.36  loss=9417.02  active=343308 feature_norm=56.65\n",
      "Iter 85  time=0.48  loss=9377.66  active=343308 feature_norm=56.93\n",
      "Iter 86  time=0.38  loss=9338.85  active=343308 feature_norm=57.09\n",
      "Iter 87  time=0.34  loss=9316.69  active=343308 feature_norm=57.17\n",
      "Iter 88  time=0.44  loss=9288.38  active=343308 feature_norm=57.32\n",
      "Iter 89  time=0.46  loss=9260.55  active=343308 feature_norm=57.47\n",
      "Iter 90  time=0.44  loss=9205.64  active=343308 feature_norm=57.69\n",
      "Iter 91  time=0.42  loss=9178.26  active=343308 feature_norm=58.08\n",
      "Iter 92  time=0.45  loss=9147.30  active=343308 feature_norm=57.92\n",
      "Iter 93  time=0.40  loss=9129.67  active=343308 feature_norm=57.86\n",
      "Iter 94  time=0.35  loss=9102.78  active=343308 feature_norm=57.84\n",
      "Iter 95  time=0.41  loss=9071.66  active=343308 feature_norm=57.89\n",
      "Iter 96  time=0.41  loss=9047.21  active=343308 feature_norm=58.03\n",
      "Iter 97  time=0.34  loss=9008.76  active=343308 feature_norm=58.17\n",
      "Iter 98  time=0.45  loss=8977.97  active=343308 feature_norm=58.29\n",
      "Iter 99  time=0.41  loss=8946.33  active=343308 feature_norm=58.28\n",
      "Iter 100 time=0.35  loss=8926.22  active=343308 feature_norm=58.18\n",
      "L-BFGS terminated with the maximum number of iterations\n",
      "Total seconds required for training: 39.741\n",
      "\n",
      "Storing the model\n",
      "Number of active features: 343308 (343308)\n",
      "Number of active attributes: 49037 (49037)\n",
      "Number of active labels: 7 (7)\n",
      "Writing labels\n",
      "Writing attributes\n",
      "Writing feature references for transitions\n",
      "Writing feature references for attributes\n",
      "Seconds required: 0.455\n",
      "\n",
      "Wall time: 54.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "crf = CRF(\n",
    "    algorithm='lbfgs',\n",
    "    max_iterations=100,\n",
    "    all_possible_transitions=True,\n",
    "    all_possible_states=True,\n",
    "    verbose=True\n",
    ")\n",
    "crf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 4: Let's see how it works!\n",
    "\n",
    "Our model is saved in a variable called `crf`. Please, take the first three sentences from the test set and check how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([['O', 'O', 'B-LOC', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O'],\n",
       "  ['O', 'B-PER', 'I-PER', 'O', 'B-LOC', 'O', 'O'],\n",
       "  ['O', 'O', 'O']],\n",
       " ['Mladost',\n",
       "  '(',\n",
       "  'BJ',\n",
       "  ')',\n",
       "  'NUMBER',\n",
       "  'NUMBER',\n",
       "  'NUMBER',\n",
       "  '2',\n",
       "  '2',\n",
       "  'NUMBER',\n",
       "  'NUMBER'],\n",
       " ['NUMBER', 'Andrea', 'Muller', '(', 'Germany', ')', 'NUMBER'],\n",
       " ['Finals', ',', 'doubles'])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "crf.predict(X_test[0:3]),sent2tokens(test_sents[0]),sent2tokens(test_sents[1]),sent2tokens(test_sents[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 5: Let's see how it ACTUALLY works!\n",
    "\n",
    "In order to evaluate our model, as you may have realized, we cannot use the regular functions from `sklearn.metrics`. We are lucky, though! The `sklearn-crfsuite` suite includes a package with metrics, especially suited for this.\n",
    "\n",
    "Could you please, then, check which is this models' macro-average F1-score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8817677592812444"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = crf.predict(X_test)\n",
    "\n",
    "metrics.flat_f1_score(y_test, y_pred,\n",
    "                      average='macro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B-LOC', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'I-LOC']"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = list(crf.classes_)\n",
    "labels.remove('O')\n",
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.863295931365789"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.flat_f1_score(y_test, y_pred,\n",
    "                      average='macro', labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "       B-LOC      0.899     0.886     0.893      1918\n",
      "       I-LOC      0.858     0.746     0.798       268\n",
      "       B-ORG      0.840     0.805     0.822      1733\n",
      "       I-ORG      0.837     0.814     0.825      1095\n",
      "       B-PER      0.897     0.898     0.897      1915\n",
      "       I-PER      0.919     0.969     0.943      1350\n",
      "\n",
      "   micro avg      0.881     0.871     0.876      8279\n",
      "   macro avg      0.875     0.853     0.863      8279\n",
      "weighted avg      0.880     0.871     0.875      8279\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sorted_labels = sorted(\n",
    "    labels,\n",
    "    key=lambda name: (name[1:], name[0])\n",
    ")\n",
    "print(metrics.flat_classification_report(\n",
    "    y_test, y_pred, labels=sorted_labels, digits=3\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "       B-LOC      0.899     0.886     0.893      1918\n",
      "       B-ORG      0.840     0.805     0.822      1733\n",
      "       B-PER      0.897     0.898     0.897      1915\n",
      "       I-LOC      0.858     0.746     0.798       268\n",
      "       I-ORG      0.837     0.814     0.825      1095\n",
      "       I-PER      0.919     0.969     0.943      1350\n",
      "           O      0.992     0.994     0.993     48257\n",
      "\n",
      "   micro avg      0.976     0.976     0.976     56536\n",
      "   macro avg      0.892     0.873     0.882     56536\n",
      "weighted avg      0.975     0.976     0.975     56536\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(metrics.flat_classification_report(\n",
    "    y_test, y_pred, digits=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nouns:\n",
    "    precision    recall  f1-score   support\n",
    "\n",
    "       B-LOC      0.873     0.794     0.832      1918\n",
    "       B-ORG      0.829     0.624     0.712      1733\n",
    "       B-PER      0.900     0.773     0.832      1915\n",
    "       I-LOC      0.743     0.668     0.703       268\n",
    "       I-ORG      0.791     0.565     0.659      1095\n",
    "       I-PER      0.903     0.889     0.896      1350\n",
    "           O      0.968     0.992     0.980     48257\n",
    "   micro avg      0.955     0.955     0.955     56536\n",
    "   macro avg      0.858     0.758     0.802     56536\n",
    "weighted avg      0.952     0.955     0.952     56536"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 6: Run your own POS tagger on the training and dev file\n",
    "\n",
    "Instead of using the POS tags we provide, which are too simplistic (it just tags every word as NOUN) you could use the real POS tagged files called `upos/train.lap-upos` and `upos/test.lap-upos`.\n",
    "\n",
    "You could also run your own POS tagger and use its predictions on the train and test file. Make sure your POS tagger produces the output the NER system expects.\n",
    "\n",
    "What accuracies to you observe? What downstream effect has the accuracy of the POS tagger on the NER task? Hypothesize and investigate!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}