14_Sequence_tagging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# PoS tagging and entity extraction\n",
    "\n",
    "Practical course material for the ASDM Class 09 (Text Mining) by Florian Leitner.\n",
    "\n",
    "© 2016-2017 Florian Leitner. All rights reserved.\n",
    "\n",
    "## Setup\n",
    "\n",
    "### NLTK\n",
    "\n",
    "NLTK-based NLP (should require no more additional steps):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk import ne_chunk, pos_tag, word_tokenize as tokenize\n",
    "from nltk.stem.snowball import EnglishStemmer\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stanford CoreNLP Taggers\n",
    "\n",
    "Some of the Stanford [CoreNLP](https://stanfordnlp.github.io/CoreNLP/index.html) tools have wrappers provided by NLTK (Note: and therefore, this part assumes/requires you have **Java** installed on your machine!):\n",
    "\n",
    "Using those taggers via NLTK requires that you download the (_basic_) Stanford [PoS tagger](http://nlp.stanford.edu/software/tagger.shtml#Download) and [NER tagger](http://nlp.stanford.edu/software/CRF-NER.shtml#Download) ZIP packages, and put the following files from those two packages into a directory named \"`stanford`\" that itself should be located in the same directory as this notebook; If you cloned the tutorial from its repo, you already have the relevant directory and files in place.\n",
    "\n",
    "From the NER tagger archive, you need these two files:\n",
    "\n",
    "* `stanford-ner.jar`\n",
    "* `english.all.3class.distsim.crf.ser.gz`\n",
    "\n",
    "And from the PoS tagger:\n",
    "\n",
    "* `english-bidirectional-distsim.tagger`\n",
    "* `stanford-postagger.jar`\n",
    "\n",
    "Add the *full path* to the directories containing `stanford-postagger.jar` and `stanford-ner.jar` to your `CLASSPATH` environment variable and set a `STANFORD_MODELS` en environment variable pointing to the *full path* of this `stanford` directory itself. Make sure you launch the Jupyter Notebook command `jupyter-notebook` in the terminal you set those two environment variables.\n",
    "\n",
    "```bash\n",
    "export STANFORD_MODELS=`pwd`/stanford\n",
    "export CLASSPATH=$STANFORD_MODELS/stanford-ner.jar:$STANFORD_MODELS/stanford-postagger.jar\n",
    "```\n",
    "\n",
    "If you get stuck, follow the [full installation instructions](https://github.com/nltk/nltk/wiki/Installing-Third-Party-Software#stanford-tagger-ner-tokenizer-and-parser) from the NLTK Wiki."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SpaCy\n",
    "\n",
    "The SpaCy NLP tool ([installation instructions](https://spacy.io/docs/usage/#installation) - [compressed form](https://spacy.io/docs/usage/lightning-tour)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this class, we will explore Python-based means to apply the sequence tagging and chunking techniques discussed in class, namely part-of-speech tagging, named entity recognition (NER), and phrase chunking.\n",
    "\n",
    "Note that NER is the hardest topic of what we will be looking into today, not the least to the extremely broad range of entity types: e.g., corporations, persons, locations & geo-political entities [GPEs], consumer products, genes & proteins, chemicals, etc. - to name just a very small selection of the more popular entity types in NLP.\n",
    "\n",
    "A more in-depth introduction to most of what we are doing here - PoS tagging, chunking, NER - is well documented in the NLTK Book, [Chapter 7](http://www.nltk.org/book/ch07.html), which you might want to have study in depth if you want to get more serious with NLP in general.\n",
    "\n",
    "## Tagging, Chunking, and NER with NLTK (w/o CoreNLP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA98AAABlCAIAAADbKMdpAAAACXBIWXMAAA3XAAAN1wFCKJt4AAAAHXRFWHRTb2Z0d2FyZQBHUEwgR2hvc3RzY3JpcHQgOS4yMcb0+xQAACAASURBVHic7d1PbNvYfi/w43+yLSexmBknmbkPdUTfGbRJF60YBw/FBWLA9KIt0EVhaj1dWAa6bi3t2qU8+xaQ7qIt0JV0F911IQ7qPBToQyzOw0Phi4u5EMfu7e2dxPeaTjL+k8i2uviNT89Q1BElUhJlfT+LxBJF8Yh/Dn88/J3DsUajwQAAAAAAIALGB10AAAAAAAD4DqJzAAAAAICoQHQOAAAAABAVk4MuAAAAQPds27ZtmzGm6/qgywIAEAK0nQMAwLAqFou6rlcqlUqloqqqqqqDLhEAQFBjGLMFAACGkW3buq5blqUoCmPMcZy7d+/ipAYAww5t5wAAMJQcx1FVlUJzxpiiKIVCYbBFAgAIDm3nAAAwrDRN03U9nU5rmjbosgAAhANt5wAAMKwsy1paWioUCrqua5pWLpcHXSIAgKDQdg4AADcBpaGbpom+oQAw1NB2DgAAQ6lYLFqWxV+qqmoYBo2uCAAwvBCdAwDAUKrVamI3UMdxyuUyGs4BYNjhaUQAADDEqGOo4zimaeZyOUTnADDskHcOAABDzHEcym/Bs0IB4GZAdA4AAAAAEBXIOwcAAAAAiApE5wAAAAAAUYHoHAAAAAAgKhCdAwAAAABEBUZUBACAaHFOTqz9fcbY2/Pzn798yd+39vd/cXR0dHLCGLu8ujp59+7dxcX7i4uTd+9ofIOpiYmJ8fHpycnY5ORcLJZ6+PDu3NxsLPa/FOXWzMyn9+/rjx8P5BcBAPiHMVsAAKBL1v6+c3LietM5Pf3Xr7765vVr8c3zev3//+IXrk++OT9/e3Z2cXXV21K2FpuYGB8fZ4zNxmJTExO3pqfp3w9v356fnb0zO/vJ/fuMsU/u3789M8MY0x4+VObmBlVaABgRiM4BAG4+c2+v+U3r4ODo22/Fd87r9W9evxabq+tXV2/PzhhjB7/5TViFGR8bmxgfr19etvrA3PR0Yna2wdi7ej0+PX1erx+fnvLPP5ifVxcWfvTJJw/m5//pyy//z1dfrT95UvzsM+fkpLCz84//9m//dXw8F4spc3ONRuOXx8f8a8cYC/eEF5uYmJma4vH6zNTU/Ozsww8/5B949tu//en9+/wlgnsA8APROQBAVNivXtmHh83vV7xia/vw0Dk9pb/rl5cUQ//q9WtXo3WI5mKxuelpxtjkxMTF5SVj7NbMzPuLi/eXl4yx8bExyaIT8bj28CH9rS4sKPE4/b32+DFj7FfHx2/fvTv49a8ZY9bBgbW/f3z905ILC+rCgra4uHTvnrqwwFNTzL29zD/8g3NykvvjP87+0R+JyzL39kq7u+Xd3ePT09Ti4p9q2qOPP/6/tVrzl09PTj6Yn5+Zmrq8upqNxf79P//TVeyp8fFbMzNn9frUxMTl5eX7y8sQW/pjk5Mfzc/zl3dmZ3/no4/4ywfz8z/69FO+ohhj6sKCeu9eWEsHgMhCdA4A0DHPjA778LD26pXHhw8OPL+Bx4gdmZmampqYeHt+7n+WB/PzYhQ4G4s9/sEPKOz75vXr83qdMfZxIvFfQjPzz1++nI3FZmMxeTlXHz2iP5R4XF1YoL8pjKa/PfO86SKksrfnnJ7ah4df/PSnfFJqcVFdWFAXFp4kk+rCAg/oXbKl0uf//M+pxcXiZ5+1+oxzclLe3S3t7tL3bzx7ll5epvJQXjttL+vgwD48/Fq4KPqDH/4wEY/fmp5+d3ExH48/uHOHtqDnqvj93/otxlj98jIeizHGPkoknJOTb8/P6cz6//7jP1qtOsZYbGJibHz8fb3e9Wl48YMPGGNTExOUeEN5OIl4PBGPs+9vCMaYMjfXal0BQKQgOgeAG4t3LnRpzuhgjFGk2PxhMXbsyOIHH7z5fgx9cXnZUVRNzcbiO2KrM3FFYIyxyfFx3r4r/lJ+keAKRl1Si4s8+0JbXKQ/7t66xf/uND2D4uDq1193HYuL7FevjL/92y8PDrb+8A+302k/BbBfvSrs7JSr1a8PD5MLC8aTJ5srK82N0LycdFPCVU5lbo7a7xdu36ZQmF+M0Yr1XKt8C2qLi/u//vXHicRHiQRj7JeOMzM19d1yr7eLc3LypdeFHHfv9u3Tev3q6opdb+XT9+/9rIFW7t+5c+/Oncnx70Zv+yiR+N0f/IBPFTc6QZ9agP5AdA4Ag+eZ0eGcnla//trjw0JGh/imJOKUiMdiC7dvu958c35+Z2aGvzyv11++eeP/O7uLquV5CzxxXGyhF1eF5CpCzCoR462162ArxFZVMca1Dw/FcHP10SNaD2uPH3eXpFHc2cmWy4yx8p//eReRYnl3t7K39+Pnz6kw6eXlzMqK5PO8jZ81pcTQKtUWFymEFX8ObSm+99I28oy8+XahjcKjYdox+BEhbnHxGlJ+W+ODW7duTU8zxuqXl9NTU/WLi/cXF4yxmamp83r91du3/taZjHhPZjYW+ziRcO3Sa9/fRki7B/AJ0TkAdGCAGR0/vHfvthAxU29F6ofH3+Qx9N25OcbYt+/eiR0c2+pFVC3BV6Z4HeI//OJZJWIhnyST/O8+tHSae3s8P0QMQCnuDBiLi5yTk8zf//1PqlXqABokyKOMl8LOzpcHB4l43Fhe3lxZ8XlxIk+J4ZcfklsB/EK0ch3E00vPbU2bmG9f2riel1LigSn2UhCPQfldIH581a+u4tft+oyxjxOJf//lLxljk+Pj8VgsxI4NPLi/e+sWveNqqhd3Zoa0exgliM4BbqBWGR2enQs9Mzra3mSX4FEjE3orfnI9cgWNCkJ/z8Zi9+/cYYy5YuixsbGOlt7nqFpCvAkQVlaJWPLBtj6Ksbj4Q1wNyaEXUtIBNAhrf7/04kXx+fPj09PkwsLmyorx5EkXu4TPlBixS2sr/MjlO48kbYbvLbSr8P3E/1JY6ytD+Y7KW/3P3r+fGB/nLej1y8upiQn+maOTE+f0lB8F8ix8IuZlyd2antYePpy8XhxrCu6Rdg/DC9E5wCB1lNHh2RTddUZHc0RL6AwnxtCMsU8fPGCMffXNN+InE/F47fuL7jSmj05ULSGGMp5ZJfJfLf5Gz6ySaLYI0q+mGNEVi/NBVHoUi7v46QAaUHl3t/TixU+qVcbY+pMn6adPjeXlIF/YXUqMH5Q2w/dDSdqMmPLOhLSZLrZXq2tOn4lVzOuyk2qYTx88oAb7t+fnlHhDxOuEN2dnvzg68tNgf3tmZnJiwv8I+mI7AmuqfJB2DwOE6BzAF5/DRfP3m9/sunOheGLjmqNY8r+Xllzt0G/Pz9+cnc0It6qbS3gjo2o5vkFbtR36TOP2zCoZulY6VyzuZ0DDPuiiA2jAxZWr1cLOzteHh4l4PPPsWfrp07C2Y/CUGD94fosrbcZzZ3alzdDlYii7rlhbeqbZtK1wPMcC4uHyV998w8bG+EDyrnpYrNx81myT4+O3rxPk/Af3rpq5uQ5E2j10DdE53CgdZXR4di7sOqNDjNhErtYX0hy50hPLU8KHPVvQEVVLiC18njGBz6ySVoMD3piTq58BDfsfi7sE7AAahLW/X9jZ4cOlp58+zTx71otNH2JKjB/8AHGlzXimvFMB+LHArzxD3xbiYdtd/1exWmvVB4MOXtcJwtVhxpXj57OHzL3bt+tCNH91dfX67KztXKypNharnebys+GphyEUiM5hMFo9ANx/RkfXnQs9Mzqaa0biqh+JWEs2Xw8gqg6RuHo907i7yCoRb1jfgFXUVvDBxfssxA6gARV3dsTh0tcePw6Y8dJW71Ji/HClzdBR1uu0mU4F7//q2buDtRvCyHXOcrX4dNFgPz87m4jH65eXZ/U6Y4wGifI/PFRzexDS7m8SROfQhv+MjtCHi/af0dGcIEgkpwpE1QPkmVXSxeCAzY+cZCN/EvIzoGGkYnGXHnUADcLncOk90p+UGD8oPHWNFMm8jlZ+kLrSZvpfs4XY/5W1vqsm/12uzkWuBnvXLVyfrU70FCpy9v79bCzGXx785jdtZyeutHvXORRp94OF6PyG6PoB4OKb3XUuDJjRwToMpxBVR1PbBq1ePHISzL092v97Mbh4//WhA2gQ5t5eaXeXMl5ouHRjeXkgTft9TonxwzVSJJOmzdDx7kqbicJ1dS/6v5LuhjoVW8eaz3Tiac7nGTz54YdXQtR3/84dsan+7tzcz1++/PbdOz9lk6fdNwf3NyYzsD8QnffbAIeL7iijY82r+ghyjkdUPXR6MThgkEdOgs8BDYdxf+5zB9AgmodLTy8vD/zqcbApMW3x+r/TB6wyocaOYI3R6/6vrNubga4GO9ftblcLnZ/723PT09QN9+jkhDHGHzTx+uzsvF6fjcVmp6YYY/6Hw2+bdu8KQoaxWgsC0bm34A8AD2u4aM6zKbpVRkeIpwpE1TfMsDxycsRJ8hl6Pbh4/w2wA2gQg814aSs6KTF+hPKA1eisfE/97P8apJySBntXtOOzwV58kNyb8/MPhOK9u7gYHxvjL8/q9Z/96ld+Ctk27d7VZ2zozk1DH50P8AHgHWV0eHYu7Onugqh6dNyAR06OOP8DGt6AWFwUnQ6gQbiGS197/DizsjLoQnmLYEqMHz16wGrEDar/a9dcqQGSkS6Zvwb7RDye/PBD/vKjROLi8pJvcWVubn52lo8gfGd2dnJiwn8/N3na/cD71PYvOvfsXNhRRke4nQs7yujo5xkRUTWwm/7IyREnSUIY4ODi/RfBDqBBOCcnxefPSy9e8IyXzZWV6EeEEU+J8aM/D1iNuCj0fw2l8CykkS4p+H5zdnZxecmuH1Z9fHp6fHrKn1HNGPv5y5dTExN8tHv/mcOdpt13uoP1MDof+7M/63QW/xkdnp0L2dAeYNQXys8nEVXfeK0OnCF95CS4iAd7NAc07Bt1a0uJx6PZATQIa3+/9OJF8fnz49PT6l/91dD9OklKTPR7BXjy+YDV1OKi9dd/PZAS9l/A/q+Vv/iLiIRb4Y502fi7vxNfurIzmptKXRkZfnIxVh89Mv/yL+WfYT2NzrOlEvMxXDQwxsy9vcreHqJqYIxlS6XhfeQktEVxzwjG4s3sV69udv1W3t3t9fjofXOD91se3tmHh5FNSRo4V//XSHWx6I7nSJfhXnw2p0JQpOdnNxv6vHMAAAAAgBtjfNAFAAAAgJ6zbdtxnE4nDeprAUbZJP1nmmbzNEVRNE0zTTOfz2uatr29He6yxYXqui6Z6ipSuMWImrGxsUql0rxCQuc4jmVZqqqqqkrv0DqnlSyf2uuyDQtxLxXXVfNUTtM0RVHkMxLbtm3bZl5HB7vefK6p2Go90tOjMpPJ2La9vb3ddhvxXYL2Itc7tD+Q5i0ur2+71rsTBJOW+UaeICTHr6qqwQ9ty7IKhYLnniaZNKivlevdjkd7mucOJjHA82lPj8E+C74aLcvKZrPMawtKJrUlnzfIN7fUaDQouWV1dXV1dTWZTKZSqdXVVfq3cU38OxSVSiWZTCaTSVpuKpXa2Ng4OjriH2hbpJtqdXW1Wq12MVens2xtbSWTyUQiQau9VqvxtV2r1eRTO13WjVSpVFZXVxOJxOq1ZDLJt13zVHpZqVTkM5JCoZBMJre2tmhDJJNJ19RUKsWn8nmx1Xqku6PSv62trUql0vZj6+vrVDGWSiU+YyqVSiQS6XS6eacqFAr0sbb1rR+SSqYXNbO8zGGdICJ1TpEcv5lMJpRDu1qtJpNJz1kkkwb1tW31aPNF83wqL1Wk9uSuBV+NR0dHjLFUKtX85ZJJbcnnDfLNrfxPdE5/8JMEBRD8c73Y8BRb8JeFQkFcStsigai7NUP7urgVxChBPhWIuObpPNRqKr3kK1AyY61WSyaTPBChI59PLZVK6+vr/OXR0ZFYN2GrDSP/22hjY8P1yXw+z6Nwcac6OjoS9yJ5fetHn6PzhrTMYZ0gonZOkRy/YR3akquyTi/Y+vC1ctGJzhu9P5+OQnTeCGM10qW755dLJrUlnzfIN3v6Lu+8Vqu52tR1XS8Wi+I72WxW0zRN03RdFzPJyuWyqqq6rquqmslkuk4yy2QydPPOf5HakpTNsizDMLRr2Wy2XC7zqaZp6rquaZqqqoZh0L1jXoxsNku36nRd13VdvKHcat5MJqPrOi0xk8kUi0X6gDgvfab5C12/hb6BbqPwxVmWpQv8bwXDMCzLanU7Rj4VXGibtrrzxRhbW1vzTGJxzeg4jqqqlLrAGFMUpVAo8A/n83nxQFAUZXt7O5/P83ew1Vy6PtiZ9KiUVwXyilE8nDu6H51Op0ulkvhOqVQyDKP5k5QO4fo54u9y1T8SfiqZVicI1ptzRPAThJ8f1arkfC7LsugzqqrS0v1U9XKS4zeUQ5tXLB1NGtTXtjrxca12PPlxLa8TOL5j+Cxtj86nPs/yXQRpkj1Z/M7mebPZLB1x2WyWfzmtQ/6dvGajD/dtNa6urrbKF5JMaks+b5Bv9uaK1ltdzDHG8vk8/V0oFHjTXaFQEO82lkol/1cPrnYRmt31jqRIbUnKRm2T/FZ1rVajPAF6Wa1WxcZIusHqyrrhH3Y1ZErmZYxRE9fGxgbNUq1WxXnlv1e8Odi8nru+1t/a2qKf37x0+VQgktbKRqORSqV4KotkRspUEafSDtmcTVGtVj23NW96x1ZzCXiwE0nF6FkVyCtG19SNjY1kMul/G4klrFQqYp3J9zd6X6xefNa3EvJ2I88TRKP354iAu7ekMG1LLubbHB0d8TXgs6r3JDl+R/PQlp/4Wu148uNaXic0rvcKqmx5FllbvT6fyg+cIEFaqz250WiIJczn8+Kkra2tRCLBb9zRT6PVValUNjY2XIvwmcMTfDWur6+L5fQ5qS35vEG+2ZPfMVtSqRS/ZhWvn/L5/Pb2Nr8ypovRTlu4uSBX2M0kZSsUCrlcjl/oqKqay+XW1tb4jMVikbdx6rqey+XEH5VKpfhFoWEY4jWcfN5MJkM/c3NzkzFGvRz8/yLTNKkBwDCMEPt/0IVvqy+UTwXbtrPXqB1U3I1t26YarXlG3sCgKEqlUnG1BFiWtbS0VCgUqPmHt+s4juN5ge5qksdW4wIe7HKtqgJ5xeiaWiwWO6r6DMPgX1UoFKgyIXx/y+fzpmmqqiqpYUKsb1udIFi0zxFyfkpOzYeKoiiKIjbrBqzqJcfvCB7akhOfJDKRHNfyOoEvNJPJbG9ve96YkhjU+TRgkNZqT6ZmddM0TdPUNI3SLDnDMGhXp59WLpfpNq+u67zDOmOM6iLP+8atBFmN5XK5+R5L20ltyecN8s2eJn1+rlWd6DhO8767tLTUXWla3YTtjqRslmW5DkXxk6Zpum5yqaqaz+f5qnetjePjY//zds00zUKhUKlUHMdxHEesXILL5XJ0N7aLqSNOURS+LzVXFjTqEWOM7wCO49D+wydtb2+7chVIJpOhis+2bX6LttUt8uY3sdVIwINdrlVVIK8YxbQl4v+eL2Nsc3OTkmpouDrxnMd3KkL3nVuFwiHWt5KgOcrnCDk/JXftWiGSHL8jdWjLT3ytdjz5cS2vEwh9sqOAkhvI+TRgkOa5J1P+j6qqtM4pwUayULE5IJfL8SRMVzamTyMelviNzluhq6Ww2jNKpVK4TcKtyiZvUtI0zbZt8bBs1WAZ7rwSVFq+cihco2bXgN9MFEXJ5XKUQNbp1BGnKIqf0Iq2nWmalUrFtZNns1lFUTKZDD+dUK6q2LRDeZMUnVObhLgtyuVycyWFrUZ6d7DLFyqpGJtDTNM0/cd5dLK0LKtUKqWlT7bLZDJijwWXcOvbVqJ8jpALt+Sdkhy/o3Nod33ikx/X8jqBWJZVLBZ1XTdNs9N9IFLn0yC7MZ2V+Hqj85f4geZOd+L9CrqDx1qPGiwXqdXYf0GfRpROp12NTHQHpIuvorUcYnuwpGzpdDqfz7t6kPAPb25uijM6juO6fSwRZF4JGpaBv6QjTSy/oijicdJFCxOFd626LsmnQkB0I5IfOLVaTdzcjuNQfx16STdbxQ5q2Ww2l8s1fy22GuvlwS5fqKRiNAxDnFosFjs9YNPpdKFQoJvvko/RZZ7npC7q2+4qmSifI5j0R4VY8u5Ijt8RObTbnvhakR/X8jqBo069YmXrXy/Op/0/AMVbc7QOXR+wLItn0VB1JJ6JqPmcBrz3s7hmwxiWWJZFPfK72G34vIwJvULz+bw4dizvCUEdIxKJBHUyODo64qPq0gdowF1K1U+lUuvr636GT3KNZesaIkdeJP8kZaPxpDc2NmiSa/RfPp70xsYG7+jQuB5fk6+NxvUIxK5hv5rn3djYYIxRPwlaInVJSSQSNC8NFib+XnGI5Xw+n0qlxK/lXTEIDcnX0VYQNwH/dTQgd9up0Gga0dy1f3qOd05bUJwk7lqJRIK6lfCtTH+IA1cT1z4mDrKOrdasu4O9IT0q21YF8oqRT11fX+cFc21lOVed6TmIPv+lfurbtjwrmbYniLaropW2ZQ5+gmj1o9qWfGtri34m3yv40ttW9T5/Mr3Dj9/RPLQlJ762O57kuG5I6wT6Hr4UWu1Btl3bqT612leDBGmSPbkhPHaDn4/EVUFfyCvJjY2N5n6fq6urHVU1NyAsodsLiUSiizH++byNRmOs0WiEcrlAl2L8CXaRIimbZJLnsxh9CjJv11/Lp0ZzK0Cn2u5FdMt1qB+O2E89Oti7XijPUBqiO7NBKpnIniPa/qjIlnxEdH14+qlC2VBt2f4fgLREzxMNNcnL28WpJ8ywrN6wfK8JvNt5Q4vOAQAAAGAUtI3Oi8VirVYbqfGFQhS0VygAAAAAjI5sNvv5558zxj7//HPXA9Fs29Y0TRzLDgF6F9B2DgAAAAAQFUHHbAEAAAAAgLAgOgcAAAAAiArknQNEgv3qVblaLb14MTM19aNPPkk/fao9fDjoQgEAAEC/Ie8cYJCck5Py7m5pd/eLn/6UMfYnv/d7E+Pj//Kznx2fniYXFownTzZXVtR79wZdTAAAAOgTROcAA0BBeWVv7yfVKmMstbiYfvo08+yZMjdHHyju7LimGk+eIEwHAAC48RCdA/QVBeU/fv6c+Qi7PYN4hOkAAAA3GKJzgH6gILu8u9tdyoorAWb9yZO1x4+N5WXe1g4AAAA3A6JzgB6y9vdLL16Uq9WvDw8pKA/Y3ZN3Hv3y4IAhTAcAALhxEJ0DhE8MyhPxuLG8TDF0iItoDtPTT5+GuwgAAADoP0TnAKERI+YeBeWeCy3s7PT0SgAAAAD6BtE5QFDOyUnx+fOBZ5v0ocEeAAAAeg3ROUCXXAOqrD56lF5ejkIKOIXpxefPeQ9UPNsIAABgWCA6B+jYsAxG3jxQDMJ0AACAiEN0DuCXK9jdXFmJZlDeLOB4jgAAANA3iM4B2rD29ws7OzejBXpYWv0BAABGFqJzAG/NnSw3V1aGNCh3wSNIAQAAIgvROcD3jNQAhXgEKQAAQNQgOgdgbOQf7oNHkAIAAEQEonMYaWg8dhnxqxQAAICBQ3QOo8gz8Trz7NnIBuXNRirDBwAAIDoQncNooaD8x8+fM/SG9Ke5d2x6eVl//HjQ5QIAALiZEJ3DaDH+5m+sgwMM+N0F/ghSY3m5+Nlngy4OAADAzYToHEaLc3KC9JWAsA4BAAB6B9E5AAAAAEBUjA+6AAAAAAAA8J3JQRcAbizTNMWXmqYpiuJ/3kKh4DgOY2xtbS2bzfqf2h+O41iWpaqqqqq8VIwxRVE0TZNPZd9fOeLH/LNt27Ztxpiu64F/DTNNM5/Pa5q2vb0d/Nu4ga8lAACAoYPMFugJ27YzmQxjzHEcx3FUVd3c3DQMw+e8hmGUy2VVVW3bdhyHYjU/U/smm82Wy2XHcWzbVhSFfi/9XS6XC4WCZKpt2/l83rIsXnLbtsvlsv8fUiwW8/k8rc9yuUzfEPxH6bruuqYKaLBrCQAAYCg1AHqpUqlsbW2FOEsXX9gjW1tbyWRSLMzW1lalUvEztdForK6u8r+r1WoymfS53Fqtlkwmj46O6OXR0VFYB7JYpLAMai0BAAAMKeSdQ79Rs7eu66qqZjIZSlAhtm3ruk4Nrvo1P1Mty6KXlmXR96uqWiwW2y6Uz8gzOrLZrGu5EoZhWJbVqr1ZPlWkaZqqqj7breleBE8TUhSlUCjwqdlsVtM0z9XLGLMsyzAM7RqtTPEDNDt9g2tevg41TctkMv4TigaylgAAAIYUonPoq2KxWKlUKBqzbXttbU1Md6HYa3t72zAM85qfqZqm0d+ZTIa+37IsHlxKFqppWi6XU1WVx5oUpouRfdtfJIlT5VM5yu7wmbNBGdvZbNayLHqHkojI2toa/6VLS0viD6GkoFwuR+unXC6bplmtVvkHvvjii7t379LUdDotfi1jLJvN0jq3LIuW4qe0pP9rCQAAYEghOoeQyROg8/n89vY2b/eldlz/oXBbuq4Xi0VFURRF4QGffKG6rvMelowx0zQ76oBIzcmtOlNKptq2nb1G3TH995q1LGtpaalQKFBLttj+TTcQKIzWNI3yXkihUMjlcjy6VVU1l8utra3xD6RSKb7SmtvdGWMU9DPGDMPoqP/oQNYSAADAMMKYLRCyTCazvb3dqoHTcZzmvqFLS0thLV2MNf0vNJfL5fN5itf5H/7lcjld11v1eW01VVEUXtouRkrJZDLUtk0JP5T1QYkrqqryMU/EFB1q8xa/xFUqeeBLQ+VUKhXq6SsG+n4MZC0BAAAMHUTnED6xzdXV/qqqarlc7nPzZ9uF6rqez+cpN6aLkfsURcnlctls1nPGVlMVReluMMRisUip4fRSVVXDMGzbpkRzcVQT0zQrlQqfUVXV5uZwn2hGHh/TJYGu6/43ZZ/XEgAAwJBCZguETFEUMSO5VCqJ7bXpdLp58PJe9/Pzs1BqPi8UCt010FKjb6tUbPnUTtVqNbEbqOM41F+TXXcY5e+LH2OMpdPpfD4vBuimafrs3EkrFXXEhAAAATpJREFUh7+koLzTWL+fawkAAGBIoe0cQlYsFg3DqFQqlA+9ubkptn3yBGJ6k5K8eSYJPROHEicoShOfjyOZSl0kLcvKZrMUOIrDq8sXSqj5XBwLRc40Tcr25uONiPk8bafSSN5UHv8jwYv4sCqmaVLHVsYYZZsYhkGrSNd1KgatJV3XNzc3aUZFUSgxnSbRqqPEGBqDnIZS4SOg3717t1Qq8YValsUXGuW1BAAAMHTwNCLoCepnKXk+KEVjHT1ANDj5Qmngl2HpdMgvUVyJH/Q+f9ymp65XfquFAgAAQFgQnQMwxlixWKzVauh3CAAAAIOFzBYYadTAf3x8zN9BgA4AAAADhLZzAAAAAICowJgtAAAAAABRgegcAAAAACAqEJ0DAAAAAEQFonMAAAAAgKhAdA4AAAAAEBX/DYGHfJ75RAzPAAAAAElFTkSuQmCC",
      "text/plain": [
       "Tree('S', [('The', 'DT'), ('economist', 'NN'), Tree('PERSON', [('Jeffrey', 'NNP'), ('Sachs', 'NNP')]), ('coined', 'VBD'), ('the', 'DT'), ('term', 'NN'), ('``', '``'), ('shock', 'NN'), ('therapy', 'NN'), (\"''\", \"''\"), ('.', '.')])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def tagger_nltk(sentence):\n",
    "    return ne_chunk(pos_tag(tokenize(sentence)))\n",
    "\n",
    "tagger_nltk('The economist Jeffrey Sachs coined the term \"shock therapy\".')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more terminal-friendly tree-view using parenthesis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  The/DT\n",
      "  economist/NN\n",
      "  (PERSON Jeffrey/NNP Sachs/NNP)\n",
      "  coined/VBD\n",
      "  the/DT\n",
      "  term/NN\n",
      "  ``/``\n",
      "  shock/NN\n",
      "  therapy/NN\n",
      "  ''/''\n",
      "  ./.)\n"
     ]
    }
   ],
   "source": [
    "tagger_nltk('The economist Jeffrey Sachs coined the term \"shock therapy\".').pprint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  The/DT\n",
      "  (ORGANIZATION Google/NNP Nexus/NNP)\n",
      "  challenges/VBZ\n",
      "  the/DT\n",
      "  (ORGANIZATION iPhone/NN)\n",
      "  's/POS\n",
      "  dominance/NN\n",
      "  ./.)\n"
     ]
    }
   ],
   "source": [
    "tagger_nltk(\"The Google Nexus challenges the iPhone's dominance.\").pprint()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note some first errors: \"Google Nexus\" and \"iPhone\" are not organizations. In general, in my personal opinion, NER entity taggers are pretty good at detecting named entities - but once you want to separate out entity types at some high level of precision (90, 95, 99, ... % -- not to mention recall issues) on arbitrary input texts, you will start to see that even the best NER tools (say, Bi-LSTM neural networks with a final CRF layer) around today still make silly mistakes a human would not.\n",
    "\n",
    "In general, my advice is: Use NER to detect all potential named entity mention (of all types - organization, persons, locations, etc.) and then work out your own expert system for the entities you need, instead of hoping that your tagger fully learns the differences between the types. Or, if you are just in need of one entity type, you can train or use an existing model just for that entity types. But even then, I strongly advise to use post-processing of the entity detections with an expert dictionary of all valid names. At the very least, you should verify their plausiblity, like checking persons against a vast collection of names, or checking if that chemical fomula you detected as chemical would even just make sense.\n",
    "\n",
    "As to NLTK's \"default\", on-board tagging functionality via `ne_chunk( pos_tag( word_tokenize( sentence )))`, that certainly provides good-enough out-of-the-box results to play around with and make experiments/prototypes and blog posts, but if you need to provide high-quality results for some commericial or scientific application, you will very quickly be \"wanting more\".\n",
    "\n",
    "Therefore, here we will take a look at the [Stanford NER Tagger](http://nlp.stanford.edu/software/CRF-NER.shtml) and its [PoS Tagger](http://nlp.stanford.edu/software/tagger.shtml) with ready-made, trained models. Second, we will look at [SpaCy](https://spacy.io/), a  high-throughput NLP tool for Python that is similarly useful and performant as the Stanford Parser. The biggest difference between the two is maybe support for Chinese, Arabic, and French. To date (2017ish), SpaCy \"only\" supports English, German, and Spanish - but with the promise of more coming. Another significant difference are licenses: CoreNLP comes with the GPL, unless you pay Stanford for a closed-source license, while SpaCy operates under the Apache 2 license.\n",
    "\n",
    "## Stanford PoS and NER Tagging with NLTK+CoreNLP\n",
    "\n",
    "We can quickly access the two Stanford taggers via an API provided from NLTK. Note that this might not necessarily be the way you want to interact with the Stanford (Java) tools in a production setting. However, it is a quick and easy way to get a research prototype up and running.\n",
    "\n",
    "Let's ensure we have everything set up as needed: The JARs and models in a local subdirectory called `stanford`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "english-bidirectional-distsim.tagger\r\n",
      "english-bidirectional-distsim.tagger.props\r\n",
      "english.all.3class.distsim.crf.ser.gz\r\n",
      "english.all.3class.distsim.prop\r\n",
      "stanford-ner.jar\r\n",
      "stanford-postagger.jar\r\n"
     ]
    }
   ],
   "source": [
    "!ls stanford/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the two environment variables set to the **full path** of the `stanford-postagger.jar`, the `stanford-ner.jar`, and the model directory (itself):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLASSPATH = /Users/fnl/Documents/ASDM Course - Text Mining/github/new_notebooks/stanford/stanford-postagger.jar:/Users/fnl/Documents/ASDM Course - Text Mining/github/new_notebooks/stanford/stanford-ner.jar\n",
      "STANFORD_MODELS =  /Users/fnl/Documents/ASDM Course - Text Mining/github/new_notebooks/stanford\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "print('CLASSPATH =', os.environ['CLASSPATH'])\n",
    "print('STANFORD_MODELS = ', os.environ['STANFORD_MODELS'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More info ambout the generic Stanford NLTK API can be found in the base class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class StanfordTagger in module nltk.tag.stanford:\n",
      "\n",
      "class StanfordTagger(nltk.tag.api.TaggerI)\n",
      " |  An interface to Stanford taggers. Subclasses must define:\n",
      " |  \n",
      " |  - ``_cmd`` property: A property that returns the command that will be\n",
      " |    executed.\n",
      " |  - ``_SEPARATOR``: Class constant that represents that character that\n",
      " |    is used to separate the tokens from their tags.\n",
      " |  - ``_JAR`` file: Class constant that represents the jar file name.\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      StanfordTagger\n",
      " |      nltk.tag.api.TaggerI\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, model_filename, path_to_jar=None, encoding='utf8', verbose=False, java_options='-mx1000m')\n",
      " |      Initialize self.  See help(type(self)) for accurate signature.\n",
      " |  \n",
      " |  parse_output(self, text, sentences=None)\n",
      " |  \n",
      " |  tag(self, tokens)\n",
      " |      Determine the most appropriate tag sequence for the given\n",
      " |      token sequence, and return a corresponding list of tagged\n",
      " |      tokens.  A tagged token is encoded as a tuple ``(token, tag)``.\n",
      " |      \n",
      " |      :rtype: list(tuple(str, str))\n",
      " |  \n",
      " |  tag_sents(self, sentences)\n",
      " |      Apply ``self.tag()`` to each element of *sentences*.  I.e.:\n",
      " |      \n",
      " |          return [self.tag(sent) for sent in sentences]\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes defined here:\n",
      " |  \n",
      " |  __abstractmethods__ = frozenset({'_cmd'})\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from nltk.tag.api.TaggerI:\n",
      " |  \n",
      " |  evaluate(self, gold)\n",
      " |      Score the accuracy of the tagger against the gold standard.\n",
      " |      Strip the tags from the gold standard text, retag it using\n",
      " |      the tagger, then compute the accuracy score.\n",
      " |      \n",
      " |      :type gold: list(list(tuple(str, str)))\n",
      " |      :param gold: The list of tagged sentences to score the tagger on.\n",
      " |      :rtype: float\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from nltk.tag.api.TaggerI:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from nltk.tag.stanford import StanfordTagger\n",
    "help(StanfordTagger)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, initialize the two taggers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.tag.stanford import StanfordNERTagger\n",
    "from nltk.tag.stanford import StanfordPOSTagger\n",
    "\n",
    "ner = StanfordNERTagger('english.all.3class.distsim.crf.ser.gz')\n",
    "pos = StanfordPOSTagger('english-bidirectional-distsim.tagger')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, let's get us word tokenization and some stemming support from NLTK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "st = EnglishStemmer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the NLTK stemmer and tokenizer, we can now set up a nice tagging pipeline: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tagger_stanford(sentence):\n",
    "    tokens = tokenize(sentence)\n",
    "    stems = map(st.stem, tokens)\n",
    "    _, tags = zip(*pos.tag(tokens))\n",
    "    _, entities = zip(*ner.tag(tokens))\n",
    "    return list(zip(tokens, stems, tags, entities))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pretty(sentence):\n",
    "    print(\"{: >15s} {: <15s} {: <5s} {}\\n\".format(\"Token\", \"Stem\", \"PoS\", \"NER\"))\n",
    "    for word in sentence:\n",
    "        print(\"{: >15s} {: <15s} {: <5s} {}\".format(*word))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Stem            PoS   NER\n",
      "\n",
      "            The the             DT    O\n",
      "      economist economist       NN    O\n",
      "        Jeffrey jeffrey         NNP   PERSON\n",
      "          Sachs sach            NNP   PERSON\n",
      "         coined coin            VBD   O\n",
      "            the the             DT    O\n",
      "           term term            NN    O\n",
      "             `` ``              ``    O\n",
      "          shock shock           NN    O\n",
      "        therapy therapi         NN    O\n",
      "             '' ''              ''    O\n",
      "              . .               .     O\n"
     ]
    }
   ],
   "source": [
    "pretty(tagger_stanford('The economist Jeffrey Sachs coined the term \"shock therapy\".'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Stem            PoS   NER\n",
      "\n",
      "            The the             DT    O\n",
      "         Google googl           NNP   O\n",
      "          Nexus nexus           NNP   O\n",
      "     challenges challeng        VBZ   O\n",
      "            the the             DT    O\n",
      "         iPhone iphon           NNP   O\n",
      "             's 's              POS   O\n",
      "      dominance domin           NN    O\n",
      "              . .               .     O\n"
     ]
    }
   ],
   "source": [
    "pretty(tagger_stanford(\"The Google Nexus challenges the iPhone's dominance.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try some real-life sentence with the other entities this model can do: location and organization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Stem            PoS   NER\n",
      "\n",
      "          India india           NNP   LOCATION\n",
      "           will will            MD    O\n",
      "             be be              VB    O\n",
      "     purchasing purchas         VBG   O\n",
      "       Predator predat          NNP   O\n",
      "         drones drone           NNS   O\n",
      "              , ,               ,     O\n",
      "              a a               DT    O\n",
      "         source sourc           NN    O\n",
      "             at at              IN    O\n",
      "       Lockheed lockhe          NNP   ORGANIZATION\n",
      "         Martin martin          NNP   ORGANIZATION\n",
      "       confirms confirm         VBZ   O\n",
      "              . .               .     O\n"
     ]
    }
   ],
   "source": [
    "pretty(tagger_stanford(\"India will be purchasing Predator drones, a source at Lockheed Martin confirms.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A little real-life trick: We replace the single-quote ASCII \"apostrophe\" with the proper Unicode apostrophe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Stem            PoS   NER\n",
      "\n",
      "            The the             DT    O\n",
      "         Google googl           NNP   O\n",
      "          Nexus nexus           NNP   O\n",
      "     challenges challeng        VBZ   O\n",
      "            the the             DT    O\n",
      "         iPhone iphon           NNP   O\n",
      "              ‘ ‘               CD    O\n",
      "              s s               NN    O\n",
      "      dominance domin           NN    O\n",
      "              . .               .     O\n"
     ]
    }
   ],
   "source": [
    "pretty(tagger_stanford('The Google Nexus challenges the iPhone‘s dominance.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Stanford's taggers correctly noticed that Google Nexus and iPhone are not person, location, or organization entities (the three entities the chosen model tags). But note the problems with non-ASCII character (the apostrophe here) (which is due to the **NLTK** tokenizer, not Stanford!); You will have to rigorously santize input if you do NLP with arbitrary text, and heavily test your setup for all [ir]regularities in the Unicode world.\n",
    "\n",
    "## PoS + NER with SpaCy\n",
    "\n",
    "Python has a great neural parser of its own, [SpaCy](https://spacy.io/). It can easily be [installed](https://spacy.io/docs/usage/) from both Conda and PIP3. And don't forget to [download, install and test the English language model](https://spacy.io/docs/usage/models) with:\n",
    "\n",
    "```bash\n",
    "python -m spacy download en\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagger_spacy = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def show(doc):\n",
    "    print(\"{: >15s} {: <15s} {}\\n\".format(\"Token\", \"Lemma\", \"PoS\"))\n",
    "    for token in doc:\n",
    "        print(\"{: >15s} {: <15s} {}:{}\".format(\n",
    "            str(token), token.lemma_, token.pos_, token.tag_))\n",
    "    if doc.ents:\n",
    "        print(\"\\nTYPED PROPER NOUNS\")\n",
    "        for entity in doc.ents:\n",
    "            print('\"{}\" -> {}'.format(entity, entity.label_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Lemma           PoS\n",
      "\n",
      "            The the             DET:DT\n",
      "      economist economist       NOUN:NN\n",
      "        Jeffrey jeffrey         PROPN:NNP\n",
      "          Sachs sachs           PROPN:NNP\n",
      "         coined coin            VERB:VBD\n",
      "            the the             DET:DT\n",
      "           term term            NOUN:NN\n",
      "              \" \"               PUNCT:``\n",
      "          shock shock           NOUN:NN\n",
      "        therapy therapy         NOUN:NN\n",
      "              \" \"               PUNCT:''\n",
      "              . .               PUNCT:.\n",
      "\n",
      "TYPED PROPER NOUNS\n",
      "\"Jeffrey Sachs\" -> PERSON\n"
     ]
    }
   ],
   "source": [
    "show(tagger_spacy('The economist Jeffrey Sachs coined the term \"shock therapy\".'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Lemma           PoS\n",
      "\n",
      "            The the             DET:DT\n",
      "         Google google          PROPN:NNP\n",
      "          Nexus nexus           PROPN:NNP\n",
      "     challenges challenge       VERB:VBZ\n",
      "            the the             DET:DT\n",
      "         iPhone iphone          PROPN:NNP\n",
      "             's 's              PART:POS\n",
      "      dominance dominance       NOUN:NN\n",
      "              . .               PUNCT:.\n",
      "\n",
      "TYPED PROPER NOUNS\n",
      "\"The Google Nexus\" -> ORG\n"
     ]
    }
   ],
   "source": [
    "show(tagger_spacy(\"The Google Nexus challenges the iPhone's dominance.\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Lemma           PoS\n",
      "\n",
      "          India india           PROPN:NNP\n",
      "           will will            VERB:MD\n",
      "             be be              VERB:VB\n",
      "     purchasing purchase        VERB:VBG\n",
      "       Predator predator        PROPN:NNP\n",
      "         drones drone           NOUN:NNS\n",
      "              , ,               PUNCT:,\n",
      "              a a               DET:DT\n",
      "         source source          NOUN:NN\n",
      "             at at              ADP:IN\n",
      "       Lockheed lockheed        PROPN:NNP\n",
      "         Martin martin          PROPN:NNP\n",
      "       confirms confirm         VERB:VBZ\n",
      "              . .               PUNCT:.\n",
      "\n",
      "TYPED PROPER NOUNS\n",
      "\"India\" -> GPE\n",
      "\"Lockheed\" -> ORG\n",
      "\"Martin\" -> PERSON\n"
     ]
    }
   ],
   "source": [
    "show(tagger_spacy(\"India will be purchasing Predator drones, a source at Lockheed Martin confirms.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SpaCy's PoS tagging and chunking abilities are quite good, but it its NER **type** support isn't very strong. But do note it correctly recognized all **proper nouns**, so it did find all named entities - it just isn't very good at assigning types to them. But as said, as far as type-specific NER solutions go, you are probably better of training your own tagger. Also note that SpaCy, just like NLTK, isn't very good at handling non-ASCII characters, either:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Lemma           PoS\n",
      "\n",
      "            The the             DET:DT\n",
      "         Google google          PROPN:NNP\n",
      "          Nexus nexus           PROPN:NNP\n",
      "     challenges challenge       VERB:VBZ\n",
      "            the the             DET:DT\n",
      "       iPhone‘s iphone‘s        ADJ:JJ\n",
      "      dominance dominance       NOUN:NN\n",
      "              . .               PUNCT:.\n",
      "\n",
      "TYPED PROPER NOUNS\n",
      "\"The Google Nexus\" -> ORG\n"
     ]
    }
   ],
   "source": [
    "show(tagger_spacy(\"The Google Nexus challenges the iPhone‘s dominance.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "Overall, between CoreNLP and SpaCy, you will find little difference except for language support and licensing, and CoreNLP's superior (and probably world-leading) NER *type* detection - any of which, of course, can be mission ciritical...\n",
    "\n",
    "Another mission-cricial issue is that most NLP systems make extensive use of letter case information; Input that is all lower- or upper-case can often have devastating effects on the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Token Lemma           PoS\n",
      "\n",
      "            THE the             DET:DT\n",
      "         GOOGLE google          PROPN:NNP\n",
      "          NEXUS nexus           PROPN:NNP\n",
      "     CHALLENGES challenges      PROPN:NNP\n",
      "            THE the             DET:DT\n",
      "        IPHONES iphones         PROPN:NNP\n",
      "             'S 's              PART:POS\n",
      "      DOMINANCE dominance       NOUN:NNS\n",
      "              . .               PUNCT:.\n",
      "\n",
      "TYPED PROPER NOUNS\n",
      "\"IPHONES\" -> ORG\n"
     ]
    }
   ],
   "source": [
    "show(tagger_spacy(\"THE GOOGLE NEXUS CHALLENGES THE IPHONES'S DOMINANCE.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    "For professional information extraction with NER and entity linking, the tools here will be insufficient and you will probably need to learn to train your own sequence taggers and expert systems that then match and **disambiguate** all found named entities against your reference data, possibly even providing special parser facilities for detecting entities that your taggers will miss. To give you the right idea: We've just only looked at the entry to the IE/NER rabbit hole...\n",
    "\n",
    "## Aside: noun-phrase chunking with SpaCy\n",
    "\n",
    "Finding all noun phrases can be extremely useful, e.g., to detect keywords or all potential named entity mentions.\n",
    "Here's how to do that with SpaCy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Google Nexus\n",
      "the iPhone‘s dominance\n"
     ]
    }
   ],
   "source": [
    "for chunk in tagger_spacy(\"The Google Nexus challenges the iPhone‘s dominance.\").noun_chunks:\n",
    "    print(chunk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "India\n",
      "Predator drones\n",
      "a source\n",
      "Lockheed Martin\n"
     ]
    }
   ],
   "source": [
    "for chunk in tagger_spacy(\"India will be purchasing Predator drones, a source at Lockheed Martin confirms.\").noun_chunks:\n",
    "    print(chunk)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}