12_Document_embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Representation learning with Doc2Vec\n",
    "\n",
    "Practical course material for the ASDM Class 09 (Text Mining) by Florian Leitner.\n",
    "\n",
    "© 2017 Florian Leitner. All rights reserved.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In addition to word embeddings we just saw, it is possible to aggregate the word embeddings and combine them with a document-specific inference step to produce [Le and Mikolov (2014)](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)'s approach for producing document embeddings, called *paragraph vectors*, aka. \"Doc2Vec\" (in analogy to \"Word2Vec\").\n",
    "\n",
    "The only requirement for this notebook is `gensim` and NTLK for the [Reuters-21578 corpus](https://archive.ics.uci.edu/ml/datasets/Reuters-21578+Text+Categorization+Collection).\n",
    "And we will be following a heavily modified version of `gensim`'s [Doc2Vec tutorial](https://nbviewer.jupyter.org/github/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-lee.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "import gensim\n",
    "import os\n",
    "import nltk\n",
    "\n",
    "# show text as HTML\n",
    "from IPython.display import HTML\n",
    "%pylab inline --no-import-all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corpus preparation: Reuters-21578\n",
    "\n",
    "We will be using the same corpus we had in use for clustering, this time including the test data, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import reuters "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the above step didn't work, first download the Reuters corpus from the GUI that pops up here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a sample document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "ASIAN EXPORTERS FEAR DAMAGE FROM U.S.-JAPAN RIFT\n",
       "  Mounting trade friction between the\n",
       "  U.S. And Japan has raised fears among many of Asia's exporting\n",
       "  nations that the row could inflict far-reachin..."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(reuters.raw(reuters.fileids()[0])[:200] + \"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The around 12,000 Reuters news articles in this corpus (the \"ApteMod\" corpus version) are assigned to the following categories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all 90 categories:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "acq, alum, barley, bop, carcass, castor-oil, cocoa, coconut, coconut-oil, coffee, copper, copra-cake, corn, cotton, cotton-oil, cpi, cpu, crude, dfl, dlr, dmk, earn, fuel, gas, gnp, gold, grain, groundnut, groundnut-oil, heat, hog, housing, income, instal-debt, interest, ipi, iron-steel, jet, jobs, l-cattle, lead, lei, lin-oil, livestock, lumber, meal-feed, money-fx, money-supply, naphtha, nat-gas, nickel, nkr, nzdlr, oat, oilseed, orange, palladium, palm-oil, palmkernel, pet-chem, platinum, potato, propane, rand, rape-oil, rapeseed, reserves, retail, rice, rubber, rye, ship, silver, sorghum, soy-meal, soy-oil, soybean, strategic-metal, sugar, sun-meal, sun-oil, sunseed, tea, tin, trade, veg-oil, wheat, wpi, yen, zinc"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"all\", len(reuters.categories()), \"categories:\")\n",
    "HTML(\", \".join(reuters.categories()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAEWCAYAAAA3h9P4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de7xVdZ3/8ddbRLyAeIEMTSUVQ01FON7FTC3vaU2Mjjp5KRlK82dmjRZj2HR3HMnRMnIUSkfxVuPojGmgoSjg4S55wQQrb4gXFElE+Pz+WN+jm8Peh3NgX9be+/18PPZjr/1dt89anHM+fL9r7c9SRGBmZpZXG9Q6ADMzs444UZmZWa45UZmZWa45UZmZWa45UZmZWa45UZmZWa45UZmZWa45UZl1kaRTJbVKWirpRUn/J+mQTq4bknapdIz1yufHinGiMusCSRcCo4EfANsAOwA/A06sZVxrI2nDWsdgtq6cqMw6SVJv4LvAuRFxZ0S8HRErIuJ/IuIbaZn9JD0q6Y3U27pa0kZp3qS0qdmpN3Zyaj9e0qy0ziOS9irY52BJMyW9Jek2SeMlfa9g/jmSnpH0mqS7JG1bMC8knStpPjBf0jWSrmh3THdJ+lqJ491D0v1p2y9L+lZq7yFptKQX0mu0pB5p3pmSHm63nfd7SZLGpjjuScc0VdLOpc6PpD6S7k7n5jVJD0ny361mExF++eVXJ17A0cB7wIYdLDMEOADYEOgPPAFcUDA/gF0KPu8DLAL2B7oBZwALgR7ARsBzwP8DugOfA94FvpfWPRxYDAxOy/8HMKndvu4HtgI2AfYDXgA2SPP7AMuAbYocRy/gReDrwMbp8/5p3neBKcCHgL7AI8C/pnlnAg+329b7xwyMBV5NsWwI3ATc0sH5+SFwbTr+7sBQQLX+WfCrui//z8Ss87YGFkfEe6UWiIjpETElIt6LiIXAL4BPdLDN4cAvImJqRKyMiHHAcrJk15bwroqs53YnMK1g3dOA6yNiRkQsBy4BDpTUv2CZH0bEaxHxt4iYBiwBjkjzTgEejIiXi8R1PPBSRFwREe9ExFsRMbVgv9+NiEUR8QpwGfCPHRxje7+JiGnpPN4EDOpg2RVAP2DHdA4eiggXKG0yTlRmnfcq0Kej6z2Sdk1DVS9JepPsWlafDra5I/D1NLT1hqQ3gO2BbdPr+XZ/mP9SML0tWY8LgIhYmmLcrsTyAOOA09P06cCvS8S1PfCnEvNW22+a3rbEssW8VDC9DOjZwbKXA88A90l6VtLFXdiPNQgnKrPOe5Sst3NSB8v8HHgSGBARmwPfAtTB8n8Bvh8RWxS8No2Im8mG3raTVLj+9gXTL5AlOgAkbUbW63u+YJn2vY8bgRMl7Q3sBvy2g7h2KjFvtf2S3VDyQpp+G9i0IKYPl9hGp6Se3NcjYifgM8CFko5Y23rWWJyozDopIpYAlwLXSDpJ0qaSuks6RtJP0mK9gDeBpZIGAl9ut5mXWT0B/BIYIWl/ZTaTdJykXmSJcSVwnqQNJZ1Idm2nzc3AWZIGpZsZfgBMTUOOpY7hr8BjZD2pOyLibyUWvRvoJ+mCdPNEL0n7F+x3pKS+kvqkc3Jjmjcb2CPFtDEwqlQsJax2ftKNJrukZL2E7Hys6uI2rc45UZl1QURcAVwIjAReIet5nMcHPZOLgFOBt8iS0Ph2mxgFjEvDfH8fEa3AOcDVwOtkw1xnpn29S3YDxReBN8iG6u4m69UREb8H/gW4g6z3tTPZdae1GQfsSelhPyLiLeBTwAlkQ3XzgU+m2d8DWoE5wFxgRmojIp4mu9ni92md1e4A7IRRFJwfYEDa1lKyxP2ziHigi9u0OidflzSrH5KmAtdGxA3rsY1DyXpAO/rGBKsH7lGZ5ZikT0j6cBr6OwPYC7h3PbbXnex29+ucpKxe+NvqZvn2MeBWYDPgWeDzEfHiumxI0m5kQ3azgbPKFqFZhXnoz8zMcs1Df2Zmlmse+quAPn36RP/+/WsdhplZXZk+ffriiOjbvt2JqgL69+9Pa2trrcMwM6srkp4r1u6hPzMzyzUnKjMzyzUnKjMzyzUnKjMzyzUnKjMzyzUnKjMzyzUnKjMzyzUnKjMzyzV/4bcC5j6/hP4X31PrMMzMqmrhj46ryHbdozIzs1xzojIzs1xzoipBUrdax2BmZk2QqCSdLmmapFmSfiGpm6SfS2qVNE/SZQXLLpT0Y0kzgGGSHkyfp0l6WtLQGh6KmVlTauhElZ5oejJwcEQMAlYCpwHfjogWssd6f0LSXgWrvRoRgyPilvR5w4jYD7gA+E4VwzczMxr/rr8jgCHAY5IANgEWAX8vaTjZ8fcDdgfmpHXGt9vGnel9OtC/1I7S9oYDdNt8jcepmJnZOmr0RCVgXERc8n6D9FHgfmDfiHhd0lhg44J13m63jeXpfSUdnK+IGAOMAejRb0Csf+hmZgYNPvQHTAA+L+lDAJK2AnYgS0ZLJG0DHFPD+MzMbC0aukcVEX+UNBK4T9IGwArgXGAm8CTwF2ByDUM0M7O1aOhEBRAR41nzutOUEsv2b/f5sILpxXRwjcrMzCqj4RNVLey5XW9aK1RKxMys2TT6NSozM6tzTlRmZpZrHvqrAFdPt2ZUqcrZZu5RmZlZrjlRmZlZrjlRmZlZrjVlopL0W0nTU/X04antaEkzJM2WNCG1bS3pvrTcdZKek9SnttGbmTWXZr2Z4uyIeE3SJmQFa/8b+CVwaEQsSKWWIKuW/nBEfFfSccAXS23QRWnNzCqjKXtUwPmSZpNVqNieLMFMiogFABHxWlruUODG1HYP8HqpDUbEmIhoiYiWbpv2rmjwZmbNpOkSlaTDgCOBAyNib7K6f7NqGpSZmZXUdIkK6A28HhHLJA0EDiB7zMeh6REgFAz9TQJOTW3HAFvWIF4zs6bWjNeo7gVGSHoCeIps+O8VsuG/O1OV9UXAp4DLgJslzQMeAf5cm5DNzJqXIvyMv86StBBoSZXUS2ppaYnW1tbqBGVm1iAkTY+IlvbtzTj0Z2ZmdaQZh/7WWfvnVZmZWeU5UVVAMxSldQFSM6sWD/2ZmVmuOVG1I6mHpN9LmiXp5FrHY2bW7Dz0t6Z9ACJiUK0DMTOzGveoJPWX9KSksZKelnSTpCMlTZY0X9J+krZKRWTnSJoiaa+07ihJ10t6UNKzks4v2O7pkqalXtEvJHWTdLak0QXLnCPpynbxfIisZNK+ad29JT0l6WNp/s2SzqnO2TEzM8jH0N8uwBXAwPQ6FTgEuAj4FtmXbmdGxF7p868K1h0IHAXsB3xHUndJuwEnAwenXtFK4DTgVuAESd3TumcB1xcGEhGLgC8BD0XEoIiYDZwHjJV0CrBlRPyy2EFIGi6pVVLrymVL1u+MmJnZ+/Iw9LcgIuYCpAoQEyIiJM0F+gM7An8HEBET06M3Nk/r3hMRy4HlkhYB2wBHAEPIqqIDbAIsioilkiYCx6eqFN3b9tuRiLhf0jDgGmDvDpYbA4wB6NFvgL9FbWZWJnlIVMsLplcVfF5FFt+KTq67Mi0vYFxEXFJk+evIemVPAjcASDoXaBvOO7b9Cqmk0m7AMrJaf3/t+HDMzKyc8jD0tzYPkQ3dtVU+XxwRb3aw/ATg8+l6E+ka144AETGV7LEepwI3p7Zr0jDfoIh4ocj2vgY8kda5oWDo0MzMqiAPPaq1GQVcL2kOWa/mjI4Wjog/ShoJ3Jd6QyuAc4Hn0iK3AoMiouSzpdqkmyi+BOwXEW9JmgSMJHugopmZVUHTFaWVdDdwZURMqNQ+evQbEP3OGL32BeuYK1OYWbmVKkpbDz2qspC0BTANmF3JJAWw53a9afUfcjOzsmiaRBURbwC71joOMzPrmqZJVNVUz0VpPaRnZnlTD3f9mZlZE3OiakfSZyRdXOs4zMws05RDf5I2jIj3is2LiLuAu6ockpmZldCQiUrSvwCnA68AfwGmA8cDs8jqCN4s6Wmy70RtBLwKnBYRL0s6E2iJiPMkjQXeBFqADwPfjIjbq3w4ZmZNreESlaR9yWoD7g10B2aQJSqAjdru0Ze0JXBAqiv4JeCbwNeLbLIfWXIbSNbTcqIyM6uihktUwMHAf0fEO8A7kv6nYN74gumPAOMl9SPrVS0osb3fRsQq4I+Stim1U0nDgeEA3Tbvuz7xm5lZgWa7meLtgun/AK6OiD2BfwI2LrFOYeFbldpwRIyJiJaIaOm2ae/1j9TMzIDGTFSTyZ47tbGknmTXporpDTyfpjusH2hmZrXTcEN/EfGYpLuAOcDLwFyg2JMMRwG3SXodmAh8tGpBmplZpzVkUVpJPdODEjcFJgHDI2JGtfZfz0VpXZnCzGql2YrSjpG0O9l1p3HVTFLgorRmZuXUkIkqIk6tdQxmZlYejXgzhZmZNZCG7FHVWr1WT/f1KTPLI/eozMws15yoipA0StJFtY7DzMyaKFFJ8jCnmVkdaqg/3pK+AFwEBNkXflcC7wD7AJMlvQksjYh/S8s/DhwfEQslfZusQsUiPqi4jqSdgWuAvsAy4JyIeLKqB2Zm1sQaJlFJ2oPssR0HRcRiSVsB/05WfPagiFgpaVSJdYcApwCDyM5JYcX1McCIiJgvaX/gZ8DhRbbhorRmZhXQMImKLHncFhGLASLiNUmktpVrWXco8JuIWAaQSjCRagUeRFZqqW3ZHsU2EBFjyJIaPfoNaLxyH2ZmNdJIiaqUworp77H6dblSFdPbbAC8ERGDyh6VmZl1SiPdTDERGCZpa4A09NfeQmBwmj+YDwrRTgJOkrSJpF7ACQAR8SawQNKwtI4k7V3RozAzs9U0TI8qIuZJ+j7wB0krgZlFFrsD+IKkecBU4Om07gxJ44HZZDdTPFawzmnAzyWNJHti8C1pOTMzq4KGrJ5eay0tLdHa2lrrMMzM6kqp6umNNPRnZmYNyInKzMxyrWGuUeVJnovSuvCsmdUb96jMzCzXGipRSdpC0lfWcxtnSrq6XDGZmdn6aahEBWwBrJGoXJDWzKx+Ndof8B8BO0uaBawgK0j7OjAQ2FXSb4HtySpS/DSVPULSWcAlwBtk35Fantr7AtcCO6TtXxARk6t3OGZm1miJ6mLg4xExSNJhwD3p84I0/+xUA3AT4DFJdwAbAZcBQ4AlwAN88GXhnwJXRsTDknYAfgfsVmzHLkprZlYZjZao2ptWkKQAzpf02TS9PTAA+DDwYES8ApAqVOyaljkS2L2gIO3mknpGxNL2O3JRWjOzymj0RPV+QdrUwzoSODAilkl6kM4VpT0gIt6pWIRmZtahRruZ4i2gV4l5vYHXU5IaCByQ2qcCn5C0taTuwLCCde4Dvtr2QZKrqJuZVVlD9agi4lVJk9OTe/8GvFww+15ghKQngKeAKWmdF9MDFR8lu5liVsE65wPXSJpDdq4mASMqfiBmZva+hkpUABFxaon25cAxJebdANxQpH0xcHJZAzQzsy5puESVB3tu15tWlyoyMyuLRrtGZWZmDcY9qgrIa1FaF6Q1s3rkHpWZmeWaE5WZmeVa7hOVpEc6scwFkjatcByDJB1byX2Ymdmacp+oIuKgTix2AdClRCWpWxdDGQQ4UZmZVVnuE5Wkpen9MEkPSrpd0pOSblLmfGBb4AFJD6RlPy3pUUkzJN0mqWdqXyjpx5JmAMMk7SzpXknTJT2UKlYgaZikxyXNljRJ0kbAd4GTJc2S5O9WmZlVSb3d9bcPsAfwAjAZODgirpJ0IfDJiFgsqQ8wEjgyIt6W9M/AhWSJBuDViBgMIGkCMCIi5kvaH/gZcDhwKXBURDwvaYuIeFfSpUBLRJxXLDBXTzczq4x6S1TTIuKvAOmZU/2Bh9stcwCwOzA5VT3fiKw8Upvxaf2ewEHAbQXV0Xuk98nAWEm3And2JjBXTzczq4x6S1TLC6ZXUjx+AfdHxD+U2EZbRfUNgDciYo1CsxExIvWwjgOmSxqyHjGbmdl6yP01qk4qrJo+BThY0i4AkjaTtGv7FSLiTWCBpGFpOUnaO03vHBFTI+JS4BWyZ1d1VJndzMwqpFES1RjgXkkPpAcgngncnKqeP0r2KPpiTgO+KGk2MA84MbVfLmluqsL+CNnj6R8ge4iib6YwM6siRfhySrm1tLREa2trrcMwM6srkqZHREv79kbpUZmZWYNyojIzs1yrt7v+6kIlq6e7ArqZNRv3qMzMLNeaOlGlkkp9irR/RtLFtYjJzMxW56G/IiLiLuCuWsdhZmZN1KNKX/y9JxWafbzgu1BfTcVr5xYUpT1T0tVpeqykayW1Snpa0vE1OwgzsybUNIkKOBp4ISL2joiPA/em9sWpSO3PgYtKrNsf2I+spNK1kjZuv4Ck4SmZta5ctqT80ZuZNalmSlRzgU+lx3wMjYi2bNJWdHY6WUIq5taIWBUR84FnKVLpIiLGRERLRLR027R3uWM3M2taTXONKiKeljSY7OGH30uP+IAPCt2WKnIL0L58h8t5mJlVSdP0qCRtCyyLiBuBy4HBXVh9mKQNJO0M7AQ8VYkYzcxsTU3TowL2JCs2uwpYAXwZuL2T6/4ZmAZsTvagxXcqE6KZmbXnorRrIWkscHdEdDapuSitmdk6cFFaMzOrS8009LdOIuLMWsdgZtbMnKgqoFJFaV2Q1syakYf+zMws1xo6UUlaWqbtDJJ0bDm2ZWZmXdPQiaqMBpF9UdjMzKqsrhOVpG9IOj9NXylpYpo+XNJNafr7qRDtFEnbpLa+ku6Q9Fh6HZza95P0qKSZkh6R9DFJGwHfBU6WNKugmK2ZmVVBXScq4CFgaJpuAXpK6p7aJgGbAVMiYu/0+Zy07E+BKyNiX+DvgOtS+5PA0IjYB7gU+EFEvJumx0fEoIgYXywQF6U1M6uMer/rbzowRNLmZDX7ZpAlrKHA+cC7wN0Fy34qTR8J7C6pbTubS+oJ9AbGSRpAVs+ve2cDiYgxwBiAHv0G+FvUZmZlUteJKiJWSFoAnAk8AswBPgnsAjwBrIgPSm8UFp3dADigfSmk9AyqByLis5L6Aw9W+BDMzGwt6n3oD7Lhv4vIhvYeAkYAM6Pj2lD3AV9t+yBpUJrsDTyfps8sWP4toFeZ4jUzsy5olETVD3g0Il4G3kltHTkfaJE0R9IfyZIbwE+AH0qayeq9zQfIhgp9M4WZWZW5KG0F9Og3IPqdMbrs23VlCjNrZKWK0tb1Naq82nO73rQ6qZiZlUUjDP2ZmVkDc4+qAspZlNbDfWbW7NyjMjOzXHOiMjOzXHOiMjOzXHOiKkGSr9+ZmeVAU/wxlvQFsuoVQVZm6V+A64E+wCvAWRHxZ0ljyb4wvA8wWdKbwA7ATul9dERcVf0jMDNrXg2fqCTtAYwEDoqIxZK2AsYB4yJinKSzgauAk9IqH0nLrpQ0ChhIVj+wF/CUpJ9HxIoi+xkODAfotnnfSh+WmVnTaIahv8OB2yJiMUBEvAYcCPxXmv9r4JCC5W+LiJUFn++JiOVp/UXANsV2EhFjIqIlIlq6bdq77AdhZtasmiFRddXb7T4vL5gurMBuZmZV0AyJaiIwTNLWAGno7xHglDT/NNZexNbMzGqk4XsHETFP0veBP0haCcwke8THDZK+QbqZopYxmplZaa6eXgEtLS3R2tpa6zDMzOpKqerpzTD0Z2ZmdcyJyszMcq3hr1HVwrpWT3eldDOzNblHZWZmueYeVRGpIsVS4OPA3RFxe20jMjNrXu5RrQNJ3Wodg5lZs3CiSiR9W9LTkh4GPlZk/kJJP5Y0AxhW/QjNzJqTh/4ASUPIKlUMIjsnM4DpRRZ9NSIGl9iGi9KamVWAe1SZocBvImJZRLwJ3FViufGlNuCitGZmleFE1TXtC9aamVmFOVFlJgEnSdpEUi/ghFoHZGZmGV+jAiJihqTxwGyyZ049VuOQzMwscVHaCnBRWjOzrnNRWjMzq0tOVGZmlmu+RlUBLkprZlY+7lGZmVmuOVEVIekwSXfXOg4zM3OiMjOznFtropLUX9KTksamoq03STpS0mRJ8yXtJ2krSb+VNEfSFEl7pXVHSbpe0oOSnpV0fsF2T5c0TdIsSb+Q1E3S2ZJGFyxzjqQri8T0ibTeLEkzJfVq3wuSdLWkM9P0Qkk/kTQ37XOX1D5W0rWSWtOxHd9uPxukY+xb8PmZts9mZlZ5ne1R7QJcAQxMr1OBQ4CLgG8BlwEzI2Kv9PlXBesOBI4C9gO+I6m7pN2Ak4GDI2IQsBI4DbgVOEFS97TuWcD1ReK5CDg3rTsU+FsnjmFJROwJXA2MLmjvn2I7DrhW0sZtMyJiFXBjig3gSGB2RLzSif2ZmVkZdDZRLYiIuekP9zxgQmTfFJ5L9of+EODXABExEdha0uZp3XsiYnlELCar+rANcAQwBHhM0qz0eaeIWApMBI6XNBDoHhFzi8QzGfj31EPbIiLe68Qx3FzwfmBB+60RsSoi5gPPkiXWQtcDX0jTZwM3FNu4pOGpZ9a6ctmSToRjZmad0dnb05cXTK8q+LwqbWNFJ9ddmZYXMC4iLimy/HVkvbInSUlB0rnAOWn+sRHxI0n3AMcCkyUdBbzH6ol3Y1YXnZhe43NE/EXSy5IOJ+t5nUYRETEGGAPQo98Al/swMyuTct1M8RDpD7ikw4DF6XEZpUwAPi/pQ2mdrSTtCBARU4HtyYYXb05t10TEoPR6QdLOqYf3Y7K6fAOB54DdJfWQtAVZL63QyQXvjxa0D0vXnnYGdgKeKhLvdWRDgLdFxMrOnBAzMyuPcn3hdxRwvaQ5wDLgjI4Wjog/ShoJ3CdpA7Ie2blkyQaya1WDIuL1Epu4QNInyXp084D/i4jlkm4FHgcWADPbrbNlim858A8F7X8GpgGbAyMi4h1J7fd3F1nvruiwn5mZVU4ui9Kmu/eujIgJZdreQqAlXScrbB8L3B0Rt69l/ZYUz9DO7K9HvwHR74zRa1+wHVemMLNmVqooba5KKKUhu2lkd9aVJUmtL0kXA1+mxLWpYvbcrjetTjpmZmWRyx5VvfNjPszMus6P+TAzs7qUq6G/RrEu1dN9fcrMrDj3qMzMLNcaIlFJeiS995f0eAX346rqZmZV1hCJKiIOqnUMZmZWGXWXqCRdKOnx9LogtS0tstweBdXZ50gakNrXqNqe2j8t6VFJMyTdJqlnaj86VY+fAXyuiodqZmbUWaKSNISsovr+wAHAOZL2KbH4COCnqcJ6C/DXUlXbJfUBRgJHRsRgoBW4MFVS/yVwAlkR3Q93EJuL0pqZVUC93fV3CPCbiHgbQNKdZI/5KOZR4NuSPgLcGRHzJRVWbQfYhKyi+wHA7mQFbgE2SusPJKscPz/t70ZgeLGduSitmVll1Fui6rSI+C9JU8meM/W/kv6JElXbJZ0A3B8R/9CufVDVAjYzs6LqauiPrEr7SZI2lbQZ8NnUtgZJOwHPRsRVwH8De1G6avsU4OCCJ/9uJmlXskeN9E+V1WH1YrZmZlYFddWjiogZqZDstNR0XUTMLFLtHODvgX+UtAJ4CfhBRLxWrGp7RExJj62/WVKPtP7IiHha0nDgHknLyJJir4odoJmZrcG1/irAtf7MzLrOtf7MzKwuOVGZmVmu1dU1qnrhorRmZuXjHpWZmeWaE5WZmeWaE9U6aKsPaGZmldcwiSp9SfceSbNTwdqTJS1MdfyQ1CLpwTTdV9L9kuZJuk7ScwXL/VbS9DRveMH2l0q6QtJs4MBaHKOZWTNqmEQFHA28EBF7R8THgXs7WPY7wMSI2AO4HdihYN7ZETGErJDt+ZK2Tu2bAVPT9h9uv0EXpTUzq4xGSlRzgU9J+rGkoRHRUbY4BLgFICLuBV4vmHd+6jVNAbYHBqT2lcAdpTYYEWMioiUiWrpt2nt9jsPMzAo0zO3pqdzRYOBY4HuSJgDv8UEy3nht25B0GHAkcGBELEtDhW3rvRMRK8seuJmZdahhelSStgWWRcSNwOXAYGAh2WM9AP6uYPHJZLUAkfRpYMvU3ht4PSWpgWSP/zAzsxpqmB4VsCdwuaRVZMVmv0z2vKn/lPSvwIMFy15GVoD2H8meO/US8BbZda0Rkp4AniIb/jMzsxpqmEQVEb8Dfldk1q5F2pYAR0XEe5IOBPaNiOVp3jEltt+zPJGamVlXNEyi6qIdgFvToz7eBc4p58b33K43rS6JZGZWFk2ZqNKj5fepdRxmZrZ2TZmoKq0rRWldjNbMrGMNc9efmZk1pqZJVKls0lRJMyUNlTRM0hOSHqh1bGZmVlozDf0dAcyNiC8BSLoXOKdYOSQzM8uPuu9RSfqCpDmpGO2vJfWXNDG1TZC0g6RBwE+AEyXNkvQdsjJK/ynpcknd0vtjab1/Ktj+NwraL6vVcZqZNau67lFJ2gMYCRwUEYslbQWMA8ZFxDhJZwNXRcRJki4FWiLivLTuJ4GLIqI1VUlfEhH7SuoBTJZ0H1mdvwHAfoCAuyQdGhGTqn+0ZmbNqa4TFXA4cFtELAaIiNfSF3g/l+b/mqwntTafBvaS9Pn0uTdZgvp0es1M7T1T+xqJKiW74QDdNu+7TgdjZmZrqvdEVS4CvpqqW3zQKB0F/DAifrG2DUTEGGAMQI9+A6IiUZqZNaF6v0Y1ERjW9syoNPT3CHBKmn8a8FAntvM74MuSuqft7Cpps9R+tqSeqX07SR8q8zGYmVkH6rpHFRHzJH0f+IOklWRDdF8FbpD0DeAV4KxObOo6oD8wQ5LSeidFxH2SdgMezZpZCpwOLCr7wZiZWVGK8ChVufXoNyD6nTG6U8u6MoWZWUbS9Ihoad9e1z2qvHJRWjOz8qn3a1RmZtbgnKjMzCzXnKjMzCzXnKjMzCzXnKjMzCzXnKjMzCzXnKjMzCzXnKjMzCzXnKjMzCzXXEKpAiS9BTxV6zg60AdYXOsgOuD41l/eY3R866dR49sxItZ4TpJLKFXGU8XqVeWFpFbHt+7yHh/kP0bHt36aLT4P/ZmZWa45UZmZWa45UVXGmFoHsBaOb/3kPT7If4yOb/00VXy+mcLMzHLNPSozM8s1JyozM8s1J6oyknS0pKckPSPp4hrGsVDSXEmzJLWmtq0k3S9pfnrfMrVL0lUp5jmSBlcopuslLZL0eEFbl/2DRokAAAY6SURBVGOSdEZafr6kMyoc3yhJz6fzOEvSsQXzLknxPSXpqIL2ivwMSNpe0gOS/ihpnqT/l9pzcQ47iC8X51DSxpKmSZqd4rsstX9U0tS0r/GSNkrtPdLnZ9L8/muLu0LxjZW0oOD8DUrtVf8dSdvuJmmmpLvT5+qcv4jwqwwvoBvwJ2AnYCNgNrB7jWJZCPRp1/YT4OI0fTHw4zR9LPB/gIADgKkViulQYDDw+LrGBGwFPJvet0zTW1YwvlHARUWW3T39+/YAPpr+3btV8mcA6AcMTtO9gKdTHLk4hx3El4tzmM5DzzTdHZiazsutwCmp/Vrgy2n6K8C1afoUYHxHcVcwvrHA54ssX/XfkbT9C4H/Au5On6ty/tyjKp/9gGci4tmIeBe4BTixxjEVOhEYl6bHAScVtP8qMlOALST1K/fOI2IS8Np6xnQUcH9EvBYRrwP3A0dXML5STgRuiYjlEbEAeIbs379iPwMR8WJEzEjTbwFPANuRk3PYQXylVPUcpvOwNH3snl4BHA7cntrbn7+283o7cIQkdRB3peIrpeq/I5I+AhwHXJc+iyqdPyeq8tkO+EvB57/S8S9qJQVwn6Tpkoantm0i4sU0/RKwTZquZdxdjakWsZ6XhlaubxtWq3V8aRhlH7L/defuHLaLD3JyDtOw1SxgEdkf8D8Bb0TEe0X29X4caf4SYOtqxhcRbefv++n8XSmpR/v42sVRyX/f0cA3gVXp89ZU6fw5UTWmQyJiMHAMcK6kQwtnRtYHz9X3EvIYE/BzYGdgEPAicEVtwwFJPYE7gAsi4s3CeXk4h0Xiy805jIiVETEI+AjZ/+IH1iqWYtrHJ+njwCVkce5LNpz3z7WITdLxwKKImF6L/TtRlc/zwPYFnz+S2qouIp5P74uA35D9Ur7cNqSX3helxWsZd1djqmqsEfFy+uOxCvglHwxR1CQ+Sd3JksBNEXFnas7NOSwWX97OYYrpDeAB4ECyIbO2mqeF+3o/jjS/N/BqleM7Og2pRkQsB26gdufvYOAzkhaSDcceDvyUap2/clxg8ysgK/D7LNkFwraLwHvUII7NgF4F04+QjVFfzuoX3X+Spo9j9Yuy0yoYW39Wv1mhSzGR/Y9yAdlF4i3T9FYVjK9fwfTXyMbWAfZg9QvCz5LdBFCxn4F0Ln4FjG7Xnotz2EF8uTiHQF9gizS9CfAQcDxwG6vfDPCVNH0uq98McGtHcVcwvn4F53c08KNa/o6kfRzGBzdTVOX8lS14v96/E+dpsrHvb9cohp3SD8JsYF5bHGTjwxOA+cDv23540w/6NSnmuUBLheK6mWzoZwXZuPQX1yUm4GyyC7DPAGdVOL5fp/3PAe5i9T+6307xPQUcU+mfAeAQsmG9OcCs9Do2L+ewg/hycQ6BvYCZKY7HgUsLfl+mpXNxG9AjtW+cPj+T5u+0trgrFN/EdP4eB27kgzsDq/47UrD9w/ggUVXl/LmEkpmZ5ZqvUZmZWa45UZmZWa45UZmZWa45UZmZWa45UZmZWa45UZnVEUkflnSLpD+lEln/K2nXMm7/MEkHlWt7ZuXgRGVWJ1JRz98AD0bEzhExhKzEzjYdr9klhwFOVJYrTlRm9eOTwIqIuLatISJmAw9LulzS48qeQ3YyvN87urttWUlXSzozTS+UdJmkGWmdgamY7Ajga+nZR0MlDUvbnS1pUhWP1ex9G659ETPLiY8DxYqCfo6s6OveQB/gsU4mlcURMVjSV8ieGfUlSdcCSyPi3wAkzQWOiojnJW1RnsMw6xr3qMzq3yHAzZEVf30Z+ANZte21aStsO52szmExk4Gxks4hq8VnVnVOVGb1Yx4wpAvLv8fqv+Mbt5u/PL2vpMToSkSMAEaSVbyeLmnrLuzfrCycqMzqx0SgR8HDMJG0F/AGcHJ68F5f4FCyQqDPAbtL6pGG7Y7oxD7eInuUfNv2d46IqRFxKfAKqz+iwawqfI3KrE5EREj6LDBa0j8D7wALgQuAnmQV8wP4ZkS8BCDpVrLK2wvIqnOvzf8At0s6Efgq2Y0VA8iqdU9I+zCrKldPNzOzXPPQn5mZ5ZoTlZmZ5ZoTlZmZ5ZoTlZmZ5ZoTlZmZ5ZoTlZmZ5ZoTlZmZ5dr/Bz8caN+y3t8JAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pairs = sorted([\n",
    "    (len(reuters.fileids(cat)), cat)\n",
    "    for cat in reuters.categories()\n",
    "])\n",
    "\n",
    "counts = [cnt for cnt, cat in pairs[-15:]]\n",
    "cats = [cat for cnt, cat in pairs[-15:]]\n",
    "plt.barh(range(len(cats)), counts)\n",
    "plt.yticks(range(len(cats)), cats)\n",
    "plt.xlabel('Counts')\n",
    "plt.title('Category counts')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's calculate the mean number of categories per document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean n. categories/document = 1.235\n"
     ]
    }
   ],
   "source": [
    "n_cat_sum = sum(len(reuters.categories(doc_id))\n",
    "                for doc_id in reuters.fileids())\n",
    "n_docs = len(reuters.fileids())\n",
    "print(\"mean n. categories/document = {:.3f}\".format(\n",
    "    n_cat_sum / n_docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we go through the usual preprocessing setup we've seen these days already: tokenization, removing accents, removing stopwords and tokens with two or less characters, lemmatization of tokens, and forming bigram collocations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n. train docs: 7769\n",
      "n. test docs: 3019\n"
     ]
    }
   ],
   "source": [
    "from gensim.models.doc2vec import TaggedDocument\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.models import Phrases\n",
    "from gensim.models.phrases import Phraser\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "stopwords_en = (frozenset(stopwords.words('english'))\n",
    "                | frozenset([\"also\"]))\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "lemmatize = lambda t: lemmatizer.lemmatize(t, pos='v')\n",
    "\n",
    "def read_and_tokenize(doc_id):\n",
    "    doc = reuters.raw(doc_id)\n",
    "    # strip accents and tokens with less than 3 characters\n",
    "    # NB: implicitly removes tokens *longer* than 15 characters\n",
    "    return simple_preprocess(doc, deacc=True, min_len=3)\n",
    "\n",
    "train_corpus = []\n",
    "test_corpus = []\n",
    "\n",
    "for doc_id in reuters.fileids():\n",
    "    doc = TaggedDocument(\n",
    "        simple_preprocess(reuters.raw(doc_id),\n",
    "                          deacc=True, min_len=3),\n",
    "        [doc_id])\n",
    "\n",
    "    if doc_id.startswith(\"train\"):\n",
    "        train_corpus.append(doc)\n",
    "    else:\n",
    "        test_corpus.append(doc)\n",
    "\n",
    "words = [d.words for d in train_corpus]\n",
    "collocations = Phraser(Phrases(words))\n",
    "\n",
    "def preprocess(doc):\n",
    "    \"\"\"\n",
    "    Stopword filtering,\n",
    "    collocation detection (joining),\n",
    "    and token lemmatization.\n",
    "    \"\"\"\n",
    "    \n",
    "    doc_filtered = filter(lambda w: w not in stopwords_en,\n",
    "                          doc.words)\n",
    "    doc_colloc = collocations[doc_filtered]\n",
    "    doc_lemmas = [lemmatize(token) for token in doc_colloc]\n",
    "    return TaggedDocument(doc_lemmas, doc.tags)\n",
    "\n",
    "train_corpus = [preprocess(d) for d in train_corpus]\n",
    "test_corpus = [preprocess(d) for d in test_corpus]\n",
    "\n",
    "print(\"n. train docs:\", len(train_corpus))\n",
    "print(\"n. test docs:\", len(test_corpus))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training/1 tags : ['training/1']\n",
      "training/1 words:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "bahia cocoa review shower continue throughout week bahia cocoa zone alleviate drought since early january improve prospect come temporao although normal humidity level restore comissaria smith say weekly review dry period mean temporao late year arrivals week_ended february bag kilos make cumulative total season mln stage last_year seem cocoa deliver..."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"training/1 tags :\", train_corpus[0].tags)\n",
    "print(\"training/1 words:\")\n",
    "HTML(\" \".join(train_corpus[0].words[:50]) + \"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the number of `tags` must be constant across all documents for Doc2Vec to work.\n",
    "And recall from the theory that `tags` is *not* a label, rather, its just an arbitrary, but unique (paragraph) ID for each document used during training (see Figures 2 and 3 in the Le & Mikolov paper).\n",
    "\n",
    "## Model training\n",
    "\n",
    "Gensim provides Le's paragraph vectors via its [Doc2Vec API](https://radimrehurek.com/gensim/models/doc2vec.html#gensim.models.doc2vec.Doc2Vec), which is based on the earlier Word2Vec API; It can be used to train the word embeddings at the same time (as the document embeddings)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec(dbow+w,d45,n20,w5,mc2,s0.001,t4)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(False, 1, 1)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models.doc2vec import Doc2Vec\n",
    "\n",
    "model = Doc2Vec(\n",
    "    workers=4, # CPU cores, \"t\" (threads)\n",
    "    epochs=10,\n",
    "    # typically, use more than Word2Vec (5-30)\n",
    "    # -> increase iter until the consistency evaluation \"works\"\n",
    "    alpha=0.025, # learning rate \n",
    "    # might optimize this (0.5-0.001)\n",
    "    # -> related with iter/epochs changes; fine-tuning\n",
    "    vector_size=len(reuters.categories()) // 2, # n. dim. \n",
    "    # -> optimize this (30-3000)\n",
    "    hs=0, negative=20, # use \"hs\" (1) or \"s\" (>0, range: 2-30)\n",
    "    # hs = hierarch. softmax\n",
    "    # s = neg. subsampling (rate)\n",
    "    window=5, # typically 5, might be optimized (2-30)\n",
    "    min_count=2, # min. freq. to add token to model\n",
    "    # filter junk and \"hard-to-learn\" words (1-10ish)\n",
    "    # -> a \"light\" regularization parameter\n",
    "    dm=0, dbow_words=1 # train with DM or DBOW (and W2V) model\n",
    ")\n",
    "# add the training documents\n",
    "model.build_vocab(train_corpus)\n",
    "print(model)\n",
    "# Doc2Vec(model,dimensions,....,threads)\n",
    "model.dm, model.dbow, model.sg\n",
    "# True 0 0 -> DM model\n",
    "# False 1 1 -> DBOW model (and words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the count for the word \"executive\" in the corpus is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "54"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.vocab['executive'].count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Traing should happen pretty fast - less than one minute on my quad-core machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:4: DeprecationWarning: Call to deprecated `iter` (Attribute will be removed in 4.0.0, use self.epochs instead).\n",
      "  after removing the cwd from sys.path.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 22s, sys: 2 s, total: 3min 24s\n",
      "Wall time: 1min 2s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "model.train(\n",
    "    train_corpus,\n",
    "    total_examples=model.corpus_count,\n",
    "    epochs=model.iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document inference\n",
    "\n",
    "With this model in place, you can now build new document embedding vectors for an unseen document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.03143727,  0.27475795,  0.30626455, -0.02935215,  0.45448834,\n",
       "        0.00536446,  0.04084774, -0.3026451 ,  0.34103075,  0.05760568,\n",
       "        0.31714466, -0.1352466 , -0.02963444,  0.02933758, -0.15975074,\n",
       "        0.06519668, -0.01539817, -0.10393609, -0.20873126, -0.02571797,\n",
       "        0.26548922,  0.14467439,  0.04403756, -0.00944956, -0.3252747 ,\n",
       "       -0.25383654,  0.21754347,  0.21752936, -0.06934156,  0.0858135 ,\n",
       "        0.094992  , -0.04853879,  0.19332312,  0.10552666, -0.3298745 ,\n",
       "        0.12288457, -0.18610416,  0.1215417 ,  0.16681048,  0.0258895 ,\n",
       "       -0.19246483,  0.15861763, -0.10338416,  0.23497216, -0.00846113],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.infer_vector([\n",
    "    'a', 'trivial', 'sample', 'document',\n",
    "    'to', 'infer', 'an', 'embedding'\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that each inference run produces slightly different results due to random sampling of the paragraph words (see the Le & Miklolov paper). Verify this by re-running the above cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model consistency check\n",
    "\n",
    "A simple way to assess the embeddings and the inference makes any sense at all is to self-test the model: That is, by infereing document vectors for each training set document, and verifying that the most similar document is the same as the input document using [`DocvecsArray.most_similar`](https://radimrehurek.com/gensim/models/doc2vec.html#gensim.models.doc2vec.DocvecsArray.most_similar).\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('training/12371', 0.956614077091217),\n",
       " ('training/14701', 0.955879807472229),\n",
       " ('training/8025', 0.955065906047821),\n",
       " ('training/13158', 0.9506606459617615),\n",
       " ('training/12366', 0.9506160616874695),\n",
       " ('training/9228', 0.9500134587287903),\n",
       " ('training/7744', 0.9498597979545593),\n",
       " ('training/11158', 0.9498116970062256),\n",
       " ('training/13148', 0.9497957229614258),\n",
       " ('training/6713', 0.9493430256843567)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.docvecs.most_similar([\n",
    "    model.infer_vector([\n",
    "        'a', 'trivial', 'sample', 'document',\n",
    "        'to', 'infer', 'an', 'embedding'\n",
    "    ])\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets run this over all documents and see where we stand; Because we must re-run the inference for each document, and then need to compare the simlilarity of the inferred vector against all known vectors, this might take a bit. To make this faster, we'll just check 10% of our documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rank 1 document matched observed 736 times (94.8%)\n",
      "rank 2 document matched observed 29 times (3.7%)\n",
      "rank 3 document matched observed 4 times (0.5%)\n",
      "rank 4 document matched observed 2 times (0.3%)\n",
      "\n",
      "remaining ranks observed 5 times (0.6%)\n",
      "mean similiarity score = 0.973\n"
     ]
    }
   ],
   "source": [
    "from random import sample\n",
    "\n",
    "n_docs = len(train_corpus)\n",
    "n_trials = n_docs // 10\n",
    "ranks = [0] * n_docs\n",
    "similarty_sum = 0.\n",
    "# alias (save screen width)\n",
    "most_similar = model.docvecs.most_similar\n",
    "\n",
    "for doc in sample(train_corpus, n_trials):\n",
    "    inferred_vector = model.infer_vector(doc.words)\n",
    "    sims = most_similar([inferred_vector], topn=n_docs)\n",
    "    doc_ids = [doc_id for doc_id, sim in sims]\n",
    "    rank = doc_ids.index(doc.tags[0])\n",
    "    ranks[rank] += 1\n",
    "    similarty_sum += sims[rank][1]\n",
    "    \n",
    "for rank, count in enumerate(ranks[:5], 1):\n",
    "    if count > 0:\n",
    "        print(\"rank\", rank, \"document matched observed\", count,\n",
    "              \"times ({:.1f}%)\".format(100 * count / n_trials))\n",
    "\n",
    "print(\"\\nremaining ranks observed {:d} times ({:.1f}%)\".format(\n",
    "    sum(ranks[5:]),\n",
    "    100 * sum(ranks[5:]) / n_trials))\n",
    "print(\"mean similiarity score = {:.3f}\".format(\n",
    "    similarty_sum / n_trials))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAakklEQVR4nO3dfZBc1X3m8e8zPT1oBGgGwVgGvUTE1prgVHjJFMbB5bXRgo28sbQVmzLxGploI2eNd015KUOcXRvvZmPiYmNCJcuuYmwLBzAEG6NNscREgL3OGsKAxbuzjCkJSdbLGJDQK9LM/PaPPj1qjbpnukfdas29z6dqqu8999x7z1WXnj59uu9pRQRmZpZdHe1ugJmZtZaD3sws4xz0ZmYZ56A3M8s4B72ZWcY56M3MMs5Bb9OOpBsk/fUU9/2kpB9XrO+W9KtNatcXJH09LS+UFJI6m3TsBamthWYcz/LFQW8tIWm9pH0pnLZK+pakk9rdrvEi4qSIeHmiOpLeJ2lTHcf6k4j4N81oV/r3+xcVx34ltXWkGce3fHHQWyv9dkScBJwLnAf8YZvb0zLN6rmbtYKD3louIrYCf0cp8AGQ9CFJP5X0hqSNkm6o2FYe9lgu6RVJv5T0R9WOLako6S5J35XUVWX7qZLWpPP8I/C2cdtD0tvT8hJJL0jaJWmzpGslnQj8b+CM9O5kt6Qz0vDRvZL+WtIbwCdrDCn9nqRfSNoi6dqK835L0h9XrI+9a5D0bWAB8L/S+T4/figotWGNpNckDUr6/Ypj3SDpHkm3p2t5XlL/xM+SZZmD3lpO0jzgMmCwongPcCXQC3wI+LeSlo3b9T3AO4DFwBcl/dq443YD3wfeBC6PiANVTv+XwH7gdOD30l8ttwGfioiTgV8HHo6IPantv0hDJydFxC9S/aXAveka7qhxzPcDi4BLgesqh2NqiYhPAK+Q3hFFxFerVPsOsAk4A/gI8CeSLq7Y/uFUpxdYA/zFZOe17HLQWyt9X9IuYCOwHfhSeUNEPBoRz0bEaEQ8A9wF/PNx+385IvZFxNPA08A5FdtmAQ8CPweuqjZ2nT64/B3gixGxJyKeA1ZP0N6DwNmSZkXE6xHx1CTX95OI+H66hn016nw5nftZ4JvAFZMcc1KS5gMXAddFxP6IWAd8ndILZ9mPI+KB9O/ybQ7/t7OccdBbKy1LveP3AWcBp5U3SHqXpEckDUnaCfxB5fZka8XyXqDyw9wLgd8AbozaM/P1AZ2UXmjKNkzQ3t8BlgAbJP1Q0rsnqMu449ZTZwOlHvjROgN4LSJ2jTv23Ir18f92M/w5Qn456K3lIuKHwLeAmyqK76Q0pDA/InqA/wGogcP+APgKsFbSnBp1hoBhYH5F2YIJ2vlERCwF3kJpSOie8qZau9TRzvHnLg/77AFmVmx7awPH/gUwW9LJ4469uY72WA456O1YuRm4RFJ5COFkSr3S/ZIuAH630QOmses7KYX9+HcDpGGL7wE3SJop6WxgebVjSeqS9HFJPRFxEHgDGE2btwGnSupptI3Af0rnfidwFXB3Kl8HLJE0W9JbgWvG7bcNqPr9/ojYCPxf4CuSZkj6DWAFMKV7Cyz7HPR2TETEEHA78MVU9GngP6cx/C9yqPfc6HH/C6Xe999Lml2lymcoDflspfSu4psTHO4TwPr0LZo/AD6ezvEzSp8hvCxph6RGhl9+SOlD6LXATRHxg1T+bUqfO6yn9O7k7nH7fQX4j+l813KkK4CFlHr39wFfioi/b6BdliPyD4+YmWWbe/RmZhnnoDczyzgHvZlZxjnozcwy7ri4geK0006LhQsXtrsZZmbTypNPPvnLiOibrN5xEfQLFy5kYGCg3c0wM5tWJE10p/cYD92YmWWcg97MLOMc9GZmGeegNzPLOAe9mVnGOejNzDLOQW9mlnHTOuifWP8aX33wZ4yOegZOM7NapnXQP71xB//90Z+z683hdjfFzOy4Na2DflZ3EYA39h1sc0vMzI5f0zroe1PQ73TQm5nVNK2DvsdBb2Y2qekd9DMd9GZmk5neQe8evZnZpDIR9Dv2OujNzGqZNOglvUPSuoq/NyRdI2m2pIckvZQeT0n1JekWSYOSnpF0fqsa310sUCzIPXozswlMGvQR8U8RcW5EnAv8JrAXuA+4HlgbEYuAtWkd4DJgUfpbCdzaioYDSKKnu+igNzObQKNDN4uBn0fEBmApsDqVrwaWpeWlwO1R8hjQK+n0prS2ilndRX+P3sxsAo0G/ceAu9LynIjYkpa3AnPS8lxgY8U+m1LZYSStlDQgaWBoaKjBZhziHr2Z2cTqDnpJXcCHgb8Zvy0iAmhowpmIWBUR/RHR39c36W/b1uSgNzObWCM9+suApyJiW1rfVh6SSY/bU/lmYH7FfvNSWUs46M3MJtZI0F/BoWEbgDXA8rS8HLi/ovzK9O2bC4GdFUM8TdfroDczm1BnPZUknQhcAnyqovhG4B5JK4ANwOWp/AFgCTBI6Rs6VzWttVX0dBd5Y/9BRkeDjg618lRmZtNSXUEfEXuAU8eVvUrpWzjj6wZwdVNaV4dZ3UUiYNebw2M3UJmZ2SHT+s5YqJgGwXfHmplVlZ2g9zi9mVlVDnozs4yb/kHvqYrNzCY0/YPePXozswk56M3MMm7aB72nKjYzm9i0D/rSVMVdDnozsxqmfdAD9HR3eqpiM7MaMhL0nu/GzKwWB72ZWcZlJuh37DvQ7maYmR2XMhP0nuvGzKy6zAT9rjeHGR1t6EeuzMxyIRNBPzZV8f7hdjfFzOy4k4mg992xZma1OejNzDIuE0HfO7MLcNCbmVVTV9BL6pV0r6SfSXpR0rslzZb0kKSX0uMpqa4k3SJpUNIzks5v7SW4R29mNpF6e/R/DjwYEWcB5wAvAtcDayNiEbA2rQNcBixKfyuBW5va4ioc9GZmtU0a9JJ6gPcCtwFExIGI2AEsBVanaquBZWl5KXB7lDwG9Eo6vektr+CgNzOrrZ4e/ZnAEPBNST+V9HVJJwJzImJLqrMVmJOW5wIbK/bflMoOI2mlpAFJA0NDQ1O/AmBGsYOuQofvjjUzq6KeoO8EzgdujYjzgD0cGqYBICICaOhupYhYFRH9EdHf19fXyK5HkMSs7qJnsDQzq6KeoN8EbIqIx9P6vZSCf1t5SCY9bk/bNwPzK/afl8paqqe700M3ZmZVTBr0EbEV2CjpHaloMfACsAZYnsqWA/en5TXAlenbNxcCOyuGeFrGM1iamVXXWWe9fwfcIakLeBm4itKLxD2SVgAbgMtT3QeAJcAgsDfVbbme7iJDu988FqcyM5tW6gr6iFgH9FfZtLhK3QCuPsp2Naynu8jg0O5jfVozs+NeJu6MBU9VbGZWS3aCfmaXpyo2M6siO0HvqYrNzKrKVNCD7441Mxsvc0Hvu2PNzA6XuaB3j97M7HAOejOzjHPQm5llnIPezCzjMhP05amKHfRmZofLTNB7qmIzs+oyE/QAvTM9g6WZ2XiZCnpPVWxmdiQHvZlZxjnozcwyLnNBv8NTFZuZHSZTQT+ru8iu/cOMeKpiM7MxdQW9pPWSnpW0TtJAKpst6SFJL6XHU1K5JN0iaVDSM5LOb+UFVCrfNLVrv3v1ZmZljfTo3x8R50ZE+ScFrwfWRsQiYG1aB7gMWJT+VgK3Nquxk/HdsWZmRzqaoZulwOq0vBpYVlF+e5Q8BvRKOv0ozlM3B72Z2ZHqDfoAfiDpSUkrU9mciNiSlrcCc9LyXGBjxb6bUlnLOejNzI7UWWe990TEZklvAR6S9LPKjRERkhr6BDS9YKwEWLBgQSO71uSgNzM7Ul09+ojYnB63A/cBFwDbykMy6XF7qr4ZmF+x+7xUNv6YqyKiPyL6+/r6pn4FFXpnOujNzMabNOglnSjp5PIycCnwHLAGWJ6qLQfuT8trgCvTt28uBHZWDPG0lHv0ZmZHqmfoZg5wn6Ry/Tsj4kFJTwD3SFoBbAAuT/UfAJYAg8Be4Kqmt7qGGcUCXZ2eqtjMrNKkQR8RLwPnVCl/FVhcpTyAq5vSuino6S6y03fHmpmNydSdseD5bszMxnPQm5llnIPezCzjHPRmZhnnoDczy7jMBb2nKjYzO1zmgr7XUxWbmR0mc0Hvu2PNzA7noDczy7jsBX2a2My/HWtmVpK9oHeP3szsMA56M7OMc9CbmWVc5oK+PFXxGw56MzMgg0EPvjvWzKySg97MLOMyGfS9DnozszGZDHr36M3MDqk76CUVJP1U0t+m9TMlPS5pUNLdkrpS+QlpfTBtX9iaptfmoDczO6SRHv1ngRcr1v8U+FpEvB14HViRylcAr6fyr6V6x9QsB72Z2Zi6gl7SPOBDwNfTuoCLgXtTldXAsrS8NK2Tti9O9Y+ZHk9VbGY2pt4e/c3A54HRtH4qsCMihtP6JmBuWp4LbARI23em+oeRtFLSgKSBoaGhKTa/uvJNU/4uvZlZHUEv6V8C2yPiyWaeOCJWRUR/RPT39fU189C+O9bMrEJnHXUuAj4saQkwA5gF/DnQK6kz9drnAZtT/c3AfGCTpE6gB3i16S2fgIPezOyQSXv0EfGHETEvIhYCHwMejoiPA48AH0nVlgP3p+U1aZ20/eGIOKaD5eWpih30ZmZH9z3664DPSRqkNAZ/Wyq/DTg1lX8OuP7omtg49+jNzA6pZ+hmTEQ8Cjyall8GLqhSZz/w0Sa0bcoc9GZmh2T2zlhw0JuZQUaDfkaxwAmeqtjMDMho0IOnQTAzK8t00PsHws3MMh707tGbmTnozcwyz0FvZpZxmQ36Wd1Ff+vGzIwMB31Pd5Fdb3qqYjOzTAc9eKpiM7PMBn2vJzYzMwMyHPSeBsHMrMRBb2aWcQ56M7OMy3zQ73DQm1nOZTboZ/lbN2ZmQIaDvjxVsYduzCzvMhv0kKZB8AyWZpZzkwa9pBmS/lHS05Kel/TlVH6mpMclDUq6W1JXKj8hrQ+m7Qtbewm1eb4bM7P6evRvAhdHxDnAucAHJV0I/CnwtYh4O/A6sCLVXwG8nsq/luq1hYPezKyOoI+S3Wm1mP4CuBi4N5WvBpal5aVpnbR9sSQ1rcUNcNCbmdU5Ri+pIGkdsB14CPg5sCMihlOVTcDctDwX2AiQtu8ETq1yzJWSBiQNDA0NHd1V1NAz00FvZlZX0EfESEScC8wDLgDOOtoTR8SqiOiPiP6+vr6jPVxVPZ6q2MyssW/dRMQO4BHg3UCvpM60aR6wOS1vBuYDpO09wKtNaW2DylMVD4+MtuP0ZmbHhXq+ddMnqTctdwOXAC9SCvyPpGrLgfvT8pq0Ttr+cES0ZVL4samK9w9PUtPMLLs6J6/C6cBqSQVKLwz3RMTfSnoB+I6kPwZ+CtyW6t8GfFvSIPAa8LEWtLsulfPdzD6xq13NMDNrq0mDPiKeAc6rUv4ypfH68eX7gY82pXVHyRObmZnl4M5YcNCbWb456M3MMs5Bb2aWcZkOek9VbGaW8aCfUSwwo+ipis0s3zId9OCpis3M8hH07tGbWY7lIuh37DvQ7maYmbVNLoJ+5z5PgWBm+ZX5oJ/lGSzNLOcyH/QeozezvMtF0O/2VMVmlmO5CHrwVMVmll+5CXoP35hZXmU+6HtnOujNLN8yH/Tu0ZtZ3jnozcwyLvNBP8tBb2Y5V8+Pg8+X9IikFyQ9L+mzqXy2pIckvZQeT0nlknSLpEFJz0g6v9UXMZGxHv1eT4NgZvlUT49+GPgPEXE2cCFwtaSzgeuBtRGxCFib1gEuAxalv5XArU1vdQNO6PRUxWaWb5MGfURsiYin0vIu4EVgLrAUWJ2qrQaWpeWlwO1R8hjQK+n0pre8Ab471szyrKExekkLgfOAx4E5EbElbdoKzEnLc4GNFbttSmXjj7VS0oCkgaGhoQab3RgHvZnlWd1BL+kk4LvANRHxRuW2iAggGjlxRKyKiP6I6O/r62tk14Y56M0sz+oKeklFSiF/R0R8LxVvKw/JpMftqXwzML9i93mprG08VbGZ5Vk937oRcBvwYkT8WcWmNcDytLwcuL+i/Mr07ZsLgZ0VQzxt4amKzSzPOuuocxHwCeBZSetS2ReAG4F7JK0ANgCXp20PAEuAQWAvcFVTWzwFvd1dHroxs9yaNOgj4seAamxeXKV+AFcfZbuaqnKq4s5C5u8RMzM7TC5Sr6e79HrmqYrNLI/yEfRpBssdvjvWzHIoH0Hv+W7MLMcc9GZmGeegNzPLuFwEfXmqYn+X3szyKBdB7x69meVZLoLeUxWbWZ7lIujBd8eaWX7lJug9g6WZ5ZWD3sws43IT9LO6i+zY66A3s/zJTdD3eKpiM8upXAW9h27MLI9yFfR7DoxwcGS03U0xMzumchT0aapi9+rNLGfyE/QzfXesmeVTfoLe0yCYWU7V8+Pg35C0XdJzFWWzJT0k6aX0eEoql6RbJA1KekbS+a1sfCMc9GaWV/X06L8FfHBc2fXA2ohYBKxN6wCXAYvS30rg1uY08+j1dHcBDnozy59Jgz4ifgS8Nq54KbA6La8GllWU3x4ljwG9kk5vVmOPRo+nKjaznJrqGP2ciNiSlrcCc9LyXGBjRb1NqewIklZKGpA0MDQ0NMVm1M9DN2aWV0f9YWxEBBBT2G9VRPRHRH9fX9/RNmNSXZ0ddBcLngbBzHJnqkG/rTwkkx63p/LNwPyKevNS2XHBd8eaWR5NNejXAMvT8nLg/oryK9O3by4EdlYM8bSdg97M8qhzsgqS7gLeB5wmaRPwJeBG4B5JK4ANwOWp+gPAEmAQ2Atc1YI2T5mD3szyaNKgj4gramxaXKVuAFcfbaNaZVZ3kU2v7213M8zMjqnc3BkLnqrYzPIpd0HvoRszy5tcBX3vTE9VbGb5k6ug992xZpZHuQx6D9+YWZ7kMuh3OOjNLEdyFfSz3KM3sxzKVdB7jN7M8iiXQe8evZnlST6D3jNYmlmO5Croy1MVu0dvZnmSq6AH3x1rZvmTy6Bf/+oe9h8caXdTzMyOidwF/cW/9haeWP8677/pUe5+4hWGPR2CmWVc7oL+ug+exZ2//y7mzJrBdd99lktv/hEPPLuF0gzLZmbZk7ugB/itt53GfZ/+Lf7nJ36TgsSn73iKpX/5D/z4pV+2u2lmZk2Xy6AHkMQH3vlWHrzmvdz00XN4dfcB/vVtj/O7f/UY6zbuaHfzzMyaRsfDkEV/f38MDAy0tQ1vDo9w5+Ov8BcPD/LqngN84J1zuPbSd7BozsltbZeZWS2SnoyI/snqtaRHL+mDkv5J0qCk61txjmY7obPAVRedyQ8//34+d8k/4x8GX+UDN/+Ia//maf/8oJlNa03v0UsqAP8PuATYBDwBXBERL9Ta53jo0Y/32p4D3ProIKt/soGI4C0nz2BmV4GZXQVmFAtpuZPuVNZdLBxa7uqku1heLjCzeKhueb27q8AJnR1Iqnr+iODAyCj7DoywN/3tOzDCvoMj7D0wPLZcLHSMO3/nofN2FZjRWaCjo/o5ppPR0eDg6CgHR4LhkdJjRFAsdNBZEMVCB8VCB4VjfK3ldg2PBAenQbs6Cx0U29gua656e/ST/jj4FFwADEbEy6kh3wGWAjWD/ng0+8Qu/uhDZ3PVRWdy+082MLTrTfYdHB4L3V/uPsDeA3tLQXywVHZguLGvanaIQy8AxQIShwX6yGhzXoTLLwLlx+Pxv/ZoBMOjwfBI6QVueGT00PJo1P1v0SFKYdYhip0ddHZ00FUQnSl4CzVeWGsJGHthKQXm1NolUQrXinaVA7cZ7RoeDQ4Oj469GDbUrtSWzhT+R9Mua9y/X7yI3z7njJaeoxVBPxfYWLG+CXjX+EqSVgIrARYsWNCCZjTHGb3dXH/ZWXXVHRmNw3rctXrieyvW9x4YYX96oYjgsN74zHHvDLqLR76LODgyOna8Q+ccTscvn394rM7xeqOYEJ0FlYK5UykIDw+d8npnRwfFzg5ERdiNjnJwOBgeHU0vFKWe/4GxdwCpV0vjL57FQkfVdlX22CdqV2Wv+mB6ETusXaMxpa/3lttV+W/UNe6dxFi7CkJS6QV0NMb+jca362B6wSjXOR4+w8u68hxcrdSKoK9LRKwCVkFp6KZd7WimQoc46YROTjqhbf+sZmZHaMWHsZuB+RXr81KZmZm1QSuC/glgkaQzJXUBHwPWtOA8ZmZWh6aPMUTEsKTPAH8HFIBvRMTzzT6PmZnVpyWDyRHxAPBAK45tZmaNye0UCGZmeeGgNzPLOAe9mVnGOejNzDLuuJi9UtIQsGGKu58G5Hki+Txff56vHfJ9/b72kl+JiL7Jdjgugv5oSBqoZ1KfrMrz9ef52iHf1+9rb+zaPXRjZpZxDnozs4zLQtCvancD2izP15/na4d8X7+vvQHTfozezMwmloUevZmZTcBBb2aWcdM66Kfjj5A3i6T1kp6VtE7S8fWDuy0g6RuStkt6rqJstqSHJL2UHk9pZxtbpca13yBpc3r+10la0s42toqk+ZIekfSCpOclfTaV5+W5r3X9DT3/03aMfio/Qp4lktYD/RGRi5tGJL0X2A3cHhG/nsq+CrwWETemF/pTIuK6drazFWpc+w3A7oi4qZ1tazVJpwOnR8RTkk4GngSWAZ8kH899reu/nAae/+ncox/7EfKIOACUf4TcMigifgS8Nq54KbA6La+m9B8gc2pcey5ExJaIeCot7wJepPS71Hl57mtdf0Omc9BX+xHyhv8BprEAfiDpyfRD63k0JyK2pOWtwJx2NqYNPiPpmTS0k8mhi0qSFgLnAY+Tw+d+3PVDA8//dA76vHtPRJwPXAZcnd7e51aUxiCn5zjk1NwKvA04F9gC/Lf2Nqe1JJ0EfBe4JiLeqNyWh+e+yvU39PxP56DP9Y+QR8Tm9LgduI/SUFbebEtjmOWxzO1tbs8xExHbImIkIkaBvyLDz7+kIqWQuyMivpeKc/PcV7v+Rp//6Rz0uf0Rckknpg9mkHQicCnw3MR7ZdIaYHlaXg7c38a2HFPlkEv+FRl9/iUJuA14MSL+rGJTLp77Wtff6PM/bb91A5C+UnQzh36E/L+2uUnHhKRfpdSLh9Lv/t6Z9WuXdBfwPkpTtG4DvgR8H7gHWEBpmuvLIyJzH1rWuPb3UXrbHsB64FMVY9aZIek9wP8BngVGU/EXKI1T5+G5r3X9V9DA8z+tg97MzCY3nYduzMysDg56M7OMc9CbmWWcg97MLOMc9GZmGeegNzPLOAe9mVnG/X/amJAWESr0CwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(list(range(25)), ranks[:25])\n",
    "plt.title('Rank distribution')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that your results will differ on each run of the above cell due to the random sampling and also because the model isn't very good due to the tiny corpus size. But you should observe a large mean (Cosine vector) similarity, like 0.8, and ideally above 0.9, and there should be a significant \"mass\" of documents that are matched with rank 1 (80% or more).\n",
    "\n",
    "## Model evaluation\n",
    "\n",
    "Next, lets use the following tactic to evaluate our model: How many of the categories of the query (i.e., test) document match with the best-matching indexed (i.e., train) document? You could use this, for example, to evaluate adding documents to category-specific clusters, or a nearest neighbor-like apporach for predicting document categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision: 2852/3744 = 76.18%\n",
      "Recall   : 2852/3659 = 77.94%\n",
      "\n",
      "mean   similiarity score = 0.886\n",
      "median similiarity score = 0.890\n"
     ]
    }
   ],
   "source": [
    "similarties = []\n",
    "n_docs = len(test_corpus)\n",
    "hits = 0\n",
    "pred_cats = 0\n",
    "ann_cats = 0\n",
    "# alias (save screen width)\n",
    "most_similar = model.docvecs.most_similar\n",
    "\n",
    "for doc in test_corpus:\n",
    "    test_id = doc.tags[0]\n",
    "    test_cats = set(reuters.categories(test_id))\n",
    "    \n",
    "    inferred_vector = model.infer_vector(doc.words)\n",
    "    \n",
    "    match_id, sim = most_similar([inferred_vector], topn=1)[0]\n",
    "    match_cats = set(reuters.categories(match_id))\n",
    "    \n",
    "    similarties.append(sim)\n",
    "    hits += len(test_cats & match_cats)\n",
    "    pred_cats += len(test_cats)\n",
    "    ann_cats += len(match_cats)\n",
    "\n",
    "similarties = sorted(similarties)\n",
    "\n",
    "print(\"Precision: {:d}/{:d} = {:.2f}%\".format(\n",
    "    hits, pred_cats,\n",
    "    100 * hits / pred_cats))\n",
    "print(\"Recall   : {:d}/{:d} = {:.2f}%\\n\".format(\n",
    "    hits, ann_cats,\n",
    "    100 * hits / ann_cats))\n",
    "print(\"mean   similiarity score = {:.3f}\".format(\n",
    "    sum(similarties) / n_docs))\n",
    "print(\"median similiarity score = {:.3f}\".format(\n",
    "    similarties[n_docs//2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step back a moment and let this sink in: We've achieved a category prediction performance that maybe does not (yet) beat the supervised methods we looked into on day one. Without ever conditioning the model on the categories, we've achieved a result that is far beyond random predictions (Reuters-21578 has 90 categories). That is, the paragraph vector model figured out the documents' semantics all on its own, and we never told it anything about the categories, yet it clearly is able to make sensible matches, retrieving documents from the same categories.\n",
    "\n",
    "Our above evaluation setup resembles that of a traditional nearest neighbor classifier. For reference, the highest score a (heavily optimized) kNN classifier ever achieved on this dataset is a [86% micro-averaged $F_1$-score](https://www.aaai.org/Papers/ICML/2003/ICML03-063.pdf) (see Figure 2). But the best kNN classifier used 2000 features, while our model only has 45 dimensions (i.e., features), those feature vectors are generated fully **unsupervised**, and they can be used to train (supervised) classifiers (see the Le & Mikolov paper, where they evaluated paragraph vectors feeding a logistic regression classifier on various datasets and tasks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Basline precision (predict earn always): 36.0%\n",
      "Basline recall    (predict earn always): 29.0%\n"
     ]
    }
   ],
   "source": [
    "# document IDs labeled \"earn\" that start with \"test/\"\n",
    "hits = sum(1 for f in reuters.fileids(\"earn\")\n",
    "           if f.startswith(\"test/\"))\n",
    "\n",
    "test_docs = [f for f in reuters.fileids()\n",
    "             if f.startswith(\"test/\")]\n",
    "\n",
    "# n. labels on of test documents\n",
    "n_cats = sum(len(reuters.categories(f)) for f in test_docs)\n",
    "\n",
    "print(\"Basline precision (predict earn always): {:.1f}%\".format(\n",
    "    100 * hits / len(test_docs)))\n",
    "\n",
    "print(\"Basline recall    (predict earn always): {:.1f}%\".format(\n",
    "    100 * hits / n_cats))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the Reuters categories are highly imbalanced, and just predicting the majority label (`earn`) gives you a 32% micro-averaged $F_1$ on this set (i.e., the best \"test dummy\" baseline, maybe).\n",
    "\n",
    "## Conclusions\n",
    "\n",
    "Finally we have arrived at the \"core\" of statistical text mining: we can now easily convert a document into small-ish (< 1000) arrays of numbers. And we've emulated supervised models using only \"represenation learning\" techniques that are fully unsupervised. It should go without saying that even more impressive results can be achieved by combining the two approaches: first, use representation learning to create document vectors and then use supervised learning to predict the correct label(s), using the document vectors as input (instead of the noisy, sparse bag-of-word vectors)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}