NLP-Lab-Starter-Kang.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing Lab\n",
    "\n",
    "In this lab we will further explore Scikit's and NLTK's capabilities to process text. We will use the 20 Newsgroup dataset, which is provided by Scikit-Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading 20news dataset. This may take a few minutes.\n",
      "Downloading dataset from https://ndownloader.figshare.com/files/5975967 (14 MB)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "categories = [\n",
    "    'alt.atheism',\n",
    "    'talk.religion.misc',\n",
    "    'comp.graphics',\n",
    "    'sci.space',\n",
    "]\n",
    "\n",
    "data_train = fetch_20newsgroups(subset='train', categories=categories,\n",
    "                                shuffle=True, random_state=42,\n",
    "                                remove=('headers', 'footers', 'quotes'))\n",
    "\n",
    "data_test = fetch_20newsgroups(subset='test', categories=categories,\n",
    "                               shuffle=True, random_state=42,\n",
    "                               remove=('headers', 'footers', 'quotes'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data inspection\n",
    "\n",
    "We have downloaded a few newsgroup categories and removed headers, footers and quotes.\n",
    "\n",
    "Because this is an SKLearn dataset, it comes with pre-split train and test sets (note we were able to call 'train' and 'test' in subset).\n",
    "\n",
    "Let's inspect them.\n",
    "\n",
    "1. What data taype is `data_train`\n",
    "> sklearn.datasets.base.Bunch\n",
    "- Is it like a list? Or like a Dictionary? or what?\n",
    "> Dict\n",
    "- How many data points does it contain?\n",
    "- Inspect the first data point, what does it look like?\n",
    "> A blurb of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sklearn.utils.Bunch"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['description', 'DESCR', 'filenames', 'target_names', 'data', 'target']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2034"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_train['data'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2034"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_train['target'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u\"Hi,\\n\\nI've noticed that if you only save a model (with all your mapping planes\\npositioned carefully) to a .3DS file that when you reload it after restarting\\n3DS, they are given a default position and orientation.  But if you save\\nto a .PRJ file their positions/orientation are preserved.  Does anyone\\nknow why this information is not stored in the .3DS file?  Nothing is\\nexplicitly said in the manual about saving texture rules in the .PRJ file. \\nI'd like to be able to read the texture rule information, does anyone have \\nthe format for the .PRJ file?\\n\\nIs the .CEL file format available from somewhere?\\n\\nRych\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train['data'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Bag of Words model\n",
    "\n",
    "Let's train a model using a simple count vectorizer\n",
    "\n",
    "1. Initialize a standard CountVectorizer and fit the training data\n",
    "- how big is the feature dictionary?\n",
    "- repeat eliminating english stop words\n",
    "- is the dictionary smaller?\n",
    "- transform the training data using the trained vectorizer\n",
    "- evaluate the performance of a Lotistic Regression on the features extracted by the CountVectorizer\n",
    "    - you will have to transform the test_set too. Be carefule to use the trained vectorizer, without re-fitting it\n",
    "\n",
    "BONUS:\n",
    "- try a couple modifications:\n",
    "    - restrict the max_features\n",
    "    - change max_df and min_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 3, 2, ..., 1, 0, 1], dtype=int64)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "vect = CountVectorizer()\n",
    "X_train_dtm = vect.fit_transform(data_train.data)\n",
    "X_test_dtm = vect.transform(data_test.data)\n",
    "y_train_dtm = data_train.target\n",
    "y_test_dtm = data_test.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1353x26879 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 129533 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26879)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'zien', u'zijn', u'zillion', u'zillions', u'zion', u'zionist', u'zip', u'zipped', u'zipping', u'zippy', u'zlumber', u'zo', u'zogeheten', u'zombie', u'zond', u'zone', u'zonker', u'zoo', u'zoology', u'zoom', u'zooming', u'zooms', u'zopfi', u'zorastrian', u'zorg', u'zorn', u'zoro', u'zoroaster', u'zoroastrian', u'zoroastrianism', u'zoroastrians', u'zsoft', u'zubin', u'zuck', u'zues', u'zug', u'zullen', u'zulu', u'zur', u'zurbrin', u'zurich', u'zurvanism', u'zus', u'zvi', u'zwaartepunten', u'zwak', u'zwakke', u'zware', u'zwarte', u'zyxel']\n"
     ]
    }
   ],
   "source": [
    "print(vect.get_feature_names()[-50:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "vect = CountVectorizer(stop_words='english')\n",
    "X_train_dtm = vect.fit_transform(data_train.data)\n",
    "X_test_dtm = vect.transform(data_test.data)\n",
    "y_train_dtm = data_train.target\n",
    "y_test_dtm = data_test.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The dictionary has shrunk after eliminating stop words, which should be expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = X_train_dtm\n",
    "X_test = X_test_dtm\n",
    "y_train = y_train_dtm\n",
    "y_test = y_test_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "lr.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "preds_lr = lr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR:  0.745011086475\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score\n",
    "\n",
    "print \"LR: \", accuracy_score(y_test,preds_lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Hashing and TF-IDF\n",
    "\n",
    "Let's see if Hashing or TF-IDF improves the accuracy.\n",
    "\n",
    "1. Initialize a HashingVectorizer and repeat the test with no restriction on the number of features\n",
    "- does the score improve with respect to the count vectorizer?\n",
    "- print out the number of features for this model\n",
    "- Initialize a TF-IDF Vectorizer and repeat the analysis above\n",
    "- print out the number of features for this model\n",
    "\n",
    "BONUS\n",
    "- Change the parameters of either (or both!) models to improve your score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vect = HashingVectorizer(stop_words='english')\n",
    "X_train_dtm = vect.fit_transform(data_train.data)\n",
    "X_test_dtm = vect.transform(data_test.data)\n",
    "y_train_dtm = data_train.target\n",
    "y_test_dtm = data_test.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = X_train_dtm\n",
    "X_test = X_test_dtm\n",
    "y_train = y_train_dtm\n",
    "y_test = y_test_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "lr.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR:  0.745011086475\n"
     ]
    }
   ],
   "source": [
    "preds_lr = lr.predict(X_test)\n",
    "print \"LR: \", accuracy_score(y_test,preds_lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2034, 26576)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_dtm.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vect = TfidfVectorizer(max_df=0.8, min_df=0.2)\n",
    "X_train_dtm = vect.fit_transform(data_train.data)\n",
    "X_test_dtm = vect.transform(data_test.data)\n",
    "y_train_dtm = data_train.target\n",
    "y_test_dtm = data_test.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = X_train_dtm\n",
    "X_test = X_test_dtm\n",
    "y_train = y_train_dtm\n",
    "y_test = y_test_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR:  0.449371766445\n"
     ]
    }
   ],
   "source": [
    "preds_lr = lr.predict(X_test)\n",
    "print \"LR: \", accuracy_score(y_test,preds_lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My score remains the exact same with a Hashing Vectorizer, while it dramatically decreases with a TFIDF vectorizer."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}