Chapter 2 - Supervised learning - Explaining Titanic Hypothesis with Decision Trees-ModifiedByTim.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Scikit-learn: Machine Learning in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook for Chapter 2: Supervised Learning: Explaining Titanic Hypothesis with Decision Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we must load the dataset. We assume it is located in the data/titanic.csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import numpy as np\n",
    "with open('data/titanic.csv', 'rb') as csvfile:\n",
    "    titanic_reader = csv.reader(csvfile, delimiter=',', quotechar='\"')\n",
    "    \n",
    "    # Header contains feature names\n",
    "    row = titanic_reader.next()\n",
    "    feature_names = np.array(row)\n",
    "    \n",
    "    # Load dataset, and target classes\n",
    "    titanic_X, titanic_y = [], []\n",
    "    for row in titanic_reader:  \n",
    "        titanic_X.append(row)\n",
    "        titanic_y.append(row[2]) # The target value is \"survived\"\n",
    "    \n",
    "    titanic_X = np.array(titanic_X)\n",
    "    titanic_y = np.array(titanic_y)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['row.names' 'pclass' 'survived' 'name' 'age' 'embarked' 'home.dest'\n",
      " 'room' 'ticket' 'boat' 'sex'] ['1' '1st' '1' 'Allen, Miss Elisabeth Walton' '29.0000' 'Southampton'\n",
      " 'St Louis, MO' 'B-5' '24160 L221' '2' 'female'] 1\n"
     ]
    }
   ],
   "source": [
    "print feature_names, titanic_X[0], titanic_y[0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep only class (1st,2nd,3rd), age (float), and sex (masc, fem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we keep the class, the age and the sex\n",
    "titanic_X = titanic_X[:, [1, 4, 10]]\n",
    "feature_names = feature_names[[1, 4, 10]]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['pclass' 'age' 'sex']\n",
      "['1st' 'NA' 'female'] 1\n"
     ]
    }
   ],
   "source": [
    "print feature_names\n",
    "print titanic_X[12], titanic_y[12]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Solve missing values ('NA') for the 'age' feature. Solution: use the mean value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We have missing values for age\n",
    "# Assign the mean value\n",
    "ages = titanic_X[:, 1]\n",
    "mean_age = np.mean(titanic_X[ages != 'NA', 1].astype(np.float))\n",
    "titanic_X[titanic_X[:, 1] == 'NA', 1] = mean_age\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['pclass' 'age' 'sex']\n",
      "['1st' '31.19418104265403' 'female'] 1\n"
     ]
    }
   ],
   "source": [
    "print feature_names\n",
    "print titanic_X[12], titanic_y[12]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Class and sex are categorical classes. Sex can be converted to a binary value (0=female,1=male):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical classes: ['female' 'male']\n",
      "Integer classes: [0 1]\n"
     ]
    }
   ],
   "source": [
    "# Encode sex \n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "enc = LabelEncoder()\n",
    "label_encoder = enc.fit(titanic_X[:, 2])\n",
    "print \"Categorical classes:\", label_encoder.classes_\n",
    "integer_classes = label_encoder.transform(label_encoder.classes_)\n",
    "print \"Integer classes:\", integer_classes\n",
    "t = label_encoder.transform(titanic_X[:, 2])\n",
    "titanic_X[:, 2] = t\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['pclass' 'age' 'sex']\n",
      "['1st' '31.19418104265403' '0'] 1\n"
     ]
    }
   ],
   "source": [
    "print feature_names\n",
    "print titanic_X[12], titanic_y[12]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have to convert the class. Since we have three different classes, we cannot convert to binary values (and using 0/1/2 values would imply an order, something we do not want). We use OneHotEncoder to get three different attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical classes: ['1st' '2nd' '3rd']\n",
      "Integer classes: [[0]\n",
      " [1]\n",
      " [2]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "enc = LabelEncoder()\n",
    "label_encoder = enc.fit(titanic_X[:, 0])\n",
    "print \"Categorical classes:\", label_encoder.classes_\n",
    "integer_classes = label_encoder.transform(label_encoder.classes_).reshape(3, 1)\n",
    "print \"Integer classes:\", integer_classes\n",
    "enc = OneHotEncoder()\n",
    "one_hot_encoder = enc.fit(integer_classes)\n",
    "# First, convert clases to 0-(N-1) integers using label_encoder\n",
    "num_of_rows = titanic_X.shape[0]\n",
    "t = label_encoder.transform(titanic_X[:, 0]).reshape(num_of_rows, 1)\n",
    "# Second, create a sparse matrix with three columns, each one indicating if the instance belongs to the class\n",
    "new_features = one_hot_encoder.transform(t)\n",
    "# Add the new features to titanix_X\n",
    "titanic_X = np.concatenate([titanic_X, new_features.toarray()], axis = 1)\n",
    "#Eliminate converted columns\n",
    "titanic_X = np.delete(titanic_X, [0], 1)\n",
    "# Update feature names\n",
    "feature_names = ['age', 'sex', 'first_class', 'second_class', 'third_class']\n",
    "# Convert to numerical values\n",
    "titanic_X = titanic_X.astype(float)\n",
    "titanic_y = titanic_y.astype(float)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['age', 'sex', 'first_class', 'second_class', 'third_class']\n",
      "[29.  0.  1.  0.  0.] 1.0\n"
     ]
    }
   ],
   "source": [
    "print feature_names\n",
    "print titanic_X[0], titanic_y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Separate training and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\apps\\Anaconda2\\lib\\site-packages\\sklearn\\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(titanic_X, titanic_y, test_size=0.25, random_state=33)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit a decision tree with the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "clf = tree.DecisionTreeClassifier(criterion='entropy', max_depth=3,min_samples_leaf=5)\n",
    "clf = clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the built tree, using pydot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'list' object has no attribute 'write_png'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-15-f7ba3bc18af1>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mtree\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexport_graphviz\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mclf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mout_file\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdot_data\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfeature_names\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'age'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m'sex'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m'1st_class'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m'2nd_class'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m'3rd_class'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mgraph\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpydot\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgraph_from_dot_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdot_data\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mgraph\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwrite_png\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'titanic.png'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mIPython\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcore\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdisplay\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mImage\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[0mImage\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m'titanic.png'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'list' object has no attribute 'write_png'"
     ]
    }
   ],
   "source": [
    "import pydot,StringIO\n",
    "dot_data = StringIO.StringIO() \n",
    "tree.export_graphviz(clf, out_file=dot_data, feature_names=['age','sex','1st_class','2nd_class','3rd_class']) \n",
    "graph = pydot.graph_from_dot_data(dot_data.getvalue()) \n",
    "graph.write_png('titanic.png') \n",
    "from IPython.core.display import Image \n",
    "Image(filename='titanic.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measure Accuracy, precision, recall, f1 in the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.838 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "def measure_performance(X,y,clf, show_accuracy=True, show_classification_report=True, show_confusion_matrix=True):\n",
    "    y_pred=clf.predict(X)   \n",
    "    if show_accuracy:\n",
    "        print \"Accuracy:{0:.3f}\".format(metrics.accuracy_score(y,y_pred)),\"\\n\"\n",
    "\n",
    "    if show_classification_report:\n",
    "        print \"Classification report\"\n",
    "        print metrics.classification_report(y,y_pred),\"\\n\"\n",
    "        \n",
    "    if show_confusion_matrix:\n",
    "        print \"Confusion matrix\"\n",
    "        print metrics.confusion_matrix(y,y_pred),\"\\n\"\n",
    "        \n",
    "measure_performance(X_train,y_train,clf, show_classification_report=False, show_confusion_matrix=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform leave-one-out cross validation to better measure performance, reducing variance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import cross_val_score, LeaveOneOut\n",
    "from scipy.stats import sem\n",
    "\n",
    "def loo_cv(X_train,y_train,clf):\n",
    "    # Perform Leave-One-Out cross validation\n",
    "    # We are preforming 1313 classifications!\n",
    "    loo = LeaveOneOut(X_train[:].shape[0])\n",
    "    scores=np.zeros(X_train[:].shape[0])\n",
    "    for train_index,test_index in loo:\n",
    "        X_train_cv, X_test_cv= X_train[train_index], X_train[test_index]\n",
    "        y_train_cv, y_test_cv= y_train[train_index], y_train[test_index]\n",
    "        clf = clf.fit(X_train_cv,y_train_cv)\n",
    "        y_pred=clf.predict(X_test_cv)\n",
    "        scores[test_index]=metrics.accuracy_score(y_test_cv.astype(int), y_pred.astype(int))\n",
    "    print (\"Mean score: {0:.3f} (+/-{1:.3f})\").format(np.mean(scores), sem(scores))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.837 (+/-0.012)\n"
     ]
    }
   ],
   "source": [
    "loo_cv(X_train, y_train,clf)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to improve performance using Random Forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score: 0.817 (+/-0.012)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "clf = RandomForestClassifier(n_estimators=10,random_state=33)\n",
    "clf = clf.fit(X_train,y_train)\n",
    "loo_cv(X_train,y_train,clf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate performance on future data, evaluate on the training set and test on the evaluation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:0.793 \n",
      "\n",
      "Classification report\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.77      0.96      0.85       202\n",
      "        1.0       0.88      0.54      0.67       127\n",
      "\n",
      "avg / total       0.81      0.79      0.78       329\n",
      "\n",
      "\n",
      "Confusion matrix\n",
      "[[193   9]\n",
      " [ 59  68]] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "clf_dt=tree.DecisionTreeClassifier(criterion='entropy', max_depth=3,min_samples_leaf=5)\n",
    "clf_dt.fit(X_train,y_train)\n",
    "measure_performance(X_test,y_test,clf_dt)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}