PyData_vanilla_v3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How “good” is your model, and how can you make it better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What distinguishes “true artists” from “one-hit wonders” in machine learning is an understanding of how a model performs with respect to different data. This hands-on tutorial will show you how to use scikit-learn’s model evaluation functions to evaluate different models in terms of accuracy and generalisability, and search for optimal parameter configurations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import scipy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn.cross_validation as cv\n",
    "\n",
    "# Extra functionality\n",
    "import visplots\n",
    "\n",
    "from sklearn import preprocessing, metrics \n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.grid_search import GridSearchCV, RandomizedSearchCV\n",
    "from scipy.stats.distributions import randint\n",
    "\n",
    "from multilayer_perceptron import multilayer_perceptron\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring and pre-processing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing you will need to do in order to work with the WDBC dataset is to read the contents from the provided .csv data file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"wdbc.csv\", header=None)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the data have been converted into a numpy array, we need to construct the data matrix X and the associated class vector y (target variable). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Convert to numpy array\n",
    "dataArray = np.array(data)\n",
    "\n",
    "# Cast the input data into type float\n",
    "X = dataArray[:,2:].astype(float)\n",
    "y = dataArray[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is always a good practice to check the dimensionality of the imported data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print (\"X dimensions:\", X.shape)\n",
    "print (\"y dimensions:\", y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important thing to understand before applying any classification algorithms is how the output labels\n",
    "are distributed. Are they evenly distributed? Imbalances in distribution of labels can often lead to poor classification results for the minority class even if the classification results for the majority class\n",
    "are very good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "yFreq = scipy.stats.itemfreq(y)\n",
    "print (yFreq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our current dataset, the y labels are categorical (they can only take one of a discrete set of values) and have a non-numeric representation (\"M\" or \"B\"). This is problematic for scikit-learn algorithms, since they assume numerical values, so we need to map the text categories to numerical representations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "le = preprocessing.LabelEncoder()\n",
    "le.fit(y)\n",
    "yTransformed = le.transform(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is usually advisable to scale your data prior to fitting a classification model. The main advantage of scaling is to avoid attributes of greater numeric ranges dominating those in smaller numeric ranges. For the purposes of this case study, we are applying auto-scaling on the whole X dataset. (Auto-scaling: mean-centering is initially applied per column, followed by scaling where the centered columns are divided by their standard deviation). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Xauto = preprocessing.StandardScaler().fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can visualise the relationship between two variables (radius and perimeter of the tumour) using a simple scatter plot. To illustrate, let’s plot the first two variables against each other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 5))\n",
    "plt.scatter(Xauto[yTransformed==0, 0], Xauto[yTransformed==0, 1], edgecolors='black', color = 'b', label='Benign')\n",
    "plt.scatter(Xauto[yTransformed==1, 0], Xauto[yTransformed==1, 1], edgecolors='black', color = 'r', label='Malignant')\n",
    "plt.xlabel(\"Radius\")\n",
    "plt.ylabel(\"Perimeter\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the distribution of the samples that belong to each distinct class given a specific feature i.e. radius"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 5))\n",
    "plt.hist(Xauto[yTransformed==0,0], color='b', alpha=.5, label='Benign', bins=15)\n",
    "plt.hist(Xauto[yTransformed==1,0], color='r', alpha=.5, label='Malignant', bins=15)\n",
    "plt.xlabel(\"Radius\")\n",
    "plt.ylabel(\"Frequency\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and testing a classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and testing a classification model on the same dataset is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data (poor generalisation). <br/> \n",
    "\n",
    "To use different datasets for training and testing, we need to split the breast cancer dataset into two disjoint sets: train and test (**Holdout method**). <br/> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "XTrain, XTest, yTrain, yTest = cv.train_test_split(Xauto, yTransformed, test_size= 0.3, random_state=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XTrain and yTrain are the two arrays you use to train your model. XTest and yTest are the two arrays that you use to evaluate your model. By default, scikit-learn splits the data so that 25% of it is used for testing, but you can also specify the proportion of data you want to use for training and testing (in this case, 30% is used for testing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the sizes of the different training and test sets by using the shape attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print (\"XTrain dimensions:\", XTrain.shape)\n",
    "print (\"yTrain dimensions:\", yTrain.shape)\n",
    "print (\"XTest dimensions:\",  XTest.shape)\n",
    "print (\"yTest dimensions:\",  yTest.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K Nearest Neighbors (KNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build KNN models using scikit-learn, you will be using the `KNeighborsClassifier` function, which allows you to set the value of K using the `n_neighbors` parameter. The optimal choice of the value K is highly data-dependent: in general a larger K suppresses the effects of noise, but makes the classification boundaries less distinct. <br/>\n",
    "\n",
    "### Uniform weights\n",
    "\n",
    "We are going to start by trying two predefined random values of K and compare their performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "knn3 = KNeighborsClassifier(n_neighbors=3)\n",
    "knn3.fit(XTrain, yTrain)\n",
    "yPredK3 = knn3.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, yPredK3))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, yPredK3), 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "knn100 = KNeighborsClassifier(n_neighbors=100)\n",
    "knn100.fit(XTrain, yTrain)\n",
    "yPredK100 = knn100.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, yPredK100))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, yPredK100), 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualisation \n",
    "\n",
    "We can visualise the classification boundary created by the KNN classifier using the following function. For easier visualisation, only the test samples have been included in the plot. Remember that the decision boundary has been built using the _training_ data! <br/> \n",
    "\n",
    "For smaller values of K the decision boundaries present many \"creases\". In this case the models may suffer from instances of overfitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visplots.knnDecisionPlot(XTrain, yTrain, XTest, yTest, n_neighbors= 3, weights=\"uniform\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that for larger values of K the decision boundaries are less distinct and tend towards linearity. In these cases the boundaries may be too simple and unable to learn thus leading to underfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visplots.knnDecisionPlot(XTrain, yTrain, XTest, yTest, n_neighbors= 100, weights=\"uniform\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distance weights\n",
    "\n",
    "Under some circumstances, it is better to give more importance (\"weight\" in computing terms) to nearer neighbours. When weights = \"distance\", weights are assigned to the training data points in a way that is proportional to the inverse of the distance from the query point. In other words, nearer neighbours contribute more to the fit. <br/>\n",
    "\n",
    "What if we use weights based on distance? Does it improve the overall performance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "knnW3 = KNeighborsClassifier(n_neighbors=3, weights='distance')\n",
    "knnW3.fit(XTrain, yTrain)\n",
    "predictedW3 = knnW3.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, predictedW3))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, predictedW3), 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than trying one-by-one predefined values of K, we can automate this process. The following function plots the accuracies of applying the KNN algorithms with different K values for both uniform and distance weights. Remember that only odd numbers are used for K to avoid ties during majority voting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(7, 6))\n",
    "visplots.plotaccuracy(XTrain, yTrain, XTest, yTest, 200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the plot, you can see that the classifier performs well throughout the selected range of K but when K is too large (K > 100) the accuracy starts to decrease when using uniform weights. When using weights = \"distance\"\n",
    "we essentially assign greater weights to neighbours that are closer in distance so that these neighbours contribute more to the fit. This reduces the impact of increasing K on the accuracy and the prediction becomes much less\n",
    "dramatic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning KNN parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sklearn library has a grid search function, `GridSearchCV`, that allows us to search for the optimum\n",
    "combination of parameters by evaluating models trained with a particular algorithm with all provided parameter combinations. You can use this function to search for a parametisation of the KNN algorithm\n",
    "that gives a more optimal model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We want to use odd numbers of K to avoid ties\n",
    "n_neighbors = np.arange(1, 50, 2)\n",
    "weights     = ['uniform','distance']\n",
    "\n",
    "# GridSearchCV accepts parameter values  only as a dictionary\n",
    "parameters = [{'n_neighbors': n_neighbors, 'weights': weights}]\n",
    "grid = GridSearchCV(KNeighborsClassifier(), parameters, cv= 10)\n",
    "grid.fit(XTrain, yTrain)\n",
    "\n",
    "print (\"The best parameters are: n_neighbors=\", grid.best_params_['n_neighbors'],\"and weight=\",grid.best_params_['weights'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/> Let us graphically represent these results using a heatmap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# plot the scores of the grid\n",
    "# grid_scores_ contains parameter settings and scores\n",
    "score_dict = grid.grid_scores_\n",
    "scores = [x[1] for x in score_dict]\n",
    "scores = np.array(scores).reshape(len(n_neighbors), len(weights))\n",
    "scores = np.transpose(scores)\n",
    "\n",
    "# Make a heatmap with the performance\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.imshow(scores, interpolation='nearest', origin='higher', cmap=plt.cm.get_cmap('jet_r'))\n",
    "plt.xticks(np.arange(len(n_neighbors)), n_neighbors)\n",
    "plt.yticks(np.arange(len(weights)), weights)\n",
    "plt.xlabel('Number of K nearest neighbors')\n",
    "plt.ylabel('Weights')\n",
    "\n",
    "cbar = plt.colorbar()\n",
    "cbar.set_label('Classification Accuracy', rotation=270, labelpad=20)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When evaluating the resulting model it is important to do it on held-out samples that were not seen during the grid search process (XTest). <br/> So, we are testing our independent XTest dataset using the optimised model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=grid.best_params_['n_neighbors'], weights = grid.best_params_['weights'])\n",
    "knn.fit(XTrain, yTrain)\n",
    "yPredKnn = knn3.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, yPredKnn))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, yPredKnn), 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Randomized search on hyperparameters. Unlike `GridSearchCV`, `RandomizedSearchCV` does not exhaustively try all the parameter settings. Instead, it samples a fixed number of parameter settings from the specified distributions. The number of parameter settings that are tried is given by `n_iter`. If all parameters are presented as a list, sampling without replacement is performed. If at least one parameter is given as a distribution, sampling with replacement is used. You should use continuous distributions for continuous parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "param_dist = {'n_neighbors': randint(1,200)}\n",
    "random_search = RandomizedSearchCV(KNeighborsClassifier(), param_distributions=param_dist, n_iter=20)\n",
    "random_search.fit(XTrain, yTrain)\n",
    "\n",
    "print (\"The best parameters are: n_neighbors=\", random_search.best_params_['n_neighbors'])\n",
    "\n",
    "neig = [score_tuple[0]['n_neighbors'] for score_tuple in random_search.grid_scores_]\n",
    "res = [score_tuple[1] for score_tuple in random_search.grid_scores_]\n",
    "plt.scatter(neig, res)\n",
    "plt.xlabel('Number of K nearest neighbors')\n",
    "plt.ylabel('Classification Accuracy')\n",
    "plt.xlim(0,200)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support Vector Machines (SVMs)\n",
    "\n",
    "SVMs attempt to build a decision boundary that accurately separates the samples of different classes by *maximizing* the margin between them.\n",
    "\n",
    "### Linear SVMs\n",
    "\n",
    "The parameter C, common to all SVM kernels, trades off misclassification of training examples against simplicity of the decision surface. A low C tolerates training misclassifications and allows softer margins, while for high C the misclassifications become more significant leading to hard-margin SVMs and potentially cases of overfitting. \n",
    "\n",
    "In this example, we will use linear SVMs with the default value for C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Default C parameter\n",
    "linearSVM = SVC(kernel='linear')\n",
    "linearSVM.fit(XTrain, yTrain)\n",
    "yPredLinear = linearSVM.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, yPredLinear))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, yPredLinear),2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualise the classification boundary created by the linear SVM using the following function. For easier visualisation, only the test samples have been included in the plot. And remember that the decision boundary has been built using the _training_ data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visplots.svmDecisionPlot(XTrain, yTrain, XTest, yTest, 'linear')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Non-linear SVMs\n",
    "\n",
    "In addition to C, which is common for all types of SVM, the gamma parameter in the RBF kernel controls the nonlinearity of the SVM bounaries. The larger the gamma, the more nonlinear the boundaries surrounding individual samples. Lower values of gamma lead to broader, more linear boundaries. <br/><br/>  In this example, we will use non-linear SVMs with the default values for C and gamma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Default C and gamma parameters\n",
    "rbfSVM = SVC(kernel='rbf', C=1.0, gamma=0.0)\n",
    "rbfSVM.fit(XTrain, yTrain)\n",
    "yPredRBF = rbfSVM.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, yPredRBF))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, yPredRBF),2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualise the classification boundary created by the RBF SVM using the following function. Once more, for easier visualisation, only the test samples have been included in the plot. And remember that the decision boundary has been built using the _training_ data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visplots.svmDecisionPlot(XTrain, yTrain, XTest, yTest, 'rbf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning\n",
    "\n",
    "Proper choice of C and gamma is critical for the performance of SVMs. Optimisation (tuning) of the hyperparameters can be achieved by applying a coarse tuning (often followed by a finer-tuning in the \"neighborhood\" of good parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Range for gamma and Cost hyperparameters\n",
    "g_range = 2. ** np.arange(-15, 5, step=2)\n",
    "C_range = 2. ** np.arange(-5, 15, step=2)\n",
    "\n",
    "grid = [{'gamma': g_range, 'C': C_range}]\n",
    "\n",
    "gridcv = GridSearchCV(SVC(), param_grid=grid, cv= cv.KFold(n=XTrain.shape[0], n_folds=5))\n",
    "gridcv.fit(XTrain, yTrain)\n",
    "\n",
    "bestG = np.log2(gridcv.best_params_['gamma']);\n",
    "bestC = np.log2(gridcv.best_params_['C']);\n",
    "\n",
    "print (\"The best parameters are: gamma=\", bestG, \" and Cost=\", bestC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the results of the grid search using a heatmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# plot the scores of the grid\n",
    "# grid_scores_ contains parameter settings and scores\n",
    "score_dict = gridcv.grid_scores_\n",
    "scores = [x[1] for x in score_dict]\n",
    "scores = np.array(scores).reshape(len(C_range), len(g_range))\n",
    "\n",
    "# Make a heatmap with the performance\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.subplots_adjust(left=0.15, right=0.95, bottom=0.15, top=0.95)\n",
    "plt.imshow(scores, interpolation='nearest', origin='higher', cmap=plt.cm.get_cmap('jet_r'))\n",
    "plt.xlabel('gamma (log2)')\n",
    "plt.ylabel('Cost (log2)')\n",
    "plt.xticks(np.arange(len(g_range)), np.log2(g_range))\n",
    "plt.yticks(np.arange(len(C_range)), np.log2(C_range))\n",
    "\n",
    "cbar = plt.colorbar()\n",
    "cbar.set_label('Classification Accuracy', rotation=270, labelpad=20)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, testing with the optimised model (best hyperparameters):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rbfSVM = SVC(kernel='rbf', C=gridcv.best_params_['C'], gamma=gridcv.best_params_['gamma'])\n",
    "rbfSVM.fit(XTrain, yTrain)\n",
    "\n",
    "predictions = rbfSVM.predict(XTest) \n",
    "\n",
    "print (metrics.classification_report(yTest, predictions))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, predictions),2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression\n",
    "\n",
    "Logistic regression predicts the probability that a sample belongs to a class based on the values of the input variables, based on a linear model. In the case of classification, we can use this to then assign the sample to the most likely class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "l_regression = LogisticRegression()\n",
    "l_regression.fit(XTrain, yTrain)\n",
    "net_prediction = l_regression.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, net_prediction))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, net_prediction),2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualise the classification boundary created by the logistic regression model using the built in visualisation function `logregDecisionPlot`. As with the above examples, only the test samples have been included in the plot. And remember that the decision boundary has been built using the _training_ data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visplots.logregDecisionPlot(XTrain, yTrain, XTest, yTest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two hyperparameters that are often tuned for logistic regression models are the norm used in penalisation (`penalty`), which can be either `l1` or `l2` (default `l2`) and the inverse of regularisation strength, `C` (default `1.0`). \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Range for gamma and Cost hyperparameters\n",
    "pen = ['l1','l2']\n",
    "C_range = 2. ** np.arange(-5, 15, step=2)\n",
    "\n",
    "grid = [{'C': C_range, 'penalty': pen}]\n",
    "\n",
    "gridcv = GridSearchCV(LogisticRegression(), param_grid=grid, cv= cv.KFold(n=XTrain.shape[0], n_folds=5))\n",
    "gridcv.fit(XTrain, yTrain)\n",
    "\n",
    "best_c = gridcv.best_params_['C']\n",
    "best_penalty = gridcv.best_params_['penalty']\n",
    "\n",
    "print (\"The best parameters are: cost=\", best_c, \" and penalty=\", best_penalty)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try these out to see how the performance metrics are affected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "l_regression = LogisticRegression(C=0.5, penalty='l2')\n",
    "l_regression.fit(XTrain, yTrain)\n",
    "net_prediction = l_regression.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, net_prediction))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, net_prediction),2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the results of the grid search with a heatmap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# plot the scores of the grid\n",
    "# grid_scores_ contains parameter settings and scores\n",
    "score_dict = gridcv.grid_scores_\n",
    "scores = [x[1] for x in score_dict]\n",
    "scores = np.array(scores).reshape(len(pen), len(C_range))\n",
    "scores = np.transpose(scores)\n",
    "\n",
    "# Make a heatmap with the performance\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.imshow(scores, interpolation='nearest', origin='higher', cmap=plt.cm.get_cmap('jet_r'))\n",
    "plt.xticks(np.arange(len(pen)), pen)\n",
    "plt.yticks(np.arange(len(C_range)), C_range)\n",
    "plt.xlabel('penalisation norm')\n",
    "plt.ylabel('inv regularisation strength')\n",
    "\n",
    "cbar = plt.colorbar()\n",
    "cbar.set_label('Classification Accuracy', rotation=270, labelpad=20)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more details on cross-validating and tuning logistic regression models, see: <br/>\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegressionCV.html\n",
    "and <br/>\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Networks\n",
    "\n",
    "A neural network is a set of connected input-output units. During training, the connections are assigned different weights. This allows the classification function to take on highly complex \"shapes\" (equivalent to complicated mathematical expressions that go beyond the linear or polynomial models of logistic regression). This might also mean that the resulting model is difficult to interpret and map to domain knowledge. (NB. even though you might think of the second layer of a neural network as just a logistic regression model, the non-linear transformation in the hidden units gives the input to output mapping a non-linear decision boundary.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nnet = multilayer_perceptron.MultilayerPerceptronClassifier(activation='logistic',\n",
    "                                                            hidden_layer_sizes=2, learning_rate_init=.5)\n",
    "nnet.fit(XTrain, yTrain)\n",
    "net_prediction = nnet.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, net_prediction))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, net_prediction),2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualise the classification boundary created by the neural network using the built in visualisation function `nnDecisionPlot`. As with the above examples, only the test samples have been included in the plot. And remember that the decision boundary has been built using the _training_ data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "visplots.nnDecisionPlot(XTrain, yTrain, XTest, yTest, 2, .5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visplots.nnDecisionPlot(XTrain, yTrain, XTest, yTest, (2,3,6), .5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural networks have many hyperparameters, all of which could potentially be tuned, including learning rate, loss function, number of training iterations, number of hidden layers and number of units within each of them, nonlinearity function, and weight initialisation. \n",
    "\n",
    "Here's a worked through example which explores the set of parameter configurations with different numbers of hidden layers and units within them (`hidden_layer_sizes`), and learning rates (`learning_rate_init`).\n",
    "\n",
    "Note the syntax to specify the number of hidden layers and units with them. If a tuple is given, each value in the tuple stands for the number of units in a layer, e.g. the tuple `(2,3,4)` would mean a network with two units in the first layer, three units in the second, and four in the third. If a single value is given, then there is only one hidden layer, and the value stands for the number of units in this layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Range for gamma and Cost hyperparameters\n",
    "layer_size_range = [(3,2),(10,10),(2,2,2),10,5] # different networks shapes\n",
    "learning_rate_range = np.linspace(.1,1,3)\n",
    "\n",
    "grid = [{'hidden_layer_sizes': layer_size_range, 'learning_rate_init': learning_rate_range}]\n",
    "\n",
    "gridcv = GridSearchCV(multilayer_perceptron.MultilayerPerceptronClassifier(), \n",
    "                      param_grid=grid, cv= cv.KFold(n=XTrain.shape[0], n_folds=5))\n",
    "gridcv.fit(XTrain, yTrain)\n",
    "\n",
    "best_size = gridcv.best_params_['hidden_layer_sizes']\n",
    "best_best_lr = gridcv.best_params_['learning_rate_init']\n",
    "\n",
    "print (\"The best parameters are: hidden_layer_sizes=\", best_size, \" and learning_rate_init=\", best_best_lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try these out to see how the performance metrics are affected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nnet = multilayer_perceptron.MultilayerPerceptronClassifier(hidden_layer_sizes=best_size, learning_rate_init=best_best_lr)\n",
    "nnet.fit(XTrain, yTrain)\n",
    "net_prediction = nnet.predict(XTest)\n",
    "\n",
    "print (metrics.classification_report(yTest, net_prediction))\n",
    "print (\"Overall Accuracy:\", round(metrics.accuracy_score(yTest, net_prediction),2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Plot the results of the grid search using a heatmap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# plot the scores of the grid\n",
    "# grid_scores_ contains parameter settings and scores\n",
    "score_dict = gridcv.grid_scores_\n",
    "scores = [x[1] for x in score_dict]\n",
    "scores = np.array(scores).reshape(len(layer_size_range), len(learning_rate_range))\n",
    "scores = np.transpose(scores)\n",
    "\n",
    "# Make a heatmap with the performance\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.imshow(scores, interpolation='nearest', origin='higher', cmap=plt.cm.get_cmap('jet_r'))\n",
    "plt.xticks(np.arange(len(layer_size_range)), layer_size_range)\n",
    "plt.yticks(np.arange(len(learning_rate_range)), learning_rate_range)\n",
    "plt.xlabel('hidden layer topology')\n",
    "plt.ylabel('learning rate')\n",
    "\n",
    "cbar = plt.colorbar()\n",
    "cbar.set_label('Classification Accuracy', rotation=270, labelpad=20)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "- *So, what is your best technique and why?*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}