CalibratedAdaMEC_ExtendedVersion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Skew-Sensitive Boosting & Calibrated AdaMEC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook provides a template implementation of the *Calibrated AdaMEC* method proposed in:\n",
    "\n",
    "*Cost-sensitive boosting algorithms: Do we really need them?  \n",
    "Nikolaos Nikolaou, Narayanan U. Edakunni, Meelis Kull, Peter A. Flach, Gavin Brown  \n",
    "Machine Learning, 104(2), pages 359-384, 2016*  \n",
    "[[Google Scholar]](https://scholar.google.co.uk/citations?view_op=view_citation&hl=el&user=R6b6Rp8AAAAJ&citation_for_view=R6b6Rp8AAAAJ:UeHWp8X0CEIC) [[BibTex]](http://www.cs.man.ac.uk/%7Enikolaon/%7Enikolaon_files/mlj2016_bib.bib).\n",
    "\n",
    "__If you are just looking for a simplified, ready-to-run, version of the code, please [go to this link](http://www.cs.man.ac.uk/~gbrown/costsensitiveboosting/), the implementation we provide [here](#Calibrated-AdaMEC-Code) is less user-friendly, but more flexible. If you make use of this either version of the code, please cite the above paper__.\n",
    "\n",
    "We included a tutorial to introduce the user to the concepts of *'skew-sensitive' classification* (a.k.a. *'asymmetric' classification*, concerned with tasks in which the *class prevalences* and/or the *misclassification costs* for different types of misclassifications are unequal) and *'calibration'*. We also give the basic concepts behind *AdaBoost* and briefly discuss the issues with the existing *skew-sensitive AdaBoost algorithms*. __For the tutorial itself, continue reading [here](Cost-Sensitive-&-Imbalanced-Class-Learning)__. For more details, we direct the reader to the paper.\n",
    "\n",
    "Finally, we allow the user to compare the standard AdaBoost algorithm against *calibrated AdaBoost using a skew-sensitive decision threshold* (*Calibrated AdaMEC* in the paper), for the purposes of skew-sensitive learning. __To perform the comparisons you need to execute the piece of code found [here](#Compare-AdaBoost-vs-Calibrated-AdaMEC)__. *Calibrated AdaMEC aims to minimize the expected misclassification cost (risk), for binary classification with given false negative & false positive costs.* As a result, it is expected to outperform AdaBoost.\n",
    "\n",
    "Users are encouraged to experiment with different datasets, cost setups, weak learners, calibration methods, calibration/training splits, ensemble sizes and the choice of AdaBoost version (discrete/real).\n",
    "\n",
    "Please, direct any questions or feedback to: __nikolaos.nikolaou[at]manchester.ac.uk__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost Sensitive & Imbalanced Class Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose -for simplicity- that we are given a binary classification task. More often than not, such a task is in some sense *'skewed'* or *'asymmetric'*. What this means is that the two types of mistakes a binary classifier can commit, *False Positives (FP)* and *False Negatives (FN)* are (i) not equally likely, and/or (ii) they are not equally costly. Simply put, usually one class is simply *'more important'* than the other.\n",
    "\n",
    "The *goal of the classifier* is therefore, generally, not just to minimize the (expected) number of misclassifications, but to *minimize the (expected) cost of misclassifications*, also known as the *risk*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example Application: Medical Diagnosis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take for example the problem of a patient being tested for a disease. There are two types of errors the doctor examining them can make:\n",
    " \n",
    "FP: \"predict that the patient has the disease, when they don't\"    \n",
    "FN: \"predict that the patient doesn't have the disease, when they do\"  \n",
    "\n",
    "The cost of these two types of mistakes is not equal. A FN is potentially fatal if the disease is life-threatenning. On the oter hand, a FP will cause some distress to the patient, but they will undergo subsequent tests that will eventually reveal the misdiagnosis. The situation can of course be reversed: a treatment resulting from a FP might have severe adverse effects that are best avoided unless the doctor is pretty sure the patient needs it.\n",
    "\n",
    "To complicate matters more, the *prior probability* of having a disease is typically lower than that of not having it. \n",
    "\n",
    "The doctor is thus faced with an asymmetric binary classification problem. They need to take a *minimum-risk decision*, weighing what they know about the prevalence of the disease in the general population, how likely the patient is to have the disease given their symptoms and what the costs of the two types of misclassifications are."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some Decision Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Formalizing the above, as a *combined measure of cost and class imbalance* we can use the *skew*: $z = \\frac{C_{FP} \\cdot Neg}{C_{FN} \\cdot Pos + C_{FP} \\cdot Neg}$, where $C_{FP}$ & $C_{FN}$ are the *costs of a single FP* and *a single FN*, respectively. $Pos$ & $Neg$ are the *numbers of positives* and *negatives* on the *training* set, respectively --here they are practically being used to *estimate the frequency of occurence of positives and negatives on the test set*. Thus, $z$ gives us an overall measure of *importance* of the negative class. \n",
    "\n",
    "It can be shown that the *minimum expected cost (minimum risk) decision rule* is:  \n",
    "*\"Assign an example ${\\mathbf{x}}$ to the positive class if $p(y=1|{\\mathbf{x}})> z$, else predict the negative class.\" (1)*  \n",
    "\n",
    "[Note: for simplicity, the case of $p(y=1|{\\mathbf{x}}) = z$, in which the classifier should *abstain from decision* is ignored.]\n",
    "\n",
    "Now, $p(y=1|{\\mathbf{x}})$, i.e. *the probability of the example ${\\mathbf{x}}$ being positive*, is usually unknown so in its place we use an *estimate* $\\hat{p}(y=1|{\\mathbf{x}})$ generated by the classifier. The problem is that the *quality* of such estimates as obtained by classifiers is often *questionable*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classifier calibration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A classifier is said to be *calibrated* if *the empirical probability of an example to which the classifier assigns a probability estimate $\\hat{p}(y=1|{\\mathbf{x}})= p$ being positive, $\\tilde{p}(y=1|{\\mathbf{x}}) = \\frac{N_{+,p}}{N_{p}}$, where $N_{+,p}$ is the number of positive examples with $\\hat{p}(y=1|{\\mathbf{x}}) = p$ and $N_{p}$ the total number of examples with $\\hat{p}(y=1|{\\mathbf{x}}) = p$, tends to the value of the estimate $p$ as the sample size grows to infinity, for all values of $p$*. For example, 65% of the examples with $\\hat{p}(y=1|{\\mathbf{x}}) = 0.65$ must be positive.\n",
    "\n",
    "Most classifiers produce *uncalibrated* probability estimates. Each model's estimates are *distorted* in ways specific to its *inductive bias*. For instance *Boosting*, the family of methods that are of interest here, tends to produce estimates that are *too extreme*, i.e. $\\hat{p}(y=1|{\\mathbf{x}}) \\rightarrow 0$ or $\\hat{p}(y=1|{\\mathbf{x}}) \\rightarrow 1$. It is obvious that such uncalibrated probability estimates -usually refered to as *'scores'* produced by the classifier on a given example ${\\mathbf{x}}$ - might be sufficient for *ranking* examples, saying whether one is *'deemed more positive'* than another, but they are *not enough when cost-sensitive decisions need to be made*, like *(1)*. In these situations we need our probability estimates to be as close to actual class probabilities as possible.\n",
    "\n",
    "*Calibration* consists of *correcting for such distortions by mapping probability estimates $\\hat{p}(y=1|{\\mathbf{x}})$ to actual empirical class probabilities $\\tilde{p}(y=1|{\\mathbf{x}})$*. This mapping is done on a held-out part of the training data (*calibration set*) to avoid overfitting. The two most popular calibration methods are *logistic calibration* (also known as *'Platt scaling'*) and *isotonic regression*. The former fits a sigmoid mapping $\\hat{p}(y=1|{\\mathbf{x}}) \\rightarrow \\tilde{p}(y=1|{\\mathbf{x}}$), so can correct distortions of a *sigmoid* shape, the latter is *non-parametric* and can correct any form of *non-decreasing* distortion.\n",
    "\n",
    "Typically isotonic regression, being non-parametric, produces *better probability estimates, but needs more data as it can overfit more easily*.\n",
    "\n",
    "In the paper we opted for logistic calibration as a proof of concept, but here we allow the user to use either, so we shall not restrict the discussion.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AdaBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*AdaBoost (Adaptive Boosting)* and the more general family of *Boosting algorithms* it is part of, is a deeply studied, practically successful and widely used learning algorithm. It is an *ensemble learning algorithm* that aims to construct a *strong predictor* -one of arbitrarily high accuracy- by *sequentially* combining multiple *weak* ones -models that are marginally more accurate than random guessing. Each successive model is constructed by *putting increased weight on training examples that the previous model misclassified*.\n",
    "\n",
    "Each *base model* (weak model) is then assigned a *confidence coefficient* -models having *lower error* also have *higher confidence*- and predictions on a new example are made by a *weighted vote* among the base models, the *weights being their associated confidence coefficients*.\n",
    "\n",
    "The problem is that *AdaBoost is skew-insensitive*, i.e. it *does not differentiate between positives and negatives*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Existing Skew-Sensitive Boosting Algorithms "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many efforts to make *skew-sensitive* can be found in the literature. In the [paper](https://scholar.google.co.uk/citations?view_op=view_citation&hl=el&user=R6b6Rp8AAAAJ&citation_for_view=R6b6Rp8AAAAJ:UeHWp8X0CEIC) we identify *a set of properties that such an algorithm should satisfy*. Without getting into too much detail, these are:  \n",
    "\n",
    "(i) *Its steps should be consistent with one another.*  \n",
    "(ii) *It should classify examples to the minimum expected cost class (i.e. acording to *(1)* above).*  \n",
    "(iii) *It should not flip the class importance during training.*  \n",
    "(iv) *It should produce calibrated probability estimates.*  \n",
    "\n",
    "We found that *no existing proposal satisfies all four of them*.   \n",
    "\n",
    "*A key issue with all existing methods is that their probability estimates are poorly calibrated.* AdaBoost might construct a strong classifier, but it simply is too confident in its assessment of how positive or negative an example is. *Once we account for this issue and properly calibrate the probability estimates of a boosting variant, we found that there are three variants that are consistent with all properties above.*\n",
    "\n",
    "Below we present the most *intuitive* and *versatile*: *Calibrated AdaMEC* "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calibrated AdaMEC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Calibrated AdaMEC* is the standard AdaBoost algorithm with *two* simple modifications:\n",
    "  \n",
    "(i) *During training: Reserve part of the dataset to calibrate the probability estimates of the ensemble. This improves the quality of the probability estimates.*  \n",
    "(ii) *When making predictions: Shift the decision threshold to match the skew as per *(1)*. This makes the method skew-sensitive.*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! You just turned AdaBoost into a skew-sensitive risk minimizer! And did so without adding any serious computational cost or any hyperparameters that need to be fine-tuned and without restraining in any way the flexibility of the original algorithm. Moreover, if the skew changes during deployment, there is no need to retrain your ensemble: Just set the decision threshold the new value of the skew."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calibrated AdaMEC Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we provide code for training a calibrated AdaBoost ensemble and making predictions using the minimum expected cost (minimum risk) decision rule, for known costs of FNs and FPs. In other words, these are the Calibrated AdaMEC training and prediction functions. If you make use of this code, please cite [Nikolaou et al 2016, \"Cost Sensitive Boosting Algorithms: do we really need them?\"](https://scholar.google.co.uk/citations?view_op=view_citation&hl=el&user=R6b6Rp8AAAAJ&citation_for_view=R6b6Rp8AAAAJ:UeHWp8X0CEIC).\n",
    "\n",
    "For the purposes of comparing Calibrated AdaMEC to AdaBoost, you do not need to execute this code, as it is provided in the separate file CalibratedAdaMEC.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def trainCalibratedAdaMEC(base_estimator, algorithm, n_estimators, calibration_method, cal_set_prcnt, X_train, y_train):\n",
    "    \"\"\"Train a Calibrated AdaBoost ensemble. \n",
    "    \n",
    "    Parameters:\n",
    "        \n",
    "    base_estimator: object, sklearn supported classifier, base learner to be used\n",
    "                    e.g. decision stump: 'tree.DecisionTreeClassifier(max_depth=1)'\n",
    "                    remember to import the relevant packages if other base learner\n",
    "                    is used\n",
    "    \n",
    "    algorithm: string, possible values: {'SAMME', 'SAMME.R'} AdaBoost algorithm:\n",
    "               \"SAMME\" for discrete AdaBoost, \"SAMME.R\" for real AdaBoost\n",
    "        \n",
    "    n_estimators: integer,  AdaBoost ensemble (maximum) size \n",
    "    \n",
    "    calibration_method: string, possible values: {'isotonic', 'sigmoid'} \n",
    "                        AdaBoost ensemble score calibration method (isotonic\n",
    "                        regression or Platt scaling (logistic calibration), resp.)\n",
    "                        \n",
    "    cal_set_prcnt: float in (0, 1), size of calibration set as a percentage of the\n",
    "                   training dataset size\n",
    "    \n",
    "    X_train: array-like, shape (n_samples, n_features), training data\n",
    "    \n",
    "    y_train: array-like, shape (n_samples,), training labels\n",
    "    \n",
    "    Returns: \n",
    "        \n",
    "    AdaBoostCal: object, a calibrated adaboost classifier\n",
    "    \"\"\"\n",
    "    #First, reserve part of the training data for calibration\n",
    "    X_train, X_cal, y_train, y_cal = train_test_split(X_train, y_train, test_size=cal_set_prcnt)\n",
    "\n",
    "    #Train an AdaBoost ensemble\n",
    "    AdaBoostUncal = AdaBoostClassifier(weak_learner, algorithm=algorithm, n_estimators=n_estimators)\n",
    "    AdaBoostUncal = AdaBoostUncal.fit(X_train, y_train)\n",
    " \n",
    "    #Now calibrate the ensemble on the data reserved for calibration\n",
    "    #cv=\"prefit\" means that the model is already fitted and only needs calibration\n",
    "    AdaBoostCal = CalibratedClassifierCV(AdaBoostUncal, cv=\"prefit\", method=calibration_method)\n",
    "    AdaBoostCal.fit(X_cal, y_cal)\n",
    "\n",
    "    return AdaBoostCal\n",
    "\n",
    " \n",
    "def predictCalibratedAdaMEC(CalibratedAdaBoostClassifier, C_FP_effective, X_test):\n",
    "    \"\"\"Output AdaMEC (AdaBoost with shifted decision threshold) predictions\n",
    "\n",
    "          Parameters:\n",
    "\n",
    "          CalibratedAdaBoostClassifier: object, a calibrated classifier object as\n",
    "                                        returned by trainCalibratedAdaMEC()\n",
    "\n",
    "          threshold: float in (0, 1), the classification threshold to be compared with\n",
    "                     the probability estimate in order to classify an example to the\n",
    "                     positive class. For minimum risk classification, it should be set\n",
    "                     equal to the skew (overall asymmetry due to both class and cost\n",
    "                     imbalance),i.e. C_FP*Neg / (C_FN*Pos + C_FP*Neg), where Pos and Neg\n",
    "                     is the number of positive and negative examples, respectively, of the\n",
    "                     training set\n",
    "                     Note: We chose to have this as an argument, as the user might want to\n",
    "                     adjust it to account for a change in costs and/or prior probability shift\n",
    "\n",
    "          X_test: array-like, shape (n_samples, n_features), test data\n",
    "\n",
    "          Returns:\n",
    "\n",
    "          y_pred_CalibratedAdaMEC: array-like, shape (n_samples), predicted classes on\n",
    "                         training data\n",
    "\n",
    "          scores_CalibratedAdaMEC: array-like, shape (n_samples), predicted scores (i.e.\n",
    "                                   calibrated probability estimates) for the positive\n",
    "                                   class for the training data \n",
    "    \n",
    "     \"\"\"    \n",
    "    scores_CalibratedAdaMEC = CalibratedAdaBoostClassifier.predict_proba(X_test)[:,1]#Positive Class scores\n",
    " \n",
    "    y_pred_CalibratedAdaMEC = np.zeros(X_test.shape[0])\n",
    "    y_pred_CalibratedAdaMEC[np.where(scores_CalibratedAdaMEC > threshold)] = 1#Classifications, AdaMEC uses a shifted decision threshold (skew-sensitive) \n",
    "\n",
    "    return (y_pred_CalibratedAdaMEC, scores_CalibratedAdaMEC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare AdaBoost vs Calibrated AdaMEC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we provide code to compare the standard AdaBoost algorithm against Calibrated AdaMEC, for the purposes of probability estimation and skew-sensitive classification.\n",
    "\n",
    "To compare the two methods, simply call the function AdaMECvsAdaBoost() with appropriate arguments. Below we explain what options we have:  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__dataset__: *string*, provided UCI datasets: {*\"survival\"*, *\"ionosphere\"*, *\"parkinsons\"*, *\"krvskp\"*, *\"liver\"*, *\"pima\"*, *\"musk2\"*, *\"congress_orig\"*, *\"landsatM\"*, *\"wdbc_orig\"*, *\"heart_orig\"*, *\"sonar_orig\"*, *\"spliceM\"*, *\"german_credit\"*, *\"semeion_orig\"*, *\"waveformM\"*, *\"spambase\"*, *\"mushroom\"*}  \n",
    "  \n",
    "__C_FP__: *float*, cost of a false positive  \n",
    "\n",
    "__C_FN__: *float*, cost of a false negative  \n",
    "\n",
    "[Note: Remember that the costs above only capture one aspect of the asymmetry. Each dataset has a different degree of class imbalance. The combined measure of cost & class imbalance (skew) is captured by $z = \\frac{C_{FP} \\cdot Neg}{C_{FN} \\cdot Pos + C_{FP} \\cdot Neg}$. It is $z$ that determines the decision threshold in *(1)*.]\n",
    "\n",
    "__base_estimator__: *object*, sklearn supported classifier, base learner to be used; e.g. for a decision stump this will be: *'tree.DecisionTreeClassifier(max_depth=1)'*  \n",
    "[Note: Remember to import the relevant package if some other base learner is used!]  \n",
    "\n",
    "__algorithm__: *string*, possible values: {*'SAMME'*, *'SAMME.R'*} AdaBoost algorithm: discrete or real AdaBoost, respectively.\n",
    "\n",
    "__n_estimators__: *integer in $\\mathcal{N^+}$*,  AdaBoost ensemble (maximum) size \n",
    "\n",
    "__calibration_method__: *string*, possible values: {*'isotonic'*, *'sigmoid'*} AdaBoost ensemble score calibration method --isotonic regression or Platt scaling (i.e. logistic calibration), respectively\n",
    "\n",
    "__test_set_prcnt__: *float in (0, 1)*, size of test set as a percentage of the *original* dataset size\n",
    "\n",
    "__cal_set_prcnt__: *float in (0, 1)*, size of calibration set as a percentage of the *training* dataset size\n",
    "\n",
    "\n",
    "All is set! Simply change the code below and run it (click on it and press Shift+Enter) to see how the two methods compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Brier Score:\n",
      "\t\t\tAdaBoost: 0.21305052218831994\n",
      "\t\t\tCalibrated AdaMEC: 0.0036629905148640616\n",
      "Negative Log-likelihood:\n",
      "\t\t\tAdaBoost: 0.6189783063692328\n",
      "\t\t\tCalibrated AdaMEC: 0.01199219380918814\n",
      "Misclassification Cost:\n",
      "\t\t\tAdaBoost: 18.626214640466422\n",
      "\t\t\tCalibrated AdaMEC: 2.5255884258259558\n",
      "Calibrated AdaMEC outperformed AdaBoost!\n"
     ]
    }
   ],
   "source": [
    "import compare# Contains script that compares the two methods\n",
    "\n",
    "#Compare the two methods (feel free to experiment--see parameters above)\n",
    "#compare.AdaMECvsAdaBoost(dataset, C_FP, C_FN, \n",
    "#                         base_estimator, algorithm, n_estimators,\n",
    "#                         calibration_method, test_set_prcnt, cal_set_prcnt)\n",
    "compare.AdaMECvsAdaBoost('mushroom', 2, 1, \n",
    "                         'tree.DecisionTreeClassifier(max_depth=1)', 'SAMME', 50,\n",
    "                         'sigmoid', 0.25, 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*More likely than not, Calibrated AdaMEC outperformed AdaBoost, not only in terms of probability estimation (lower Brier score and negative log-likelihood) but also in terms of skew-sensitive classification (lower misclassification cost).* In case this did not happen, run the comparison again. In expectation Calibrated AdaMEC should produce better results.  \n",
    "  \n",
    "Still, this might not happen, for *two main reasons*:    \n",
    "(i) The problem is relatively *balanced*.  \n",
    "(ii) AdaBoost finds an almost perfect solution i.e. the *classes are pretty separable*.  \n",
    "\n",
    "In these cases, calibrating the estimates and shifting the decision threshold is not really necesary. In fact, reserving part of the dataset to perform calibration can actually harm classification.  \n",
    "\n",
    "These effects become less and less important as the number of training examples grow, however. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Extending the Paper's Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the [paper](https://scholar.google.co.uk/citations?view_op=view_citation&hl=el&user=R6b6Rp8AAAAJ&citation_for_view=R6b6Rp8AAAAJ:UeHWp8X0CEIC) we fixed the class imbalance and only discussed cost asymmetry -- although we did mention that our results can extend to either or a combination of both. Here our analysis and code covers the more general case.\n",
    "\n",
    "Also, the paper only included experiments using discrete AdaBoost, univariate logistic regression as the weak learner, Platt scaling as the calibration method of choice, and a 50/50 training/calibration dataset split, for proof of concept. Again, the code here allows for more general empirical support to our hypotheses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Thanks for reading our tutorial! Happy Boosting!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}