Hackathon_Challenge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspiration and contributions\n",
    "\n",
    "The following challenge was inspired by _Python Machine Learning: 1st Edition_ by _Sebastian Raschka_.\n",
    "\n",
    "Contributions to coding challenge (alphabetically): Mateusz Bieniek (KCL, London, UK), Matthew Wai Heng Chung (KCL, London, UK), Lisa Grant (UCL, London, UK), Anna Laddach (Francis Crick Institute, London, UK), Marius Kausas (Francis Crick Institute & KCL, London, UK), Paul Smith (KCL, London, UK)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Hackathon Challenge\n",
    "\n",
    "Based on previous medical data, can we predict the diagnosis for the unseen breast cancer biopsy?\n",
    "\n",
    "Let's use Supervised Machine Learning to classify breast cancer biopsies as either benign or malignant. We'll use two different methods - Logistic Regression and Random Forests - and compare our models' predictive power to that of human specialists."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge data set\n",
    "\n",
    "1. Title: Mammographic Mass Data\n",
    "\n",
    "2. Sources:\n",
    "\n",
    "   (a) Original owners of database:\n",
    "        Prof. Dr. Reudiger Schulz-Wendtland\n",
    "        Institute of Radiology, Gynaecological Radiology, University Erlangen-Nuremberg\n",
    "        Universitaetsstrasse 21-23\n",
    "        91054 Erlangen, Germany\n",
    "        \n",
    "   (b) Donor of database:\n",
    "        Matthias Elter\n",
    "        Fraunhofer Institute for Integrated Circuits (IIS)\n",
    "        Image Processing and Medical Engineering Department (BMT) \n",
    "        Am Wolfsmantel 33\n",
    "        91058 Erlangen, Germany\n",
    "        matthias.elter@iis.fraunhofer.de\n",
    "        (49) 9131-7767327 \n",
    "        \n",
    "   (c) Date received: October 2007\n",
    " \n",
    "3. Past Usage:\n",
    "    M. Elter, R. Schulz-Wendtland and T. Wittenberg (2007)\n",
    "    The prediction of breast cancer biopsy outcomes using two CAD approaches that both emphasize an intelligible decision process.\n",
    "    Medical Physics 34(11), pp. 4164-4172\n",
    "\n",
    "4. Relevant Information:\n",
    "    Mammography is the most effective method for breast cancer screening\n",
    "    available today. However, the low positive predictive value of breast\n",
    "    biopsy resulting from mammogram interpretation leads to approximately\n",
    "    70% unnecessary biopsies with benign outcomes. To reduce the high\n",
    "    number of unnecessary breast biopsies, several computer-aided diagnosis\n",
    "    (CAD) systems have been proposed in the last years.These systems\n",
    "    help physicians in their decision to perform a breast biopsy on a suspicious\n",
    "    lesion seen in a mammogram or to perform a short term follow-up\n",
    "    examination instead.\n",
    "    This data set can be used to predict the severity (benign or malignant)\n",
    "    of a mammographic mass lesion from BI-RADS attributes and the patient's age.\n",
    "    It contains a BI-RADS assessment, the patient's age and three BI-RADS attributes\n",
    "    together with the ground truth (the severity field) for 516 benign and\n",
    "    445 malignant masses that have been identified on full field digital mammograms\n",
    "    collected at the Institute of Radiology of the\n",
    "    University Erlangen-Nuremberg between 2003 and 2006.\n",
    "    Each instance has an associated BI-RADS assessment ranging from 1 (definitely benign)\n",
    "    to 5 (highly suggestive of malignancy) assigned in a double-review process by\n",
    "    physicians. Assuming that all cases with BI-RADS assessments greater or equal\n",
    "    a given value (varying from 1 to 5), are malignant and the other cases benign,\n",
    "    sensitivities and associated specificities can be calculated. These can be an\n",
    "    indication of how well a CAD system performs compared to the radiologists.\n",
    "\n",
    "5. Number of Instances: 961\n",
    "\n",
    "6. Number of Attributes: 6 (1 goal field, 1 non-predictive, 4 predictive attributes)\n",
    "\n",
    "7. Attribute Information:\n",
    "   1. BI-RADS assessment: 1 to 5 (ordinal)  \n",
    "   2. Age: patient's age in years (integer)\n",
    "   3. Shape: mass shape: round=1 oval=2 lobular=3 irregular=4 (nominal)\n",
    "   4. Margin: mass margin: circumscribed=1 microlobulated=2 obscured=3 ill-defined=4 spiculated=5 (nominal)\n",
    "   5. Density: mass density high=1 iso=2 low=3 fat-containing=4 (ordinal)\n",
    "   6. Severity: benign=0 or malignant=1 (binominal)\n",
    "\n",
    "8. Missing Attribute Values: Yes\n",
    "    - BI-RADS assessment:    2\n",
    "    - Age:                   5\n",
    "    - Shape:                31\n",
    "    - Margin:               48\n",
    "    - Density:              76\n",
    "    - Severity:              0\n",
    "\n",
    "9. Class Distribution: benign: 516; malignant: 445\n",
    "\n",
    "For more information, please follow to: [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/datasets/mammographic+mass)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Hackathon tasks\n",
    "\n",
    "#### Interacting with Python libraries through Application Programming Interface (API)\n",
    "\n",
    "Your task is try to grasp the internal structure of Python libraries, how to interact with an API and look-up useful methods for your research.\n",
    "\n",
    "#### Data pre-processing\n",
    "\n",
    "Your task is understand the nature of the dataset. Can you tell what will give you most predictive power?\n",
    "\n",
    "1. Understand what type of data do you have.\n",
    "2. Check for missing data values.\n",
    "3. Investigate dataset values by plotting histograms or scatter plots.\n",
    "\n",
    "#### Data encoding\n",
    "\n",
    "Why would you want to encode your data labels?\n",
    "\n",
    "#### Data preparation\n",
    "\n",
    "Why data preparation is important?\n",
    "\n",
    "1. Why we need training and tests sets?\n",
    "2. How does one split data into training and test sets using _scikit-learn_?\n",
    "\n",
    "#### Correlations\n",
    "\n",
    "Why would you investigate your data using correlations?\n",
    "\n",
    "1. What types of correlation there are?\n",
    "2. Using _scipy_ or _pandas_, calculate various types of correlations between any given two features.\n",
    "3. Calculate pair-wise correlations between dataset features and plot a heatmap using.\n",
    "\n",
    "#### Logistic regression\n",
    "\n",
    "Perform Logistic Regression.\n",
    "\n",
    "1. What is a Logistic Regression?\n",
    "2. What are the differences between Linear and Logistic regression?\n",
    "3. What are advantages and disadvantages of the method?\n",
    "4. Using _scikit-learn_, how do you set-up a Logistic Regression?\n",
    "\n",
    "#### Random Forests\n",
    "\n",
    "Investigate the use of Random Trees/Forests for data classification.\n",
    "\n",
    "1. What are Random Forests?\n",
    "2. What are advantages and disadvantages of the method?\n",
    "3. Using _scikit-learn_, how do you set-up a Random Forest model?\n",
    "4. How do you get feature importance from Random Forests?\n",
    "5. How would you optimize Random Forests' hyperparameters?\n",
    "\n",
    "#### Model performance\n",
    "\n",
    "What statistical measures can be used to quantify models performance?\n",
    "\n",
    "Investigate the following metrics:\n",
    "\n",
    "1. Accuracy.\n",
    "2. Precision.\n",
    "3. Recall.\n",
    "4. F1 score.\n",
    "5. Receiver operator characteristic (ROC).\n",
    "6. Area under the curve (AUC).\n",
    "7. Matthew's correlation coefficient.\n",
    "8. Confusion matrix.\n",
    "9. Out-of-bag error.\n",
    "\n",
    "#### Plotting\n",
    "\n",
    "Basics of _matloptlib_ and _seaborn_:\n",
    "\n",
    "1. Learn how to generate a basic plot.\n",
    "2. Change default _matplotlib_ or _seaborn_ plotting styles.\n",
    "3. Set basic plot attributes, such as labels, titles and colors.\n",
    "4. Save figures as .png format with a certain quality.\n",
    "5. Avoid typing plt.show() and set interactive plotting within IPython. \n",
    "6. Change figure dimensions within Ipython.\n",
    "\n",
    "#### Too easy?\n",
    "\n",
    "Cross-compare the performance of available classification algorithms and tell which works the best!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Learning material for this Hackathon\n",
    "\n",
    "* _scikit_learn_ tutorial: [text](https://scikit-learn.org/stable/tutorial/index.html)\n",
    "* Logistic Regression: [video](https://www.youtube.com/watch?v=yIYKR4sgzI8)\n",
    "* Decision Trees:[video](https://www.youtube.com/watch?v=7VeUPuFGJHk)\n",
    "* Random Forests: [video](https://www.youtube.com/watch?v=J4Wdy0Wc_xQ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading Python libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data manipulation libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Correlation coefficients\n",
    "from scipy import stats\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import mean_squared_error, accuracy_score, precision_score, recall_score\n",
    "from sklearn.metrics import roc_curve, f1_score, roc_auc_score, matthews_corrcoef, confusion_matrix\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# Data pre-processing\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Regression models\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Random Forests\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# Plotting libraries\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use(\"ggplot\")                    # Set Matplotlib styling\n",
    "from matplotlib.pylab import rcParams      \n",
    "rcParams['figure.figsize'] = 10, 9          # Set figure size\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data loading and processing\n",
    "\n",
    "In the beginning of every data analysis project, one must pre-process available data. Usually the pre-processing involves raw data inspection, visualization, dealing with missing values and preparing correct input for machine learning algortithms. If the data is from a publicly available research repository, reading the associated research articles often is very useful. \n",
    "\n",
    "The role of a researcher is to investigate and understand the available data set and choose the most appropriate modelling technique for the investigation of a given hypothesis. \n",
    "\n",
    "Thus, one would need to invest a considerable amount of time on the following section.  \n",
    " \n",
    "In this section, we will use _pandas_ and _numpy_ functionallities to load the [Mammographic Masses dataset](http://archive.ics.uci.edu/ml/datasets/mammographic+mass) and prepare the correct input for a machine learning model in _scikit_learn_.\n",
    "\n",
    "The data set of interest contains some missing values and some outliers. Can you detect them? How would you proceed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List current directory files\n",
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define column names\n",
    "df_col_names = [\"BI-RADS assessment\", \"Age\", \"Shape\", \"Margin\", \"Density\", \"Severity\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have you really understood the types of data you have?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Using _pandas_, load mammographics_masses.data file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data\n",
    "df = pd.read_csv(\"./input-data/mammographic_masses.data\", names = df_col_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Inspect underlying data set for missing values - what are they?\n",
    "- Check which columns contain missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out head of the pandas DataFrame\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check which columns contain missing values\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Identify indices of rows that contain missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterate over pandas DataFrame rows\n",
    "inds = []\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Separate your initial data set into complete and missing data sets.\n",
    "\n",
    "___Hint:___ Given the fact that _pandas_ are built connected to a _numpy_ interface, we can start working with _numpy_ arrays. \n",
    "\n",
    "To separate two data sets, we can generate a so-called \"mask\". In our case, \"mask\" will be a boolean array with True or False values indicating present and missing values, respectively. \n",
    "\n",
    "Due to the _numpy_ indexing ability, use the mask to extract complete data set. Subsequently, use the previously extracted indices to extract missing data set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a mask\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Complete data \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Missing data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert complete data numpy array string values to float64 representation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "-  From hereon, we are focusing on the complete data set. Given enough time, you should tihink about how to treat missing data and incorporate additional information into your model.\n",
    "- Separate features and diagnosis columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Separate into features and diagnosis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Perform summary statistics on the feature data set.\n",
    "- To do that convert features array back to _pandas_ DataFrame and search for its summary statistics method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert numpy array back to a pandas dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get summary statistics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Perform visual inspection of your data set by plotting all values of a single column as a distribution, bar or histogram plots. Plot scatter plots between two different features. Plot anything that might give you a better understanding about your data set.\n",
    "\n",
    "- Which plots are suited for which data?\n",
    "\n",
    "- Investigate what one can plot using _matplotlib_ and _seaborn_ visualisation libraries. As _seaborn_ provides a high-level interface to _matplotlib_, it is suggested to start of with _seaborn_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize single feature values as a bar plot\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# another bar plot\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Perfom a cross-correlation analysis of your features. Investigate which features do or not cross-correlate. Given all you know about the data set, can you tell why do you observe such behaviour?\n",
    "\n",
    "- Perhaps, can you tell which features will be more important in modelling cancer biopsy outcomes?\n",
    "\n",
    "- What assumptions are used to calculate correlation scores? Why they might be important here?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Correlation heatmap using pandas and seaborn functionalities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Manual calculation by creating empty numpy array and assigning correlation coefficients\n",
    "\n",
    "# Set the number of features\n",
    "\n",
    "# Define a zero 2D numpy matrix of shape (number_of_features, number_of_features)\n",
    "\n",
    "# Calculate pair-wise correlations between each features\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot heatmaps of pair-wise correlations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preparation of training and test data sets\n",
    "\n",
    "In this section, we will prepare training and test data sets. We will train our selected machine learning models on a training data set (i.e a subset of the complete data set). Once, training is complete, we will predict diagnosis outcomes of a test data set (i.e the rest of the complete data set).\n",
    "\n",
    "Preparation of training and test sets is a crucial point of a machine learning protocols that will allow us to quantify how well our model works. We will be able to quantify model's performance using a set of statistical metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Transform diagnosis data set into appropriate labels: benign=0 or malignant=1. Use the available functionality of LabelEncoder() to do that, even, if the initial diagnosis data set has the right labels. LabelEncoder() will return data in a right array shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use method of LabelEncoder()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Prepare training and test datasets using _scikit-learn_ by controlling the ration of data splitting.\n",
    "- Once you finish with a first classifier, investigate the effect of ratio splitting on your model's ability to predict the outcome of biopsy outcome. In such case, what is the minimum training data ratio is required to effectively predict biopsy outcome?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the proportion between training and test datasets\n",
    "proportion = 0.2\n",
    "\n",
    "# Split the dataset (random_state for analysis reproducibility)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic regression\n",
    "\n",
    "Let's choose logistic regression for training and classifying biopsy outcomes.\n",
    "\n",
    "__Tasks:__\n",
    "\n",
    "- Initialize a LogisticRegression model. Select Limited-memory BFGS algorithm as a solver for regression minimizaition. Set maximum number of iterations taken for the solver to converge to 5000.\n",
    "\n",
    "- Fit a LogisticRegression model on training data set.\n",
    "\n",
    "- Predict diagnosis labels for feature test data set.\n",
    "\n",
    "- Investigate other LogisticRegression model parameters. Is there anything you can do to improve the prediction accuracy? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialise Logistic Regression\n",
    "logreg = LogisticRegression()\n",
    "\n",
    "# Fit logistic regression model based on training data\n",
    "\n",
    "# Predict labels based on test dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Investigate the performance of your model by calculating accuracy of your regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the results\n",
    "# The mean accuracy on the given test data and labels\n",
    "print('Accuracy of logistic regression classifier on test set: {:.2f}'.format())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Print out a confusion matrix to explain the performance of a learning algorithm. Confusion matrix is simply a square matrix that reports the counts of the true positives, true negatives, false positives and false negatives.\n",
    "- Make sure to understand which parts of confusion matrix are what!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate confusion matrix\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot confusion matrix heatmap\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Use classification report to provide global overview of prediction in terms of precision, recall and f1-score. See _scikit-learn_ for more information. \n",
    "- One will be able to observe, how well prediction is performed for specific labels. Why a certain label has a higher prediction score than the other?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out classification_report\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__ \n",
    "\n",
    "- Plot a receiver operating characteristic (ROC) plot. Such graph is useful for selecting models based for classification based on their performance with respect to the false positive and true positive rates.  \n",
    "- Once, you have multiple classification models, plot multiple ROC curves to compare classification accuracies. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate ROC curve\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot ROC curve\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forests\n",
    "\n",
    "_Tour de force_ of Logistic Regression allowed us to use multiple tools for setting up and analysing a basic machine learning algorithm. Now, we will introduce another classifier - Random Forests - to perform the prediction of breast cancer biopsies.\n",
    "\n",
    "Intuitively, Random Forests can be considered as an ensemble of Decision Trees. A Decision Tree is a model where one breaks down our data of interest by making decision based on asking a series of questions. By running multiple Decision Trees one asign class labels by a majority vote."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Split your data set in training and test sets.\n",
    "- Initialize a Random Forest classifier.\n",
    "- Train and predict biopsy outcomes using a new classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data for Random Forest\n",
    "rf_proportion = 0.2\n",
    "\n",
    "# Initialize a Random Forest\n",
    "rf = RandomForestClassifier()\n",
    "\n",
    "# Train and predict with your Random Forest model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Calculate the following  scores:     \n",
    "    * model accuracy\n",
    "    * precision\n",
    "    * recall\n",
    "    * F1\n",
    "    * ROC\n",
    "    * area under the curve\n",
    "    * Matthew's correlation coefficient\n",
    "    * Matthew's correlation\n",
    "    * Confusion matrix\n",
    "    * Out-of-bag error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get metrics on model performance\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Random forest classifier can be useful, because it provides importance of training features for classification prediction. Calculate the importance of each feature. \n",
    "- Does this make sense with what we already know?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get feature importances of random forest classifier\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Plot importance of features as a bar plot with error bars."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the feature importances of the forest\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Tasks:__\n",
    "\n",
    "- Perform hyperparameter tuning for RandomForest Classifier using GridSearch. Find the best set of hyperparameters to perform model training. \n",
    "- What are other ways to search for hyperparameters? Which ones could be better and why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grid search for best parameters\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}