Practice4 - Data Cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Cleaning in Python\n",
    "The examples and code in this notebook are taken from the [Kaggle Data Cleaning Challenge](https://www.kaggle.com/getting-started/52652)\n",
    "\n",
    "Detailed explanations for important code snippets are provided by Mervat Abuelkheir as part of the CSEN1095 Data Engineering Course.\n",
    "\n",
    "The goal of this notebook is to show how to handle missing values and noisy data.\n",
    "\n",
    "Pay attention to the <span style=\"color:red\"> <b> paragraphs in bold red</b></span>; they ask you to do something and provide input!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspecting the data\n",
    "\n",
    "First thing we need to do is import a dataset.\n",
    "\n",
    "<br>We will work in this exercise with the <a href=\"https://www.kaggle.com/maxhorowitz/nflplaybyplay2009to2016#NFL%20Play%20by%20Play%202009-2017%20(v4).csv\"> NFL dataset</a>. The NFL dataset is available in the repository's data folder, and is made available on Kaggle. It contains all the regular season plays from the 2009-2016 NFL seasons. The dataset has 100 columns. Each play contains information on: game situation, players involved, results, and advanced metrics such as expected point and win probability values.\n",
    "\n",
    "### Important note\n",
    "\n",
    "<b>The NFL dataset is very big (200+ MB) and is not made available in the `data` folder in this repository. Download the dataset from the link provided, store it in your own data or working directory, and proceed with running the code below, taking care to change your directory according to your own data storage location.</b>\n",
    "\n",
    "<b>For large datasets, you will be provided with a link through which you can download the data. Another option is to read from the html link, provided the data is well-organized to help with proper parsing.</b>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] File b'data/NFL Play by Play 2009-2017 (v4).csv' does not exist: b'data/NFL Play by Play 2009-2017 (v4).csv'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-2-f79fcd54144f>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;31m# read in all our data\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m \u001b[0mnfl_data\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"data/NFL Play by Play 2009-2017 (v4).csv\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;31m# set seed for reproducibility\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\Anaconda3\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[1;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[0;32m    700\u001b[0m                     skip_blank_lines=skip_blank_lines)\n\u001b[0;32m    701\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 702\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    703\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    704\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\Anaconda3\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36m_read\u001b[1;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[0;32m    427\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    428\u001b[0m     \u001b[1;31m# Create the parser.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 429\u001b[1;33m     \u001b[0mparser\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    430\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    431\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\Anaconda3\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[0;32m    893\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'has_index_names'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'has_index_names'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    894\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 895\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    896\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    897\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\Anaconda3\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[1;34m(self, engine)\u001b[0m\n\u001b[0;32m   1120\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_make_engine\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m'c'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1121\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'c'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1122\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mCParserWrapper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mf\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1123\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1124\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'python'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\Anaconda3\\lib\\site-packages\\pandas\\io\\parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, src, **kwds)\u001b[0m\n\u001b[0;32m   1851\u001b[0m         \u001b[0mkwds\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'usecols'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0musecols\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1852\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1853\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_reader\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mparsers\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTextReader\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1854\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0munnamed_cols\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_reader\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0munnamed_cols\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1855\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.__cinit__\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._setup_parser_source\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] File b'data/NFL Play by Play 2009-2017 (v4).csv' does not exist: b'data/NFL Play by Play 2009-2017 (v4).csv'"
     ]
    }
   ],
   "source": [
    "# modules we'll use\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# read in all our data\n",
    "nfl_data = pd.read_csv(\"data/NFL Play by Play 2009-2017 (v4).csv\")\n",
    "\n",
    "# set seed for reproducibility\n",
    "np.random.seed(0) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>GameID</th>\n",
       "      <th>Drive</th>\n",
       "      <th>qtr</th>\n",
       "      <th>down</th>\n",
       "      <th>time</th>\n",
       "      <th>TimeUnder</th>\n",
       "      <th>TimeSecs</th>\n",
       "      <th>PlayTimeDiff</th>\n",
       "      <th>SideofField</th>\n",
       "      <th>...</th>\n",
       "      <th>yacEPA</th>\n",
       "      <th>Home_WP_pre</th>\n",
       "      <th>Away_WP_pre</th>\n",
       "      <th>Home_WP_post</th>\n",
       "      <th>Away_WP_post</th>\n",
       "      <th>Win_Prob</th>\n",
       "      <th>WPA</th>\n",
       "      <th>airWPA</th>\n",
       "      <th>yacWPA</th>\n",
       "      <th>Season</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>244485</th>\n",
       "      <td>2014-10-26</td>\n",
       "      <td>2014102607</td>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>00:39</td>\n",
       "      <td>1</td>\n",
       "      <td>939.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>TB</td>\n",
       "      <td>...</td>\n",
       "      <td>1.240299</td>\n",
       "      <td>0.225647</td>\n",
       "      <td>0.774353</td>\n",
       "      <td>0.245582</td>\n",
       "      <td>0.754418</td>\n",
       "      <td>0.225647</td>\n",
       "      <td>0.019935</td>\n",
       "      <td>-0.018156</td>\n",
       "      <td>0.038091</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115340</th>\n",
       "      <td>2011-11-20</td>\n",
       "      <td>2011112000</td>\n",
       "      <td>22</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>06:47</td>\n",
       "      <td>7</td>\n",
       "      <td>407.0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>OAK</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.056036</td>\n",
       "      <td>0.943964</td>\n",
       "      <td>0.042963</td>\n",
       "      <td>0.957037</td>\n",
       "      <td>0.943964</td>\n",
       "      <td>0.013073</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68357</th>\n",
       "      <td>2010-11-14</td>\n",
       "      <td>2010111401</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>00:23</td>\n",
       "      <td>1</td>\n",
       "      <td>1823.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>CLE</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.365307</td>\n",
       "      <td>0.634693</td>\n",
       "      <td>0.384697</td>\n",
       "      <td>0.615303</td>\n",
       "      <td>0.634693</td>\n",
       "      <td>-0.019390</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368377</th>\n",
       "      <td>2017-09-24</td>\n",
       "      <td>2017092405</td>\n",
       "      <td>24</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>08:48</td>\n",
       "      <td>9</td>\n",
       "      <td>528.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>CLE</td>\n",
       "      <td>...</td>\n",
       "      <td>1.075660</td>\n",
       "      <td>0.935995</td>\n",
       "      <td>0.064005</td>\n",
       "      <td>0.921231</td>\n",
       "      <td>0.078769</td>\n",
       "      <td>0.064005</td>\n",
       "      <td>0.014764</td>\n",
       "      <td>0.003866</td>\n",
       "      <td>0.010899</td>\n",
       "      <td>2017</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 102 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              Date      GameID  Drive  qtr  down   time  TimeUnder  TimeSecs  \\\n",
       "244485  2014-10-26  2014102607     18    3   1.0  00:39          1     939.0   \n",
       "115340  2011-11-20  2011112000     22    4   1.0  06:47          7     407.0   \n",
       "68357   2010-11-14  2010111401      8    2   NaN  00:23          1    1823.0   \n",
       "368377  2017-09-24  2017092405     24    4   1.0  08:48          9     528.0   \n",
       "\n",
       "        PlayTimeDiff SideofField  ...    yacEPA  Home_WP_pre  Away_WP_pre  \\\n",
       "244485          12.0          TB  ...  1.240299     0.225647     0.774353   \n",
       "115340          44.0         OAK  ...       NaN     0.056036     0.943964   \n",
       "68357            0.0         CLE  ...       NaN     0.365307     0.634693   \n",
       "368377           8.0         CLE  ...  1.075660     0.935995     0.064005   \n",
       "\n",
       "        Home_WP_post  Away_WP_post  Win_Prob       WPA    airWPA    yacWPA  \\\n",
       "244485      0.245582      0.754418  0.225647  0.019935 -0.018156  0.038091   \n",
       "115340      0.042963      0.957037  0.943964  0.013073       NaN       NaN   \n",
       "68357       0.384697      0.615303  0.634693 -0.019390       NaN       NaN   \n",
       "368377      0.921231      0.078769  0.064005  0.014764  0.003866  0.010899   \n",
       "\n",
       "        Season  \n",
       "244485    2014  \n",
       "115340    2011  \n",
       "68357     2010  \n",
       "368377    2017  \n",
       "\n",
       "[4 rows x 102 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Look at a few rows of the nfl_data file. A handful of missing data!\n",
    "nfl_data.sample(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## See how many missing data points we have\n",
    "\n",
    "Let's see how many we have in each column. Let's start with the NFL dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Date                                0\n",
      "GameID                              0\n",
      "Drive                               0\n",
      "qtr                                 0\n",
      "down                            61154\n",
      "time                              224\n",
      "TimeUnder                           0\n",
      "TimeSecs                          224\n",
      "PlayTimeDiff                      444\n",
      "SideofField                       528\n",
      "yrdln                             840\n",
      "yrdline100                        840\n",
      "ydstogo                             0\n",
      "ydsnet                              0\n",
      "GoalToGo                          840\n",
      "FirstDown                       28811\n",
      "posteam                         24992\n",
      "DefensiveTeam                   24992\n",
      "desc                                2\n",
      "PlayAttempted                       0\n",
      "Yards.Gained                        0\n",
      "sp                                  0\n",
      "Touchdown                           0\n",
      "ExPointResult                  397578\n",
      "TwoPointConv                   407083\n",
      "DefTwoPoint                    407664\n",
      "Safety                              0\n",
      "Onsidekick                          0\n",
      "PuntResult                     385317\n",
      "PlayType                            0\n",
      "                                ...  \n",
      "AwayTeam                            0\n",
      "Timeout_Indicator                   0\n",
      "Timeout_Team                        0\n",
      "posteam_timeouts_pre                0\n",
      "HomeTimeouts_Remaining_Pre          0\n",
      "AwayTimeouts_Remaining_Pre          0\n",
      "HomeTimeouts_Remaining_Post         0\n",
      "AwayTimeouts_Remaining_Post         0\n",
      "No_Score_Prob                     176\n",
      "Opp_Field_Goal_Prob               176\n",
      "Opp_Safety_Prob                   176\n",
      "Opp_Touchdown_Prob                176\n",
      "Field_Goal_Prob                   176\n",
      "Safety_Prob                       176\n",
      "Touchdown_Prob                    176\n",
      "ExPoint_Prob                        0\n",
      "TwoPoint_Prob                       0\n",
      "ExpPts                            176\n",
      "EPA                               369\n",
      "airEPA                         248394\n",
      "yacEPA                         248498\n",
      "Home_WP_pre                     24954\n",
      "Away_WP_pre                     24954\n",
      "Home_WP_post                    26587\n",
      "Away_WP_post                    26587\n",
      "Win_Prob                        25009\n",
      "WPA                              5541\n",
      "airWPA                         248501\n",
      "yacWPA                         248762\n",
      "Season                              0\n",
      "Length: 102, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# get the number of missing data points per column\n",
    "missing_values_count_nfl = nfl_data.isnull().sum()\n",
    "\n",
    "# look at the # of missing points in the first ten columns\n",
    "print(missing_values_count_nfl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of missing values it seems! Let's take a look at the percentage of the missing values in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24.87214126835169\n"
     ]
    }
   ],
   "source": [
    "# How many total missing values do we have?\n",
    "# shape returns the dimentionality of a dataframe (rows and columns), can you guess what product will do?\n",
    "total_cells_nfl = np.product(nfl_data.shape) \n",
    "total_missing_nfl = missing_values_count_nfl.sum()\n",
    "\n",
    "# percent of data that is missing\n",
    "percentage_missign_values_nfl = (total_missing_nfl/total_cells_nfl) * 100\n",
    "print(percentage_missign_values_nfl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost a quarter of the cells in this dataset are empty! \n",
    "\n",
    "In the next step, we're going to take a closer look at some of the columns with missing values and try to figure out what might be going on with them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining the missing values\n",
    "\n",
    "This is the point where you should develop some data intuition and look at the data to try and figure out why it is the way it is and how that will affect your analysis. For dealing with missing values, you'll need to use your intution to figure out why the value is missing. One of the most important question you can ask yourself to help figure this out is this:\n",
    "\n",
    "**Is this value missing becuase it wasn't recorded or becuase it dosen't exist?**\n",
    "\n",
    "If a value is missing becuase it doens't exist (like the height of the oldest child of someone who doesn't have any children) then it doesn't make sense to try and guess what it might be. These values you probably do want to keep as `NaN`. On the other hand, if a value is missing becuase it wasn't recorded, then you can try to guess what it might have been based on the other values in that column and row. (This is called \"imputation\" and we'll learn how to do it next! :)\n",
    "\n",
    "Let's work through an example. Looking at the number of missing values in the nfl_data dataframe, notice that the column TimesSec has a lot of missing values in it: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date                0\n",
       "GameID              0\n",
       "Drive               0\n",
       "qtr                 0\n",
       "down            61154\n",
       "time              224\n",
       "TimeUnder           0\n",
       "TimeSecs          224\n",
       "PlayTimeDiff      444\n",
       "SideofField       528\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the # of missing points in the first ten columns\n",
    "missing_values_count_nfl[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `TimeSecs` column has information on the number of seconds left in the game when the play was made. This means that these values are probably missing because they were not recorded, rather than because they don't exist. So, it would make sense for us to try and guess what they should be rather than just leaving them as NA's.\n",
    "\n",
    "On the other hand, there are other fields, like `PenalizedTeam` that also have lot of missing fields. In this case, though, the field is missing because if there was no penalty then it doesn't make sense to say which team was penalized. For this column, it would make more sense to either leave it empty or to add a third value like \"neither\" and use that to replace the NA's.\n",
    "\n",
    "**Tip: This is a great place to read over the dataset documentation if you haven't already! If you're working with a dataset that you've gotten from another person, you can also try reaching out to them to get more information.**\n",
    "\n",
    "You should look at each column individually to figure out the best strategy for filling those missing values. For the rest of this notebook, we'll cover some \"quick and dirty\" techniques that can help you with missing values but will probably also end up removing some useful information or adding some noise to your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropping the missing values\n",
    "\n",
    "If you don't have a reason to figure out why your values are missing, one option you have is to just remove any rows or columns that contain missing values. (Note: DO NOT do this approch for important projects! It's usually worth it to take the time to go through your data and really look at all the columns with missing values one-by-one to really get to know your dataset.)  \n",
    "\n",
    "If you're sure you want to drop rows with missing values, pandas does have a handy function, `dropna()` to help you do this. Let's try it out on our NFL dataset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>GameID</th>\n",
       "      <th>Drive</th>\n",
       "      <th>qtr</th>\n",
       "      <th>down</th>\n",
       "      <th>time</th>\n",
       "      <th>TimeUnder</th>\n",
       "      <th>TimeSecs</th>\n",
       "      <th>PlayTimeDiff</th>\n",
       "      <th>SideofField</th>\n",
       "      <th>...</th>\n",
       "      <th>yacEPA</th>\n",
       "      <th>Home_WP_pre</th>\n",
       "      <th>Away_WP_pre</th>\n",
       "      <th>Home_WP_post</th>\n",
       "      <th>Away_WP_post</th>\n",
       "      <th>Win_Prob</th>\n",
       "      <th>WPA</th>\n",
       "      <th>airWPA</th>\n",
       "      <th>yacWPA</th>\n",
       "      <th>Season</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>0 rows × 102 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [Date, GameID, Drive, qtr, down, time, TimeUnder, TimeSecs, PlayTimeDiff, SideofField, yrdln, yrdline100, ydstogo, ydsnet, GoalToGo, FirstDown, posteam, DefensiveTeam, desc, PlayAttempted, Yards.Gained, sp, Touchdown, ExPointResult, TwoPointConv, DefTwoPoint, Safety, Onsidekick, PuntResult, PlayType, Passer, Passer_ID, PassAttempt, PassOutcome, PassLength, AirYards, YardsAfterCatch, QBHit, PassLocation, InterceptionThrown, Interceptor, Rusher, Rusher_ID, RushAttempt, RunLocation, RunGap, Receiver, Receiver_ID, Reception, ReturnResult, Returner, BlockingPlayer, Tackler1, Tackler2, FieldGoalResult, FieldGoalDistance, Fumble, RecFumbTeam, RecFumbPlayer, Sack, Challenge.Replay, ChalReplayResult, Accepted.Penalty, PenalizedTeam, PenaltyType, PenalizedPlayer, Penalty.Yards, PosTeamScore, DefTeamScore, ScoreDiff, AbsScoreDiff, HomeTeam, AwayTeam, Timeout_Indicator, Timeout_Team, posteam_timeouts_pre, HomeTimeouts_Remaining_Pre, AwayTimeouts_Remaining_Pre, HomeTimeouts_Remaining_Post, AwayTimeouts_Remaining_Post, No_Score_Prob, Opp_Field_Goal_Prob, Opp_Safety_Prob, Opp_Touchdown_Prob, Field_Goal_Prob, Safety_Prob, Touchdown_Prob, ExPoint_Prob, TwoPoint_Prob, ExpPts, EPA, airEPA, yacEPA, Home_WP_pre, Away_WP_pre, Home_WP_post, Away_WP_post, Win_Prob, WPA, airWPA, ...]\n",
       "Index: []\n",
       "\n",
       "[0 rows x 102 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove all the rows that contain a missing value\n",
    "nfl_data.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops! That's removed all our data! <span style=\"color:red\"><b>Why do you think this happened?</b></span>\n",
    "\n",
    "We might have better luck removing all the columns that have at least one missing value instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>GameID</th>\n",
       "      <th>Drive</th>\n",
       "      <th>qtr</th>\n",
       "      <th>TimeUnder</th>\n",
       "      <th>ydstogo</th>\n",
       "      <th>ydsnet</th>\n",
       "      <th>PlayAttempted</th>\n",
       "      <th>Yards.Gained</th>\n",
       "      <th>sp</th>\n",
       "      <th>...</th>\n",
       "      <th>Timeout_Indicator</th>\n",
       "      <th>Timeout_Team</th>\n",
       "      <th>posteam_timeouts_pre</th>\n",
       "      <th>HomeTimeouts_Remaining_Pre</th>\n",
       "      <th>AwayTimeouts_Remaining_Pre</th>\n",
       "      <th>HomeTimeouts_Remaining_Post</th>\n",
       "      <th>AwayTimeouts_Remaining_Post</th>\n",
       "      <th>ExPoint_Prob</th>\n",
       "      <th>TwoPoint_Prob</th>\n",
       "      <th>Season</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2009-09-10</td>\n",
       "      <td>2009091000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2009-09-10</td>\n",
       "      <td>2009091000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2009-09-10</td>\n",
       "      <td>2009091000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>-3</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2009-09-10</td>\n",
       "      <td>2009091000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2009-09-10</td>\n",
       "      <td>2009091000</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date      GameID  Drive  qtr  TimeUnder  ydstogo  ydsnet  \\\n",
       "0  2009-09-10  2009091000      1    1         15        0       0   \n",
       "1  2009-09-10  2009091000      1    1         15       10       5   \n",
       "2  2009-09-10  2009091000      1    1         15        5       2   \n",
       "3  2009-09-10  2009091000      1    1         14        8       2   \n",
       "4  2009-09-10  2009091000      1    1         14        8       2   \n",
       "\n",
       "   PlayAttempted  Yards.Gained  sp  ...  Timeout_Indicator  Timeout_Team  \\\n",
       "0              1            39   0  ...                  0          None   \n",
       "1              1             5   0  ...                  0          None   \n",
       "2              1            -3   0  ...                  0          None   \n",
       "3              1             0   0  ...                  0          None   \n",
       "4              1             0   0  ...                  0          None   \n",
       "\n",
       "   posteam_timeouts_pre HomeTimeouts_Remaining_Pre AwayTimeouts_Remaining_Pre  \\\n",
       "0                     3                          3                          3   \n",
       "1                     3                          3                          3   \n",
       "2                     3                          3                          3   \n",
       "3                     3                          3                          3   \n",
       "4                     3                          3                          3   \n",
       "\n",
       "   HomeTimeouts_Remaining_Post  AwayTimeouts_Remaining_Post  ExPoint_Prob  \\\n",
       "0                            3                            3           0.0   \n",
       "1                            3                            3           0.0   \n",
       "2                            3                            3           0.0   \n",
       "3                            3                            3           0.0   \n",
       "4                            3                            3           0.0   \n",
       "\n",
       "   TwoPoint_Prob  Season  \n",
       "0            0.0    2009  \n",
       "1            0.0    2009  \n",
       "2            0.0    2009  \n",
       "3            0.0    2009  \n",
       "4            0.0    2009  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove all columns with at least one missing value\n",
    "columns_with_na_dropped = nfl_data.dropna(axis=1) # axis 1 indicates operation along columns\n",
    "columns_with_na_dropped.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check how much data we lost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns in original dataset: 102 \n",
      "\n",
      "Columns with na's dropped: 41\n"
     ]
    }
   ],
   "source": [
    "print(\"Columns in original dataset: %d \\n\" % nfl_data.shape[1])\n",
    "print(\"Columns with na's dropped: %d\" % columns_with_na_dropped.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've lost quite a bit of data, but at this point we have successfully removed all the `NaN`'s from our data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Filling in missing values automatically\n",
    "\n",
    "Another option is to try and fill in the missing values. Let's get a small sub-section of the NFL data so that it will print well and work with that to avoid having to work with the entire dataset and consume CPU time. You can work on the entire dataset at home!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPA</th>\n",
       "      <th>airEPA</th>\n",
       "      <th>yacEPA</th>\n",
       "      <th>Home_WP_pre</th>\n",
       "      <th>Away_WP_pre</th>\n",
       "      <th>Home_WP_post</th>\n",
       "      <th>Away_WP_post</th>\n",
       "      <th>Win_Prob</th>\n",
       "      <th>WPA</th>\n",
       "      <th>airWPA</th>\n",
       "      <th>yacWPA</th>\n",
       "      <th>Season</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.014474</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.485675</td>\n",
       "      <td>0.514325</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.453567</td>\n",
       "      <td>0.485675</td>\n",
       "      <td>0.060758</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.077907</td>\n",
       "      <td>-1.068169</td>\n",
       "      <td>1.146076</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.453567</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>0.448912</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.004655</td>\n",
       "      <td>-0.032244</td>\n",
       "      <td>0.036899</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.402760</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>0.448912</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>0.489207</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>-0.040295</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.712583</td>\n",
       "      <td>3.318841</td>\n",
       "      <td>-5.031425</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>0.489207</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.538783</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>-0.049576</td>\n",
       "      <td>0.106663</td>\n",
       "      <td>-0.156239</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.097796</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.538783</td>\n",
       "      <td>0.558929</td>\n",
       "      <td>0.441071</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.097712</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        EPA    airEPA    yacEPA  Home_WP_pre  Away_WP_pre  Home_WP_post  \\\n",
       "0  2.014474       NaN       NaN     0.485675     0.514325      0.546433   \n",
       "1  0.077907 -1.068169  1.146076     0.546433     0.453567      0.551088   \n",
       "2 -1.402760       NaN       NaN     0.551088     0.448912      0.510793   \n",
       "3 -1.712583  3.318841 -5.031425     0.510793     0.489207      0.461217   \n",
       "4  2.097796       NaN       NaN     0.461217     0.538783      0.558929   \n",
       "\n",
       "   Away_WP_post  Win_Prob       WPA    airWPA    yacWPA  Season  \n",
       "0      0.453567  0.485675  0.060758       NaN       NaN    2009  \n",
       "1      0.448912  0.546433  0.004655 -0.032244  0.036899    2009  \n",
       "2      0.489207  0.551088 -0.040295       NaN       NaN    2009  \n",
       "3      0.538783  0.510793 -0.049576  0.106663 -0.156239    2009  \n",
       "4      0.441071  0.461217  0.097712       NaN       NaN    2009  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get a small subset of the NFL dataset\n",
    "subset_nfl_data = nfl_data.loc[:, 'EPA':'Season'].head()\n",
    "subset_nfl_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the Panda's fillna() function to fill in missing values in a dataframe for us. One option we have is to specify what we want the `NaN` values to be replaced with. Here, I'm saying that I would like to replace all the `NaN` values with 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPA</th>\n",
       "      <th>airEPA</th>\n",
       "      <th>yacEPA</th>\n",
       "      <th>Home_WP_pre</th>\n",
       "      <th>Away_WP_pre</th>\n",
       "      <th>Home_WP_post</th>\n",
       "      <th>Away_WP_post</th>\n",
       "      <th>Win_Prob</th>\n",
       "      <th>WPA</th>\n",
       "      <th>airWPA</th>\n",
       "      <th>yacWPA</th>\n",
       "      <th>Season</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.014474</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.485675</td>\n",
       "      <td>0.514325</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.453567</td>\n",
       "      <td>0.485675</td>\n",
       "      <td>0.060758</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.077907</td>\n",
       "      <td>-1.068169</td>\n",
       "      <td>1.146076</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.453567</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>0.448912</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.004655</td>\n",
       "      <td>-0.032244</td>\n",
       "      <td>0.036899</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.402760</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>0.448912</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>0.489207</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>-0.040295</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.712583</td>\n",
       "      <td>3.318841</td>\n",
       "      <td>-5.031425</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>0.489207</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.538783</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>-0.049576</td>\n",
       "      <td>0.106663</td>\n",
       "      <td>-0.156239</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.097796</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.538783</td>\n",
       "      <td>0.558929</td>\n",
       "      <td>0.441071</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.097712</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        EPA    airEPA    yacEPA  Home_WP_pre  Away_WP_pre  Home_WP_post  \\\n",
       "0  2.014474  0.000000  0.000000     0.485675     0.514325      0.546433   \n",
       "1  0.077907 -1.068169  1.146076     0.546433     0.453567      0.551088   \n",
       "2 -1.402760  0.000000  0.000000     0.551088     0.448912      0.510793   \n",
       "3 -1.712583  3.318841 -5.031425     0.510793     0.489207      0.461217   \n",
       "4  2.097796  0.000000  0.000000     0.461217     0.538783      0.558929   \n",
       "\n",
       "   Away_WP_post  Win_Prob       WPA    airWPA    yacWPA  Season  \n",
       "0      0.453567  0.485675  0.060758  0.000000  0.000000    2009  \n",
       "1      0.448912  0.546433  0.004655 -0.032244  0.036899    2009  \n",
       "2      0.489207  0.551088 -0.040295  0.000000  0.000000    2009  \n",
       "3      0.538783  0.510793 -0.049576  0.106663 -0.156239    2009  \n",
       "4      0.441071  0.461217  0.097712  0.000000  0.000000    2009  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace all NA's with 0\n",
    "subset_nfl_data.fillna(0) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also be a bit more savvy and replace missing values with whatever value comes directly after it in the same column. (This makes a lot of sense for datasets where the observations have some sort of logical order to them.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EPA</th>\n",
       "      <th>airEPA</th>\n",
       "      <th>yacEPA</th>\n",
       "      <th>Home_WP_pre</th>\n",
       "      <th>Away_WP_pre</th>\n",
       "      <th>Home_WP_post</th>\n",
       "      <th>Away_WP_post</th>\n",
       "      <th>Win_Prob</th>\n",
       "      <th>WPA</th>\n",
       "      <th>airWPA</th>\n",
       "      <th>yacWPA</th>\n",
       "      <th>Season</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.014474</td>\n",
       "      <td>-1.068169</td>\n",
       "      <td>1.146076</td>\n",
       "      <td>0.485675</td>\n",
       "      <td>0.514325</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.453567</td>\n",
       "      <td>0.485675</td>\n",
       "      <td>0.060758</td>\n",
       "      <td>-0.032244</td>\n",
       "      <td>0.036899</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.077907</td>\n",
       "      <td>-1.068169</td>\n",
       "      <td>1.146076</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.453567</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>0.448912</td>\n",
       "      <td>0.546433</td>\n",
       "      <td>0.004655</td>\n",
       "      <td>-0.032244</td>\n",
       "      <td>0.036899</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.402760</td>\n",
       "      <td>3.318841</td>\n",
       "      <td>-5.031425</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>0.448912</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>0.489207</td>\n",
       "      <td>0.551088</td>\n",
       "      <td>-0.040295</td>\n",
       "      <td>0.106663</td>\n",
       "      <td>-0.156239</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.712583</td>\n",
       "      <td>3.318841</td>\n",
       "      <td>-5.031425</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>0.489207</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.538783</td>\n",
       "      <td>0.510793</td>\n",
       "      <td>-0.049576</td>\n",
       "      <td>0.106663</td>\n",
       "      <td>-0.156239</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.097796</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.538783</td>\n",
       "      <td>0.558929</td>\n",
       "      <td>0.441071</td>\n",
       "      <td>0.461217</td>\n",
       "      <td>0.097712</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        EPA    airEPA    yacEPA  Home_WP_pre  Away_WP_pre  Home_WP_post  \\\n",
       "0  2.014474 -1.068169  1.146076     0.485675     0.514325      0.546433   \n",
       "1  0.077907 -1.068169  1.146076     0.546433     0.453567      0.551088   \n",
       "2 -1.402760  3.318841 -5.031425     0.551088     0.448912      0.510793   \n",
       "3 -1.712583  3.318841 -5.031425     0.510793     0.489207      0.461217   \n",
       "4  2.097796  0.000000  0.000000     0.461217     0.538783      0.558929   \n",
       "\n",
       "   Away_WP_post  Win_Prob       WPA    airWPA    yacWPA  Season  \n",
       "0      0.453567  0.485675  0.060758 -0.032244  0.036899    2009  \n",
       "1      0.448912  0.546433  0.004655 -0.032244  0.036899    2009  \n",
       "2      0.489207  0.551088 -0.040295  0.106663 -0.156239    2009  \n",
       "3      0.538783  0.510793 -0.049576  0.106663 -0.156239    2009  \n",
       "4      0.441071  0.461217  0.097712  0.000000  0.000000    2009  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace all NA's the value that comes directly after it in the same column, \n",
    "# then replace all the reamining na's with 0\n",
    "subset_nfl_data.fillna(method = 'bfill', axis=0).fillna(0) # here you may want to explore other methods like 'mean' if it makes sense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trying multivariate imputation with sklearn\n",
    "\n",
    "sklearn has an `Imputer` and a `SimpleImputer` modules which perform imputation on one column at a time using the mean, median, or mode. \n",
    "\n",
    "sklearn also has a multivariate imputation module, `IterativeImputer`, that can help us impute values across multiple columns, but it is experimental and only available through developer release.\n",
    "\n",
    "Let's see how an sklearn's imputation works for the NFL dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "H:\\DataScienceWork\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:58: DeprecationWarning: Class Imputer is deprecated; Imputer was deprecated in version 0.20 and will be removed in 0.22. Import impute.SimpleImputer from sklearn instead.\n",
      "  warnings.warn(msg, category=DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "EPA             0\n",
       "airEPA          3\n",
       "yacEPA          3\n",
       "Home_WP_pre     0\n",
       "Away_WP_pre     0\n",
       "Home_WP_post    0\n",
       "Away_WP_post    0\n",
       "Win_Prob        0\n",
       "WPA             0\n",
       "airWPA          3\n",
       "yacWPA          3\n",
       "Season          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import Imputer\n",
    "#from sklearn.impute import SimpleImputer # pretty much the same thing, you can try yourself\n",
    "mean_imputer = Imputer(missing_values=np.nan, strategy='mean')\n",
    "#mean_simpleimputer = SimpleImputer(missing_values=np.nan, strategy='mean') # Same use for sklearn SimpleImputer\n",
    "\n",
    "# Again, let's get a small subset of the NFL dataset\n",
    "subset_nfl_data = nfl_data.loc[:, 'EPA':'Season'].head()\n",
    "\n",
    "# Check the presence of null values in the subset chosen\n",
    "subset_nfl_data.isnull().sum()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EPA             0\n",
       "airEPA          3\n",
       "yacEPA          3\n",
       "Home_WP_pre     0\n",
       "Away_WP_pre     0\n",
       "Home_WP_post    0\n",
       "Away_WP_post    0\n",
       "Win_Prob        0\n",
       "WPA             0\n",
       "airWPA          3\n",
       "yacWPA          0\n",
       "Season          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now let's impute missing values in yacWPA column with the mean\n",
    "mean_imputer.fit(subset_nfl_data[[\"yacWPA\"]])\n",
    "subset_nfl_data[[\"yacWPA\"]]=mean_imputer.transform(subset_nfl_data[[\"yacWPA\"]]).ravel()\n",
    "#imputed_subset_nfl_data = pd.DataFrame(mean_simpleimputer.fit_transform(subset_nfl_data), columns=subset_nfl_data.columns) # fitting and transforming can be done in one step\n",
    "\n",
    "# Let's check the presence null values again to see if imputation worked for the column\n",
    "subset_nfl_data.isnull().sum()\n",
    "# add you own counting of nulls for the dataframe imputed by SimpleImputer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        EPA    airEPA    yacEPA  Home_WP_pre  Away_WP_pre  Home_WP_post  \\\n",
      "0  2.014474       NaN       NaN     0.485675     0.514325      0.546433   \n",
      "1  0.077907 -1.068169  1.146076     0.546433     0.453567      0.551088   \n",
      "2 -1.402760       NaN       NaN     0.551088     0.448912      0.510793   \n",
      "3 -1.712583  3.318841 -5.031425     0.510793     0.489207      0.461217   \n",
      "4  2.097796       NaN       NaN     0.461217     0.538783      0.558929   \n",
      "\n",
      "   Away_WP_post  Win_Prob       WPA    airWPA    yacWPA  Season  \n",
      "0      0.453567  0.485675  0.060758       NaN -0.059670    2009  \n",
      "1      0.448912  0.546433  0.004655 -0.032244  0.036899    2009  \n",
      "2      0.489207  0.551088 -0.040295       NaN -0.059670    2009  \n",
      "3      0.538783  0.510793 -0.049576  0.106663 -0.156239    2009  \n",
      "4      0.441071  0.461217  0.097712       NaN -0.059670    2009  \n"
     ]
    }
   ],
   "source": [
    "# Print the dataframe \n",
    "print(subset_nfl_data)\n",
    "#print(imputed_subset_nfl_data) # if you want to try SimpleImputer code\n",
    "# You may want to try the same code snippets above with the entire dataset and not just the subset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try the more advanced `IterativeImputer` in sklearn. It provides means to do multivariate imputation using a variety of ML models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'sklearn.experimental'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-17-9a28e0d7e0a4>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# This code snippet may not work, so do not worry about it, and just examine the similarities with the SimpleImputer\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;31m# To use this experimental feature, we need to explicitly ask for it:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexperimental\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0menable_iterative_imputer\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mimpute\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mIterativeImputer\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlinear_model\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mBayesianRidge\u001b[0m \u001b[1;31m# Imputation via linear regression\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'sklearn.experimental'"
     ]
    }
   ],
   "source": [
    "# This code snippet may not work, so do not worry about it, and just examine the similarities with the SimpleImputer\n",
    "# To use this experimental feature, we need to explicitly ask for it:\n",
    "from sklearn.experimental import enable_iterative_imputer  \n",
    "from sklearn.impute import IterativeImputer\n",
    "from sklearn.linear_model import BayesianRidge # Imputation via linear regression\n",
    "from sklearn.neighbors import KNeighborsRegressor # Imputation via KNN\n",
    "# Define imputer\n",
    "reg_imputer = IterativeImputer(BayesianRidge, max_iter=5, random_state=0)\n",
    "subset_nfl_data = nfl_data.loc[:, 'EPA':'Season'].head()\n",
    "imputed_subset_nfl_data = pd.DataFrame(mean_simpleimputer.fit_transform(subset_nfl_data), columns=subset_nfl_data.columns)\n",
    "print(imputed_subset_nfl_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\"> Exercise 1: Missing value handling on the San Francisco Building Permits Dataset </span>\n",
    "\n",
    "<span style=\"color:red\"><b> The <a href=\"https://www.kaggle.com/aparnashastry/building-permit-applications-data#Building_Permits.csv\"> Building permits dataset</a> pertains to all types of structural permits from Jan 1, 2013-Feb 25th 2018 in San Francisco. Data includes details on application/permit numbers, job addresses, supervisorial districts, and the current status of the applications. A data dictionary for the dataset is available in the data folder in this repository.</b></span>\n",
    "\n",
    "<span style=\"color:red\"> <b>The dataset is rather big and does not exist in the \"data\" folder in this repository. Download the dataset from the link provided and store it in your own data or working folder. Import the dataset in the next cell, and work your way through the steps in the subsequent cells.</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Permit Number</th>\n",
       "      <th>Permit Type</th>\n",
       "      <th>Permit Type Definition</th>\n",
       "      <th>Permit Creation Date</th>\n",
       "      <th>Block</th>\n",
       "      <th>Lot</th>\n",
       "      <th>Street Number</th>\n",
       "      <th>Street Number Suffix</th>\n",
       "      <th>Street Name</th>\n",
       "      <th>Street Suffix</th>\n",
       "      <th>...</th>\n",
       "      <th>Existing Construction Type</th>\n",
       "      <th>Existing Construction Type Description</th>\n",
       "      <th>Proposed Construction Type</th>\n",
       "      <th>Proposed Construction Type Description</th>\n",
       "      <th>Site Permit</th>\n",
       "      <th>Supervisor District</th>\n",
       "      <th>Neighborhoods - Analysis Boundaries</th>\n",
       "      <th>Zipcode</th>\n",
       "      <th>Location</th>\n",
       "      <th>Record ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>201505065519</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>05/06/2015</td>\n",
       "      <td>0326</td>\n",
       "      <td>023</td>\n",
       "      <td>140</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ellis</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.785719256680785, -122.40852313194863)</td>\n",
       "      <td>1380611233945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>201604195146</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>04/19/2016</td>\n",
       "      <td>0306</td>\n",
       "      <td>007</td>\n",
       "      <td>440</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Geary</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.78733980600732, -122.41063199757738)</td>\n",
       "      <td>1420164406718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>201605278609</td>\n",
       "      <td>3</td>\n",
       "      <td>additions alterations or repairs</td>\n",
       "      <td>05/27/2016</td>\n",
       "      <td>0595</td>\n",
       "      <td>203</td>\n",
       "      <td>1647</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>Av</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>constr type 1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>constr type 1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Russian Hill</td>\n",
       "      <td>94109.0</td>\n",
       "      <td>(37.7946573324287, -122.42232562979227)</td>\n",
       "      <td>1424856504716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>201611072166</td>\n",
       "      <td>8</td>\n",
       "      <td>otc alterations permit</td>\n",
       "      <td>11/07/2016</td>\n",
       "      <td>0156</td>\n",
       "      <td>011</td>\n",
       "      <td>1230</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>Av</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>wood frame (5)</td>\n",
       "      <td>5.0</td>\n",
       "      <td>wood frame (5)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Nob Hill</td>\n",
       "      <td>94109.0</td>\n",
       "      <td>(37.79595867909168, -122.41557405519474)</td>\n",
       "      <td>1443574295566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>201611283529</td>\n",
       "      <td>6</td>\n",
       "      <td>demolitions</td>\n",
       "      <td>11/28/2016</td>\n",
       "      <td>0342</td>\n",
       "      <td>001</td>\n",
       "      <td>950</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Market</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.78315261897309, -122.40950883997789)</td>\n",
       "      <td>144548169992</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  Permit Number  Permit Type            Permit Type Definition  \\\n",
       "0  201505065519            4                      sign - erect   \n",
       "1  201604195146            4                      sign - erect   \n",
       "2  201605278609            3  additions alterations or repairs   \n",
       "3  201611072166            8            otc alterations permit   \n",
       "4  201611283529            6                       demolitions   \n",
       "\n",
       "  Permit Creation Date Block  Lot  Street Number Street Number Suffix  \\\n",
       "0           05/06/2015  0326  023            140                  NaN   \n",
       "1           04/19/2016  0306  007            440                  NaN   \n",
       "2           05/27/2016  0595  203           1647                  NaN   \n",
       "3           11/07/2016  0156  011           1230                  NaN   \n",
       "4           11/28/2016  0342  001            950                  NaN   \n",
       "\n",
       "  Street Name Street Suffix  ...  Existing Construction Type  \\\n",
       "0       Ellis            St  ...                         3.0   \n",
       "1       Geary            St  ...                         3.0   \n",
       "2     Pacific            Av  ...                         1.0   \n",
       "3     Pacific            Av  ...                         5.0   \n",
       "4      Market            St  ...                         3.0   \n",
       "\n",
       "  Existing Construction Type Description Proposed Construction Type  \\\n",
       "0                          constr type 3                        NaN   \n",
       "1                          constr type 3                        NaN   \n",
       "2                          constr type 1                        1.0   \n",
       "3                         wood frame (5)                        5.0   \n",
       "4                          constr type 3                        NaN   \n",
       "\n",
       "  Proposed Construction Type Description Site Permit Supervisor District  \\\n",
       "0                                    NaN         NaN                 3.0   \n",
       "1                                    NaN         NaN                 3.0   \n",
       "2                          constr type 1         NaN                 3.0   \n",
       "3                         wood frame (5)         NaN                 3.0   \n",
       "4                                    NaN         NaN                 6.0   \n",
       "\n",
       "  Neighborhoods - Analysis Boundaries  Zipcode  \\\n",
       "0                          Tenderloin  94102.0   \n",
       "1                          Tenderloin  94102.0   \n",
       "2                        Russian Hill  94109.0   \n",
       "3                            Nob Hill  94109.0   \n",
       "4                          Tenderloin  94102.0   \n",
       "\n",
       "                                    Location      Record ID  \n",
       "0  (37.785719256680785, -122.40852313194863)  1380611233945  \n",
       "1   (37.78733980600732, -122.41063199757738)  1420164406718  \n",
       "2    (37.7946573324287, -122.42232562979227)  1424856504716  \n",
       "3   (37.79595867909168, -122.41557405519474)  1443574295566  \n",
       "4   (37.78315261897309, -122.40950883997789)   144548169992  \n",
       "\n",
       "[5 rows x 43 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# modules we'll use\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# read in all our data\n",
    "sanFran_permits = pd.read_csv(\"data/building-permit-applications-data/Building_Permits.csv\")\n",
    "sanFran_permits.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\"> <b>Find out what percent of the building permits dataset is missing. </b></span>\n",
    "<span style=\"color:red\"> <b><br>Write down your code in the following cell. </b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26.26002315058403"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_values = sanFran_permits.isnull().sum().sum()\n",
    "all_data = np.product(sanFran_permits.shape)\n",
    "(missing_values/all_data) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\"><b>Write a code to check the values of the `Street Number Suffix` and `Zipcode` from the building permits datasets. Both of these contain missing values. Which, if either, of these are missing because they don't exist? Which, if either, are missing because they weren't recorded?</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       Street Number Suffix  Zipcode\n",
      "0                       NaN  94102.0\n",
      "1                       NaN  94102.0\n",
      "2                       NaN  94109.0\n",
      "3                       NaN  94109.0\n",
      "4                       NaN  94102.0\n",
      "5                       NaN  94107.0\n",
      "6                       NaN  94122.0\n",
      "7                       NaN  94124.0\n",
      "8                       NaN  94117.0\n",
      "9                       NaN  94117.0\n",
      "10                      NaN  94114.0\n",
      "11                      NaN  94102.0\n",
      "12                      NaN  94114.0\n",
      "13                      NaN  94131.0\n",
      "14                      NaN  94115.0\n",
      "15                      NaN  94108.0\n",
      "16                      NaN  94104.0\n",
      "17                      NaN  94114.0\n",
      "18                      NaN  94123.0\n",
      "19                      NaN  94114.0\n",
      "20                      NaN  94117.0\n",
      "21                      NaN  94110.0\n",
      "22                      NaN  94133.0\n",
      "23                      NaN  94123.0\n",
      "24                      NaN  94123.0\n",
      "25                      NaN  94122.0\n",
      "26                      NaN  94121.0\n",
      "27                      NaN  94117.0\n",
      "28                      NaN  94122.0\n",
      "29                      NaN  94102.0\n",
      "...                     ...      ...\n",
      "198870                  NaN  94109.0\n",
      "198871                  NaN  94117.0\n",
      "198872                  NaN  94114.0\n",
      "198873                  NaN  94110.0\n",
      "198874                  NaN  94131.0\n",
      "198875                  NaN  94102.0\n",
      "198876                  NaN  94117.0\n",
      "198877                  NaN  94112.0\n",
      "198878                  NaN  94110.0\n",
      "198879                  NaN  94112.0\n",
      "198880                  NaN  94122.0\n",
      "198881                  NaN  94133.0\n",
      "198882                  NaN  94112.0\n",
      "198883                  NaN  94102.0\n",
      "198884                  NaN  94103.0\n",
      "198885                  NaN  94116.0\n",
      "198886                  NaN      NaN\n",
      "198887                  NaN      NaN\n",
      "198888                  NaN      NaN\n",
      "198889                  NaN      NaN\n",
      "198890                  NaN      NaN\n",
      "198891                  NaN      NaN\n",
      "198892                  NaN      NaN\n",
      "198893                  NaN      NaN\n",
      "198894                  NaN      NaN\n",
      "198895                  NaN      NaN\n",
      "198896                  NaN      NaN\n",
      "198897                  NaN      NaN\n",
      "198898                  NaN      NaN\n",
      "198899                  NaN      NaN\n",
      "\n",
      "[198900 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "print(sanFran_permits[['Street Number Suffix', 'Zipcode']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\"><b>See if you can drop the missing values from the building permits dataset and still keep some data, or drop the columns with missing values and see if you have columns still.</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Permit Number</th>\n",
       "      <th>Permit Type</th>\n",
       "      <th>Permit Type Definition</th>\n",
       "      <th>Permit Creation Date</th>\n",
       "      <th>Block</th>\n",
       "      <th>Lot</th>\n",
       "      <th>Street Number</th>\n",
       "      <th>Street Name</th>\n",
       "      <th>Current Status</th>\n",
       "      <th>Current Status Date</th>\n",
       "      <th>Filed Date</th>\n",
       "      <th>Record ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>201505065519</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>05/06/2015</td>\n",
       "      <td>0326</td>\n",
       "      <td>023</td>\n",
       "      <td>140</td>\n",
       "      <td>Ellis</td>\n",
       "      <td>expired</td>\n",
       "      <td>12/21/2017</td>\n",
       "      <td>05/06/2015</td>\n",
       "      <td>1380611233945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>201604195146</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>04/19/2016</td>\n",
       "      <td>0306</td>\n",
       "      <td>007</td>\n",
       "      <td>440</td>\n",
       "      <td>Geary</td>\n",
       "      <td>issued</td>\n",
       "      <td>08/03/2017</td>\n",
       "      <td>04/19/2016</td>\n",
       "      <td>1420164406718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>201605278609</td>\n",
       "      <td>3</td>\n",
       "      <td>additions alterations or repairs</td>\n",
       "      <td>05/27/2016</td>\n",
       "      <td>0595</td>\n",
       "      <td>203</td>\n",
       "      <td>1647</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>withdrawn</td>\n",
       "      <td>09/26/2017</td>\n",
       "      <td>05/27/2016</td>\n",
       "      <td>1424856504716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>201611072166</td>\n",
       "      <td>8</td>\n",
       "      <td>otc alterations permit</td>\n",
       "      <td>11/07/2016</td>\n",
       "      <td>0156</td>\n",
       "      <td>011</td>\n",
       "      <td>1230</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>complete</td>\n",
       "      <td>07/24/2017</td>\n",
       "      <td>11/07/2016</td>\n",
       "      <td>1443574295566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>201611283529</td>\n",
       "      <td>6</td>\n",
       "      <td>demolitions</td>\n",
       "      <td>11/28/2016</td>\n",
       "      <td>0342</td>\n",
       "      <td>001</td>\n",
       "      <td>950</td>\n",
       "      <td>Market</td>\n",
       "      <td>issued</td>\n",
       "      <td>12/01/2017</td>\n",
       "      <td>11/28/2016</td>\n",
       "      <td>144548169992</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Permit Number  Permit Type            Permit Type Definition  \\\n",
       "0  201505065519            4                      sign - erect   \n",
       "1  201604195146            4                      sign - erect   \n",
       "2  201605278609            3  additions alterations or repairs   \n",
       "3  201611072166            8            otc alterations permit   \n",
       "4  201611283529            6                       demolitions   \n",
       "\n",
       "  Permit Creation Date Block  Lot  Street Number Street Name Current Status  \\\n",
       "0           05/06/2015  0326  023            140       Ellis        expired   \n",
       "1           04/19/2016  0306  007            440       Geary         issued   \n",
       "2           05/27/2016  0595  203           1647     Pacific      withdrawn   \n",
       "3           11/07/2016  0156  011           1230     Pacific       complete   \n",
       "4           11/28/2016  0342  001            950      Market         issued   \n",
       "\n",
       "  Current Status Date  Filed Date      Record ID  \n",
       "0          12/21/2017  05/06/2015  1380611233945  \n",
       "1          08/03/2017  04/19/2016  1420164406718  \n",
       "2          09/26/2017  05/27/2016  1424856504716  \n",
       "3          07/24/2017  11/07/2016  1443574295566  \n",
       "4          12/01/2017  11/28/2016   144548169992  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns_with_na_dropped = sanFran_permits.dropna(axis=1) # axis 1 indicates operation along columns\n",
    "columns_with_na_dropped.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\"><b>Try replacing all the `NaN`'s in the building permits data with the one that comes directly after it and then replacing any remaining `NaN`'s with 0.</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Permit Number                             0\n",
       "Permit Type                               0\n",
       "Permit Type Definition                    0\n",
       "Permit Creation Date                      0\n",
       "Block                                     0\n",
       "Lot                                       0\n",
       "Street Number                             0\n",
       "Street Number Suffix                      0\n",
       "Street Name                               0\n",
       "Street Suffix                             0\n",
       "Unit                                      0\n",
       "Unit Suffix                               0\n",
       "Description                               0\n",
       "Current Status                            0\n",
       "Current Status Date                       0\n",
       "Filed Date                                0\n",
       "Issued Date                               0\n",
       "Completed Date                            0\n",
       "First Construction Document Date          0\n",
       "Structural Notification                   0\n",
       "Number of Existing Stories                0\n",
       "Number of Proposed Stories                0\n",
       "Voluntary Soft-Story Retrofit             0\n",
       "Fire Only Permit                          0\n",
       "Permit Expiration Date                    0\n",
       "Estimated Cost                            0\n",
       "Revised Cost                              0\n",
       "Existing Use                              0\n",
       "Existing Units                            0\n",
       "Proposed Use                              0\n",
       "Proposed Units                            0\n",
       "Plansets                                  0\n",
       "TIDF Compliance                           0\n",
       "Existing Construction Type                0\n",
       "Existing Construction Type Description    0\n",
       "Proposed Construction Type                0\n",
       "Proposed Construction Type Description    0\n",
       "Site Permit                               0\n",
       "Supervisor District                       0\n",
       "Neighborhoods - Analysis Boundaries       0\n",
       "Zipcode                                   0\n",
       "Location                                  0\n",
       "Record ID                                 0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sanFran_permits.bfill(axis=0, inplace = True)\n",
    "sanFran_permits.fillna(0, inplace=True)\n",
    "# sanFran_permits.fillna(method = 'bfill',.fillna(0, inplace=True)\n",
    "sanFran_permits.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\"><b>Create a new dataset at the ending with all missing values handled and store as a CSV file.</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "sanFran_permits.to_csv(\"data\\PA4EX1.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\"> Exercise 2: Missing value handling on the  Pima Indians Diabetes Dataset</span>\n",
    "\n",
    "<span style=\"color:red\"><b> The <a href= \"https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv\"> Pima Indians Diabetes Dataset</a> involves predicting the onset of diabetes within 5 years in Pima Indians given medical details.</b></span>\n",
    "\n",
    "<span style=\"color:red\">There are 768 observations with 8 input variables and 1 output variable. The variable names are as follows:</span>\n",
    "\n",
    "<br><span style=\"color:red\">0. Number of times pregnant.</span>\n",
    "<br><span style=\"color:red\">1. Plasma glucose concentration a 2 hours in an oral glucose tolerance test.</span>\n",
    "<br><span style=\"color:red\">2. Diastolic blood pressure (mm Hg).</span>\n",
    "<br><span style=\"color:red\">3. Triceps skinfold thickness (mm).</span>\n",
    "<br><span style=\"color:red\">4. 2-Hour serum insulin (mu U/ml).</span>\n",
    "<br><span style=\"color:red\">5. Body mass index (weight in kg/(height in m)^2).</span>\n",
    "<br><span style=\"color:red\">6. Diabetes pedigree function.</span>\n",
    "<br><span style=\"color:red\">7. Age (years).\n",
    "<br><span style=\"color:red\">8. Class variable (0 or 1).</span>\n",
    "\n",
    "<span style=\"color:red\"> <b>This is a medical dataset, so an important thing we will do with this dataset is that we will see if the statistical summaries can reveal something about the missing values.</b></span>\n",
    "\n",
    "<span style=\"color:red\"> <b>The dataset is available in the `data` folder in this repository. Import the dataset in the next cell, and work your way through the steps in the subsequent cells.</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pregnant_no</th>\n",
       "      <th>plasma</th>\n",
       "      <th>blood_pressure</th>\n",
       "      <th>triceps</th>\n",
       "      <th>isulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>diabetes</th>\n",
       "      <th>age</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pregnant_no  plasma  blood_pressure  triceps  isulin   BMI  diabetes  age  \\\n",
       "0            6     148              72       35       0  33.6     0.627   50   \n",
       "1            1      85              66       29       0  26.6     0.351   31   \n",
       "2            8     183              64        0       0  23.3     0.672   32   \n",
       "3            1      89              66       23      94  28.1     0.167   21   \n",
       "4            0     137              40       35     168  43.1     2.288   33   \n",
       "\n",
       "   class  \n",
       "0      1  \n",
       "1      0  \n",
       "2      1  \n",
       "3      0  \n",
       "4      1  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "columns=[\"pregnant_no\", \"plasma\", \"blood_pressure\", \"triceps\", \"isulin\", \"BMI\", \"diabetes\", \"age\", \"class\"]\n",
    "data = pd.read_csv(\"data/pima-indians-diabetes.data.csv\", header=None,  names=columns)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\"> <b>Use the describe method used for pandas data frames to obtain a statistucal summary of the dataset.</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pregnant_no</th>\n",
       "      <th>plasma</th>\n",
       "      <th>blood_pressure</th>\n",
       "      <th>triceps</th>\n",
       "      <th>isulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>diabetes</th>\n",
       "      <th>age</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.845052</td>\n",
       "      <td>120.894531</td>\n",
       "      <td>69.105469</td>\n",
       "      <td>20.536458</td>\n",
       "      <td>79.799479</td>\n",
       "      <td>31.992578</td>\n",
       "      <td>0.471876</td>\n",
       "      <td>33.240885</td>\n",
       "      <td>0.348958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.369578</td>\n",
       "      <td>31.972618</td>\n",
       "      <td>19.355807</td>\n",
       "      <td>15.952218</td>\n",
       "      <td>115.244002</td>\n",
       "      <td>7.884160</td>\n",
       "      <td>0.331329</td>\n",
       "      <td>11.760232</td>\n",
       "      <td>0.476951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.300000</td>\n",
       "      <td>0.243750</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>140.250000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>127.250000</td>\n",
       "      <td>36.600000</td>\n",
       "      <td>0.626250</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>199.000000</td>\n",
       "      <td>122.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>846.000000</td>\n",
       "      <td>67.100000</td>\n",
       "      <td>2.420000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       pregnant_no      plasma  blood_pressure     triceps      isulin  \\\n",
       "count   768.000000  768.000000      768.000000  768.000000  768.000000   \n",
       "mean      3.845052  120.894531       69.105469   20.536458   79.799479   \n",
       "std       3.369578   31.972618       19.355807   15.952218  115.244002   \n",
       "min       0.000000    0.000000        0.000000    0.000000    0.000000   \n",
       "25%       1.000000   99.000000       62.000000    0.000000    0.000000   \n",
       "50%       3.000000  117.000000       72.000000   23.000000   30.500000   \n",
       "75%       6.000000  140.250000       80.000000   32.000000  127.250000   \n",
       "max      17.000000  199.000000      122.000000   99.000000  846.000000   \n",
       "\n",
       "              BMI    diabetes         age       class  \n",
       "count  768.000000  768.000000  768.000000  768.000000  \n",
       "mean    31.992578    0.471876   33.240885    0.348958  \n",
       "std      7.884160    0.331329   11.760232    0.476951  \n",
       "min      0.000000    0.078000   21.000000    0.000000  \n",
       "25%     27.300000    0.243750   24.000000    0.000000  \n",
       "50%     32.000000    0.372500   29.000000    0.000000  \n",
       "75%     36.600000    0.626250   41.000000    1.000000  \n",
       "max     67.100000    2.420000   81.000000    1.000000  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\"><b>There are columns that have a minimum value of zero (0). On some columns, a value of zero does not make sense and indicates an invalid or missing value. For example, the `BMI` for a person cannot be 0. See if there are other columns/attributes whose value of 0 does not make sense. An understanding of the medical columns may be needed.</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pregnant_no</th>\n",
       "      <th>plasma</th>\n",
       "      <th>blood_pressure</th>\n",
       "      <th>triceps</th>\n",
       "      <th>isulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>diabetes</th>\n",
       "      <th>age</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>116</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25.6</td>\n",
       "      <td>0.201</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>10</td>\n",
       "      <td>115</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35.3</td>\n",
       "      <td>0.134</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>8</td>\n",
       "      <td>125</td>\n",
       "      <td>96</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.232</td>\n",
       "      <td>54</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4</td>\n",
       "      <td>110</td>\n",
       "      <td>92</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>37.6</td>\n",
       "      <td>0.191</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>10</td>\n",
       "      <td>168</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>0.537</td>\n",
       "      <td>34</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>10</td>\n",
       "      <td>139</td>\n",
       "      <td>80</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27.1</td>\n",
       "      <td>1.441</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>7</td>\n",
       "      <td>100</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.484</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>7</td>\n",
       "      <td>107</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29.6</td>\n",
       "      <td>0.254</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>8</td>\n",
       "      <td>99</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35.4</td>\n",
       "      <td>0.388</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>7</td>\n",
       "      <td>196</td>\n",
       "      <td>90</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>39.8</td>\n",
       "      <td>0.451</td>\n",
       "      <td>41</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>9</td>\n",
       "      <td>119</td>\n",
       "      <td>80</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.263</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>7</td>\n",
       "      <td>147</td>\n",
       "      <td>76</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>39.4</td>\n",
       "      <td>0.257</td>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>5</td>\n",
       "      <td>117</td>\n",
       "      <td>92</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>34.1</td>\n",
       "      <td>0.337</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>5</td>\n",
       "      <td>109</td>\n",
       "      <td>75</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>0.546</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>6</td>\n",
       "      <td>92</td>\n",
       "      <td>92</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>19.9</td>\n",
       "      <td>0.188</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>10</td>\n",
       "      <td>122</td>\n",
       "      <td>78</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>27.6</td>\n",
       "      <td>0.512</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>11</td>\n",
       "      <td>138</td>\n",
       "      <td>76</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>33.2</td>\n",
       "      <td>0.420</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>9</td>\n",
       "      <td>102</td>\n",
       "      <td>76</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>32.9</td>\n",
       "      <td>0.665</td>\n",
       "      <td>46</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>2</td>\n",
       "      <td>90</td>\n",
       "      <td>68</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>38.2</td>\n",
       "      <td>0.503</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>7</td>\n",
       "      <td>133</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40.2</td>\n",
       "      <td>0.696</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>7</td>\n",
       "      <td>106</td>\n",
       "      <td>92</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>22.7</td>\n",
       "      <td>0.235</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>7</td>\n",
       "      <td>159</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27.4</td>\n",
       "      <td>0.294</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>0</td>\n",
       "      <td>180</td>\n",
       "      <td>66</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>1.893</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>1</td>\n",
       "      <td>146</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29.7</td>\n",
       "      <td>0.564</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>2</td>\n",
       "      <td>71</td>\n",
       "      <td>70</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.586</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>7</td>\n",
       "      <td>103</td>\n",
       "      <td>66</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>39.1</td>\n",
       "      <td>0.344</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>7</td>\n",
       "      <td>105</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.305</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>712</th>\n",
       "      <td>10</td>\n",
       "      <td>129</td>\n",
       "      <td>62</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>41.2</td>\n",
       "      <td>0.441</td>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>714</th>\n",
       "      <td>3</td>\n",
       "      <td>102</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29.5</td>\n",
       "      <td>0.121</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>717</th>\n",
       "      <td>10</td>\n",
       "      <td>94</td>\n",
       "      <td>72</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>23.1</td>\n",
       "      <td>0.595</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>719</th>\n",
       "      <td>5</td>\n",
       "      <td>97</td>\n",
       "      <td>76</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>35.6</td>\n",
       "      <td>0.378</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>720</th>\n",
       "      <td>4</td>\n",
       "      <td>83</td>\n",
       "      <td>86</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>29.3</td>\n",
       "      <td>0.317</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>1</td>\n",
       "      <td>111</td>\n",
       "      <td>94</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>32.8</td>\n",
       "      <td>0.265</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>725</th>\n",
       "      <td>4</td>\n",
       "      <td>112</td>\n",
       "      <td>78</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>39.4</td>\n",
       "      <td>0.236</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>727</th>\n",
       "      <td>0</td>\n",
       "      <td>141</td>\n",
       "      <td>84</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>32.4</td>\n",
       "      <td>0.433</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>2</td>\n",
       "      <td>175</td>\n",
       "      <td>88</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22.9</td>\n",
       "      <td>0.326</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>729</th>\n",
       "      <td>2</td>\n",
       "      <td>92</td>\n",
       "      <td>52</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.141</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>731</th>\n",
       "      <td>8</td>\n",
       "      <td>120</td>\n",
       "      <td>86</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>28.4</td>\n",
       "      <td>0.259</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>734</th>\n",
       "      <td>2</td>\n",
       "      <td>105</td>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.560</td>\n",
       "      <td>53</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>735</th>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>60</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>35.4</td>\n",
       "      <td>0.284</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>737</th>\n",
       "      <td>8</td>\n",
       "      <td>65</td>\n",
       "      <td>72</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.600</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>739</th>\n",
       "      <td>1</td>\n",
       "      <td>102</td>\n",
       "      <td>74</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>39.5</td>\n",
       "      <td>0.293</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>743</th>\n",
       "      <td>9</td>\n",
       "      <td>140</td>\n",
       "      <td>94</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>32.7</td>\n",
       "      <td>0.734</td>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>746</th>\n",
       "      <td>1</td>\n",
       "      <td>147</td>\n",
       "      <td>94</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>49.3</td>\n",
       "      <td>0.358</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>749</th>\n",
       "      <td>6</td>\n",
       "      <td>162</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24.3</td>\n",
       "      <td>0.178</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>750</th>\n",
       "      <td>4</td>\n",
       "      <td>136</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>31.2</td>\n",
       "      <td>1.182</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>752</th>\n",
       "      <td>3</td>\n",
       "      <td>108</td>\n",
       "      <td>62</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.223</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>754</th>\n",
       "      <td>8</td>\n",
       "      <td>154</td>\n",
       "      <td>78</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>32.4</td>\n",
       "      <td>0.443</td>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>756</th>\n",
       "      <td>7</td>\n",
       "      <td>137</td>\n",
       "      <td>90</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.391</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>757</th>\n",
       "      <td>0</td>\n",
       "      <td>123</td>\n",
       "      <td>72</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36.3</td>\n",
       "      <td>0.258</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>758</th>\n",
       "      <td>1</td>\n",
       "      <td>106</td>\n",
       "      <td>76</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>37.5</td>\n",
       "      <td>0.197</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>759</th>\n",
       "      <td>6</td>\n",
       "      <td>190</td>\n",
       "      <td>92</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35.5</td>\n",
       "      <td>0.278</td>\n",
       "      <td>66</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>761</th>\n",
       "      <td>9</td>\n",
       "      <td>170</td>\n",
       "      <td>74</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>0.403</td>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>762</th>\n",
       "      <td>9</td>\n",
       "      <td>89</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22.5</td>\n",
       "      <td>0.142</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>2</td>\n",
       "      <td>122</td>\n",
       "      <td>70</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>36.8</td>\n",
       "      <td>0.340</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>766</th>\n",
       "      <td>1</td>\n",
       "      <td>126</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.349</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>1</td>\n",
       "      <td>93</td>\n",
       "      <td>70</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>30.4</td>\n",
       "      <td>0.315</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>374 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     pregnant_no  plasma  blood_pressure  triceps  isulin   BMI  diabetes  \\\n",
       "0              6     148              72       35       0  33.6     0.627   \n",
       "1              1      85              66       29       0  26.6     0.351   \n",
       "2              8     183              64        0       0  23.3     0.672   \n",
       "5              5     116              74        0       0  25.6     0.201   \n",
       "7             10     115               0        0       0  35.3     0.134   \n",
       "9              8     125              96        0       0   0.0     0.232   \n",
       "10             4     110              92        0       0  37.6     0.191   \n",
       "11            10     168              74        0       0  38.0     0.537   \n",
       "12            10     139              80        0       0  27.1     1.441   \n",
       "15             7     100               0        0       0  30.0     0.484   \n",
       "17             7     107              74        0       0  29.6     0.254   \n",
       "21             8      99              84        0       0  35.4     0.388   \n",
       "22             7     196              90        0       0  39.8     0.451   \n",
       "23             9     119              80       35       0  29.0     0.263   \n",
       "26             7     147              76        0       0  39.4     0.257   \n",
       "29             5     117              92        0       0  34.1     0.337   \n",
       "30             5     109              75       26       0  36.0     0.546   \n",
       "33             6      92              92        0       0  19.9     0.188   \n",
       "34            10     122              78       31       0  27.6     0.512   \n",
       "36            11     138              76        0       0  33.2     0.420   \n",
       "37             9     102              76       37       0  32.9     0.665   \n",
       "38             2      90              68       42       0  38.2     0.503   \n",
       "41             7     133              84        0       0  40.2     0.696   \n",
       "42             7     106              92       18       0  22.7     0.235   \n",
       "44             7     159              64        0       0  27.4     0.294   \n",
       "45             0     180              66       39       0  42.0     1.893   \n",
       "46             1     146              56        0       0  29.7     0.564   \n",
       "47             2      71              70       27       0  28.0     0.586   \n",
       "48             7     103              66       32       0  39.1     0.344   \n",
       "49             7     105               0        0       0   0.0     0.305   \n",
       "..           ...     ...             ...      ...     ...   ...       ...   \n",
       "712           10     129              62       36       0  41.2     0.441   \n",
       "714            3     102              74        0       0  29.5     0.121   \n",
       "717           10      94              72       18       0  23.1     0.595   \n",
       "719            5      97              76       27       0  35.6     0.378   \n",
       "720            4      83              86       19       0  29.3     0.317   \n",
       "724            1     111              94        0       0  32.8     0.265   \n",
       "725            4     112              78       40       0  39.4     0.236   \n",
       "727            0     141              84       26       0  32.4     0.433   \n",
       "728            2     175              88        0       0  22.9     0.326   \n",
       "729            2      92              52        0       0  30.1     0.141   \n",
       "731            8     120              86        0       0  28.4     0.259   \n",
       "734            2     105              75        0       0  23.3     0.560   \n",
       "735            4      95              60       32       0  35.4     0.284   \n",
       "737            8      65              72       23       0  32.0     0.600   \n",
       "739            1     102              74        0       0  39.5     0.293   \n",
       "743            9     140              94        0       0  32.7     0.734   \n",
       "746            1     147              94       41       0  49.3     0.358   \n",
       "749            6     162              62        0       0  24.3     0.178   \n",
       "750            4     136              70        0       0  31.2     1.182   \n",
       "752            3     108              62       24       0  26.0     0.223   \n",
       "754            8     154              78       32       0  32.4     0.443   \n",
       "756            7     137              90       41       0  32.0     0.391   \n",
       "757            0     123              72        0       0  36.3     0.258   \n",
       "758            1     106              76        0       0  37.5     0.197   \n",
       "759            6     190              92        0       0  35.5     0.278   \n",
       "761            9     170              74       31       0  44.0     0.403   \n",
       "762            9      89              62        0       0  22.5     0.142   \n",
       "764            2     122              70       27       0  36.8     0.340   \n",
       "766            1     126              60        0       0  30.1     0.349   \n",
       "767            1      93              70       31       0  30.4     0.315   \n",
       "\n",
       "     age  class  \n",
       "0     50      1  \n",
       "1     31      0  \n",
       "2     32      1  \n",
       "5     30      0  \n",
       "7     29      0  \n",
       "9     54      1  \n",
       "10    30      0  \n",
       "11    34      1  \n",
       "12    57      0  \n",
       "15    32      1  \n",
       "17    31      1  \n",
       "21    50      0  \n",
       "22    41      1  \n",
       "23    29      1  \n",
       "26    43      1  \n",
       "29    38      0  \n",
       "30    60      0  \n",
       "33    28      0  \n",
       "34    45      0  \n",
       "36    35      0  \n",
       "37    46      1  \n",
       "38    27      1  \n",
       "41    37      0  \n",
       "42    48      0  \n",
       "44    40      0  \n",
       "45    25      1  \n",
       "46    29      0  \n",
       "47    22      0  \n",
       "48    31      1  \n",
       "49    24      0  \n",
       "..   ...    ...  \n",
       "712   38      1  \n",
       "714   32      0  \n",
       "717   56      0  \n",
       "719   52      1  \n",
       "720   34      0  \n",
       "724   45      0  \n",
       "725   38      0  \n",
       "727   22      0  \n",
       "728   22      0  \n",
       "729   22      0  \n",
       "731   22      1  \n",
       "734   53      0  \n",
       "735   28      0  \n",
       "737   42      0  \n",
       "739   42      1  \n",
       "743   45      1  \n",
       "746   27      1  \n",
       "749   50      1  \n",
       "750   22      1  \n",
       "752   25      0  \n",
       "754   45      1  \n",
       "756   39      0  \n",
       "757   52      1  \n",
       "758   26      0  \n",
       "759   66      1  \n",
       "761   43      1  \n",
       "762   33      0  \n",
       "764   27      0  \n",
       "766   47      1  \n",
       "767   23      0  \n",
       "\n",
       "[374 rows x 9 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data.loc[data['BMI'] == 0]\n",
    "# data.loc[data['blood_pressure'] == 0]\n",
    "data.loc[data['isulin'] == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\"><b>Now you can proceed to check the missing values and remove/impute as in exercise 1. Create a new dataset at the ending with all missing values handled and store as a CSV file.</b></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pregnant_no</th>\n",
       "      <th>plasma</th>\n",
       "      <th>blood_pressure</th>\n",
       "      <th>triceps</th>\n",
       "      <th>isulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>diabetes</th>\n",
       "      <th>age</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>116</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25.6</td>\n",
       "      <td>0.201</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>32</td>\n",
       "      <td>88</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.248</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>10</td>\n",
       "      <td>115</td>\n",
       "      <td>69.690885</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35.3</td>\n",
       "      <td>0.134</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>197</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>45</td>\n",
       "      <td>543</td>\n",
       "      <td>30.5</td>\n",
       "      <td>0.158</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4</td>\n",
       "      <td>110</td>\n",
       "      <td>92.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>37.6</td>\n",
       "      <td>0.191</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>10</td>\n",
       "      <td>168</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>0.537</td>\n",
       "      <td>34</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>10</td>\n",
       "      <td>139</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>27.1</td>\n",
       "      <td>1.441</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>189</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>23</td>\n",
       "      <td>846</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.398</td>\n",
       "      <td>59</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>5</td>\n",
       "      <td>166</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>19</td>\n",
       "      <td>175</td>\n",
       "      <td>25.8</td>\n",
       "      <td>0.587</td>\n",
       "      <td>51</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>7</td>\n",
       "      <td>100</td>\n",
       "      <td>69.690885</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.484</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0</td>\n",
       "      <td>118</td>\n",
       "      <td>84.000000</td>\n",
       "      <td>47</td>\n",
       "      <td>230</td>\n",
       "      <td>45.8</td>\n",
       "      <td>0.551</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>7</td>\n",
       "      <td>107</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29.6</td>\n",
       "      <td>0.254</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>103</td>\n",
       "      <td>30.000000</td>\n",
       "      <td>38</td>\n",
       "      <td>83</td>\n",
       "      <td>43.3</td>\n",
       "      <td>0.183</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>115</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>30</td>\n",
       "      <td>96</td>\n",
       "      <td>34.6</td>\n",
       "      <td>0.529</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>3</td>\n",
       "      <td>126</td>\n",
       "      <td>88.000000</td>\n",
       "      <td>41</td>\n",
       "      <td>235</td>\n",
       "      <td>39.3</td>\n",
       "      <td>0.704</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>8</td>\n",
       "      <td>99</td>\n",
       "      <td>84.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35.4</td>\n",
       "      <td>0.388</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>7</td>\n",
       "      <td>196</td>\n",
       "      <td>90.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>39.8</td>\n",
       "      <td>0.451</td>\n",
       "      <td>41</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>9</td>\n",
       "      <td>119</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.263</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>11</td>\n",
       "      <td>143</td>\n",
       "      <td>94.000000</td>\n",
       "      <td>33</td>\n",
       "      <td>146</td>\n",
       "      <td>36.6</td>\n",
       "      <td>0.254</td>\n",
       "      <td>51</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>10</td>\n",
       "      <td>125</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>26</td>\n",
       "      <td>115</td>\n",
       "      <td>31.1</td>\n",
       "      <td>0.205</td>\n",
       "      <td>41</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>7</td>\n",
       "      <td>147</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>39.4</td>\n",
       "      <td>0.257</td>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>1</td>\n",
       "      <td>97</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>15</td>\n",
       "      <td>140</td>\n",
       "      <td>23.2</td>\n",
       "      <td>0.487</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>13</td>\n",
       "      <td>145</td>\n",
       "      <td>82.000000</td>\n",
       "      <td>19</td>\n",
       "      <td>110</td>\n",
       "      <td>22.2</td>\n",
       "      <td>0.245</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>5</td>\n",
       "      <td>117</td>\n",
       "      <td>92.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>34.1</td>\n",
       "      <td>0.337</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>5</td>\n",
       "      <td>109</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>0.546</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>738</th>\n",
       "      <td>2</td>\n",
       "      <td>99</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>17</td>\n",
       "      <td>160</td>\n",
       "      <td>36.6</td>\n",
       "      <td>0.453</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>739</th>\n",
       "      <td>1</td>\n",
       "      <td>102</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>39.5</td>\n",
       "      <td>0.293</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>740</th>\n",
       "      <td>11</td>\n",
       "      <td>120</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>37</td>\n",
       "      <td>150</td>\n",
       "      <td>42.3</td>\n",
       "      <td>0.785</td>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>741</th>\n",
       "      <td>3</td>\n",
       "      <td>102</td>\n",
       "      <td>44.000000</td>\n",
       "      <td>20</td>\n",
       "      <td>94</td>\n",
       "      <td>30.8</td>\n",
       "      <td>0.400</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>742</th>\n",
       "      <td>1</td>\n",
       "      <td>109</td>\n",
       "      <td>58.000000</td>\n",
       "      <td>18</td>\n",
       "      <td>116</td>\n",
       "      <td>28.5</td>\n",
       "      <td>0.219</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>743</th>\n",
       "      <td>9</td>\n",
       "      <td>140</td>\n",
       "      <td>94.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>32.7</td>\n",
       "      <td>0.734</td>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>744</th>\n",
       "      <td>13</td>\n",
       "      <td>153</td>\n",
       "      <td>88.000000</td>\n",
       "      <td>37</td>\n",
       "      <td>140</td>\n",
       "      <td>40.6</td>\n",
       "      <td>1.174</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>745</th>\n",
       "      <td>12</td>\n",
       "      <td>100</td>\n",
       "      <td>84.000000</td>\n",
       "      <td>33</td>\n",
       "      <td>105</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.488</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>746</th>\n",
       "      <td>1</td>\n",
       "      <td>147</td>\n",
       "      <td>94.000000</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>49.3</td>\n",
       "      <td>0.358</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>747</th>\n",
       "      <td>1</td>\n",
       "      <td>81</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>41</td>\n",
       "      <td>57</td>\n",
       "      <td>46.3</td>\n",
       "      <td>1.096</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>748</th>\n",
       "      <td>3</td>\n",
       "      <td>187</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>22</td>\n",
       "      <td>200</td>\n",
       "      <td>36.4</td>\n",
       "      <td>0.408</td>\n",
       "      <td>36</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>749</th>\n",
       "      <td>6</td>\n",
       "      <td>162</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24.3</td>\n",
       "      <td>0.178</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>750</th>\n",
       "      <td>4</td>\n",
       "      <td>136</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>31.2</td>\n",
       "      <td>1.182</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>751</th>\n",
       "      <td>1</td>\n",
       "      <td>121</td>\n",
       "      <td>78.000000</td>\n",
       "      <td>39</td>\n",
       "      <td>74</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0.261</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>752</th>\n",
       "      <td>3</td>\n",
       "      <td>108</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.223</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>753</th>\n",
       "      <td>0</td>\n",
       "      <td>181</td>\n",
       "      <td>88.000000</td>\n",
       "      <td>44</td>\n",
       "      <td>510</td>\n",
       "      <td>43.3</td>\n",
       "      <td>0.222</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>754</th>\n",
       "      <td>8</td>\n",
       "      <td>154</td>\n",
       "      <td>78.000000</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>32.4</td>\n",
       "      <td>0.443</td>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>755</th>\n",
       "      <td>1</td>\n",
       "      <td>128</td>\n",
       "      <td>88.000000</td>\n",
       "      <td>39</td>\n",
       "      <td>110</td>\n",
       "      <td>36.5</td>\n",
       "      <td>1.057</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>756</th>\n",
       "      <td>7</td>\n",
       "      <td>137</td>\n",
       "      <td>90.000000</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.391</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>757</th>\n",
       "      <td>0</td>\n",
       "      <td>123</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>36.3</td>\n",
       "      <td>0.258</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>758</th>\n",
       "      <td>1</td>\n",
       "      <td>106</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>37.5</td>\n",
       "      <td>0.197</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>759</th>\n",
       "      <td>6</td>\n",
       "      <td>190</td>\n",
       "      <td>92.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35.5</td>\n",
       "      <td>0.278</td>\n",
       "      <td>66</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>760</th>\n",
       "      <td>2</td>\n",
       "      <td>88</td>\n",
       "      <td>58.000000</td>\n",
       "      <td>26</td>\n",
       "      <td>16</td>\n",
       "      <td>28.4</td>\n",
       "      <td>0.766</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>761</th>\n",
       "      <td>9</td>\n",
       "      <td>170</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>44.0</td>\n",
       "      <td>0.403</td>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>762</th>\n",
       "      <td>9</td>\n",
       "      <td>89</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22.5</td>\n",
       "      <td>0.142</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>763</th>\n",
       "      <td>10</td>\n",
       "      <td>101</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>48</td>\n",
       "      <td>180</td>\n",
       "      <td>32.9</td>\n",
       "      <td>0.171</td>\n",
       "      <td>63</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>2</td>\n",
       "      <td>122</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>36.8</td>\n",
       "      <td>0.340</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>765</th>\n",
       "      <td>5</td>\n",
       "      <td>121</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23</td>\n",
       "      <td>112</td>\n",
       "      <td>26.2</td>\n",
       "      <td>0.245</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>766</th>\n",
       "      <td>1</td>\n",
       "      <td>126</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.349</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>1</td>\n",
       "      <td>93</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>30.4</td>\n",
       "      <td>0.315</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>757 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     pregnant_no  plasma  blood_pressure  triceps  isulin   BMI  diabetes  \\\n",
       "0              6     148       72.000000       35       0  33.6     0.627   \n",
       "1              1      85       66.000000       29       0  26.6     0.351   \n",
       "2              8     183       64.000000        0       0  23.3     0.672   \n",
       "3              1      89       66.000000       23      94  28.1     0.167   \n",
       "4              0     137       40.000000       35     168  43.1     2.288   \n",
       "5              5     116       74.000000        0       0  25.6     0.201   \n",
       "6              3      78       50.000000       32      88  31.0     0.248   \n",
       "7             10     115       69.690885        0       0  35.3     0.134   \n",
       "8              2     197       70.000000       45     543  30.5     0.158   \n",
       "10             4     110       92.000000        0       0  37.6     0.191   \n",
       "11            10     168       74.000000        0       0  38.0     0.537   \n",
       "12            10     139       80.000000        0       0  27.1     1.441   \n",
       "13             1     189       60.000000       23     846  30.1     0.398   \n",
       "14             5     166       72.000000       19     175  25.8     0.587   \n",
       "15             7     100       69.690885        0       0  30.0     0.484   \n",
       "16             0     118       84.000000       47     230  45.8     0.551   \n",
       "17             7     107       74.000000        0       0  29.6     0.254   \n",
       "18             1     103       30.000000       38      83  43.3     0.183   \n",
       "19             1     115       70.000000       30      96  34.6     0.529   \n",
       "20             3     126       88.000000       41     235  39.3     0.704   \n",
       "21             8      99       84.000000        0       0  35.4     0.388   \n",
       "22             7     196       90.000000        0       0  39.8     0.451   \n",
       "23             9     119       80.000000       35       0  29.0     0.263   \n",
       "24            11     143       94.000000       33     146  36.6     0.254   \n",
       "25            10     125       70.000000       26     115  31.1     0.205   \n",
       "26             7     147       76.000000        0       0  39.4     0.257   \n",
       "27             1      97       66.000000       15     140  23.2     0.487   \n",
       "28            13     145       82.000000       19     110  22.2     0.245   \n",
       "29             5     117       92.000000        0       0  34.1     0.337   \n",
       "30             5     109       75.000000       26       0  36.0     0.546   \n",
       "..           ...     ...             ...      ...     ...   ...       ...   \n",
       "738            2      99       60.000000       17     160  36.6     0.453   \n",
       "739            1     102       74.000000        0       0  39.5     0.293   \n",
       "740           11     120       80.000000       37     150  42.3     0.785   \n",
       "741            3     102       44.000000       20      94  30.8     0.400   \n",
       "742            1     109       58.000000       18     116  28.5     0.219   \n",
       "743            9     140       94.000000        0       0  32.7     0.734   \n",
       "744           13     153       88.000000       37     140  40.6     1.174   \n",
       "745           12     100       84.000000       33     105  30.0     0.488   \n",
       "746            1     147       94.000000       41       0  49.3     0.358   \n",
       "747            1      81       74.000000       41      57  46.3     1.096   \n",
       "748            3     187       70.000000       22     200  36.4     0.408   \n",
       "749            6     162       62.000000        0       0  24.3     0.178   \n",
       "750            4     136       70.000000        0       0  31.2     1.182   \n",
       "751            1     121       78.000000       39      74  39.0     0.261   \n",
       "752            3     108       62.000000       24       0  26.0     0.223   \n",
       "753            0     181       88.000000       44     510  43.3     0.222   \n",
       "754            8     154       78.000000       32       0  32.4     0.443   \n",
       "755            1     128       88.000000       39     110  36.5     1.057   \n",
       "756            7     137       90.000000       41       0  32.0     0.391   \n",
       "757            0     123       72.000000        0       0  36.3     0.258   \n",
       "758            1     106       76.000000        0       0  37.5     0.197   \n",
       "759            6     190       92.000000        0       0  35.5     0.278   \n",
       "760            2      88       58.000000       26      16  28.4     0.766   \n",
       "761            9     170       74.000000       31       0  44.0     0.403   \n",
       "762            9      89       62.000000        0       0  22.5     0.142   \n",
       "763           10     101       76.000000       48     180  32.9     0.171   \n",
       "764            2     122       70.000000       27       0  36.8     0.340   \n",
       "765            5     121       72.000000       23     112  26.2     0.245   \n",
       "766            1     126       60.000000        0       0  30.1     0.349   \n",
       "767            1      93       70.000000       31       0  30.4     0.315   \n",
       "\n",
       "     age  class  \n",
       "0     50      1  \n",
       "1     31      0  \n",
       "2     32      1  \n",
       "3     21      0  \n",
       "4     33      1  \n",
       "5     30      0  \n",
       "6     26      1  \n",
       "7     29      0  \n",
       "8     53      1  \n",
       "10    30      0  \n",
       "11    34      1  \n",
       "12    57      0  \n",
       "13    59      1  \n",
       "14    51      1  \n",
       "15    32      1  \n",
       "16    31      1  \n",
       "17    31      1  \n",
       "18    33      0  \n",
       "19    32      1  \n",
       "20    27      0  \n",
       "21    50      0  \n",
       "22    41      1  \n",
       "23    29      1  \n",
       "24    51      1  \n",
       "25    41      1  \n",
       "26    43      1  \n",
       "27    22      0  \n",
       "28    57      0  \n",
       "29    38      0  \n",
       "30    60      0  \n",
       "..   ...    ...  \n",
       "738   21      0  \n",
       "739   42      1  \n",
       "740   48      1  \n",
       "741   26      0  \n",
       "742   22      0  \n",
       "743   45      1  \n",
       "744   39      0  \n",
       "745   46      0  \n",
       "746   27      1  \n",
       "747   32      0  \n",
       "748   36      1  \n",
       "749   50      1  \n",
       "750   22      1  \n",
       "751   28      0  \n",
       "752   25      0  \n",
       "753   26      1  \n",
       "754   45      1  \n",
       "755   37      1  \n",
       "756   39      0  \n",
       "757   52      1  \n",
       "758   26      0  \n",
       "759   66      1  \n",
       "760   22      0  \n",
       "761   43      1  \n",
       "762   33      0  \n",
       "763   63      0  \n",
       "764   27      0  \n",
       "765   30      0  \n",
       "766   47      1  \n",
       "767   23      0  \n",
       "\n",
       "[757 rows x 9 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputed = data.dropna(axis=1)\n",
    "imputed = imputed.loc[imputed['BMI'] != 0]\n",
    "imputed[\"blood_pressure\"] = imputed[\"blood_pressure\"].mask(imputed[\"blood_pressure\"] == 0).fillna(imputed[\"blood_pressure\"].mean())\n",
    "imputed.to_csv(\"data\\PA4EX2.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}