02_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding the data first\n",
    "\n",
    "In this section we show how it's important to understand how our data is structured and what can we get from it before we even start dealing with machine learning. The next part is on data sets preparation and data processing. The last part is about quality metrics as it's important to understand the output of our models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Static input data analysis\n",
    "\n",
    "In this example we go through the Boston housing data set that is a well-known set of house prices in Boston. It has the following features:\n",
    "\n",
    "* **crim**  per capita crime rate by town.\n",
    "* **zn** proportion of residential land zoned for lots over 25,000 sq.ft.\n",
    "* **indus** proportion of non-retail business acres per town.\n",
    "* **chas** Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).\n",
    "* **nox** nitrogen oxides concentration (parts per 10 million).\n",
    "* **rm** average number of rooms per dwelling.\n",
    "* **age** proportion of owner-occupied units built prior to 1940.\n",
    "* **dis** weighted mean of distances to five Boston employment centres.\n",
    "* **rad** index of accessibility to radial highways.\n",
    "* **tax** full-value property-tax rate per \\$10000\n",
    "* **ptratio** pupil-teacher ratio by town.\n",
    "* **black** 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town.\n",
    "* **lstat** lower status of the population (percent).\n",
    "* **medv** median value of owner-occupied homes in \\$1000s.\n",
    "\n",
    "We can load the data with Pandas to check what's in the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
       "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
       "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
       "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
       "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
       "\n",
       "   PTRATIO       B  LSTAT  \n",
       "0     15.3  396.90   4.98  \n",
       "1     17.8  396.90   9.14  \n",
       "2     17.8  392.83   4.03  \n",
       "3     18.7  394.63   2.94  \n",
       "4     18.7  396.90   5.33  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import load_boston\n",
    "\n",
    "# Loading the dataset with pandas\n",
    "\n",
    "boston_data = load_boston(return_X_y=False)\n",
    "\n",
    "boston_housing_df = pd.DataFrame(boston_data.data,columns=boston_data.feature_names)\n",
    "#boston_housing_df.drop(\"ID\", axis=1, inplace=True)\n",
    "boston_housing_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step that you should do before going into any kind of preprocessing or training, is to check the profile of your data. Take a look how to profile your data with Pandas below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "76a6cc0d50de499d904ac6d7662a7aeb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Tab(children=(HTML(value='<div id=\"overview-content\" class=\"row variable spacing\">\\n    <div class=\"row\">\\n   …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Report generated with <a href=\"https://github.com/pandas-profiling/pandas-profiling\">pandas-profiling</a>."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas_profiling as pp\n",
    "\n",
    "pp.ProfileReport(boston_housing_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset preparation\n",
    "\n",
    "Quality measure are more about the input and output of the model. We can take the dataset and depending on the way how we divided it, we can measure the quality of our model. One of the common problem that each data scientist has is about how to divide the data set into training and testing data sets. To understand the following equations we need to introduce new designations. Let $L_{n}$ be our training data set of size $n$, $T_{m}$ our testing data set of size $m$, $M_{e}$ the number of misclassified cases, $I$ a function that return 1 if there is a match between predicted and label value and $e(d)$ the error rate of classifier $d$. We use also $X$ and $Y$ sets that we have already explained. We can write the error rate like following:\n",
    "\\begin{equation}\n",
    "e(d)=\\frac{M_{e}}{m}.\n",
    "\\end{equation}\n",
    "It is the opposite to accuracy that is described later in this section. Error rate can be calculated differently depending on which method of data set preparation is used. There are few commonly used approached of how we can handle the training, testing and validation data sets:\n",
    "\n",
    "- resubstitution -- R-method,\n",
    "- hold-out -- H-method,\n",
    "- cross-validation -- $\\pi$-method,\n",
    "- bootstrap,\n",
    "- jackknife.\n",
    "\n",
    "The first method is a very simple one. We have the same data set for training and testing. It is not the best solution if we consider to have a solid classifier. The error rate can be written as following:\n",
    "\\begin{equation}\n",
    "e_{R}(d)=\\frac{1}{n}\\sum_{j=1}^{n}I(d(X_{j};L_{n})\\neq Y_{j}).\n",
    "\\end{equation}\n",
    "It means that we calculate the error rate for each element $j$ of our training data set and add 1 for each well predicted case. We need to divide it with $n$ which is the number of elements in the training data set. \n",
    "\n",
    "The second method is about dividing a data set into two data sets. It can be divided by half or other proportions. One set is our training data set and the second training data set. We can swap those sets and calculate the average of both sets. The error rate can be calculated as following:\n",
    "\\begin{equation}\n",
    "e_{\\tau}(\\hat{d})=\\frac{1}{m}\\sum_{j=1}^{m}I(\\hat{d}(X_{j}^{t};L_{n}\\neq Y_{j}^{t}).\n",
    "\\end{equation}\n",
    "Compared to resubstitution method it uses the testing data set only.\n",
    "Cross-validation is the most common approach. It can be also called as rotation method. We need to divide the data set into $k$ subsets. The elements in each set are randomly chosen. One of those sets are taken as a testing set where the other sets are merged into a  training set. It should be repeated $k$ times for each $k$ subset. The error rate can be calculated like following:\n",
    "\\begin{equation}\n",
    "e_{CV}(d)=\\frac{1}{n}\\sum_{j=1}^{n}I(\\hat{d}(X_{j};L_{n}^{(-j)}\\neq Y_{j}).\n",
    "\\end{equation}\n",
    "A special case is when $k=m$. It means that we have subsets where each consist of just one element. This approach is known as leave-one-out or U-method.\n",
    "\n",
    "Bootstrap method can be considered as an extension of resubstitution. The goal is to generate multiple sets from the main set by randomly selection. We use resubstitution method on each set and calculate an average error at the end:\n",
    "\\begin{equation}\n",
    "e_{B}(d)=\\frac{1}{B}\\sum_{b=1}^{B}\\frac{\\sum_{j=1}^{n}I(Z_{j}\\notin L_{n}^{\\star b})I(d(X_{j};L^{\\star b}_{n})\\not Y_{j})}{\\sum_{j=1}^{n}(Z_{j}\\notin L^{\\star b}_{n})}.\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preprocessing\n",
    "\n",
    "In this section we go through another popular data set - the Titanic data set. You can grab it from [https://www.kaggle.com/c/titanic/data](https://www.kaggle.com/c/titanic/data). The data consists of several features and the binary class of survival. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Pclass                                               Name  \\\n",
       "0         0       3                            Braund, Mr. Owen Harris   \n",
       "1         1       1  Cumings, Mrs. John Bradley (Florence Briggs Th...   \n",
       "2         1       3                             Heikkinen, Miss. Laina   \n",
       "3         1       1       Futrelle, Mrs. Jacques Heath (Lily May Peel)   \n",
       "4         0       3                           Allen, Mr. William Henry   \n",
       "\n",
       "      Sex   Age  SibSp  Parch            Ticket     Fare Cabin Embarked  \n",
       "0    male  22.0      1      0         A/5 21171   7.2500   NaN        S  \n",
       "1  female  38.0      1      0          PC 17599  71.2833   C85        C  \n",
       "2  female  26.0      0      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3  female  35.0      1      0            113803  53.1000  C123        S  \n",
       "4    male  35.0      0      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read the train dataset of Titanic dataset\n",
    "titanic_df = pd.read_csv(\"./datasets/titanic-train.csv\")\n",
    "titanic_df.drop(\"PassengerId\", axis=1, inplace=True)\n",
    "titanic_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some values are not avaialble (NaN), we cannot use such data to build a model. We need clean it up and drop three features that seems not to give any information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "FEATURES_TO_DROP = [\"Name\", \"Ticket\", \"Cabin\"]\n",
    "\n",
    "# Fill missing values\n",
    "titanic_df.fillna(titanic_df.mean(), inplace=True)\n",
    "\n",
    "# Transform the dataset and perform one-hot encoding\n",
    "titanic_dummies_df = pd.get_dummies(titanic_df.drop(FEATURES_TO_DROP, axis=1))\n",
    "\n",
    "# Divide the dataset to have train and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(titanic_dummies_df.drop(\"Survived\",                                                                             axis=1),\n",
    "                                                    titanic_df[\"Survived\"],\n",
    "                                                    test_size=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we will create a basic logistic regression classifier, that will be a benchmark for the further analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7871148459383753"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Create a logistic regression model\n",
    "lr_classifier = LogisticRegression()\n",
    "lr_classifier.fit(scaler.fit_transform(X_train), \n",
    "                  y_train)\n",
    "\n",
    "# Calculate the accuracy score\n",
    "y_pred = lr_classifier.predict(scaler.transform(X_test))\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pclass       -0.883955\n",
       "Age          -0.384756\n",
       "SibSp        -0.265138\n",
       "Parch        -0.200861\n",
       "Fare          0.128376\n",
       "Sex_female    0.678576\n",
       "Sex_male     -0.678576\n",
       "Embarked_C    0.059250\n",
       "Embarked_Q    0.035060\n",
       "Embarked_S   -0.177997\n",
       "dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display feature importance\n",
    "pd.Series(lr_classifier.coef_[0], \n",
    "          index=X_test.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a next step we are going to include some more properties from the previously dropped textual columns. We will consider the following features:\n",
    "\n",
    "* **Name**\n",
    "* **Ticket**\n",
    "* **Cabin**\n",
    "\n",
    "Let's dive into the variables, for some better understanding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "770               Lievens, Mr. Rene Aime\n",
       "853            Lines, Miss. Mary Conover\n",
       "144           Andrew, Mr. Edgardo Samuel\n",
       "141             Nysten, Miss. Anna Sofia\n",
       "261    Asplund, Master. Edvin Rojj Felix\n",
       "11              Bonnell, Miss. Elizabeth\n",
       "794                Dantcheff, Mr. Ristiu\n",
       "390           Carter, Mr. William Ernest\n",
       "15      Hewlett, Mrs. (Mary D Kingcome) \n",
       "675       Edvardsson, Mr. Gustaf Hjalmar\n",
       "Name: Name, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display some names\n",
    "titanic_df[\"Name\"].sample(10, random_state=43)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some words seem to occur quite often. We can analyze the most frequent ones to see if they may enrich the data we already collected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                         [braund,, mr., owen, harris]\n",
       "1    [cumings,, mrs., john, bradley, (florence, bri...\n",
       "2                           [heikkinen,, miss., laina]\n",
       "3    [futrelle,, mrs., jacques, heath, (lily, may, ...\n",
       "4                        [allen,, mr., william, henry]\n",
       "5                                 [moran,, mr., james]\n",
       "6                         [mccarthy,, mr., timothy, j]\n",
       "7                  [palsson,, master., gosta, leonard]\n",
       "8    [johnson,, mrs., oscar, w, (elisabeth, vilhelm...\n",
       "9            [nasser,, mrs., nicholas, (adele, achem)]\n",
       "Name: Name, dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split the names into pieces\n",
    "titanic_df[\"Name\"].str.lower()\\\n",
    "    .str.split()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>braund,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mr.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>owen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>harris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cumings,</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       word\n",
       "0   braund,\n",
       "1       mr.\n",
       "2      owen\n",
       "3    harris\n",
       "4  cumings,"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a dataframe with a word as a single column\n",
    "words_df = titanic_df[\"Name\"].str.lower()\\\n",
    "    .str.split()\\\n",
    "    .apply(pd.Series)\\\n",
    "    .stack()\\\n",
    "    .reset_index(drop=True)\\\n",
    "    .to_frame(name=\"word\")\n",
    "words_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "word\n",
       "mr.        517\n",
       "miss.      182\n",
       "mrs.       125\n",
       "william     62\n",
       "john        44\n",
       "master.     40\n",
       "henry       33\n",
       "james       24\n",
       "charles     23\n",
       "george      22\n",
       "Name: word, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display most frequent words descending\n",
    "words_df.groupby(\"word\")[\"word\"].count()\\\n",
    "    .sort_values(ascending=False).head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "      <th>Name_mr.</th>\n",
       "      <th>Name_miss.</th>\n",
       "      <th>Name_mrs.</th>\n",
       "      <th>Name_master.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Pclass   Age  SibSp  Parch     Fare  Sex_female  Sex_male  \\\n",
       "0         0       3  22.0      1      0   7.2500           0         1   \n",
       "1         1       1  38.0      1      0  71.2833           1         0   \n",
       "2         1       3  26.0      0      0   7.9250           1         0   \n",
       "3         1       1  35.0      1      0  53.1000           1         0   \n",
       "4         0       3  35.0      0      0   8.0500           0         1   \n",
       "\n",
       "   Embarked_C  Embarked_Q  Embarked_S  Name_mr.  Name_miss.  Name_mrs.  \\\n",
       "0           0           0           1         1           0          0   \n",
       "1           1           0           0         0           0          1   \n",
       "2           0           0           1         0           1          0   \n",
       "3           0           0           1         0           0          1   \n",
       "4           0           0           1         1           0          0   \n",
       "\n",
       "   Name_master.  \n",
       "0             0  \n",
       "1             0  \n",
       "2             0  \n",
       "3             0  \n",
       "4             0  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SPECIAL_WORDS = [\"mr.\", \"miss.\", \"mrs.\", \"master.\"]\n",
    "\n",
    "# Define new columns by the presence of special words\n",
    "for special_word in SPECIAL_WORDS:\n",
    "    titanic_dummies_df[\"Name_%s\" % special_word] = titanic_df[\"Name\"].str.contains(\n",
    "        special_word, case=False, regex=False).astype(int)\n",
    "    \n",
    "# Display some first rows\n",
    "titanic_dummies_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the **Ticket** column we can also try to extract some more properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "869               347742\n",
       "685        SC/Paris 2123\n",
       "163               315093\n",
       "374               349909\n",
       "487                11771\n",
       "260               384461\n",
       "495                 2627\n",
       "400    STON/O 2. 3101289\n",
       "610               347082\n",
       "264               382649\n",
       "Name: Ticket, dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show 10 randomly selected rows\n",
    "titanic_df[\"Ticket\"].sample(10, random_state=214)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          [A]\n",
       "1         [PC]\n",
       "2    [STON, O]\n",
       "3           []\n",
       "4           []\n",
       "5           []\n",
       "6           []\n",
       "7           []\n",
       "8           []\n",
       "9           []\n",
       "Name: Ticket, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract all the names from the \n",
    "titanic_df[\"Ticket\"].str.findall(r\"[A-Za-z]+\")\\\n",
    "    .head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>PC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>STON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PP</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   word\n",
       "0     A\n",
       "1    PC\n",
       "2  STON\n",
       "3     O\n",
       "4    PP"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a dataframe with a word as a single column\n",
    "words_df = titanic_df[\"Ticket\"].str.findall(r\"[A-Za-z]+\")\\\n",
    "    .apply(pd.Series)\\\n",
    "    .stack()\\\n",
    "    .reset_index(drop=True)\\\n",
    "    .to_frame(name=\"word\")\n",
    "words_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>...</th>\n",
       "      <th>Ticket_Q</th>\n",
       "      <th>Ticket_PARIS</th>\n",
       "      <th>Ticket_Fa</th>\n",
       "      <th>Ticket_LINE</th>\n",
       "      <th>Ticket_F</th>\n",
       "      <th>Ticket_SW</th>\n",
       "      <th>Ticket_SCO</th>\n",
       "      <th>Ticket_AH</th>\n",
       "      <th>Ticket_Basle</th>\n",
       "      <th>Ticket_WE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Pclass   Age  SibSp  Parch     Fare  Sex_female  Sex_male  \\\n",
       "0         0       3  22.0      1      0   7.2500           0         1   \n",
       "1         1       1  38.0      1      0  71.2833           1         0   \n",
       "2         1       3  26.0      0      0   7.9250           1         0   \n",
       "3         1       1  35.0      1      0  53.1000           1         0   \n",
       "4         0       3  35.0      0      0   8.0500           0         1   \n",
       "\n",
       "   Embarked_C  Embarked_Q    ...      Ticket_Q  Ticket_PARIS  Ticket_Fa  \\\n",
       "0           0           0    ...             0             0          0   \n",
       "1           1           0    ...             0             0          0   \n",
       "2           0           0    ...             0             0          0   \n",
       "3           0           0    ...             0             0          0   \n",
       "4           0           0    ...             0             0          0   \n",
       "\n",
       "   Ticket_LINE  Ticket_F  Ticket_SW  Ticket_SCO  Ticket_AH  Ticket_Basle  \\\n",
       "0            0         0          0           0          0             0   \n",
       "1            0         0          0           0          0             0   \n",
       "2            0         0          0           0          0             0   \n",
       "3            0         0          0           0          0             0   \n",
       "4            0         0          0           0          0             0   \n",
       "\n",
       "   Ticket_WE  \n",
       "0          0  \n",
       "1          0  \n",
       "2          0  \n",
       "3          0  \n",
       "4          0  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert the presence of the phrases to features\n",
    "for ticket_word in words_df[\"word\"].unique():\n",
    "    titanic_dummies_df[\"Ticket_%s\" % ticket_word] = titanic_df[\"Ticket\"].str.contains(\n",
    "        ticket_word, case=False, regex=False).astype(int)\n",
    "    \n",
    "# Display some first rows\n",
    "titanic_dummies_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the last column is *Cabin* where we could expect some similar process to be taken."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "867                A24\n",
       "248                D35\n",
       "31                 B78\n",
       "61                 B28\n",
       "311    B57 B59 B63 B66\n",
       "572                E25\n",
       "170                B19\n",
       "128              F E69\n",
       "765                D11\n",
       "434                E44\n",
       "698                C68\n",
       "62                 C83\n",
       "195                B80\n",
       "366                D37\n",
       "88         C23 C25 C27\n",
       "329                B18\n",
       "393                D36\n",
       "835                E49\n",
       "297            C22 C26\n",
       "849                C92\n",
       "Name: Cabin, dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filter out NaNs and show 10 randomly selected samples\n",
    "not_null_mask = titanic_df[\"Cabin\"].notna()\n",
    "titanic_df[not_null_mask][\"Cabin\"].sample(20, random_state=134)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example of *Cabin* the letter may indicate closeness of different rooms. We may keep just this information - it won't tell us anything about an exact location of the cabins, but just a rough estimate of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['C', 'E', 'G', 'D', 'A', 'B', 'F', 'T'], dtype=object)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get all the letters that occur in the cabin number\n",
    "letters_df = titanic_df[not_null_mask][\"Cabin\"].str.findall(r\"[A-Za-z]\")\\\n",
    "    .apply(pd.Series)\\\n",
    "    .stack()\\\n",
    "    .reset_index(drop=True)\\\n",
    "    .to_frame(name=\"letter\")\n",
    "# Display unique values\n",
    "letters_df[\"letter\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>...</th>\n",
       "      <th>Ticket_Basle</th>\n",
       "      <th>Ticket_WE</th>\n",
       "      <th>Cabin_C</th>\n",
       "      <th>Cabin_E</th>\n",
       "      <th>Cabin_G</th>\n",
       "      <th>Cabin_D</th>\n",
       "      <th>Cabin_A</th>\n",
       "      <th>Cabin_B</th>\n",
       "      <th>Cabin_F</th>\n",
       "      <th>Cabin_T</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 49 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Pclass   Age  SibSp  Parch     Fare  Sex_female  Sex_male  \\\n",
       "0         0       3  22.0      1      0   7.2500           0         1   \n",
       "1         1       1  38.0      1      0  71.2833           1         0   \n",
       "2         1       3  26.0      0      0   7.9250           1         0   \n",
       "3         1       1  35.0      1      0  53.1000           1         0   \n",
       "4         0       3  35.0      0      0   8.0500           0         1   \n",
       "\n",
       "   Embarked_C  Embarked_Q   ...     Ticket_Basle  Ticket_WE  Cabin_C  Cabin_E  \\\n",
       "0           0           0   ...                0          0        0        0   \n",
       "1           1           0   ...                0          0        1        0   \n",
       "2           0           0   ...                0          0        0        0   \n",
       "3           0           0   ...                0          0        1        0   \n",
       "4           0           0   ...                0          0        0        0   \n",
       "\n",
       "   Cabin_G  Cabin_D  Cabin_A  Cabin_B  Cabin_F  Cabin_T  \n",
       "0        0        0        0        0        0        0  \n",
       "1        0        0        0        0        0        0  \n",
       "2        0        0        0        0        0        0  \n",
       "3        0        0        0        0        0        0  \n",
       "4        0        0        0        0        0        0  \n",
       "\n",
       "[5 rows x 49 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert the presence of the letter to features\n",
    "for letter in letters_df[\"letter\"].unique():\n",
    "    titanic_dummies_df[\"Cabin_%s\" % letter] = titanic_df[\"Cabin\"].str.contains(\n",
    "        letter, case=False, regex=False, na=False).astype(int)\n",
    "    \n",
    "# Display some first rows\n",
    "titanic_dummies_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression with engineered features\n",
    "\n",
    "We have created another model, which includes some more features than the original one. We can directly include them for the model training, so we can then compare the achieved accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace the datasets with their extensions\n",
    "X_train = titanic_dummies_df.iloc[X_train.index].drop(\"Survived\", \n",
    "                                                      axis=1)\n",
    "X_test = titanic_dummies_df.iloc[X_test.index].drop(\"Survived\", \n",
    "                                                    axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.803921568627451"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler = StandardScaler()\n",
    "\n",
    "# Create a logistic regression model\n",
    "lr_classifier = LogisticRegression()\n",
    "lr_classifier.fit(scaler.fit_transform(X_train), \n",
    "                  y_train)\n",
    "\n",
    "# Calculate the accuracy score\n",
    "y_pred = lr_classifier.predict(scaler.transform(X_test))\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pclass         -0.538172\n",
       "Age            -0.238012\n",
       "SibSp          -0.604627\n",
       "Parch          -0.427661\n",
       "Fare            0.328578\n",
       "Sex_female      0.542784\n",
       "Sex_male       -0.542784\n",
       "Embarked_C      0.152972\n",
       "Embarked_Q      0.001463\n",
       "Embarked_S     -0.204464\n",
       "Name_mr.       -0.698937\n",
       "Name_miss.     -0.156069\n",
       "Name_mrs.       0.189722\n",
       "Name_master.    0.490672\n",
       "Ticket_A       -0.482318\n",
       "Ticket_PC      -0.171329\n",
       "Ticket_STON     0.866803\n",
       "Ticket_O       -0.559273\n",
       "Ticket_PP       0.847821\n",
       "Ticket_C        0.753048\n",
       "Ticket_SC       0.267054\n",
       "Ticket_Paris    0.051459\n",
       "Ticket_S       -0.314472\n",
       "Ticket_CA      -0.115545\n",
       "Ticket_P       -0.537001\n",
       "Ticket_SO       0.391967\n",
       "Ticket_W       -0.419061\n",
       "Ticket_SOTON   -0.096758\n",
       "Ticket_OQ      -0.324310\n",
       "Ticket_E       -0.085299\n",
       "Ticket_Q        0.308176\n",
       "Ticket_PARIS    0.051459\n",
       "Ticket_Fa      -0.107655\n",
       "Ticket_LINE    -0.114152\n",
       "Ticket_F        0.082929\n",
       "Ticket_SW       0.086848\n",
       "Ticket_SCO     -0.059066\n",
       "Ticket_AH       0.126617\n",
       "Ticket_Basle    0.013820\n",
       "Ticket_WE       0.000000\n",
       "Cabin_C         0.079679\n",
       "Cabin_E         0.467601\n",
       "Cabin_G        -0.373012\n",
       "Cabin_D         0.343571\n",
       "Cabin_A         0.054824\n",
       "Cabin_B         0.151409\n",
       "Cabin_F         0.127218\n",
       "Cabin_T         0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display feature importance\n",
    "pd.Series(lr_classifier.coef_[0], \n",
    "          index=X_test.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model performance\n",
    "\n",
    "The model performance can be measure in different ways. On of such measure is the loss function measurement. We have different methods to do it.\n",
    "\n",
    "### Loss function\n",
    "\n",
    "It is a method of evaluating how well your algorithm models your dataset. If your predictions are totally off, your loss function will output a higher number. If they’re pretty good, it’ll output a lower one.\n",
    "\n",
    "Examples:\n",
    "\n",
    "**Mean Squared Error** (MSE) - the workhorse of basic loss functions: it’s easy to understand and implement, and generally works pretty well. To calculate MSE, you take the difference between your predictions and the ground truth, square it, and average it out across the whole dataset. Often used in regression.\n",
    "\n",
    "$MSE = \\frac{1}{n} \\sum_{i=1}^{n} (y_{i}^{true} - y_{i}^{pred})^2$\n",
    "\n",
    "**Cross Entropy** (log loss) - measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label.\n",
    "\n",
    "$H = -\\frac{1}{n} \\sum_{i=1}^{n} [y_{i}^{true} \\log(y_{i}^{pred}) + (1 - y_{i}^{true}) \\log(1 - y_{i}^{pred})]$\n",
    "\n",
    "### Gradient based optimization\n",
    "\n",
    "Gradient descent is a first-order iterative optimization algorithm for finding the minimum of a function. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or approximate gradient) of the function at the current point. If, instead, one takes steps proportional to the positive of the gradient, one approaches a local maximum of that function; the procedure is then known as gradient ascent.\n",
    "\n",
    "![Minimum and maximum of the function](images/minmax.png)\n",
    "\n",
    "Vanilla gradient descent (batch gradient descent) computes the gradient of the cost function with respect to the parameters for the entire training dataset:\n",
    "\n",
    "$\\theta = \\theta - \\eta \\cdot \\nabla_\\theta J(\\theta)$\n",
    "\n",
    "where:\n",
    "\n",
    "- $\\theta$ - parameters\n",
    "- $\\eta$ - learning rate\n",
    "- $J$ - cost function\n",
    "\n",
    "\n",
    "**Stochastic Gradient Descent** (SGD)\n",
    "\n",
    "Stochastic gradient descent in contrast performs a parameter update for each training example:\n",
    "\n",
    "$\\theta = \\theta - \\eta \\cdot \\nabla_\\theta J( \\theta, x_i, y_i)$\n",
    "\n",
    "where:\n",
    "\n",
    "- $x_i$ - example\n",
    "- $y_i$ - label\n",
    "\n",
    "Batch gradient descent performs redundant computations for large datasets, as it recomputes gradients for similar examples before each parameter update. SGD does away with this redundancy by performing one update at a time. It is therefore usually much faster and can also be used to learn online.\n",
    "\n",
    "**Mini-batch gradient descent**\n",
    "\n",
    "Mini-batch gradient descent finally takes the best of both worlds and performs an update for every mini-batch of N training examples:\n",
    "\n",
    "$\\theta = \\theta - \\eta \\cdot \\nabla_\\theta J( \\theta, x_{(i:i + N)}, y_{(i:i + N)})$\n",
    "\n",
    "\n",
    "Mini-batch approach reduces the variance of the parameter updates (more stable convergence) and make use of highly optimized matrix operations. It is typically the algorithm of choice when training a neural network and the term **SGD** usually is employed also when mini-batches are used.\n",
    "\n",
    "**SGD optimizations**\n",
    "\n",
    "There are many modifications to standard SGD method that improve robustness, reduce oscillation and gain faster convergence.\n",
    "\n",
    "**Other**\n",
    "\n",
    "There are plenty of optimizers like **Momentum**, **Nesterov Accelerated Gradient** (NAG), **Adagrad**, **Adadelta**, **RMSprop**, or **Adam**. Take a look at a comparison.\n",
    "\n",
    "Gradient based method visualization |\n",
    "\n",
    "![SGD variants summary](images/gradsummary.gif) \n",
    "![SGD variants summary](images/gradsaddlesummary.gif)\n",
    "\n",
    "Let's take a look how to use SGD optimizer with Keras. It can be done in a similar way in Tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras.optimizers import SGD\n",
    "from keras.utils.np_utils import to_categorical\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "features, labels = load_iris(return_X_y=True)\n",
    "features = normalize(features)\n",
    "labels = to_categorical(labels)\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size=0.15, shuffle=True)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(units=10, activation='relu', input_dim=x_train.shape[1]))\n",
    "model.add(Dense(units=7, activation='relu'))\n",
    "model.add(Dense(units=3, activation='softmax'))\n",
    "\n",
    "optimizer = SGD(lr=0.05, momentum=0.7)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=150, batch_size=16, verbose=0)\n",
    "loss, accuracy = model.evaluate(x_test, y_test)\n",
    "\n",
    "print('accuracy on test set: {accuracy * 100}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can apply different optimizers or loss functions to the same or different classification problems. It does not depends on the neural network architecture. A more complex example using MNIST dataset is shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Conv2D, Dense, Dropout, Flatten, MaxPool2D\n",
    "from keras.optimizers import Adam\n",
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "\n",
    "FEATURES_SHAPE = (-1, 28, 28, 1)\n",
    "MAX_FEATURE = 255\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train = (x_train / MAX_FEATURE).astype(np.float16).reshape(FEATURES_SHAPE)\n",
    "y_train = to_categorical(y_train)\n",
    "x_test = (x_test / MAX_FEATURE).astype(np.float16).reshape(FEATURES_SHAPE)\n",
    "y_test = to_categorical(y_test)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(filters=6, kernel_size=5, activation='relu', input_shape=FEATURES_SHAPE[1:]))\n",
    "model.add(MaxPool2D())\n",
    "model.add(Conv2D(filters=16, kernel_size=3, activation='relu'))\n",
    "model.add(MaxPool2D())\n",
    "model.add(Flatten())\n",
    "model.add(Dense(units=120, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(units=84, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(units=10, activation='softmax'))\n",
    "\n",
    "optimizer = Adam(lr=0.005)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=10, batch_size=128, verbose=1)\n",
    "loss, accuracy = model.evaluate(x_test, y_test)\n",
    "\n",
    "print('accuracy on test set: {accuracy * 100}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output quality metrics\n",
    "\n",
    "There are several metrics to show the quality of our classification model:\n",
    "\n",
    "- ROC that stands for Receiver Operating Characteristic curve,\n",
    "- AUC -- Area Under Curve,\n",
    "- $F_{1}$ score,\n",
    "- Precision,\n",
    "- Recall.\n",
    "\n",
    "To calculate the metrics we ned \n",
    "\n",
    "|                      |condition positive |condition negative |\n",
    "|----------------------|-------------------|-------------------|\n",
    "|**predicted positive**|True Positive (TP) |False Positive (FP)|         \n",
    "|**predicted negative**|False Negative (FN)|True Negative (TN) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most common metric is the accuracy. It can be calculated like following:\n",
    "\\begin{equation}\n",
    "ACC=\\frac{\\#TP+\\#TN}{\\#TP+\\#TN+\\#FP+\\#FN}.\n",
    "\\end{equation}\n",
    "First one that we describe is called True Positive Rate (TPR). It can be calculated like following:\n",
    "\\begin{equation}\n",
    "TPR=\\frac{\\#TP}{\\#TP+\\#FN}.\n",
    "\\end{equation}\n",
    "TPR is also called sensitivity or recall and is a measure of good predictions within a set of cases. By $\\#TP, \\#FP$ we mean the number of True Positive and False Positive cases. An opposite to it is specificity. It is also called TNR what stands for True Negative Rate. It can be calculated as following:\n",
    "\\begin{equation}\n",
    "TNR=\\frac{\\#TN}{\\#TN+\\#FP}.\n",
    "\\end{equation}\n",
    "It is a measure that says how good we are at predicting negative scenario. Another important metric is precision that is also known as Positive Predictive Value (PPV):\n",
    "\\begin{equation}\n",
    "PPV=\\frac{\\#TP}{\\#TP+\\#FP}.\n",
    "\\end{equation}\n",
    "It is a ratio of positive cases that that were well predicted to all positive cases, even those that are not well predicted. The opposite to it is the Negative Predictive Value:\n",
    "\\begin{equation}\n",
    "NPV=\\frac{TN}{TN+FN}.\n",
    "\\end{equation}\n",
    "We can also calculate the False Positive Rate metric known as fall-out. It is about how bad we are on predicting positive cases:\n",
    "\\begin{equation}\n",
    "FPR=1-TNR.\n",
    "\\end{equation}\n",
    "The opposite to FPR is False Negative Rate:\n",
    "\\begin{equation}\n",
    "FNR=1-TPR.\n",
    "\\end{equation}\n",
    "Another popular metric is called $F_{1}$ score and it is a weighted accuracy measure. It takes PPV and TPR to calculate the score:\n",
    "\\begin{equation}\n",
    "F_{1}=2\\frac{PPV\\cdot TPR}{TPR+PPV}.\n",
    "\\end{equation}\n",
    "The $F_{1}$ value as in case of all previous metrics between 1 and 0, where 1 is the best. \n",
    "A interesting measure is Matthews Correlation Coefficient measure that is about the correlation between observed and predicted values. The value of MCC is between -1 and 1. If we have a perfect classifier we get MCC=1. A random classifier is when we have MCC=0 and a totally bad classifier if have MCC=-1. This measure can be calculated as following:\n",
    "\\begin{equation}\n",
    "MCC=\\frac{\\#TP\\cdot\\#TN-\\#FP\\cdot\\#FN}{\\sqrt{(\\#TP+\\#FP)(\\#TP+\\#FN)(\\#TN+\\#FP)(\\#TN+\\#FN)}}.\n",
    "\\end{equation}\n",
    "\n",
    "Let's generate a test and train dataset to measure the metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets.samples_generator import make_blobs\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# build the train dataset\n",
    "X, y = make_blobs(n_samples=100000, centers=2, n_features=2, random_state=1)\n",
    "scalar = MinMaxScaler()\n",
    "scalar.fit(X)\n",
    "X = scalar.transform(X)\n",
    "\n",
    "# build the test dataset\n",
    "Xtest, ytest = make_blobs(n_samples=5000, centers=2, n_features=2, random_state=1)\n",
    "Xtest = scalar.transform(Xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "\n",
    "# build a simple neural network with thre layers\n",
    "model = Sequential()\n",
    "model.add(Dense(4, input_dim=2, activation='relu'))\n",
    "model.add(Dense(4, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# compile it with adam optimizer\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam')\n",
    "model.fit(X, y, epochs=10, verbose=0)\n",
    "\n",
    "ypred = model.predict_classes(Xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(ypred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_quality_metrics(y,ypredicted):\n",
    "    tn = 0\n",
    "    tp = 0\n",
    "    fn = 0\n",
    "    fp = 0\n",
    "    for i in range(len(y)):\n",
    "        if y[i] > 0:\n",
    "            if y[i] == ypredicted[i]:\n",
    "                tp = tp + 1\n",
    "            else:\n",
    "                fp = fp + 1\n",
    "        else:\n",
    "            if y[i] == ypredicted[i]:\n",
    "                tn = tn + 1\n",
    "            else:\n",
    "                fn = fn + 1\n",
    "    acc = ((tp + tn) * 1.0) / ((tp + tn + fp + fn) * 1.0)\n",
    "    tpr = tp * 1.0 / (tp + fn) * 1.0\n",
    "    tnr = tn * 1.0 / (tn + fp) * 1.0\n",
    "    ppv = tp / (tp + fp) * 1.0\n",
    "    npv = tn / (tn + fn) * 1.0\n",
    "    fpr = 1.0 - tnr\n",
    "    fnr = 1.0 - tpr\n",
    "    f1 = 2 * (ppv * tpr * 1.0 / (tpr + ppv * 1.0))\n",
    "    mcc = (tp * tn - fp * fn) / (math.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)) * 1.0)    print(\"Accuracy: \"+str(acc))\n",
    "    print(\"TPR: \"+str(tpr))\n",
    "    print(\"TNR: \"+str(tnr))\n",
    "    print(\"PPC: \"+str(ppv))\n",
    "    print(\"NPV: \"+str(npv))\n",
    "    print(\"FPR: \"+str(fpr))\n",
    "    print(\"FNR: \"+str(fnr))\n",
    "    print(\"MCC: \"+str(mcc))\n",
    "    print(\"F1: \"+str(f1))    \n",
    "    return [acc, tpr, tnr, ppv, npv, fpr, fnr, mcc, f1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "calculate_quality_metrics(ytest,ypred)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}