Hamoye Stage C - Amit Purswani.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hamoye Stage C Tag-along Project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset Description\n",
    "Stability of the Grid System\n",
    "Electrical grids require a balance between electricity supply and demand in order to be stable. Conventional systems achieve this balance through demand-driven electricity production. For future grids with a high share of inflexible (i.e., renewable) energy source, the concept of demand response is a promising solution. This implies changes in electricity consumption in relation to electricity price changes. In this work, we’ll build a binary classification model to predict if a grid is stable or unstable using the UCI Electrical Grid Stability Simulated dataset.\n",
    "\n",
    "Dataset: https://archive.ics.uci.edu/ml/datasets/Electrical+Grid+Stability+Simulated+Data+\n",
    "\n",
    "It has 12 primary predictive features and two dependent variables.\n",
    "\n",
    "Predictive features:\n",
    "\n",
    "'tau1' to 'tau4': the reaction time of each network participant, a real value within the range 0.5 to 10 ('tau1' corresponds to the supplier node, 'tau2' to 'tau4' to the consumer nodes);\n",
    "'p1' to 'p4': nominal power produced (positive) or consumed (negative) by each network participant, a real value within the range -2.0 to -0.5 for consumers ('p2' to 'p4'). As the total power consumed equals the total power generated, p1 (supplier node) = - (p2 + p3 + p4);\n",
    "'g1' to 'g4': price elasticity coefficient for each network participant, a real value within the range 0.05 to 1.00 ('g1' corresponds to the supplier node, 'g2' to 'g4' to the consumer nodes; 'g' stands for 'gamma');\n",
    "Dependent variables:\n",
    "\n",
    "'stab': the maximum real part of the characteristic differential equation root (if positive, the system is linearly unstable; if negative, linearly stable);\n",
    "'stabf': a categorical (binary) label ('stable' or 'unstable').\n",
    "Because of the direct relationship between 'stab' and 'stabf' ('stabf' = 'stable' if 'stab' <= 0, 'unstable' otherwise), 'stab' should be dropped and 'stabf' will remain as the sole dependent variable (binary classification).\n",
    "\n",
    "Split the data into an 80-20 train-test split with a random state of “1”. Use the standard scaler to transform the train set (x_train, y_train) and the test set (x_test). Use scikit learn to train a random forest and extra trees classifier. And use xgboost and lightgbm to train an extreme boosting model and a light gradient boosting model. Use random_state = 1 for training all models and evaluate on the test set.\n",
    "\n",
    "Also, to improve the Extra Trees Classifier, you will use the following parameters (number of estimators, minimum number of samples, minimum number of samples for leaf node and the number of features to consider when looking for the best split) for the hyperparameter grid needed to run a Randomized Cross Validation Search (RandomizedSearchCV).\n",
    "\n",
    "n_estimators = [50, 100, 300, 500, 1000]\n",
    "\n",
    "min_samples_split = [2, 3, 5, 7, 9]\n",
    "\n",
    "min_samples_leaf = [1, 2, 4, 6, 8]\n",
    "\n",
    "max_features = ['auto', 'sqrt', 'log2', None]\n",
    "\n",
    "hyperparameter_grid = {'n_estimators': n_estimators,\n",
    "\n",
    "'min_samples_leaf': min_samples_leaf,\n",
    "\n",
    "'min_samples_split': min_samples_split,\n",
    "\n",
    "'max_features': max_features}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import necessary libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import data set\n",
    "df = pd.read_csv('D:\\People\\Amit\\Hamoye\\stageC\\\\Data_for_UCI_named.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tau1</th>\n",
       "      <th>tau2</th>\n",
       "      <th>tau3</th>\n",
       "      <th>tau4</th>\n",
       "      <th>p1</th>\n",
       "      <th>p2</th>\n",
       "      <th>p3</th>\n",
       "      <th>p4</th>\n",
       "      <th>g1</th>\n",
       "      <th>g2</th>\n",
       "      <th>g3</th>\n",
       "      <th>g4</th>\n",
       "      <th>stab</th>\n",
       "      <th>stabf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.959060</td>\n",
       "      <td>3.079885</td>\n",
       "      <td>8.381025</td>\n",
       "      <td>9.780754</td>\n",
       "      <td>3.763085</td>\n",
       "      <td>-0.782604</td>\n",
       "      <td>-1.257395</td>\n",
       "      <td>-1.723086</td>\n",
       "      <td>0.650456</td>\n",
       "      <td>0.859578</td>\n",
       "      <td>0.887445</td>\n",
       "      <td>0.958034</td>\n",
       "      <td>0.055347</td>\n",
       "      <td>unstable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.304097</td>\n",
       "      <td>4.902524</td>\n",
       "      <td>3.047541</td>\n",
       "      <td>1.369357</td>\n",
       "      <td>5.067812</td>\n",
       "      <td>-1.940058</td>\n",
       "      <td>-1.872742</td>\n",
       "      <td>-1.255012</td>\n",
       "      <td>0.413441</td>\n",
       "      <td>0.862414</td>\n",
       "      <td>0.562139</td>\n",
       "      <td>0.781760</td>\n",
       "      <td>-0.005957</td>\n",
       "      <td>stable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.971707</td>\n",
       "      <td>8.848428</td>\n",
       "      <td>3.046479</td>\n",
       "      <td>1.214518</td>\n",
       "      <td>3.405158</td>\n",
       "      <td>-1.207456</td>\n",
       "      <td>-1.277210</td>\n",
       "      <td>-0.920492</td>\n",
       "      <td>0.163041</td>\n",
       "      <td>0.766689</td>\n",
       "      <td>0.839444</td>\n",
       "      <td>0.109853</td>\n",
       "      <td>0.003471</td>\n",
       "      <td>unstable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.716415</td>\n",
       "      <td>7.669600</td>\n",
       "      <td>4.486641</td>\n",
       "      <td>2.340563</td>\n",
       "      <td>3.963791</td>\n",
       "      <td>-1.027473</td>\n",
       "      <td>-1.938944</td>\n",
       "      <td>-0.997374</td>\n",
       "      <td>0.446209</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>0.929381</td>\n",
       "      <td>0.362718</td>\n",
       "      <td>0.028871</td>\n",
       "      <td>unstable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.134112</td>\n",
       "      <td>7.608772</td>\n",
       "      <td>4.943759</td>\n",
       "      <td>9.857573</td>\n",
       "      <td>3.525811</td>\n",
       "      <td>-1.125531</td>\n",
       "      <td>-1.845975</td>\n",
       "      <td>-0.554305</td>\n",
       "      <td>0.797110</td>\n",
       "      <td>0.455450</td>\n",
       "      <td>0.656947</td>\n",
       "      <td>0.820923</td>\n",
       "      <td>0.049860</td>\n",
       "      <td>unstable</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       tau1      tau2      tau3      tau4        p1        p2        p3  \\\n",
       "0  2.959060  3.079885  8.381025  9.780754  3.763085 -0.782604 -1.257395   \n",
       "1  9.304097  4.902524  3.047541  1.369357  5.067812 -1.940058 -1.872742   \n",
       "2  8.971707  8.848428  3.046479  1.214518  3.405158 -1.207456 -1.277210   \n",
       "3  0.716415  7.669600  4.486641  2.340563  3.963791 -1.027473 -1.938944   \n",
       "4  3.134112  7.608772  4.943759  9.857573  3.525811 -1.125531 -1.845975   \n",
       "\n",
       "         p4        g1        g2        g3        g4      stab     stabf  \n",
       "0 -1.723086  0.650456  0.859578  0.887445  0.958034  0.055347  unstable  \n",
       "1 -1.255012  0.413441  0.862414  0.562139  0.781760 -0.005957    stable  \n",
       "2 -0.920492  0.163041  0.766689  0.839444  0.109853  0.003471  unstable  \n",
       "3 -0.997374  0.446209  0.976744  0.929381  0.362718  0.028871  unstable  \n",
       "4 -0.554305  0.797110  0.455450  0.656947  0.820923  0.049860  unstable  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 14)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tau1     0\n",
       "tau2     0\n",
       "tau3     0\n",
       "tau4     0\n",
       "p1       0\n",
       "p2       0\n",
       "p3       0\n",
       "p4       0\n",
       "g1       0\n",
       "g2       0\n",
       "g3       0\n",
       "g4       0\n",
       "stab     0\n",
       "stabf    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 10000 entries, 0 to 9999\n",
      "Data columns (total 14 columns):\n",
      " #   Column  Non-Null Count  Dtype  \n",
      "---  ------  --------------  -----  \n",
      " 0   tau1    10000 non-null  float64\n",
      " 1   tau2    10000 non-null  float64\n",
      " 2   tau3    10000 non-null  float64\n",
      " 3   tau4    10000 non-null  float64\n",
      " 4   p1      10000 non-null  float64\n",
      " 5   p2      10000 non-null  float64\n",
      " 6   p3      10000 non-null  float64\n",
      " 7   p4      10000 non-null  float64\n",
      " 8   g1      10000 non-null  float64\n",
      " 9   g2      10000 non-null  float64\n",
      " 10  g3      10000 non-null  float64\n",
      " 11  g4      10000 non-null  float64\n",
      " 12  stab    10000 non-null  float64\n",
      " 13  stabf   10000 non-null  object \n",
      "dtypes: float64(13), object(1)\n",
      "memory usage: 1.1+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tau1</th>\n",
       "      <th>tau2</th>\n",
       "      <th>tau3</th>\n",
       "      <th>tau4</th>\n",
       "      <th>p1</th>\n",
       "      <th>p2</th>\n",
       "      <th>p3</th>\n",
       "      <th>p4</th>\n",
       "      <th>g1</th>\n",
       "      <th>g2</th>\n",
       "      <th>g3</th>\n",
       "      <th>g4</th>\n",
       "      <th>stab</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "      <td>10000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.250000</td>\n",
       "      <td>5.250001</td>\n",
       "      <td>5.250004</td>\n",
       "      <td>5.249997</td>\n",
       "      <td>3.750000</td>\n",
       "      <td>-1.250000</td>\n",
       "      <td>-1.250000</td>\n",
       "      <td>-1.250000</td>\n",
       "      <td>0.525000</td>\n",
       "      <td>0.525000</td>\n",
       "      <td>0.525000</td>\n",
       "      <td>0.525000</td>\n",
       "      <td>0.015731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.742548</td>\n",
       "      <td>2.742549</td>\n",
       "      <td>2.742549</td>\n",
       "      <td>2.742556</td>\n",
       "      <td>0.752160</td>\n",
       "      <td>0.433035</td>\n",
       "      <td>0.433035</td>\n",
       "      <td>0.433035</td>\n",
       "      <td>0.274256</td>\n",
       "      <td>0.274255</td>\n",
       "      <td>0.274255</td>\n",
       "      <td>0.274255</td>\n",
       "      <td>0.036919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.500793</td>\n",
       "      <td>0.500141</td>\n",
       "      <td>0.500788</td>\n",
       "      <td>0.500473</td>\n",
       "      <td>1.582590</td>\n",
       "      <td>-1.999891</td>\n",
       "      <td>-1.999945</td>\n",
       "      <td>-1.999926</td>\n",
       "      <td>0.050009</td>\n",
       "      <td>0.050053</td>\n",
       "      <td>0.050054</td>\n",
       "      <td>0.050028</td>\n",
       "      <td>-0.080760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.874892</td>\n",
       "      <td>2.875140</td>\n",
       "      <td>2.875522</td>\n",
       "      <td>2.874950</td>\n",
       "      <td>3.218300</td>\n",
       "      <td>-1.624901</td>\n",
       "      <td>-1.625025</td>\n",
       "      <td>-1.624960</td>\n",
       "      <td>0.287521</td>\n",
       "      <td>0.287552</td>\n",
       "      <td>0.287514</td>\n",
       "      <td>0.287494</td>\n",
       "      <td>-0.015557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.250004</td>\n",
       "      <td>5.249981</td>\n",
       "      <td>5.249979</td>\n",
       "      <td>5.249734</td>\n",
       "      <td>3.751025</td>\n",
       "      <td>-1.249966</td>\n",
       "      <td>-1.249974</td>\n",
       "      <td>-1.250007</td>\n",
       "      <td>0.525009</td>\n",
       "      <td>0.525003</td>\n",
       "      <td>0.525015</td>\n",
       "      <td>0.525002</td>\n",
       "      <td>0.017142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>7.624690</td>\n",
       "      <td>7.624893</td>\n",
       "      <td>7.624948</td>\n",
       "      <td>7.624838</td>\n",
       "      <td>4.282420</td>\n",
       "      <td>-0.874977</td>\n",
       "      <td>-0.875043</td>\n",
       "      <td>-0.875065</td>\n",
       "      <td>0.762435</td>\n",
       "      <td>0.762490</td>\n",
       "      <td>0.762440</td>\n",
       "      <td>0.762433</td>\n",
       "      <td>0.044878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>9.999469</td>\n",
       "      <td>9.999837</td>\n",
       "      <td>9.999450</td>\n",
       "      <td>9.999443</td>\n",
       "      <td>5.864418</td>\n",
       "      <td>-0.500108</td>\n",
       "      <td>-0.500072</td>\n",
       "      <td>-0.500025</td>\n",
       "      <td>0.999937</td>\n",
       "      <td>0.999944</td>\n",
       "      <td>0.999982</td>\n",
       "      <td>0.999930</td>\n",
       "      <td>0.109403</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               tau1          tau2          tau3          tau4            p1  \\\n",
       "count  10000.000000  10000.000000  10000.000000  10000.000000  10000.000000   \n",
       "mean       5.250000      5.250001      5.250004      5.249997      3.750000   \n",
       "std        2.742548      2.742549      2.742549      2.742556      0.752160   \n",
       "min        0.500793      0.500141      0.500788      0.500473      1.582590   \n",
       "25%        2.874892      2.875140      2.875522      2.874950      3.218300   \n",
       "50%        5.250004      5.249981      5.249979      5.249734      3.751025   \n",
       "75%        7.624690      7.624893      7.624948      7.624838      4.282420   \n",
       "max        9.999469      9.999837      9.999450      9.999443      5.864418   \n",
       "\n",
       "                 p2            p3            p4            g1            g2  \\\n",
       "count  10000.000000  10000.000000  10000.000000  10000.000000  10000.000000   \n",
       "mean      -1.250000     -1.250000     -1.250000      0.525000      0.525000   \n",
       "std        0.433035      0.433035      0.433035      0.274256      0.274255   \n",
       "min       -1.999891     -1.999945     -1.999926      0.050009      0.050053   \n",
       "25%       -1.624901     -1.625025     -1.624960      0.287521      0.287552   \n",
       "50%       -1.249966     -1.249974     -1.250007      0.525009      0.525003   \n",
       "75%       -0.874977     -0.875043     -0.875065      0.762435      0.762490   \n",
       "max       -0.500108     -0.500072     -0.500025      0.999937      0.999944   \n",
       "\n",
       "                 g3            g4          stab  \n",
       "count  10000.000000  10000.000000  10000.000000  \n",
       "mean       0.525000      0.525000      0.015731  \n",
       "std        0.274255      0.274255      0.036919  \n",
       "min        0.050054      0.050028     -0.080760  \n",
       "25%        0.287514      0.287494     -0.015557  \n",
       "50%        0.525015      0.525002      0.017142  \n",
       "75%        0.762440      0.762433      0.044878  \n",
       "max        0.999982      0.999930      0.109403  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unstable    63.8\n",
       "stable      36.2\n",
       "Name: stabf, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['stabf'].value_counts()/df.shape[0]*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop('stab', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tau1</th>\n",
       "      <th>tau2</th>\n",
       "      <th>tau3</th>\n",
       "      <th>tau4</th>\n",
       "      <th>p1</th>\n",
       "      <th>p2</th>\n",
       "      <th>p3</th>\n",
       "      <th>p4</th>\n",
       "      <th>g1</th>\n",
       "      <th>g2</th>\n",
       "      <th>g3</th>\n",
       "      <th>g4</th>\n",
       "      <th>stabf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.959060</td>\n",
       "      <td>3.079885</td>\n",
       "      <td>8.381025</td>\n",
       "      <td>9.780754</td>\n",
       "      <td>3.763085</td>\n",
       "      <td>-0.782604</td>\n",
       "      <td>-1.257395</td>\n",
       "      <td>-1.723086</td>\n",
       "      <td>0.650456</td>\n",
       "      <td>0.859578</td>\n",
       "      <td>0.887445</td>\n",
       "      <td>0.958034</td>\n",
       "      <td>unstable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.304097</td>\n",
       "      <td>4.902524</td>\n",
       "      <td>3.047541</td>\n",
       "      <td>1.369357</td>\n",
       "      <td>5.067812</td>\n",
       "      <td>-1.940058</td>\n",
       "      <td>-1.872742</td>\n",
       "      <td>-1.255012</td>\n",
       "      <td>0.413441</td>\n",
       "      <td>0.862414</td>\n",
       "      <td>0.562139</td>\n",
       "      <td>0.781760</td>\n",
       "      <td>stable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.971707</td>\n",
       "      <td>8.848428</td>\n",
       "      <td>3.046479</td>\n",
       "      <td>1.214518</td>\n",
       "      <td>3.405158</td>\n",
       "      <td>-1.207456</td>\n",
       "      <td>-1.277210</td>\n",
       "      <td>-0.920492</td>\n",
       "      <td>0.163041</td>\n",
       "      <td>0.766689</td>\n",
       "      <td>0.839444</td>\n",
       "      <td>0.109853</td>\n",
       "      <td>unstable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.716415</td>\n",
       "      <td>7.669600</td>\n",
       "      <td>4.486641</td>\n",
       "      <td>2.340563</td>\n",
       "      <td>3.963791</td>\n",
       "      <td>-1.027473</td>\n",
       "      <td>-1.938944</td>\n",
       "      <td>-0.997374</td>\n",
       "      <td>0.446209</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>0.929381</td>\n",
       "      <td>0.362718</td>\n",
       "      <td>unstable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.134112</td>\n",
       "      <td>7.608772</td>\n",
       "      <td>4.943759</td>\n",
       "      <td>9.857573</td>\n",
       "      <td>3.525811</td>\n",
       "      <td>-1.125531</td>\n",
       "      <td>-1.845975</td>\n",
       "      <td>-0.554305</td>\n",
       "      <td>0.797110</td>\n",
       "      <td>0.455450</td>\n",
       "      <td>0.656947</td>\n",
       "      <td>0.820923</td>\n",
       "      <td>unstable</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       tau1      tau2      tau3      tau4        p1        p2        p3  \\\n",
       "0  2.959060  3.079885  8.381025  9.780754  3.763085 -0.782604 -1.257395   \n",
       "1  9.304097  4.902524  3.047541  1.369357  5.067812 -1.940058 -1.872742   \n",
       "2  8.971707  8.848428  3.046479  1.214518  3.405158 -1.207456 -1.277210   \n",
       "3  0.716415  7.669600  4.486641  2.340563  3.963791 -1.027473 -1.938944   \n",
       "4  3.134112  7.608772  4.943759  9.857573  3.525811 -1.125531 -1.845975   \n",
       "\n",
       "         p4        g1        g2        g3        g4     stabf  \n",
       "0 -1.723086  0.650456  0.859578  0.887445  0.958034  unstable  \n",
       "1 -1.255012  0.413441  0.862414  0.562139  0.781760    stable  \n",
       "2 -0.920492  0.163041  0.766689  0.839444  0.109853  unstable  \n",
       "3 -0.997374  0.446209  0.976744  0.929381  0.362718  unstable  \n",
       "4 -0.554305  0.797110  0.455450  0.656947  0.820923  unstable  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#splitting features and target variable\n",
    "x = df.drop('stabf', axis=1)\n",
    "y = df['stabf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#split the data into training and testing sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size= 0.2 , random_state= 1 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Scaling dataset using standard scaler\n",
    "# train datset\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "std_train_df = scaler.fit_transform(x_train)\n",
    "std_train_df = pd.DataFrame(std_train_df, columns=x_train.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tau1</th>\n",
       "      <th>tau2</th>\n",
       "      <th>tau3</th>\n",
       "      <th>tau4</th>\n",
       "      <th>p1</th>\n",
       "      <th>p2</th>\n",
       "      <th>p3</th>\n",
       "      <th>p4</th>\n",
       "      <th>g1</th>\n",
       "      <th>g2</th>\n",
       "      <th>g3</th>\n",
       "      <th>g4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.367327</td>\n",
       "      <td>-0.986042</td>\n",
       "      <td>0.650447</td>\n",
       "      <td>1.547527</td>\n",
       "      <td>-0.291490</td>\n",
       "      <td>0.061535</td>\n",
       "      <td>1.293862</td>\n",
       "      <td>-0.845074</td>\n",
       "      <td>0.160918</td>\n",
       "      <td>0.339859</td>\n",
       "      <td>0.585568</td>\n",
       "      <td>0.492239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.064659</td>\n",
       "      <td>0.089437</td>\n",
       "      <td>1.035079</td>\n",
       "      <td>-1.641494</td>\n",
       "      <td>0.619865</td>\n",
       "      <td>-0.067235</td>\n",
       "      <td>-1.502925</td>\n",
       "      <td>0.486613</td>\n",
       "      <td>-0.293143</td>\n",
       "      <td>-1.558488</td>\n",
       "      <td>1.429649</td>\n",
       "      <td>-1.443521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.467850</td>\n",
       "      <td>1.298418</td>\n",
       "      <td>-0.502536</td>\n",
       "      <td>1.166046</td>\n",
       "      <td>-0.180521</td>\n",
       "      <td>0.490603</td>\n",
       "      <td>0.682560</td>\n",
       "      <td>-0.855302</td>\n",
       "      <td>1.399350</td>\n",
       "      <td>1.451534</td>\n",
       "      <td>-1.045743</td>\n",
       "      <td>0.492489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.820081</td>\n",
       "      <td>0.529920</td>\n",
       "      <td>1.299657</td>\n",
       "      <td>-1.141975</td>\n",
       "      <td>-0.812854</td>\n",
       "      <td>-0.763632</td>\n",
       "      <td>1.521579</td>\n",
       "      <td>0.658780</td>\n",
       "      <td>-0.958319</td>\n",
       "      <td>1.361958</td>\n",
       "      <td>1.604140</td>\n",
       "      <td>0.275303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.665424</td>\n",
       "      <td>-1.425627</td>\n",
       "      <td>0.312300</td>\n",
       "      <td>0.919137</td>\n",
       "      <td>-1.614296</td>\n",
       "      <td>0.760315</td>\n",
       "      <td>1.422019</td>\n",
       "      <td>0.639243</td>\n",
       "      <td>1.676895</td>\n",
       "      <td>0.695660</td>\n",
       "      <td>1.137504</td>\n",
       "      <td>-1.312575</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       tau1      tau2      tau3      tau4        p1        p2        p3  \\\n",
       "0  0.367327 -0.986042  0.650447  1.547527 -0.291490  0.061535  1.293862   \n",
       "1 -0.064659  0.089437  1.035079 -1.641494  0.619865 -0.067235 -1.502925   \n",
       "2 -1.467850  1.298418 -0.502536  1.166046 -0.180521  0.490603  0.682560   \n",
       "3  0.820081  0.529920  1.299657 -1.141975 -0.812854 -0.763632  1.521579   \n",
       "4  0.665424 -1.425627  0.312300  0.919137 -1.614296  0.760315  1.422019   \n",
       "\n",
       "         p4        g1        g2        g3        g4  \n",
       "0 -0.845074  0.160918  0.339859  0.585568  0.492239  \n",
       "1  0.486613 -0.293143 -1.558488  1.429649 -1.443521  \n",
       "2 -0.855302  1.399350  1.451534 -1.045743  0.492489  \n",
       "3  0.658780 -0.958319  1.361958  1.604140  0.275303  \n",
       "4  0.639243  1.676895  0.695660  1.137504 -1.312575  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "std_train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scaling test data set\n",
    "std_test_df = scaler.fit_transform(x_test)\n",
    "std_test_df = pd.DataFrame(std_test_df, columns=x_test.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tau1</th>\n",
       "      <th>tau2</th>\n",
       "      <th>tau3</th>\n",
       "      <th>tau4</th>\n",
       "      <th>p1</th>\n",
       "      <th>p2</th>\n",
       "      <th>p3</th>\n",
       "      <th>p4</th>\n",
       "      <th>g1</th>\n",
       "      <th>g2</th>\n",
       "      <th>g3</th>\n",
       "      <th>g4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.592163</td>\n",
       "      <td>-0.420565</td>\n",
       "      <td>1.472472</td>\n",
       "      <td>1.093036</td>\n",
       "      <td>0.426786</td>\n",
       "      <td>-1.504594</td>\n",
       "      <td>-0.792677</td>\n",
       "      <td>1.600201</td>\n",
       "      <td>-0.925703</td>\n",
       "      <td>1.175287</td>\n",
       "      <td>-1.492644</td>\n",
       "      <td>1.086291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.199183</td>\n",
       "      <td>0.364543</td>\n",
       "      <td>-0.190076</td>\n",
       "      <td>-0.518473</td>\n",
       "      <td>-0.229402</td>\n",
       "      <td>-1.071766</td>\n",
       "      <td>0.427103</td>\n",
       "      <td>1.052337</td>\n",
       "      <td>-1.655910</td>\n",
       "      <td>-0.395949</td>\n",
       "      <td>1.412703</td>\n",
       "      <td>1.227535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.086035</td>\n",
       "      <td>-0.321834</td>\n",
       "      <td>-0.873505</td>\n",
       "      <td>0.011761</td>\n",
       "      <td>-0.977094</td>\n",
       "      <td>0.094896</td>\n",
       "      <td>0.813041</td>\n",
       "      <td>0.751381</td>\n",
       "      <td>1.450284</td>\n",
       "      <td>-1.444370</td>\n",
       "      <td>0.654216</td>\n",
       "      <td>-1.679799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.087014</td>\n",
       "      <td>-1.113357</td>\n",
       "      <td>0.361518</td>\n",
       "      <td>-1.684316</td>\n",
       "      <td>0.792280</td>\n",
       "      <td>-1.649041</td>\n",
       "      <td>0.410662</td>\n",
       "      <td>-0.084473</td>\n",
       "      <td>0.066085</td>\n",
       "      <td>-1.679450</td>\n",
       "      <td>-0.349573</td>\n",
       "      <td>1.057439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.873004</td>\n",
       "      <td>1.425833</td>\n",
       "      <td>0.080476</td>\n",
       "      <td>1.681022</td>\n",
       "      <td>-0.154247</td>\n",
       "      <td>-0.024315</td>\n",
       "      <td>-0.197525</td>\n",
       "      <td>0.485988</td>\n",
       "      <td>0.119716</td>\n",
       "      <td>-1.475773</td>\n",
       "      <td>0.957057</td>\n",
       "      <td>-0.817608</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       tau1      tau2      tau3      tau4        p1        p2        p3  \\\n",
       "0  0.592163 -0.420565  1.472472  1.093036  0.426786 -1.504594 -0.792677   \n",
       "1  0.199183  0.364543 -0.190076 -0.518473 -0.229402 -1.071766  0.427103   \n",
       "2 -1.086035 -0.321834 -0.873505  0.011761 -0.977094  0.094896  0.813041   \n",
       "3 -0.087014 -1.113357  0.361518 -1.684316  0.792280 -1.649041  0.410662   \n",
       "4  0.873004  1.425833  0.080476  1.681022 -0.154247 -0.024315 -0.197525   \n",
       "\n",
       "         p4        g1        g2        g3        g4  \n",
       "0  1.600201 -0.925703  1.175287 -1.492644  1.086291  \n",
       "1  1.052337 -1.655910 -0.395949  1.412703  1.227535  \n",
       "2  0.751381  1.450284 -1.444370  0.654216 -1.679799  \n",
       "3 -0.084473  0.066085 -1.679450 -0.349573  1.057439  \n",
       "4  0.485988  0.119716 -1.475773  0.957057 -0.817608  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "std_test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier()"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Random Forest Classifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "random_forest = RandomForestClassifier(n_estimators=100)\n",
    "random_forest.fit(std_train_df, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_rfc = random_forest.predict(std_test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['unstable', 'unstable', 'stable', ..., 'stable', 'stable',\n",
       "       'unstable'], dtype=object)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_rfc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 626,   86],\n",
       "       [  66, 1222]], dtype=int64)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Confusion Matrix for Random Forest\n",
    "from sklearn.metrics import recall_score, accuracy_score, precision_score, f1_score, confusion_matrix\n",
    "cnf_mat = confusion_matrix(y_true=y_test, y_pred=y_pred_rfc, labels=[ 'stable', 'unstable' ])\n",
    "cnf_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.924\n"
     ]
    }
   ],
   "source": [
    "#Random Forest Accuracy\n",
    "accuracy = accuracy_score(y_test,y_pred_rfc)\n",
    "print(f\"Accuracy: {round(accuracy,4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14:16:51] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.4.0/src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "              colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
       "              importance_type='gain', interaction_constraints='',\n",
       "              learning_rate=0.300000012, max_delta_step=0, max_depth=6,\n",
       "              min_child_weight=1, missing=nan, monotone_constraints='()',\n",
       "              n_estimators=100, n_jobs=4, num_parallel_tree=1, random_state=1,\n",
       "              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,\n",
       "              tree_method='exact', validate_parameters=1, verbosity=None)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#xgboost classifier\n",
    "from xgboost import XGBClassifier\n",
    "xgb = XGBClassifier(random_state=1)\n",
    "xgb.fit(std_train_df, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_xgb = xgb.predict(std_test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 647,   65],\n",
       "       [  43, 1245]], dtype=int64)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Confusion Matrix for XGB\n",
    "cnf_mat_xgb = confusion_matrix(y_true=y_test, y_pred=y_pred_xgb, labels=[ 'stable', 'unstable' ])\n",
    "cnf_mat_xgb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.946\n"
     ]
    }
   ],
   "source": [
    "#XGBoost Accuracy\n",
    "accuracy = accuracy_score(y_test,y_pred_xgb)\n",
    "print(f\"Accuracy: {round(accuracy,4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Light GBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Lightgbm Model\n",
    "import lightgbm as lgb\n",
    "lgb_clf = lgb.LGBMClassifier(random_state=1)\n",
    "lgb_clf.fit(std_train_df, y_train)\n",
    "y_pred_lgb = lgb_clf.predict(std_test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 636,   76],\n",
       "       [  51, 1237]], dtype=int64)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Confusion Matrix for XGB\n",
    "cnf_mat_lgb = confusion_matrix(y_true=y_test, y_pred=y_pred_lgb, labels=[ 'stable', 'unstable' ])\n",
    "cnf_mat_lgb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9365\n"
     ]
    }
   ],
   "source": [
    "#Lightgbm Accuracy\n",
    "accuracy = accuracy_score(y_test,y_pred_lgb)\n",
    "print(f\"Accuracy: {round(accuracy,4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra Trees Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "etc = ExtraTreesClassifier(random_state = 1)\n",
    "etc.fit(std_train_df, y_train)\n",
    "y_pred_etc = etc.predict(std_test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 602,  110],\n",
       "       [  38, 1250]], dtype=int64)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Confusion Matrix for Extra Tree Classifier\n",
    "cnf_mat_etc = confusion_matrix(y_true=y_test, y_pred=y_pred_etc, labels=[ 'stable', 'unstable' ])\n",
    "cnf_mat_etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.926\n"
     ]
    }
   ],
   "source": [
    "#Extra Tree Classifier Accuracy\n",
    "accuracy = accuracy_score(y_test,y_pred_etc)\n",
    "print(f\"Accuracy: {round(accuracy,4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuning Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_estimators = [50,100,300,500,1000] \n",
    "min_samples_split = [2,3,5,7,9] \n",
    "min_samples_leaf = [1,2,4,6,8] \n",
    "max_features = ['auto','sqrt','log2',None] \n",
    "hyperparameter_grid = {'n_estimators':n_estimators,\n",
    "                       'min_samples_leaf':min_samples_leaf,\n",
    "                       'min_samples_split':min_samples_split,\n",
    "                       'max_features':max_features}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Randomized CrossValidation Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "ran_search = RandomizedSearchCV(estimator = etc,param_distributions=hyperparameter_grid,random_state=1)\n",
    "ran_search = ran_search.fit(std_train_df, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': 1000,\n",
       " 'min_samples_split': 2,\n",
       " 'min_samples_leaf': 8,\n",
       " 'max_features': None}"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ran_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# New CLassifier using Extra Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classifier = ExtraTreesClassifier(random_state=1,n_estimators = 1000,\n",
    "                                       min_samples_leaf= 8,max_features=None,\n",
    "                                       min_samples_split=2)\n",
    "n_classifier.fit(std_train_df, y_train)\n",
    "new_pred = n_classifier.predict(std_test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9285\n"
     ]
    }
   ],
   "source": [
    "accuracy = accuracy_score(y_test,new_pred)\n",
    "print(f\"Accuracy: {round(accuracy,4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.13723975, 0.1405075 , 0.13468029, 0.13541676, 0.00368342,\n",
       "       0.00533686, 0.00542927, 0.00496249, 0.10256244, 0.10757765,\n",
       "       0.11306268, 0.10954089])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imp = ran_search.best_estimator_.feature_importances_\n",
    "imp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0.14050750384993677, 'tau2'),\n",
       " (0.13723974766109256, 'tau1'),\n",
       " (0.1354167630909727, 'tau4'),\n",
       " (0.13468028520386593, 'tau3'),\n",
       " (0.11306267999167334, 'g3'),\n",
       " (0.10954089174337298, 'g4'),\n",
       " (0.10757764577478764, 'g2'),\n",
       " (0.10256244080927947, 'g1'),\n",
       " (0.005429268421191957, 'p3'),\n",
       " (0.005336864710946151, 'p2'),\n",
       " (0.004962486591192238, 'p4'),\n",
       " (0.003683422151688322, 'p1')]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(zip(imp,x),reverse = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}