t81_558_class2_python_ml.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 2: Python for Machine Learning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* [Part 2.1: Dealing with Data in Python with Pandas](https://www.youtube.com/watch?v=Bj2m6hvRoNk&index=6&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Part 2.2: Machine Learning Background for Deep Learning, Keras and Tensorflow](https://www.youtube.com/watch?v=WCXzchgxi9c&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Part 2.3: Pandas and Machine Learning](https://www.youtube.com/watch?v=eZGunTjrHyA&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "\n",
    "Weekly video update:\n",
    "\n",
    "* [In Class Meeting #1, January 22, 2018](https://www.youtube.com/watch?v=J2lEeCoYQqw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions\n",
    "\n",
    "You will see these at the top of every module.  These are simply a set of reusable functions that we will make use of.  Each of them will be explained as the semester progresses.  They are explained in greater detail as the course progresses.  Class 4 contains a complete overview of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import os\n",
    "import requests\n",
    "import base64\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = \"{}-{}\".format(name, x)\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = \"{}-{}\".format(name, tv)\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df.as_matrix(result).astype(np.float32), dummies.as_matrix().astype(np.float32)\n",
    "    else:\n",
    "        # Regression\n",
    "        return df.as_matrix(result).astype(np.float32), df.as_matrix([target]).astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred,y,sort=True):\n",
    "    t = pd.DataFrame({'pred' : pred, 'y' : y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'],inplace=True)\n",
    "    a = plt.plot(t['y'].tolist(),label='expected')\n",
    "    b = plt.plot(t['pred'].tolist(),label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean()) >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "               * (normalized_high - normalized_low) + normalized_low\n",
    "        \n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - Pandas dataframe output.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.  \n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    r = requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={'csv':base64.b64encode(data.to_csv(index=False).encode('ascii')).decode(\"ascii\"),\n",
    "        'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code == 200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas\n",
    "======\n",
    "[Pandas](http://pandas.pydata.org/) is an open source library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.  It is based on the [dataframe](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) concept found in the [R programming language](https://www.r-project.org/about.html).  For this class, Pandas will be the primary means by which data is manipulated in conjunction with neural networks.\n",
    "\n",
    "The dataframe is a key component of Pandas.  We will use it to access the [auto-mpg dataset](https://archive.ics.uci.edu/ml/datasets/Auto+MPG).  This dataset can be found on the UCI machine learning repository.  For this class we will use a version of the Auto MPG dataset where I added column headers.  You can find my version [here](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/data/auto-mpg.csv).\n",
    "\n",
    "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University. The dataset was used in the 1983 American Statistical Association Exposition.  It contains data for 398 cars, including [mpg](https://en.wikipedia.org/wiki/Fuel_economy_in_automobiles), [cylinders](https://en.wikipedia.org/wiki/Cylinder_(engine)), [displacement](https://en.wikipedia.org/wiki/Engine_displacement), [horsepower](https://en.wikipedia.org/wiki/Horsepower) , weight, acceleration, model year, origin and the car's name.\n",
    "\n",
    "The following code loads the MPG dataset into a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    mpg  cylinders  displacement horsepower  weight  acceleration  year  \\\n",
      "0  18.0          8         307.0        130    3504          12.0    70   \n",
      "1  15.0          8         350.0        165    3693          11.5    70   \n",
      "2  18.0          8         318.0        150    3436          11.0    70   \n",
      "3  16.0          8         304.0        150    3433          12.0    70   \n",
      "4  17.0          8         302.0        140    3449          10.5    70   \n",
      "\n",
      "   origin                       name  \n",
      "0       1  chevrolet chevelle malibu  \n",
      "1       1          buick skylark 320  \n",
      "2       1         plymouth satellite  \n",
      "3       1              amc rebel sst  \n",
      "4       1                ford torino  \n"
     ]
    }
   ],
   "source": [
    "# Simple dataframe\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read)\n",
    "print(df[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'mpg', 'mean': 23.514572864321615, 'var': 61.089610774274405, 'sdev': 7.815984312565782}\n",
      "{'name': 'cylinders', 'mean': 5.454773869346734, 'var': 2.8934154399199943, 'sdev': 1.7010042445332094}\n",
      "{'name': 'displacement', 'mean': 193.42587939698493, 'var': 10872.199152247364, 'sdev': 104.26983817119581}\n",
      "{'name': 'horsepower', 'mean': 104.46938775510205, 'var': 1481.5693929745862, 'sdev': 38.49115993282855}\n",
      "{'name': 'weight', 'mean': 2970.424623115578, 'var': 717140.9905256768, 'sdev': 846.8417741973271}\n",
      "{'name': 'acceleration', 'mean': 15.568090452261291, 'var': 7.604848233611381, 'sdev': 2.7576889298126757}\n",
      "{'name': 'year', 'mean': 76.01005025125629, 'var': 13.672442818627143, 'sdev': 3.697626646732623}\n",
      "{'name': 'origin', 'mean': 1.5728643216080402, 'var': 0.6432920268850575, 'sdev': 0.8020548777266163}\n"
     ]
    }
   ],
   "source": [
    "# Perform basic statistics on a dataframe.\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "# Strip non-numerics\n",
    "df = df.select_dtypes(include=['int', 'float'])\n",
    "\n",
    "headers = list(df.columns.values)\n",
    "fields = []\n",
    "\n",
    "for field in headers:\n",
    "    fields.append( {\n",
    "        'name' : field,\n",
    "        'mean': df[field].mean(),\n",
    "        'var': df[field].var(),\n",
    "        'sdev': df[field].std()\n",
    "    })\n",
    "    \n",
    "for field in fields:\n",
    "    print(field)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sorting and Shuffling Dataframes\n",
    "It is possable to sort and shuffle.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>27.0</td>\n",
       "      <td>4</td>\n",
       "      <td>151.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2950</td>\n",
       "      <td>17.3</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet camaro</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>22.0</td>\n",
       "      <td>4</td>\n",
       "      <td>122.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>2395</td>\n",
       "      <td>16.0</td>\n",
       "      <td>72</td>\n",
       "      <td>1</td>\n",
       "      <td>ford pinto (sw)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28.0</td>\n",
       "      <td>4</td>\n",
       "      <td>98.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>2164</td>\n",
       "      <td>15.0</td>\n",
       "      <td>72</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge colt (sw)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>32.3</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>2065</td>\n",
       "      <td>17.8</td>\n",
       "      <td>81</td>\n",
       "      <td>3</td>\n",
       "      <td>subaru</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>37.0</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>2025</td>\n",
       "      <td>18.2</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>mazda glc custom l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>30.0</td>\n",
       "      <td>4</td>\n",
       "      <td>79.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2074</td>\n",
       "      <td>19.5</td>\n",
       "      <td>71</td>\n",
       "      <td>2</td>\n",
       "      <td>peugeot 304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>11.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>210.0</td>\n",
       "      <td>4382</td>\n",
       "      <td>13.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge d200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>18.0</td>\n",
       "      <td>6</td>\n",
       "      <td>250.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>3139</td>\n",
       "      <td>14.5</td>\n",
       "      <td>71</td>\n",
       "      <td>1</td>\n",
       "      <td>ford mustang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>38.0</td>\n",
       "      <td>6</td>\n",
       "      <td>262.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>3015</td>\n",
       "      <td>17.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>oldsmobile cutlass ciera (diesel)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>400.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3761</td>\n",
       "      <td>9.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet monte carlo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>14.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>4077</td>\n",
       "      <td>14.0</td>\n",
       "      <td>72</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite custom (sw)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>37.0</td>\n",
       "      <td>4</td>\n",
       "      <td>85.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>1975</td>\n",
       "      <td>19.4</td>\n",
       "      <td>81</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun 210 mpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>28.0</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2264</td>\n",
       "      <td>15.5</td>\n",
       "      <td>71</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet vega 2300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14.0</td>\n",
       "      <td>8</td>\n",
       "      <td>454.0</td>\n",
       "      <td>220.0</td>\n",
       "      <td>4354</td>\n",
       "      <td>9.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet impala</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>30.5</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>78.0</td>\n",
       "      <td>2190</td>\n",
       "      <td>14.1</td>\n",
       "      <td>77</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen dasher</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>32.0</td>\n",
       "      <td>4</td>\n",
       "      <td>144.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>2665</td>\n",
       "      <td>13.9</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>toyota celica gt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>34.0</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2245</td>\n",
       "      <td>16.9</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>toyota corolla</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>12.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>4499</td>\n",
       "      <td>12.5</td>\n",
       "      <td>73</td>\n",
       "      <td>1</td>\n",
       "      <td>oldsmobile vista cruiser</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>43.1</td>\n",
       "      <td>4</td>\n",
       "      <td>90.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1985</td>\n",
       "      <td>21.5</td>\n",
       "      <td>78</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen rabbit custom diesel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>34.1</td>\n",
       "      <td>4</td>\n",
       "      <td>86.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>1975</td>\n",
       "      <td>15.2</td>\n",
       "      <td>79</td>\n",
       "      <td>3</td>\n",
       "      <td>maxda glc deluxe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21.6</td>\n",
       "      <td>4</td>\n",
       "      <td>121.0</td>\n",
       "      <td>115.0</td>\n",
       "      <td>2795</td>\n",
       "      <td>15.7</td>\n",
       "      <td>78</td>\n",
       "      <td>2</td>\n",
       "      <td>saab 99gle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22.0</td>\n",
       "      <td>6</td>\n",
       "      <td>232.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>2835</td>\n",
       "      <td>14.7</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford granada l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>38.0</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1965</td>\n",
       "      <td>15.0</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>honda civic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>16.0</td>\n",
       "      <td>6</td>\n",
       "      <td>250.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>3781</td>\n",
       "      <td>17.0</td>\n",
       "      <td>74</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu classic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>23.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>3900</td>\n",
       "      <td>17.4</td>\n",
       "      <td>79</td>\n",
       "      <td>1</td>\n",
       "      <td>cadillac eldorado</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>13.0</td>\n",
       "      <td>8</td>\n",
       "      <td>400.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>4464</td>\n",
       "      <td>12.0</td>\n",
       "      <td>73</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet caprice classic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>14.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>4096</td>\n",
       "      <td>13.0</td>\n",
       "      <td>71</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth fury iii</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>21.0</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>2587</td>\n",
       "      <td>16.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford maverick</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>400.0</td>\n",
       "      <td>230.0</td>\n",
       "      <td>4278</td>\n",
       "      <td>9.5</td>\n",
       "      <td>73</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac grand prix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>19.4</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>3735</td>\n",
       "      <td>13.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge diplomat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>26.0</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1835</td>\n",
       "      <td>20.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen 1131 deluxe sedan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>369</th>\n",
       "      <td>39.4</td>\n",
       "      <td>4</td>\n",
       "      <td>85.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2070</td>\n",
       "      <td>18.6</td>\n",
       "      <td>78</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun b210 gx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>370</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>371</th>\n",
       "      <td>25.0</td>\n",
       "      <td>6</td>\n",
       "      <td>181.0</td>\n",
       "      <td>110.0</td>\n",
       "      <td>2945</td>\n",
       "      <td>16.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>buick century limited</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>22.0</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>2408</td>\n",
       "      <td>19.0</td>\n",
       "      <td>71</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet vega (sw)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>373</th>\n",
       "      <td>37.3</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>14.7</td>\n",
       "      <td>79</td>\n",
       "      <td>2</td>\n",
       "      <td>fiat strada custom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>374</th>\n",
       "      <td>36.0</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2160</td>\n",
       "      <td>14.5</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>nissan stanza xe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>17.5</td>\n",
       "      <td>6</td>\n",
       "      <td>250.0</td>\n",
       "      <td>110.0</td>\n",
       "      <td>3520</td>\n",
       "      <td>16.4</td>\n",
       "      <td>77</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet concours</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>14.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>4209</td>\n",
       "      <td>12.0</td>\n",
       "      <td>71</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet impala</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>26.0</td>\n",
       "      <td>4</td>\n",
       "      <td>98.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2265</td>\n",
       "      <td>15.5</td>\n",
       "      <td>73</td>\n",
       "      <td>2</td>\n",
       "      <td>fiat 124 sport coupe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>22.0</td>\n",
       "      <td>4</td>\n",
       "      <td>121.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>2945</td>\n",
       "      <td>14.5</td>\n",
       "      <td>75</td>\n",
       "      <td>2</td>\n",
       "      <td>volvo 244dl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>15.5</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>4140</td>\n",
       "      <td>13.7</td>\n",
       "      <td>77</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge monaco brougham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>380</th>\n",
       "      <td>26.0</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>78.0</td>\n",
       "      <td>2300</td>\n",
       "      <td>14.5</td>\n",
       "      <td>74</td>\n",
       "      <td>2</td>\n",
       "      <td>opel manta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>35.1</td>\n",
       "      <td>4</td>\n",
       "      <td>81.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>1760</td>\n",
       "      <td>16.1</td>\n",
       "      <td>81</td>\n",
       "      <td>3</td>\n",
       "      <td>honda civic 1300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>22.0</td>\n",
       "      <td>6</td>\n",
       "      <td>198.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2833</td>\n",
       "      <td>15.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth duster</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>383</th>\n",
       "      <td>23.0</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2957</td>\n",
       "      <td>17.0</td>\n",
       "      <td>75</td>\n",
       "      <td>2</td>\n",
       "      <td>peugeot 504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>23.7</td>\n",
       "      <td>3</td>\n",
       "      <td>70.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2420</td>\n",
       "      <td>12.5</td>\n",
       "      <td>80</td>\n",
       "      <td>3</td>\n",
       "      <td>mazda rx-7 gs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>385</th>\n",
       "      <td>25.0</td>\n",
       "      <td>4</td>\n",
       "      <td>90.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>2223</td>\n",
       "      <td>16.5</td>\n",
       "      <td>75</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen dasher</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>4440</td>\n",
       "      <td>14.0</td>\n",
       "      <td>75</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet bel air</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>35.0</td>\n",
       "      <td>4</td>\n",
       "      <td>72.0</td>\n",
       "      <td>69.0</td>\n",
       "      <td>1613</td>\n",
       "      <td>18.0</td>\n",
       "      <td>71</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun 1200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>23.0</td>\n",
       "      <td>4</td>\n",
       "      <td>115.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2694</td>\n",
       "      <td>15.0</td>\n",
       "      <td>75</td>\n",
       "      <td>2</td>\n",
       "      <td>audi 100ls</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>24.2</td>\n",
       "      <td>6</td>\n",
       "      <td>146.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>2930</td>\n",
       "      <td>13.8</td>\n",
       "      <td>81</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun 810 maxima</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>13.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>3169</td>\n",
       "      <td>12.0</td>\n",
       "      <td>75</td>\n",
       "      <td>1</td>\n",
       "      <td>ford mustang ii</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>31.5</td>\n",
       "      <td>4</td>\n",
       "      <td>89.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>1990</td>\n",
       "      <td>14.9</td>\n",
       "      <td>78</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen scirocco</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>392</th>\n",
       "      <td>13.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>4294</td>\n",
       "      <td>16.0</td>\n",
       "      <td>72</td>\n",
       "      <td>1</td>\n",
       "      <td>ford gran torino (sw)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>15.0</td>\n",
       "      <td>6</td>\n",
       "      <td>250.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>3336</td>\n",
       "      <td>17.0</td>\n",
       "      <td>74</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet nova</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>18.0</td>\n",
       "      <td>3</td>\n",
       "      <td>70.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2124</td>\n",
       "      <td>13.5</td>\n",
       "      <td>73</td>\n",
       "      <td>3</td>\n",
       "      <td>maxda rx3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395</th>\n",
       "      <td>36.4</td>\n",
       "      <td>5</td>\n",
       "      <td>121.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>2950</td>\n",
       "      <td>19.9</td>\n",
       "      <td>80</td>\n",
       "      <td>2</td>\n",
       "      <td>audi 5000s (diesel)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>17.5</td>\n",
       "      <td>6</td>\n",
       "      <td>258.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>3193</td>\n",
       "      <td>17.8</td>\n",
       "      <td>76</td>\n",
       "      <td>1</td>\n",
       "      <td>amc pacer d/l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>19.2</td>\n",
       "      <td>8</td>\n",
       "      <td>305.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>3425</td>\n",
       "      <td>13.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet monte carlo landau</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>398 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "0    27.0          4         151.0        90.0    2950          17.3    82   \n",
       "1    22.0          4         122.0        86.0    2395          16.0    72   \n",
       "2    28.0          4          98.0        80.0    2164          15.0    72   \n",
       "3    32.3          4          97.0        67.0    2065          17.8    81   \n",
       "4    37.0          4          91.0        68.0    2025          18.2    82   \n",
       "5    30.0          4          79.0        70.0    2074          19.5    71   \n",
       "6    11.0          8         318.0       210.0    4382          13.5    70   \n",
       "7    18.0          6         250.0        88.0    3139          14.5    71   \n",
       "8    38.0          6         262.0        85.0    3015          17.0    82   \n",
       "9    15.0          8         400.0       150.0    3761           9.5    70   \n",
       "10   14.0          8         318.0       150.0    4077          14.0    72   \n",
       "11   37.0          4          85.0        65.0    1975          19.4    81   \n",
       "12   28.0          4         140.0        90.0    2264          15.5    71   \n",
       "13   14.0          8         454.0       220.0    4354           9.0    70   \n",
       "14   30.5          4          97.0        78.0    2190          14.1    77   \n",
       "15   32.0          4         144.0        96.0    2665          13.9    82   \n",
       "16   34.0          4         108.0        70.0    2245          16.9    82   \n",
       "17   12.0          8         350.0       180.0    4499          12.5    73   \n",
       "18   43.1          4          90.0        48.0    1985          21.5    78   \n",
       "19   34.1          4          86.0        65.0    1975          15.2    79   \n",
       "20   21.6          4         121.0       115.0    2795          15.7    78   \n",
       "21   22.0          6         232.0       112.0    2835          14.7    82   \n",
       "22   38.0          4          91.0        67.0    1965          15.0    82   \n",
       "23   16.0          6         250.0       100.0    3781          17.0    74   \n",
       "24   23.0          8         350.0       125.0    3900          17.4    79   \n",
       "25   13.0          8         400.0       150.0    4464          12.0    73   \n",
       "26   14.0          8         318.0       150.0    4096          13.0    71   \n",
       "27   21.0          6         200.0        85.0    2587          16.0    70   \n",
       "28   16.0          8         400.0       230.0    4278           9.5    73   \n",
       "29   19.4          8         318.0       140.0    3735          13.2    78   \n",
       "..    ...        ...           ...         ...     ...           ...   ...   \n",
       "368  26.0          4          97.0        46.0    1835          20.5    70   \n",
       "369  39.4          4          85.0        70.0    2070          18.6    78   \n",
       "370  16.0          8         304.0       150.0    3433          12.0    70   \n",
       "371  25.0          6         181.0       110.0    2945          16.4    82   \n",
       "372  22.0          4         140.0        72.0    2408          19.0    71   \n",
       "373  37.3          4          91.0        69.0    2130          14.7    79   \n",
       "374  36.0          4         120.0        88.0    2160          14.5    82   \n",
       "375  17.5          6         250.0       110.0    3520          16.4    77   \n",
       "376  14.0          8         350.0       165.0    4209          12.0    71   \n",
       "377  26.0          4          98.0        90.0    2265          15.5    73   \n",
       "378  22.0          4         121.0        98.0    2945          14.5    75   \n",
       "379  15.5          8         318.0       145.0    4140          13.7    77   \n",
       "380  26.0          4          97.0        78.0    2300          14.5    74   \n",
       "381  35.1          4          81.0        60.0    1760          16.1    81   \n",
       "382  22.0          6         198.0        95.0    2833          15.5    70   \n",
       "383  23.0          4         120.0        88.0    2957          17.0    75   \n",
       "384  23.7          3          70.0       100.0    2420          12.5    80   \n",
       "385  25.0          4          90.0        71.0    2223          16.5    75   \n",
       "386  15.0          8         350.0       145.0    4440          14.0    75   \n",
       "387  35.0          4          72.0        69.0    1613          18.0    71   \n",
       "388  23.0          4         115.0        95.0    2694          15.0    75   \n",
       "389  24.2          6         146.0       120.0    2930          13.8    81   \n",
       "390  13.0          8         302.0       129.0    3169          12.0    75   \n",
       "391  31.5          4          89.0        71.0    1990          14.9    78   \n",
       "392  13.0          8         302.0       140.0    4294          16.0    72   \n",
       "393  15.0          6         250.0       100.0    3336          17.0    74   \n",
       "394  18.0          3          70.0        90.0    2124          13.5    73   \n",
       "395  36.4          5         121.0        67.0    2950          19.9    80   \n",
       "396  17.5          6         258.0        95.0    3193          17.8    76   \n",
       "397  19.2          8         305.0       145.0    3425          13.2    78   \n",
       "\n",
       "     origin                               name  \n",
       "0         1                   chevrolet camaro  \n",
       "1         1                    ford pinto (sw)  \n",
       "2         1                    dodge colt (sw)  \n",
       "3         3                             subaru  \n",
       "4         3                 mazda glc custom l  \n",
       "5         2                        peugeot 304  \n",
       "6         1                         dodge d200  \n",
       "7         1                       ford mustang  \n",
       "8         1  oldsmobile cutlass ciera (diesel)  \n",
       "9         1              chevrolet monte carlo  \n",
       "10        1     plymouth satellite custom (sw)  \n",
       "11        3                     datsun 210 mpg  \n",
       "12        1                chevrolet vega 2300  \n",
       "13        1                   chevrolet impala  \n",
       "14        2                  volkswagen dasher  \n",
       "15        3                   toyota celica gt  \n",
       "16        3                     toyota corolla  \n",
       "17        1           oldsmobile vista cruiser  \n",
       "18        2    volkswagen rabbit custom diesel  \n",
       "19        3                   maxda glc deluxe  \n",
       "20        2                         saab 99gle  \n",
       "21        1                     ford granada l  \n",
       "22        3                        honda civic  \n",
       "23        1  chevrolet chevelle malibu classic  \n",
       "24        1                  cadillac eldorado  \n",
       "25        1          chevrolet caprice classic  \n",
       "26        1                  plymouth fury iii  \n",
       "27        1                      ford maverick  \n",
       "28        1                 pontiac grand prix  \n",
       "29        1                     dodge diplomat  \n",
       "..      ...                                ...  \n",
       "368       2       volkswagen 1131 deluxe sedan  \n",
       "369       3                     datsun b210 gx  \n",
       "370       1                      amc rebel sst  \n",
       "371       1              buick century limited  \n",
       "372       1                chevrolet vega (sw)  \n",
       "373       2                 fiat strada custom  \n",
       "374       3                   nissan stanza xe  \n",
       "375       1                 chevrolet concours  \n",
       "376       1                   chevrolet impala  \n",
       "377       2               fiat 124 sport coupe  \n",
       "378       2                        volvo 244dl  \n",
       "379       1              dodge monaco brougham  \n",
       "380       2                         opel manta  \n",
       "381       3                   honda civic 1300  \n",
       "382       1                    plymouth duster  \n",
       "383       2                        peugeot 504  \n",
       "384       3                      mazda rx-7 gs  \n",
       "385       2                  volkswagen dasher  \n",
       "386       1                  chevrolet bel air  \n",
       "387       3                        datsun 1200  \n",
       "388       2                         audi 100ls  \n",
       "389       3                  datsun 810 maxima  \n",
       "390       1                    ford mustang ii  \n",
       "391       2                volkswagen scirocco  \n",
       "392       1              ford gran torino (sw)  \n",
       "393       1                     chevrolet nova  \n",
       "394       3                          maxda rx3  \n",
       "395       2                audi 5000s (diesel)  \n",
       "396       1                      amc pacer d/l  \n",
       "397       1       chevrolet monte carlo landau  \n",
       "\n",
       "[398 rows x 9 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "#np.random.seed(42) # Uncomment this line to get the same shuffle each time\n",
    "df = df.reindex(np.random.permutation(df.index))\n",
    "df.reset_index(inplace=True, drop=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first car is: amc ambassador brougham\n",
      "      mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
      "96   13.0          8         360.0       175.0    3821          11.0    73   \n",
      "9    15.0          8         390.0       190.0    3850           8.5    70   \n",
      "66   17.0          8         304.0       150.0    3672          11.5    72   \n",
      "315  24.3          4         151.0        90.0    3003          20.1    80   \n",
      "257  19.4          6         232.0        90.0    3210          17.2    78   \n",
      "\n",
      "     origin                     name  \n",
      "96        1  amc ambassador brougham  \n",
      "9         1       amc ambassador dpl  \n",
      "66        1       amc ambassador sst  \n",
      "315       1              amc concord  \n",
      "257       1              amc concord  \n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "df = df.sort_values(by='name',ascending=True)\n",
    "print(\"The first car is: {}\".format(df['name'].iloc[0]))\n",
    "print(df[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving a Dataframe\n",
    "\n",
    "Many of the assignments in this course will require that you save a dataframe to submit to the instructor.  The following code performs a shuffle and then saves a new copy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "filename_write = os.path.join(path,\"auto-mpg-shuffle.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "df = df.reindex(np.random.permutation(df.index))\n",
    "df.to_csv(filename_write,index=False) # Specify index = false to not write row numbers\n",
    "print(\"Done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropping Fields\n",
    "\n",
    "Some fields are of no value to the neural network and can be dropped.  The following code removes the name column from the MPG dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before drop: Index(['mpg', 'cylinders', 'displacement', 'horsepower', 'weight',\n",
      "       'acceleration', 'year', 'origin', 'name'],\n",
      "      dtype='object')\n",
      "After drop: Index(['mpg', 'cylinders', 'displacement', 'horsepower', 'weight',\n",
      "       'acceleration', 'year', 'origin'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "\n",
    "print(\"Before drop: {}\".format(df.columns))\n",
    "df.drop('name',1,inplace=True)\n",
    "print(\"After drop: {}\".format(df.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculated Fields\n",
    "\n",
    "It is possible to add new fields to the dataframe that are calculated from the other fields.  We can create a new column that gives the weight in kilograms.  The equation to calculate a metric weight, given a weight in pounds is:\n",
    "\n",
    "$ m_{(kg)} = m_{(lb)} \\times 0.45359237 $\n",
    "\n",
    "This can be used with the following Python code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>weight_kg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>1589</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1675</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>1558</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>1557</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>1564</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1969</td>\n",
       "      <td>8</td>\n",
       "      <td>429.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>4341</td>\n",
       "      <td>10.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford galaxie 500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1974</td>\n",
       "      <td>8</td>\n",
       "      <td>454.0</td>\n",
       "      <td>220.0</td>\n",
       "      <td>4354</td>\n",
       "      <td>9.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet impala</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1955</td>\n",
       "      <td>8</td>\n",
       "      <td>440.0</td>\n",
       "      <td>215.0</td>\n",
       "      <td>4312</td>\n",
       "      <td>8.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth fury iii</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>14.0</td>\n",
       "      <td>2007</td>\n",
       "      <td>8</td>\n",
       "      <td>455.0</td>\n",
       "      <td>225.0</td>\n",
       "      <td>4425</td>\n",
       "      <td>10.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac catalina</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1746</td>\n",
       "      <td>8</td>\n",
       "      <td>390.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>3850</td>\n",
       "      <td>8.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc ambassador dpl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1616</td>\n",
       "      <td>8</td>\n",
       "      <td>383.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>3563</td>\n",
       "      <td>10.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge challenger se</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1637</td>\n",
       "      <td>8</td>\n",
       "      <td>340.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>3609</td>\n",
       "      <td>8.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth 'cuda 340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>15.0</td>\n",
       "      <td>1705</td>\n",
       "      <td>8</td>\n",
       "      <td>400.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3761</td>\n",
       "      <td>9.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet monte carlo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14.0</td>\n",
       "      <td>1399</td>\n",
       "      <td>8</td>\n",
       "      <td>455.0</td>\n",
       "      <td>225.0</td>\n",
       "      <td>3086</td>\n",
       "      <td>10.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick estate wagon (sw)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>24.0</td>\n",
       "      <td>1075</td>\n",
       "      <td>4</td>\n",
       "      <td>113.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2372</td>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "      <td>3</td>\n",
       "      <td>toyota corona mark ii</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>22.0</td>\n",
       "      <td>1285</td>\n",
       "      <td>6</td>\n",
       "      <td>198.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2833</td>\n",
       "      <td>15.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth duster</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>18.0</td>\n",
       "      <td>1258</td>\n",
       "      <td>6</td>\n",
       "      <td>199.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>2774</td>\n",
       "      <td>15.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc hornet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>21.0</td>\n",
       "      <td>1173</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>2587</td>\n",
       "      <td>16.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford maverick</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>27.0</td>\n",
       "      <td>966</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>14.5</td>\n",
       "      <td>70</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun pl510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>26.0</td>\n",
       "      <td>832</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1835</td>\n",
       "      <td>20.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen 1131 deluxe sedan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>25.0</td>\n",
       "      <td>1211</td>\n",
       "      <td>4</td>\n",
       "      <td>110.0</td>\n",
       "      <td>87.0</td>\n",
       "      <td>2672</td>\n",
       "      <td>17.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>peugeot 504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>24.0</td>\n",
       "      <td>1102</td>\n",
       "      <td>4</td>\n",
       "      <td>107.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2430</td>\n",
       "      <td>14.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>audi 100 ls</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>25.0</td>\n",
       "      <td>1077</td>\n",
       "      <td>4</td>\n",
       "      <td>104.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2375</td>\n",
       "      <td>17.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>saab 99e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>26.0</td>\n",
       "      <td>1013</td>\n",
       "      <td>4</td>\n",
       "      <td>121.0</td>\n",
       "      <td>113.0</td>\n",
       "      <td>2234</td>\n",
       "      <td>12.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>bmw 2002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>21.0</td>\n",
       "      <td>1201</td>\n",
       "      <td>6</td>\n",
       "      <td>199.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2648</td>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc gremlin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>10.0</td>\n",
       "      <td>2093</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>215.0</td>\n",
       "      <td>4615</td>\n",
       "      <td>14.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford f250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1984</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>4376</td>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy c20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>11.0</td>\n",
       "      <td>1987</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>210.0</td>\n",
       "      <td>4382</td>\n",
       "      <td>13.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge d200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2146</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>193.0</td>\n",
       "      <td>4732</td>\n",
       "      <td>18.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>hi 1200d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>27.0</td>\n",
       "      <td>966</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>14.5</td>\n",
       "      <td>71</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun pl510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>27.0</td>\n",
       "      <td>1197</td>\n",
       "      <td>4</td>\n",
       "      <td>112.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2640</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet cavalier wagon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>369</th>\n",
       "      <td>34.0</td>\n",
       "      <td>1086</td>\n",
       "      <td>4</td>\n",
       "      <td>112.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2395</td>\n",
       "      <td>18.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet cavalier 2-door</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>370</th>\n",
       "      <td>31.0</td>\n",
       "      <td>1168</td>\n",
       "      <td>4</td>\n",
       "      <td>112.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>2575</td>\n",
       "      <td>16.2</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac j2000 se hatchback</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>371</th>\n",
       "      <td>29.0</td>\n",
       "      <td>1145</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2525</td>\n",
       "      <td>16.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge aries se</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>27.0</td>\n",
       "      <td>1240</td>\n",
       "      <td>4</td>\n",
       "      <td>151.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2735</td>\n",
       "      <td>18.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac phoenix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>373</th>\n",
       "      <td>24.0</td>\n",
       "      <td>1299</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>2865</td>\n",
       "      <td>16.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford fairmont futura</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>374</th>\n",
       "      <td>23.0</td>\n",
       "      <td>1376</td>\n",
       "      <td>4</td>\n",
       "      <td>151.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3035</td>\n",
       "      <td>20.5</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>amc concord dl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>36.0</td>\n",
       "      <td>898</td>\n",
       "      <td>4</td>\n",
       "      <td>105.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>1980</td>\n",
       "      <td>15.3</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen rabbit l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>37.0</td>\n",
       "      <td>918</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>2025</td>\n",
       "      <td>18.2</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>mazda glc custom l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>31.0</td>\n",
       "      <td>893</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>1970</td>\n",
       "      <td>17.6</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>mazda glc custom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>38.0</td>\n",
       "      <td>963</td>\n",
       "      <td>4</td>\n",
       "      <td>105.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>2125</td>\n",
       "      <td>14.7</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth horizon miser</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>36.0</td>\n",
       "      <td>963</td>\n",
       "      <td>4</td>\n",
       "      <td>98.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2125</td>\n",
       "      <td>17.3</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>mercury lynx l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>380</th>\n",
       "      <td>36.0</td>\n",
       "      <td>979</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2160</td>\n",
       "      <td>14.5</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>nissan stanza xe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>36.0</td>\n",
       "      <td>1000</td>\n",
       "      <td>4</td>\n",
       "      <td>107.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>2205</td>\n",
       "      <td>14.5</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>honda accord</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>34.0</td>\n",
       "      <td>1018</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2245</td>\n",
       "      <td>16.9</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>toyota corolla</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>383</th>\n",
       "      <td>38.0</td>\n",
       "      <td>891</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1965</td>\n",
       "      <td>15.0</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>honda civic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>32.0</td>\n",
       "      <td>891</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1965</td>\n",
       "      <td>15.7</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>honda civic (auto)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>385</th>\n",
       "      <td>38.0</td>\n",
       "      <td>904</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1995</td>\n",
       "      <td>16.2</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun 310 gx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>25.0</td>\n",
       "      <td>1335</td>\n",
       "      <td>6</td>\n",
       "      <td>181.0</td>\n",
       "      <td>110.0</td>\n",
       "      <td>2945</td>\n",
       "      <td>16.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>buick century limited</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>38.0</td>\n",
       "      <td>1367</td>\n",
       "      <td>6</td>\n",
       "      <td>262.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>3015</td>\n",
       "      <td>17.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>oldsmobile cutlass ciera (diesel)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>26.0</td>\n",
       "      <td>1172</td>\n",
       "      <td>4</td>\n",
       "      <td>156.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>2585</td>\n",
       "      <td>14.5</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chrysler lebaron medallion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>22.0</td>\n",
       "      <td>1285</td>\n",
       "      <td>6</td>\n",
       "      <td>232.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>2835</td>\n",
       "      <td>14.7</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford granada l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>32.0</td>\n",
       "      <td>1208</td>\n",
       "      <td>4</td>\n",
       "      <td>144.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>2665</td>\n",
       "      <td>13.9</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>toyota celica gt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>36.0</td>\n",
       "      <td>1075</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2370</td>\n",
       "      <td>13.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge charger 2.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>392</th>\n",
       "      <td>27.0</td>\n",
       "      <td>1338</td>\n",
       "      <td>4</td>\n",
       "      <td>151.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2950</td>\n",
       "      <td>17.3</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet camaro</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>27.0</td>\n",
       "      <td>1265</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>2790</td>\n",
       "      <td>15.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford mustang gl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>44.0</td>\n",
       "      <td>966</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>24.6</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>vw pickup</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395</th>\n",
       "      <td>32.0</td>\n",
       "      <td>1040</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2295</td>\n",
       "      <td>11.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge rampage</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>28.0</td>\n",
       "      <td>1190</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>2625</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford ranger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>31.0</td>\n",
       "      <td>1233</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>2720</td>\n",
       "      <td>19.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy s-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>398 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  weight_kg  cylinders  displacement  horsepower  weight  \\\n",
       "0    18.0       1589          8         307.0       130.0    3504   \n",
       "1    15.0       1675          8         350.0       165.0    3693   \n",
       "2    18.0       1558          8         318.0       150.0    3436   \n",
       "3    16.0       1557          8         304.0       150.0    3433   \n",
       "4    17.0       1564          8         302.0       140.0    3449   \n",
       "5    15.0       1969          8         429.0       198.0    4341   \n",
       "6    14.0       1974          8         454.0       220.0    4354   \n",
       "7    14.0       1955          8         440.0       215.0    4312   \n",
       "8    14.0       2007          8         455.0       225.0    4425   \n",
       "9    15.0       1746          8         390.0       190.0    3850   \n",
       "10   15.0       1616          8         383.0       170.0    3563   \n",
       "11   14.0       1637          8         340.0       160.0    3609   \n",
       "12   15.0       1705          8         400.0       150.0    3761   \n",
       "13   14.0       1399          8         455.0       225.0    3086   \n",
       "14   24.0       1075          4         113.0        95.0    2372   \n",
       "15   22.0       1285          6         198.0        95.0    2833   \n",
       "16   18.0       1258          6         199.0        97.0    2774   \n",
       "17   21.0       1173          6         200.0        85.0    2587   \n",
       "18   27.0        966          4          97.0        88.0    2130   \n",
       "19   26.0        832          4          97.0        46.0    1835   \n",
       "20   25.0       1211          4         110.0        87.0    2672   \n",
       "21   24.0       1102          4         107.0        90.0    2430   \n",
       "22   25.0       1077          4         104.0        95.0    2375   \n",
       "23   26.0       1013          4         121.0       113.0    2234   \n",
       "24   21.0       1201          6         199.0        90.0    2648   \n",
       "25   10.0       2093          8         360.0       215.0    4615   \n",
       "26   10.0       1984          8         307.0       200.0    4376   \n",
       "27   11.0       1987          8         318.0       210.0    4382   \n",
       "28    9.0       2146          8         304.0       193.0    4732   \n",
       "29   27.0        966          4          97.0        88.0    2130   \n",
       "..    ...        ...        ...           ...         ...     ...   \n",
       "368  27.0       1197          4         112.0        88.0    2640   \n",
       "369  34.0       1086          4         112.0        88.0    2395   \n",
       "370  31.0       1168          4         112.0        85.0    2575   \n",
       "371  29.0       1145          4         135.0        84.0    2525   \n",
       "372  27.0       1240          4         151.0        90.0    2735   \n",
       "373  24.0       1299          4         140.0        92.0    2865   \n",
       "374  23.0       1376          4         151.0         NaN    3035   \n",
       "375  36.0        898          4         105.0        74.0    1980   \n",
       "376  37.0        918          4          91.0        68.0    2025   \n",
       "377  31.0        893          4          91.0        68.0    1970   \n",
       "378  38.0        963          4         105.0        63.0    2125   \n",
       "379  36.0        963          4          98.0        70.0    2125   \n",
       "380  36.0        979          4         120.0        88.0    2160   \n",
       "381  36.0       1000          4         107.0        75.0    2205   \n",
       "382  34.0       1018          4         108.0        70.0    2245   \n",
       "383  38.0        891          4          91.0        67.0    1965   \n",
       "384  32.0        891          4          91.0        67.0    1965   \n",
       "385  38.0        904          4          91.0        67.0    1995   \n",
       "386  25.0       1335          6         181.0       110.0    2945   \n",
       "387  38.0       1367          6         262.0        85.0    3015   \n",
       "388  26.0       1172          4         156.0        92.0    2585   \n",
       "389  22.0       1285          6         232.0       112.0    2835   \n",
       "390  32.0       1208          4         144.0        96.0    2665   \n",
       "391  36.0       1075          4         135.0        84.0    2370   \n",
       "392  27.0       1338          4         151.0        90.0    2950   \n",
       "393  27.0       1265          4         140.0        86.0    2790   \n",
       "394  44.0        966          4          97.0        52.0    2130   \n",
       "395  32.0       1040          4         135.0        84.0    2295   \n",
       "396  28.0       1190          4         120.0        79.0    2625   \n",
       "397  31.0       1233          4         119.0        82.0    2720   \n",
       "\n",
       "     acceleration  year  origin                               name  \n",
       "0            12.0    70       1          chevrolet chevelle malibu  \n",
       "1            11.5    70       1                  buick skylark 320  \n",
       "2            11.0    70       1                 plymouth satellite  \n",
       "3            12.0    70       1                      amc rebel sst  \n",
       "4            10.5    70       1                        ford torino  \n",
       "5            10.0    70       1                   ford galaxie 500  \n",
       "6             9.0    70       1                   chevrolet impala  \n",
       "7             8.5    70       1                  plymouth fury iii  \n",
       "8            10.0    70       1                   pontiac catalina  \n",
       "9             8.5    70       1                 amc ambassador dpl  \n",
       "10           10.0    70       1                dodge challenger se  \n",
       "11            8.0    70       1                 plymouth 'cuda 340  \n",
       "12            9.5    70       1              chevrolet monte carlo  \n",
       "13           10.0    70       1            buick estate wagon (sw)  \n",
       "14           15.0    70       3              toyota corona mark ii  \n",
       "15           15.5    70       1                    plymouth duster  \n",
       "16           15.5    70       1                         amc hornet  \n",
       "17           16.0    70       1                      ford maverick  \n",
       "18           14.5    70       3                       datsun pl510  \n",
       "19           20.5    70       2       volkswagen 1131 deluxe sedan  \n",
       "20           17.5    70       2                        peugeot 504  \n",
       "21           14.5    70       2                        audi 100 ls  \n",
       "22           17.5    70       2                           saab 99e  \n",
       "23           12.5    70       2                           bmw 2002  \n",
       "24           15.0    70       1                        amc gremlin  \n",
       "25           14.0    70       1                          ford f250  \n",
       "26           15.0    70       1                          chevy c20  \n",
       "27           13.5    70       1                         dodge d200  \n",
       "28           18.5    70       1                           hi 1200d  \n",
       "29           14.5    71       3                       datsun pl510  \n",
       "..            ...   ...     ...                                ...  \n",
       "368          18.6    82       1           chevrolet cavalier wagon  \n",
       "369          18.0    82       1          chevrolet cavalier 2-door  \n",
       "370          16.2    82       1         pontiac j2000 se hatchback  \n",
       "371          16.0    82       1                     dodge aries se  \n",
       "372          18.0    82       1                    pontiac phoenix  \n",
       "373          16.4    82       1               ford fairmont futura  \n",
       "374          20.5    82       1                     amc concord dl  \n",
       "375          15.3    82       2                volkswagen rabbit l  \n",
       "376          18.2    82       3                 mazda glc custom l  \n",
       "377          17.6    82       3                   mazda glc custom  \n",
       "378          14.7    82       1             plymouth horizon miser  \n",
       "379          17.3    82       1                     mercury lynx l  \n",
       "380          14.5    82       3                   nissan stanza xe  \n",
       "381          14.5    82       3                       honda accord  \n",
       "382          16.9    82       3                     toyota corolla  \n",
       "383          15.0    82       3                        honda civic  \n",
       "384          15.7    82       3                 honda civic (auto)  \n",
       "385          16.2    82       3                      datsun 310 gx  \n",
       "386          16.4    82       1              buick century limited  \n",
       "387          17.0    82       1  oldsmobile cutlass ciera (diesel)  \n",
       "388          14.5    82       1         chrysler lebaron medallion  \n",
       "389          14.7    82       1                     ford granada l  \n",
       "390          13.9    82       3                   toyota celica gt  \n",
       "391          13.0    82       1                  dodge charger 2.2  \n",
       "392          17.3    82       1                   chevrolet camaro  \n",
       "393          15.6    82       1                    ford mustang gl  \n",
       "394          24.6    82       2                          vw pickup  \n",
       "395          11.6    82       1                      dodge rampage  \n",
       "396          18.6    82       1                        ford ranger  \n",
       "397          19.4    82       1                         chevy s-10  \n",
       "\n",
       "[398 rows x 10 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "df.insert(1,'weight_kg',(df['weight']*0.45359237).astype(int))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Field Transformation & Preprocessing\n",
    "\n",
    "The data fed into a machine learning model rarely bares much similarity to the data that the data scientist originally received. One common transformation is to normalize the inputs.  A normalization allows numbers to be put in a standard form so that two values can easily be compared.  Consider if a friend told you that he received a $10 discount.  Is this a good deal?  Maybe.  But the value is not normalized.  If your friend purchased a car, then the discount is not that good.  If your friend purchased dinner, this is a very good discount!\n",
    "\n",
    "Percentages are a very common form of normalization.  If your friend tells you they got 10% off, we know that this is a better discount than 5%.  It does not matter how much the purchase price was.  One very common machine learning normalization is the Z-Score:\n",
    "\n",
    "$z = {x- \\mu \\over \\sigma} $\n",
    "\n",
    "To calculate the Z-Score you need to also calculate the mean($\\mu$) and the standard deviation ($\\sigma$).  The mean is calculated as follows:\n",
    "\n",
    "$\\mu = \\bar{x} = \\frac{x_1+x_2+\\cdots +x_n}{n}$\n",
    "\n",
    "The standard deviation is calculated as follows:\n",
    "\n",
    "$\\sigma = \\sqrt{\\frac{1}{N} \\sum_{i=1}^N (x_i - \\mu)^2}, {\\rm \\ \\ where\\ \\ } \\mu = \\frac{1}{N} \\sum_{i=1}^N x_i$\n",
    "\n",
    "The following Python code replaces the mpg with a z-score.  Cars with average MPG will be near zero, above zero is above average, and below zero is below average.  Z-Scores above/below -3/3 are very rare, these are outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.706439</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.090751</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.706439</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.962647</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.834543</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-1.090751</td>\n",
       "      <td>8</td>\n",
       "      <td>429.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>4341</td>\n",
       "      <td>10.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford galaxie 500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-1.218855</td>\n",
       "      <td>8</td>\n",
       "      <td>454.0</td>\n",
       "      <td>220.0</td>\n",
       "      <td>4354</td>\n",
       "      <td>9.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet impala</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-1.218855</td>\n",
       "      <td>8</td>\n",
       "      <td>440.0</td>\n",
       "      <td>215.0</td>\n",
       "      <td>4312</td>\n",
       "      <td>8.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth fury iii</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.218855</td>\n",
       "      <td>8</td>\n",
       "      <td>455.0</td>\n",
       "      <td>225.0</td>\n",
       "      <td>4425</td>\n",
       "      <td>10.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac catalina</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-1.090751</td>\n",
       "      <td>8</td>\n",
       "      <td>390.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>3850</td>\n",
       "      <td>8.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc ambassador dpl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-1.090751</td>\n",
       "      <td>8</td>\n",
       "      <td>383.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>3563</td>\n",
       "      <td>10.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge challenger se</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-1.218855</td>\n",
       "      <td>8</td>\n",
       "      <td>340.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>3609</td>\n",
       "      <td>8.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth 'cuda 340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.090751</td>\n",
       "      <td>8</td>\n",
       "      <td>400.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3761</td>\n",
       "      <td>9.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet monte carlo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>-1.218855</td>\n",
       "      <td>8</td>\n",
       "      <td>455.0</td>\n",
       "      <td>225.0</td>\n",
       "      <td>3086</td>\n",
       "      <td>10.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick estate wagon (sw)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.062185</td>\n",
       "      <td>4</td>\n",
       "      <td>113.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2372</td>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "      <td>3</td>\n",
       "      <td>toyota corona mark ii</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.194023</td>\n",
       "      <td>6</td>\n",
       "      <td>198.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2833</td>\n",
       "      <td>15.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth duster</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>-0.706439</td>\n",
       "      <td>6</td>\n",
       "      <td>199.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>2774</td>\n",
       "      <td>15.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc hornet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.322127</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>2587</td>\n",
       "      <td>16.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford maverick</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.446497</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>14.5</td>\n",
       "      <td>70</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun pl510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.318393</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1835</td>\n",
       "      <td>20.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen 1131 deluxe sedan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.190289</td>\n",
       "      <td>4</td>\n",
       "      <td>110.0</td>\n",
       "      <td>87.0</td>\n",
       "      <td>2672</td>\n",
       "      <td>17.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>peugeot 504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.062185</td>\n",
       "      <td>4</td>\n",
       "      <td>107.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2430</td>\n",
       "      <td>14.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>audi 100 ls</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.190289</td>\n",
       "      <td>4</td>\n",
       "      <td>104.0</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2375</td>\n",
       "      <td>17.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>saab 99e</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.318393</td>\n",
       "      <td>4</td>\n",
       "      <td>121.0</td>\n",
       "      <td>113.0</td>\n",
       "      <td>2234</td>\n",
       "      <td>12.5</td>\n",
       "      <td>70</td>\n",
       "      <td>2</td>\n",
       "      <td>bmw 2002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>-0.322127</td>\n",
       "      <td>6</td>\n",
       "      <td>199.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2648</td>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc gremlin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>-1.731270</td>\n",
       "      <td>8</td>\n",
       "      <td>360.0</td>\n",
       "      <td>215.0</td>\n",
       "      <td>4615</td>\n",
       "      <td>14.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford f250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>-1.731270</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>4376</td>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy c20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>-1.603167</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>210.0</td>\n",
       "      <td>4382</td>\n",
       "      <td>13.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge d200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>-1.859374</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>193.0</td>\n",
       "      <td>4732</td>\n",
       "      <td>18.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>hi 1200d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.446497</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>14.5</td>\n",
       "      <td>71</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun pl510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>0.446497</td>\n",
       "      <td>4</td>\n",
       "      <td>112.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2640</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet cavalier wagon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>369</th>\n",
       "      <td>1.343225</td>\n",
       "      <td>4</td>\n",
       "      <td>112.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2395</td>\n",
       "      <td>18.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet cavalier 2-door</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>370</th>\n",
       "      <td>0.958913</td>\n",
       "      <td>4</td>\n",
       "      <td>112.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>2575</td>\n",
       "      <td>16.2</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac j2000 se hatchback</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>371</th>\n",
       "      <td>0.702705</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2525</td>\n",
       "      <td>16.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge aries se</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>0.446497</td>\n",
       "      <td>4</td>\n",
       "      <td>151.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2735</td>\n",
       "      <td>18.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac phoenix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>373</th>\n",
       "      <td>0.062185</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>2865</td>\n",
       "      <td>16.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford fairmont futura</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>374</th>\n",
       "      <td>-0.065919</td>\n",
       "      <td>4</td>\n",
       "      <td>151.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3035</td>\n",
       "      <td>20.5</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>amc concord dl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>1.599433</td>\n",
       "      <td>4</td>\n",
       "      <td>105.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>1980</td>\n",
       "      <td>15.3</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>volkswagen rabbit l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>1.727537</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>2025</td>\n",
       "      <td>18.2</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>mazda glc custom l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>0.958913</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>1970</td>\n",
       "      <td>17.6</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>mazda glc custom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>1.855641</td>\n",
       "      <td>4</td>\n",
       "      <td>105.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>2125</td>\n",
       "      <td>14.7</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth horizon miser</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>1.599433</td>\n",
       "      <td>4</td>\n",
       "      <td>98.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2125</td>\n",
       "      <td>17.3</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>mercury lynx l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>380</th>\n",
       "      <td>1.599433</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>2160</td>\n",
       "      <td>14.5</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>nissan stanza xe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>1.599433</td>\n",
       "      <td>4</td>\n",
       "      <td>107.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>2205</td>\n",
       "      <td>14.5</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>honda accord</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>1.343225</td>\n",
       "      <td>4</td>\n",
       "      <td>108.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2245</td>\n",
       "      <td>16.9</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>toyota corolla</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>383</th>\n",
       "      <td>1.855641</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1965</td>\n",
       "      <td>15.0</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>honda civic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>1.087017</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1965</td>\n",
       "      <td>15.7</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>honda civic (auto)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>385</th>\n",
       "      <td>1.855641</td>\n",
       "      <td>4</td>\n",
       "      <td>91.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>1995</td>\n",
       "      <td>16.2</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>datsun 310 gx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>0.190289</td>\n",
       "      <td>6</td>\n",
       "      <td>181.0</td>\n",
       "      <td>110.0</td>\n",
       "      <td>2945</td>\n",
       "      <td>16.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>buick century limited</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>1.855641</td>\n",
       "      <td>6</td>\n",
       "      <td>262.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>3015</td>\n",
       "      <td>17.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>oldsmobile cutlass ciera (diesel)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>0.318393</td>\n",
       "      <td>4</td>\n",
       "      <td>156.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>2585</td>\n",
       "      <td>14.5</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chrysler lebaron medallion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>-0.194023</td>\n",
       "      <td>6</td>\n",
       "      <td>232.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>2835</td>\n",
       "      <td>14.7</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford granada l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>1.087017</td>\n",
       "      <td>4</td>\n",
       "      <td>144.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>2665</td>\n",
       "      <td>13.9</td>\n",
       "      <td>82</td>\n",
       "      <td>3</td>\n",
       "      <td>toyota celica gt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>1.599433</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2370</td>\n",
       "      <td>13.0</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge charger 2.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>392</th>\n",
       "      <td>0.446497</td>\n",
       "      <td>4</td>\n",
       "      <td>151.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2950</td>\n",
       "      <td>17.3</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet camaro</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>0.446497</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>2790</td>\n",
       "      <td>15.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford mustang gl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>2.624265</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>24.6</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>vw pickup</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395</th>\n",
       "      <td>1.087017</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>2295</td>\n",
       "      <td>11.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge rampage</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>0.574601</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>2625</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford ranger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>0.958913</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>2720</td>\n",
       "      <td>19.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy s-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>398 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          mpg  cylinders  displacement  horsepower  weight  acceleration  \\\n",
       "0   -0.706439          8         307.0       130.0    3504          12.0   \n",
       "1   -1.090751          8         350.0       165.0    3693          11.5   \n",
       "2   -0.706439          8         318.0       150.0    3436          11.0   \n",
       "3   -0.962647          8         304.0       150.0    3433          12.0   \n",
       "4   -0.834543          8         302.0       140.0    3449          10.5   \n",
       "5   -1.090751          8         429.0       198.0    4341          10.0   \n",
       "6   -1.218855          8         454.0       220.0    4354           9.0   \n",
       "7   -1.218855          8         440.0       215.0    4312           8.5   \n",
       "8   -1.218855          8         455.0       225.0    4425          10.0   \n",
       "9   -1.090751          8         390.0       190.0    3850           8.5   \n",
       "10  -1.090751          8         383.0       170.0    3563          10.0   \n",
       "11  -1.218855          8         340.0       160.0    3609           8.0   \n",
       "12  -1.090751          8         400.0       150.0    3761           9.5   \n",
       "13  -1.218855          8         455.0       225.0    3086          10.0   \n",
       "14   0.062185          4         113.0        95.0    2372          15.0   \n",
       "15  -0.194023          6         198.0        95.0    2833          15.5   \n",
       "16  -0.706439          6         199.0        97.0    2774          15.5   \n",
       "17  -0.322127          6         200.0        85.0    2587          16.0   \n",
       "18   0.446497          4          97.0        88.0    2130          14.5   \n",
       "19   0.318393          4          97.0        46.0    1835          20.5   \n",
       "20   0.190289          4         110.0        87.0    2672          17.5   \n",
       "21   0.062185          4         107.0        90.0    2430          14.5   \n",
       "22   0.190289          4         104.0        95.0    2375          17.5   \n",
       "23   0.318393          4         121.0       113.0    2234          12.5   \n",
       "24  -0.322127          6         199.0        90.0    2648          15.0   \n",
       "25  -1.731270          8         360.0       215.0    4615          14.0   \n",
       "26  -1.731270          8         307.0       200.0    4376          15.0   \n",
       "27  -1.603167          8         318.0       210.0    4382          13.5   \n",
       "28  -1.859374          8         304.0       193.0    4732          18.5   \n",
       "29   0.446497          4          97.0        88.0    2130          14.5   \n",
       "..        ...        ...           ...         ...     ...           ...   \n",
       "368  0.446497          4         112.0        88.0    2640          18.6   \n",
       "369  1.343225          4         112.0        88.0    2395          18.0   \n",
       "370  0.958913          4         112.0        85.0    2575          16.2   \n",
       "371  0.702705          4         135.0        84.0    2525          16.0   \n",
       "372  0.446497          4         151.0        90.0    2735          18.0   \n",
       "373  0.062185          4         140.0        92.0    2865          16.4   \n",
       "374 -0.065919          4         151.0         NaN    3035          20.5   \n",
       "375  1.599433          4         105.0        74.0    1980          15.3   \n",
       "376  1.727537          4          91.0        68.0    2025          18.2   \n",
       "377  0.958913          4          91.0        68.0    1970          17.6   \n",
       "378  1.855641          4         105.0        63.0    2125          14.7   \n",
       "379  1.599433          4          98.0        70.0    2125          17.3   \n",
       "380  1.599433          4         120.0        88.0    2160          14.5   \n",
       "381  1.599433          4         107.0        75.0    2205          14.5   \n",
       "382  1.343225          4         108.0        70.0    2245          16.9   \n",
       "383  1.855641          4          91.0        67.0    1965          15.0   \n",
       "384  1.087017          4          91.0        67.0    1965          15.7   \n",
       "385  1.855641          4          91.0        67.0    1995          16.2   \n",
       "386  0.190289          6         181.0       110.0    2945          16.4   \n",
       "387  1.855641          6         262.0        85.0    3015          17.0   \n",
       "388  0.318393          4         156.0        92.0    2585          14.5   \n",
       "389 -0.194023          6         232.0       112.0    2835          14.7   \n",
       "390  1.087017          4         144.0        96.0    2665          13.9   \n",
       "391  1.599433          4         135.0        84.0    2370          13.0   \n",
       "392  0.446497          4         151.0        90.0    2950          17.3   \n",
       "393  0.446497          4         140.0        86.0    2790          15.6   \n",
       "394  2.624265          4          97.0        52.0    2130          24.6   \n",
       "395  1.087017          4         135.0        84.0    2295          11.6   \n",
       "396  0.574601          4         120.0        79.0    2625          18.6   \n",
       "397  0.958913          4         119.0        82.0    2720          19.4   \n",
       "\n",
       "     year  origin                               name  \n",
       "0      70       1          chevrolet chevelle malibu  \n",
       "1      70       1                  buick skylark 320  \n",
       "2      70       1                 plymouth satellite  \n",
       "3      70       1                      amc rebel sst  \n",
       "4      70       1                        ford torino  \n",
       "5      70       1                   ford galaxie 500  \n",
       "6      70       1                   chevrolet impala  \n",
       "7      70       1                  plymouth fury iii  \n",
       "8      70       1                   pontiac catalina  \n",
       "9      70       1                 amc ambassador dpl  \n",
       "10     70       1                dodge challenger se  \n",
       "11     70       1                 plymouth 'cuda 340  \n",
       "12     70       1              chevrolet monte carlo  \n",
       "13     70       1            buick estate wagon (sw)  \n",
       "14     70       3              toyota corona mark ii  \n",
       "15     70       1                    plymouth duster  \n",
       "16     70       1                         amc hornet  \n",
       "17     70       1                      ford maverick  \n",
       "18     70       3                       datsun pl510  \n",
       "19     70       2       volkswagen 1131 deluxe sedan  \n",
       "20     70       2                        peugeot 504  \n",
       "21     70       2                        audi 100 ls  \n",
       "22     70       2                           saab 99e  \n",
       "23     70       2                           bmw 2002  \n",
       "24     70       1                        amc gremlin  \n",
       "25     70       1                          ford f250  \n",
       "26     70       1                          chevy c20  \n",
       "27     70       1                         dodge d200  \n",
       "28     70       1                           hi 1200d  \n",
       "29     71       3                       datsun pl510  \n",
       "..    ...     ...                                ...  \n",
       "368    82       1           chevrolet cavalier wagon  \n",
       "369    82       1          chevrolet cavalier 2-door  \n",
       "370    82       1         pontiac j2000 se hatchback  \n",
       "371    82       1                     dodge aries se  \n",
       "372    82       1                    pontiac phoenix  \n",
       "373    82       1               ford fairmont futura  \n",
       "374    82       1                     amc concord dl  \n",
       "375    82       2                volkswagen rabbit l  \n",
       "376    82       3                 mazda glc custom l  \n",
       "377    82       3                   mazda glc custom  \n",
       "378    82       1             plymouth horizon miser  \n",
       "379    82       1                     mercury lynx l  \n",
       "380    82       3                   nissan stanza xe  \n",
       "381    82       3                       honda accord  \n",
       "382    82       3                     toyota corolla  \n",
       "383    82       3                        honda civic  \n",
       "384    82       3                 honda civic (auto)  \n",
       "385    82       3                      datsun 310 gx  \n",
       "386    82       1              buick century limited  \n",
       "387    82       1  oldsmobile cutlass ciera (diesel)  \n",
       "388    82       1         chrysler lebaron medallion  \n",
       "389    82       1                     ford granada l  \n",
       "390    82       3                   toyota celica gt  \n",
       "391    82       1                  dodge charger 2.2  \n",
       "392    82       1                   chevrolet camaro  \n",
       "393    82       1                    ford mustang gl  \n",
       "394    82       2                          vw pickup  \n",
       "395    82       1                      dodge rampage  \n",
       "396    82       1                        ford ranger  \n",
       "397    82       1                         chevy s-10  \n",
       "\n",
       "[398 rows x 9 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "df['mpg'] = zscore(df['mpg'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing Values\n",
    "\n",
    "Missing values are a reality of machine learning.  Ideally every row of data will have values for all columns.  However, this is rarely the case.  Most of the values are present in the MPG database.  However, there are missing values in the horsepower column.  A common practice is to replace missing values with the median value for that column.  The median is calculated as described [here](https://www.mathsisfun.com/median.html).  The following code replaces any NA values in horsepower with the median:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "horsepower has na? False\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "med = df['horsepower'].median()\n",
    "df['horsepower'] = df['horsepower'].fillna(med)\n",
    "# df = df.dropna() # you can also simply drop NA values\n",
    "print(\"horsepower has na? {}\".format(pd.isnull(df['horsepower']).values.any()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concatenating Rows and Columns\n",
    "Rows and columns can be concatenated together to form new data frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>horsepower</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "      <td>130.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>buick skylark 320</td>\n",
       "      <td>165.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>plymouth satellite</td>\n",
       "      <td>150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>amc rebel sst</td>\n",
       "      <td>150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ford torino</td>\n",
       "      <td>140.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ford galaxie 500</td>\n",
       "      <td>198.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>chevrolet impala</td>\n",
       "      <td>220.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>plymouth fury iii</td>\n",
       "      <td>215.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>pontiac catalina</td>\n",
       "      <td>225.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>amc ambassador dpl</td>\n",
       "      <td>190.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>dodge challenger se</td>\n",
       "      <td>170.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>plymouth 'cuda 340</td>\n",
       "      <td>160.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>chevrolet monte carlo</td>\n",
       "      <td>150.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>buick estate wagon (sw)</td>\n",
       "      <td>225.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>toyota corona mark ii</td>\n",
       "      <td>95.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>plymouth duster</td>\n",
       "      <td>95.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>amc hornet</td>\n",
       "      <td>97.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>ford maverick</td>\n",
       "      <td>85.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>datsun pl510</td>\n",
       "      <td>88.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>volkswagen 1131 deluxe sedan</td>\n",
       "      <td>46.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>peugeot 504</td>\n",
       "      <td>87.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>audi 100 ls</td>\n",
       "      <td>90.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>saab 99e</td>\n",
       "      <td>95.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>bmw 2002</td>\n",
       "      <td>113.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>amc gremlin</td>\n",
       "      <td>90.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>ford f250</td>\n",
       "      <td>215.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>chevy c20</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>dodge d200</td>\n",
       "      <td>210.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>hi 1200d</td>\n",
       "      <td>193.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>datsun pl510</td>\n",
       "      <td>88.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>368</th>\n",
       "      <td>chevrolet cavalier wagon</td>\n",
       "      <td>88.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>369</th>\n",
       "      <td>chevrolet cavalier 2-door</td>\n",
       "      <td>88.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>370</th>\n",
       "      <td>pontiac j2000 se hatchback</td>\n",
       "      <td>85.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>371</th>\n",
       "      <td>dodge aries se</td>\n",
       "      <td>84.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>372</th>\n",
       "      <td>pontiac phoenix</td>\n",
       "      <td>90.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>373</th>\n",
       "      <td>ford fairmont futura</td>\n",
       "      <td>92.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>374</th>\n",
       "      <td>amc concord dl</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>volkswagen rabbit l</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>376</th>\n",
       "      <td>mazda glc custom l</td>\n",
       "      <td>68.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>377</th>\n",
       "      <td>mazda glc custom</td>\n",
       "      <td>68.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>plymouth horizon miser</td>\n",
       "      <td>63.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>mercury lynx l</td>\n",
       "      <td>70.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>380</th>\n",
       "      <td>nissan stanza xe</td>\n",
       "      <td>88.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>honda accord</td>\n",
       "      <td>75.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>382</th>\n",
       "      <td>toyota corolla</td>\n",
       "      <td>70.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>383</th>\n",
       "      <td>honda civic</td>\n",
       "      <td>67.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>honda civic (auto)</td>\n",
       "      <td>67.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>385</th>\n",
       "      <td>datsun 310 gx</td>\n",
       "      <td>67.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>buick century limited</td>\n",
       "      <td>110.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>oldsmobile cutlass ciera (diesel)</td>\n",
       "      <td>85.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>chrysler lebaron medallion</td>\n",
       "      <td>92.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>ford granada l</td>\n",
       "      <td>112.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>toyota celica gt</td>\n",
       "      <td>96.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>dodge charger 2.2</td>\n",
       "      <td>84.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>392</th>\n",
       "      <td>chevrolet camaro</td>\n",
       "      <td>90.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>ford mustang gl</td>\n",
       "      <td>86.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>vw pickup</td>\n",
       "      <td>52.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395</th>\n",
       "      <td>dodge rampage</td>\n",
       "      <td>84.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>ford ranger</td>\n",
       "      <td>79.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>chevy s-10</td>\n",
       "      <td>82.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>398 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  name  horsepower\n",
       "0            chevrolet chevelle malibu       130.0\n",
       "1                    buick skylark 320       165.0\n",
       "2                   plymouth satellite       150.0\n",
       "3                        amc rebel sst       150.0\n",
       "4                          ford torino       140.0\n",
       "5                     ford galaxie 500       198.0\n",
       "6                     chevrolet impala       220.0\n",
       "7                    plymouth fury iii       215.0\n",
       "8                     pontiac catalina       225.0\n",
       "9                   amc ambassador dpl       190.0\n",
       "10                 dodge challenger se       170.0\n",
       "11                  plymouth 'cuda 340       160.0\n",
       "12               chevrolet monte carlo       150.0\n",
       "13             buick estate wagon (sw)       225.0\n",
       "14               toyota corona mark ii        95.0\n",
       "15                     plymouth duster        95.0\n",
       "16                          amc hornet        97.0\n",
       "17                       ford maverick        85.0\n",
       "18                        datsun pl510        88.0\n",
       "19        volkswagen 1131 deluxe sedan        46.0\n",
       "20                         peugeot 504        87.0\n",
       "21                         audi 100 ls        90.0\n",
       "22                            saab 99e        95.0\n",
       "23                            bmw 2002       113.0\n",
       "24                         amc gremlin        90.0\n",
       "25                           ford f250       215.0\n",
       "26                           chevy c20       200.0\n",
       "27                          dodge d200       210.0\n",
       "28                            hi 1200d       193.0\n",
       "29                        datsun pl510        88.0\n",
       "..                                 ...         ...\n",
       "368           chevrolet cavalier wagon        88.0\n",
       "369          chevrolet cavalier 2-door        88.0\n",
       "370         pontiac j2000 se hatchback        85.0\n",
       "371                     dodge aries se        84.0\n",
       "372                    pontiac phoenix        90.0\n",
       "373               ford fairmont futura        92.0\n",
       "374                     amc concord dl         NaN\n",
       "375                volkswagen rabbit l        74.0\n",
       "376                 mazda glc custom l        68.0\n",
       "377                   mazda glc custom        68.0\n",
       "378             plymouth horizon miser        63.0\n",
       "379                     mercury lynx l        70.0\n",
       "380                   nissan stanza xe        88.0\n",
       "381                       honda accord        75.0\n",
       "382                     toyota corolla        70.0\n",
       "383                        honda civic        67.0\n",
       "384                 honda civic (auto)        67.0\n",
       "385                      datsun 310 gx        67.0\n",
       "386              buick century limited       110.0\n",
       "387  oldsmobile cutlass ciera (diesel)        85.0\n",
       "388         chrysler lebaron medallion        92.0\n",
       "389                     ford granada l       112.0\n",
       "390                   toyota celica gt        96.0\n",
       "391                  dodge charger 2.2        84.0\n",
       "392                   chevrolet camaro        90.0\n",
       "393                    ford mustang gl        86.0\n",
       "394                          vw pickup        52.0\n",
       "395                      dodge rampage        84.0\n",
       "396                        ford ranger        79.0\n",
       "397                         chevy s-10        82.0\n",
       "\n",
       "[398 rows x 2 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new dataframe from name and horsepower\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "col_horsepower = df['horsepower']\n",
    "col_name = df['name']\n",
    "result = pd.concat([col_name,col_horsepower],axis=1)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         chevrolet chevelle malibu\n",
       "1                 buick skylark 320\n",
       "2                plymouth satellite\n",
       "3                     amc rebel sst\n",
       "4                       ford torino\n",
       "5                  ford galaxie 500\n",
       "6                  chevrolet impala\n",
       "7                 plymouth fury iii\n",
       "8                  pontiac catalina\n",
       "9                amc ambassador dpl\n",
       "10              dodge challenger se\n",
       "11               plymouth 'cuda 340\n",
       "12            chevrolet monte carlo\n",
       "13          buick estate wagon (sw)\n",
       "14            toyota corona mark ii\n",
       "15                  plymouth duster\n",
       "16                       amc hornet\n",
       "17                    ford maverick\n",
       "18                     datsun pl510\n",
       "19     volkswagen 1131 deluxe sedan\n",
       "20                      peugeot 504\n",
       "21                      audi 100 ls\n",
       "22                         saab 99e\n",
       "23                         bmw 2002\n",
       "24                      amc gremlin\n",
       "25                        ford f250\n",
       "26                        chevy c20\n",
       "27                       dodge d200\n",
       "28                         hi 1200d\n",
       "29                     datsun pl510\n",
       "                   ...             \n",
       "368                              88\n",
       "369                              88\n",
       "370                              85\n",
       "371                              84\n",
       "372                              90\n",
       "373                              92\n",
       "374                             NaN\n",
       "375                              74\n",
       "376                              68\n",
       "377                              68\n",
       "378                              63\n",
       "379                              70\n",
       "380                              88\n",
       "381                              75\n",
       "382                              70\n",
       "383                              67\n",
       "384                              67\n",
       "385                              67\n",
       "386                             110\n",
       "387                              85\n",
       "388                              92\n",
       "389                             112\n",
       "390                              96\n",
       "391                              84\n",
       "392                              90\n",
       "393                              86\n",
       "394                              52\n",
       "395                              84\n",
       "396                              79\n",
       "397                              82\n",
       "Length: 796, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new dataframe from name and horsepower, but this time by row\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "col_horsepower = df['horsepower']\n",
    "col_name = df['name']\n",
    "result = pd.concat([col_name,col_horsepower])\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Training and Validation\n",
    "\n",
    "It is very important that we evaluate a machine learning model based on its ability to predict data that it has never seen before.  Because of this we often divide the training data into a validation and training set.  The machine learning model will learn from the training data, but ultimately be evaluated based on the validation data.\n",
    "\n",
    "* **Training Data** - **In Sample Data** - The data that the machine learning model was fit to/created from. \n",
    "* **Validation Data** - **Out of Sample Data** - The data that the machine learning model is evaluated upon after it is fit to the training data.\n",
    "\n",
    "There are two predominant means of dealing with training and validation data:\n",
    "\n",
    "* **Training/Validation Split** - The data are split according to some ratio between a training and validation (hold-out) set.  Common ratios are 80% training and 20% validation.\n",
    "* **K-Fold Cross Validation** - The data are split into a number of folds and models.  Because a number of models equal to the folds is created out-of-sample predictions can be generated for the entire dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training/Validation Split\n",
    "\n",
    "The code below performs a split of the MPG data into a training and validation set.  The training set uses 80% of the data and the validation set uses 20%.\n",
    "\n",
    "The following image shows how a model is trained on 80% of the data and then validated against the remaining 20%.\n",
    "\n",
    "![Training and Validation](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_train_val.png \"Training and Validation\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training DF: 322\n",
      "Validation DF: 76\n"
     ]
    }
   ],
   "source": [
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "df = df.reindex(np.random.permutation(df.index)) # Usually a good idea to shuffle\n",
    "mask = np.random.rand(len(df)) < 0.8\n",
    "trainDF = pd.DataFrame(df[mask])\n",
    "validationDF = pd.DataFrame(df[~mask])\n",
    "\n",
    "print(\"Training DF: {}\".format(len(trainDF)))\n",
    "print(\"Validation DF: {}\".format(len(validationDF)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### K-Fold Cross Validation\n",
    "\n",
    "There are several types of cross validation; however, k-fold is the most common.  The value K specifies the number of folds.  The two most common values for K are either 5 or 10.  For this course we will always use a K value of 5, or a 5-fold cross validation. A 5-fold validation is illustrated by the following diagram:\n",
    "\n",
    "![K-Fold Crossvalidation](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_kfold.png \"K-Fold Crossvalidation\")\n",
    "\n",
    "First, the data are split into 5 equal (or close to, due to rounding) folds.  These folds are used to generate 5 training/validation set combinations.  Each of the folds becomes the validation set once, and the remaining folds become the training sets.  This allows the validated results to be appended together to produce a final out-of-sample prediction for the entire dataset.  \n",
    "\n",
    "\n",
    "The following code demonstrates a 5-fold cross validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1, Training Size: 318, Validation Size: 80\n",
      "Fold #2, Training Size: 318, Validation Size: 80\n",
      "Fold #3, Training Size: 318, Validation Size: 80\n",
      "Fold #4, Training Size: 319, Validation Size: 79\n",
      "Fold #5, Training Size: 319, Validation Size: 79\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:15: DeprecationWarning: \n",
      ".ix is deprecated. Please use\n",
      ".loc for label based indexing or\n",
      ".iloc for positional indexing\n",
      "\n",
      "See the documentation here:\n",
      "http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated\n",
      "  from ipykernel import kernelapp as app\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from sklearn.model_selection import KFold\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "df = pd.read_csv(filename_read,na_values=['NA','?'])\n",
    "df = df.reindex(np.random.permutation(df.index))\n",
    "kf = KFold(5)\n",
    "\n",
    "fold = 1\n",
    "for train_index, validate_index in kf.split(df):        \n",
    "    trainDF = pd.DataFrame(df.ix[train_index,:])\n",
    "    validateDF = pd.DataFrame(df.ix[validate_index])\n",
    "    print(\"Fold #{}, Training Size: {}, Validation Size: {}\".format(fold,len(trainDF),len(validateDF)))\n",
    "    fold+=1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessing Files Directly\n",
    "========================\n",
    "It is possible to access files directly, rather than using Pandas.  For class assignments you should use Pandas; however, direct access is possible.  Using the CSV package, you can read the files in, line-by-line and process them.  Accessing a file line-by-line can allow you to process very large files that would not fit into memory.  For the purposes of this class, all files will fit into memory, and you should use Pandas for all class assignments.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mpg,cylinders,displacement,horsepower,weight,acceleration,year,origin,name\n",
      "18,8,307,130,3504,12,70,1,chevrolet chevelle malibu\n",
      "15,8,350,165,3693,11.5,70,1,buick skylark 320\n",
      "18,8,318,150,3436,11,70,1,plymouth satellite\n",
      "16,8,304,150,3433,12,70,1,amc rebel sst\n"
     ]
    }
   ],
   "source": [
    "# Read a raw text file (avoid this)\n",
    "import codecs\n",
    "import os\n",
    "\n",
    "path = \"./data\"\n",
    "\n",
    "# Always specify your encoding! There is no such thing as \"its just a text file\".\n",
    "# See... http://www.joelonsoftware.com/articles/Unicode.html\n",
    "# Also see... http://www.utf8everywhere.org/\n",
    "encoding = 'utf-8'\n",
    "filename = os.path.join(path,\"auto-mpg.csv\")\n",
    "\n",
    "c = 0\n",
    "\n",
    "with codecs.open(filename, \"r\", encoding) as fh:\n",
    "    # Iterate over this line by line...\n",
    "    for line in fh:\n",
    "        c+=1 # Only the first 5 lines\n",
    "        if c>5: break\n",
    "        print(line.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['mpg', 'cylinders', 'displacement', 'horsepower', 'weight', 'acceleration', 'year', 'origin', 'name']\n",
      "['18', '8', '307', '130', '3504', '12', '70', '1', 'chevrolet chevelle malibu']\n",
      "['15', '8', '350', '165', '3693', '11.5', '70', '1', 'buick skylark 320']\n",
      "['18', '8', '318', '150', '3436', '11', '70', '1', 'plymouth satellite']\n",
      "['16', '8', '304', '150', '3433', '12', '70', '1', 'amc rebel sst']\n"
     ]
    }
   ],
   "source": [
    "# Read a CSV file\n",
    "import codecs\n",
    "import os\n",
    "import csv\n",
    "\n",
    "encoding = 'utf-8'\n",
    "path = \"./data/\"\n",
    "filename = os.path.join(path,\"auto-mpg.csv\")\n",
    "\n",
    "c = 0\n",
    "\n",
    "with codecs.open(filename, \"r\", encoding) as fh:\n",
    "    reader = csv.reader(fh)\n",
    "    for row in reader:\n",
    "        c+=1\n",
    "        if c>5: break\n",
    "        print(row)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Car Name: chevrolet chevelle malibu\n",
      "Car Name: buick skylark 320\n",
      "Car Name: plymouth satellite\n",
      "Car Name: amc rebel sst\n",
      "Car Name: ford torino\n"
     ]
    }
   ],
   "source": [
    "# Read a CSV, symbolic headers\n",
    "import codecs\n",
    "import os\n",
    "import csv\n",
    "\n",
    "path = \"./data\"\n",
    "\n",
    "encoding = 'utf-8'\n",
    "filename = os.path.join(path,\"auto-mpg.csv\")\n",
    "\n",
    "c = 0\n",
    "\n",
    "with codecs.open(filename, \"r\", encoding) as fh:\n",
    "    reader = csv.reader(fh)\n",
    "\n",
    "    # Generate header index using comprehension.\n",
    "    # Comprehension is cool, but not necessarily a beginners feature of Python.\n",
    "    header_idx = {key: value for (value, key) in enumerate(next(reader))}\n",
    "    \n",
    "    for row in reader:\n",
    "        c+=1\n",
    "        if c>5: break\n",
    "        print( \"Car Name: {}\".format(row[header_idx['name']]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acceleration:{'count': 398, 'variance': 7.585740574732961, 'mean': 15.568090452261291, 'sdev': 2.7542223175940177}\n",
      "cylinders:{'count': 398, 'variance': 2.8861455518799946, 'mean': 5.454773869346734, 'sdev': 1.698865960539558}\n",
      "displacement:{'count': 398, 'variance': 10844.882068950259, 'mean': 193.42587939698493, 'sdev': 104.13876352708563}\n",
      "horsepower:{'count': 392, 'variance': 1477.7898792169979, 'mean': 104.46938775510205, 'sdev': 38.442032714425984}\n",
      "mpg:{'count': 398, 'variance': 60.93611928991693, 'mean': 23.514572864321615, 'sdev': 7.806159061274433}\n",
      "name:{'count': 0, 'sum': 0}\n",
      "origin:{'count': 398, 'variance': 0.6416757152597181, 'mean': 1.5728643216080402, 'sdev': 0.801046637381194}\n",
      "weight:{'count': 398, 'variance': 715339.1287404363, 'mean': 2970.424623115578, 'sdev': 845.7772335198177}\n",
      "year:{'count': 398, 'variance': 13.638089947223559, 'mean': 76.01005025125629, 'sdev': 3.6929784655780975}\n"
     ]
    }
   ],
   "source": [
    "# Read a CSV, manual stats\n",
    "import codecs\n",
    "import os\n",
    "import csv\n",
    "import math\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "encoding = 'utf-8'\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")\n",
    "filename_write = os.path.join(path,\"auto-mpg-norm.csv\")\n",
    "\n",
    "c = 0\n",
    "\n",
    "with codecs.open(filename_read, \"r\", encoding) as fh:\n",
    "    reader = csv.reader(fh)\n",
    "\n",
    "    # Generate header index using comprehension.\n",
    "    # Comprehension is cool, but not necessarily a beginners feature of Python.\n",
    "    header_idx = {key: value for (value, key) in enumerate(next(reader))}\n",
    "    headers = header_idx.keys()\n",
    "    \n",
    "    #print([(key,{'count':0}) for key in headers])\n",
    "    \n",
    "    fields = {key: value for (key, value) in [(key,{'count':0,'sum':0,'variance':0}) for key in headers] }\n",
    "    \n",
    "    # Pass 1, means\n",
    "    row_count = 0\n",
    "    for row in reader:\n",
    "        row_count += 1\n",
    "        for name in headers:\n",
    "            try:\n",
    "                value = float(row[header_idx[name]])\n",
    "                field = fields[name]\n",
    "                field['count'] += 1\n",
    "                field['sum'] += value\n",
    "            except ValueError:\n",
    "                pass\n",
    "    \n",
    "    # Calculate means, toss sums (part of pass 1)\n",
    "    for field in fields.values():\n",
    "        # If 90% are not missing (or non-numeric) calculate a mean\n",
    "        if (field['count']/row_count)>0.9:\n",
    "            field['mean'] = field['sum'] / field['count']\n",
    "            del field['sum']\n",
    "    \n",
    "    # Pass 2, standard deviation & variance\n",
    "    fh.seek(0)\n",
    "    for row in reader:\n",
    "        for name in headers:\n",
    "            try:\n",
    "                value = float(row[header_idx[name]])\n",
    "                field = fields[name]\n",
    "                # If we failed to calculate a mean, no variance.\n",
    "                if 'mean' in field:\n",
    "                    field['variance'] += (value - field['mean'])**2\n",
    "            except ValueError:\n",
    "                pass\n",
    "            \n",
    "    # Calculate standard deviation, keep variance (part of pass 2)\n",
    "    for field in fields.values():\n",
    "        # If no variance, then no standard deviation\n",
    "        if 'mean' in field:\n",
    "            field['variance'] /= field['count']\n",
    "            field['sdev'] = math.sqrt(field['variance'])\n",
    "        else:\n",
    "            del field['variance']\n",
    "    \n",
    "    # Print summary stats\n",
    "    for key in sorted(fields.keys()):\n",
    "        print(\"{}:{}\".format(key,fields[key]))\n",
    "        \n",
    "        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Module 2 Assignment\n",
    "\n",
    "You can find the first assignmeht here: [assignment 2](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class2.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}