t81_558_class12_security.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 12: Deep Learning and Security**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* [Part 12.1: Security and Information Assurance with Deep Learning](https://www.youtube.com/watch?v=UI8HX5GzpGQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=35)\n",
    "* [Part 12.2: Programming KDD99 with Keras TensorFlow, Intrusion Detection System (IDS)](https://www.youtube.com/watch?v=2PAFVKA-OWY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=36)\n",
    "* Part 12.3: Security Project (coming soon)\n",
    "\n",
    "Weekly video update:\n",
    "\n",
    "* *Will be posted week of this class*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions\n",
    "\n",
    "You will see these at the top of every module.  These are simply a set of reusable functions that we will make use of.  Each of them will be explained as the semester progresses.  They are explained in greater detail as the course progresses.  Class 4 contains a complete overview of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import os\n",
    "import requests\n",
    "import base64\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = \"{}-{}\".format(name, x)\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = \"{}-{}\".format(name, tv)\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df.as_matrix(result).astype(np.float32), dummies.as_matrix().astype(np.float32)\n",
    "    else:\n",
    "        # Regression\n",
    "        return df.as_matrix(result).astype(np.float32), df.as_matrix([target]).astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred,y,sort=True):\n",
    "    t = pd.DataFrame({'pred' : pred, 'y' : y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'],inplace=True)\n",
    "    a = plt.plot(t['y'].tolist(),label='expected')\n",
    "    b = plt.plot(t['pred'].tolist(),label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean()) >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "               * (normalized_high - normalized_low) + normalized_low\n",
    "        \n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - Pandas dataframe output.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.  \n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    r = requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={'csv':base64.b64encode(data.to_csv(index=False).encode('ascii')).decode(\"ascii\"),\n",
    "        'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code == 200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The KDD-99 Dataset\n",
    "\n",
    "The [KDD-99 dataset](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html) is very famous in the security field and almost a \"hello world\" of intrusion detection systems in machine learning.\n",
    "\n",
    "# Read in Raw KDD-99 Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/jeff/.keras/datasets/kddcup.data_10_percent.gz\n",
      "Read 494021 rows.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>181</td>\n",
       "      <td>5450</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>239</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>19</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>235</td>\n",
       "      <td>1337</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>219</td>\n",
       "      <td>1337</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>39</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>217</td>\n",
       "      <td>2032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>49</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type service flag  src_bytes  dst_bytes  land  \\\n",
       "0         0           tcp    http   SF        181       5450     0   \n",
       "1         0           tcp    http   SF        239        486     0   \n",
       "2         0           tcp    http   SF        235       1337     0   \n",
       "3         0           tcp    http   SF        219       1337     0   \n",
       "4         0           tcp    http   SF        217       2032     0   \n",
       "\n",
       "   wrong_fragment  urgent  hot   ...     dst_host_srv_count  \\\n",
       "0               0       0    0   ...                      9   \n",
       "1               0       0    0   ...                     19   \n",
       "2               0       0    0   ...                     29   \n",
       "3               0       0    0   ...                     39   \n",
       "4               0       0    0   ...                     49   \n",
       "\n",
       "   dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
       "0                     1.0                     0.0   \n",
       "1                     1.0                     0.0   \n",
       "2                     1.0                     0.0   \n",
       "3                     1.0                     0.0   \n",
       "4                     1.0                     0.0   \n",
       "\n",
       "   dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
       "0                         0.11                          0.0   \n",
       "1                         0.05                          0.0   \n",
       "2                         0.03                          0.0   \n",
       "3                         0.03                          0.0   \n",
       "4                         0.02                          0.0   \n",
       "\n",
       "   dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
       "0                   0.0                       0.0                   0.0   \n",
       "1                   0.0                       0.0                   0.0   \n",
       "2                   0.0                       0.0                   0.0   \n",
       "3                   0.0                       0.0                   0.0   \n",
       "4                   0.0                       0.0                   0.0   \n",
       "\n",
       "   dst_host_srv_rerror_rate  outcome  \n",
       "0                       0.0  normal.  \n",
       "1                       0.0  normal.  \n",
       "2                       0.0  normal.  \n",
       "3                       0.0  normal.  \n",
       "4                       0.0  normal.  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.utils.data_utils import get_file\n",
    "\n",
    "try:\n",
    "    path = get_file('kddcup.data_10_percent.gz', origin='http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path) \n",
    "\n",
    "# This file is a CSV, just no CSV extension or headers\n",
    "# Download from: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html\n",
    "df = pd.read_csv(path, header=None)\n",
    "\n",
    "print(\"Read {} rows.\".format(len(df)))\n",
    "# df = df.sample(frac=0.1, replace=False) # Uncomment this line to sample only 10% of the dataset\n",
    "df.dropna(inplace=True,axis=1) # For now, just drop NA's (rows with missing values)\n",
    "\n",
    "# The CSV file has no column heads, so add them\n",
    "df.columns = [\n",
    "    'duration',\n",
    "    'protocol_type',\n",
    "    'service',\n",
    "    'flag',\n",
    "    'src_bytes',\n",
    "    'dst_bytes',\n",
    "    'land',\n",
    "    'wrong_fragment',\n",
    "    'urgent',\n",
    "    'hot',\n",
    "    'num_failed_logins',\n",
    "    'logged_in',\n",
    "    'num_compromised',\n",
    "    'root_shell',\n",
    "    'su_attempted',\n",
    "    'num_root',\n",
    "    'num_file_creations',\n",
    "    'num_shells',\n",
    "    'num_access_files',\n",
    "    'num_outbound_cmds',\n",
    "    'is_host_login',\n",
    "    'is_guest_login',\n",
    "    'count',\n",
    "    'srv_count',\n",
    "    'serror_rate',\n",
    "    'srv_serror_rate',\n",
    "    'rerror_rate',\n",
    "    'srv_rerror_rate',\n",
    "    'same_srv_rate',\n",
    "    'diff_srv_rate',\n",
    "    'srv_diff_host_rate',\n",
    "    'dst_host_count',\n",
    "    'dst_host_srv_count',\n",
    "    'dst_host_same_srv_rate',\n",
    "    'dst_host_diff_srv_rate',\n",
    "    'dst_host_same_src_port_rate',\n",
    "    'dst_host_srv_diff_host_rate',\n",
    "    'dst_host_serror_rate',\n",
    "    'dst_host_srv_serror_rate',\n",
    "    'dst_host_rerror_rate',\n",
    "    'dst_host_srv_rerror_rate',\n",
    "    'outcome'\n",
    "]\n",
    "\n",
    "# display 5 rows\n",
    "df[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyzing a Dataset\n",
    "\n",
    "The following script can be used to give a high level overview of how a dataset appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ENCODING = 'utf-8'\n",
    "\n",
    "def expand_categories(values):\n",
    "    result = []\n",
    "    s = values.value_counts()\n",
    "    t = float(len(values))\n",
    "    for v in s.index:\n",
    "        result.append(\"{}:{}%\".format(v,round(100*(s[v]/t),2)))\n",
    "    return \"[{}]\".format(\",\".join(result))\n",
    "        \n",
    "def analyze(filename):\n",
    "    print()\n",
    "    print(\"Analyzing: {}\".format(filename))\n",
    "    df = pd.read_csv(filename,encoding=ENCODING)\n",
    "    cols = df.columns.values\n",
    "    total = float(len(df))\n",
    "\n",
    "    print(\"{} rows\".format(int(total)))\n",
    "    for col in cols:\n",
    "        uniques = df[col].unique()\n",
    "        unique_count = len(uniques)\n",
    "        if unique_count>100:\n",
    "            print(\"** {}:{} ({}%)\".format(col,unique_count,int(((unique_count)/total)*100)))\n",
    "        else:\n",
    "            print(\"** {}:{}\".format(col,expand_categories(df[col])))\n",
    "            expand_categories(df[col])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Analyze KDD-99\n",
    "\n",
    "import tensorflow.contrib.learn as skflow\n",
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "\n",
    "path = \"./data/\"\n",
    "\n",
    "filename_read = os.path.join(path,\"auto-mpg.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Encode the feature vector\n",
    "Encode every row in the database.  This is not instant!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>...</th>\n",
       "      <th>flag-S3</th>\n",
       "      <th>flag-SF</th>\n",
       "      <th>flag-SH</th>\n",
       "      <th>land-0</th>\n",
       "      <th>land-1</th>\n",
       "      <th>logged_in-0</th>\n",
       "      <th>logged_in-1</th>\n",
       "      <th>is_host_login-0</th>\n",
       "      <th>is_guest_login-0</th>\n",
       "      <th>is_guest_login-1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002879</td>\n",
       "      <td>0.138664</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002820</td>\n",
       "      <td>-0.011578</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002824</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002840</td>\n",
       "      <td>0.014179</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.067792</td>\n",
       "      <td>-0.002842</td>\n",
       "      <td>0.035214</td>\n",
       "      <td>-0.04772</td>\n",
       "      <td>-0.002571</td>\n",
       "      <td>-0.044136</td>\n",
       "      <td>-0.009782</td>\n",
       "      <td>-0.005679</td>\n",
       "      <td>-0.010552</td>\n",
       "      <td>-0.004676</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 121 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration  src_bytes  dst_bytes  wrong_fragment    urgent       hot  \\\n",
       "0 -0.067792  -0.002879   0.138664        -0.04772 -0.002571 -0.044136   \n",
       "1 -0.067792  -0.002820  -0.011578        -0.04772 -0.002571 -0.044136   \n",
       "2 -0.067792  -0.002824   0.014179        -0.04772 -0.002571 -0.044136   \n",
       "3 -0.067792  -0.002840   0.014179        -0.04772 -0.002571 -0.044136   \n",
       "4 -0.067792  -0.002842   0.035214        -0.04772 -0.002571 -0.044136   \n",
       "\n",
       "   num_failed_logins  num_compromised  root_shell  su_attempted  \\\n",
       "0          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "1          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "2          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "3          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "4          -0.009782        -0.005679   -0.010552     -0.004676   \n",
       "\n",
       "         ...         flag-S3  flag-SF  flag-SH  land-0  land-1  logged_in-0  \\\n",
       "0        ...               0        1        0       1       0            0   \n",
       "1        ...               0        1        0       1       0            0   \n",
       "2        ...               0        1        0       1       0            0   \n",
       "3        ...               0        1        0       1       0            0   \n",
       "4        ...               0        1        0       1       0            0   \n",
       "\n",
       "   logged_in-1  is_host_login-0  is_guest_login-0  is_guest_login-1  \n",
       "0            1                1                 1                 0  \n",
       "1            1                1                 1                 0  \n",
       "2            1                1                 1                 0  \n",
       "3            1                1                 1                 0  \n",
       "4            1                1                 1                 0  \n",
       "\n",
       "[5 rows x 121 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now encode the feature vector\n",
    "\n",
    "encode_numeric_zscore(df, 'duration')\n",
    "encode_text_dummy(df, 'protocol_type')\n",
    "encode_text_dummy(df, 'service')\n",
    "encode_text_dummy(df, 'flag')\n",
    "encode_numeric_zscore(df, 'src_bytes')\n",
    "encode_numeric_zscore(df, 'dst_bytes')\n",
    "encode_text_dummy(df, 'land')\n",
    "encode_numeric_zscore(df, 'wrong_fragment')\n",
    "encode_numeric_zscore(df, 'urgent')\n",
    "encode_numeric_zscore(df, 'hot')\n",
    "encode_numeric_zscore(df, 'num_failed_logins')\n",
    "encode_text_dummy(df, 'logged_in')\n",
    "encode_numeric_zscore(df, 'num_compromised')\n",
    "encode_numeric_zscore(df, 'root_shell')\n",
    "encode_numeric_zscore(df, 'su_attempted')\n",
    "encode_numeric_zscore(df, 'num_root')\n",
    "encode_numeric_zscore(df, 'num_file_creations')\n",
    "encode_numeric_zscore(df, 'num_shells')\n",
    "encode_numeric_zscore(df, 'num_access_files')\n",
    "encode_numeric_zscore(df, 'num_outbound_cmds')\n",
    "encode_text_dummy(df, 'is_host_login')\n",
    "encode_text_dummy(df, 'is_guest_login')\n",
    "encode_numeric_zscore(df, 'count')\n",
    "encode_numeric_zscore(df, 'srv_count')\n",
    "encode_numeric_zscore(df, 'serror_rate')\n",
    "encode_numeric_zscore(df, 'srv_serror_rate')\n",
    "encode_numeric_zscore(df, 'rerror_rate')\n",
    "encode_numeric_zscore(df, 'srv_rerror_rate')\n",
    "encode_numeric_zscore(df, 'same_srv_rate')\n",
    "encode_numeric_zscore(df, 'diff_srv_rate')\n",
    "encode_numeric_zscore(df, 'srv_diff_host_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_count')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_count')\n",
    "encode_numeric_zscore(df, 'dst_host_same_srv_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_diff_srv_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_same_src_port_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_diff_host_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_serror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_serror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_rerror_rate')\n",
    "encode_numeric_zscore(df, 'dst_host_srv_rerror_rate')\n",
    "outcomes = encode_text_index(df, 'outcome')\n",
    "num_classes = len(outcomes)\n",
    "\n",
    "# display 5 rows\n",
    "\n",
    "df.dropna(inplace=True,axis=1)\n",
    "df[0:5]\n",
    "# This is the numeric feature vector, as it goes to the neural net\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 494021 samples, validate on 123506 samples\n",
      "Epoch 1/1000\n",
      " - 30s - loss: 0.0968 - val_loss: 0.0381\n",
      "Epoch 2/1000\n",
      " - 29s - loss: 0.0342 - val_loss: 0.0299\n",
      "Epoch 3/1000\n",
      " - 30s - loss: 0.0272 - val_loss: 0.0242\n",
      "Epoch 4/1000\n",
      " - 29s - loss: 0.0248 - val_loss: 0.0229\n",
      "Epoch 5/1000\n",
      " - 30s - loss: 0.0228 - val_loss: 0.0203\n",
      "Epoch 6/1000\n",
      " - 33s - loss: 0.0212 - val_loss: 0.0205\n",
      "Epoch 7/1000\n",
      " - 29s - loss: 0.0208 - val_loss: 0.0204\n",
      "Epoch 8/1000\n",
      " - 32s - loss: 0.0193 - val_loss: 0.0194\n",
      "Epoch 9/1000\n",
      " - 33s - loss: 0.0188 - val_loss: 0.0195\n",
      "Epoch 10/1000\n",
      " - 31s - loss: 0.0185 - val_loss: 0.0177\n",
      "Epoch 11/1000\n",
      " - 30s - loss: 0.0178 - val_loss: 0.0179\n",
      "Epoch 12/1000\n",
      " - 32s - loss: 0.0173 - val_loss: 0.0167\n",
      "Epoch 13/1000\n",
      " - 31s - loss: 0.0170 - val_loss: 0.0167\n",
      "Epoch 14/1000\n",
      " - 31s - loss: 0.0170 - val_loss: 0.0164\n",
      "Epoch 15/1000\n",
      " - 31s - loss: 0.0157 - val_loss: 0.0143\n",
      "Epoch 16/1000\n",
      " - 31s - loss: 0.0156 - val_loss: 0.0135\n",
      "Epoch 17/1000\n",
      " - 29s - loss: 0.0153 - val_loss: 0.0146\n",
      "Epoch 18/1000\n",
      " - 29s - loss: 0.0154 - val_loss: 0.0139\n",
      "Epoch 19/1000\n",
      " - 32s - loss: 0.0156 - val_loss: 0.0151\n",
      "Epoch 20/1000\n",
      " - 30s - loss: 0.0153 - val_loss: 0.0142\n",
      "Epoch 00020: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1a19dc2da0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Activation\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "# Break into X (predictors) & y (prediction)\n",
    "x, y = to_xy(df,'outcome')\n",
    "\n",
    "# Create a test/train split.  25% test\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Create neural net\n",
    "model = Sequential()\n",
    "model.add(Dense(10, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))\n",
    "model.add(Dense(50, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))\n",
    "model.add(Dense(10, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))\n",
    "model.add(Dense(1, kernel_initializer='normal'))\n",
    "model.add(Dense(y.shape[1],activation='softmax'))\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, verbose=1, mode='auto')\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),callbacks=[monitor],verbose=2,epochs=1000)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation score: 0.9982510971126909\n"
     ]
    }
   ],
   "source": [
    "# Measure accuracy\n",
    "pred = model.predict(x_test)\n",
    "pred = np.argmax(pred,axis=1)\n",
    "y_eval = np.argmax(y_test,axis=1)\n",
    "score = metrics.accuracy_score(y_eval, pred)\n",
    "print(\"Validation score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}