Part 2a - Music speech classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Music vs. Speech Classification with Deep Learning\n",
    "\n",
    "This tutorial shows how different Convolutional Neural Network architectures are used for the task of discriminating a piece of audio whether it is music or speech (binary classification).\n",
    "\n",
    "The data set used is the [Music Speech](http://marsyasweb.appspot.com/download/data_sets/) data set compiled by George Tzanetakis. It consists of 128 tracks, each 30 seconds long. Each class (music/speech) has 64 examples. The tracks are all 22050Hz Mono 16-bit audio files in .wav format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of Contents\n",
    "\n",
    "This tutorial contains:\n",
    "* Loading and Preprocessing of Audio files\n",
    "* Loading class files from CSV and using Label Encoder\n",
    "* Audio Preprocessing: Generating log Mel spectrograms\n",
    "* Standardization of Data\n",
    "* Convolutional Neural Networks: single, stacked, parallel\n",
    "* ReLU Activation\n",
    "* Dropout\n",
    "* Train/Test set split\n",
    "\n",
    "You can execute the following code blocks by pressing SHIFT+Enter consecutively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Data\n",
    "\n",
    "The data set can be downloaded from [here](http://opihi.cs.uvic.ca/sound/music_speech.tar.gz).\n",
    "\n",
    "Please unpack it (it is .tar.gz compressed)\n",
    "\n",
    "Set the path to the unpacked folder in the next box:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "DATA_PATH = '/data/music/GTZAN_music_speech'\n",
    "AUDIO_PATH = os.path.join(DATA_PATH, 'wav')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# General Imports\n",
    "\n",
    "import argparse\n",
    "import csv\n",
    "import datetime\n",
    "import glob\n",
    "import math\n",
    "import sys\n",
    "import time\n",
    "import numpy as np\n",
    "import pandas as pd # Pandas for reading CSV files and easier Data handling in preparation\n",
    "\n",
    "# Deep Learning\n",
    "\n",
    "import keras\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Input, Convolution2D, MaxPooling2D, Dense, Dropout, Activation, Flatten, merge\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import StratifiedShuffleSplit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# local imports for audio reading and processing\n",
    "import audio_spectrogram as rp\n",
    "from audiofile_read import audiofile_read"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Metadata\n",
    "\n",
    "The tab-separated file contains pairs of filename TAB class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>speech/stupid.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/teachers2.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/danie1.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/oneday.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/jvoice.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/relation.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/geography.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/pulp2.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/greek1.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speech/conversion.wav</th>\n",
       "      <td>speech</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            1\n",
       "0                            \n",
       "speech/stupid.wav      speech\n",
       "speech/teachers2.wav   speech\n",
       "speech/danie1.wav      speech\n",
       "speech/oneday.wav      speech\n",
       "speech/jvoice.wav      speech\n",
       "speech/relation.wav    speech\n",
       "speech/geography.wav   speech\n",
       "speech/pulp2.wav       speech\n",
       "speech/greek1.wav      speech\n",
       "speech/conversion.wav  speech"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_file = os.path.join(DATA_PATH,'filelist_wav_relpath_wclasses.txt')\n",
    "metadata = pd.read_csv(csv_file, index_col=0, sep='\\t', header=None)\n",
    "metadata.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create two lists: one with filenames and one with associated classes\n",
    "filelist = metadata.index.tolist()\n",
    "classes = metadata[1].values.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encode Labels to Numbers\n",
    "\n",
    "String labels need to be encoded as numbers. We use the LabelEncoder from the scikit-learn package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['speech', 'speech', 'speech', 'speech', 'speech']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classes[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 classes: music, speech\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "labelencoder = LabelEncoder()\n",
    "labelencoder.fit(classes)\n",
    "print(len(labelencoder.classes_), \"classes:\", \", \".join(list(labelencoder.classes_)))\n",
    "\n",
    "classes_num = labelencoder.transform(classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check first few classes (speech)\n",
    "classes_num[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check first few classes (music)\n",
    "classes_num[-5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: In order to correctly re-transform any predicted numbers into strings, we keep the labelencoder for later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Audio Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "................................................................................................................................\n",
      "Read 128 audio files\n"
     ]
    }
   ],
   "source": [
    "list_spectrograms = [] # spectrograms are put into a list first\n",
    "\n",
    "# desired output parameters\n",
    "n_mel_bands = 40   # y axis\n",
    "frames = 80        # x axis\n",
    "\n",
    "# some FFT parameters\n",
    "fft_window_size=512\n",
    "fft_overlap = 0.5\n",
    "hop_size = int(fft_window_size*(1-fft_overlap))\n",
    "segment_size = fft_window_size + (frames-1) * hop_size # segment size for desired # frames\n",
    "\n",
    "for filename in filelist:\n",
    "    print(\".\", end='')\n",
    "    filepath = os.path.join(AUDIO_PATH, filename)\n",
    "    samplerate, samplewidth, wavedata = audiofile_read(filepath,verbose=False)\n",
    "    sample_length = wavedata.shape[0]\n",
    "\n",
    "    # make Mono (in case of multiple channels / stereo)\n",
    "    if wavedata.ndim > 1:\n",
    "        wavedata = np.mean(wavedata, 1)\n",
    "        \n",
    "    # take only a segment; choose start position:\n",
    "    #pos = 0 # beginning\n",
    "    pos = int(wavedata.shape[0]/2 - segment_size/2) # center minus half segment size\n",
    "    wav_segment = wavedata[pos:pos+segment_size]\n",
    "\n",
    "    # 1) FFT spectrogram \n",
    "    spectrogram = rp.calc_spectrogram(wav_segment,fft_window_size,fft_overlap)\n",
    "\n",
    "    # 2) Transform to perceptual Mel scale (uses librosa.filters.mel)\n",
    "    spectrogram = rp.transform2mel(spectrogram,samplerate,fft_window_size,n_mel_bands)\n",
    "        \n",
    "    # 3) Log 10 transform\n",
    "    spectrogram = np.log10(spectrogram)\n",
    "    \n",
    "    list_spectrograms.append(spectrogram)\n",
    "        \n",
    "print(\"\\nRead\", len(filelist), \"audio files\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.00320435, -0.05340576, -0.09249878, ...,  0.14624023,\n",
       "        0.19299316,  0.18325806])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wav_segment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "128"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list_spectrograms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40, 80)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spectrogram.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "An audio segment is 0.94 seconds long\n"
     ]
    }
   ],
   "source": [
    "print(\"An audio segment is\", round(float(segment_size) / samplerate, 2), \"seconds long\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: For simplicity of this tutorial, here we load only 1 single segment of ~ 1 second length from each audio file.\n",
    "In a real setting, one would create training instances of as many audio segments as possible to be fed to a Neural Network.\n",
    "\n",
    "### Show Waveform and Spectrogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'music/echoes.wav'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XecE3X6B/DPs5Xel16WrihIWRcUpUgHFfVQQe/Eyqmo51k5sZ3oTxT1bJxnPbEXbHggHRRBUHoT2KVIb0uvy+5+f39ksjtJZpJJZtI/79eLF8lkkpnMTp75zvNtopQCEREll5Ro7wAREUUegz8RURJi8CciSkIM/kRESYjBn4goCTH4ExElIQZ/IqIkxOBPRJSEGPyJiJJQWrR3wEytWrVUdnZ2tHeDiCiuLFmyZL9SKivQejEb/LOzs7F48eJo7wYRUVwRkT+srMe0DxFREmLwJyJKQgz+RERJiMGfiCgJMfgTESUhBn8ioiTE4E9ElIQY/ImIYsi8vH34o+B42LcTs528iIiS0V/e/RUAsGXsoLBuhyV/IqIkxOBPRJSEGPyJiJIQgz8RURJi8CciipL8vceglIrKthn8iYiiYOX2Q+j90o94e96mqGyfwZ+IKAq2HTgJAFi+7VBUts/gT0QUBQq+6Z49R05FbPsM/kREUSQQAIBSCp3/b1bEtsvgT0QUA/47f0tEt8fgT0QUBb9sLPB4PnXN7ohun8GfiMgBp84UY9y0dTh1ptjS+h8v2up6IGHcKT8Y/ImIHPDe/M0YP2cj3gm16WaEm/sz+BMROeD0mRIAQGFxdDptBYvBn4goCTkS/EWkv4isF5F8ERnlZ70/iYgSkRwntktEFO+OnSqKynZtB38RSQUwHsAAAG0ADBORNgbrVQbwNwCL7G6TiChR/LhhHwqOnY74dp0o+ecCyFdKbVJKFQL4DMBgg/XGAHgOQOS6sBERxYGC44WGPX7DyYng3wDANt3z7dqyUiLSEUAjpdRkB7ZHRBQ1c9btRfaoydiy33OeXXfoDqXlZjRae4a9wldEUgC8BOB+C+uOEJHFIrJ437594d41IqKgfbt8B4DoDcjmFCeC/w4AjXTPG2rL3CoDOBfAXBHZAqALgElGlb5KqbeUUjlKqZysrCwHdo2IiIw4Efx/A9BSRJqKSAaAoQAmuV9USh1WStVSSmUrpbIBLARwuVJqsQPbJiKKexKFvI/t4K+UKgJwF4BpAH4H8IVSao2IPCUil9v9fCKiWORsBa0g0hN6pTnxIUqpKQCmeC173GTdHk5sk4goGgIV0kMpxY/539qQ9sUO9vAlIgpCKAX0SSt24p5Pl5U+P3zijMfrP27YF+mhfRj8iYgc4Sdvc8+nyzBpxc7S5wNfnReJPfKLwZ+IDB09dQa3vP8b9kZwasF4EDDtY6HV/o5DJ4P+XKcx+BORoW+W7cCsdXvx2uz8aO9KUmDah4iIwo7Bn4jIAWYl95OF1mb2UhFu68ngT0R+RXrAsXjn3dTztdl50dmRABj8iYjC6Phpa+P1S4S7+TL4E5FfVlqvhGrYWwvRdezssH1+KJRS+Py3rQGD9oQFf+CDX7bo3ue7ztTVu/H9yl2WtxtJDP5EFDW/bCowbPYYTYs2H8DDX63CE5PW+F1v+bZDePy7NSgsKjFd5/aPluDA8UJL291zJLITujD4E5FfyZbzP1HoKvHvtzi71ps/bnRku5G+CDL4E5GhaEwwEguCTXMdPuk5VEO8HDcGf6IoWbf7SMTzvG6Dx89Ht+fn+F0nucr7yYfBnygKfs7bj/4vz8PHi7ZGZfsrth3C1gMnLK0bzgrfWGZ2XTZrlWMlPeavfiDSGPwpaqav2e0zD2qy2Fzg+t6/7zoS8W2vCHL6wWTL+buvdaF+a38tNt31CbGAwZ+iZsSHS3DJi3OjvRtJZ/2eo5bWi3R5XykVE4PIuVNxZik5O6m6KGX5DDH4U1SVxNCPIZZNWbULvV/6ESVOHDCLHxHpP81/529B7v/NQp7Fi1O4fLJoGwBgXt7+kN5/8kwxskdNxjfLtvu89o+vV9naNycx+BPFgQe+XIH8vcdw8oy1cWKsajX6B3y91DdI6UUq5z8/3xVs/yiwVhcRLvsCNPEM1BN312HX3cvfP1/h89rUNbtD3zGHMfgThVGLR6Zg2FsLbX+OO9yUOJw3KCwuwZMBOjNFKudvdSuFRSWWh0yIBHcnLqt/ms9/i04lvzcGf6IwKipR+GVTgenrVgNeilbadCIMWw3mkcz5ny4qu6PRF6yPnjrjMyrm1f9ZgHOemBbSdg4eL8SZYmdb3Hy9bIfn86U7TNZ0GTdtg6PbDxWDP1E8cLdACUNLwSOnijzGqDGy6/BJTF0dvpRF60enGrZ8avvkdFzs1R9hxfbDIW+nw5gZuO8L33SMXrgvelZ7Docbgz9RFHkHms37j6Pfv37CQa/xYI5paQ4nUjDHT/vWGzz+nSv1s2Djfvx7br62rTJXjl+A2z9aYnvb/rhz5Wt2el4EnA6W3+vm0k1mDP5EUeQdyt+Ym4/1e45i+tqyUvaZ4pLSfHKgxj4vTl8fcJTMsVPXmb523duL8PzU9T7Ld+uaYB4/XYT7Pl/uc4FyylcBKqAD2X/sNNbsLLs7KNAuHk71pp71+x7D5fHWcI3BnyjG6XPUgQLYa7PzAw4QZrWXqVn649Nft+LrZTtsze07eeUuzFxrHERTbI5r3//leRj06s8AgIWbCtDp6ZmYunq3Y23sj5yKncpmOxj8KaKmrt6N7FGTLQ9zG69GfrIUfV76MeB6wYa5YOLXd8t3YMkfB4PcQmSM/GQpbv1gseFrdnPu+jTRyu2u3syLtxzwWGf8nHyfAdlKtx/CDmzZfxx7Dke/g1ow0qK9A5Rc3pu/GQCwIcodeZw0dfUuHDtdjCGdGpYum+xnAo9JK3ZilRaUgi2MBtPU82+fLQcAbBk7KMitWLNgY2idoAIJR/pExPNzx01bj437juGla9oHfO+W/cdRISMVtauUM12nxwtz7e9khLHk75CSEoXXZuXh8Anj0gS5xjX5dfOBwCtatP/YaRTHQBfh2z9aige+LGtBEig1c8+ny/DFYlde+5NFWz1aubjf6l3pWbaCq1lk17GzMXvdHmwtOIHvlvtvWggAXyzehivGzw+4XjDW7T6KjfuOOfqZwbAyMbq/P8UJXcV3q0d/wAvTfOs6AFdgz/2/WUHvX6xj8HfIj3n78OKMDXh80uqA6yql8MXibTjlcG/NWLdxb9kgbkOD7Pi09+gprNp+uDT/ffB4IXKenonn/FRe2nH01JmQB+F6e94mw+UfLfwDgw0CsPtYrN15BAs2uvoEfPCLa937Pl/usa4CsOPgSew4dBJPfb8Wl73+c2kJ35+HJq7E8iAHdLPimFf+e6tDvXPPFJcEvLCv2mG9yaeI+FyUp67ZjVajfwDgqgd5fU7gOoxwVXJHgyPBX0T6i8h6EckXkVEGr98nImtFZKWIzBKRJk5sN5ac0SrRjJrReZu7fh8emrgybIErVtlpppj7zCxc9vrPuOuTpQCAAydcP0KzSkO72j45HV1CLO3NXLvXcPmj3642HFHz1JlibN5/HANfnedRWbti2yGfDkT6tI+ImOat7crfewwFBoFu31HzZpeTVuxEt3Fz8NOGfba333L0D2j+yBTbn+OmlPHZV1hcElRAP5FABTbbwV9EUgGMBzAAQBsAw0SkjddqywDkKKXaAZgI4Hm7242krmNn45nJawG4bp/nrjf+cbsEDnBHTrl+sPuPRa8Ucfx0EbJHTcb4AKWdldsPYftBa6W5g8cLcehEeL/TtDWuYF9ainO4R45SqvSzg23Vse/oaRQVlwS9T6eLStDTYs64REWmSWHvl37EyzPzfJb7mwBm+VbXhc1ufY7Vw/fZr9aHSdh/rBATFmwxfK3DmBmWPyeROFHyzwWQr5TapJQqBPAZgMH6FZRSc5RS7giyEEBDxJEdh07i7XmuisqHJq7Ejf/9zWedQIM9xZqDWpD+JMBkIpe/Ph8XPed/xie3DmNmoP1TkfkhuWO/v2aBq3ccDnhx8zbo1Z8tlzhPFhZjjq4gcP4zM9Fi9A9Bj5cfDH3qIlITg+izJf4GlnPflaSI4GRhcdjuSty874qMuHf9m2U78PTk38O6P/HGieDfAMA23fPt2jIztwD4wYHtRtVOk7bUwbQl9g5by7YejKkBq/ScqJ8wG61RKYUvg6wDKSkN/ubrXPrazxhnUolnZu2uI5aHmX7su9W4yaAgcDqMQVkplKZVvNvzR2pKyCLdAXrkm1U4WehKW5UFf6Dvyz/ivH9OD+nzjb7FyE+Woqi4BJ8GKO3Py7OfcoqvYlzoItrUU0T+DCAHQHeT10cAGAEAjRs3juCeBe/CsbPx04M90bhmBQCukzOQcdPWoahEoU29Kj6vHTpRiCv/vQC9z66Dd4bnBLUvuw6fRL2q5YN6T7Cen7oe8/L24ZWhHdCmvu/+B7Lz0Enc/ekyw9dm/r4XD05ciQ17jmL0IO+MoTF9KbOouAS/bj6AC1vUCnq/9KYFOdxuNIYJePirldhjMOGJUgo/hGnsHe+bq7E/lNVVrdl5BJe//jPy9h5DeqprxZQUwbYD/jua+bPZYHa3ySt3oXlWJbw6yzMVpZTyuOte+of9u66lW8N35xZLnCj57wDQSPe8obbMg4j0BjAawOVKKcNaI6XUW0qpHKVUTlZWlgO7Fl57jpb9CK3cgo+fsxFv/rjJ8O7g1BnX+1ftcJ14U1fvthRcFuTvxwXPzvbbrtwJM37fjby9x/CvmaGNSHjLBOMOPYCrZQ0QXB2IvuLz1Vl5uO6dRVjoZ/TMQOau34u/fhjc2DXhLOGbWbCxABv3+QZHpfxXxoZT3l5Xc88zxa6/SaAUaKh3KN6BHwC+9zrv9Zt24i7AmxOV2bHCieD/G4CWItJURDIADAUwSb+CiHQA8CZcgd9fbWlcMTvFV2w7hGen/B7USe5ui7DnyGkMfv1n3P7REtOSMuBq6509ajJGf+tqWrpsq/+enKeLig33p7C4BC9MW1/arFEphelrduPA8UKP8VFKtDhnlGbJ3+vZ1tvoQuR0Okuf89+olRTdwW/NzsPobaF3rd67P282XG400uTU1bsDjoLpzT1RSbgoIOgUlzejEjcQ/NSDRbrhKMbPyfcZQnmugwHUu15Bf3o+9m3gZtfeNvnpt1BUXBJTM3HZZTv4K6WKANwFYBqA3wF8oZRaIyJPicjl2mrjAFQC8KWILBeRSSYfF1eMCjgKwODx8/HmT5sC5o4nrdhpWHdgZcjaIyddwdTsB6t3uqgYrR+dimd/8G1auu/oabw+J7+0YvSbZTsw4sMl6DhmRun4KEBZac27gvUHbXrBqavLAr5RCszpCUFKg7/BGfzS9A0+FyQzeXuOYq1ZhyoAA16Z5zNEwu0fLSkdBdOqcPWGdVNKlY78CQBz1hmXsdx3WUYe/mqlI/vy4cI/Sh+Pm7YeH+ueA8D2g6GnhLx5/wTttrv45/drTV+LfndCZznSzl8pNUUp1Uop1Vwp9Yy27HGl1CTtcW+lVB2lVHvt3+X+PzE+LNx0wKcjir50baVH5brdrsBjd6o8ESB71GS8ON239Heq0FXycjeNKywq8UlTudNOu00m0N6pjVviDv7j5+Tjs1+34vfdR7Xv4dm8TyllmJs24j5kq3ccxp4jp1BcolBSorBoUwGyR002HKXSnfYpOFZY+qt8xSAt4M/CTQXo86+fMPDVeX7Xs9rUNVyKLNRAe69x0/tlFdH6u662T5pXwu42GZsm2IDqXdI/eabs+T2fLgupRO7PzkMnMUPr7+Gdcgr2ruXHBErrBJLUY/scPnEGEKBq+fSQ3u++zR7Zs4Xh66t2HMbybYfQpEYFVK+YUbp8hlfHpDnr96JKueD2wawk/drsfNzft7Xf9+Y8PcOnDfsW7Q7CaDhfPfdvy/3d77nE+Lt/uWQ7Hpq4Et/ceSE6NK5u6eKWt/cYuo+bg9NFJWhWqyLqaGOpGI1S6e7ktUsXsNyl/WKLv/j3dKkef/n7aDfjbecnYLv5a176ssV6mq0HnLnIpfo5XpMcriQXcTVH3n/sNLaMHeQx3MSWghPI2+vcGFJv/rjRsc+KBUkd/M97yvWjsjPwlXfaxTvsXDF+PpplVcTs+3uULpu8qixFcqKwGHd9Yp7bP1NcghKlkJmWGvI+ejPqvLTfYi/H/63chf+tnOyzvMCrsna61nImb+8xdGhc3XLax30HsnHfcdStaj6Q1hk/wXru+uBLb/7GHNKHspEfB27VZSTcrTCH/OcX09dOnbFXMR3svm/xatK7OohhGIKllOconoe8xtYKNKViIFN0v9UXpsfG9ItOSfixfdxDCOsrctbvPmo6IUNxiTJtwx+qTQatM9z8VeoCQM8X5qL1o1N9lp/2+kFb+YGGI/58p5XkPvTK68783ZVzfmjiSi2N49w2C46dxvrd0RkVVH/hThYfB+gIGMjkVbvwwJcrAk4UH4r/eJXGne7rcGeIF/t4kPAlf/eP9eWZeXh1WAcAQL+XfzJd//mp6/DmT5uw6JFepWmHQPQtOcwGo9p71DifGuhc1VeOLdt6EFPX7MYd3ZvjkW8CtzpYvu0Qrhg/H9UqBE4plYQ4OqZZxy29Tk/P8CmRWWGWKsp5ZqbHcdMPt2H1xz/yk6WYHuS4QNtspEVW2ph31q5Y6Hw+cYm92bnMOFl5nGwSPvi7TVqxE49d2gZZlTP9rveVdptYcKzQcvC//p1FpY/n5Rm36uj+/FxrO+rHlf9eAAD4fvnO0gpYN6OQ565w1gdes+AYzAiJwQol8Pvj/RWO64b2Pf+ZmQHfP2PtnqD6Rdz96TKUKGVp9EwzjWqEtxMeuUS7fiaeJGTaZ+nWg6Uj9elPBe9WCHru9JA7fygC/LKxIODk0VbvMv2NiRIs78AfzH4s9dMfwGi44Ugwa2IYyu84UEexk4XFuM1kBil/7AR+AGhQjcGfYktCBv+r/r0A17xpXgFmxKg37bC3F+IaPxVpgP3Jpp3iniErEH+tWsI5IJk/Vpoy2vHQxLKJVoqcrHwIwhtzo9dSJJnKwv7GeiJPCRn8AVcrE++Bwt6Yu9Fwgo6lWw+aVuRtstCJKlKS7Y7WLIUWrC8Wb8ex00UY+fFSn1ZJkXLcwqxTZF9GWsKGNMcldM7/i8XbPJ5/uPAPww47cw16Q8bi6JpWUzsvz9yAu0z6HiSrIW8swLrdR5HJ4JDQpqwKz+B2iSihg79RsFz8h2/O26iSyF+76Wh4LYjeqy/PzEPD6hV8X1CuKQCTkbsH8ortyTFio16yVIKOttACjsokfDHI57w3uCD4mxAkVrw4I7gOJkajjB49XYQHJzozfku8MhoRM9GVhLuHWYyw2x8h2SR08DeK6UcN0jlWu7XP8Tt9Y2xRUExxEADYGuqaEldCRwerBR6rLXaMZm2KZclR3qNANuyxNsIpJZeEDv7JrKhYRWyOVyKKPwkd/OMglR82T4RhHBUiShwJ3don2Ak3iIiSRUKX/ImIyFjCBX+nh3QlIkpECRf8iYgoMAZ/IqIklHDBn1kfIqLAEi74ExFRYAkX/FnwJyIKLOGCPxERBcbgT0QUYwacWzfs20i44M92/kQU786pXyXs20i44E9EFO8iMQFPwgV/lvuJiAJzJPiLSH8RWS8i+SIyyuD1TBH5XHt9kYhkO7FdIiIKje3gLyKpAMYDGACgDYBhItLGa7VbABxUSrUA8C8Az9ndrhmm/ImIAnOi5J8LIF8ptUkpVQjgMwCDvdYZDGCC9ngigF4SpqTWqaLicHwsEVFCcSL4NwCwTfd8u7bMcB2lVBGAwwBqOrBtHydOM/gTEQUSUxW+IjJCRBaLyOJ9+/ZFe3eIiBKWE8F/B4BGuucNtWWG64hIGoCqAAq8P0gp9ZZSKkcplZOVleXArhERkREngv9vAFqKSFMRyQAwFMAkr3UmARiuPR4CYLYKU28sxcaeRBTnIjH/uO05fJVSRSJyF4BpAFIBvKeUWiMiTwFYrJSaBOBdAB+KSD6AA3BdIIiIKEocmcBdKTUFwBSvZY/rHp8CcLUT2wpEEIFLJhFRGEUijsVUhS8REUUGgz8RURJi8CciijGRqPBl8CciijEHjheGfRsJF/zZ1JOI4t3klbvCvo2EC/5ERBRYwgV/NvUkIgos4YI/EREFxuBPRJSEGPyJiGIMm3oSESWhSMxIyOBPRJSEEi74166cGe1dICKyhWmfEKSkBH/U/tSxYRj2hIgoNAz+EfLiNedFexei4qH+raO9C0QUJQz+Ji47r360dyGsBrWthzt7tAjpvQ/3P8vhvSGiSGPw13w3sis+uDm39PlrwzpgcPvEvABUq5CO8dd39FnesXG1KOwNEUUDg7/mvEbV0K2V56TxrwztgOeHtMPIns3Dvv1+59QJ+zYCSUvl6eDP33u3ivYuUJLgTF4x4JqcRniwX/jTHOc1Kit1X5MTnQroihmpHs9fGdo+KvsRq+6MQCGACGCFryOWPNo72rsQtL92j1yQObteldLHrw7r4PHa4PYNIrYf8SCdd0aUQBL+bK5RMaP0ce+za2NIp9hs1hnu27yv7rjAcPnnf+1S+rhyuXRkpHmeEmkhNJ1NZO0bsV6EEkNCBv/eZ9cufSy6+6e0lBT8Y4D/FM5Xd1yI2fd391n+xGVtcOOF2Y7to1Me6GstD52Rmopnr2oLAPjols6ly6uUSw9qe81qVSx9PKJbs6DeG06pYb5I3dQ1GwDwyW2dMfv+7miWVdH/G4hiXEIG/3eGnx/yezs1qY5mWZV8lt/UtSn6tAlfpay+wjeYMNaoRgXL6w7LbYwtYwfh3AZVLb+nViXPHtNf3l52BxGJvKRV59SvEnglLwtGXWJ53ScuOwcAUCEjDc2yKmH2/T2C3h6RVZH4aSVk8LfqtWEdsPSxPpbXD+cfxOiCAwBjrjjXkc9vUsv8IvHi1efhf3dfZPjal7dfgOeHtCt9Xr1CWRoNChgV4E4qUkIZCKtiRprf16tVCO6uKNJevpYV8hS6hA3+lTLTAqYlKpdL86gTiLaZ93XHjL9381iW6UAl4+Xn1feb3vlTp4amdwONalTANTmNSp97D58Rairszh7OVmqHY+7mlrUrISuGx4pqWoupJwpdwgb/1f/sh0cGnu2xzOk0hdMBrEXtSmhZp7Lp6y9cbX0Yiu/vMi7JOypKaZ8aFTPQpVkNj2VODYGbobvYvvHnTph+bzdMuqurMx8eQbGUkqPgSQT+gAkb/I14Bwi7B9i7U5hTzCovg+lx3LZhWUn+opa1bO9Td9137aVVqA9qW6+0NZBTpdDc7BqBV4Jn66hJd3Ut/du2qeeb+zf6zFsvahpwG7UqZaJ6xQy0axibLXwUgKs6GDfHvbdXWUOA70YGf/GKZsqrSjn/6ThyRlIFf2/KZnExXNfmxrpKXAWFypmuH0OqwcXK6CvkPTPA47k+bROI2Xd678bzSz+3VZ3K2DJ2ENo1rIa01BSs/mc/PDX4HMvbsKuvV8V7u4bVSpM+zw9ph0cHed7xZab7nuYD2tZDea9ObfHG6vkXTKMAt0hMJmImHKXecbp6q3hgNzZZYSv4i0gNEZkhInna/9UN1mkvIr+IyBoRWSki19rZph3e55Tdw2v0/rdvyLH8/k9u62y4XEQ8evnOuK87Ph/RxXS4av02G1Yvb6szkrunsfcw16kpYvq5lTLTDC9Mdi1/3LgyXl8J/letXsf9Y0kRQeemNT3Wr6ZVUudm1/BoqpqRloJ6Vct5rBvMRSzWK4TdQvnLJFr/Dn93prGYIouHtM8oALOUUi0BzNKeezsB4Aal1DkA+gN4WURi8z46gPS0wIerZW3jVjtu+qBulKJwu79va/RtUweXtquPulXLoXMzV0BbN6a/z7ALtSo5V2n97vAcfDuya0wNc10pMw0TdU1M9Reh7l6pNxHfyt8G1crj25Fd8cEtuahqErDLp7vuAnKyfcovppY91gfXd25sad15D/Usffz+TaE3Rfbnzb90Kn2sPwYpIQSSWy+OXh+OcFx3/B2CYPu6JAq7wX8wgAna4wkArvBeQSm1QSmVpz3eCWAvgPAky4MVZNE/p4lnYAjlHNX/EP1d3etUKYe3bshBxUzP/Ge59FSP9/U7py7OqlsFmdqFybuHbrAql0sPey9Ws2alAAwPaooAOSZ1Ae4/ofsu2eyQtm9UDeXS9Wke+7fVIoIH+1mbE0GfeunRurafNUNXWZcr98gaWDxR7+3dsvRxBZOUmNU6GTsqBGiCS86wG/zrKKV2aY93A/DbC0pEcgFkANhoc7uOCLZ5oIgg/5kBGHtVW4wxSQ8EU8iyW8K5tJ0rb10+IxXrxvTH33q1xH9v9CxV2r0YWBbEdwmmk5np5nxSeErbDd8d0a/rdKHSX8/iFgHuAkPRIYzDbl+pqzw2O48rh7Eydvb93XHrRU1N06F2+DvnYjLtE4FtBIwMIjJTRFYb/BusX0+5kq6m0VRE6gH4EMBNSqkSk3VGiMhiEVm8b9++IL9KZKSlpmBobmP85YLsoN737ciu+OnBnrhPNxxDqHk9o8ogEcHf+7RCk5pluc0Zf++G+Q9b78VqR04TZ0qE+oo5f+MdXXKWq/TcqLqrRK0v+furK6uk3eJ7p0Ja13U1sS0fZKmzcrl0vHD1eTi3gW8Kz30h/vS2Lj6vhapvm7qW1vMo+NuMJP+83PnK/Cn3XOyzrFlWJTx6aRuPc9gpmWmpeOzSNj7Lz6pr3rQ60QU805VSpsNiisgeEamnlNqlBfe9JutVATAZwGil1EI/23oLwFsAkJOTE8X2Bs4zSqXYbm0U4Fftr8+A0zLSUpBVORP7jp629Tn6H36FTFfq4f6+vqmVWy5qij91bIjqWie9EuUu+fsvjb8wpB0+/22bz9/jhavPw76jp9GgWvmg93lIp4aYunoXVu844rG8UY0K2DJ2UNCfFwz9GdRWV7o9euqMY9u4on0DPDFpjWOfBwBt6ldBh8bVUKdyOUxds9vRzwZcx2LVjsN+11nxRF9kpqWgy7OzHN9+PLCbE5gEYLj2eDjEHGbOAAASD0lEQVSA77xXEJEMAN8A+EApNdHm9kJyVUfX7WxNr4pRu62prJbce59tnA3LTEvF/X1a+c+BG2hWy5VOOD+IyslIGH9dRzSoVt4n9aT3uEHpy0x6agq2jB2E4Qa9iEWkNPADwHWdmwAAalcph3PqV8Hfe7fCDRc08Xlf7SrlcHevlj5/uwoZqbigeU2f9aPlmzsvDPo9lXUVlxMWbCl9bLXgb5QizKqcaVpJ/vGt9tIz39zZFf/RVVI76V/XBm6wULV8Osqlp+KunmXTmYZ7gECrIlHytRv8xwLoIyJ5AHprzyEiOSLyjrbONQC6AbhRRJZr/yI6KMmLV5+HZ69qi9EDPQOPU01pOzXxH4T7tDGu4MtIS8HdvVoGnQNv27Aqfn64J/7SxTe4RVNu0xqYP+oS9DzLvELTKJA74ZaLmmLL2EGoWj4dIoK/9W6J+/q0QrdWWbi5a+AOXbFm/e6jltYzC1V3X9LS5BVj1Sqko17VsrueLlrrsle08YMGt6+P5/7U1uM9nZuWpfrs1G88dmkbn45d/c+xlt7y5q6ozkyz3odD37Lpb72CO27hEhM5f3+UUgVKqV5KqZZKqd5KqQPa8sVKqVu1xx8ppdKVUu11/5Y7sfNWiQiG5TZG+YxUj6kK3WmFULWuWxkiwN2XtPDomOXmHmmyTT37FZzeGlavEJG2wPGsWoUMfHBzbkyPz2PmiIW0jb9Sqv4i69nKyZj3BdLdke/CFq7e4a8M7YBrz/ds1qo//7xbpZnxni0OcF24Vz7Zz2NZSoiRyWPgwSC8cPV5PuNUDTg3tAuQIyLw0066NlVVy6fj9es6YNehU7igmb3b/Krl07H5WXdO1zcn6v5thGPQMUpsVib3Oad+FdMzq0bFDGx+diCU8h2MLxzqVsnECgvr/e+ei/Hb5gMB13N//3FD2mFLwXGMn2O/geA1OQ2xcFMBZqzd4/PakE4NMaRTQ7w+O6902dDcxvhhtfP1EbEiKYd3uLRdfdzWrVnYS87uE9g7vfTkZW2CGqcnkcTavcqrwzqge6ss1Kni2dM3+PFlnP1m+lPzaoPZ5/7a3XX++tuqiFgO/Hb3fpxu0MGZ9/lOhuTWtFZFXHN+4OFG3N8/Mz01pE5YRj/tyuXSA/bA18eE5lkVUdfrvIiUmE/7kH8P9muNKuXSfPKhN3ZtileGdjB5V2KLtXug87NrYMLNuR4plKn3XozZD/QI8pOc+2Ytalfy2B99nVLdqq4UVn0tP+/UkOT6vQ+l0lMfoJ3o39CkpiuNWqNChumRbVCtPJ650nO+C/excqqj2J+7WOvB7bR4qPAlL/pSfrdWWVj5ZD/L+dBEZNYjNNYuAnpn1a3iM4NZJD0/pJ1Hye9aXUn5ivYN8M4NOaWV/Y21IDks1/rgff7Mur87Fv6jlyOfZce9vVvh3eE5PiPS6of5mD/qElzfuazRwxd/dU08NOWeiz0uigPbeubuH+5vPgGRT+fBWD5RbWLwp7Aa6lBQSja5usHp9KkIEUHvNnV80jlpodaQemmeZX0CG6v3BxueHhB4JS/pqSnopTWRtrqd3KY1UC49FW20hhYTbs7Fwn/0wr+v92xOekeP5qb9L7xHjE1kDP4OYwMccoK7bX39qpHJOTt92uon24nWb6J7qyzUDfL4tahd2aOjXyL/nhn8KaLE6//E4XCFr/Z/SYC0Q7BpiQk35xp/TnAfE5CV1kqWP0v3UZHOwkQr7ROJ30fyJqMprBpWL49TZ4o9lo0acFZpusLfb+r7uy7Cos0FYdy72Ocee8jpZsLeQ2Db9eRlbdDRTyfHa3IaOh7IrAzdEC4ZaSkoLDIcmizuMPhTWPysDSj39dLtpcuyLQ7Y1bZhVY9pKJORO6UfqOQf7bTEjSa9p91zX1zR3niayVAphbDPqVy/WjnsOHQS6akpUT++4cTgHwbl0lMwIoqTYVA0BC6h9zunTukopIG4hydokRV8s8k+Eaq09BcYXxjSDu/8vBmdm9XUzbJmf5tKKZ/+OWOuONewh32o/vPnTpi/scCn70ckRSLbxOAfBuvGBN+6IVHp21tXKW/vdOt9dh2s2Rmd230nvPkX61N8Vq2Qjo9v7Rz0uE9mrVge7Nca9avZD2ZWhooAXAPoPTLQNZdyUbG9UKbPuxt9ktNjXNWslInLz0v8TpgM/hRWfdvUweiBZ6NGxQxc2LyszXYohcB3hlsPnpEXWrH2laHtcfB4IZ78fq3PJ3VtUcv4TSEYqRu50o4xV5yLyat2BV5Rx8me9E6W8JMdgz+FVUqK4LZuvimwBO47E5TBWk78svPq4+YJi7Fi26GYPjZO9SgOxV+6NEH9EOZbcFQs/3GCxKaeRDGgZpR6FFfX+hO4h1MIhxRxzZj1cohDmvTVhncelhv5oRb8DU8eTmzqGYcSuTs4OadWpUzsP2Y861kwP/x2DV0zkoWaIlr6WB8s3XoQHRtbnxjow1ty8emvWy2vLyKYem+3UHYPgGswuHDPiGbmnPqedS6JNEIvS/4UVbnZNXB79+YAgGZZzs/dGindW3kG30Dz9n5/d1e8f5PnjGf39m6JjNSUoAZGa9+oGlb/sx/6hzj2vIigU5MaQeXlL26ZhX9f34nzSYTA6iihbO0Th/h7CE731lkY2bMFbru4KcobTPQRL/7cpQkGtq2HTk/PBICAU0LWq1reY+YsAOjZujY2PBN8S7FKMTJw4P/uvggFxwujvRsxoWuLmpifX9ZR8Y4ezfHG3I2oUj4Nu4/4eaNbBKI/S/4UE2pWynRsGN5oEJGo5e1jxbkNqjregzhenVW3isfzhtVdF3r9KKR+RaAQyeDvkLPqVgYAlLfYDpooGC1qV0LloCeYoWjxjt1VyqVjy9hBYZvDOhQ8mxzyr2vbY/WOw6gdxV6BFH1f3XEhdh0+6fjn+psdi8Lrvzeej5ve/w2A9QYd8VAtzJK/QypmpqGzzTmBKf51alIdl7ZL/N6hySTU5p4LRl2C3mf7vvfe3i19lhmtF24M/hQV1+U2RuMaFXBVR2cH/iKKlMw0/+GzfrXyyDRIA6enxkbYjY29oKTTqEYF/PRQT58WL0SxzIm5tzMY/ImI4sugdvVsf8ZlVgaNY1NPIqLYFGqfnrpVy+G6zt5DVUS+gxBb+xARBVAxIxWVwtrU1quoH4FrAYM/EVEAK5/sF+1dcByDPxFRAKlOTEMWY2zl/EWkhojMEJE87X/ToQFFpIqIbBeR1+1sk4gomt67MQePDDzLZ3l2EMNiB+wsFgcVvqMAzFJKtQQwS3tuZgyAn2xuj4goqi45qw5GdGsO0SXmt4wdhLkP9ix9HvzQ7uL3aTjYTfsMBtBDezwBwFwAD3uvJCKdANQBMBVALM/FR5TwZt7XDcu3HUaP1hyEzQkP9/e9Cwgs+gNA2C3511FKuSf03A1XgPcgIikAXgTwgM1tEZEDWtSujCGdGqJWko9C6pQbLvAdqdPdDPTanEYAgA6NqwX3oRG4NgQs+YvITABGM0WM1j9RSikRMdrlOwFMUUptDzT5g4iMADACABo3jvyUbUREVvkLZ+60T7dWWRZnIYv8nUDA4K+U6m32mojsEZF6SqldIlIPwF6D1S4AcLGI3AmgEoAMETmmlPKpH1BKvQXgLQDIycmJ/n0REZGJeJ+y1W7aZxKA4drj4QC+815BKXW9UqqxUiobrtTPB0aBn4goWRhdOJ658tyyJ3EwmctYAH1EJA9Ab+05RCRHRN6xu3NERLHKzpStvsFf0LpOZd0KoX+2VbZa+yilCgD0Mli+GMCtBsvfB/C+nW0SESWiSGeROLAbEZEN8Zr6Z/AnIgqBv6xP1xbBz+qn/7xIXFAY/ImIHNbrbJ8uTx6UV3i/1IF5AoLF4E9EFILx13dE56Y1UMFgqsZgbPq/gbiiQ+SnM+WonkREIejRujZ6tLY38fpzf2qLFIMRQyMxhihL/kREMYY5fyKiBBQLvYMZ/ImIokSiMHevG4M/EVGEGRX8Iz3KKoM/EVGEVdYmgy+XUdZSKLtWxdLHKgJ5Ibb2ISKKsAf7tUadKuUwqK1x+/5Aw987gcGfiCjCKmSk4fbuzaO6D0z7EBElIQZ/IqIYE4mcP4M/EVESYvAnIkpCDP5EREmIwZ+IKAkx+BMRJSEGfyKiJMTgT0SUhBj8iYhiTCRGfObwDkREDnmwX2t0bVEr5PeLuMb6b1WnsoN7ZYwlfyIih4zs2QLtG1UL+f2XaNNCXpPTyKldMsXgT0QUIzLTXSE5zWBeX6cx7UNEFCOevqItsmtWRLdWWWHfFoM/EVGMqFExAw/1Pysi22Lah4goCdkK/iJSQ0RmiEie9n91k/Uai8h0EfldRNaKSLad7RIRkT12S/6jAMxSSrUEMEt7buQDAOOUUmcDyAWw1+Z2iYjIBrvBfzCACdrjCQCu8F5BRNoASFNKzQAApdQxpdQJm9slIiIb7Ab/OkqpXdrj3QDqGKzTCsAhEflaRJaJyDgRSTVYj4iIIiRgax8RmQmgrsFLo/VPlFJKRIx6JacBuBhABwBbAXwO4EYA7xpsawSAEQDQuHHjQLtGREQhChj8lVK9zV4TkT0iUk8ptUtE6sE4l78dwHKl1CbtPd8C6AKD4K+UegvAWwCQk5MTieEtiIiSkt20zyQAw7XHwwF8Z7DObwCqiYi718IlANba3C4REdkgdmaJF5GaAL4A0BjAHwCuUUodEJEcALcrpW7V1usD4EUAAmAJgBFKqcIAn71P+8xQ1QKw38b7kwGPUWA8RtbwOAUWqWPURCkVsIuwreAfy0RksVIqJ9r7Ect4jALjMbKGxymwWDtG7OFLRJSEGPyJiJJQIgf/t6K9A3GAxygwHiNreJwCi6ljlLA5fyIiMpfIJX8iIjKRcMFfRPqLyHoRyRcRs4HmEpaIbBGRVSKyXEQWa8sMR18Vl1e1Y7VSRDrqPme4tn6eiAw32168EJH3RGSviKzWLXPsuIhIJ+2452vvDf9UTA4zOUZPisgO7XxaLiIDda/9Q/u+60Wkn2654W9QRJqKyCJt+ecikhG5b+cMEWkkInO00YnXiMjftOXxdy4ppRLmH4BUABsBNAOQAWAFgDbR3q8IH4MtAGp5LXsewCjt8SgAz2mPBwL4Aa7+F10ALNKW1wCwSfu/uva4erS/m83j0g1ARwCrw3FcAPyqrSvaewdE+zs7dIyeBPCAwbpttN9XJoCm2u8u1d9vEK4+QUO1x/8BcEe0v3MIx6gegI7a48oANmjHIu7OpUQr+ecCyFdKbVKuTmSfwTXyaLIzG311MIAPlMtCuHpi1wPQD8AMpdQBpdRBADMA9I/0TjtJKfUTgANeix05LtprVZRSC5Xr1/sBDEa4jXUmx8jMYACfKaVOK6U2A8iH6/dn+BvUSq+XAJiovd9wFOBYp5TapZRaqj0+CuB3AA0Qh+dSogX/BgC26Z5v15YlEwVguogs0QbKA8xHXzU7XslyHJ06Lg20x97LE8VdWsriPSmbsCnYY1QTwCGlVJHX8rglrkmpOgBYhDg8lxIt+BNwkVKqI4ABAEaKSDf9i1ppgk28vPC4mHoDQHMA7QHsgmuYlqQnIpUAfAXgXqXUEf1r8XIuJVrw3wGgke55Q21Z0lBK7dD+3wvgG7huw/dot5MQz9FXzY5XshxHp47LDu2x9/K4p5Tao5QqVkqVAHgbrvMJCP4YFcCV8kjzWh53RCQdrsD/sVLqa21x3J1LiRb8fwPQUmtVkAFgKFwjjyYFEakoIpXdjwH0BbAa5qOvTgJwg9YioQuAw9qt6zQAfUWkunab31dblmgcOS7aa0dEpIuW274BxiPcxh13QNNcCdf5BLiO0VARyRSRpgBawlVRafgb1ErDcwAM0d5vNgpwTNP+vu8C+F0p9ZLupfg7l6Jde+70P7hq1zfA1eJgdLT3J8LfvRlcrStWAFjj/v5w5VtnAcgDMBNADW25ABivHatVAHJ0n3UzXJV4+QBuivZ3c+DYfApX2uIMXHnUW5w8LgBy4AqMGwG8Dq0DZTz9MzlGH2rHYCVcgayebv3R2vddD12LFLPfoHZ+/qoduy8BZEb7O4dwjC6CK6WzEsBy7d/AeDyX2MOXiCgJJVrah4iILGDwJyJKQgz+RERJiMGfiCgJMfgTESUhBn8ioiTE4E9ElIQY/ImIktD/Aw4BBH9FkENNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# you can skip this if you do not have matplotlib installed\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline \n",
    "\n",
    "# show 1 sec wave segment\n",
    "plt.plot(wav_segment)\n",
    "filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAAC8CAYAAABPAdTWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnX2QntV53q+XlbSLVtKKldhdJCStpF30gWQWsSAigSU+HIixMbQwCa5dux1a4s649XjcprUzTe1pmqST1hmnbeJWM05cWje2YjCOKRAZEAiKYAUSYCGhrxVCsj4XraQFraT12z9oxN6/c9jnfTOTyaPO9fvv3vd5zjnPOec5enTd97lPpVqtyhhjzN8+F/1tN8AYY8z7eEE2xpiS4AXZGGNKghdkY4wpCV6QjTGmJHhBNsaYkuAF2RhjSoIXZGOMKQlekI0xpiSMq+fiSmVSVWod9Rfu8hufu6vgmnOw34V9MeyGD2/gh8I6zsKekLnnDOxG2COwJ8Hmc0jps7MM2k2whwvaJElDsCfDHiioIzcl2Be8hmPyC9i5/uWz8Br2Fa9nm3J1sB0ck4mw2f/NmTI5lwZhXwKb78ipTJmc46yDv3P+0s69hxwzvpfsX7Y7Ny9YL8ssekdOZ8pk29kOzrXcezaa3Ddnve824XhI6fzjOO84Wq1WLy0ouL4F+f3F+CujbE6ctsw97OCZsLlAvAx7UaYN9XII9uGCNknSfthdsNnuVbD5HJLUDvsk7GOw+ew7C9okSS8WtOv7BWWwjZLUX3DNFNjvwc71L8vkNZfD3lFwf2emDraDY7IMNsd0ZaZMzqXHYN8Nm+/Ic5kylxa0g/OAbeBczfU33xu+l/2w2e7ce8d2sEzO59Ww38iUybnFRZ9z7dVMGaPhP7pS+qxF7zbheEnp/OM4//LegkIlWbIwxpjSUOcXcrOk5aNs/gu5XMXwX7xu2PxXll8xOU4U/M5/AfmvbO6/IKAJ/9KeLvpvIr8wpPTZ+eXDLzp+lXykoE4p/bLkF8cXYbPvcmXyb2z327Br+ermV9y0aE7Hz8dRxrncVyCZW/A727Uadu7rqxM228Ex5Jjlvr7w3oxDf5/bg+s5X/nFlpt7nBccd861a2BzjKX0WdgOrg987zJf3ePQX+fYTraDZa6Gnetv/k+efcMve9bB91hKv8Q57rXhL2RjjCkJXpCNMaYkeEE2xpiSUKknQX2l0luV+kb9hVoLdEBJqd5CTYiaG/VKhm3VAvUcanD8PVPHOFyTyHIMt2G7qZ9JxeFJ1KrQV5OgdZ3KaVlF2iAjO2rRx6ilUl/k752wOcaZejrwbEtw+WbYR1leZh5PRRjWcfYNtViOB+eNpCbo0qe34gL05zjqkxkYeZhExnEu8Tnoh8nB+coJDXdSJ+Z/TkI+V/T+cy5Rz835FopgmZhb49D/53LzuSgEkLDv+A5JybN04OeDlU3VarW3oCJ/IRtjTFnwgmyMMSXBC7IxxpQEL8jGGFMS6nPqjeutanLfh19wnCK/pCYI/fQlJIHfdLBBcOd2eCnj3KJzBr834ffclnrubaAT6XhBmVMzZR6HnfQFbG6OYDtz23pYBh1E7L9TcBgtzDhauGO7EzYdPqc5p7hVXdIk1NOD31fDXgM7ceplKEqdQJtjdjzzbqzGfHy6oA107uT2bHCc+2Gfxns1He8Un7M/l98BDkw6PAnnKtsopf1H5yTfoYOwc2PIMtlfibMsU8Zosu8IxxV9wXuSMcutmShjIX7eZqeeMcZcUHhBNsaYkuAF2RhjSkJ9yYUuVgzYp5xMvVhKtSeqKA8jsLsHgd3UQKlnSplkLGwDfqc+Rr0nWwbs4yiTOl4uMya1VupfrJMaWy2pixONGHpXE7SuU9g40p8pkyRaLDV8NKwpo0tTr30BNseZ2mGR/ptpRtLfLJP9m9NZt2XqGQ3nCduQ84EU7R3ZmdtwNQo+R39uIw42evAdKCozB3039KMU6b85PZ3vSG5cR5O8lzXcz3FlO/rxXJ14Lr5DUjqubFfRvPl/+AvZGGNKghdkY4wpCV6QjTGmJNSnIVcV9RbqJrnSqOEUJenpL7g+pwlRT2S7aB+Frno0owlRA2a7COvIJWOhHk7Nks9BHaogH4ykVCe9Ac9G3b8L+liu3Xw2toPaIdt1fabM12EzmVBRvGktM5dzJRdvPho+Z05nZTtYB22OeS26aVF/k0RvL9CcpbSdRVo3x0uSLqe2qrHt/oI2SMUactE7wHcmp9kXvWd8LpbB55LS+Zo7OrEG/IVsjDElwQuyMcaUBC/IxhhTEurTkEcUtZGjBfF6OTbwD4g7phbD+L2cJtQJmzpUP+wO6Kq5/fCF+9kL6sjt/S/KM1EUF8vrczoVY6qpg1LPZf/m4k/7YVPXK9K6c1psUX/ynqLcCrk4T44rY+DZhlpim1kv51pRzpOOGt6RbfBx9GC+sp2cB52ZMuk74LNx3KkZZw9DKHiWolwtOV2ayfffLlgf2P/8PTf3uIYU5cPg77k1iM+Wu6YG/IVsjDElwQuyMcaUBC/IxhhTEurTkM8qajbUjHN6DXVQanCnEDNJDYgtzLW4qA7q1rw+l0uA1xTpTNRucxppUTupqRXpUIzdlYrjpfkcbFOuzhtg89kYB0uNMzcvqGGyjFwukKIySZFWWJRbIZt7GzbHnXWcriFPNnXnJQW5Fqib1nCOamGeYZK0M6MXF8XaFsUU59p9HJpxkR+lKF9JZ6aOfmj0nejvflzPMc6tQUk+kcw1NeAvZGOMKQlekI0xpiR4QTbGmJLgBdkYY0pCfU69SsEdOZE/SewDm84bOraKAtil4uDwomRDuaT3pxGg3jQlc9EYZeYS2RQl1ClKTENnxMJMUqSiDS206bDI9UVRkpiixDS1JJHhPChygNbiOGQSKSZa4njQIZRzmpLNsIsSRuUO9uTfeOBrf0GZtYwh3wleU7SZJzefWSafg2PGeZBLZFWUTGwnDuVVG9pUcGCplDrxWAc3TyXJtDJlJocbZK6pAX8hG2NMSfCCbIwxJcELsjHGlIT6NOSJiglaqAF1Zu4pCr5nGdSqGDyea3E/bGqvLIN6Ty6B+lFoxkWJatju3MYGaqv9sDsLrn8b2ldOg+OzMulO0YaLWjbeUOMs2iiS1fEKrik6iHYd7Fwyp1MFSaRYB+dqzl9B3Zn1sm+oQz+dKZOwLzph/3WSC/FvnBfsm7tgJ0nBlM6LW2E/DZtzM6enFyV4uhUH5vLZOaZ8T3N1FCX9KtpEI6Xj3F/DPRn8hWyMMSXBC7IxxpQEL8jGGFMSKtVqtfiqv7p4Vm9VXxqV6ZoaZi7+kYlpGLdJPeczsKmn1ZKInMm4qW0VJSzJtasoRpgaW053os7EdvB3amy1KP7UNKm1UkOm5lbL4YzUtjmmvwY7pxVy7nCecJypP/L+XMwwn+1h2KthPw27Fv2xKMadY5ZLys4xYX8V+Tyehp17D6ltFx0iy/7OtbtIi2X/sS+ezpTJZytK6kU6Yef6gnPlBdh89lrikIt05t+sbKpWq7kZFfAXsjHGlAQvyMYYUxK8IBtjTEmoT0Oe01vVV0cJtNTLVqcb3jtmHgj2wV1z4wU7EStatK+fOqCU6mHUe/8aydBbboji1eDvo5KieN/cXnbG0rJeamyPwaaO94lMHdBNL7p9KNht7TEXwMH9M+INRzMCGTViPluShwJzimMs6aKe2K4b258N9vpNt6POWOavzH8o2JN1MqnjgOKzbVj/sXhBZzy4c/yk94I9Y1qcu5K0dzsGEfrj5Z/bEexzagj2bO1LyjykGFvbrjhGL25fFW+gNo7xWLTg5aSON7YvC3Zr1/5gr2h4PrlnNH/x3L3pHzvj+z5pajoGozm17dJgtyxJBeLOxv5gb/lJFJWbbhgI9um1rSgABeYOGeD7z/cSc03H4/xt7Yl9J0lTG+KgTNS7wX69cp01ZGOMuZDwgmyMMSXBC7IxxpSEujTkxt4l1cv61p63VyjqTrs0P7lnFjSzk5oc7J0QIGco6na9CCqm3iZJxxQPSn1XE8f8faleKyyT7WAZl+H391DnYm1NymS79mnWmPewr9bqnmDfpKeSOvhsDRoJNnXVw8gny/HK/W1YE4K9CQGnnBe8XpIe1R3BvlOP4J7GYFMjZt/l2n0G9W7V4mC36XCwqd3m4LxgO49D0J0Gp8g4jIckvaKrg8150Kdrgv1zjCHnYk5P5/zkXOS84XPwuSVpJ9531sExeU1LkzLIVAjkG7V8zHbwHaE9PxOIzHnA+XoJ2nAUwf3v6eKkTPYf2/Glyn+1hmyMMRcSXpCNMaYkeEE2xpiS4AXZGGNKQl0J6ru1Q2v1QcA+RX06WiRpomKwPZ0FbXCknIIYfqPihgE6qXJl8poGZP64alsM3h9YmG6GOAYhf4KGx2wnnWU5JwiFfjoTu+CAmKAzwb5RzwS7V5uSOjpeH4x/iD4n7eiOO1r6EUmfazedXycRbU9HzHJtDPYZNkLSDP082Ev1arA5b6YNxuda2/KpYN934EdJHcI5tUwAU8WemGE0c3PzVUmRPUNbgt0U9ynopVkxcw37k/NEkjq1J9if2vZEsLsWxnkxCU67OduOxAJzCaLwpu/tiZs0+A7tgrP97xz+30mRA23xveE9dKh1Imt7zvFNViMD0TrdEuwbtSvY7O9cHXsw53uH4kaaM03jg/1uw9jOeEm69lHs2orLhb6U3JHHX8jGGFMSvCAbY0xJ8IJsjDEloa6NIT29DdW/7PtAT6H+SI1IkrbrimBTq+qCBvS8VgSbeiSDtCVplt4KNjXLWcMxQL2/sTPYDO6X0uB6bjLgPdSu2DdSqrV2Du0NdlOUEhON85XWRcFetuONpI713dcFm33zsO4O9gPD3w72hNO/SMocD530LPK5jECfbIBWu7ElJraRpKuHY8aiVxpjFqkb9iFBDh71bNwvoH0t6Ymk1KE7DkQdesuM7mBT48/p6fO2jZ0x/UR31B+nHIgJjAZmpf6KcSPR53HxKSQ9eg43LILNadCtFLgWkn00TNreEs0dbcymJXW/Hk8JOLEoPvu5hphYafJgTEY0PpP0fvfKOI6H4WfhhhZuPuF7S9+PJLX2IQkaE4PhfOOaYH/inaj8urwxxBhjLiS8IBtjTEnwgmyMMSWhrjjkEV0UYlCpsb2bSbqRarFRe2XimYuR2Hn+0O5YR3PUkKRUK5x3YGydb3hGbAMTwEjS7IEY28kY1deao+jG+FJq5VLaF9ubo9g3d1F/rLMhVtpIPSwzeqsOvBjsE+1R1/vysT8K9sG2KBaebIw6qiRNaIn1MrlNLi5zNNN0LPnb0caYrKlIM2b47rGW2O4ZQ+mYN6X5hgITZ8R585GBGJ9+tDXNbr53YYzfzSXyGc3ZKVEPzsVkjzREDXnK5niPYi7/dNynqRhowpxKp9G/jMXt3sNTZSUO65Qfot2CzYNA56VFzns9juOMudE+0Bw15nmHMe6PosC0u5NnT4DPBPm3hLD894H/R80FdXwI/kI2xpiS4AXZGGNKghdkY4wpCXVpyI1D5zRv4weazVBPXM+b96QxrMcWxrhh7m+fMRi12q6mGJfcFGU9TeyJGrOU6rUd+xB0iXDSZSshUFL/kYR802pCbu1rG2MQ5e6eqG0t25PGCJMjc6NGOXkwam6TocHta4VWG0OM3yfm89eUE9DxoPt1XB/7qmMVA1YlnNOpM9Dg5yOWnIywgNzfqP1h3PXxaHY8iXbm8jdshI346YkL41yqbIi/XzouLfTsyvg3xmgnYIw6WtL+rc7GH9gX1CxZJ84NTrRaKdWZ8U40oQ1Nb2HeZKZFEnvL/P7MJcLuzGmxWJGa0M55LdCMOU/4LuN8WEmpvssyGMfNs5sxjyRJX47mWdbxjcw9GfyFbIwxJcELsjHGlAQvyMYYUxLq0pDVKFVH6SvbGxeEn2ctTAM/lw5GLbW/Je6JZzxp+0AUq3b0xOvPZfRIxioPLI/5AkbwmDywdGpbzDEhSTuXx1zPjDelzTwVJ2bHOE4pzavKfNLTFWNxT7bEMphrIclpIGn/qihwzfwBBMfrcQM0usdmpaLb7TvWB5vPeunmKA6eWDp2PlkpHccdD8RxZmz4JQNRyGPf5OroGAfhE/3FvLgzl8a+qjJ2V9J4pqCGvlhF/1YYa5vRd99pjfO19Z/FQqlHHm+JvgfGgfOdk9L3rKMx9s3PZsWGzm+N71SOJnTvz2bEMq7ciDLw6lYzcciVF/AHxhGjyBP3xXmwryH2BdcGSWpCXHcVmnEF/qOaYoqhh5/sYc4SCtF5/IVsjDElwQuyMcaUBC/IxhhTErwgG2NMSagrQf283kuq3+j74JDBqXon/E7nmZQmdid07jD5StH9UnroI5Nab9XiYDPpPRMaSelmk5uSwxZvDfZS7CSZmCmTmyHoVOpHhD8PZ6XzZnJmN8RhZELhAaVX65Vgv6q442Vu5pCBFXoe7ewMNpP1s3/pRJXSA17pxBuHg2o5Rm8qOpQTh6fS5OU8NLYdOxmY/Pxx3ZaUebceCvbMfdER+L1Z8fBVtuESpQ7k7XgWjjPL4HjwUAdeL0kbFTP68zk43zmmnIuStHgw7qh4uuWGYPPACj7XAm1PyuR7xQOLeRgu33UmK8v1BdcpvjN8lzlvJg+l711/85xgs91XVN52gnpjjLmQ8IJsjDElwQuyMcaUhLo05O7eKdVv9l37ob/nNM3v6PPBvlsPB5ubDIoOLM3pvWt0f7Dv1I+D/V19Nti36fFgP6uPJmVS3ypKoLNOtwSbmrOUJrGnHtYIreootNd26ME5XY+aGqEeRo2edUrSMRwsS02YffOpfU/EAjIx8UOz47dAw7mYmGpXc9w1QH2XfdmZyRC1WVcHe7WeCjYT7VNv7FMq+fVAg+cGodbD8WG/1xY15ZyGzHegZ2hLsJks/pmGG4NNTTSXNJ91bFQ8DJfvDPs7d/gE++s1fSTY9KPQh0LtXEp9AV1IRsZx5yEZ9Cdx7kr1a8ZFbcoxbSj6Fi6e5ENOjTHmgsILsjHGlAQvyMYYUxLq0pBn9HZUH+j73Hm7R5vD7zntignp52yMCel3L8ehhU8gATUPGMwkBD94MxKnrI9ZT07cEDW4hnMjsNPE+k2shwlHkHhmy8qYoYT6pCQt0tZgzzkc+yJJOM8QSibSziUiZ39lEuSgUYEjS9KDPS89HH0D+9tiAiPq+q0PQjTmIZ2S1AObz86kMldE8+BCjPkjmcTvMSxWlR/g93vxOxPDL1cKx4TtxBjt+HxMmtQ1kB4WWmHTcchAkhyeU4uHdubmBRi4FwmNOGbU/XFAgCTh9dfZlWPXOZ59l0nac3AuxnUjHh5nNAxNi9+U6xpjHDP1Ykm6RevGbCeZPhJPdTjakPpZGHf8GuL7P1l50hqyMcZcSHhBNsaYkuAF2RhjSkJdGvLS3vHVH/WlcX1/RS4OmfGL1E2rSH9BPW393BgvmcttwT3z3LdPGiG6TePJn0rjTZnXgPGkK0ZifoHhBoqL0iZdM2a9a3VPsD+O0y6nI9/DlYfT5Nsvt0VRmPkamDuEOSOY90OS2lAGY8Of0upgMyfHrRnNjjGs7F/2/y51BZtxtbncIYuh2TN/BtkMYfsd5HfI3xNjnW/UM8FmbpDc3LxRzwabOR84X69RzJLPPCDULyVpot4L9iFoq10Fcfa5ecExoA+J/cnxyJVJfwTXFNbxqY2IeYcuPbCEieKl1i9BIIcWrimw74f9J0mRqa4Pv0nlTxyHbIwxFxRekI0xpiR4QTbGmJJQl4bc03tR9cm+D/RD5lDdp9nJPdxr3j4S9cgpb50du1Joyqe700uYi3Thnr3B3j03xjqneYejjiqlWjW1LWqDMzcicDkNQ05iVk9MSw9CHQ3jHamnXbon1exzB3OOpsLzL4vilKU01pb3MIaYcbH8XVIVqYaPtsb450ufw8Gp1499qGnHtjQOGSlLkjHZfS9i4J9EDPyStMgftH0i2IzFp5Y98zDmBQ6VlaSXemJF1655PV7ANB1sF+daJlaf9Z79TLTHx1QWaSx0jlbYuTk/GoYEp8tFGkPNucTYZXQV9d8jd2bi6r+H9+Y5XIA1phpT4aiyJiky7QtMx8pXrCEbY8wFhRdkY4wpCV6QjTGmJHhBNsaYkpCeSjoGp3WxtmrheZvJn+nky/1te0PMErN4bgwWZ+D8HSNxc0QusUcC85EMRmfN1pYYkM7NEVKaCLvwYFQ4vg7OSr1lTNBNZ+K0wdjwyS3R+ZBLSE8qTOTDW7gHht2Z+gmTWbKjLSbM6R5Ewhy2IZNEpoJ2TGfFqHPyIJy/Lej/TB1JO2AnB6M2xHnCJEpSupmBm5Im0AvFNqT7FJKNN4kjizaddtzv0qAUzINjLUji04iXhitDrn/5N84lOoM5FzMOZTql97SO7ZCf3BM3ivCQU27ckaTb7ove3jP3xXt2YhMSN97s+hfxdyl30AM20H3lD5J7cvgL2RhjSoIXZGOMKQlekI0xpiTUtTHk4t5F1a6+7563/61+M/zer87kHmpshBtHmFiFOl8uIQkPHeShm9SlH81m245Q++ZzUDNinblkN+nBnLFM9h/1SCbYobYlSW/i4Mi7cKgsn4uJaHKHDLDd1MJXKCZW+u5gjKS/teWnSZnsH9bBAzDv1kPB5iG0RQmlpHQTBxMacS7mElkx4dACvRlsHlzLQ2dzm5B4aCznBe8pGkPqqFKaXIjJnTj3OMbXwYeSg3OHc5F15g457UMCrtsUkwe9h2Rle9BuHi78bT2Q1MHkTJxL92htsJmUKnfIKdtB/9pdlSe8McQYYy4kvCAbY0xJ8IJsjDEloa445HMap6Oj9K5v6Yvhd+omUqovUgOmbsfE2UwEX0sioFcQe8gk+bw+Vyb1xgO6LNiMXX5eK4LNuGUpTdBNvZy6Ksuknp6L+yZv4nTQYQSHsv9pS2kCdR4cuU7xYMnelqjRUTuUUt20yJdAm+QOGeCBAOx/JvxnHbkx3IlnnYh5wuT91B95YIAkXYZnpxZbpI9zHuT8ANRJGUf/rG4M9ruKyZs4BySFtUCSpmMMcmMymiRmW9IM/RxlxjGiVsv+5nv7G/q9pA4eqEDoz2DfUF+X0jHgOlYr/kI2xpiS4AXZGGNKghdkY4wpCXVpyE0aDoch/n399/A7tVpJuhqazv/Qp4P9n3b982DPmx91vjv1rWDzYEUp1V6LDrdkHHJO46T+Rc2TOtIjujPYq/VUUiaT2vPAUcaTck/9Gz9bFuxPX/n1pA5qsXy2nZof7F7EZLLvcjBm9RlobDzUdDViQ6W0P6nb0eYYM1Y3p/d+XD8JNv0ZzHOwC31zS+ZwVnIX4qN5WCjjYHPtbNPhYC8djKcITGuJczH3DsTf30n+Rm2WeVGW68Vg/5l+Ndi/qj9LynxIdwWbh/ISzu/cXHsV13AMXtNHgk1fDvXeHIzJpkbPuXkxrs/tt6AfpWj/xYfhL2RjjCkJXpCNMaYkeEE2xpiSUJeGPE7nQhwl9UpqRLm/vYf4xuvmPxNsxgRTm8nFWDKekTGBzD/AOnK60yTUM4KuoobMOh/W3UmZzHvLuOJGnYEdn2vOlduCncvrsR1xx4sV9citWhTs47ok2NRRJektxOcyJ21RvpFczgLqdNRRN2z8WLwBIdfnFsT+Z5y4lMbFEuqRHNNNSlMPcM6/gWdlLDP9ALkx43y8qSX6H57STcGmPknNOOcTeRE6Ketk/gv6g3Kxu0+jXexv+kgYV5+D/UONmO/yd4b/QbB7GqPPKpdDnPN39/445ztmxjHmc9WyFyKXy6YW/IVsjDElwQuyMcaUBC/IxhhTErwgG2NMSajLqXf8zCX6872jAsbnxN/pCJOkP917f7C/MOcPgz0BjqwN+6ODbdnMvmAvzgTFP6uPxnZC+Gdid4ryW356fVLmL93yZLC5UYHJWpbqNVyfOpn4rA/uj4ncV818OthFzppcwp2bsCHld/Z/NdhNk6Kz4UfH74ltmJMmk6djhI5Z2kxMQ0ejJO342VXB/rtXPhgvwDAvWv5ysJlUhk4rSdo6FB1EPc3R4bN5qCfYdzXHZP6bFX+XUgclNxBxTJgcJzlYVWnCJ85fzj06Xuk8yx3aS/oORYdlb3t8z+g8o1Nbkr7/08/Fdt4y9mYezv+/+O17kzKXfO2lYP+u/mWwV+i5YDeMi8/K9y63+YTvSC/WmD44czmfj4+kSb2mNRSPcy34C9kYY0qCF2RjjCkJXpCNMaYk1KUhq1LVRU0f6EDUSf788c8kt4zvPRHsH+uTwaYG94WZ/yXYDLhmoiApDegv0jSToO2mpMiknjRwfuwyT2U0pM0oc9LUqLmv3786Ngt678RJMcnJQH/U7CRp6vwovrZMj/byxpjc5mRLbCc1TylNoM4NFYR9waRKknTmyqibPnQobqTpvm1LsKn7c8xfHUw3Jc1tifXyUNgVzTHZUK+ilsgE7O+XEecrdWZqltR7c9oiyyxKbn4UB0FcrVeCzSTuUnoY62XtcUz5zlBDzh34esMtfxlsat3cRMPEQau+9lhSJvuPhyEwydGshuhXuRuH+vagb6RUD6dmz+RB3Iz2Ww1pUq8H9O1gM7nY95M78vgL2RhjSoIXZGOMKQlekI0xpiRUqtVqzRe39HZVr+/7D+ftKxCLyyTXkrRUrwabybWp3ywY2hHsR5t/BeXFeF8p1QYZo0pNiPGQTHQjpXoYExi1IxkO25CLBWW8KfXEf6XfCfZn9d1gM6E6E65LxbGfHI95+w7GAtIzTqXBaB755UnBfly3BZtabC3Jm6i5f0v/NNj3a02wP6lHgv1mJoHRHyM5/MMH4uEIyRmbb8F+TikMnW2GvSeaf7Qyxur++sCfJkVWkP9mS1t3sDnXOG86B9+Ov7ekTpFLBk4H+63WS4M9Y/BIsNe01NBu5G66pzseWLH2cIyzf7ktJra6eiAmvpKkSgwV17ab42aHhRv3BnuoJ35TPtIYfVT3Hf5RUsdAW+wfHs7MxErsf64NkrTsAJ4Fr39ltjZVq9U0WxXwF7IxxpQEL8jGGFMSvCAbY0xJqCsOeZqO6vP6znn7VsW8B9MHTiWMdbUJAAAJ0ElEQVT3VKA/ssaRWTGedGRc/DfiGuiROW22tS/qY9V50e44F2Na97ZF/Yz6ryStHtwQ7K0tUdejjtc1HA+3PNCY6kyMpW1Hwvp/rW/ENgytD/ZrzfFAUsZGS2mM5aoDUdc/S81zd1JAylA0Oc4bW68LNnXqnObGGFX2zT/Rfw42D6Zk3CxjeaX0AFedwAWcm2wmNWVJ2gGbIcPQkL8wF9or75dSHRplTh2KDZ3YFP0E4wfi9RPHxfkvSSdbxgf753jYOcNRQ+aBpZVU7k00+C92xzw1eHW1bCkKwbySJA7jwgNja8Z8zzj/X2pbklRRlGeCPg/6vei/kKQfzoi+rtQvhVwtH4K/kI0xpiR4QTbGmJLgBdkYY0pCXXHI3b1Tqt/su/a8zUM5c3kmblQ8xHT2QNSq3mmNMYETh6L+1RSlLGXOndTA3FgGDzFkjlrmFd6ka5IyGeN75b4otu6YdXmwu/fEWNANc5clZbJe6qDzB6NeNh4a24n2qANO2XE2qWP3wo5gzzuMOGOWOTuWOXkwLTPRDynX0hNB+YwaqZTqszFEVVvmRs1+7kh/sKcciu08OKMlqWLaYNRex1NDJjwPkxqzlPTfkZUxJpuHdK7ahtj89MxNqRU2Y8EZLz07U8YoTrM8SQea47zgu8vcLN3Pxfmcq/OxWauCfQ00+0t3RF9DFfOikm4pSHXldItALDNOE21tnRfs3IGkC3fE9yw5ExZpxk/Mi+/IQw3pAcaM/39Edwb7wco/dhyyMcZcSHhBNsaYkuAF2RhjSoIXZGOMKQl1OfV6L6tU+/7hqD8gCF5fztz0JOybYdOB8WPYDLxP47wlnlH6Q9gM+Of1uSQyjK2H0yl5djqp0vMxJQTwi84ySv4sEw6MxBkhCXtNUofQYdh0uKW+sbRMOl6Yy4Znr+aceuwLOgbpZ2Wd7P9cu3kP51rG+RVg3+Xu4ZiwTvqUcpsh6DzMOf5Gw3eC86Do/tw1LLMW5xrfiQbY22BzHnEMpXR+clw5lxbC5lzM5frnPCjaIMQ2cMyltD/Rf5XVTi5kjDEXFF6QjTGmJHhBNsaYklDfIafDislRqL3kEtPwmidgUzOiFsPfkcBaUpqwhRoRtSraOa2QZb4Om9oU20k71y5qxAU6VHJ/TtfjPex//t4IO7cZgv1DDa1IJ033CxVr28/Apj5Jctosn40UbcDIzYuiucQy+Vy5NlEX5VvJTRksk3Xm3mr6RDjObAO12Jwu3Zf522iK3u2i8ZHSd6RIP+dz0lchJb6bV/Fuc7riSGQcrfA+3E71XuaaWvAXsjHGlAQvyMYYUxK8IBtjTEmoT0OeKoWcGdSEcjrTLbCL9F2W8QLsXIt5D6/5LGxqX3OVwvhGxlSyjiKNLgejEqm58fei+MlaYBwttddc3CbjMJmkh8/OvsnNC+qHHANqf4xLvgL2TzJ1cG7RD8D+K9Lbc/A52FdFmr0kTYJNjZ7zYkpBmTlfDvVwtou/c57k4rzpw2DSKc4lauHLM2VyT8B62NSMC3TqQxvTKor0Xt7C6Z5z3fTDZsh1rfgL2RhjSoIXZGOMKQlekI0xpiTUpyFfpBivyNjFXOJs6Fv7b47i1MxBiIWMO7wBdnp+Y9oOaq3UYqmf5TRk6nLc3844TZLb716k17JO6qi8vpaYYdqsg/purt3U7agvsp3MZTFPKWjHQFvs0NYn40DvvTkeTDuCwOR5QiJ+KdWVOWbUPNkXuRhWxkNT/+Wzs/9zejrbxTHg2cEUMYti96VivZxlUjPO5f3gO1FwoDHzuzzWvUrk9hkQjdkOPhvbhTa1L02qSNpxOcvgXgeuJ8zhkbsHc+vrzOnzIfgL2RhjSoIXZGOMKQlekI0xpiTUlQ+5q7el+u/7Vp63T2py+H2dbk3u4QGj1P7u0dpgP6sbg81DI/cNUaST5jfvCvbdeijYPMBxAoS9f3foa0mZ5BfHIVQ3xWjGL8z5w2CvOXZ/UsbZF2IA6aTV8cDXqc3Hg/32/visl8+MYu4EHFQppYct8tk5ZiMjcTwWN2xNypyvncE+jN3+7M9d6hqzTTl4QC4PhN2wK86tm+dHgfjZYx9Nyrx7WpwHaw/dE+zJU2NWgsF18SDQi3rTQOTF7bF/Xv/JtcG+6o4YOH8AouaR9amj5fJVMXEKx/2qmVGg3LI/iqDLZsbA+n0ZZ86RjfFv47tidO3Zg5ibnXFunno4aviSpBsQ0fsH8TDQjm/Gg4EPH4pj/ItzaYKS62bGKOAX966IZc7JJQH/gC7FtWDryOL0moY4n1/cFLXsJde8FOzjmhrst9enjpaLFsa5Mr891rGj0uN8yMYYcyHhBdkYY0qCF2RjjCkJXpCNMaYk1LUxZPe5efq1Y//rvN01LQrXS/Vacs/zx6Io3zAuRrHPaoki/brh6Lx591R0Sp1dx8wq0pYl0eGwq3N+sKc1xyhtOhYXtG9Pyjyq6cHuao/Ohv+zKZ7WunZOdBixbyTp6B2xzF5kOdo4gmwrp+KOgXdHYl/QYSdJ0xuOBvur+u1gv4d7frfhN5IyyM/hmKIz8XmtDHYXnIB0JOaYnE37/QEd82OmoJEapm6/OoM9rT3OgyPbo6Or9fb9wX731MVJmYsFp9706NQ7owmxnXCatlyfbmAZxi6ZT8yMJ/3uUpzPdPIdw1ydkexkkE4ujDtYTm+IuyGW3bEh1jkcHbO/9PfSnQ1bh6PDrOH3Y1ajuUi5c0f7o8FeoeeTMjmnvwhn+f/Up2Mb4PTn+Iw0pI5DrlNt18TdJuw/OmaP90YnnyQNn45jeL/WBLv4LXsffyEbY0xJ8IJsjDElwQuyMcaUhLo2hsztba3+m76Pnbcf1cfD79RvJKkTOhKvoT7DTQTcWPLl4f+Y1LG4MZZ5haImvEBvBrsRGxme0k1JmV/XbwV7qt4J9rwDUQvcMiMGi69RujGEmx2oMz2u24I9EamzGaCe0+BmDb0d7KY1uAAHPA78cdSpfxxOIHif5xX9ABxD6n7UbnPt5LzI9ddoqPuTnG5KuPlkAeYJn+NqvZKUQb33NcXsNe1ITc7rb9W6pMwFI7EdU3bHDRf7u6PeexSZgBqh6U9Lsialc4vPdg79e9VmnPKbS6aF7jkblwONLzqA9AeZMu+Djb0561deF+yTyO40Dtn552OjiCR1vx7fkbPYazae7boNdmYq/rcZnwn2P9r4YLAr18sbQ4wx5kLCC7IxxpQEL8jGGFMS6tKQK5XKEUl7/+aaY4wx/18yp1qtZjI0RepakI0xxvzNYcnCGGNKghdkY4wpCV6QjTGmJHhBNsaYkuAF2RhjSoIXZGOMKQlekI0xpiR4QTbGmJLgBdkYY0rC/wUBylQU4X4rJwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show spectrogram\n",
    "fig = plt.imshow(spectrogram, origin='lower')\n",
    "fig.set_cmap('jet')\n",
    "fig.axes.get_xaxis().set_visible(False)\n",
    "fig.axes.get_yaxis().set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make 1 big array of list of spectrograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'float32'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert the input data to the right data type used by Keras Deep Learning (GPU)\n",
    "dtype = keras.backend.floatx()\n",
    "dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(128, 40, 80)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a list of many 40x80 spectrograms is made into 1 big array\n",
    "data = np.array(list_spectrograms, dtype=dtype)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standardization\n",
    "\n",
    "<b>Always standardize</b> the data before feeding it into the Neural Network!\n",
    "\n",
    "We use <b>Zero-mean Unit-variance standardization</b> (also known as Z-score normalization).\n",
    "Here, we use <b>attribute-wise standardization</b>, i.e. each pixel is standardized individually, as opposed to computing a single mean and single standard deviation of all values.\n",
    "\n",
    "('Flat' standardization would also be possible, but we have seen benefits of attribut-wise standardization in our experiments).\n",
    "\n",
    "We use the StandardScaler from the scikit-learn package for our purpose.\n",
    "As it works typically on vector data, we have to vectorize (i.e. reshape) our matrices first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(128, 3200)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# vectorize\n",
    "N, ydim, xdim = data.shape\n",
    "data = data.reshape(N, xdim*ydim)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# standardize\n",
    "scaler = preprocessing.StandardScaler()\n",
    "data = scaler.fit_transform(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([-3.983009 , -3.997586 , -3.9986486, ..., -8.036928 , -8.029473 ,\n",
       "        -8.038338 ], dtype=float32),\n",
       " array([1.0415441, 1.0475882, 1.0341036, ..., 1.0074264, 0.9896529,\n",
       "        1.0046793], dtype=float32))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show mean and standard deviation: two vectors with same length as data.shape[1]\n",
    "scaler.mean_, scaler.scale_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Train & Test Set \n",
    "\n",
    "We split the original full data set into two parts: Train Set (75%) and Test Set (25%).\n",
    "\n",
    "Here we compare Random Split vs. Stratified Split:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "testset_size = 0.25 # % portion of whole data set to keep for testing, i.e. 75% is used for training\n",
    "\n",
    "# Normal (random) split of data set into 2 parts\n",
    "# from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_set, test_set, train_classes, test_classes = train_test_split(data, classes_num, test_size=testset_size, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0,\n",
       "       0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1,\n",
       "       1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1,\n",
       "       0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0,\n",
       "       1, 0, 0, 0, 0, 0, 1, 1])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0,\n",
       "       1, 0, 0, 0, 0, 1, 0, 0, 1, 0])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class Counts: Class 0: 49 Class 1: 47\n"
     ]
    }
   ],
   "source": [
    "# The two classes may be unbalanced\n",
    "print(\"Class Counts: Class 0:\", sum(train_classes==0), \"Class 1:\", sum(train_classes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TRAIN INDEX: [ 97 102   0  34  13  44  60  20  75  22   6  24  99 116  47  61  14 126\n",
      "  68 104  89  17  95   7  37   3  93   8  40   9  69 122  72 119  71  82\n",
      "  33  74  51  49  78  29  76  98   5  50 121  23  12 105  36  96  25  90\n",
      " 101  55  92  21  80   2 125  16  32 127 124  48  28 112  63 110  77 109\n",
      "  54  79  45  53  66 115  86 120  52  59  81 118 106  19  43  94  65  15\n",
      "  26  84  30 107  10  91]\n",
      "TEST INDEX: [  1  27  64 117  88  85  35  18  46 100 111  11  83 103  87   4  70  31\n",
      " 108  58  57 123  38  73  56 113  67  39 114  42  41  62]\n"
     ]
    }
   ],
   "source": [
    "# better: Stratified Split retains the class balance in both sets\n",
    "# from sklearn.model_selection import StratifiedShuffleSplit\n",
    "\n",
    "splitter = StratifiedShuffleSplit(n_splits=1, test_size=testset_size, random_state=0)\n",
    "splits = splitter.split(data, classes_num)\n",
    "\n",
    "for train_index, test_index in splits:\n",
    "    print(\"TRAIN INDEX:\", train_index)\n",
    "    print(\"TEST INDEX:\", test_index)\n",
    "    train_set = data[train_index]\n",
    "    test_set = data[test_index]\n",
    "    train_classes = classes_num[train_index]\n",
    "    test_classes = classes_num[test_index]\n",
    "# Note: this for loop is only executed once, if n_iter==1 resp. n_splits==1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(96, 3200)\n",
      "(32, 3200)\n"
     ]
    }
   ],
   "source": [
    "print(train_set.shape)\n",
    "print(test_set.shape)\n",
    "# Note: we will reshape the data later back to matrix form "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class Counts: Class 0: 48 Class 1: 48\n"
     ]
    }
   ],
   "source": [
    "print(\"Class Counts: Class 0:\", sum(train_classes==0), \"Class 1:\", sum(train_classes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Networks\n",
    "\n",
    "A Convolutional Neural Network (ConvNet or CNN) is a type of (deep) Neural Network that is well-suited for 2D axes data, such as images or spectrograms, as it is optimized for learning from spatial proximity. Its core elements are 2D filter kernels which essentially learn the weights of the Neural Network, and downscaling functions such as Max Pooling.\n",
    "\n",
    "A CNN can have one or more Convolution layers, each of them having an arbitrary number of N filters (which define the depth of the CNN layer), following typically by a pooling step, which aggregates neighboring pixels together and thus reduces the image resolution by retaining only the maximum values of neighboring pixels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Data\n",
    "\n",
    "### Adding the channel\n",
    "\n",
    "As CNNs were initially made for image data, we need to add a dimension for the color channel to the data. RGB images typically have a 3rd dimension with the color. \n",
    "\n",
    "<b>Spectrograms, however, are considered like greyscale images, as in the previous tutorial.\n",
    "Likewise we need to add an extra dimension for compatibility with the CNN implementation.</b>\n",
    "\n",
    "For greyscale images, we add the number 1 as the depth of the additional dimension of the input shape (for RGB color images, the number of channels is 3).\n",
    "\n",
    "<i>Note on Tensorflow vs. Theano:</i>\n",
    "\n",
    "In Theano, traditionally the color channel was the <b>first</b> dimension in the image shape. \n",
    "In Tensorflow, the color channel is the <b>last</b> dimension in the image shape. \n",
    "\n",
    "This can be configured in ~/.keras/keras.json: \"image_dim_ordering\": \"th\" or \"tf\" (for Theano or Tensorflow) *or* with \"image_data_format\" set to \"channels_first\" or \"channels_last\".\n",
    "\n",
    "Tensorflow is now the default image ordering for Kears (\"tf\" and/or \"channels_last\").\n",
    "To be on the safe side, we added the if statement below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'channels_last'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " keras.backend.image_data_format()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_channels = 1 # for grey-scale, 3 for RGB, but usually already present in the data\n",
    "\n",
    "if keras.backend.image_data_format() == 'channels_last':  # TENSORFLOW\n",
    "    # Tensorflow ordering (~/.keras/keras.json: \"image_dim_ordering\": \"tf\")\n",
    "    train_set = train_set.reshape(train_set.shape[0], ydim, xdim, n_channels)\n",
    "    test_set = test_set.reshape(test_set.shape[0], ydim, xdim, n_channels)\n",
    "else: # THEANO\n",
    "    # Theano ordering (~/.keras/keras.json: \"image_dim_ordering\": \"th\")\n",
    "    train_set = train_set.reshape(train_set.shape[0], n_channels, ydim, xdim)\n",
    "    test_set = test_set.reshape(test_set.shape[0], n_channels, ydim, xdim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(96, 40, 80, 1)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32, 40, 80, 1)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40, 80, 1)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we store the new shape of the images in the 'input_shape' variable.\n",
    "# take all dimensions except the 0th one (which is the number of images)\n",
    "input_shape = train_set.shape[1:]  \n",
    "input_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Neural Network Models in Keras\n",
    "\n",
    "## Sequential Models\n",
    "\n",
    "In Keras, one can choose between a Sequential model and a Graph model. Sequential models are the standard case. Graph models are for parallel networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Single Layer and a Two Layer CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try: (comment/uncomment code in the following code block)\n",
    "* 1 Layer\n",
    "* 2 Layer\n",
    "* more conv_filters\n",
    "* Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#np.random.seed(0) # make results repeatable\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "#conv_filters = 16   # number of convolution filters (= CNN depth)\n",
    "conv_filters = 32   # number of convolution filters (= CNN depth)\n",
    "\n",
    "# Layer 1\n",
    "model.add(Convolution2D(conv_filters, (3, 3), input_shape=input_shape))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2))) \n",
    "model.add(Dropout(0.25)) \n",
    "\n",
    "# Layer 2\n",
    "model.add(Convolution2D(conv_filters, (3, 3)))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2))) \n",
    "\n",
    "# After Convolution, we have a 16*x*y matrix output\n",
    "# In order to feed this to a Full(Dense) layer, we need to flatten all data\n",
    "# Note: Keras does automatic shape inference, i.e. it knows how many (flat) input units the next layer will need,\n",
    "# so no parameter is needed for the Flatten() layer.\n",
    "model.add(Flatten()) \n",
    "\n",
    "# Full layer\n",
    "model.add(Dense(256, activation='sigmoid')) \n",
    "\n",
    "# Output layer\n",
    "# For binary/2-class problems use ONE sigmoid unit, \n",
    "# for multi-class/multi-label problems use n output units and activation='softmax!'\n",
    "model.add(Dense(1,activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get OverflowError: Range exceeds valid bounds in the above box, check the correct Theano vs. Tensorflow ordering in the box before and your keras.json configuration file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 38, 78, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 19, 39, 32)        0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 19, 39, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 17, 37, 32)        9248      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 8, 18, 32)         0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 4608)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 256)               1179904   \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 1,189,729\n",
      "Trainable params: 1,189,729\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a loss function \n",
    "loss = 'binary_crossentropy'  # 'categorical_crossentropy' for multi-class problems\n",
    "\n",
    "# Optimizer = Stochastic Gradient Descent\n",
    "optimizer = 'sgd' \n",
    "\n",
    "# Compiling the model\n",
    "model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.7460 - acc: 0.5104\n",
      "Epoch 2/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6705 - acc: 0.5312\n",
      "Epoch 3/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6426 - acc: 0.6354\n",
      "Epoch 4/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6114 - acc: 0.7188\n",
      "Epoch 5/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5936 - acc: 0.7188\n",
      "Epoch 6/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5708 - acc: 0.7188\n",
      "Epoch 7/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5583 - acc: 0.7396\n",
      "Epoch 8/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5415 - acc: 0.7396\n",
      "Epoch 9/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5273 - acc: 0.7604\n",
      "Epoch 10/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5169 - acc: 0.7604\n",
      "Epoch 11/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5113 - acc: 0.7604\n",
      "Epoch 12/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5054 - acc: 0.7708\n",
      "Epoch 13/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.4921 - acc: 0.7708\n",
      "Epoch 14/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.4829 - acc: 0.7812\n",
      "Epoch 15/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.4783 - acc: 0.7917\n"
     ]
    }
   ],
   "source": [
    "# TRAINING the model\n",
    "epochs = 15\n",
    "history = model.fit(train_set, train_classes, batch_size=32, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy goes up pretty quickly for 1 layer on Train set! Also on Test set?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verifying Accuracy on Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# always execute this, and then a box of accuracy_score below to print the result\n",
    "test_pred = model.predict_classes(test_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.71875"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1 layer\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.71875"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2 layer\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.71875"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2 layer + 32 convolution filters\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.71875"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2 layer + 32 convolution filters + Dropout\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Parameters & Techniques\n",
    "\n",
    "Try out more parameters and techniques: (comment/uncomment code blocks below)\n",
    "* Adding ReLU activation\n",
    "* Adding Batch normalization\n",
    "* Adding Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "conv_filters = 16   # number of convolution filters (= CNN depth)\n",
    "filter_size = (3,3)\n",
    "pool_size = (2,2)\n",
    "\n",
    "# Layer 1\n",
    "model.add(Convolution2D(conv_filters, filter_size, padding='valid', input_shape=input_shape))\n",
    "#model.add(BatchNormalization())\n",
    "#model.add(Activation('relu')) \n",
    "model.add(MaxPooling2D(pool_size=pool_size)) \n",
    "#model.add(Dropout(0.3))\n",
    "\n",
    "# Layer 2\n",
    "model.add(Convolution2D(conv_filters, filter_size, padding='valid', input_shape=input_shape))\n",
    "#model.add(BatchNormalization())\n",
    "#model.add(Activation('relu')) \n",
    "model.add(MaxPooling2D(pool_size=pool_size)) \n",
    "#model.add(Dropout(0.1))\n",
    "\n",
    "# In order to feed this to a Full(Dense) layer, we need to flatten all data\n",
    "model.add(Flatten()) \n",
    "\n",
    "# Full layer\n",
    "model.add(Dense(256))  \n",
    "#model.add(Activation('relu'))\n",
    "#model.add(Dropout(0.1))\n",
    "\n",
    "# Output layer\n",
    "# For binary/2-class problems use ONE sigmoid unit, \n",
    "# for multi-class/multi-label problems use n output units and activation='softmax!'\n",
    "model.add(Dense(1,activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MIREX 2015 model\n",
    "model = Sequential()\n",
    "\n",
    "conv_filters = 15   # number of convolution filters (= CNN depth)\n",
    "\n",
    "# Layer 1\n",
    "model.add(Convolution2D(conv_filters, (12, 8), padding='valid', input_shape=input_shape))\n",
    "#model.add(BatchNormalization())\n",
    "#model.add(Activation('relu')) \n",
    "model.add(Activation('sigmoid')) \n",
    "model.add(MaxPooling2D(pool_size=(2, 1))) \n",
    "#model.add(Dropout(0.3))\n",
    "\n",
    "\n",
    "# In order to feed this to a Full(Dense) layer, we need to flatten all data\n",
    "model.add(Flatten()) \n",
    "\n",
    "# Full layer\n",
    "model.add(Dense(200, activation='sigmoid'))  \n",
    "#model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "\n",
    "# Output layer\n",
    "# For binary/2-class problems use ONE sigmoid unit, \n",
    "# for multi-class/multi-label problems use n output units and activation='softmax!'\n",
    "model.add(Dense(1,activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compiling and training the model\n",
    "model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.7729 - acc: 0.4688\n",
      "Epoch 2/15\n",
      "96/96 [==============================] - 0s 1ms/step - loss: 0.7734 - acc: 0.4583\n",
      "Epoch 3/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.7648 - acc: 0.5417\n",
      "Epoch 4/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.7545 - acc: 0.4583\n",
      "Epoch 5/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6950 - acc: 0.5417\n",
      "Epoch 6/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6950 - acc: 0.5833\n",
      "Epoch 7/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6554 - acc: 0.5833\n",
      "Epoch 8/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6468 - acc: 0.6146\n",
      "Epoch 9/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6807 - acc: 0.6250\n",
      "Epoch 10/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6628 - acc: 0.6042\n",
      "Epoch 11/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.7091 - acc: 0.6146\n",
      "Epoch 12/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6760 - acc: 0.6042\n",
      "Epoch 13/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6157 - acc: 0.6667\n",
      "Epoch 14/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.6132 - acc: 0.6771\n",
      "Epoch 15/15\n",
      "96/96 [==============================] - 0s 2ms/step - loss: 0.5837 - acc: 0.7396\n"
     ]
    }
   ],
   "source": [
    "epochs = 15\n",
    "history = model.fit(train_set, train_classes, batch_size=32, epochs=epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.71875"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verifying Accuracy on Test Set\n",
    "\n",
    "test_pred = model.predict_classes(test_set)\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Parallel CNNs\n",
    "\n",
    "It has been discovered, that CNNs for music work best, when they have one filter that is detecting frequencies in the vertical axis, and nother filter that is focused on the time axis, i.e. detecting rhythm. Consequently, this is realized in a parallel CNN, where 2 layers are not stacked after each other, but first run independently in parallel with their output being merged later.\n",
    "\n",
    "To create parallel CNNs we need a \"graph-based\" model. In Keras 1.x this is realized via the functional API of the Model() class.\n",
    "We use it to create two CNN layers that run in parallel to each other and are merged subsequently.\n",
    "In the functional API, you pass the name of the previous layer in (brackets) after defining the next layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers.merge import concatenate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input only specifies the input shape\n",
    "input = Input(input_shape)\n",
    "\n",
    "# CNN layers\n",
    "# specify desired number of filters\n",
    "n_filters = 16 \n",
    "# The functional API allows to specify the predecessor in (brackets) after the new Layer function call\n",
    "conv_layer1 = Convolution2D(16, (10, 2))(input)  # a vertical filter\n",
    "conv_layer2 = Convolution2D(25, (2, 10))(input)  # a horizontal filter\n",
    "\n",
    "# possibly add Activation('relu') here\n",
    "\n",
    "# Pooling layers\n",
    "maxpool1 = MaxPooling2D(pool_size=(1,2))(conv_layer1) # horizontal pooling\n",
    "maxpool2 = MaxPooling2D(pool_size=(2,1))(conv_layer2) # vertical pooling\n",
    "\n",
    "# we have to flatten the Pooling output in order to be concatenated\n",
    "flat1 = Flatten()(maxpool1)\n",
    "flat2 = Flatten()(maxpool2)\n",
    "\n",
    "# Merge the 2\n",
    "merged = concatenate([flat1, flat2])\n",
    "\n",
    "full = Dense(256, activation='relu')(merged)\n",
    "output_layer = Dense(1, activation='sigmoid')(full)\n",
    "\n",
    "# finally create the model\n",
    "model = Model(inputs=input, outputs=output_layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 40, 80, 1)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_6 (Conv2D)               (None, 31, 79, 16)   336         input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_7 (Conv2D)               (None, 39, 71, 25)   525         input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling2d_6 (MaxPooling2D)  (None, 31, 39, 16)   0           conv2d_6[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling2d_7 (MaxPooling2D)  (None, 19, 71, 25)   0           conv2d_7[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "flatten_4 (Flatten)             (None, 19344)        0           max_pooling2d_6[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "flatten_5 (Flatten)             (None, 33725)        0           max_pooling2d_7[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 53069)        0           flatten_4[0][0]                  \n",
      "                                                                 flatten_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_7 (Dense)                 (None, 256)          13585920    concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_8 (Dense)                 (None, 1)            257         dense_7[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 13,587,038\n",
      "Trainable params: 13,587,038\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a loss function \n",
    "loss = 'binary_crossentropy'  # 'categorical_crossentropy' for multi-class problems\n",
    "\n",
    "# Optimizer = Stochastic Gradient Descent\n",
    "optimizer = 'sgd' \n",
    "\n",
    "# Compiling the model\n",
    "model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "96/96 [==============================] - 0s 5ms/step - loss: 0.5242 - acc: 0.7188\n",
      "Epoch 2/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.2932 - acc: 0.8958\n",
      "Epoch 3/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.2043 - acc: 0.9375\n",
      "Epoch 4/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.1615 - acc: 0.9688\n",
      "Epoch 5/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.1361 - acc: 0.9896\n",
      "Epoch 6/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.1095 - acc: 0.9896\n",
      "Epoch 7/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0953 - acc: 0.9896\n",
      "Epoch 8/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0815 - acc: 1.0000\n",
      "Epoch 9/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0726 - acc: 1.0000\n",
      "Epoch 10/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0641 - acc: 1.0000\n",
      "Epoch 11/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0574 - acc: 1.0000\n",
      "Epoch 12/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0516 - acc: 1.0000\n",
      "Epoch 13/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0468 - acc: 1.0000\n",
      "Epoch 14/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0429 - acc: 1.0000\n",
      "Epoch 15/15\n",
      "96/96 [==============================] - 0s 3ms/step - loss: 0.0403 - acc: 1.0000\n"
     ]
    }
   ],
   "source": [
    "# TRAINING the model\n",
    "epochs = 15\n",
    "history = model.fit(train_set, train_classes, batch_size=32, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verifying Accuracy on Test Set\n",
    "\n",
    "Note: The functional API, i.e. Model() does not have a convenience method `.predict_classes()`. We therefore do 'raw' predictions with `predict()`, which returns values between 0 and 1, and then round to the nearest value (0 or 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.8116090e-03, 9.9629503e-01, 8.9173570e-02, 1.3832313e-02,\n",
       "       1.5312824e-02, 6.2414613e-02, 9.4741523e-01, 2.8874350e-01,\n",
       "       6.6524655e-02, 7.7041835e-01, 5.9007388e-03, 8.8411587e-01,\n",
       "       7.7622104e-03, 5.5290854e-01, 7.2940448e-03, 9.7821641e-01,\n",
       "       1.9546174e-03, 1.2710236e-01, 2.5445417e-02, 9.2131579e-01,\n",
       "       5.5363268e-01, 7.3306054e-02, 9.8920244e-01, 8.6430952e-02,\n",
       "       8.9812309e-01, 6.6113585e-05, 2.7863439e-03, 9.9372864e-01,\n",
       "       2.8969401e-02, 9.8893112e-01, 7.3728962e-03, 6.7004067e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_pred = model.predict(test_set)\n",
    "test_pred[0:35,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.78125"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_pred = np.round(test_pred)\n",
    "accuracy_score(test_classes, test_pred)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}