Neural_Nets_Project_Starter_Code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gesture Recognition\n",
    "In this group project, you are going to build a 3D Conv model that will be able to predict the 5 gestures correctly. Please import the following libraries to get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "from scipy.misc import imread, imresize\n",
    "from skimage.transform import resize\n",
    "# from imageio import imread\n",
    "import datetime\n",
    "import os\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, GRU, Flatten, TimeDistributed, Flatten, BatchNormalization, Activation, Dropout,Input\n",
    "from keras.layers.convolutional import Conv3D, MaxPooling3D\n",
    "from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\n",
    "from keras import optimizers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set the random seed so that the results don't vary drastically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(30)\n",
    "import random as rn\n",
    "rn.seed(30)\n",
    "from keras import backend as K\n",
    "import tensorflow as tf\n",
    "tf.set_random_seed(30)\n",
    "import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this block, you read the folder names for training and validation. You also set the `batch_size` here. Note that you set the batch size in such a way that you are able to use the GPU in full capacity. You keep increasing the batch size until the machine throws an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Errno 2] No such file or directory: 'Project_data/'\n",
      "/mnt/disks/user/project/PROJECT/Project_data\n"
     ]
    }
   ],
   "source": [
    "cd Project_data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_doc = np.random.permutation(open('train.csv').readlines())\n",
    "val_doc = np.random.permutation(open('val.csv').readlines())\n",
    "batch_size = 10 #experiment with the batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator\n",
    "This is one of the most important part of the code. The overall structure of the generator has been given. In the generator, you are going to preprocess the images as you have images of 2 different dimensions as well as create a batch of video frames. You have to experiment with `img_idx`, `y`,`z` and normalization such that you get high accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 30 # number of frames\n",
    "y = 120 # image width\n",
    "z = 120 # image height"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 30 # number of frames\n",
    "y = 120 # image width\n",
    "z = 120 # image height\n",
    "\n",
    "def generator(source_path, folder_list, batch_size):\n",
    "#     print('3')\n",
    "    print( 'Source path = ', source_path, '; batch size =', batch_size)\n",
    "    img_idx = [x for x in range(0,x)] #create a list of image numbers you want to use for a particular video\n",
    "    while True:\n",
    "        t = np.random.permutation(folder_list)\n",
    "        num_batches = len(folder_list)//batch_size # calculate the number of batches\n",
    "        for batch in range(num_batches): # we iterate over the number of batches\n",
    "            batch_data = np.zeros((batch_size,x,y,z,3)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "                    \n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    \n",
    "                    temp = resize(image,(120,120))\n",
    "                    temp = temp/127.5-1 #Normalize data\n",
    "                    \n",
    "                    batch_data[folder,idx,:,:,0] = (temp[:,:,0]) #normalise and feed in the image\n",
    "                    batch_data[folder,idx,:,:,1] = (temp[:,:,1]) #normalise and feed in the image\n",
    "                    batch_data[folder,idx,:,:,2] = (temp[:,:,2]) #normalise and feed in the image\n",
    "                    \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do\n",
    "\n",
    "#             print('1')\n",
    "        # write the code for the remaining data points which are left after full batches\n",
    "        if (len(folder_list) != batch_size*num_batches):\n",
    "            print(\"Batch: \",num_batches+1,\"Index:\", batch_size)\n",
    "            batch_size = len(folder_list) - (batch_size*num_batches)\n",
    "            batch_data = np.zeros((batch_size,x,y,z,3)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "                    \n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    temp = resize(image,(120,120))\n",
    "                    temp = temp/127.5-1 #Normalize data\n",
    "                    \n",
    "                    batch_data[folder,idx,:,:,0] = (temp[:,:,0])\n",
    "                    batch_data[folder,idx,:,:,1] = (temp[:,:,1])\n",
    "                    batch_data[folder,idx,:,:,2] = (temp[:,:,2])\n",
    "                   \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "#                 print('2')\n",
    "            yield batch_data, batch_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note here that a video is represented above in the generator as (number of images, height, width, number of channels). Take this into consideration while creating the model architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# training sequences = 663\n",
      "# validation sequences = 100\n",
      "# epochs = 10\n"
     ]
    }
   ],
   "source": [
    "curr_dt_time = datetime.datetime.now()\n",
    "train_path = 'train'\n",
    "val_path = 'val'\n",
    "num_train_sequences = len(train_doc)\n",
    "print('# training sequences =', num_train_sequences)\n",
    "num_val_sequences = len(val_doc)\n",
    "print('# validation sequences =', num_val_sequences)\n",
    "num_epochs = 10 # choose the number of epochs\n",
    "print ('# epochs =', num_epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model 1\n",
    "Here you make the model using different functionalities that Keras provides. Remember to use `Conv3D` and `MaxPooling3D` and not `Conv2D` and `Maxpooling2D` for a 3D convolution model. You would want to use `TimeDistributed` while building a Conv2D + RNN model. Also remember that the last layer is the softmax. Design the network in such a way that the model is able to give good accuracy on the least number of parameters so that it can fit in the memory of the webcam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#write your model here\n",
    "model1 = Sequential()\n",
    "\n",
    "model1.add(Conv3D(32, #number of filters \n",
    "                 kernel_size=(3,3,3), \n",
    "                 input_shape=(30, 120, 120, 3),\n",
    "                 padding='same'))\n",
    "model1.add(BatchNormalization())\n",
    "model1.add(Activation('relu'))\n",
    "\n",
    "model1.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model1.add(Conv3D(64, #Number of filters, \n",
    "                 kernel_size=(3,3,3), \n",
    "                 padding='same'))\n",
    "model1.add(BatchNormalization())\n",
    "model1.add(Activation('relu'))\n",
    "\n",
    "model1.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model1.add(Conv3D(128, #Number of filters \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model1.add(BatchNormalization())\n",
    "model1.add(Activation('relu'))\n",
    "\n",
    "model1.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "model1.add(Conv3D(256, #Number pf filters \n",
    "                 kernel_size=(1,3,3), \n",
    "                 padding='same'))\n",
    "model1.add(BatchNormalization())\n",
    "model1.add(Activation('relu'))\n",
    "\n",
    "model1.add(MaxPooling3D(pool_size=(2,2,2)))\n",
    "\n",
    "#Flatten Layers\n",
    "model1.add(Flatten())\n",
    "\n",
    "model1.add(Dense(1000, activation='relu'))\n",
    "model1.add(Dropout(0.25))\n",
    "\n",
    "model1.add(Dense(500, activation='relu'))\n",
    "model1.add(Dropout(0.25))\n",
    "\n",
    "#softmax layer\n",
    "model1.add(Dense(5, activation='softmax'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have written the model, the next step is to `compile` the model. When you print the `summary` of the model, you'll see the total number of parameters you have to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_5 (Conv3D)            (None, 30, 120, 120, 32)  2624      \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 30, 120, 120, 32)  128       \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 30, 120, 120, 32)  0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_5 (MaxPooling3 (None, 15, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_6 (Conv3D)            (None, 15, 60, 60, 64)    55360     \n",
      "_________________________________________________________________\n",
      "batch_normalization_6 (Batch (None, 15, 60, 60, 64)    256       \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 15, 60, 60, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_6 (MaxPooling3 (None, 7, 30, 30, 64)     0         \n",
      "_________________________________________________________________\n",
      "conv3d_7 (Conv3D)            (None, 7, 30, 30, 128)    73856     \n",
      "_________________________________________________________________\n",
      "batch_normalization_7 (Batch (None, 7, 30, 30, 128)    512       \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 7, 30, 30, 128)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_7 (MaxPooling3 (None, 3, 15, 15, 128)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_8 (Conv3D)            (None, 3, 15, 15, 256)    295168    \n",
      "_________________________________________________________________\n",
      "batch_normalization_8 (Batch (None, 3, 15, 15, 256)    1024      \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 3, 15, 15, 256)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_8 (MaxPooling3 (None, 1, 7, 7, 256)      0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 12544)             0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1000)              12545000  \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 5)                 2505      \n",
      "=================================================================\n",
      "Total params: 13,476,933\n",
      "Trainable params: 13,475,973\n",
      "Non-trainable params: 960\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "optimiser = optimizers.Adam(lr=0.001) #write your optimizer\n",
    "model1.compile(optimizer=optimiser, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "print (model1.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create the `train_generator` and the `val_generator` which will be used in `.fit_generator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator(train_path, train_doc, batch_size)\n",
    "val_generator = generator(val_path, val_doc, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'model_init' + '_' + str(curr_dt_time).replace(' ','').replace(':','_') + '/'\n",
    "    \n",
    "if not os.path.exists(model_name):\n",
    "    os.mkdir(model_name)\n",
    "        \n",
    "filepath = model_name + 'model-{epoch:05d}-{loss:.5f}-{categorical_accuracy:.5f}-{val_loss:.5f}-{val_categorical_accuracy:.5f}.h5'\n",
    "\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=False, save_weights_only=False, mode='auto', period=1)\n",
    "\n",
    "LR = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=2, cooldown=1, verbose=1)# write the REducelronplateau code here\n",
    "callbacks_list = [checkpoint, LR]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `steps_per_epoch` and `validation_steps` are used by `fit_generator` to decide the number of next() calls it need to make."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (num_train_sequences%batch_size) == 0:\n",
    "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
    "else:\n",
    "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
    "\n",
    "if (num_val_sequences%batch_size) == 0:\n",
    "    validation_steps = int(num_val_sequences/batch_size)\n",
    "else:\n",
    "    validation_steps = (num_val_sequences//batch_size) + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now fit the model. This will start training the model and with the help of the checkpoints, you'll be able to save the model at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# validation_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  val ; batch size = 10\n",
      "Source path =  train ; batch size = 10\n",
      "Epoch 1/10\n",
      "65/67 [============================>.] - ETA: 3s - loss: 12.9520 - categorical_accuracy: 0.1831Batch:  67 Index: 10\n",
      "67/67 [==============================] - 131s 2s/step - loss: 12.9739 - categorical_accuracy: 0.1821 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2615_56_14.668735/model-00001-12.94119-0.18401-12.57211-0.22000.h5\n",
      "Epoch 2/10\n",
      "67/67 [==============================] - 44s 650ms/step - loss: 12.9105 - categorical_accuracy: 0.1990 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2615_56_14.668735/model-00002-12.91051-0.19900-12.57211-0.22000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 44s 651ms/step - loss: 12.7501 - categorical_accuracy: 0.2090 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2615_56_14.668735/model-00003-12.75014-0.20896-12.57211-0.22000.h5\n",
      "\n",
      "Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 44s 663ms/step - loss: 13.2313 - categorical_accuracy: 0.1791 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2615_56_14.668735/model-00004-13.23127-0.17910-12.57211-0.22000.h5\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 45s 666ms/step - loss: 13.1511 - categorical_accuracy: 0.1841 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2615_56_14.668735/model-00005-13.15108-0.18408-12.57211-0.22000.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 47s 701ms/step - loss: 12.5898 - categorical_accuracy: 0.2189 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2615_56_14.668735/model-00006-12.58976-0.21891-12.57211-0.22000.h5\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 45s 674ms/step - loss: 12.8303 - categorical_accuracy: 0.2040 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2615_56_14.668735/model-00007-12.83032-0.20398-12.57211-0.22000.h5\n",
      "\n",
      "Epoch 00007: ReduceLROnPlateau reducing learning rate to 0.0001250000059371814.\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 44s 652ms/step - loss: 13.3115 - categorical_accuracy: 0.1741 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2615_56_14.668735/model-00008-13.31146-0.17413-12.57211-0.22000.h5\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 46s 688ms/step - loss: 12.7501 - categorical_accuracy: 0.2090 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2615_56_14.668735/model-00009-12.75014-0.20896-12.57211-0.22000.h5\n",
      "\n",
      "Epoch 00009: ReduceLROnPlateau reducing learning rate to 6.25000029685907e-05.\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 43s 646ms/step - loss: 13.1511 - categorical_accuracy: 0.1841 - val_loss: 12.5721 - val_categorical_accuracy: 0.2200\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2615_56_14.668735/model-00010-13.15108-0.18408-12.57211-0.22000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6920106c88>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model1.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = 5 #left swipe, right swipe, thumbs up, thumbs down, stop\n",
    "channel = 3 #RGB\n",
    "x = 30 # number of frames\n",
    "y = 120 # image width\n",
    "z = 120 # image height\n",
    "\n",
    "def generator_ex(source_path, folder_list, batch_size):\n",
    "    print( 'Source path = ', source_path, '; batch size =', batch_size)\n",
    "    img_idx = [x for x in range(0,x)] #create a list of image numbers you want to use for a particular video\n",
    "    while True:\n",
    "        t = np.random.permutation(folder_list)\n",
    "        num_batches = len(folder_list)//batch_size # calculate the number of batches\n",
    "        for batch in range(num_batches): # we iterate over the number of batches\n",
    "            batch_data = np.zeros((batch_size,x,y,z,channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,classes)) # batch_labels is the one hot representation of the output\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "                    \n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    \n",
    "                    temp = resize(image,(y,z))\n",
    "                    #Converting to gray scale\n",
    "                    temp = temp.mean(axis=-1,keepdims=1) \n",
    "                    temp = temp/127.5-1 #Normalize data\n",
    "                    batch_data[folder,idx] = temp #normalise and feed in the image\n",
    "                    \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "                \n",
    "            yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do\n",
    "\n",
    "        \n",
    "        # write the code for the remaining data points which are left after full batches\n",
    "        if (len(folder_list) != batch_size*num_batches):\n",
    "            print(\"Batch: \",num_batches+1,\"Index:\", batch_size)\n",
    "            batch_size = len(folder_list) - (batch_size*num_batches)\n",
    "            batch_data = np.zeros((batch_size,x,y,z,channel)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
    "            batch_labels = np.zeros((batch_size,classes)) # batch_labels is the one hot representation of the output\n",
    "            for folder in range(batch_size): # iterate over the batch_size\n",
    "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
    "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
    "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
    "                    \n",
    "                    #crop the images and resize them. Note that the images are of 2 different shape \n",
    "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
    "                    temp = resize(image,(y,z))\n",
    "                    #Converting to gray scale\n",
    "                    temp = temp.mean(axis=-1,keepdims=1) \n",
    "                    temp = temp/127.5-1 #Normalize data\n",
    "                    \n",
    "                    batch_data[folder,idx] = temp\n",
    "                   \n",
    "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
    "            yield batch_data, batch_labels\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_9 (Conv3D)            (None, 30, 120, 120, 32)  2624      \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 30, 120, 120, 32)  0         \n",
      "_________________________________________________________________\n",
      "conv3d_10 (Conv3D)           (None, 30, 120, 120, 32)  27680     \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 30, 120, 120, 32)  0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_9 (MaxPooling3 (None, 10, 40, 40, 32)    0         \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 10, 40, 40, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_11 (Conv3D)           (None, 10, 40, 40, 64)    55360     \n",
      "_________________________________________________________________\n",
      "activation_11 (Activation)   (None, 10, 40, 40, 64)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_12 (Conv3D)           (None, 10, 40, 40, 64)    110656    \n",
      "_________________________________________________________________\n",
      "activation_12 (Activation)   (None, 10, 40, 40, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_10 (MaxPooling (None, 4, 14, 14, 64)     0         \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 4, 14, 14, 64)     0         \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 50176)             0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 512)               25690624  \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 5)                 2565      \n",
      "=================================================================\n",
      "Total params: 25,889,509\n",
      "Trainable params: 25,889,509\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.losses import categorical_crossentropy\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "# Define 2nd model \n",
    "model2 = Sequential()\n",
    "model2.add(Conv3D(32, kernel_size=(3, 3, 3), input_shape=(x,y,z,channel), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(Conv3D(32, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Dropout(0.25))\n",
    "\n",
    "model2.add(Conv3D(64, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(Conv3D(64, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Dropout(0.25))\n",
    "\n",
    "model2.add(Flatten())\n",
    "model2.add(Dense(512, activation='relu'))\n",
    "model2.add(Dropout(0.5))\n",
    "model2.add(Dense(classes, activation='softmax'))\n",
    "\n",
    "model2.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model2.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "train_generator = generator_ex(train_path, train_doc, batch_size)\n",
    "val_generator = generator_ex(val_path, val_doc, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path = Source path =  train ; batch size = Epoch 1/10\n",
      " 10\n",
      "val ; batch size = 10\n",
      "65/67 [============================>.] - ETA: 4s - loss: 1.7055 - categorical_accuracy: 0.2123Batch:  67 Index: 10\n",
      "67/67 [==============================] - 152s 2s/step - loss: 1.7025 - categorical_accuracy: 0.2159 - val_loss: 1.6089 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2615_56_14.668735/model-00001-1.70366-0.21116-1.60891-0.21000.h5\n",
      "Epoch 2/10\n",
      "67/67 [==============================] - 49s 739ms/step - loss: 1.6109 - categorical_accuracy: 0.1692 - val_loss: 1.6095 - val_categorical_accuracy: 0.1800\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2615_56_14.668735/model-00002-1.61092-0.16915-1.60948-0.18000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 50s 751ms/step - loss: 1.6126 - categorical_accuracy: 0.1891 - val_loss: 1.6068 - val_categorical_accuracy: 0.2300\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2615_56_14.668735/model-00003-1.61259-0.18905-1.60680-0.23000.h5\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 50s 747ms/step - loss: 1.6093 - categorical_accuracy: 0.2239 - val_loss: 1.6062 - val_categorical_accuracy: 0.2900\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2615_56_14.668735/model-00004-1.60934-0.22388-1.60621-0.29000.h5\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 52s 777ms/step - loss: 1.6538 - categorical_accuracy: 0.2537 - val_loss: 1.6096 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2615_56_14.668735/model-00005-1.65385-0.25373-1.60959-0.21000.h5\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 51s 765ms/step - loss: 1.6072 - categorical_accuracy: 0.1990 - val_loss: 1.6057 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2615_56_14.668735/model-00006-1.60716-0.19900-1.60571-0.21000.h5\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 51s 765ms/step - loss: 1.6113 - categorical_accuracy: 0.1891 - val_loss: 1.6062 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2615_56_14.668735/model-00007-1.61134-0.18905-1.60617-0.21000.h5\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 51s 761ms/step - loss: 1.6157 - categorical_accuracy: 0.1443 - val_loss: 1.6076 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2615_56_14.668735/model-00008-1.61567-0.14428-1.60765-0.21000.h5\n",
      "\n",
      "Epoch 00008: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 51s 757ms/step - loss: 1.6101 - categorical_accuracy: 0.2537 - val_loss: 1.6085 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2615_56_14.668735/model-00009-1.61007-0.25373-1.60846-0.21000.h5\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 52s 779ms/step - loss: 1.6102 - categorical_accuracy: 0.2189 - val_loss: 1.6079 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2615_56_14.668735/model-00010-1.61017-0.21891-1.60792-0.21000.h5\n",
      "\n",
      "Epoch 00010: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f692014c898>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = 'green' >  trial 1 - Resizing changing x, y , z value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    " \n",
    "x = 30 # number of frames\n",
    "y = 60 # image width\n",
    "z = 60 # image height"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_13 (Conv3D)           (None, 30, 60, 60, 32)    2624      \n",
      "_________________________________________________________________\n",
      "activation_13 (Activation)   (None, 30, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_14 (Conv3D)           (None, 30, 60, 60, 32)    27680     \n",
      "_________________________________________________________________\n",
      "activation_14 (Activation)   (None, 30, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_11 (MaxPooling (None, 10, 20, 20, 32)    0         \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (None, 10, 20, 20, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_15 (Conv3D)           (None, 10, 20, 20, 64)    55360     \n",
      "_________________________________________________________________\n",
      "activation_15 (Activation)   (None, 10, 20, 20, 64)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_16 (Conv3D)           (None, 10, 20, 20, 64)    110656    \n",
      "_________________________________________________________________\n",
      "activation_16 (Activation)   (None, 10, 20, 20, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_12 (MaxPooling (None, 4, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "dropout_9 (Dropout)          (None, 4, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_4 (Flatten)          (None, 12544)             0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 512)               6423040   \n",
      "_________________________________________________________________\n",
      "dropout_10 (Dropout)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 5)                 2565      \n",
      "=================================================================\n",
      "Total params: 6,621,925\n",
      "Trainable params: 6,621,925\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define model b\n",
    "model2 = Sequential()\n",
    "model2.add(Conv3D(32, kernel_size=(3, 3, 3), input_shape=(x,y,z,channel), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(Conv3D(32, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Dropout(0.25))\n",
    "\n",
    "model2.add(Conv3D(64, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(Conv3D(64, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Dropout(0.25))\n",
    "\n",
    "model2.add(Flatten())\n",
    "model2.add(Dense(512, activation='relu'))\n",
    "model2.add(Dropout(0.5))\n",
    "model2.add(Dense(classes, activation='softmax'))\n",
    "\n",
    "model2.compile(optimizer=keras.optimizers.Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "train_generator = generator_ex(train_path, train_doc, batch_size)\n",
    "val_generator = generator_ex(val_path, val_doc, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  val ; batch size = 10\n",
      "Source path =  Epoch 1/10\n",
      "train ; batch size = 10\n",
      "65/67 [============================>.] - ETA: 2s - loss: 1.6132 - categorical_accuracy: 0.2246Batch:  67 Index: 10\n",
      "67/67 [==============================] - 99s 1s/step - loss: 1.6120 - categorical_accuracy: 0.2259 - val_loss: 1.5663 - val_categorical_accuracy: 0.1700\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2615_56_14.668735/model-00001-1.61206-0.22474-1.56632-0.17000.h5\n",
      "Epoch 2/10\n",
      "67/67 [==============================] - 34s 505ms/step - loss: 1.6058 - categorical_accuracy: 0.1990 - val_loss: 1.6128 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2615_56_14.668735/model-00002-1.60576-0.19900-1.61283-0.21000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 34s 505ms/step - loss: 1.6116 - categorical_accuracy: 0.2289 - val_loss: 1.6098 - val_categorical_accuracy: 0.2100\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2615_56_14.668735/model-00003-1.61162-0.22886-1.60979-0.21000.h5\n",
      "\n",
      "Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 34s 515ms/step - loss: 1.5980 - categorical_accuracy: 0.2090 - val_loss: 1.9018 - val_categorical_accuracy: 0.2300\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2615_56_14.668735/model-00004-1.59803-0.20896-1.90181-0.23000.h5\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 36s 530ms/step - loss: 1.5924 - categorical_accuracy: 0.2587 - val_loss: 1.5052 - val_categorical_accuracy: 0.2000\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2615_56_14.668735/model-00005-1.59238-0.25871-1.50517-0.20000.h5\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 36s 535ms/step - loss: 1.5017 - categorical_accuracy: 0.3930 - val_loss: 1.4502 - val_categorical_accuracy: 0.4100\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2615_56_14.668735/model-00006-1.50172-0.39303-1.45018-0.41000.h5\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 36s 543ms/step - loss: 1.4634 - categorical_accuracy: 0.3831 - val_loss: 1.4061 - val_categorical_accuracy: 0.3900\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2615_56_14.668735/model-00007-1.46335-0.38308-1.40611-0.39000.h5\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 35s 529ms/step - loss: 1.4119 - categorical_accuracy: 0.3930 - val_loss: 1.3504 - val_categorical_accuracy: 0.3500\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2615_56_14.668735/model-00008-1.41195-0.39303-1.35042-0.35000.h5\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 35s 525ms/step - loss: 1.2218 - categorical_accuracy: 0.4826 - val_loss: 1.3221 - val_categorical_accuracy: 0.4400\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2615_56_14.668735/model-00009-1.22177-0.48259-1.32212-0.44000.h5\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 36s 537ms/step - loss: 1.3685 - categorical_accuracy: 0.3483 - val_loss: 1.2460 - val_categorical_accuracy: 0.4600\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2615_56_14.668735/model-00010-1.36851-0.34826-1.24602-0.46000.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6944a4c4a8>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color='green'> Trial 2. Batch size = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator_ex(train_path, train_doc, 20)\n",
    "val_generator = generator_ex(val_path, val_doc, 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  val ; batch size = 20\n",
      "Source path =  train ; batch size = Epoch 1/10\n",
      "20\n",
      "32/67 [=============>................] - ETA: 1:46 - loss: 1.1137 - categorical_accuracy: 0.5203Batch:  34 Index: 20\n",
      "67/67 [==============================] - 116s 2s/step - loss: 1.1097 - categorical_accuracy: 0.5401 - val_loss: 1.1047 - val_categorical_accuracy: 0.5600\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2615_56_14.668735/model-00001-1.10931-0.52625-1.10469-0.56000.h5\n",
      "Epoch 2/10\n",
      "67/67 [==============================] - 45s 678ms/step - loss: 1.0326 - categorical_accuracy: 0.5672 - val_loss: 1.1624 - val_categorical_accuracy: 0.4900\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2615_56_14.668735/model-00002-1.03264-0.56716-1.16239-0.49000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 46s 694ms/step - loss: 1.0659 - categorical_accuracy: 0.5274 - val_loss: 1.1534 - val_categorical_accuracy: 0.5100\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2615_56_14.668735/model-00003-1.06591-0.52736-1.15336-0.51000.h5\n",
      "\n",
      "Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 46s 686ms/step - loss: 0.9952 - categorical_accuracy: 0.6119 - val_loss: 1.0627 - val_categorical_accuracy: 0.5400\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2615_56_14.668735/model-00004-0.99517-0.61194-1.06267-0.54000.h5\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 45s 678ms/step - loss: 0.7637 - categorical_accuracy: 0.6617 - val_loss: 1.0662 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2615_56_14.668735/model-00005-0.76373-0.66169-1.06616-0.57000.h5\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 46s 685ms/step - loss: 0.7774 - categorical_accuracy: 0.7015 - val_loss: 1.0772 - val_categorical_accuracy: 0.5000\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2615_56_14.668735/model-00006-0.77741-0.70149-1.07717-0.50000.h5\n",
      "\n",
      "Epoch 00006: ReduceLROnPlateau reducing learning rate to 0.0001250000059371814.\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 46s 694ms/step - loss: 0.8710 - categorical_accuracy: 0.6318 - val_loss: 1.0210 - val_categorical_accuracy: 0.5600\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2615_56_14.668735/model-00007-0.87099-0.63184-1.02103-0.56000.h5\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 46s 683ms/step - loss: 0.7233 - categorical_accuracy: 0.6915 - val_loss: 0.9926 - val_categorical_accuracy: 0.5600\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2615_56_14.668735/model-00008-0.72335-0.69154-0.99264-0.56000.h5\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 46s 693ms/step - loss: 0.6919 - categorical_accuracy: 0.7214 - val_loss: 1.0519 - val_categorical_accuracy: 0.5400\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2615_56_14.668735/model-00009-0.69186-0.72139-1.05190-0.54000.h5\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 46s 687ms/step - loss: 0.6621 - categorical_accuracy: 0.7313 - val_loss: 1.0323 - val_categorical_accuracy: 0.5800\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2615_56_14.668735/model-00010-0.66214-0.73134-1.03231-0.58000.h5\n",
      "\n",
      "Epoch 00010: ReduceLROnPlateau reducing learning rate to 6.25000029685907e-05.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6944a4cda0>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "model2.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color='green'> Trial 3. Batchsize = 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator_ex(train_path, train_doc, 30)\n",
    "val_generator = generator_ex(val_path, val_doc, 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  val ; batch size = 30\n",
      "Source path =  train ; batch size = 30\n",
      "Epoch 1/10\n",
      " 3/67 [>.............................] - ETA: 6:58 - loss: 0.6375 - categorical_accuracy: 0.8000Batch:  4 Index: 30\n",
      "21/67 [========>.....................] - ETA: 3:36 - loss: 0.5676 - categorical_accuracy: 0.7905Batch:  23 Index: 30\n",
      "67/67 [==============================] - 121s 2s/step - loss: 0.5862 - categorical_accuracy: 0.7576 - val_loss: 1.0238 - val_categorical_accuracy: 0.6000\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2615_56_14.668735/model-00001-0.57141-0.78239-1.02382-0.60000.h5\n",
      "Epoch 2/10\n",
      "67/67 [==============================] - 36s 535ms/step - loss: 0.5804 - categorical_accuracy: 0.7413 - val_loss: 1.0149 - val_categorical_accuracy: 0.5400\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2615_56_14.668735/model-00002-0.58036-0.74129-1.01490-0.54000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 37s 550ms/step - loss: 0.5388 - categorical_accuracy: 0.7811 - val_loss: 1.0149 - val_categorical_accuracy: 0.6200\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2615_56_14.668735/model-00003-0.53881-0.78109-1.01494-0.62000.h5\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 33s 492ms/step - loss: 0.4918 - categorical_accuracy: 0.8408 - val_loss: 1.0727 - val_categorical_accuracy: 0.5600\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2615_56_14.668735/model-00004-0.49176-0.84080-1.07271-0.56000.h5\n",
      "\n",
      "Epoch 00004: ReduceLROnPlateau reducing learning rate to 3.125000148429535e-05.\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 36s 532ms/step - loss: 0.4698 - categorical_accuracy: 0.8209 - val_loss: 1.0047 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2615_56_14.668735/model-00005-0.46984-0.82090-1.00473-0.57000.h5\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 35s 522ms/step - loss: 0.5583 - categorical_accuracy: 0.7612 - val_loss: 1.0379 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2615_56_14.668735/model-00006-0.55829-0.76119-1.03790-0.57000.h5\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 35s 515ms/step - loss: 0.4897 - categorical_accuracy: 0.8060 - val_loss: 1.0430 - val_categorical_accuracy: 0.5600\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2615_56_14.668735/model-00007-0.48967-0.80597-1.04303-0.56000.h5\n",
      "\n",
      "Epoch 00007: ReduceLROnPlateau reducing learning rate to 1.5625000742147677e-05.\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 35s 517ms/step - loss: 0.5262 - categorical_accuracy: 0.7960 - val_loss: 0.9633 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2615_56_14.668735/model-00008-0.52617-0.79602-0.96334-0.57000.h5\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 36s 535ms/step - loss: 0.4957 - categorical_accuracy: 0.8159 - val_loss: 1.0508 - val_categorical_accuracy: 0.5600\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2615_56_14.668735/model-00009-0.49575-0.81592-1.05083-0.56000.h5\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 35s 521ms/step - loss: 0.5192 - categorical_accuracy: 0.8060 - val_loss: 1.1190 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2615_56_14.668735/model-00010-0.51924-0.80597-1.11898-0.57000.h5\n",
      "\n",
      "Epoch 00010: ReduceLROnPlateau reducing learning rate to 7.812500371073838e-06.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6944a4c128>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color = 'green'> Trial 4. Batchsize = 40 and epoch =20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator_ex(train_path, train_doc, 40)\n",
    "val_generator = generator_ex(val_path, val_doc, 40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  val ; batch size = 40\n",
      "Source path =  Epoch 1/20train ; batch size = 40\n",
      "\n",
      " 2/67 [..............................] - ETA: 10:01 - loss: 0.4257 - categorical_accuracy: 0.8625Batch:  3 Index: 40\n",
      "15/67 [=====>........................] - ETA: 5:54 - loss: 0.4590 - categorical_accuracy: 0.8283Batch:  17 Index: 40\n",
      "44/67 [==================>...........] - ETA: 1:40 - loss: 0.4795 - categorical_accuracy: 0.8169Batch:  29 Index: 23\n",
      "67/67 [==============================] - 249s 4s/step - loss: 0.4798 - categorical_accuracy: 0.8128 - val_loss: 0.9900 - val_categorical_accuracy: 0.5875\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2615_56_14.668735/model-00001-0.47740-0.81565-0.99002-0.58750.h5\n",
      "Epoch 2/20\n",
      "12/67 [====>.........................] - ETA: 3:22 - loss: 0.4582 - categorical_accuracy: 0.8421Batch:  35 Index: 19\n",
      "67/67 [==============================] - 174s 3s/step - loss: 0.4655 - categorical_accuracy: 0.8227 - val_loss: 1.0475 - val_categorical_accuracy: 0.5550\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2615_56_14.668735/model-00002-0.46524-0.82318-1.04748-0.55500.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 157s 2s/step - loss: 0.4698 - categorical_accuracy: 0.8209 - val_loss: 1.0050 - val_categorical_accuracy: 0.5950\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2615_56_14.668735/model-00003-0.46978-0.82090-1.00497-0.59500.h5\n",
      "\n",
      "Epoch 00003: ReduceLROnPlateau reducing learning rate to 3.906250185536919e-06.\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 157s 2s/step - loss: 0.4834 - categorical_accuracy: 0.7989 - val_loss: 1.0567 - val_categorical_accuracy: 0.5850\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2615_56_14.668735/model-00004-0.48338-0.79895-1.05673-0.58500.h5\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 159s 2s/step - loss: 0.4709 - categorical_accuracy: 0.8209 - val_loss: 1.0697 - val_categorical_accuracy: 0.5550\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2615_56_14.668735/model-00005-0.47088-0.82090-1.06972-0.55500.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 1.9531250927684596e-06.\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 157s 2s/step - loss: 0.4770 - categorical_accuracy: 0.8147 - val_loss: 1.0696 - val_categorical_accuracy: 0.5800\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2615_56_14.668735/model-00006-0.47704-0.81475-1.06961-0.58000.h5\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 163s 2s/step - loss: 0.4539 - categorical_accuracy: 0.8314 - val_loss: 1.0180 - val_categorical_accuracy: 0.5850\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2615_56_14.668735/model-00007-0.45388-0.83143-1.01799-0.58500.h5\n",
      "\n",
      "Epoch 00007: ReduceLROnPlateau reducing learning rate to 9.765625463842298e-07.\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 178s 3s/step - loss: 0.4609 - categorical_accuracy: 0.8156 - val_loss: 1.0344 - val_categorical_accuracy: 0.5950\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2615_56_14.668735/model-00008-0.46088-0.81563-1.03437-0.59500.h5\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 167s 2s/step - loss: 0.4360 - categorical_accuracy: 0.8341 - val_loss: 1.0785 - val_categorical_accuracy: 0.5650\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2615_56_14.668735/model-00009-0.43596-0.83406-1.07846-0.56500.h5\n",
      "\n",
      "Epoch 00009: ReduceLROnPlateau reducing learning rate to 4.882812731921149e-07.\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 171s 3s/step - loss: 0.4969 - categorical_accuracy: 0.7946 - val_loss: 1.0510 - val_categorical_accuracy: 0.5750\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2615_56_14.668735/model-00010-0.49694-0.79456-1.05096-0.57500.h5\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 158s 2s/step - loss: 0.4623 - categorical_accuracy: 0.8121 - val_loss: 1.0542 - val_categorical_accuracy: 0.5800\n",
      "\n",
      "Epoch 00011: saving model to model_init_2020-03-2615_56_14.668735/model-00011-0.46231-0.81212-1.05419-0.58000.h5\n",
      "\n",
      "Epoch 00011: ReduceLROnPlateau reducing learning rate to 2.4414063659605745e-07.\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 164s 2s/step - loss: 0.4622 - categorical_accuracy: 0.8270 - val_loss: 1.0032 - val_categorical_accuracy: 0.6100\n",
      "\n",
      "Epoch 00012: saving model to model_init_2020-03-2615_56_14.668735/model-00012-0.46216-0.82704-1.00317-0.61000.h5\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 166s 2s/step - loss: 0.4868 - categorical_accuracy: 0.8112 - val_loss: 1.0756 - val_categorical_accuracy: 0.5550\n",
      "\n",
      "Epoch 00013: saving model to model_init_2020-03-2615_56_14.668735/model-00013-0.48679-0.81124-1.07559-0.55500.h5\n",
      "\n",
      "Epoch 00013: ReduceLROnPlateau reducing learning rate to 1.2207031829802872e-07.\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 164s 2s/step - loss: 0.4807 - categorical_accuracy: 0.8191 - val_loss: 1.1111 - val_categorical_accuracy: 0.5550\n",
      "\n",
      "Epoch 00014: saving model to model_init_2020-03-2615_56_14.668735/model-00014-0.48066-0.81914-1.11108-0.55500.h5\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 158s 2s/step - loss: 0.4857 - categorical_accuracy: 0.8191 - val_loss: 1.0152 - val_categorical_accuracy: 0.5900\n",
      "\n",
      "Epoch 00015: saving model to model_init_2020-03-2615_56_14.668735/model-00015-0.48569-0.81914-1.01521-0.59000.h5\n",
      "\n",
      "Epoch 00015: ReduceLROnPlateau reducing learning rate to 6.103515914901436e-08.\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 157s 2s/step - loss: 0.4500 - categorical_accuracy: 0.8174 - val_loss: 1.0047 - val_categorical_accuracy: 0.5950\n",
      "\n",
      "Epoch 00016: saving model to model_init_2020-03-2615_56_14.668735/model-00016-0.45001-0.81738-1.00471-0.59500.h5\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 158s 2s/step - loss: 0.4554 - categorical_accuracy: 0.8306 - val_loss: 1.0680 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00018: saving model to model_init_2020-03-2615_56_14.668735/model-00018-0.45537-0.83055-1.06801-0.57000.h5\n",
      "Epoch 19/20\n",
      "67/67 [==============================] - 157s 2s/step - loss: 0.4574 - categorical_accuracy: 0.8314 - val_loss: 1.0239 - val_categorical_accuracy: 0.6050\n",
      "\n",
      "Epoch 00019: saving model to model_init_2020-03-2615_56_14.668735/model-00019-0.45735-0.83143-1.02391-0.60500.h5\n",
      "\n",
      "Epoch 00019: ReduceLROnPlateau reducing learning rate to 1.525878978725359e-08.\n",
      "Epoch 20/20\n",
      "67/67 [==============================] - 158s 2s/step - loss: 0.4772 - categorical_accuracy: 0.8209 - val_loss: 1.0460 - val_categorical_accuracy: 0.5750\n",
      "\n",
      "Epoch 00020: saving model to model_init_2020-03-2615_56_14.668735/model-00020-0.47715-0.82090-1.04600-0.57500.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6944a4c7b8>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color ='green'> Trial 5 . optimizer = Adadelta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_13 (Conv3D)           (None, 30, 60, 60, 32)    2624      \n",
      "_________________________________________________________________\n",
      "activation_13 (Activation)   (None, 30, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_14 (Conv3D)           (None, 30, 60, 60, 32)    27680     \n",
      "_________________________________________________________________\n",
      "activation_14 (Activation)   (None, 30, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_11 (MaxPooling (None, 10, 20, 20, 32)    0         \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (None, 10, 20, 20, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_15 (Conv3D)           (None, 10, 20, 20, 64)    55360     \n",
      "_________________________________________________________________\n",
      "activation_15 (Activation)   (None, 10, 20, 20, 64)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_16 (Conv3D)           (None, 10, 20, 20, 64)    110656    \n",
      "_________________________________________________________________\n",
      "activation_16 (Activation)   (None, 10, 20, 20, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_12 (MaxPooling (None, 4, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "dropout_9 (Dropout)          (None, 4, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_4 (Flatten)          (None, 12544)             0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 512)               6423040   \n",
      "_________________________________________________________________\n",
      "dropout_10 (Dropout)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 5)                 2565      \n",
      "=================================================================\n",
      "Total params: 6,621,925\n",
      "Trainable params: 6,621,925\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model2.compile(optimizer=keras.optimizers.Adadelta(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator_ex(train_path, train_doc, 40)\n",
    "val_generator = generator_ex(val_path, val_doc, 40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  val ; batch size = 40\n",
      "Source path =  train Epoch 1/10\n",
      "; batch size = 40\n",
      " 1/67 [..............................] - ETA: 13:44 - loss: 0.4402 - categorical_accuracy: 0.8250Batch:  3 Index: 40\n",
      "15/67 [=====>........................] - ETA: 5:54 - loss: 0.7246 - categorical_accuracy: 0.7500Batch:  17 Index: 40\n",
      "44/67 [==================>...........] - ETA: 1:41 - loss: 0.6156 - categorical_accuracy: 0.7671Batch:  29 Index: 23\n",
      "67/67 [==============================] - 251s 4s/step - loss: 0.5409 - categorical_accuracy: 0.7940 - val_loss: 1.1986 - val_categorical_accuracy: 0.5792\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2615_56_14.668735/model-00001-0.57459-0.78551-1.19856-0.57917.h5\n",
      "Epoch 2/10\n",
      "12/67 [====>.........................] - ETA: 3:17 - loss: 0.4659 - categorical_accuracy: 0.8640Batch:  35 Index: 19\n",
      "67/67 [==============================] - 179s 3s/step - loss: 0.3950 - categorical_accuracy: 0.8540 - val_loss: 1.7383 - val_categorical_accuracy: 0.5900\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2615_56_14.668735/model-00002-0.39675-0.85408-1.73828-0.59000.h5\n",
      "Epoch 3/10\n",
      "67/67 [==============================] - 174s 3s/step - loss: 0.2942 - categorical_accuracy: 0.8859 - val_loss: 1.6281 - val_categorical_accuracy: 0.5550\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2615_56_14.668735/model-00003-0.29421-0.88586-1.62810-0.55500.h5\n",
      "\n",
      "Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.5.\n",
      "Epoch 4/10\n",
      "67/67 [==============================] - 171s 3s/step - loss: 0.1300 - categorical_accuracy: 0.9535 - val_loss: 1.7470 - val_categorical_accuracy: 0.6150\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2615_56_14.668735/model-00004-0.12999-0.95347-1.74701-0.61500.h5\n",
      "Epoch 5/10\n",
      "67/67 [==============================] - 157s 2s/step - loss: 0.1243 - categorical_accuracy: 0.9579 - val_loss: 1.6487 - val_categorical_accuracy: 0.6350\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2615_56_14.668735/model-00005-0.12432-0.95786-1.64868-0.63500.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.25.\n",
      "Epoch 6/10\n",
      "67/67 [==============================] - 160s 2s/step - loss: 0.0805 - categorical_accuracy: 0.9719 - val_loss: 1.8347 - val_categorical_accuracy: 0.6150\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2615_56_14.668735/model-00006-0.08053-0.97191-1.83470-0.61500.h5\n",
      "Epoch 7/10\n",
      "67/67 [==============================] - 158s 2s/step - loss: 0.0564 - categorical_accuracy: 0.9851 - val_loss: 1.7766 - val_categorical_accuracy: 0.6350\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2615_56_14.668735/model-00007-0.05642-0.98507-1.77657-0.63500.h5\n",
      "\n",
      "Epoch 00007: ReduceLROnPlateau reducing learning rate to 0.125.\n",
      "Epoch 8/10\n",
      "67/67 [==============================] - 166s 2s/step - loss: 0.0426 - categorical_accuracy: 0.9868 - val_loss: 2.2462 - val_categorical_accuracy: 0.5950\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2615_56_14.668735/model-00008-0.04260-0.98683-2.24624-0.59500.h5\n",
      "Epoch 9/10\n",
      "67/67 [==============================] - 157s 2s/step - loss: 0.0387 - categorical_accuracy: 0.9895 - val_loss: 1.9869 - val_categorical_accuracy: 0.6350\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2615_56_14.668735/model-00009-0.03871-0.98946-1.98691-0.63500.h5\n",
      "\n",
      "Epoch 00009: ReduceLROnPlateau reducing learning rate to 0.0625.\n",
      "Epoch 10/10\n",
      "67/67 [==============================] - 176s 3s/step - loss: 0.0348 - categorical_accuracy: 0.9868 - val_loss: 2.0457 - val_categorical_accuracy: 0.6250\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2615_56_14.668735/model-00010-0.03477-0.98683-2.04569-0.62500.h5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6944a4c278>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color ='green'> Trial 6. epoch = 20 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_13 (Conv3D)           (None, 30, 60, 60, 32)    2624      \n",
      "_________________________________________________________________\n",
      "activation_13 (Activation)   (None, 30, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_14 (Conv3D)           (None, 30, 60, 60, 32)    27680     \n",
      "_________________________________________________________________\n",
      "activation_14 (Activation)   (None, 30, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_11 (MaxPooling (None, 10, 20, 20, 32)    0         \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (None, 10, 20, 20, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_15 (Conv3D)           (None, 10, 20, 20, 64)    55360     \n",
      "_________________________________________________________________\n",
      "activation_15 (Activation)   (None, 10, 20, 20, 64)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_16 (Conv3D)           (None, 10, 20, 20, 64)    110656    \n",
      "_________________________________________________________________\n",
      "activation_16 (Activation)   (None, 10, 20, 20, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_12 (MaxPooling (None, 4, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "dropout_9 (Dropout)          (None, 4, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "flatten_4 (Flatten)          (None, 12544)             0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 512)               6423040   \n",
      "_________________________________________________________________\n",
      "dropout_10 (Dropout)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 5)                 2565      \n",
      "=================================================================\n",
      "Total params: 6,621,925\n",
      "Trainable params: 6,621,925\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 20\n",
    "model2.compile(optimizer=keras.optimizers.Adadelta(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator_ex(train_path, train_doc, 40)\n",
    "val_generator = generator_ex(val_path, val_doc, 40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  val Source path =  ; batch size = 40\n",
      "Epoch 1/20\n",
      "train ; batch size = 40\n",
      " 2/67 [..............................] - ETA: 9:19 - loss: 0.0364 - categorical_accuracy: 0.9875 Batch:  3 Index: 40\n",
      "15/67 [=====>........................] - ETA: 5:51 - loss: 0.1595 - categorical_accuracy: 0.9450Batch:  17 Index: 40\n",
      "44/67 [==================>...........] - ETA: 1:38 - loss: 0.1277 - categorical_accuracy: 0.9599Batch:  29 Index: 23\n",
      "67/67 [==============================] - 245s 4s/step - loss: 0.1186 - categorical_accuracy: 0.9642 - val_loss: 2.0928 - val_categorical_accuracy: 0.6417\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2615_56_14.668735/model-00001-0.12469-0.96116-2.09280-0.64167.h5\n",
      "Epoch 2/20\n",
      "12/67 [====>.........................] - ETA: 3:11 - loss: 0.1022 - categorical_accuracy: 0.9693Batch:  35 Index: 19\n",
      "67/67 [==============================] - 158s 2s/step - loss: 0.1117 - categorical_accuracy: 0.9665 - val_loss: 2.0317 - val_categorical_accuracy: 0.6200\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2615_56_14.668735/model-00002-0.11157-0.96652-2.03175-0.62000.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 173s 3s/step - loss: 0.0694 - categorical_accuracy: 0.9754 - val_loss: 2.1078 - val_categorical_accuracy: 0.5700\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2615_56_14.668735/model-00003-0.06938-0.97542-2.10783-0.57000.h5\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 159s 2s/step - loss: 0.0635 - categorical_accuracy: 0.9789 - val_loss: 2.2750 - val_categorical_accuracy: 0.6550\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2615_56_14.668735/model-00004-0.06346-0.97893-2.27500-0.65500.h5\n",
      "\n",
      "Epoch 00004: ReduceLROnPlateau reducing learning rate to 0.5.\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 176s 3s/step - loss: 0.0351 - categorical_accuracy: 0.9895 - val_loss: 2.1503 - val_categorical_accuracy: 0.6350\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2615_56_14.668735/model-00005-0.03512-0.98946-2.15027-0.63500.h5\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 157s 2s/step - loss: 0.0196 - categorical_accuracy: 0.9921 - val_loss: 2.4744 - val_categorical_accuracy: 0.6400\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2615_56_14.668735/model-00006-0.01965-0.99210-2.47441-0.64000.h5\n",
      "\n",
      "Epoch 00006: ReduceLROnPlateau reducing learning rate to 0.25.\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 165s 2s/step - loss: 0.0217 - categorical_accuracy: 0.9912 - val_loss: 2.3739 - val_categorical_accuracy: 0.6100\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2615_56_14.668735/model-00007-0.02174-0.99122-2.37392-0.61000.h5\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 164s 2s/step - loss: 0.0186 - categorical_accuracy: 0.9956 - val_loss: 2.5866 - val_categorical_accuracy: 0.6200\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2615_56_14.668735/model-00008-0.01863-0.99561-2.58658-0.62000.h5\n",
      "\n",
      "Epoch 00008: ReduceLROnPlateau reducing learning rate to 0.125.\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 163s 2s/step - loss: 0.0168 - categorical_accuracy: 0.9947 - val_loss: 2.3962 - val_categorical_accuracy: 0.6300\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2615_56_14.668735/model-00009-0.01681-0.99473-2.39616-0.63000.h5\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 163s 2s/step - loss: 0.0104 - categorical_accuracy: 0.9965 - val_loss: 2.3751 - val_categorical_accuracy: 0.6400\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2615_56_14.668735/model-00010-0.01036-0.99649-2.37512-0.64000.h5\n",
      "\n",
      "Epoch 00010: ReduceLROnPlateau reducing learning rate to 0.0625.\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 161s 2s/step - loss: 0.0093 - categorical_accuracy: 0.9965 - val_loss: 2.7286 - val_categorical_accuracy: 0.6200\n",
      "\n",
      "Epoch 00011: saving model to model_init_2020-03-2615_56_14.668735/model-00011-0.00932-0.99649-2.72858-0.62000.h5\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 169s 3s/step - loss: 0.0103 - categorical_accuracy: 0.9974 - val_loss: 2.2706 - val_categorical_accuracy: 0.6550\n",
      "\n",
      "Epoch 00012: saving model to model_init_2020-03-2615_56_14.668735/model-00012-0.01032-0.99737-2.27057-0.65500.h5\n",
      "\n",
      "Epoch 00012: ReduceLROnPlateau reducing learning rate to 0.03125.\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 163s 2s/step - loss: 0.0058 - categorical_accuracy: 0.9991 - val_loss: 2.8012 - val_categorical_accuracy: 0.6100\n",
      "\n",
      "Epoch 00013: saving model to model_init_2020-03-2615_56_14.668735/model-00013-0.00579-0.99912-2.80122-0.61000.h5\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 165s 2s/step - loss: 0.0130 - categorical_accuracy: 0.9974 - val_loss: 2.5670 - val_categorical_accuracy: 0.6300\n",
      "\n",
      "Epoch 00014: saving model to model_init_2020-03-2615_56_14.668735/model-00014-0.01301-0.99737-2.56701-0.63000.h5\n",
      "\n",
      "Epoch 00014: ReduceLROnPlateau reducing learning rate to 0.015625.\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 158s 2s/step - loss: 0.0060 - categorical_accuracy: 0.9982 - val_loss: 2.5014 - val_categorical_accuracy: 0.6550\n",
      "\n",
      "Epoch 00015: saving model to model_init_2020-03-2615_56_14.668735/model-00015-0.00597-0.99824-2.50137-0.65500.h5\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 169s 3s/step - loss: 0.0112 - categorical_accuracy: 0.9965 - val_loss: 2.6926 - val_categorical_accuracy: 0.6100\n",
      "\n",
      "Epoch 00016: saving model to model_init_2020-03-2615_56_14.668735/model-00016-0.01116-0.99649-2.69264-0.61000.h5\n",
      "\n",
      "Epoch 00016: ReduceLROnPlateau reducing learning rate to 0.0078125.\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 162s 2s/step - loss: 0.0071 - categorical_accuracy: 0.9991 - val_loss: 2.5845 - val_categorical_accuracy: 0.6250\n",
      "\n",
      "Epoch 00017: saving model to model_init_2020-03-2615_56_14.668735/model-00017-0.00713-0.99912-2.58454-0.62500.h5\n",
      "Epoch 18/20\n",
      "67/67 [==============================] - 161s 2s/step - loss: 0.0068 - categorical_accuracy: 0.9982 - val_loss: 2.8753 - val_categorical_accuracy: 0.6200\n",
      "\n",
      "Epoch 00018: saving model to model_init_2020-03-2615_56_14.668735/model-00018-0.00681-0.99824-2.87532-0.62000.h5\n",
      "\n",
      "Epoch 00018: ReduceLROnPlateau reducing learning rate to 0.00390625.\n",
      "Epoch 19/20\n",
      "36/67 [===============>..............] - ETA: 1:16 - loss: 0.0048 - categorical_accuracy: 1.0000"
     ]
    }
   ],
   "source": [
    "model2.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extra"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_generator = generator_ex(train_path, train_doc, 64)\n",
    "val_generator = generator_ex(val_path, val_doc, 64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv3d_19 (Conv3D)           (None, 30, 60, 60, 32)    2624      \n",
      "_________________________________________________________________\n",
      "activation_19 (Activation)   (None, 30, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_20 (Conv3D)           (None, 30, 60, 60, 32)    27680     \n",
      "_________________________________________________________________\n",
      "activation_20 (Activation)   (None, 30, 60, 60, 32)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_12 (MaxPooling (None, 10, 20, 20, 32)    0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 10, 20, 20, 32)    128       \n",
      "_________________________________________________________________\n",
      "conv3d_21 (Conv3D)           (None, 10, 20, 20, 64)    55360     \n",
      "_________________________________________________________________\n",
      "activation_21 (Activation)   (None, 10, 20, 20, 64)    0         \n",
      "_________________________________________________________________\n",
      "conv3d_22 (Conv3D)           (None, 10, 20, 20, 64)    110656    \n",
      "_________________________________________________________________\n",
      "activation_22 (Activation)   (None, 10, 20, 20, 64)    0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_13 (MaxPooling (None, 4, 7, 7, 64)       0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_6 (Batch (None, 4, 7, 7, 64)       256       \n",
      "_________________________________________________________________\n",
      "conv3d_23 (Conv3D)           (None, 4, 7, 7, 128)      221312    \n",
      "_________________________________________________________________\n",
      "activation_23 (Activation)   (None, 4, 7, 7, 128)      0         \n",
      "_________________________________________________________________\n",
      "conv3d_24 (Conv3D)           (None, 4, 7, 7, 128)      442496    \n",
      "_________________________________________________________________\n",
      "activation_24 (Activation)   (None, 4, 7, 7, 128)      0         \n",
      "_________________________________________________________________\n",
      "max_pooling3d_14 (MaxPooling (None, 2, 3, 3, 128)      0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_7 (Batch (None, 2, 3, 3, 128)      512       \n",
      "_________________________________________________________________\n",
      "flatten_5 (Flatten)          (None, 2304)              0         \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 512)               1180160   \n",
      "_________________________________________________________________\n",
      "dropout_13 (Dropout)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 64)                32832     \n",
      "_________________________________________________________________\n",
      "dense_13 (Dense)             (None, 5)                 325       \n",
      "=================================================================\n",
      "Total params: 2,074,341\n",
      "Trainable params: 2,073,893\n",
      "Non-trainable params: 448\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define model b\n",
    "model2 = Sequential()\n",
    "model2.add(Conv3D(32, kernel_size=(3, 3, 3), input_shape=(x,y,z,channel), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(Conv3D(32, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model2.add(BatchNormalization())\n",
    "\n",
    "model2.add(Conv3D(64, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(Conv3D(64, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model2.add(BatchNormalization())\n",
    "\n",
    "model2.add(Conv3D(128, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(Conv3D(128, kernel_size=(3, 3, 3), padding='same'))\n",
    "model2.add(Activation('relu'))\n",
    "model2.add(MaxPooling3D(pool_size=(3, 3, 3), padding='same'))\n",
    "model2.add(BatchNormalization())\n",
    "\n",
    "model2.add(Flatten())\n",
    "model2.add(Dense(512, activation='relu'))\n",
    "model2.add(Dropout(0.5))\n",
    "model2.add(Dense(64, activation='relu'))\n",
    "model2.add(Dense(classes, activation='softmax'))\n",
    "\n",
    "model2.compile(optimizer=keras.optimizers.Adam(), loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source path =  val ; batch size =Source path =  train ; batch size = Epoch 1/20 64\n",
      "64\n",
      "\n",
      "Batch:  2 Index: 64\n",
      " 2/67 [..............................] - ETA: 16:26 - loss: 2.1660 - categorical_accuracy: 0.2344Batch:  3 Index: 36\n",
      " 4/67 [>.............................] - ETA: 14:53 - loss: 1.9235 - categorical_accuracy: 0.3047Batch:  4 Index: 28\n",
      " 9/67 [===>..........................] - ETA: 11:06 - loss: 1.6732 - categorical_accuracy: 0.3333Batch:  11 Index: 64\n",
      "38/67 [================>.............] - ETA: 2:23 - loss: 1.4649 - categorical_accuracy: 0.4005Batch:  29 Index: 23\n",
      "67/67 [==============================] - 261s 4s/step - loss: 1.3593 - categorical_accuracy: 0.4530 - val_loss: 3.0489 - val_categorical_accuracy: 0.3101\n",
      "\n",
      "Epoch 00001: saving model to model_init_2020-03-2607_25_29.400376/model-00001-1.42852-0.42469-3.04888-0.31013.h5\n",
      "Epoch 2/20\n",
      " 4/67 [>.............................] - ETA: 2:29 - loss: 1.0707 - categorical_accuracy: 0.4342Batch:  7 Index: 16\n",
      " 6/67 [=>............................] - ETA: 2:50 - loss: 1.2287 - categorical_accuracy: 0.4035Batch:  35 Index: 19\n",
      "67/67 [==============================] - 146s 2s/step - loss: 1.1367 - categorical_accuracy: 0.5308 - val_loss: 2.1508 - val_categorical_accuracy: 0.3100\n",
      "\n",
      "Epoch 00002: saving model to model_init_2020-03-2607_25_29.400376/model-00002-1.13795-0.52905-2.15082-0.31000.h5\n",
      "Epoch 3/20\n",
      "67/67 [==============================] - 146s 2s/step - loss: 0.9393 - categorical_accuracy: 0.6093 - val_loss: 0.9489 - val_categorical_accuracy: 0.6000\n",
      "\n",
      "Epoch 00003: saving model to model_init_2020-03-2607_25_29.400376/model-00003-0.93928-0.60931-0.94894-0.60000.h5\n",
      "Epoch 4/20\n",
      "67/67 [==============================] - 143s 2s/step - loss: 0.7152 - categorical_accuracy: 0.7243 - val_loss: 2.1009 - val_categorical_accuracy: 0.5000\n",
      "\n",
      "Epoch 00004: saving model to model_init_2020-03-2607_25_29.400376/model-00004-0.71519-0.72432-2.10085-0.50000.h5\n",
      "Epoch 5/20\n",
      "67/67 [==============================] - 154s 2s/step - loss: 0.5614 - categorical_accuracy: 0.7761 - val_loss: 2.0016 - val_categorical_accuracy: 0.3750\n",
      "\n",
      "Epoch 00005: saving model to model_init_2020-03-2607_25_29.400376/model-00005-0.56136-0.77612-2.00160-0.37500.h5\n",
      "\n",
      "Epoch 00005: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
      "Epoch 6/20\n",
      "67/67 [==============================] - 160s 2s/step - loss: 0.3685 - categorical_accuracy: 0.8648 - val_loss: 0.5473 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00006: saving model to model_init_2020-03-2607_25_29.400376/model-00006-0.36845-0.86479-0.54726-0.80000.h5\n",
      "Epoch 7/20\n",
      "67/67 [==============================] - 147s 2s/step - loss: 0.2024 - categorical_accuracy: 0.9271 - val_loss: 0.5500 - val_categorical_accuracy: 0.7750\n",
      "\n",
      "Epoch 00007: saving model to model_init_2020-03-2607_25_29.400376/model-00007-0.20235-0.92713-0.54996-0.77500.h5\n",
      "Epoch 8/20\n",
      "67/67 [==============================] - 144s 2s/step - loss: 0.1249 - categorical_accuracy: 0.9614 - val_loss: 1.0412 - val_categorical_accuracy: 0.7250\n",
      "\n",
      "Epoch 00008: saving model to model_init_2020-03-2607_25_29.400376/model-00008-0.12491-0.96137-1.04121-0.72500.h5\n",
      "\n",
      "Epoch 00008: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n",
      "Epoch 9/20\n",
      "67/67 [==============================] - 142s 2s/step - loss: 0.0768 - categorical_accuracy: 0.9754 - val_loss: 1.1760 - val_categorical_accuracy: 0.7250\n",
      "\n",
      "Epoch 00009: saving model to model_init_2020-03-2607_25_29.400376/model-00009-0.07685-0.97542-1.17599-0.72500.h5\n",
      "Epoch 10/20\n",
      "67/67 [==============================] - 169s 3s/step - loss: 0.0417 - categorical_accuracy: 0.9921 - val_loss: 0.8455 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00010: saving model to model_init_2020-03-2607_25_29.400376/model-00010-0.04168-0.99210-0.84549-0.80000.h5\n",
      "\n",
      "Epoch 00010: ReduceLROnPlateau reducing learning rate to 0.0001250000059371814.\n",
      "Epoch 11/20\n",
      "67/67 [==============================] - 155s 2s/step - loss: 0.0283 - categorical_accuracy: 0.9930 - val_loss: 1.2287 - val_categorical_accuracy: 0.6500\n",
      "\n",
      "Epoch 00011: saving model to model_init_2020-03-2607_25_29.400376/model-00011-0.02827-0.99298-1.22874-0.65000.h5\n",
      "Epoch 12/20\n",
      "67/67 [==============================] - 143s 2s/step - loss: 0.0258 - categorical_accuracy: 0.9939 - val_loss: 0.5280 - val_categorical_accuracy: 0.8500\n",
      "\n",
      "Epoch 00012: saving model to model_init_2020-03-2607_25_29.400376/model-00012-0.02582-0.99385-0.52801-0.85000.h5\n",
      "Epoch 13/20\n",
      "67/67 [==============================] - 145s 2s/step - loss: 0.0243 - categorical_accuracy: 0.9947 - val_loss: 0.7794 - val_categorical_accuracy: 0.7250\n",
      "\n",
      "Epoch 00013: saving model to model_init_2020-03-2607_25_29.400376/model-00013-0.02429-0.99473-0.77935-0.72500.h5\n",
      "Epoch 14/20\n",
      "67/67 [==============================] - 142s 2s/step - loss: 0.0280 - categorical_accuracy: 0.9903 - val_loss: 0.5852 - val_categorical_accuracy: 0.8000\n",
      "\n",
      "Epoch 00014: saving model to model_init_2020-03-2607_25_29.400376/model-00014-0.02801-0.99034-0.58519-0.80000.h5\n",
      "\n",
      "Epoch 00014: ReduceLROnPlateau reducing learning rate to 6.25000029685907e-05.\n",
      "Epoch 15/20\n",
      "67/67 [==============================] - 160s 2s/step - loss: 0.0297 - categorical_accuracy: 0.9912 - val_loss: 1.1838 - val_categorical_accuracy: 0.7250\n",
      "\n",
      "Epoch 00015: saving model to model_init_2020-03-2607_25_29.400376/model-00015-0.02973-0.99122-1.18375-0.72500.h5\n",
      "Epoch 16/20\n",
      "67/67 [==============================] - 144s 2s/step - loss: 0.0153 - categorical_accuracy: 0.9965 - val_loss: 0.5696 - val_categorical_accuracy: 0.8500\n",
      "\n",
      "Epoch 00016: saving model to model_init_2020-03-2607_25_29.400376/model-00016-0.01525-0.99649-0.56959-0.85000.h5\n",
      "\n",
      "Epoch 00016: ReduceLROnPlateau reducing learning rate to 3.125000148429535e-05.\n",
      "Epoch 17/20\n",
      "67/67 [==============================] - 158s 2s/step - loss: 0.0109 - categorical_accuracy: 1.0000 - val_loss: 0.7720 - val_categorical_accuracy: 0.7500\n",
      "\n",
      "Epoch 00017: saving model to model_init_2020-03-2607_25_29.400376/model-00017-0.01091-1.00000-0.77197-0.75000.h5\n",
      "Epoch 18/20\n",
      "67/67 [==============================] - 163s 2s/step - loss: 0.0164 - categorical_accuracy: 0.9982 - val_loss: 0.9446 - val_categorical_accuracy: 0.7750\n",
      "\n",
      "Epoch 00018: saving model to model_init_2020-03-2607_25_29.400376/model-00018-0.01644-0.99824-0.94460-0.77500.h5\n",
      "\n",
      "Epoch 00018: ReduceLROnPlateau reducing learning rate to 1.5625000742147677e-05.\n",
      "Epoch 19/20\n",
      "67/67 [==============================] - 151s 2s/step - loss: 0.0139 - categorical_accuracy: 0.9974 - val_loss: 0.9672 - val_categorical_accuracy: 0.6750\n",
      "\n",
      "Epoch 00019: saving model to model_init_2020-03-2607_25_29.400376/model-00019-0.01395-0.99737-0.96725-0.67500.h5\n",
      "Epoch 20/20\n",
      "67/67 [==============================] - 146s 2s/step - loss: 0.0131 - categorical_accuracy: 0.9991 - val_loss: 1.1733 - val_categorical_accuracy: 0.7250\n",
      "\n",
      "Epoch 00020: saving model to model_init_2020-03-2607_25_29.400376/model-00020-0.01310-0.99912-1.17329-0.72500.h5\n",
      "\n",
      "Epoch 00020: ReduceLROnPlateau reducing learning rate to 7.812500371073838e-06.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f9988308630>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=20, verbose=1, \n",
    "                    callbacks=callbacks_list, validation_data=val_generator, \n",
    "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}