10. Convolutional Neural Networks II.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "10. Convolutional Neural Networks II.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DataScienceUB/DeepLearningMaster2019/blob/master/10.%20Convolutional%20Neural%20Networks%20II.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "zlMderMf6iaX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 10. Convolutional Neural Networks II\n",
        "## Large Convolutional Networks\n",
        "\n",
        "There are several architectures in the field of Convolutional Networks that have a name. The most common are:\n",
        "\n",
        "+ **LeNet**, 1990’s. \n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/conv2.png?raw=1\" alt=\"\" style=\"width: 600px;\"/> \n",
        "</center>\n",
        "\n",
        "\n",
        "\n",
        "+ **AlexNet**. 2012.\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/alexnet.png?raw=1\" alt=\"\" style=\"width: 700px;\"/>\n",
        "(Source: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)\n",
        "</center>\n",
        "\n",
        "> AlexNet has about 60 million parameters!\n",
        "\n",
        "\n",
        "+ **ZF Net**. The ILSVRC 2013 winner was a Convolutional Network from Matthew Zeiler and Rob Fergus. It became known as the ZFNet (short for Zeiler & Fergus Net). It was an improvement on AlexNet by tweaking the architecture hyperparameters, in particular by expanding the size of the middle convolutional layers and making the stride and filter size on the first layer smaller.\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/zfnet.png?raw=1\" alt=\"\" style=\"width: 700px;\"/> \n",
        "(Source: https://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf)\n",
        "</center>\n",
        "\n",
        "\n",
        "+ **VGGNet**. The runner-up in ILSVRC 2014 was the network from Karen Simonyan and Andrew Zisserman that became known as the VGGNet. Its main contribution was in showing that the depth of the network is a critical component for good performance. Their final best network contains 16 CONV/FC layers and, appealingly, features an **extremely homogeneous architecture that only performs 3x3 convolutions and 2x2 pooling from the beginning to the end**. A downside of the VGGNet is that it is more expensive to evaluate and uses a lot more memory and parameters. Most of these parameters are in the first fully connected layer, and it was since found that these FC layers can be removed with no performance downgrade, significantly reducing the number of necessary parameters.\n",
        "\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/vgg16.png?raw=1\" alt=\"\" style=\"width: 600px;\"/> \n",
        "(Source: https://blog.heuritech.com/2016/02/29/a-brief-report-of-the-heuritech-deep-learning-meetup-5/)\n",
        "</center>"
      ]
    },
    {
      "metadata": {
        "id": "xZnRS5306iaa",
        "colab_type": "code",
        "outputId": "fe4da8fb-84f5-4ad5-fd59-0b6c1284232e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1129
        }
      },
      "cell_type": "code",
      "source": [
        "# Small VGG-like convnet in Keras\n",
        "\n",
        "import numpy as np\n",
        "import keras\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, Flatten\n",
        "from keras.layers import Conv2D, MaxPooling2D\n",
        "from keras.optimizers import SGD\n",
        "\n",
        "# Generate dummy data\n",
        "\n",
        "def to_categorical(y, num_classes=None):\n",
        "    \"\"\"\n",
        "    Converts a class vector (integers) to binary class matrix.\n",
        "    \"\"\"\n",
        "    y = np.array(y, dtype='int').ravel()\n",
        "    if not num_classes:\n",
        "        num_classes = np.max(y) + 1\n",
        "    n = y.shape[0]\n",
        "    categorical = np.zeros((n, num_classes))\n",
        "    categorical[np.arange(n), y] = 1\n",
        "    return categorical\n",
        "\n",
        "x_train = np.random.random((100, 100, 100, 3))\n",
        "y_train = to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)\n",
        "x_test = np.random.random((20, 100, 100, 3))\n",
        "y_test = to_categorical(np.random.randint(10, size=(20, 1)), num_classes=10)\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(100, 100, 3)))\n",
        "model.add(Conv2D(32, (3, 3), activation=\"relu\"))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Dropout(0.25))\n",
        "\n",
        "model.add(Conv2D(32, (3, 3), activation=\"relu\"))\n",
        "model.add(Conv2D(32, (3, 3), activation=\"relu\"))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Dropout(0.25))\n",
        "\n",
        "model.add(Flatten())\n",
        "model.add(Dense(256, activation='relu'))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(10, activation='softmax'))\n",
        "\n",
        "sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)\n",
        "model.compile(loss='categorical_crossentropy', optimizer=sgd)\n",
        "print(model.summary())\n",
        "\n",
        "model.fit(x_train, y_train, batch_size=32, epochs=10)\n",
        "score = model.evaluate(x_test, y_test, batch_size=32)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_1 (Conv2D)            (None, 98, 98, 32)        896       \n",
            "_________________________________________________________________\n",
            "conv2d_2 (Conv2D)            (None, 96, 96, 32)        9248      \n",
            "_________________________________________________________________\n",
            "max_pooling2d_1 (MaxPooling2 (None, 48, 48, 32)        0         \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 48, 48, 32)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_3 (Conv2D)            (None, 46, 46, 32)        9248      \n",
            "_________________________________________________________________\n",
            "conv2d_4 (Conv2D)            (None, 44, 44, 32)        9248      \n",
            "_________________________________________________________________\n",
            "max_pooling2d_2 (MaxPooling2 (None, 22, 22, 32)        0         \n",
            "_________________________________________________________________\n",
            "dropout_2 (Dropout)          (None, 22, 22, 32)        0         \n",
            "_________________________________________________________________\n",
            "flatten_1 (Flatten)          (None, 15488)             0         \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 256)               3965184   \n",
            "_________________________________________________________________\n",
            "dropout_3 (Dropout)          (None, 256)               0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 10)                2570      \n",
            "=================================================================\n",
            "Total params: 3,996,394\n",
            "Trainable params: 3,996,394\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "Epoch 1/10\n",
            "100/100 [==============================] - 6s 57ms/step - loss: 2.3112\n",
            "Epoch 2/10\n",
            "100/100 [==============================] - 0s 747us/step - loss: 2.3700\n",
            "Epoch 3/10\n",
            "100/100 [==============================] - 0s 746us/step - loss: 2.3099\n",
            "Epoch 4/10\n",
            "100/100 [==============================] - 0s 719us/step - loss: 2.2850\n",
            "Epoch 5/10\n",
            "100/100 [==============================] - 0s 725us/step - loss: 2.2990\n",
            "Epoch 6/10\n",
            "100/100 [==============================] - 0s 740us/step - loss: 2.3016\n",
            "Epoch 7/10\n",
            "100/100 [==============================] - 0s 778us/step - loss: 2.3113\n",
            "Epoch 8/10\n",
            "100/100 [==============================] - 0s 792us/step - loss: 2.2885\n",
            "Epoch 9/10\n",
            "100/100 [==============================] - 0s 758us/step - loss: 2.2925\n",
            "Epoch 10/10\n",
            "100/100 [==============================] - 0s 867us/step - loss: 2.2957\n",
            "20/20 [==============================] - 0s 6ms/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WMiG37Tg6iah",
        "colab_type": "code",
        "outputId": "33c25635-a733-4069-a217-fae2215c046e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "# how to compute the numer of trainable and non trainable weights in a model\n",
        "\n",
        "from keras import backend as K\n",
        "import numpy\n",
        "\n",
        "trainable_count = int(numpy.sum([K.count_params(p) for p in set(model.trainable_weights)]))\n",
        "\n",
        "non_trainable_count = int(numpy.sum([K.count_params(p) for p in set(model.non_trainable_weights)]))\n",
        "\n",
        "print(trainable_count,non_trainable_count)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "3996394 0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "D3R-Gg0c6iak",
        "colab_type": "code",
        "outputId": "22fb793c-cd46-433f-afc4-c0e605890fd3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "# how to compute the memory allocated by the activations of a model\n",
        "\n",
        "batch = 1\n",
        "shapes_count = int(numpy.sum([numpy.prod(numpy.array([s if isinstance(s, int) \n",
        "                                                      else 1 for s in l.output_shape])) \n",
        "                              for l in model.layers]))\n",
        "memory = shapes_count * 4 * batch\n",
        "\n",
        "print(memory)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "3643436\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "31DIGKll6iap",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Exercise**\n",
        "\n",
        "+ Why do we have 896 parameters in the ``convolution2d_1`` layer of the previous example?\n",
        "\n",
        "+ Compute the number of parameters of the original VGG16 (all CONV layers are 3x3).\n",
        "> The VGG16 architecture is: INPUT: [224x224x3] $\\rightarrow$ CONV3-64: [224x224x64] $\\rightarrow$ CONV3-64: [224x224x64] $\\rightarrow$ POOL2: [112x112x64] $\\rightarrow$ CONV3-128: [112x112x128]  $\\rightarrow$ CONV3-128: [112x112x128]  $\\rightarrow$ POOL2: [56x56x128] $\\rightarrow$ CONV3-256: [56x56x256] $\\rightarrow$ CONV3-256: [56x56x256]  $\\rightarrow$ CONV3-256: [56x56x256]  $\\rightarrow$ POOL2: [28x28x256]  $\\rightarrow$ CONV3-512: [28x28x512]  $\\rightarrow$ CONV3-512: [28x28x512]  $\\rightarrow$ CONV3-512: [28x28x512]  $\\rightarrow$ POOL2: [14x14x512] $\\rightarrow$ CONV3-512: [14x14x512]  $\\rightarrow$ CONV3-512: [14x14x512]  $\\rightarrow$ CONV3-512: [14x14x512]  $\\rightarrow$ POOL2: [7x7x512]  $\\rightarrow$ FC: [1x1x4096]  $\\rightarrow$ FC: [1x1x4096]  $\\rightarrow$ FC: [1x1x1000].\n",
        "\n",
        "+ The largest bottleneck to be aware of when constructing ConvNet architectures is the memory bottleneck. What is the necessary memory size (supposing that we need 4 bytes for each element) to store intermediate data?\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "oyrajG_-6iaq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# your code here\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aYqr5XV66ias",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## More Large Convolutional Networks\n",
        "\n",
        "\n",
        "+ **GoogLeNet**. The ILSVRC 2014 winner was a Convolutional Network from Szegedy et al. from Google. Its main contribution was the development of an **Inception Module** that dramatically reduced the number of parameters in the network (4M, compared to VGG with 138,357,544). Additionally, this paper uses Average Pooling instead of Fully Connected layers at the top of the ConvNet, eliminating a large amount of parameters that do not seem to matter much. There are also several followup versions to the GoogLeNet, most recently Inception-v4.\n",
        "\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/googlenet.png?raw=1\" alt=\"\" style=\"width: 400px;\"/> \n",
        "GoogLeNet Architecture. Source: https://arxiv.org/pdf/1409.4842v1.pdf\n",
        "</center>\n",
        "\n",
        "Blue Box: Convolution | Red Box: Pooling | Yelow Box: Softmax | Green Box: Normalization\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/inception.png?raw=1\" alt=\"\" style=\"width: 400px;\"/> \n",
        "Inception Layer. Source: https://arxiv.org/pdf/1409.4842v1.pdf\n",
        "</center>\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/googlenet2.png?raw=1\" alt=\"\" style=\"width: 600px;\"/> \n",
        "GoogLeNet parameters and ops. Source: https://arxiv.org/pdf/1409.4842v1.pdf\n",
        "</center>\n",
        "\n",
        "> What is the role of 1x1 convolutions?\n",
        "\n",
        "+ **ResNet**. Residual Network developed by Kaiming He et al. was the winner of ILSVRC 2015. It features special **skip connections** and a heavy use of batch normalization. A Residual Network, or ResNet is a neural network architecture which solves the problem of vanishing gradients in the simplest way possible. If there is trouble sending the gradient signal backwards, why not provide the network with a shortcut at each layer to make things happen more smoothly? The architecture is also missing fully connected layers at the end of the network. \n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/res1.png?raw=1\" alt=\"\" style=\"width: 400px;\"/> \n",
        "(Source: https://arxiv.org/pdf/1512.03385.pdf)\n",
        "</center>\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/resnet.png?raw=1\" alt=\"\" style=\"width: 400px;\"/> \n",
        "    \n",
        "(Source: https://arxiv.org/pdf/1512.03385.pdf)\n",
        "</center>"
      ]
    },
    {
      "metadata": {
        "id": "XOHYi9jh6iat",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##  Deeper is better?\n",
        "\n",
        "When it comes to neural network design, the trend in the past few years has pointed in one direction: deeper. \n",
        "\n",
        "Whereas the state of the art only a few years ago consisted of networks which were roughly twelve layers deep, it is now not surprising to come across networks which are hundreds of layers deep. \n",
        "\n",
        "This move hasn’t just consisted of greater depth for depths sake. For many applications, the most prominent of which being object classification, the deeper the neural network, the better the performance.\n",
        "\n",
        "So the problem is to design a network in which the gradient can more easily reach all the layers of a network which might be dozens, or even hundreds of layers deep. This is the goal behind some of state of the art architectures: ResNets, HighwayNets, and DenseNets.\n",
        "\n",
        "**HighwayNets** builds on the ResNet in a pretty intuitive way. The Highway Network preserves the shortcuts introduced in the ResNet, but augments them with a learnable parameter to determine to what extent each layer should be a skip connection or a nonlinear connection. Layers in a Highway Network are defined as follows:\n",
        "\n",
        " $$ y = H(x, W_H) \\cdot T(x,W_T) + x \\cdot C(x, W_C) $$\n",
        " \n",
        "In this equation we can see an outline of two kinds of layers discussed: $y = H(x,W_H)$ mirrors the traditional layer, and $y = H(x,W_H) + x$ mirrors our residual unit. \n",
        "\n",
        "The traditional layer can be implemented as:\n",
        "\n",
        "```python\n",
        "def dense(x, input_size, output_size, activation):\n",
        "  W = tf.Variable(tf.truncated_normal([input_size, output_size], stddev=0.1), name=\"weight\")\n",
        "  b = tf.Variable(tf.constant(0.1, shape=[output_size]), name=\"bias\")\n",
        "  y = activation(tf.matmul(x, W) + b)\n",
        "  return y\n",
        "```\n",
        "\n",
        "What is new is the $T(x,W_t)$, the transform gate function and $C(x,W_C) = 1 - T(x,W_t)$, the carry gate function. What happens is that when the transform gate is 1, we pass through our activation (H) and suppress the carry gate (since it will be 0). When the carry gate is 1, we pass through the unmodified input (x), while the activation is suppressed.\n",
        "\n",
        "```python\n",
        "def highway(x, size, activation, carry_bias=-1.0):\n",
        "  W_T = tf.Variable(tf.truncated_normal([size, size], stddev=0.1), name=\"weight_transform\")\n",
        "  b_T = tf.Variable(tf.constant(carry_bias, shape=[size]), name=\"bias_transform\")\n",
        "\n",
        "  W = tf.Variable(tf.truncated_normal([size, size], stddev=0.1), name=\"weight\")\n",
        "  b = tf.Variable(tf.constant(0.1, shape=[size]), name=\"bias\")\n",
        "\n",
        "  T = tf.sigmoid(tf.matmul(x, W_T) + b_T, name=\"transform_gate\")\n",
        "  H = activation(tf.matmul(x, W) + b, name=\"activation\")\n",
        "  C = tf.sub(1.0, T, name=\"carry_gate\")\n",
        "\n",
        "  y = tf.add(tf.mul(H, T), tf.mul(x, C), \"y\")\n",
        "  return y\n",
        "```\n",
        "\n",
        "With this kind of network you can train models with hundreds of layers.\n",
        "\n",
        "**DenseNet** takes the insights of the skip connection to the extreme. The idea here is that if connecting a skip connection from the previous layer improves performance, why not connect every layer to every other layer? That way there is always a direct route for the information backwards through the network.\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/densenet.png?raw=1\" alt=\"\" style=\"width: 700px;\"/> \n",
        "(Source: https://arxiv.org/abs/1608.06993)\n",
        "</center>\n",
        "\n",
        "Instead of using an addition however, the DenseNet relies on stacking of layers. Mathematically this looks like:\n",
        "\n",
        "$$ y = f(x, x-1, x-2, \\dots, x-n) $$\n",
        "\n",
        "This architecture makes intuitive sense in both the feedforward and feed backward settings. In the feed-forward setting, a task may benefit from being able to get low-level feature activations in addition to high level feature activations. In classifying objects for example, a lower layer of the network may determine edges in an image, whereas a higher layer would determine larger-scale features such as presence of faces. There may be cases where being able to use information about edges can help in determining the correct object in a complex scene. In the backwards case, having all the layers connected allows us to quickly send gradients to their respective places in the network easily.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Zg29A7wJ6iau",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Fully Convolutional Networks\n",
        "\n",
        "(Source: http://cs231n.github.io/convolutional-networks/#convert)\n",
        "\n",
        "The only difference between **Fully Connected (FC)** and **Convolutional (CONV)** layers is that the neurons in the CONV layer are connected only to a local region in the input, and that many of the neurons in a CONV volume share parameters. \n",
        "\n",
        "However, the neurons in both layers still compute dot products, so their functional form is identical.\n",
        "\n",
        "Then, it is easy to see that for any CONV layer there is an FC layer that implements the same forward function. The weight matrix would be a large matrix that is mostly zero except for at certain blocks (due to local connectivity) where the weights in many of the blocks are equal (due to parameter sharing).\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/t10.png?raw=1\" alt=\"\" style=\"width: 800px;\"/> \n",
        "</center>\n",
        "\n",
        "\n",
        "Conversely, any FC layer can be converted to a CONV layer. \n",
        "\n",
        "Let $F$ be the receptive field size of the CONV layer neurons and $K$ the depth (number of bands) of the CONV layer.\n",
        "\n",
        "For example, an FC layer with $K=4096$ that is looking at some input volume of size $7×7×512$ (this is a tensor with size $(7×7×512, 4096$) can be equivalently expressed as $4096$ CONV layers with $F=7,K=512$ (this are $4096$ $(7,7,512)$ matrices). \n",
        "\n",
        "This can be very useful, bacause now we can apply the network to arbitrary large images!"
      ]
    },
    {
      "metadata": {
        "id": "B3HZ8p-g6iau",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Object Detection and Segmentation\n",
        "(Source: https://blog.athelas.com/a-brief-history-of-cnns-in-image-segmentation-from-r-cnn-to-mask-r-cnn-34ea83205de4)\n",
        "\n",
        "In classification, there’s generally an image with a single object as the focus and the task is to say what that image is. But when we look at the world around us, we see complicated sights with multiple overlapping objects, and different backgrounds and we not only classify these different objects but also identify their boundaries, differences, and relations to one another!\n",
        "\n",
        "To what extent do CNN generalize to object detection? Object detection is the task of finding the different objects in an image and classifying them.\n",
        "\n",
        "### R-CNN\n",
        "\n",
        "A team, comprised of Ross Girshick (a name we’ll see again), Jeff Donahue, and Trevor Darrel found that this problem can be solved with AlexNet by testing on the PASCAL VOC Challenge, a popular object detection challenge akin to ImageNet.\n",
        "\n",
        "The goal of R-CNN is to take in an image, and correctly identify where the main objects (via a bounding box) in the image.\n",
        "\n",
        ">Inputs: Image\n",
        "\n",
        ">Outputs: Bounding boxes + labels for each object in the image.\n",
        "\n",
        "But how do we find out where these bounding boxes are? R-CNN proposes a bunch of boxes in the image and see if any of them actually correspond to an object.\n",
        "\n",
        "R-CNN creates these bounding boxes, or region proposals, using a process called Selective Search (see http://www.cs.cornell.edu/courses/cs7670/2014sp/slides/VisionSeminar14.pdf). \n",
        "\n",
        "At a high level, Selective Search looks at the image through windows of different sizes, and for each size tries to group together adjacent pixels by texture, color, or intensity to identify objects.\n",
        "\n",
        "Once the proposals are created, R-CNN warps the region to a standard square size and passes it through to a modified version of AlexNet.\n",
        "\n",
        "On the final layer of the CNN, R-CNN adds a Support Vector Machine (SVM) that simply classifies whether this is an object, and if so what object. \n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/rcnn.png?raw=1\" alt=\"\" style=\"width: 800px;\"/> \n",
        "</center>\n",
        "\n",
        "Now, having found the object in the box, can we tighten the box to fit the true dimensions of the object? We can, and this is the final step of R-CNN. R-CNN runs a simple linear regression on the region proposal to generate tighter bounding box coordinates to get our final result. Here are the inputs and outputs of this regression model:\n",
        "\n",
        "> Inputs: sub-regions of the image corresponding to objects.\n",
        "\n",
        "> Outputs: New bounding box coordinates for the object in the sub-region.\n",
        "\n",
        "\n",
        "\n",
        "### Fast R-CNN\n",
        "\n",
        "R-CNN works really well, but is really quite slow for a few simple reasons:\n",
        "+ It requires a forward pass of the CNN (AlexNet) for every single region proposal for every single image (that’s around 2000 forward passes per image!).\n",
        "+ It has to train three different models separately - the CNN to generate image features, the classifier that predicts the class, and the regression model to tighten the bounding boxes. This makes the pipeline extremely hard to train.\n",
        "\n",
        "In 2015, Ross Girshick, the first author of R-CNN, solved both these problems, leading to Fast R-CNN. \n",
        "\n",
        "For the forward pass of the CNN, Girshick realized that for each image, a lot of proposed regions for the image invariably overlapped causing us to run the same CNN computation again and again (~2000 times!). His insight was simple — Why not run the CNN just once per image and then find a way to share that computation across the ~2000 proposals?\n",
        "\n",
        "This is exactly what Fast R-CNN does using a technique known as **RoIPool** (Region of Interest Pooling). At its core, RoIPool shares the forward pass of a CNN for an image across its subregions. In the image below, notice how the CNN features for each region are obtained by selecting a corresponding region from the CNN’s feature map. Then, the features in each region are pooled (usually using max pooling). So all it takes us is one pass of the original image as opposed to ~2000!\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/fastrcnn.png?raw=1\" alt=\"\" style=\"width: 600px;\"/> \n",
        "(Source: Stanford’s CS231N slides by Fei Fei Li, Andrei Karpathy, and Justin Johnson)\n",
        "</center>\n",
        "\n",
        "\n",
        "The second insight of Fast R-CNN is to jointly train the CNN, classifier, and bounding box regressor in a single model. Where earlier we had different models to extract image features (CNN), classify (SVM), and tighten bounding boxes (regressor), Fast R-CNN instead used a single network to compute all three.\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/fastrcnn2.png?raw=1\" alt=\"\" style=\"width: 400px;\"/> \n",
        "(Source: https://www.slideshare.net/simplyinsimple/detection-52781995)\n",
        "</center>\n",
        "\n",
        "### Faster R-CNN\n",
        "\n",
        "Even with all these advancements, there was still one remaining bottleneck in the Fast R-CNN process — the region proposer. As we saw, the very first step to detecting the locations of objects is generating a bunch of potential bounding boxes or regions of interest to test. In Fast R-CNN, these proposals were created using Selective Search, a fairly slow process that was found to be the bottleneck of the overall process.\n",
        "\n",
        "In the middle 2015, a team at Microsoft Research composed of Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, found a way to make the region proposal step almost cost free through an architecture they (creatively) named Faster R-CNN.\n",
        "\n",
        "The insight of Faster R-CNN was that region proposals depended on features of the image that were already calculated with the forward pass of the CNN (first step of classification). So why not reuse those same CNN results for region proposals instead of running a separate selective search algorithm?\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/fasterrcnn.png?raw=1\" alt=\"\" style=\"width: 400px;\"/> \n",
        "(Source:  https://arxiv.org/abs/1506.01497)\n",
        "</center>\n",
        "\n",
        "Here are the inputs and outputs of their model:\n",
        "\n",
        "> Inputs: Images (Notice how region proposals are not needed).\n",
        "\n",
        "> Outputs: Classifications and bounding box coordinates of objects in the images.\n",
        "\n",
        "### Mask R-CNN\n",
        "\n",
        "So far, we’ve seen how we’ve been able to use CNN features in many interesting ways to effectively locate different objects in an image with bounding boxes.\n",
        "\n",
        "Can we extend such techniques to go one step further and locate exact pixels of each object instead of just bounding boxes? This problem, known as image segmentation, is what Kaiming He and a team of researchers, including Girshick, explored at Facebook AI using an architecture known as Mask R-CNN.\n",
        "\n",
        "Given that Faster R-CNN works so well for object detection, could we extend it to also carry out pixel level segmentation? \n",
        "\n",
        "Mask R-CNN does this by adding a branch to Faster R-CNN that outputs a binary mask that says whether or not a given pixel is part of an object. The branch (in white in the above image), as before, is just a Fully Convolutional Network on top of a CNN based feature map. Here are its inputs and outputs:\n",
        "\n",
        "> Inputs: CNN Feature Map.\n",
        "> Outputs: Matrix with 1s on all locations where the pixel belongs to the object and 0s elsewhere (this is known as a binary mask).\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/pixelrcnn.png?raw=1\" alt=\"\" style=\"width: 700px;\"/> \n",
        "(Source:  https://arxiv.org/abs/1703.06870)\n",
        "</center>"
      ]
    },
    {
      "metadata": {
        "id": "4QlgQgrl6iav",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 1D-Conv for text classification\n",
        "\n",
        "**IMDB Movie reviews sentiment classification**: Dataset of 25,000 movies reviews from IMDB, labeled by sentiment (positive/negative). Reviews have been preprocessed, and each review is encoded as a sequence of word indexes (integers). For convenience, words are indexed by overall frequency in the dataset, so that for instance the integer \"3\" encodes the 3rd most frequent word in the data. This allows for quick filtering operations such as: \"only consider the top 10,000 most common words, but eliminate the top 20 most common words\".\n",
        "\n",
        "The seminal research paper on this subject was published by Yoon Kim on 2014. In this paper Yoon Kim has laid the foundations for how to model and process text by convolutional neural networks for the purpose of sentiment analysis. He has shown that by simple one-dimentional convolutional networks, one can develops very simple neural networks that reach 90% accuracy very quickly.\n",
        "\n",
        "Here is the text of an example review from our dataset:\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/review1.png?raw=1\" alt=\"\" style=\"width: 600px;\"/> \n",
        "<center>"
      ]
    },
    {
      "metadata": {
        "id": "pp2zq6BWCoVi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "a1cc3933-5049-4c7f-d0e2-a6e6bc959a52"
      },
      "cell_type": "code",
      "source": [
        "!pip install numpy==1.16.2"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: numpy==1.16.2 in /usr/local/lib/python3.6/dist-packages (1.16.2)\n",
            "<module 'numpy.version' from '/usr/local/lib/python3.6/dist-packages/numpy/version.py'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "6sgz-wbhCuF2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7151d57f-31c9-4484-a20e-85f3b83527a5"
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "print(np.__version__)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.16.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bZolXdO46iaw",
        "colab_type": "code",
        "outputId": "4584bda2-8dd1-44c0-a561-c7d0b66c5226",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 575
        }
      },
      "cell_type": "code",
      "source": [
        "'''\n",
        "This example demonstrates the use of Convolution1D for text classification.\n",
        "'''\n",
        "\n",
        "from __future__ import print_function\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "np.random.seed(1337)  # for reproducibility\n",
        "\n",
        "from tensorflow.keras.preprocessing import sequence\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten\n",
        "from tensorflow.keras.layers import Embedding\n",
        "from tensorflow.keras.layers import Conv1D, MaxPooling1D\n",
        "from tensorflow.keras.datasets import imdb\n",
        "\n",
        "\n",
        "# set parameters:\n",
        "max_features = 5000\n",
        "maxlen = 100\n",
        "batch_size = 32\n",
        "embedding_dims = 100\n",
        "nb_filter = 250\n",
        "filter_length = 3\n",
        "hidden_dims = 250\n",
        "nb_epoch = 10\n",
        "\n",
        "print('Loading data...')\n",
        "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=max_features)\n",
        "print(len(X_train), ' train sequences \\n')\n",
        "print(len(X_test), ' test sequences \\n')\n",
        "\n",
        "print('Pad sequences (samples x time)')\n",
        "X_train = sequence.pad_sequences(X_train, maxlen=maxlen)\n",
        "X_test = sequence.pad_sequences(X_test, maxlen=maxlen)\n",
        "print('X_train shape:', X_train.shape)\n",
        "print('X_test shape:', X_test.shape)\n",
        "\n",
        "print('Build model...')\n",
        "model = Sequential()\n",
        "\n",
        "# we start off with an efficient embedding layer which maps\n",
        "# our vocab indices into embedding_dims dimensions\n",
        "model.add(Embedding(max_features, embedding_dims, input_length=maxlen))\n",
        "model.add(Dropout(0.25))\n",
        "\n",
        "# we add a Convolution1D, which will learn nb_filter\n",
        "# word group filters of size filter_length:\n",
        "model.add(Conv1D(padding=\"valid\", \n",
        "                 kernel_size=3, \n",
        "                 filters=250, \n",
        "                 strides=1, \n",
        "                 activation=\"relu\"))\n",
        "# we use standard max pooling (halving the output of the previous layer):\n",
        "model.add(MaxPooling1D(pool_size=2))\n",
        "\n",
        "model.add(Conv1D(padding=\"valid\", \n",
        "                 kernel_size=3, \n",
        "                 filters=250, \n",
        "                 strides=1, \n",
        "                 activation=\"relu\"))\n",
        "model.add(MaxPooling1D(pool_size=2))\n",
        "\n",
        "\n",
        "# We flatten the output of the conv layer,\n",
        "# so that we can add a vanilla dense layer:\n",
        "model.add(Flatten())\n",
        "\n",
        "# We add a vanilla hidden layer:\n",
        "model.add(Dense(hidden_dims))\n",
        "model.add(Dropout(0.25))\n",
        "model.add(Activation('relu'))\n",
        "\n",
        "# We project onto a single unit output layer, and squash it with a sigmoid:\n",
        "model.add(Dense(1))\n",
        "model.add(Activation('sigmoid'))\n",
        "\n",
        "model.compile(loss='binary_crossentropy',\n",
        "              optimizer='rmsprop',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(X_train, y_train,\n",
        "          batch_size=batch_size,\n",
        "          epochs=nb_epoch,\n",
        "          validation_data=(X_test, y_test))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading data...\n",
            "25000  train sequences \n",
            "\n",
            "25000  test sequences \n",
            "\n",
            "Pad sequences (samples x time)\n",
            "X_train shape: (25000, 100)\n",
            "X_test shape: (25000, 100)\n",
            "Build model...\n",
            "Train on 25000 samples, validate on 25000 samples\n",
            "Epoch 1/10\n",
            "25000/25000 [==============================] - 10s 384us/sample - loss: 0.4533 - acc: 0.7702 - val_loss: 0.3769 - val_acc: 0.8333\n",
            "Epoch 2/10\n",
            "25000/25000 [==============================] - 6s 222us/sample - loss: 0.3199 - acc: 0.8620 - val_loss: 0.3270 - val_acc: 0.8564\n",
            "Epoch 3/10\n",
            "25000/25000 [==============================] - 6s 223us/sample - loss: 0.2845 - acc: 0.8836 - val_loss: 0.3240 - val_acc: 0.8582\n",
            "Epoch 4/10\n",
            "25000/25000 [==============================] - 6s 236us/sample - loss: 0.2559 - acc: 0.8992 - val_loss: 0.3555 - val_acc: 0.8459\n",
            "Epoch 5/10\n",
            "25000/25000 [==============================] - 6s 243us/sample - loss: 0.2326 - acc: 0.9072 - val_loss: 0.4883 - val_acc: 0.8101\n",
            "Epoch 6/10\n",
            "25000/25000 [==============================] - 6s 223us/sample - loss: 0.2097 - acc: 0.9195 - val_loss: 0.5515 - val_acc: 0.7976\n",
            "Epoch 7/10\n",
            "25000/25000 [==============================] - 6s 224us/sample - loss: 0.1858 - acc: 0.9300 - val_loss: 0.4102 - val_acc: 0.8372\n",
            "Epoch 8/10\n",
            "25000/25000 [==============================] - 6s 221us/sample - loss: 0.1644 - acc: 0.9378 - val_loss: 0.3814 - val_acc: 0.8463\n",
            "Epoch 9/10\n",
            "25000/25000 [==============================] - 6s 223us/sample - loss: 0.1430 - acc: 0.9480 - val_loss: 0.5229 - val_acc: 0.8366\n",
            "Epoch 10/10\n",
            "25000/25000 [==============================] - 6s 220us/sample - loss: 0.1291 - acc: 0.9537 - val_loss: 0.4734 - val_acc: 0.8504\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f1cd7eacc18>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    }
  ]
}