MobileNet_Aerial_Image_Recognition.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.6"
    },
    "colab": {
      "name": "MobileNet-Aerial Image Recognition.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "hr2okOEwAbUf",
        "HzsT9kWdAbUk",
        "IQRLOJyPSVkc"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/xiaoya27/Aerial-Image-Recognition/blob/main/MobileNet_Aerial_Image_Recognition.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ciutS3R-AbUT"
      },
      "source": [
        "# Aerial imagery Project"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "__AM9JhGAbUe"
      },
      "source": [
        "<div>\n",
        "    <h2>Introduction </h2>\n",
        "     <br>\n",
        "Aerial imagery has been a primary source of geographic data for quite a long time. With technology progress, aerial imagery became really practical for remote sensing : the science of obtaining information about an object, area or phenomenon.\n",
        "Nowadays, there are many uses of image recognition spanning from robotics/drone vision to autonomous driving vehicules or face detection.\n",
        "<br>\n",
        "In this challenge, we will use pre-processed data, coming from landscape images. The goal is to learn to differentiate common and uncommon landscapes such as a beach, a lake or a meadow.\n",
        "    Data comes from part of the data set (NWPU-RESISC45) originally used in <a href=\"https://arxiv.org/pdf/1703.00121.pdf?fbclid=IwAR16qo-EX_Z05ZpxvWG8F-oBU0SlnY-3BPCWBVVOGPyJcVy7BBqCKjnsvJo\">Remote Sensing Image Scene Classification</a>. This data set contains 45 categories while we only kept 13 out of them.\n",
        "\n",
        "References and credits: \n",
        "Yuliya Tarabalka, Guillaume Charpiat, Nicolas Girard for the data sets presentation.<br>\n",
        "Gong Cheng, Junwei Han, and Xiaoqiang Lu, for the original article on the chosen data set.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "44Fc3NVlD3ia"
      },
      "source": [
        "scene_class = ['beach', 'chaparral','cloud','desert','forest','island','lake','meadow','mountain','river','sea','snowberg','wetland']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gBDAs5DhD35S"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bKnScagFTeIT",
        "outputId": "b9fbaa68-b1f1-40f1-82e6-73eaa04b0989"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hr2okOEwAbUf"
      },
      "source": [
        "### Step1 Install Requirements \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M3Q20uZlij2Y"
      },
      "source": [
        "!pip install neptune-client==0.4.130\r\n",
        "\r\n",
        "import neptune\r\n",
        "from google.colab import output\r\n",
        "output.clear() "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8MZmGYXtAbUf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ceaa9ce5-e540-4d9e-9dcb-4576d8b160f5"
      },
      "source": [
        "%cd /content/drive/MyDrive/workingfolder/2021/starting_kit/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/MyDrive/workingfolder/2021/starting_kit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s2CilwrhiytB",
        "outputId": "45f8a4af-3754-408d-a1a3-4e941a9d2b3c"
      },
      "source": [
        "neptune.init(\r\n",
        "    api_token=\"xx\",\r\n",
        "    project_qualified_name=\"xiaoya27/AerialImageClassification\"\r\n",
        ")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Project(xiaoya27/AerialImageClassification)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lvg3FbP1AbUg"
      },
      "source": [
        "import numpy as np\n",
        "import random\n",
        "import re\n",
        "from matplotlib import pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WME90f50AbUg"
      },
      "source": [
        "model_dir = \"sample_code_submission\"\n",
        "result_dir = 'sample_result_submission/' \n",
        "problem_dir = 'ingestion_program/'  \n",
        "score_dir = 'scoring_program/'\n",
        "from sys import path; path.append(model_dir); path.append(problem_dir); path.append(score_dir)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_87P-1wkq3o7"
      },
      "source": [
        "def data_loading():\r\n",
        "  #data_dir = 'sample_data'\r\n",
        "  data_dir = '../public_data' # download \"public_data\" from the challenge website\r\n",
        "  data_name = 'Areal'\r\n",
        "  from data_manager import DataManager\r\n",
        "  D = DataManager(data_name, data_dir, replace_missing=False, verbose=True)\r\n",
        "  print(D)\r\n",
        "  X_train = D.data['X_train']\r\n",
        "  Y_train = D.data['Y_train']\r\n",
        "  X_valid = D.data['X_valid']\r\n",
        "  X_test = D.data['X_test']\r\n",
        "  with open('data.npy', 'wb') as f:\r\n",
        "      np.save(f, X_train)\r\n",
        "      np.save(f, Y_train)\r\n",
        "      np.save(f, X_valid )\r\n",
        "      np.save(f, X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GOrt6PerMwdB"
      },
      "source": [
        "# run it once\r\n",
        "#data_loading()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qiGqpMpRmQjK",
        "outputId": "a1da2a96-4cc7-4e55-d81d-59ff87304cf4"
      },
      "source": [
        "with open('data.npy', 'rb') as f:\r\n",
        "    X_train = np.load(f)\r\n",
        "    Y_train = np.load(f)\r\n",
        "    X_valid = np.load(f)\r\n",
        "    X_test = np.load(f)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5200, 1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zrTM3z-levXt"
      },
      "source": [
        "val_percent = 0.05\r\n",
        "train_size=int((1-val_percent)*len(X_train))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fTZ6ui2HORPd"
      },
      "source": [
        "#data_dir = 'sample_data'\r\n",
        "data_dir = '../public_data' # download \"public_data\" from the challenge website\r\n",
        "data_name = 'Areal'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QvReAXpWh4k7"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GBpyWbCXh4-H"
      },
      "source": [
        "\r\n",
        "params={}\r\n",
        "params['batch_size']=16\r\n",
        "params['val_percent']=val_percent\r\n",
        "\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HzsT9kWdAbUk"
      },
      "source": [
        "### Step 2 : Building training dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4rL9sJGk_ejq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1aaa9916-291c-47a1-d5be-1a87a58322c2"
      },
      "source": [
        "#X_train.reshape((5200,128,128,3))\r\n",
        "\r\n",
        "%tensorflow_version 2.x\r\n",
        "import tensorflow as tf\r\n",
        "print(\"Tensorflow version \" + tf.__version__)\r\n",
        "\r\n",
        "from tensorflow.keras.applications import MobileNetV2 as MobileNet\r\n",
        "\r\n",
        "feature_extractor_name='MobileNet'"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tensorflow version 2.4.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UZdYYigqmMH4"
      },
      "source": [
        "# check AUTOTUNE in https://www.tensorflow.org/guide/data_performance"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DUNVP_2-jE-Z"
      },
      "source": [
        "#View images from the dataset\r\n",
        "\r\n",
        "def view_image(ds):\r\n",
        "    image, label = next(iter(ds)) # extract 1 batch from the dataset\r\n",
        "    image = image.numpy()\r\n",
        "    label = label.numpy()\r\n",
        "\r\n",
        "    fig = plt.figure(figsize=(22,11))\r\n",
        "    for i in range(8):\r\n",
        "        ax = fig.add_subplot(2, 4, i+1, xticks=[], yticks=[])\r\n",
        "        ax.imshow(image[i])\r\n",
        "        ax.set_title(f\"Label: {label[i]}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DedZ8PbFk5pt"
      },
      "source": [
        "#view_image(dataset)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B99Qz-hFSUy5"
      },
      "source": [
        "def process_image(image, label):\r\n",
        "    # cast and normalize image\r\n",
        "    image = tf.reshape(image, (128,128,3))\r\n",
        "    image = tf.image.resize(image, [224, 224],method='nearest')\r\n",
        "    image = tf.image.convert_image_dtype(image, tf.float32)\r\n",
        "    image = tf.image.random_flip_left_right(image)\r\n",
        "    image = tf.keras.applications.mobilenet.preprocess_input(image)\r\n",
        "    return image, label\r\n",
        "\r\n",
        "\r\n",
        "dataset = tf.data.Dataset.from_tensor_slices((X_train, Y_train)).shuffle(buffer_size=1024)\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BKtSPImiegaR"
      },
      "source": [
        "train_dataset = dataset.take(train_size)\r\n",
        "val_dataset = dataset.skip(train_size)\r\n",
        "\r\n",
        "train_dataset = dataset.map(process_image).batch(params['batch_size']).cache().prefetch(tf.data.experimental.AUTOTUNE)\r\n",
        "\r\n",
        "val_dataset = dataset.map(process_image).batch(params['batch_size']).cache().prefetch(tf.data.experimental.AUTOTUNE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2PbPxpOdi5tS",
        "outputId": "d8a4defe-17b8-480d-f3d8-0f31ba8ea6b0"
      },
      "source": [
        "from tensorflow.keras.callbacks import Callback\r\n",
        "class NeptuneLogger(Callback):\r\n",
        "\r\n",
        "    def on_batch_end(self, batch, logs={}):\r\n",
        "        for log_name, log_value in logs.items():\r\n",
        "            neptune.log_metric(f'batch_{log_name}', log_value)\r\n",
        "\r\n",
        "    def on_epoch_end(self, epoch, logs={}):\r\n",
        "        for log_name, log_value in logs.items():\r\n",
        "            neptune.log_metric(f'epoch_{log_name}', log_value)\r\n",
        "\r\n",
        "neptune.create_experiment(name='Aerial-image',\r\n",
        "                          params=params,\r\n",
        "                          tags=['Tensorflow',feature_extractor_name],\r\n",
        "                          )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "https://ui.neptune.ai/xiaoya27/AerialImageClassification/e/AER-5\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Experiment(AER-5)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IQRLOJyPSVkc"
      },
      "source": [
        "### Step 3 . Building&training model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VxobL70wSZrc"
      },
      "source": [
        "feature_extractor = MobileNet(input_shape=(224,224,3),\r\n",
        "                                          include_top=False,\r\n",
        "                                          weights='imagenet')\r\n",
        "# what is input shape of backend model\r\n",
        "from tensorflow.keras import layers\r\n",
        "\r\n",
        "data_augmentation = tf.keras.Sequential([\r\n",
        "                                          layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\"),\r\n",
        "                                          layers.experimental.preprocessing.RandomRotation(0.2),])\r\n",
        "input = tf.keras.Input(shape=(224,224,3))\r\n",
        "x = data_augmentation(input)\r\n",
        "x = feature_extractor(input,training=True)\r\n",
        "\r\n",
        "x = tf.keras.layers.GlobalAveragePooling2D()(x)\r\n",
        "x = tf.keras.layers.Flatten()(x)\r\n",
        "x = tf.keras.layers.Dense(1024, activation=\"relu\")(x)\r\n",
        "x = tf.keras.layers.Dense(512, activation=\"relu\")(x)\r\n",
        "x = tf.keras.layers.Dense(13, activation=\"softmax\", name=\"classification\")(x)                                  \r\n",
        "\r\n",
        "checkpoint_path = \"training_1/cp.ckpt\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gPf8DmjUSZvA",
        "outputId": "9b8c5b3b-5cc4-49e7-a1dd-e2f81e35ca93"
      },
      "source": [
        "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\r\n",
        "                                                 save_weights_only=True,\r\n",
        "                                                 verbose=1)\r\n",
        "\r\n",
        "with tf.device('/device:GPU:0'):\r\n",
        "  model = tf.keras.Model(inputs=input, outputs=x)\r\n",
        "\r\n",
        "  model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics='accuracy', run_eagerly=True)\r\n",
        "  model.fit(train_dataset,\r\n",
        "            epochs=15,\r\n",
        "            validation_data=val_dataset,\r\n",
        "            callbacks=[cp_callback,NeptuneLogger()]\r\n",
        "            )\r\n",
        "# TBC https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/transfer_learning.ipynb#scrollTo=DgzQX6Veb2WT"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/15\n",
            "325/325 [==============================] - 83s 249ms/step - loss: 1.5463 - accuracy: 0.5367 - val_loss: 0.7686 - val_accuracy: 0.7673\n",
            "\n",
            "Epoch 00001: saving model to training_1/cp.ckpt\n",
            "Epoch 2/15\n",
            "325/325 [==============================] - 67s 205ms/step - loss: 0.8798 - accuracy: 0.7315 - val_loss: 0.6921 - val_accuracy: 0.7925\n",
            "\n",
            "Epoch 00002: saving model to training_1/cp.ckpt\n",
            "Epoch 3/15\n",
            "325/325 [==============================] - 67s 206ms/step - loss: 0.6606 - accuracy: 0.8012 - val_loss: 0.5738 - val_accuracy: 0.8246\n",
            "\n",
            "Epoch 00003: saving model to training_1/cp.ckpt\n",
            "Epoch 4/15\n",
            "325/325 [==============================] - 66s 204ms/step - loss: 0.5347 - accuracy: 0.8357 - val_loss: 0.5015 - val_accuracy: 0.8494\n",
            "\n",
            "Epoch 00004: saving model to training_1/cp.ckpt\n",
            "Epoch 5/15\n",
            "325/325 [==============================] - 66s 205ms/step - loss: 0.4734 - accuracy: 0.8475 - val_loss: 0.5337 - val_accuracy: 0.8490\n",
            "\n",
            "Epoch 00005: saving model to training_1/cp.ckpt\n",
            "Epoch 6/15\n",
            "325/325 [==============================] - 66s 204ms/step - loss: 0.3838 - accuracy: 0.8849 - val_loss: 0.4216 - val_accuracy: 0.8733\n",
            "\n",
            "Epoch 00006: saving model to training_1/cp.ckpt\n",
            "Epoch 7/15\n",
            "325/325 [==============================] - 68s 209ms/step - loss: 0.3778 - accuracy: 0.8873 - val_loss: 0.4794 - val_accuracy: 0.8754\n",
            "\n",
            "Epoch 00007: saving model to training_1/cp.ckpt\n",
            "Epoch 8/15\n",
            "325/325 [==============================] - 68s 209ms/step - loss: 0.2819 - accuracy: 0.9130 - val_loss: 0.4893 - val_accuracy: 0.8602\n",
            "\n",
            "Epoch 00008: saving model to training_1/cp.ckpt\n",
            "Epoch 9/15\n",
            "325/325 [==============================] - 66s 205ms/step - loss: 0.2545 - accuracy: 0.9243 - val_loss: 0.5082 - val_accuracy: 0.8596\n",
            "\n",
            "Epoch 00009: saving model to training_1/cp.ckpt\n",
            "Epoch 10/15\n",
            "325/325 [==============================] - 66s 203ms/step - loss: 0.2728 - accuracy: 0.9188 - val_loss: 0.4284 - val_accuracy: 0.8848\n",
            "\n",
            "Epoch 00010: saving model to training_1/cp.ckpt\n",
            "Epoch 11/15\n",
            "325/325 [==============================] - 67s 205ms/step - loss: 0.2174 - accuracy: 0.9433 - val_loss: 0.4803 - val_accuracy: 0.8787\n",
            "\n",
            "Epoch 00011: saving model to training_1/cp.ckpt\n",
            "Epoch 12/15\n",
            "325/325 [==============================] - 66s 204ms/step - loss: 0.2434 - accuracy: 0.9213 - val_loss: 0.3881 - val_accuracy: 0.9004\n",
            "\n",
            "Epoch 00012: saving model to training_1/cp.ckpt\n",
            "Epoch 13/15\n",
            "325/325 [==============================] - 66s 203ms/step - loss: 0.1631 - accuracy: 0.9470 - val_loss: 0.3474 - val_accuracy: 0.9056\n",
            "\n",
            "Epoch 00013: saving model to training_1/cp.ckpt\n",
            "Epoch 14/15\n",
            "325/325 [==============================] - 66s 204ms/step - loss: 0.1881 - accuracy: 0.9412 - val_loss: 0.4717 - val_accuracy: 0.8888\n",
            "\n",
            "Epoch 00014: saving model to training_1/cp.ckpt\n",
            "Epoch 15/15\n",
            "325/325 [==============================] - 67s 206ms/step - loss: 0.2250 - accuracy: 0.9345 - val_loss: 0.3083 - val_accuracy: 0.9140\n",
            "\n",
            "Epoch 00015: saving model to training_1/cp.ckpt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T2ZnUowZu0PB"
      },
      "source": [
        "neptune.stop()"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}