MLA-CV-DAY3-Final-Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning Accelerator Computer Vision - Final Project - Day 3\n",
    "\n",
    "In this project, you will train Convolutional Neural Networks to correctly classify images of luggages. The competition is hosted here: https://mlu.corp.amazon.com/contests/redirect/50\n",
    "\n",
    "We have a total of 5 classes and they are:\n",
    "* Class 0: Inconclusive\n",
    "* Class 1: Two wheels\n",
    "* Class 2: Four wheels\n",
    "* Class 3: Not a luggage\n",
    "* Class 4: Zero wheels\n",
    "\n",
    "__You can try to improve your previous model or try VGGNet and ResNet. MLA-CV-DAY2-AlexNet.ipynb and MLA-CV-DAY3-ResNet.ipynb will be helpful__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import all the libraries. Before that, make sure you have installed the required version our libraries as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -q torch==1.8.0\n",
    "! pip install -q torchvision==0.9.0\n",
    "! pip install -q d2l==0.16.01\n",
    "! pip install -q numpy==1.19.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from glob import glob\n",
    "import os\n",
    "from shutil import copyfile\n",
    "from torch.utils.data import Dataset\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "from numpy.random import permutation\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from torchvision import transforms\n",
    "from torchvision.datasets import ImageFolder\n",
    "from torchvision.models import resnet18,resnet34,densenet121\n",
    "from torchvision.models.inception import inception_v3\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.autograd import Variable\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import pickle\n",
    "%matplotlib inline\n",
    "import multiprocessing\n",
    "cpu_count = multiprocessing.cpu_count()\n",
    "is_cuda = torch.cuda.is_available()\n",
    "is_cuda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Reading the dataset and creating loaders:\n",
    "\n",
    "Your final project dataset is now under the __\"data/final_project_dataset\"__ folder. Over there you will see the __training__, __validation__ and __test__ folders. Let's start creating the data transforms and loaders below. In this project, images come in different sizes and we will resize them to 224x224 and normalize pixel values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "path = '../../data/final_project_dataset'\n",
    "train_path = os.path.join(path, 'train')\n",
    "val_path = os.path.join(path, 'validate')\n",
    "test_path = os.path.join(path, 'test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_images_in_folder(path):\n",
    "    import os\n",
    "    ans = {}\n",
    "    for clas in os.listdir(path):\n",
    "        ans[clas] = len(os.listdir(os.path.join(path, clas)))\n",
    "    return ans "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dist = count_images_in_folder(train_path)\n",
    "print(\"Training distribution:\", train_dist)\n",
    "val_dist = count_images_in_folder(val_path)\n",
    "print(\"Validation distribution:\", val_dist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "simple_tranform = transforms.Compose([\n",
    "    transforms.Resize((224, 224)),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))\n",
    "])\n",
    "\n",
    "simple_tranform_incp = transforms.Compose([\n",
    "    transforms.Resize((299, 299)),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))\n",
    "])\n",
    "\n",
    "train_transform = transforms.Compose([\n",
    "    transforms.Resize((224, 224)),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.RandomResizedCrop(size=(224, 224)),\n",
    "    transforms.ColorJitter(brightness=.5,contrast=0.3, saturation=0.2, hue=.3),\n",
    "    transforms.RandomRotation(degrees=(0, 30)),\n",
    "    transforms.RandomAdjustSharpness(sharpness_factor=2),\n",
    "    transforms.RandomAutocontrast(),\n",
    "    transforms.RandomHorizontalFlip(p=0.5),\n",
    "    transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 512\n",
    "train_loader = DataLoader(\n",
    "    ImageFolder(train_path, transform=simple_tranform),\n",
    "    batch_size=batch_size, shuffle=False )\n",
    "\n",
    "val_loader = DataLoader(\n",
    "    ImageFolder(val_path, transform=simple_tranform),\n",
    "    batch_size=batch_size, shuffle=False)\n",
    "\n",
    "test_loader = DataLoader(\n",
    "    ImageFolder(test_path, transform=simple_tranform),\n",
    "    batch_size=batch_size, shuffle=False)\n",
    "\n",
    "train_loader_incp = DataLoader(\n",
    "    ImageFolder(train_path, transform=simple_tranform_incp),\n",
    "    batch_size=batch_size, shuffle=False )\n",
    "\n",
    "val_loader_incp = DataLoader(\n",
    "    ImageFolder(val_path, transform=simple_tranform_incp),\n",
    "    batch_size=batch_size, shuffle=False)\n",
    "\n",
    "test_loader_incp = DataLoader(\n",
    "    ImageFolder(test_path, transform=simple_tranform_incp),\n",
    "    batch_size=batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Training and Validation\n",
    "\n",
    "In this part, you will use a simple Convolutional Neural Network. You can start with a network that we discuss in the class today. You will use the __train_loader__ and __validation_loader__ from above. \n",
    "\n",
    "Some important notes:\n",
    "* There are 5 classes in this project, so adjust the network's last dense layer for that\n",
    "* Once the tranining runs without errors, experiment with batch_size (in the data loaders part) and learning rate in your code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeaturesDataset(Dataset):\n",
    "    \n",
    "    def __init__(self,featlst1,featlst2,featlst3,labellst):\n",
    "        self.featlst1 = featlst1\n",
    "        self.featlst2 = featlst2\n",
    "        self.featlst3 = featlst3\n",
    "        self.labellst = labellst\n",
    "        \n",
    "    def __getitem__(self,index):\n",
    "        return (self.featlst1[index],self.featlst2[index],self.featlst3[index],self.labellst[index])\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.labellst)\n",
    "    \n",
    "def fit(epoch,model,data_loader,phase='training',volatile=False):\n",
    "    if phase == 'training':\n",
    "        model.train()\n",
    "    if phase == 'validation':\n",
    "        model.eval()\n",
    "        volatile=True\n",
    "    running_loss = 0.0\n",
    "    running_correct = 0\n",
    "    for batch_idx , (data1,data2,data3,target) in enumerate(data_loader):\n",
    "        if is_cuda:\n",
    "            data1,data2,data3,target = data1.cuda(),data2.cuda(),data3.cuda(),target.cuda()\n",
    "        data1,data2,data3,target = Variable(data1,volatile),Variable(data2,volatile),Variable(data3,volatile),Variable(target)\n",
    "        if phase == 'training':\n",
    "            optimizer.zero_grad()\n",
    "        output = model(data1,data2,data3)\n",
    "        w = torch.tensor([0.9495, 0.5931, 0.6861, 0.9784, 0.7929]).cuda()\n",
    "        loss = F.cross_entropy(output,target,w)\n",
    "        \n",
    "        \n",
    "        \n",
    "        running_loss += F.cross_entropy(output,target,size_average=False).data.item()\n",
    "        preds = output.data.max(dim=1,keepdim=True)[1]\n",
    "        running_correct += preds.eq(target.data.view_as(preds)).cpu().sum()\n",
    "        if phase == 'training':\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "    \n",
    "    loss = running_loss/len(data_loader.dataset)\n",
    "    accuracy = 100. * running_correct/len(data_loader.dataset)\n",
    "    \n",
    "    print(f'{phase} loss is {loss:{5}.{2}} and {phase} accuracy is {running_correct}/{len(data_loader.dataset)}{accuracy:{10}.{4}}')\n",
    "    return loss,accuracy\n",
    "\n",
    "class LayerActivations():\n",
    "    features=[]\n",
    "    \n",
    "    def __init__(self,model):\n",
    "        self.features = []\n",
    "        self.hook = model.register_forward_hook(self.hook_fn)\n",
    "    \n",
    "    def hook_fn(self,module,input,output):\n",
    "        #out = F.avg_pool2d(output, kernel_size=8)\n",
    "        self.features.extend(output.view(output.size(0),-1).cpu().data)\n",
    "\n",
    "    \n",
    "    def remove(self):\n",
    "        \n",
    "        self.hook.remove()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create ResNet model\n",
    "my_resnet = resnet34(pretrained=True)\n",
    "\n",
    "if is_cuda:\n",
    "    my_resnet = my_resnet.cuda()\n",
    "\n",
    "my_resnet = nn.Sequential(*list(my_resnet.children())[:-1])\n",
    "\n",
    "for p in my_resnet.parameters():\n",
    "    p.requires_grad = False\n",
    "\n",
    "#Create inception model\n",
    "\n",
    "my_inception = inception_v3(pretrained=True)\n",
    "my_inception.aux_logits = False\n",
    "if is_cuda:\n",
    "    my_inception = my_inception.cuda()\n",
    "for p in my_inception.parameters():\n",
    "    p.requires_grad = False\n",
    "\n",
    "#Create densenet model\n",
    "\n",
    "my_densenet = densenet121(pretrained=True).features\n",
    "if is_cuda:\n",
    "    my_densenet = my_densenet.cuda()\n",
    "    \n",
    "for p in my_densenet.parameters():\n",
    "    p.requires_grad = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### For ResNet\n",
    "\n",
    "trn_labels = []\n",
    "trn_resnet_features = []\n",
    "for d,la in train_loader:\n",
    "    o = my_resnet(Variable(d.cuda()))\n",
    "    o = o.view(o.size(0),-1)\n",
    "    trn_labels.extend(la)\n",
    "    trn_resnet_features.extend(o.cpu().data)\n",
    "val_labels = []\n",
    "val_resnet_features = []\n",
    "for d,la in val_loader:\n",
    "    o = my_resnet(Variable(d.cuda()))\n",
    "    o = o.view(o.size(0),-1)\n",
    "    val_labels.extend(la)\n",
    "    val_resnet_features.extend(o.cpu().data)\n",
    "\n",
    "# ### For Inception\n",
    "\n",
    "trn_inception_features = LayerActivations(my_inception.Mixed_7c)\n",
    "for da,la in train_loader_incp:\n",
    "    _ = my_inception(Variable(da.cuda()))\n",
    "\n",
    "trn_inception_features.remove()\n",
    "\n",
    "val_inception_features = LayerActivations(my_inception.Mixed_7c)\n",
    "for da,la in val_loader_incp:\n",
    "    _ = my_inception(Variable(da.cuda()))\n",
    "\n",
    "val_inception_features.remove()\n",
    "\n",
    "### For Densenet\n",
    "\n",
    "\n",
    "trn_densenet_features = []\n",
    "for d,la in train_loader:\n",
    "    o = my_densenet(Variable(d.cuda()))\n",
    "    o = o.view(o.size(0),-1)\n",
    "    \n",
    "    trn_densenet_features.extend(o.cpu().data)\n",
    "    \n",
    "\n",
    "val_densenet_features = []\n",
    "for d,la in val_loader:\n",
    "    o = my_densenet(Variable(d.cuda()))\n",
    "    o = o.view(o.size(0),-1)\n",
    "    val_densenet_features.extend(o.cpu().data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_feat_dset = FeaturesDataset(trn_resnet_features,trn_inception_features.features,trn_densenet_features,trn_labels)\n",
    "val_feat_dset = FeaturesDataset(val_resnet_features,val_inception_features.features,val_densenet_features,val_labels)\n",
    "trn_feat_loader = DataLoader(trn_feat_dset,batch_size=512,shuffle=True)\n",
    "val_feat_loader = DataLoader(val_feat_dset,batch_size=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EnsembleModel(nn.Module):\n",
    "    \n",
    "    def __init__(self,out_size,training=True):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(512,512)\n",
    "        self.fc2 = nn.Linear(131072,512)\n",
    "        self.fc3 = nn.Linear(50176,512)\n",
    "        self.fc4 = nn.Linear(512,out_size)\n",
    "\n",
    "    def forward(self,inp1,inp2,inp3):\n",
    "        out1 = self.fc1(F.dropout(inp1,training=self.training))\n",
    "        out2 = self.fc2(F.dropout(inp2,training=self.training))\n",
    "        out3 = self.fc3(F.dropout(inp3,training=self.training))\n",
    "        out = out1 + out2 + out3\n",
    "        out = self.fc4(F.dropout(out,training=self.training, p=0.75))\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.seed()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "em = EnsembleModel(5)\n",
    "if is_cuda:\n",
    "    em = em.cuda()\n",
    "optimizer = optim.Adam(em.parameters(),lr=0.01)\n",
    "scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_losses , train_accuracy = [],[]\n",
    "val_losses , val_accuracy = [],[]\n",
    "for epoch in range(1,20):\n",
    "    epoch_loss, epoch_accuracy = fit(epoch,em,trn_feat_loader,phase='training')\n",
    "    val_epoch_loss , val_epoch_accuracy = fit(epoch,em,val_feat_loader,phase='validation')\n",
    "    train_losses.append(epoch_loss)\n",
    "    train_accuracy.append(epoch_accuracy)\n",
    "    val_losses.append(val_epoch_loss)\n",
    "    val_accuracy.append(val_epoch_accuracy)\n",
    "    scheduler.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Reading the test data and getting predictions\n",
    "\n",
    "After you trained your model, it is time to read our test set and make predictions with it. Then, we will write our predictions to a csv file that thas \"ID\" and \"label\" columns. This will be our final project submission file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "test_df = pd.read_csv(\"../../data/final_project_dataset/public_test_features.csv\")\n",
    "print(test_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "file_index = 0\n",
    "files = ImageFolder(test_path, transform=simple_tranform).samples\n",
    "test_df[\"label\"] = np.nan\n",
    "# test_df[\"conf\"] = np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### For ResNet\n",
    "\n",
    "test_resnet_features = []\n",
    "for d,la in test_loader:\n",
    "    o = my_resnet(Variable(d.cuda()))\n",
    "    o = o.view(o.size(0),-1)\n",
    "    test_resnet_features.extend(o.cpu().data)\n",
    "\n",
    "# ### For Inception\n",
    "\n",
    "\n",
    "test_inception_features = LayerActivations(my_inception.Mixed_7c)\n",
    "for da,la in test_loader_incp:\n",
    "    _ = my_inception(Variable(da.cuda()))\n",
    "\n",
    "test_inception_features.remove()\n",
    "\n",
    "### For Densenet\n",
    "\n",
    "test_densenet_features = []\n",
    "for d,la in test_loader:\n",
    "    o = my_densenet(Variable(d.cuda()))\n",
    "    o = o.view(o.size(0),-1)\n",
    "    test_densenet_features.extend(o.cpu().data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeaturesDatasetTest(Dataset):\n",
    "    \n",
    "    def __init__(self,featlst1,featlst2,featlst3):\n",
    "        self.featlst1 = featlst1\n",
    "        self.featlst2 = featlst2\n",
    "        self.featlst3 = featlst3\n",
    "        \n",
    "        \n",
    "    def __getitem__(self,index):\n",
    "        return (self.featlst1[index],self.featlst2[index],self.featlst3[index])\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.featlst1)\n",
    "\n",
    "\n",
    "\n",
    "test_feat_dset = FeaturesDatasetTest(test_resnet_features,test_inception_features.features,test_densenet_features)\n",
    "\n",
    "test_feat_loader = DataLoader(test_feat_dset,batch_size=512, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "   \n",
    "    em.eval()\n",
    "    volatile=True\n",
    "\n",
    "    for batch_idx , (data1,data2,data3) in enumerate(test_feat_loader):\n",
    "        if is_cuda:\n",
    "            data1,data2,data3 = data1.cuda(),data2.cuda(),data3.cuda()\n",
    "        data1,data2,data3 = Variable(data1,volatile),Variable(data2,volatile),Variable(data3,volatile)\n",
    "\n",
    "        output = em(data1,data2,data3)\n",
    "        preds = output.data.max(dim=1,keepdim=True)[1]\n",
    "        confs = output.data.max(dim=1,keepdim=True)[0]\n",
    "        \n",
    "        for pred, conf in zip(preds, confs):\n",
    "            test_df.loc[test_df[\"ASIN\"]==files[file_index][0].split(\"/\")[-1].split(\".jpg\")[0], \"label\"] = int(pred.data.item())\n",
    "#             test_df.loc[test_df[\"ASIN\"]==files[file_index][0].split(\"/\")[-1].split(\".jpg\")[0], \"conf\"] = conf.data.item()\n",
    "            file_index += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the test images and start creating our csv file. Here, we add a new column named \"label\". We write model's predictions to this column. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Submit your predictions\n",
    "\n",
    "This is the last step. The code block below will create a csv file: __final_project.csv__ with ID and label (your predictions) columns and then you will submit it to the leaderboard: https://mlu.corp.amazon.com/contests/redirect/50 \"ASIN\" column is not needed, so we drop it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "test_df.drop(columns=[\"ASIN\"], inplace=True)\n",
    "\n",
    "test_df.to_csv(\"final_project.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df.to_csv(\"final_project.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}