introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### We will be using opencv to handle everything from loading network configuration and weights, to display the results."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2 as cv\n",
    "from copy import deepcopy\n",
    "import numpy as np\n",
    "import src.utility as ut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### File paths\n",
    "\n",
    "During the workshop, we've simplyfied it such that you only have to provide the name of the file you want to work with. \n",
    "\n",
    "* Any **weight** files you want to use should be placed in the **data/weights** folder in this project\n",
    "* Any **configuration** files you want to use should be places in the **data/cfg** folder in this project\n",
    "* Any **images** you want to use should be places in the **data/images** folder in this project"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "weigths_path = \"data/weights/\"\n",
    "cfg_path = \"data/cfg/\"\n",
    "class_path = \"data/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Download necessary files\n",
    "\n",
    "If the files are already downloaded and placed correctly, no downloads will start.\n",
    "\n",
    "The download could take a few minutes..."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "ut.download(\"https://raw.githubusercontent.com/pjreddie/darknet/master/cfg/yolov3.cfg\", cfg_path, \"yolov3.cfg\")\n",
    "ut.download(\"https://pjreddie.com/media/files/yolov3.weights\", weigths_path, \"yolov3.weights\")\n",
    "ut.download(\"https://raw.githubusercontent.com/pjreddie/darknet/master/data/coco.names\", class_path, \"coco.names\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Loading weights and configuration files, using OpenCV\n",
    "\n",
    "Using provided variables, the function ```load``` will create a network for us."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load(weights, cfg):\n",
    "    weights = weigths_path + weights\n",
    "    cfg = cfg_path + cfg\n",
    "    return cv.dnn.readNet(weights, cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Extract output layers\n",
    "\n",
    "The last preperation we need to do for the network, is to extract the output layers. These will provide us with the detections which we will use to draw boxes on the image."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_output_layers(net):\n",
    "    names = net.getLayerNames()\n",
    "    return [names[layer[0] - 1] for layer in net.getUnconnectedOutLayers()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Loading classes\n",
    "\n",
    "Next, we want to load the classes we will use to label the output. These could techically be any list of 80 strings. It has to be atleast 80, as we will index this list from the 80 possible classes this network was trained on originally. "
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_classes(classes):\n",
    "    with(open(class_path + classes, \"r\")) as f:\n",
    "        return [line.strip() for line in f.readlines()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Creating a blob from image\n",
    "\n",
    "This method will expect a preloaded image (utility functions for this will be shown later) which will be converted to a blob. This is the expected format for YOLO.\n",
    "\n",
    "A few things to note about this:\n",
    "* **size** defines the dimensions of the final input we send to our network.\n",
    "* **scalefactor** lets us scale each pixel in the image. As we have 255 possibilities per channel (RGB), we will scale the pixels by 1/255 which is part of the exected input for YOLO.\n",
    "* **swapRB** simply means that we will swap R and B channels, as OpenCV for some reason works with BGR instead of RGB.\n",
    "\n",
    "**Important note about size:**\n",
    "\n",
    "If you decide to change this parameter (which you should try, by the way) later in the workshop, please note that YOLO downsamples the input by 32. This means that the size of the input must be some multiple of 32 in both the width and height"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "def image_to_blob(image, size=(416, 416)):\n",
    "    return cv.dnn.blobFromImage(image, scalefactor=1/255, size=size, swapRB=True, crop=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Forwarding the image\n",
    "\n",
    "This is the main part. This will forward our blob through the network and return the output."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward(net, blob, output_layers):\n",
    "    net.setInput(blob)\n",
    "    return net.forward(output_layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Drawing box detections\n",
    "\n",
    "This is a utility function we use to draw boxes on a given image. We will use this during the postprocess."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "def draw_box(image, label, box, box_color, text_color, font_scale, font, thickness):\n",
    "    x,y,w,h = box\n",
    "    tw, th = cv.getTextSize(text=label, fontFace=font, fontScale=font_scale, thickness=thickness)[0]\n",
    "    cv.rectangle(image, (x, y), (x + w, y + h), box_color, thickness=thickness)\n",
    "    cv.rectangle(image, (x, y), (x + tw + 10, y - th - 10), color=box_color, thickness=cv.FILLED)\n",
    "    cv.putText(image, label, (x + 5, y - 5), fontFace=font, fontScale=font_scale, color=text_color, thickness=thickness)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Process the output\n",
    "\n",
    "Given an image, network outputs and the classes (labels) we decide to use - this function will handle things like overlapping boxes and sorting out detections under a certain threshold. \n",
    "\n",
    "You might be confused by the lines such as ```center_x = detection[0] * width```. Why would we have to multiply the output by the width of the image? This is because YOLO will use numbers between 0 and 1, which is scaled down by the width and height of the input, as its output. We simply multiply to translate back to values we can work with."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "def postprocess(image, outputs, classes, threshold=0.8, nms_threshold=0.7, box_color=(0, 0, 0), text_color=(255, 255, 255), font_scale=0.5, font=cv.FONT_HERSHEY_SIMPLEX, thickness=2):\n",
    "    height, width, _ = image.shape\n",
    "\n",
    "    image = deepcopy(image)\n",
    "\n",
    "    class_ids = []\n",
    "    confidences = []\n",
    "    boxes = []\n",
    "\n",
    "    for o in outputs:\n",
    "        for detection in o:\n",
    "            center_x = detection[0] * width\n",
    "            center_y = detection[1] * height\n",
    "            w = detection[2] * width\n",
    "            h = detection[3] * height\n",
    "            x = center_x - w/2\n",
    "            y = center_y - h/2\n",
    "\n",
    "            scores = detection[5:]\n",
    "            class_id = np.argmax(scores)\n",
    "            confidence = scores[class_id]\n",
    "\n",
    "            boxes.append([int(x), int(y), int(w), int(h)])\n",
    "            confidences.append(float(confidence))\n",
    "            class_ids.append(class_id)\n",
    "\n",
    "    indices = cv.dnn.NMSBoxes(boxes, confidences, threshold, nms_threshold)\n",
    "\n",
    "    for i in indices:\n",
    "        i = i[0]\n",
    "        label = str(classes[class_ids[i]])\n",
    "        draw_box(image, label, boxes[i], box_color, text_color, font_scale, font, thickness)\n",
    "        \n",
    "    return image, class_ids, confidences, boxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Example time!\n",
    "\n",
    "Using the functions above, this next block will load a network, load an image and forward it through the network. Then we will utilize the postprocess function to create an image with the detections displayed on it. Finally we show this image."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = \"yolov3.weights\"\n",
    "cfg = \"yolov3.cfg\"\n",
    "classes = \"coco.names\"\n",
    "image = \"demo.jpg\"\n",
    "\n",
    "net = load(weights, cfg)\n",
    "\n",
    "classes = prepare_classes(classes)\n",
    "\n",
    "output_layers = extract_output_layers(net)\n",
    "\n",
    "img = ut.load_image(image)\n",
    "\n",
    "blob = image_to_blob(img)\n",
    "\n",
    "outputs = forward(net, blob, output_layers)\n",
    "\n",
    "processed_image,_,_,_ = postprocess(img, outputs, classes, 0.2)\n",
    "\n",
    "ut.show_image(processed_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "outputs": [],
   "source": [
    "### What's next?\n",
    "\n",
    "We have prepared two more notebooks. \n",
    "* **image_workshop** here you will get access to the same you did here, but in a more condensed package. This is where you should go to experiment with different parameters.\n",
    "* **video_workshop** this workshop revolves around performing object detection on videos from youtube. "
   ]
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": 3
 },
 "nbformat": 4,
 "nbformat_minor": 2
}