yaltai_V8x.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59cae60a-0c8a-4fe6-977d-02cd37885729",
   "metadata": {},
   "source": [
    "# Training a segmentation model for layout recognition with YALTAi and YOLOv8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44215395-2636-4fa3-9d49-14d4a0fa5fc7",
   "metadata": {},
   "source": [
    "This example has been used to produce an object classification model for images of books printed around the 16th - 18th centuries. The aim is to facilitate the process of image occlusion, while preserving the semantic value of the layout."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d46160c-fa81-4c9b-aec4-65ea69d9e82f",
   "metadata": {},
   "source": [
    "Resources :\n",
    "- [**YALTAi**](https://github.com/ponteineptique/yaltai) : CLI for kraken engine aptation with YOLOv8 API\n",
    "- [**Yolov8**](https://docs.ultralytics.com/) : Deep learning vision engin for object detection and image segmention. Provides by ultralytics API\n",
    "- [**SegmOnto**](https://segmonto.github.io/) : Controlled vocabulary for the description of printed and manuscript books to describe content and layout"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f202921e-5c3e-40b3-a726-1791d03833a6",
   "metadata": {},
   "source": [
    "Dataset :\n",
    "- Gallic(orpor)a : [link](https://github.com/Gallicorpora)\n",
    "- FoNDUE : [link](https://github.com/FoNDUE-HTR)\n",
    "- SETAF-Pierre-de-Vingle : [link](https://github.com/SETAFDH/HTR-SETAF-Pierre-de-Vingle)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "610426e6-ada1-4bf3-ae05-0c46f59a8c99",
   "metadata": {},
   "source": [
    "*The computations were performed at University of Geneva using Baobab HPC service.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfada4a2-c363-4286-a8f6-aa5c9df58655",
   "metadata": {},
   "source": [
    "## Producing classification vision model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a232f6-639e-47d5-8cbc-cc966afb6d7f",
   "metadata": {},
   "source": [
    "### Prepare dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd427b89-77dc-4752-b800-2974f6c592c3",
   "metadata": {},
   "source": [
    "You need to group your XML-ALTO data in a single folder so that you can transform it into the YOLO format used by the Ultralytics API. Check that your zone names are consistent, so as not to multiply the number of classes, thereby reducing model quality. Ideally, your dataset should be structured as follows: `dataset/[BookId]/[IdImg]/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8ca0565c-a128-463b-b7e4-26d871bf2e50",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = 'PATH/TO/DATA/'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b31fa6a0-2049-479f-ba64-bd07a18dae42",
   "metadata": {},
   "source": [
    "For virtual environnement dedicated to yaltai:\n",
    "`pip install yaltai`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "98da6e86-c368-4967-b8c5-c6bd5d0cf37d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash: line 1: yaltai: command not found\n"
     ]
    }
   ],
   "source": [
    "# convert alto data to yolo data\n",
    "\n",
    "!yaltai alto-to-yolo PATH/TO/ALTOorPAGE/*.xml my-dataset --shuffle .1 --segmonto region"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eb14b0d-33b4-4209-9fa7-75a8bb781c44",
   "metadata": {},
   "source": [
    "You'll find your new dataset, converted to YOLO format, in my-dataset or under the name you specified.\n",
    "\n",
    "It may be necessary to modify certain parameters if you move your dataset to exercise the model, for example on a calculation server. You need to change the values of `train` and `val` to indicate the new absolute path within the `config.yml`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "148c5cf6-bd46-4746-9b42-4f1f55d755e5",
   "metadata": {},
   "source": [
    "###  Training a model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dfc4abb-3419-4848-b1a4-4740b0dfe7ee",
   "metadata": {},
   "source": [
    "You can use the YALTAi library CLI to facilitate model training. In our case, we've chosen to use the ultralytics API directly, to go a step further in refining the model.\n",
    "\n",
    "If you wish to use this method, we recommend that you install ultralytics in a new environment to ensure that you have the latest updates.\n",
    "\n",
    "`pip install ultralytics`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea39f3db-5b80-4352-94a6-210c6479ea73",
   "metadata": {},
   "source": [
    "#### Dashboard"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f498e114-e438-4ff6-a084-fa1d594777a6",
   "metadata": {},
   "source": [
    "You can use the *Comet* dashboard to track your training progress. The implementation is native to the ultralytics API. All you need to do is install the library.\n",
    "Here, we indicate the various pieces of information when instantiating a class, although it is possible to import this information as an environment variable.\n",
    "For documentation, please refer to https://www.comet.com/docs/v2/\n",
    "\n",
    "`pip install comet-ml`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5d649ea-319e-45db-a5ec-422527d934cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import comet_ml\n",
    "\n",
    "comet_ml.init(api_key='API-KEY', project_name='NOM_PROJET', workspace='PSEUDO')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd7f2f8-6e7c-4557-b131-db8e95776d64",
   "metadata": {},
   "source": [
    "#### TRAINING"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7db49eda-b5a5-4bc4-94aa-a44c16347408",
   "metadata": {},
   "source": [
    "Here are the parameters we used to produce the model.\n",
    "To do this, we used the following resources:\n",
    "- GPU : 2 RTX Titan with 24 Gb RAM\n",
    "- CPU : 12 Cores\n",
    "- RAM : 25 Gb\n",
    "\n",
    "time : 08:55:47\n",
    "\n",
    "Pour les utilisateur du serveur de calcul de l'HPC de l'Université de Genève, vous pouvez retrouver l'exemple de script de lancement `SBATCH` ici : [FoNDUE](https://github.com/FoNDUE-HTR/Documentation/blob/master/CLUSTERS.md) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90c02899-72af-4b4f-bb29-ca40fe289aa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "from ultralytics import YOLO\n",
    "\n",
    "# empty CUDA cache\n",
    "torch.cuda.empty_cache()\n",
    "\n",
    "# load pretrained model\n",
    "model = YOLO('yolov8x.pt') #it's huge model, you can use other little model if you want.\n",
    "model.to('cuda')\n",
    "\n",
    "# get config path\n",
    "dataset_path = os.getcwd() + dataset + 'config.yml'\n",
    "\n",
    "\n",
    "#train\n",
    "model.train(data=dataset_path, \n",
    "                      epochs=300, \n",
    "                      patience=150, \n",
    "                      imgsz=896, #image size pixel\n",
    "                      batch=32, # it's big batch depending to your GPU Ram capacity\n",
    "                      cache=True, # RAM caching\n",
    "                      device=[0,1]) # for multiprocessing GPU, you can use only one GPU. You need only to put number device (generally 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "579baf8f-f2a1-40a6-8876-cf476b963f72",
   "metadata": {},
   "source": [
    "# VAL/PREDICT/Benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e09d477-0503-40a7-bc08-ce4d5cedaa3a",
   "metadata": {},
   "source": [
    "Once you have completed your training, you can retrieve most of your models at the following address: `runs/detect/train/weights/`. \n",
    "Unless you've saved by iteration, you'll have two models. The `best.pt`, which corresponds to the best performing model according to ultralytics (based on the results of the metrics and your loss curve). The second, `last.pt`, corresponds to the model from the last epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c68e05de-afab-418b-bc07-d785a573f320",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best.pt\n"
     ]
    }
   ],
   "source": [
    "!ls models/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc4670c9-995b-413b-9f9d-b75b2ae60296",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ultralytics import YOLO\n",
    "\n",
    "#path_model = \".PATH/TO/WEIGHTS/best.pt\"\n",
    "path_model = \"models/best.pt\"\n",
    "\n",
    "model = YOLO(path_model)\n",
    "model.to('cuda') # to use GPU\n",
    "model.info()\n",
    "#model.fuse()    #Fuse PyTorch Conv2d and BatchNorm2d layers. This improves inference time and therefore execution time. \n",
    "#These two layers are generally the cause of high RAM usage in the case of large batches. Its use remains situational."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3822de31-3914-4ca4-9f43-a708ff9b7dfa",
   "metadata": {},
   "source": [
    "## Eval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acd3899b-4284-4f02-9bec-96910631185b",
   "metadata": {},
   "source": [
    "You can directly evaluate the quality of your model using its `val()` class method, by entering your annotated dataset in the data parameter. You can reuse your initial dataset to give results on the evaluation data. It's best to use a third-party dataset for testing, so you can quickly visualize biases and, in particular, the risk of overfitting your model.\n",
    "If the plots option is enabled, visualizations will be saved as `runs/detect/valn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7043c849-ce0a-43ed-a8dc-c61a863ba054",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = 'PATH/TO/DATASET/config.yml'\n",
    "\n",
    "metrics = model.val(data = '/home/rayondemiel/Grand_Siecle/yolov8-testing/data/dataset-GallicorporaXVIXVIIIxFonduexSETAF/config.yml', plots=True)  # no arguments needed, dataset and settings remembered\n",
    "metrics.box.map    # map50-95\n",
    "metrics.box.map50  # map50\n",
    "metrics.box.map75  # map75\n",
    "metrics.box.maps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f465d0d8-2879-49b2-8b79-6ac3480d27ec",
   "metadata": {},
   "source": [
    "To better understand the results:\n",
    "- `Class`: objects to be detected, in this case Segmonto zones\n",
    "- `Images`: number of images used for evaluation\n",
    "- `Box(P)`: Accuracy of your class relative to a box. Simply put, has the model succeeded in determining the correct class for this zone?\n",
    "- `R`: Recall your class. This is used to determine whether your model has missed too many zones in its identification, thereby distorting the accuracy rate.\n",
    "- `mAP50`: Mean Average Precision with a confidence level of 50. It evaluates the ability of an object detection model to accurately locate objects in an image when the prediction confidence is equal to or greater than 50%.\n",
    "- `mAP50-95`: Confidence threshold between 50 and 95. It indicates that the model is capable of accurately detecting objects over a wide range of confidence thresholds."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f370c872-0590-41b7-92be-f9e4b2b763ad",
   "metadata": {},
   "source": [
    "## Predict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5a9b2ed-3ff0-456c-8169-cf5b13201835",
   "metadata": {},
   "source": [
    "Script for making predictions and visualizing model capacity from web images, especially iiif."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e90f4f9-1303-4b4f-b9d1-39f1fbfb316f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from io import BytesIO\n",
    "from PIL import Image\n",
    "\n",
    "def get_img(url: str):\n",
    "    \"\"\"Web image download function\"\"\"\n",
    "    # download image\n",
    "    response = requests.get(url)\n",
    "    # open image with PIL library\n",
    "    image = Image.open(BytesIO(response.content))\n",
    "    return image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf7e84d3-bfa9-4384-8d69-73421c502abc",
   "metadata": {},
   "source": [
    "##### One image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f169bb5-19a2-457b-9e12-5b066c1c11a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "img = get_img(\"https://gallica.bnf.fr/iiif/ark:/12148/bpt6k97850416/f11/full/,640/0/native.jpg\")\n",
    "results = model(img) # Prediction\n",
    "\n",
    "# Print Result\n",
    "for r in results:\n",
    "    im_array = r.plot(conf=True)  # plot a BGR numpy array of predictions\n",
    "    im = Image.fromarray(im_array[..., ::-1])  # RGB PIL image\n",
    "    plt.imshow(im)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fbf860f-4c92-4a85-96e9-bd0c46520c69",
   "metadata": {},
   "source": [
    "##### Multiple images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1c7e79a-569c-4ad2-8666-a84e7435800e",
   "metadata": {},
   "outputs": [],
   "source": [
    "img = get_img(\"https://gallica.bnf.fr/iiif/ark:/12148/bpt6k9767960q/f24/full/,640/0/native.jpg\")\n",
    "img1 = get_img(\"https://gallica.bnf.fr/iiif/ark:/12148/bpt6k9767960q/f11/full/,640/0/native.jpg\")\n",
    "img2 = get_img(\"https://gallica.bnf.fr/iiif/ark:/12148/bpt6k9767960q/f28/full/,640/0/native.jpg\")\n",
    "img3 = get_img(\"https://gallica.bnf.fr/iiif/ark:/12148/bpt6k9767960q/f24/full/,640/0/native.jpg\")\n",
    "\n",
    "# Run inference on 'bus.jpg'\n",
    "results = model([img, img1, img2, img3])  # results list\n",
    "#results[0].boxes.data  # to check tensor\n",
    "\n",
    "# Show the results\n",
    "for r in results:\n",
    "    im_array = r.plot(conf=True)  # plot a BGR numpy array of predictions\n",
    "    im = Image.fromarray(im_array[..., ::-1])  # RGB PIL image\n",
    "    plt.imshow(im)\n",
    "    plt.show()\n",
    "    #im.save('results.jpg')  # save image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff868459-309d-4b62-b0fe-13dc65cd4e19",
   "metadata": {},
   "source": [
    "## EXPORT ONNX"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "213ef74c-53b3-4b0a-b92e-de5b28fc14c7",
   "metadata": {},
   "source": [
    "Convert the model to ONNX (Open Neural Network Exchange) format, making it interoperable.\n",
    "You can then easily reprocess your model to further improve its predictions, notably in terms of execution time, using [Nvidia TensorRT](https://developer.nvidia.com/tensorrt)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d4b87dc-0463-4e43-af29-43eab04892df",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.export(format=\"onnx\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}