tutorials/notebooks/27_stac_ml_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# STAC ML-Dataset extension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the following line to install eotdl if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install eotdl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main extension used by EOTDL for Q2 datasets is the ML-Dataset extension. It enhances the STAC metadata of a dataset including information such as data splits (train, validation, test), quality metrics, etc.\n",
    "\n",
    "Let's see how to generate a Q2 dataset using the EOTDL library for the EuroSAT dataset. Q2 datasets are generated from Q1 datasets, datasets with STAC metadata. We already showed how to generate a Q1 dataset in the previous section.\n",
    "\n",
    "The addition of the `ml-dataset` STAC extension to a STAC catalog is pretty straightforward, so it can be done with a simple function called `add_ml_extension`. Sounds easy, right? Let's see what we need.\n",
    "- `catalog`: the path to the STAC catalog, or the pystac Catalog itself, we want to add the extension. In our case, `data/sentinel_2_stac/catalog.json`.\n",
    "- `destination`: if we want we can define an output folder to save the catalog, but by default the function generates it in the same folder of the given catalog. In our case, `data/sentinel_2_q2`.\n",
    "- `splits`: we should put is as `True` if we want to split the labels. By default is `False`, and the default values for the splits are `Train`, `Test` and `Validation` in a `80, 10, 10` proportion. In our case is `True`, and we are fine with the default proportions.\n",
    "- `splits_collection_id`: the id of the collection we want to make the splits to. In our case, `labels`, which is the default option.\n",
    "- `name`: the name of the dataset. In our case, `Q2 Dataset`, but feel free to customize it at your own.\n",
    "- `tasks`: the tasks of the dataset. In our case, `[segmentation]`.\n",
    "- `inputs_type`: the type of the dataset inputs. In our case, `[satellite imagery]`.\n",
    "- `annotations_type`: the type of the annotations. In our case, `raster`.\n",
    "- `version`: the version of our dataset. In our case, `0.1.0`.\n",
    "\n",
    "Let's add the extension!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating splits...\n",
      "Total size: 4\n",
      "Train size: 3\n",
      "Test size: 0\n",
      "Validation size: 0\n",
      "Generating Training split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 29399.33it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating Validation split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0it [00:00, ?it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating Test split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success on splits generation!\n",
      "Validating and saving...\n",
      "Success!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from eotdl.curation.stac.extensions import add_ml_extension\n",
    "\n",
    "catalog = 'example_data/jaca_dataset_stac_labels/catalog.json'\n",
    "\n",
    "add_ml_extension(\n",
    "\tcatalog,\n",
    "\tdestination='data/jaca_dataset_q2',\n",
    "\tsplits=True,\n",
    "\tsplits_collection_id=\"labels\",\n",
    "\tname='Q2 Dataset',\n",
    "\ttasks=['segmentation'],\n",
    "\tinputs_type=['satellite imagery'],\n",
    "\tannotations_type='raster',\n",
    "\tversion='0.1.0'\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sen12floods",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}