tutorials/notebooks/04_q2_datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q2 Training Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training Datasets (TDS) in EOTDL are categorized into different [quality levels](https://eotdl.com/docs/datasets/quality), which in turn will impact the range of functionality that will be available for each dataset.\n",
    "\n",
    "In this tutorial you will learn about Q2 datsets, datasets with STAC metadata and EOTDL's custom STAC extensions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The ML-Dataset Extension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main extension used by EOTDL for Q2 datasets is the ML-Dataset extension. It enhances the STAC metadata of a dataset including information such as data splits (train, validation, test), quality metrics, etc.\n",
    "\n",
    "Let's see how to generate a Q2 dataset using the EOTDL library for the EuroSAT dataset. Q2 datasets are generated from Q1 datasets, datasets with STAC metadata. We already showed how to generate a Q1 dataset in the previous tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['EuroSAT-RGB-small-STAC',\n",
       " 'jaca_dataset_stac',\n",
       " 'eurosat_rgb_dataset',\n",
       " 'jaca_dataset_q2',\n",
       " 'labels_scaneo',\n",
       " 'jaca_dataset_structured',\n",
       " 'EuroSAT-RGB-small',\n",
       " 'jaca_dataset',\n",
       " 'EuroSAT-small',\n",
       " 'jaca_dataset_stac_labels',\n",
       " 'sample_stacdataframe.csv',\n",
       " 'eurosat_rgb_stac',\n",
       " 'eurosat_rgb_stac_labels']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os \n",
    "\n",
    "os.listdir('example_data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating splits...\n",
      "Total size: 100\n",
      "Train size: 80\n",
      "Test size: 10\n",
      "Validation size: 10\n",
      "Generating Training split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 80/80 [00:00<00:00, 4559.65it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating Validation split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:00<00:00, 3770.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating Test split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:00<00:00, 3216.98it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success on splits generation!\n",
      "Validating and saving...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success!\n"
     ]
    }
   ],
   "source": [
    "from eotdl.curation.stac.extensions import add_ml_extension\n",
    "\n",
    "catalog = 'data/EuroSAT-small-STAC/catalog.json'\n",
    "\n",
    "add_ml_extension(\n",
    "\tcatalog,\n",
    "\tdestination='data/EuroSAT-Q2-small',\n",
    "\tsplits=True,\n",
    "\tsplits_collection_id=\"labels\",\n",
    "\tname='EuroSAT Q2 (small) Dataset',\n",
    "\ttasks=['image classification'],\n",
    "\tinputs_type=['satellite imagery'],\n",
    "\tannotations_type='raster',\n",
    "\tversion='0.1.0'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When ingesting a Q2 dataset, EOTDL will automatically compute quality metrics on your dataset, that will be reported in the metadata. Optionally, you can compute them to analyse your dataset before ingesting it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Looking for spatial duplicates...: 0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Looking for spatial duplicates...: 400it [00:00, 4398.40it/s]\n",
      "Calculating classes balance...: 400it [00:00, 168260.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validating and saving...\n",
      "Success!\n"
     ]
    }
   ],
   "source": [
    "from eotdl.curation.stac.extensions import MLDatasetQualityMetrics\n",
    "\n",
    "catalog = 'data/EuroSAT-Q2-small/catalog.json'\n",
    "\n",
    "MLDatasetQualityMetrics.calculate(catalog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, however, that the metrics will be computed automatically when ingesting the dataset, so you don't need to do it yourself. These metrics incude aspects such as the number of samples, duplicates, missing values, class imbalance, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting Q2 datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the metadata has been generated, you can ingest, explore and stage a Q2 dataset as any other dataset.\n",
    "\n",
    "> Change the id of the catalog if you want to give it a specific name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading STAC catalog...\n",
      "New version created, version: 2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 400/400 [01:16<00:00,  5.23it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting STAC catalog...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "from eotdl.datasets import ingest_dataset\n",
    "\n",
    "ingest_dataset('data/EuroSAT-Q2-small')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}