Collab 9 - Validación de datos con TensorFlow Data Validation (TFDV).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5AoMQHXm_W6F"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/joanby/tensorflow2/blob/master/Collab%209%20-%20Validación%20de%20datos%20con%20TensorFlow%20Data%20Validation%20(TFDV).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wf6nxdDBtGN5"
   },
   "source": [
    "## Paso 1:Instalar todas las dependencias y configurar el entorno"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "id": "q8b1TeZBs_2S",
    "outputId": "94a39afa-6a37-4f53-8574-56556e8e9742"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "python-dev is already the newest version (2.7.15~rc1-1).\n",
      "The following NEW packages will be installed:\n",
      "  python-snappy\n",
      "0 upgraded, 1 newly installed, 0 to remove and 22 not upgraded.\n",
      "Need to get 10.8 kB of archives.\n",
      "After this operation, 39.9 kB of additional disk space will be used.\n",
      "Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-snappy amd64 0.5-1.1build2 [10.8 kB]\n",
      "Fetched 10.8 kB in 0s (35.6 kB/s)\n",
      "Selecting previously unselected package python-snappy.\n",
      "(Reading database ... 144618 files and directories currently installed.)\n",
      "Preparing to unpack .../python-snappy_0.5-1.1build2_amd64.deb ...\n",
      "Unpacking python-snappy (0.5-1.1build2) ...\n",
      "Setting up python-snappy (0.5-1.1build2) ...\n"
     ]
    }
   ],
   "source": [
    "!apt-get install python-dev python-snappy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 734
    },
    "id": "7bMUsv3JwU2A",
    "outputId": "84fe5c87-810b-43be-d818-b14421f46a49"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |████████████████████████████████| 1.3MB 3.3MB/s \n",
      "\u001b[K     |████████████████████████████████| 296kB 13.3MB/s \n",
      "\u001b[K     |████████████████████████████████| 378kB 19.0MB/s \n",
      "\u001b[K     |████████████████████████████████| 8.6MB 22.0MB/s \n",
      "\u001b[K     |████████████████████████████████| 1.8MB 62.7MB/s \n",
      "\u001b[K     |████████████████████████████████| 63.8MB 47kB/s \n",
      "\u001b[K     |████████████████████████████████| 71kB 10.9MB/s \n",
      "\u001b[K     |████████████████████████████████| 61kB 9.5MB/s \n",
      "\u001b[K     |████████████████████████████████| 1.4MB 55.9MB/s \n",
      "\u001b[K     |████████████████████████████████| 829kB 53.4MB/s \n",
      "\u001b[K     |████████████████████████████████| 81kB 12.0MB/s \n",
      "\u001b[K     |████████████████████████████████| 51kB 8.2MB/s \n",
      "\u001b[K     |████████████████████████████████| 153kB 58.7MB/s \n",
      "\u001b[K     |████████████████████████████████| 153kB 59.9MB/s \n",
      "\u001b[K     |████████████████████████████████| 256kB 58.7MB/s \n",
      "\u001b[K     |████████████████████████████████| 276kB 57.3MB/s \n",
      "\u001b[K     |████████████████████████████████| 122kB 64.5MB/s \n",
      "\u001b[K     |████████████████████████████████| 184kB 65.0MB/s \n",
      "\u001b[K     |████████████████████████████████| 440kB 50.2MB/s \n",
      "\u001b[K     |████████████████████████████████| 92kB 13.2MB/s \n",
      "\u001b[K     |████████████████████████████████| 174kB 62.7MB/s \n",
      "\u001b[K     |████████████████████████████████| 174kB 57.2MB/s \n",
      "\u001b[K     |████████████████████████████████| 112kB 64.7MB/s \n",
      "\u001b[K     |████████████████████████████████| 1.2MB 44.1MB/s \n",
      "\u001b[K     |████████████████████████████████| 266kB 60.1MB/s \n",
      "\u001b[K     |████████████████████████████████| 153kB 62.4MB/s \n",
      "\u001b[?25h  Building wheel for future (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for oauth2client (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for avro-python3 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for hdfs (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for dill (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for google-apitools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for grpc-google-iam-v1 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for idna-ssl (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[31mERROR: pydrive 1.3.1 has requirement oauth2client>=4.0.0, but you'll have oauth2client 3.0.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: multiprocess 0.70.10 has requirement dill>=0.3.2, but you'll have dill 0.3.1.1 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-colab 1.0.0 has requirement google-auth~=1.17.2, but you'll have google-auth 1.22.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-colab 1.0.0 has requirement requests~=2.23.0, but you'll have requests 2.24.0 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-spanner 1.19.0 has requirement google-cloud-core<2.0dev,>=1.4.1, but you'll have google-cloud-core 1.0.3 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: google-cloud-bigtable 1.5.0 has requirement google-cloud-core<2.0dev,>=1.4.1, but you'll have google-cloud-core 1.0.3 which is incompatible.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install -q --upgrade tensorflow_data_validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Yji0YgSFHoei"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.kill(os.getpid(), 9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FO47q651xVKp"
   },
   "source": [
    "## Paso 2: Importar las librerías para el proyecto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "v15-TQQ4wf4v"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import tensorflow_data_validation as tfdv\n",
    "\n",
    "from __future__ import print_function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S5LUUynDxvak"
   },
   "source": [
    "## Paso 3: Análisis descriptivo del dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "v7r7kstFxfRQ"
   },
   "outputs": [],
   "source": [
    "dataset = pd.read_csv(\"https://raw.githubusercontent.com/joanby/tensorflow2/master/datasets/pollution-small.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "Ha53nh-HyqF3",
    "outputId": "a4cd4544-b0d9-46b7-be33-a5f97db1a1d0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2188, 5)"
      ]
     },
     "execution_count": 3,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "ej93eK0Aysv4"
   },
   "outputs": [],
   "source": [
    "training_data = dataset[:1600]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "TTZVB628y38P",
    "outputId": "dbe98c23-fa8f-40f4-e960-80c6fec459eb"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pm10</th>\n",
       "      <th>no2</th>\n",
       "      <th>so2</th>\n",
       "      <th>soot</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1600.000000</td>\n",
       "      <td>1600.000000</td>\n",
       "      <td>1600.000000</td>\n",
       "      <td>1600.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>49.656494</td>\n",
       "      <td>30.980519</td>\n",
       "      <td>16.229981</td>\n",
       "      <td>21.551956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>35.211906</td>\n",
       "      <td>12.400788</td>\n",
       "      <td>10.621896</td>\n",
       "      <td>12.127354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>6.380000</td>\n",
       "      <td>9.740000</td>\n",
       "      <td>4.010000</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>28.345000</td>\n",
       "      <td>22.567500</td>\n",
       "      <td>9.777500</td>\n",
       "      <td>14.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>38.835000</td>\n",
       "      <td>28.715000</td>\n",
       "      <td>13.275000</td>\n",
       "      <td>18.630000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>58.050000</td>\n",
       "      <td>36.370000</td>\n",
       "      <td>19.282500</td>\n",
       "      <td>24.072500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>277.250000</td>\n",
       "      <td>138.010000</td>\n",
       "      <td>123.130000</td>\n",
       "      <td>107.650000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              pm10          no2          so2         soot\n",
       "count  1600.000000  1600.000000  1600.000000  1600.000000\n",
       "mean     49.656494    30.980519    16.229981    21.551956\n",
       "std      35.211906    12.400788    10.621896    12.127354\n",
       "min       6.380000     9.740000     4.010000     6.000000\n",
       "25%      28.345000    22.567500     9.777500    14.400000\n",
       "50%      38.835000    28.715000    13.275000    18.630000\n",
       "75%      58.050000    36.370000    19.282500    24.072500\n",
       "max     277.250000   138.010000   123.130000   107.650000"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "lNepyJ0gy9IZ"
   },
   "outputs": [],
   "source": [
    "test_set = dataset[1600:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "O9zmE9FEzgOH",
    "outputId": "79f2425d-5e37-4070-ebb5-3c5421aa3ff0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pm10</th>\n",
       "      <th>no2</th>\n",
       "      <th>so2</th>\n",
       "      <th>soot</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>588.000000</td>\n",
       "      <td>588.000000</td>\n",
       "      <td>588.000000</td>\n",
       "      <td>588.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>44.648248</td>\n",
       "      <td>37.296922</td>\n",
       "      <td>13.605170</td>\n",
       "      <td>18.441310</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>28.992087</td>\n",
       "      <td>10.940050</td>\n",
       "      <td>5.098944</td>\n",
       "      <td>6.596459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>11.900000</td>\n",
       "      <td>15.070000</td>\n",
       "      <td>4.990000</td>\n",
       "      <td>8.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>28.337500</td>\n",
       "      <td>29.217500</td>\n",
       "      <td>10.122500</td>\n",
       "      <td>14.410000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>35.555000</td>\n",
       "      <td>35.815000</td>\n",
       "      <td>12.345000</td>\n",
       "      <td>17.090000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>50.812500</td>\n",
       "      <td>43.872500</td>\n",
       "      <td>15.855000</td>\n",
       "      <td>20.962500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>273.770000</td>\n",
       "      <td>106.030000</td>\n",
       "      <td>38.030000</td>\n",
       "      <td>87.210000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             pm10         no2         so2        soot\n",
       "count  588.000000  588.000000  588.000000  588.000000\n",
       "mean    44.648248   37.296922   13.605170   18.441310\n",
       "std     28.992087   10.940050    5.098944    6.596459\n",
       "min     11.900000   15.070000    4.990000    8.000000\n",
       "25%     28.337500   29.217500   10.122500   14.410000\n",
       "50%     35.555000   35.815000   12.345000   17.090000\n",
       "75%     50.812500   43.872500   15.855000   20.962500\n",
       "max    273.770000  106.030000   38.030000   87.210000"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iGqBm9LZzyqi"
   },
   "source": [
    "## Paso 3b: Análisis descriptivo y validación con TFDV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wAVQXEGG0RKH"
   },
   "source": [
    "### Generar análisis descriptivo del dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "WByxycNeziDA"
   },
   "outputs": [],
   "source": [
    "train_stats = tfdv.generate_statistics_from_dataframe(dataframe=training_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BitwkaZ00_iL"
   },
   "source": [
    "### Inferir el esquema "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "26We0ahE1CKP"
   },
   "outputs": [],
   "source": [
    "schema = tfdv.infer_schema(statistics=train_stats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 235
    },
    "id": "2ZHnaOxg24t3",
    "outputId": "da160cbc-9f3f-4110-8461-4ce55ed56d2f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "      <th>Presence</th>\n",
       "      <th>Valency</th>\n",
       "      <th>Domain</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Feature name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>'Date'</th>\n",
       "      <td>BYTES</td>\n",
       "      <td>required</td>\n",
       "      <td></td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'pm10'</th>\n",
       "      <td>FLOAT</td>\n",
       "      <td>required</td>\n",
       "      <td></td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'no2'</th>\n",
       "      <td>FLOAT</td>\n",
       "      <td>required</td>\n",
       "      <td></td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'so2'</th>\n",
       "      <td>FLOAT</td>\n",
       "      <td>required</td>\n",
       "      <td></td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'soot'</th>\n",
       "      <td>FLOAT</td>\n",
       "      <td>required</td>\n",
       "      <td></td>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Type  Presence Valency Domain\n",
       "Feature name                                \n",
       "'Date'        BYTES  required              -\n",
       "'pm10'        FLOAT  required              -\n",
       "'no2'         FLOAT  required              -\n",
       "'so2'         FLOAT  required              -\n",
       "'soot'        FLOAT  required              -"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tfdv.display_schema(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uWtUFaJQ3Tuc"
   },
   "source": [
    "### Calcular los estadísticos  descriptivos del conjunto de testing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "ok3jQTVt2-zI"
   },
   "outputs": [],
   "source": [
    "test_stats = tfdv.generate_statistics_from_dataframe(dataframe=test_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6BnDXqiz3vy6"
   },
   "source": [
    "## Paso 4: Comparar estadísticos del conjunto de test con Schema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kWlDe6VJ37ko"
   },
   "source": [
    "### Buscar anomalías en los nuevos datos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "UAfRZBnL3qx3"
   },
   "outputs": [],
   "source": [
    "anomalies = tfdv.validate_statistics(statistics=test_stats, schema=schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "shY10N0N3_DD"
   },
   "source": [
    "### Mostrar todas las anomalías detectadas\n",
    "\n",
    "- Número entero mayor que  10\n",
    "- Tipo STRING cuando se esperaba un tipo INT\n",
    "- Tipo FLOAT cuando se esperaba un tipo INT\n",
    "- Número entero menor que 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 57
    },
    "id": "pexizEYr4Bqn",
    "outputId": "1ee49e70-5237-452e-9b81-18297420912a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4 style=\"color:green;\">No anomalies found.</h4>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tfdv.display_anomalies(anomalies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2aWeFdYQ4Cb5"
   },
   "source": [
    "### Nuevos datos CON anomalías"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "PQoDCrd04HK2"
   },
   "outputs": [],
   "source": [
    "test_set_copy = test_set.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "tldZL65465hm"
   },
   "outputs": [],
   "source": [
    "test_set_copy.drop(\"soot\", axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O6AbhtEY4Hp-"
   },
   "source": [
    "### Estadísticos basados en datos con anomalías"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "DNXMz0gD4Skv"
   },
   "outputs": [],
   "source": [
    "test_set_copy_stats = tfdv.generate_statistics_from_dataframe(dataframe=test_set_copy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "wJQS09K17XKf"
   },
   "outputs": [],
   "source": [
    "anomalies_new = tfdv.validate_statistics(statistics=test_set_copy_stats, schema=schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 165
    },
    "id": "texSxcnO7h8Q",
    "outputId": "6b902d62-e961-4666-cf35-a40cc91a33d5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/tensorflow_data_validation/utils/display_util.py:173: FutureWarning: Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use None to not limit the column width.\n",
      "  pd.set_option('max_colwidth', -1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Anomaly short description</th>\n",
       "      <th>Anomaly long description</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Feature name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>'soot'</th>\n",
       "      <td>Column dropped</td>\n",
       "      <td>Column is completely missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Anomaly short description      Anomaly long description\n",
       "Feature name                                                        \n",
       "'soot'        Column dropped            Column is completely missing"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tfdv.display_anomalies(anomalies_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IlrBt63h7_UB"
   },
   "source": [
    "## Paso 5: Preparar el esquema para subir a producción"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "MgWLB_TW8Epa"
   },
   "outputs": [],
   "source": [
    "schema.default_environment.append(\"TRAINING\")\n",
    "schema.default_environment.append(\"SERVING\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BSuPxRpr8Cxg"
   },
   "source": [
    "### Eliminar la columna objetivo del esquema de producción"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "8_H9a7WD7l3A"
   },
   "outputs": [],
   "source": [
    "tfdv.get_feature(schema, \"soot\").not_in_environment.append(\"SERVING\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QonMtSXb8KWY"
   },
   "source": [
    "### Comprobar anomalías entre el entorno del servidor y nuevos datos entrantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "ZXdZ9tk58Rh_"
   },
   "outputs": [],
   "source": [
    "serving_env_anomalies = tfdv.validate_statistics(test_set_copy_stats, schema, environment=\"SERVING\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 57
    },
    "id": "05jDXv8o_J0O",
    "outputId": "67445f03-065b-4a1f-aacc-a21a7f9f6424"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4 style=\"color:green;\">No anomalies found.</h4>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tfdv.display_anomalies(serving_env_anomalies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KFR3LpXSdlct"
   },
   "source": [
    "## Paso 6: Congelar el esquema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "-Ok0E4AwzbH0"
   },
   "outputs": [],
   "source": [
    "tfdv.write_schema_text(schema = schema, output_path = \"pollution_schema.pbtxt\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Collab 9 - Validación de datos con TensorFlow Data Validation (TFDV).ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}