readi_reproduction.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nicolashernandez/READI-LREC22/blob/main/readi_reproduction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a6OK4ukoLyFI"
      },
      "source": [
        "# Introduction\n",
        "\n",
        "This notebook comes from the git repository available [here](https://github.com/nicolashernandez/READI-LREC22/)  \n",
        "It will show how to reproduce the contents of the READI paper available [here](https://cental.uclouvain.be/readi2022/accepted.html), then present an introduction on how to manipulate the library.  \n",
        "In order to speed up deep learning applications significantly, please enable GPU in this notebook's parameters :  \n",
        "Edit -> Notebook Settings -> Hardware Accelerator : GPU\n",
        " \n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Setup : Importing library and corpuses"
      ],
      "metadata": {
        "id": "xBexs2rPciL2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "# 1. Download project and set current directory\n",
        "!git clone https://github.com/nicolashernandez/READI-LREC22/\n",
        "%cd READI-LREC22/"
      ],
      "metadata": {
        "id": "jDdveq3ZUI__"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "# 2. Install module, should take around a minute to install every dependency\n",
        "!pip install ."
      ],
      "metadata": {
        "id": "Z_XksS_gUJOQ"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 3. Add project directory to the path (not needed but helps Colab editor with \n",
        "# auto-completion if you wish to try the library)\n",
        "import sys,os\n",
        "sys.path.append(os.getcwd())\n",
        "sys.path.append(os.path.join(os.getcwd(),\"readability\"))\n",
        "sys.path.append(os.path.join(os.getcwd(),\"readability\",\"readability\"))"
      ],
      "metadata": {
        "id": "y_GOTUVpUJb8"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "1dlJDld-LyFO"
      },
      "outputs": [],
      "source": [
        "import readability"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Recreating experiments"
      ],
      "metadata": {
        "id": "nTaUch-_Yd53"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Six files are located in the git repository that was cloned : in the READI-LREC22/readability/data folder.  \n",
        "These contain the cleaned and formatted content of the corpuses used in our project, and will be used for the demonstrations."
      ],
      "metadata": {
        "id": "2m7Wg8S1Plye"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "with open(os.path.join(os.getcwd(),\"readability\",\"data\",\"tokens_split.pkl\"), \"rb\") as file:\n",
        "    corpus_ljl = pickle.load(file)\n",
        "with open(os.path.join(os.getcwd(),\"readability\",\"data\",\"bibebook.com.pkl\"), \"rb\") as file:\n",
        "    corpus_bb = pickle.load(file)\n",
        "with open(os.path.join(os.getcwd(),\"readability\",\"data\",\"JeLisLibre_md.pkl\"), \"rb\") as file:\n",
        "    corpus_jll = pickle.load(file)\n"
      ],
      "metadata": {
        "id": "7FMcd7YaQfhZ"
      },
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you wish to view the content, simply treat it as a dictionary containing texts, classes can be known by doing dict.keys().  \n",
        "Each text being a list of sentences, which are lists of tokens.  \n",
        "For instance: corpus_ljl['level1'][0][0] would give you the first sentence of the first text in the ljl corpus, for the \"level1\" class.  \n"
      ],
      "metadata": {
        "id": "vnILar8cQ6-1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "corpus_ljl['level1'][0][0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sMMUSlzgQy-s",
        "outputId": "1190d19b-9bb6-4d19-cb7f-90fe8c60c642"
      },
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"Aujourd'hui\",\n",
              " ',',\n",
              " 'toute',\n",
              " 'la',\n",
              " 'famille',\n",
              " 'est',\n",
              " 'allée',\n",
              " 'à',\n",
              " 'la',\n",
              " 'fête',\n",
              " 'foraine',\n",
              " '.']"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for level in corpus_bb.keys():\n",
        "  for text in corpus_bb[level][:]:\n",
        "    if len(text)==0:\n",
        "      corpus_bb[level].remove(text)\n",
        "\n",
        "for level in corpus_jll.keys():\n",
        "  for text in corpus_jll[level][:]:\n",
        "    if len(text)==0:\n",
        "      corpus_jll[level].remove(text)"
      ],
      "metadata": {
        "id": "Aqw3YoaZUYlT"
      },
      "execution_count": 61,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Introducing the first main component of the library\n",
        "\n",
        "The class that takes care of calling the processes developped for handling various text readability tasks is called the \"ReadabilityProcessor\".\n",
        "\n",
        "When initializing it, it loads external resources that may be needed such as NLP processors, language models, or dataframes containing data such as word lists.\n",
        "\n",
        "Each measure is enabled by default, but can be excluded, alongside their dependencies, on a case-by-case basis."
      ],
      "metadata": {
        "id": "sNHq8U-R6Ydn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "readability_processor = readability.Readability(exclude=[\"cosine_similarity_LDA\"])"
      ],
      "metadata": {
        "id": "xMDxuAnjcF1o",
        "outputId": "bacd33cd-e009-40bc-b260-7febfab1f066",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Acquiring Natural Language Processor...\n",
            "DEBUG: Spacy model location (already installed):  /usr/local/lib/python3.7/dist-packages/fr_core_news_sm/fr_core_news_sm-3.2.0\n",
            "importing GPT2 model..\n",
            "imported GPT2 model\n",
            "importing lexique data as dataframe\n",
            "lexique dataframe imported\n",
            "importing dubois-buyse data as dataframe\n",
            "dubois-buyse dataframe imported\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Introducing the second main component of the library\n",
        "\n",
        "The ParsedText class, and its extension ParsedCollection can be created from a readability processor when supplied with a collection.\n",
        "\n",
        "They're not only an interface between texts and the readability processor, these are also used to store the measures obtained from the readability processor, alongside various common statistics, which can be re-used across several of the processor's functions in order to speed up the process.\n"
      ],
      "metadata": {
        "id": "DNib045T9lhn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "processed_corpus_ljl = readability_processor.parseCollection(corpus_ljl)\n",
        "processed_corpus_jll = readability_processor.parseCollection(corpus_jll)\n",
        "processed_corpus_bb = readability_processor.parseCollection(corpus_bb)"
      ],
      "metadata": {
        "id": "8lenRpQVcF0V"
      },
      "execution_count": 63,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Reproducing the contents of table 2"
      ],
      "metadata": {
        "id": "EptHo7EPMqGk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "metadata": {
        "id": "SF_IJ5WbpmSW"
      },
      "execution_count": 64,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def reproduce_table2(processed_corpus):\n",
        "  needed_labels = [\"totalTexts\",\"totalSentences\",\"totalWords\"]\n",
        "  stats = [[],[],[]]\n",
        "  for class_label in list(processed_corpus.statistics.keys()):\n",
        "    for index,stat_label in enumerate(needed_labels):\n",
        "      stats[index].append(processed_corpus.statistics[class_label][stat_label])\n",
        "\n",
        "  for stat in stats:\n",
        "    stat.append(sum(stat))\n",
        "\n",
        "  df = pd.DataFrame([stats[0],stats[1],stats[2]],columns=(list(processed_corpus.statistics.keys())+[\"total\"]))\n",
        "  df.index = [\"Nombre de fichiers artificiel\",\"Nombre de phrases total\",\"Nombre de tokens\"]\n",
        "  return df"
      ],
      "metadata": {
        "id": "u4ZlVxo6jCwu"
      },
      "execution_count": 65,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_table2_ljl = reproduce_table2(processed_corpus_ljl)\n",
        "original_documents = [240,314,134,58,746]\n",
        "df_table2_ljl.loc[\"Nombre de fichiers original\"] = original_documents\n",
        "\n",
        "df_table2_bb = reproduce_table2(processed_corpus_bb)\n",
        "original_documents = [52,91,65,208]\n",
        "df_table2_bb.loc[\"Nombre de fichiers original\"] = original_documents\n",
        "\n",
        "df_table2_jll = reproduce_table2(processed_corpus_jll)\n",
        "original_documents = [13,12,10,9,44]\n",
        "df_table2_jll.loc[\"Nombre de fichiers original\"] = original_documents"
      ],
      "metadata": {
        "id": "AkFTrCDwe1Ds"
      },
      "execution_count": 66,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_table2_ljl"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "PhYb8GUBqpPt",
        "outputId": "491b6d14-7959-4682-c403-7ca0729d9495"
      },
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                               level1  level2  level3  level4   total\n",
              "Nombre de fichiers artificiel     240     628     670     522    2060\n",
              "Nombre de phrases total          4880   13049   10354    7743   36026\n",
              "Nombre de tokens                38976  128019  124901  101165  393061\n",
              "Nombre de fichiers original       240     314     134      58     746"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-464baca7-c1d8-4b75-ac20-fe6223ad114f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>level1</th>\n",
              "      <th>level2</th>\n",
              "      <th>level3</th>\n",
              "      <th>level4</th>\n",
              "      <th>total</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Nombre de fichiers artificiel</th>\n",
              "      <td>240</td>\n",
              "      <td>628</td>\n",
              "      <td>670</td>\n",
              "      <td>522</td>\n",
              "      <td>2060</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de phrases total</th>\n",
              "      <td>4880</td>\n",
              "      <td>13049</td>\n",
              "      <td>10354</td>\n",
              "      <td>7743</td>\n",
              "      <td>36026</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de tokens</th>\n",
              "      <td>38976</td>\n",
              "      <td>128019</td>\n",
              "      <td>124901</td>\n",
              "      <td>101165</td>\n",
              "      <td>393061</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de fichiers original</th>\n",
              "      <td>240</td>\n",
              "      <td>314</td>\n",
              "      <td>134</td>\n",
              "      <td>58</td>\n",
              "      <td>746</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-464baca7-c1d8-4b75-ac20-fe6223ad114f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-464baca7-c1d8-4b75-ac20-fe6223ad114f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-464baca7-c1d8-4b75-ac20-fe6223ad114f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_table2_bb"
      ],
      "metadata": {
        "id": "2lIxicsKAjZp",
        "outputId": "d47eac2a-c309-4f3e-f065-fc5e78ee30ff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        }
      },
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                               intermédiaire  avancée  aisée   total\n",
              "Nombre de fichiers artificiel            1729      1253     986    3968\n",
              "Nombre de phrases total                 22088     15762   12274   50124\n",
              "Nombre de tokens                       315369    232604  173939  721912\n",
              "Nombre de fichiers original                52        91      65     208"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6fb818c3-42dd-4722-9c56-d569c6a4e130\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>intermédiaire</th>\n",
              "      <th>avancée</th>\n",
              "      <th>aisée</th>\n",
              "      <th>total</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Nombre de fichiers artificiel</th>\n",
              "      <td>1729</td>\n",
              "      <td>1253</td>\n",
              "      <td>986</td>\n",
              "      <td>3968</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de phrases total</th>\n",
              "      <td>22088</td>\n",
              "      <td>15762</td>\n",
              "      <td>12274</td>\n",
              "      <td>50124</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de tokens</th>\n",
              "      <td>315369</td>\n",
              "      <td>232604</td>\n",
              "      <td>173939</td>\n",
              "      <td>721912</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de fichiers original</th>\n",
              "      <td>52</td>\n",
              "      <td>91</td>\n",
              "      <td>65</td>\n",
              "      <td>208</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6fb818c3-42dd-4722-9c56-d569c6a4e130')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6fb818c3-42dd-4722-9c56-d569c6a4e130 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6fb818c3-42dd-4722-9c56-d569c6a4e130');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_table2_jll"
      ],
      "metadata": {
        "id": "gOx0ioQfAjl_",
        "outputId": "e9ede618-cffe-45af-bc2b-442ce0fc6842",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        }
      },
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                               cycle4_3e  cycle4_4e  cycle4_5e  cycle3_6e  \\\n",
              "Nombre de fichiers artificiel        986        989       1187       1283   \n",
              "Nombre de phrases total            14689      13553      13818      13463   \n",
              "Nombre de tokens                  188091     195375     211099     256573   \n",
              "Nombre de fichiers original           13         12         10          9   \n",
              "\n",
              "                                total  \n",
              "Nombre de fichiers artificiel    4445  \n",
              "Nombre de phrases total         55523  \n",
              "Nombre de tokens               851138  \n",
              "Nombre de fichiers original        44  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-bfceaf6c-c04c-431c-a042-7d214576b0dc\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>cycle4_3e</th>\n",
              "      <th>cycle4_4e</th>\n",
              "      <th>cycle4_5e</th>\n",
              "      <th>cycle3_6e</th>\n",
              "      <th>total</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Nombre de fichiers artificiel</th>\n",
              "      <td>986</td>\n",
              "      <td>989</td>\n",
              "      <td>1187</td>\n",
              "      <td>1283</td>\n",
              "      <td>4445</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de phrases total</th>\n",
              "      <td>14689</td>\n",
              "      <td>13553</td>\n",
              "      <td>13818</td>\n",
              "      <td>13463</td>\n",
              "      <td>55523</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de tokens</th>\n",
              "      <td>188091</td>\n",
              "      <td>195375</td>\n",
              "      <td>211099</td>\n",
              "      <td>256573</td>\n",
              "      <td>851138</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Nombre de fichiers original</th>\n",
              "      <td>13</td>\n",
              "      <td>12</td>\n",
              "      <td>10</td>\n",
              "      <td>9</td>\n",
              "      <td>44</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bfceaf6c-c04c-431c-a042-7d214576b0dc')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-bfceaf6c-c04c-431c-a042-7d214576b0dc button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-bfceaf6c-c04c-431c-a042-7d214576b0dc');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Reproducing the contents of table 3"
      ],
      "metadata": {
        "id": "rYsSYkENM_Kf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Traditional scores"
      ],
      "metadata": {
        "id": "KzhNGCxTfiyu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from scipy.stats import pearsonr"
      ],
      "metadata": {
        "id": "0aGvHf3MyVYX"
      },
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def reproduce_table3(processed_corpus):\n",
        "  needed_traditional_scores = [\"gfi\",\"ari\",\"fre\",\"fkgl\",\"smog\",\"rel\"]\n",
        "  pearson = []\n",
        "  for score in needed_traditional_scores:\n",
        "    labels = []\n",
        "    scores_as_list = []\n",
        "    for label in list(processed_corpus.content.keys()):\n",
        "      for text in processed_corpus.content[label]:\n",
        "        scores_as_list.append(text.traditional_score(score))\n",
        "        labels.append(list(processed_corpus.content.keys()).index(label))\n",
        "    pearson.append(pearsonr(scores_as_list,labels)[0])\n",
        "\n",
        "  \n",
        "  math_formulas = pd.DataFrame([processed_corpus.gfi(),processed_corpus.ari(),\n",
        "                                processed_corpus.fre(),processed_corpus.fkgl(),\n",
        "                                processed_corpus.smog(),processed_corpus.rel()],\n",
        "                                columns=list(processed_corpus.content.keys()))\n",
        "\n",
        "  math_formulas.index = [\"The Gunning fog index GFI\", \"The Automated readability index ARI\",\"The Flesch reading ease FRE\",\"The Flesch-Kincaid grade level FKGL\",\"The Simple Measure of Gobbledygook SMOG\",\"Reading Ease Level\"]\n",
        "  math_formulas['Pearson Score'] = pearson\n",
        "  math_formulas.columns.name = \"Mean values\"\n",
        "  return math_formulas"
      ],
      "metadata": {
        "id": "TWaWcth2kZyT"
      },
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "scores_ljl = reproduce_table3(processed_corpus_ljl)\n",
        "scores_bb = reproduce_table3(processed_corpus_bb)\n",
        "scores_jll = reproduce_table3(processed_corpus_jll)"
      ],
      "metadata": {
        "id": "TkNED87Dlwxc"
      },
      "execution_count": 72,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "scores_ljl"
      ],
      "metadata": {
        "id": "terfgr2XrhYq",
        "outputId": "fcfa8a45-34c9-4632-e299-56ec290b6b25",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Mean values                                 level1     level2     level3  \\\n",
              "The Gunning fog index GFI                45.132518  67.697721  91.866336   \n",
              "The Automated readability index ARI      14.238996  19.932585  25.719148   \n",
              "The Flesch reading ease FRE              72.037523  61.563843  54.090075   \n",
              "The Flesch-Kincaid grade level FKGL       5.183396   7.152952   8.740208   \n",
              "The Simple Measure of Gobbledygook SMOG  16.250487  18.911327  21.336397   \n",
              "Reading Ease Level                       88.681859  79.299771  72.508344   \n",
              "\n",
              "Mean values                                  level4  Pearson Score  \n",
              "The Gunning fog index GFI                105.669951       0.475915  \n",
              "The Automated readability index ARI       27.757700       0.472037  \n",
              "The Flesch reading ease FRE               50.257194      -0.404092  \n",
              "The Flesch-Kincaid grade level FKGL        9.439848       0.457392  \n",
              "The Simple Measure of Gobbledygook SMOG   22.355208       0.486627  \n",
              "Reading Ease Level                        69.086158      -0.410960  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3de078d3-137c-4e74-b5a1-e3962b7b1c15\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Mean values</th>\n",
              "      <th>level1</th>\n",
              "      <th>level2</th>\n",
              "      <th>level3</th>\n",
              "      <th>level4</th>\n",
              "      <th>Pearson Score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>The Gunning fog index GFI</th>\n",
              "      <td>45.132518</td>\n",
              "      <td>67.697721</td>\n",
              "      <td>91.866336</td>\n",
              "      <td>105.669951</td>\n",
              "      <td>0.475915</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Automated readability index ARI</th>\n",
              "      <td>14.238996</td>\n",
              "      <td>19.932585</td>\n",
              "      <td>25.719148</td>\n",
              "      <td>27.757700</td>\n",
              "      <td>0.472037</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch reading ease FRE</th>\n",
              "      <td>72.037523</td>\n",
              "      <td>61.563843</td>\n",
              "      <td>54.090075</td>\n",
              "      <td>50.257194</td>\n",
              "      <td>-0.404092</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch-Kincaid grade level FKGL</th>\n",
              "      <td>5.183396</td>\n",
              "      <td>7.152952</td>\n",
              "      <td>8.740208</td>\n",
              "      <td>9.439848</td>\n",
              "      <td>0.457392</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Simple Measure of Gobbledygook SMOG</th>\n",
              "      <td>16.250487</td>\n",
              "      <td>18.911327</td>\n",
              "      <td>21.336397</td>\n",
              "      <td>22.355208</td>\n",
              "      <td>0.486627</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Reading Ease Level</th>\n",
              "      <td>88.681859</td>\n",
              "      <td>79.299771</td>\n",
              "      <td>72.508344</td>\n",
              "      <td>69.086158</td>\n",
              "      <td>-0.410960</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3de078d3-137c-4e74-b5a1-e3962b7b1c15')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3de078d3-137c-4e74-b5a1-e3962b7b1c15 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3de078d3-137c-4e74-b5a1-e3962b7b1c15');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "scores_bb"
      ],
      "metadata": {
        "id": "Gyja19fVq1ex",
        "outputId": "dea0e900-94fb-46c9-e3cf-4ce7ccb7c222",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Mean values                              intermédiaire    avancée  \\\n",
              "The Gunning fog index GFI                    128.933829  122.607686   \n",
              "The Automated readability index ARI           36.714696   36.263151   \n",
              "The Flesch reading ease FRE                   53.810381   55.380706   \n",
              "The Flesch-Kincaid grade level FKGL            9.987963    9.753552   \n",
              "The Simple Measure of Gobbledygook SMOG       24.227734   24.180287   \n",
              "Reading Ease Level                            71.622888   72.997206   \n",
              "\n",
              "Mean values                                  aisée  Pearson Score  \n",
              "The Gunning fog index GFI                122.606339      -0.037663  \n",
              "The Automated readability index ARI       35.554898      -0.024102  \n",
              "The Flesch reading ease FRE               54.790151       0.024818  \n",
              "The Flesch-Kincaid grade level FKGL        9.742124      -0.024763  \n",
              "The Simple Measure of Gobbledygook SMOG   24.041178      -0.012431  \n",
              "Reading Ease Level                        72.533265       0.024926  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-0d0bf679-8fad-4f4b-b3d8-d90b842c4bb8\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Mean values</th>\n",
              "      <th>intermédiaire</th>\n",
              "      <th>avancée</th>\n",
              "      <th>aisée</th>\n",
              "      <th>Pearson Score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>The Gunning fog index GFI</th>\n",
              "      <td>128.933829</td>\n",
              "      <td>122.607686</td>\n",
              "      <td>122.606339</td>\n",
              "      <td>-0.037663</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Automated readability index ARI</th>\n",
              "      <td>36.714696</td>\n",
              "      <td>36.263151</td>\n",
              "      <td>35.554898</td>\n",
              "      <td>-0.024102</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch reading ease FRE</th>\n",
              "      <td>53.810381</td>\n",
              "      <td>55.380706</td>\n",
              "      <td>54.790151</td>\n",
              "      <td>0.024818</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch-Kincaid grade level FKGL</th>\n",
              "      <td>9.987963</td>\n",
              "      <td>9.753552</td>\n",
              "      <td>9.742124</td>\n",
              "      <td>-0.024763</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Simple Measure of Gobbledygook SMOG</th>\n",
              "      <td>24.227734</td>\n",
              "      <td>24.180287</td>\n",
              "      <td>24.041178</td>\n",
              "      <td>-0.012431</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Reading Ease Level</th>\n",
              "      <td>71.622888</td>\n",
              "      <td>72.997206</td>\n",
              "      <td>72.533265</td>\n",
              "      <td>0.024926</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0d0bf679-8fad-4f4b-b3d8-d90b842c4bb8')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-0d0bf679-8fad-4f4b-b3d8-d90b842c4bb8 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-0d0bf679-8fad-4f4b-b3d8-d90b842c4bb8');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "scores_jll"
      ],
      "metadata": {
        "id": "LV0kBMScsHTf",
        "outputId": "8da04531-332a-445f-aab2-963c25454d66",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 316
        }
      },
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Mean values                               cycle4_3e   cycle4_4e   cycle4_5e  \\\n",
              "The Gunning fog index GFI                104.065761  102.421179  132.388006   \n",
              "The Automated readability index ARI       34.358482   36.119010   40.754610   \n",
              "The Flesch reading ease FRE               73.376908   76.957310   57.766046   \n",
              "The Flesch-Kincaid grade level FKGL        7.147536    6.922545    9.912481   \n",
              "The Simple Measure of Gobbledygook SMOG   21.718341   22.137552   24.655992   \n",
              "Reading Ease Level                        88.704426   91.673517   74.811220   \n",
              "\n",
              "Mean values                               cycle3_6e  Pearson Score  \n",
              "The Gunning fog index GFI                119.822886       0.111777  \n",
              "The Automated readability index ARI       46.380268       0.195434  \n",
              "The Flesch reading ease FRE               75.527410      -0.050773  \n",
              "The Flesch-Kincaid grade level FKGL        8.215760       0.138531  \n",
              "The Simple Measure of Gobbledygook SMOG   23.986362       0.175971  \n",
              "Reading Ease Level                        89.848486      -0.062848  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3cb4edc9-210a-4cac-ad68-096f9fe68605\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Mean values</th>\n",
              "      <th>cycle4_3e</th>\n",
              "      <th>cycle4_4e</th>\n",
              "      <th>cycle4_5e</th>\n",
              "      <th>cycle3_6e</th>\n",
              "      <th>Pearson Score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>The Gunning fog index GFI</th>\n",
              "      <td>104.065761</td>\n",
              "      <td>102.421179</td>\n",
              "      <td>132.388006</td>\n",
              "      <td>119.822886</td>\n",
              "      <td>0.111777</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Automated readability index ARI</th>\n",
              "      <td>34.358482</td>\n",
              "      <td>36.119010</td>\n",
              "      <td>40.754610</td>\n",
              "      <td>46.380268</td>\n",
              "      <td>0.195434</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch reading ease FRE</th>\n",
              "      <td>73.376908</td>\n",
              "      <td>76.957310</td>\n",
              "      <td>57.766046</td>\n",
              "      <td>75.527410</td>\n",
              "      <td>-0.050773</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch-Kincaid grade level FKGL</th>\n",
              "      <td>7.147536</td>\n",
              "      <td>6.922545</td>\n",
              "      <td>9.912481</td>\n",
              "      <td>8.215760</td>\n",
              "      <td>0.138531</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Simple Measure of Gobbledygook SMOG</th>\n",
              "      <td>21.718341</td>\n",
              "      <td>22.137552</td>\n",
              "      <td>24.655992</td>\n",
              "      <td>23.986362</td>\n",
              "      <td>0.175971</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Reading Ease Level</th>\n",
              "      <td>88.704426</td>\n",
              "      <td>91.673517</td>\n",
              "      <td>74.811220</td>\n",
              "      <td>89.848486</td>\n",
              "      <td>-0.062848</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3cb4edc9-210a-4cac-ad68-096f9fe68605')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3cb4edc9-210a-4cac-ad68-096f9fe68605 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3cb4edc9-210a-4cac-ad68-096f9fe68605');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Pseudo-perplexity (takes around an hour to calculate)"
      ],
      "metadata": {
        "id": "4cpHqWCZfg_5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def add_perplexity_to_table3(processed_corpus,scores_dataframe):\n",
        "  pearson = []\n",
        "  labels = []\n",
        "  perplexity_as_list = []\n",
        "  for label in list(processed_corpus.content.keys()):\n",
        "    for text in processed_corpus.content[label]:\n",
        "      perplexity_as_list.append(text.perplexity())\n",
        "      labels.append(list(processed_corpus.content.keys()).index(label))\n",
        "  pearson.append(pearsonr(perplexity_as_list,labels)[0])\n",
        "\n",
        "  scores_dataframe.loc[\"Pseudo_perplexity\"] = list(processed_corpus.perplexity().values()) + pearson\n",
        "  return scores_dataframe"
      ],
      "metadata": {
        "id": "n6b7NOr9xRen"
      },
      "execution_count": 76,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "add_perplexity_to_table3(processed_corpus_ljl, scores_ljl)"
      ],
      "metadata": {
        "id": "d8bYyvLTweps",
        "outputId": "a8882f1b-5852-4927-caa6-e236ba44493e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 348
        }
      },
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "error",
          "ename": "IndexError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-77-e6c12f0fa91d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0madd_perplexity_to_table3\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprocessed_corpus_ljl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscores_ljl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-76-c171805ff04c>\u001b[0m in \u001b[0;36madd_perplexity_to_table3\u001b[0;34m(processed_corpus, scores_dataframe)\u001b[0m\n\u001b[1;32m      5\u001b[0m   \u001b[0;32mfor\u001b[0m \u001b[0mlabel\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprocessed_corpus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mtext\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mprocessed_corpus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m       \u001b[0mperplexity_as_list\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mperplexity\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m       \u001b[0mlabels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprocessed_corpus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m   \u001b[0mpearson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpearsonr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mperplexity_as_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mlabels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/parsed_text/parsed_text.py\u001b[0m in \u001b[0;36mperplexity\u001b[0;34m(self, force)\u001b[0m\n\u001b[1;32m    199\u001b[0m         \u001b[0;34m:\u001b[0m\u001b[0mrtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    200\u001b[0m         \"\"\"\n\u001b[0;32m--> 201\u001b[0;31m         \u001b[0;32mreturn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcall_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"pppl\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mforce\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    202\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mstub_rsrs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mforce\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/parsed_text/parsed_text.py\u001b[0m in \u001b[0;36mcall_score\u001b[0;34m(self, score_name, arguments, force)\u001b[0m\n\u001b[1;32m    104\u001b[0m                 \u001b[0;31m#print(\"WARNING: defaulting to default arguments :\", arguments)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 106\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscores\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mscore_name\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marguments\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    107\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscores\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mscore_name\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    108\u001b[0m         \u001b[0;31m# If function is unavailable, return None to indicate so.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/readability.py\u001b[0m in \u001b[0;36mperplexity\u001b[0;34m(self, content)\u001b[0m\n\u001b[1;32m    345\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"measure 'pppl' cannot be calculated, please try ReadabilityProcessor.load('pppl') and try again.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    346\u001b[0m         \u001b[0;31m#print(\"Please be patient, pseudo-perplexity takes a lot of time to calculate.\")\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 347\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mperplexity\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPPPL_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdependencies\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"GPT2_LM\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    348\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    349\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mstub_rsrs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/stats/perplexity.py\u001b[0m in \u001b[0;36mPPPL_score\u001b[0;34m(GPT2_LM, content)\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mPPPL_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mGPT2_LM\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m     \u001b[0mcontent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_text_to_string\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     19\u001b[0m     \u001b[0mtokenize_input\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mGPT2_LM\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"tokenizer\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0mtensor_input\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtokenize_input\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mGPT2_LM\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"max_length\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/utils/utils.py\u001b[0m in \u001b[0;36mconvert_text_to_string\u001b[0;34m(text)\u001b[0m\n\u001b[1;32m     96\u001b[0m         \u001b[0mdoc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m' '\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtext\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;31m# Make first sentence not start with a whitespace, and remove whitespace between text and last punctuation mark.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     97\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0msentence\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtext\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 98\u001b[0;31m             \u001b[0mdoc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdoc\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m' '\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m' '\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentence\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0msentence\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;31m# Remove whitespace between text and last punctuation mark.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     99\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    100\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mIndexError\u001b[0m: list index out of range"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "add_perplexity_to_table3(processed_corpus_bb, scores_bb)"
      ],
      "metadata": {
        "id": "WE0D1BvGwlLx",
        "outputId": "08733950-10e4-45ca-b55e-300244aa5e77",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        }
      },
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Mean values                              intermédiaire    avancée  \\\n",
              "The Gunning fog index GFI                    128.933829  122.607686   \n",
              "The Automated readability index ARI           36.714696   36.263151   \n",
              "The Flesch reading ease FRE                   53.810381   55.380706   \n",
              "The Flesch-Kincaid grade level FKGL            9.987963    9.753552   \n",
              "The Simple Measure of Gobbledygook SMOG       24.227734   24.180287   \n",
              "Reading Ease Level                            71.622888   72.997206   \n",
              "Pseudo_perplexity                            356.681890  144.490090   \n",
              "\n",
              "Mean values                                  aisée  Pearson Score  \n",
              "The Gunning fog index GFI                122.606339      -0.037663  \n",
              "The Automated readability index ARI       35.554898      -0.024102  \n",
              "The Flesch reading ease FRE               54.790151       0.024818  \n",
              "The Flesch-Kincaid grade level FKGL        9.742124      -0.024763  \n",
              "The Simple Measure of Gobbledygook SMOG   24.041178      -0.012431  \n",
              "Reading Ease Level                        72.533265       0.024926  \n",
              "Pseudo_perplexity                        139.707477      -0.124918  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e30f4cae-6ef4-406d-8bcd-653888b37c4b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Mean values</th>\n",
              "      <th>intermédiaire</th>\n",
              "      <th>avancée</th>\n",
              "      <th>aisée</th>\n",
              "      <th>Pearson Score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>The Gunning fog index GFI</th>\n",
              "      <td>128.933829</td>\n",
              "      <td>122.607686</td>\n",
              "      <td>122.606339</td>\n",
              "      <td>-0.037663</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Automated readability index ARI</th>\n",
              "      <td>36.714696</td>\n",
              "      <td>36.263151</td>\n",
              "      <td>35.554898</td>\n",
              "      <td>-0.024102</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch reading ease FRE</th>\n",
              "      <td>53.810381</td>\n",
              "      <td>55.380706</td>\n",
              "      <td>54.790151</td>\n",
              "      <td>0.024818</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch-Kincaid grade level FKGL</th>\n",
              "      <td>9.987963</td>\n",
              "      <td>9.753552</td>\n",
              "      <td>9.742124</td>\n",
              "      <td>-0.024763</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Simple Measure of Gobbledygook SMOG</th>\n",
              "      <td>24.227734</td>\n",
              "      <td>24.180287</td>\n",
              "      <td>24.041178</td>\n",
              "      <td>-0.012431</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Reading Ease Level</th>\n",
              "      <td>71.622888</td>\n",
              "      <td>72.997206</td>\n",
              "      <td>72.533265</td>\n",
              "      <td>0.024926</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pseudo_perplexity</th>\n",
              "      <td>356.681890</td>\n",
              "      <td>144.490090</td>\n",
              "      <td>139.707477</td>\n",
              "      <td>-0.124918</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e30f4cae-6ef4-406d-8bcd-653888b37c4b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e30f4cae-6ef4-406d-8bcd-653888b37c4b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e30f4cae-6ef4-406d-8bcd-653888b37c4b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "add_perplexity_to_table3(processed_corpus_jll, scores_jll)"
      ],
      "metadata": {
        "id": "D7bDcG35wlkJ",
        "outputId": "553548e3-377b-409b-98e4-2c5d6dd524c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 348
        }
      },
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Mean values                               cycle4_3e   cycle4_4e   cycle4_5e  \\\n",
              "The Gunning fog index GFI                104.065761  102.421179  132.388006   \n",
              "The Automated readability index ARI       34.358482   36.119010   40.754610   \n",
              "The Flesch reading ease FRE               73.376908   76.957310   57.766046   \n",
              "The Flesch-Kincaid grade level FKGL        7.147536    6.922545    9.912481   \n",
              "The Simple Measure of Gobbledygook SMOG   21.718341   22.137552   24.655992   \n",
              "Reading Ease Level                        88.704426   91.673517   74.811220   \n",
              "Pseudo_perplexity                        158.024309  155.570485   99.319846   \n",
              "\n",
              "Mean values                               cycle3_6e  Pearson Score  \n",
              "The Gunning fog index GFI                119.822886       0.111777  \n",
              "The Automated readability index ARI       46.380268       0.195434  \n",
              "The Flesch reading ease FRE               75.527410      -0.050773  \n",
              "The Flesch-Kincaid grade level FKGL        8.215760       0.138531  \n",
              "The Simple Measure of Gobbledygook SMOG   23.986362       0.175971  \n",
              "Reading Ease Level                        89.848486      -0.062848  \n",
              "Pseudo_perplexity                        160.707202      -0.034147  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e0328c91-da54-4a66-81b9-9275d653f8f4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Mean values</th>\n",
              "      <th>cycle4_3e</th>\n",
              "      <th>cycle4_4e</th>\n",
              "      <th>cycle4_5e</th>\n",
              "      <th>cycle3_6e</th>\n",
              "      <th>Pearson Score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>The Gunning fog index GFI</th>\n",
              "      <td>104.065761</td>\n",
              "      <td>102.421179</td>\n",
              "      <td>132.388006</td>\n",
              "      <td>119.822886</td>\n",
              "      <td>0.111777</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Automated readability index ARI</th>\n",
              "      <td>34.358482</td>\n",
              "      <td>36.119010</td>\n",
              "      <td>40.754610</td>\n",
              "      <td>46.380268</td>\n",
              "      <td>0.195434</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch reading ease FRE</th>\n",
              "      <td>73.376908</td>\n",
              "      <td>76.957310</td>\n",
              "      <td>57.766046</td>\n",
              "      <td>75.527410</td>\n",
              "      <td>-0.050773</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Flesch-Kincaid grade level FKGL</th>\n",
              "      <td>7.147536</td>\n",
              "      <td>6.922545</td>\n",
              "      <td>9.912481</td>\n",
              "      <td>8.215760</td>\n",
              "      <td>0.138531</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>The Simple Measure of Gobbledygook SMOG</th>\n",
              "      <td>21.718341</td>\n",
              "      <td>22.137552</td>\n",
              "      <td>24.655992</td>\n",
              "      <td>23.986362</td>\n",
              "      <td>0.175971</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Reading Ease Level</th>\n",
              "      <td>88.704426</td>\n",
              "      <td>91.673517</td>\n",
              "      <td>74.811220</td>\n",
              "      <td>89.848486</td>\n",
              "      <td>-0.062848</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pseudo_perplexity</th>\n",
              "      <td>158.024309</td>\n",
              "      <td>155.570485</td>\n",
              "      <td>99.319846</td>\n",
              "      <td>160.707202</td>\n",
              "      <td>-0.034147</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e0328c91-da54-4a66-81b9-9275d653f8f4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e0328c91-da54-4a66-81b9-9275d653f8f4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e0328c91-da54-4a66-81b9-9275d653f8f4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Reproducing the contents of table 4 for MLP and SVM"
      ],
      "metadata": {
        "id": "qbko_k2ZNKH3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This should take around 50 minutes to compute on Colab."
      ],
      "metadata": {
        "id": "5eY5WVYGqZLP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from readability.methods import methods"
      ],
      "metadata": {
        "id": "7qM-v7vhNJAq"
      },
      "execution_count": 80,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "methods.demo_doMethods(corpus_ljl,plot=False)"
      ],
      "metadata": {
        "id": "q7-_trjtbo1A",
        "outputId": "00cca929-6de7-4bf2-eb82-da69bf613208",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MLP RESULTS\n",
            "cross-validation result for 5 runs = 0.479126213592233\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.45      0.46      0.45       240\n",
            "      level2       0.47      0.63      0.54       628\n",
            "      level3       0.47      0.47      0.47       670\n",
            "      level4       0.54      0.32      0.40       522\n",
            "\n",
            "    accuracy                           0.48      2060\n",
            "   macro avg       0.48      0.47      0.47      2060\n",
            "weighted avg       0.49      0.48      0.47      2060\n",
            "\n",
            "SVM RESULTS\n",
            "cross-validation result for 5 runs = 0.4757281553398058\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.46      0.42      0.44       240\n",
            "      level2       0.46      0.60      0.52       628\n",
            "      level3       0.47      0.49      0.48       670\n",
            "      level4       0.53      0.33      0.41       522\n",
            "\n",
            "    accuracy                           0.48      2060\n",
            "   macro avg       0.48      0.46      0.46      2060\n",
            "weighted avg       0.48      0.48      0.47      2060\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "methods.demo_doMethods(corpus_bb,plot=False)"
      ],
      "metadata": {
        "id": "-H2e8lqOe0iS",
        "outputId": "3d466876-f38d-4fa3-dce0-5fa46260934e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 82,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MLP RESULTS\n",
            "cross-validation result for 5 runs = 0.4977301387137453\n",
            "                precision    recall  f1-score   support\n",
            "\n",
            "intermédiaire       0.52      0.60      0.56      1729\n",
            "      avancée       0.51      0.48      0.50      1253\n",
            "        aisée       0.43      0.33      0.37       986\n",
            "\n",
            "      accuracy                           0.50      3968\n",
            "     macro avg       0.48      0.47      0.48      3968\n",
            "  weighted avg       0.49      0.50      0.49      3968\n",
            "\n",
            "SVM RESULTS\n",
            "cross-validation result for 5 runs = 0.5176462180095991\n",
            "                precision    recall  f1-score   support\n",
            "\n",
            "intermédiaire       0.52      0.66      0.59      1729\n",
            "      avancée       0.55      0.46      0.50      1253\n",
            "        aisée       0.46      0.33      0.39       986\n",
            "\n",
            "      accuracy                           0.52      3968\n",
            "     macro avg       0.51      0.49      0.49      3968\n",
            "  weighted avg       0.51      0.52      0.51      3968\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "methods.demo_doMethods(corpus_jll,plot=False)"
      ],
      "metadata": {
        "id": "DD3NzSyzkY44",
        "outputId": "20d9d414-0588-4584-e12f-9225bb4b36e4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MLP RESULTS\n",
            "cross-validation result for 5 runs = 0.604949381327334\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "   cycle4_3e       0.56      0.58      0.57       986\n",
            "   cycle4_4e       0.41      0.38      0.39       989\n",
            "   cycle4_5e       0.80      0.61      0.69      1187\n",
            "   cycle3_6e       0.64      0.79      0.71      1283\n",
            "\n",
            "    accuracy                           0.60      4445\n",
            "   macro avg       0.60      0.59      0.59      4445\n",
            "weighted avg       0.61      0.60      0.60      4445\n",
            "\n",
            "SVM RESULTS\n",
            "cross-validation result for 5 runs = 0.5739032620922384\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "   cycle4_3e       0.52      0.48      0.50       986\n",
            "   cycle4_4e       0.42      0.31      0.36       989\n",
            "   cycle4_5e       0.75      0.61      0.67      1187\n",
            "   cycle3_6e       0.57      0.81      0.67      1283\n",
            "\n",
            "    accuracy                           0.57      4445\n",
            "   macro avg       0.56      0.55      0.55      4445\n",
            "weighted avg       0.57      0.57      0.56      4445\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## How to reproduce the results in table 4 for fastText and CamemBERT"
      ],
      "metadata": {
        "id": "EJ3NnlP0yybH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from readability.models import models, fasttext, bert"
      ],
      "metadata": {
        "id": "rbQjwXU4ZuwM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following demonstration uses the csv files available in the data/ folder, encoded in one-hot vector format.  \n",
        "It relies on the ktrain library (wrapping around Keras) to help configure and train models for deep learning use.  \n",
        "Please enable the GPU to make these much faster :  \n",
        "Edit -> Notebook Settings -> Hardware Accelerator : GPU"
      ],
      "metadata": {
        "id": "ZbrwZaRfLtjf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###fastText"
      ],
      "metadata": {
        "id": "dsbPH6vmLpBh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fasttext.demo_doFastText(\"ljl\") #Can pass \"ljl\", \"bibebook.com\", \"JeLisLibre\", or \"all\" as a parameter\n",
        "# Takes around 15 minutes without GPU for the ljl corpus (default parameter) on free colab\n",
        "# Takes around 3 minute with GPU enabled."
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F1Lqu_ZOv2ON",
        "outputId": "35397bf8-bcb2-450a-cd3d-ce7008f2b56e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "detected encoding: utf-8 (if wrong, set manually)\n",
            "['level1', 'level2', 'level3', 'level4']\n",
            "      level1  level2  level3  level4\n",
            "1859       0       0       0       1\n",
            "697        0       1       0       0\n",
            "1691       0       0       0       1\n",
            "41         1       0       0       0\n",
            "62         1       0       0       0\n",
            "['level1', 'level2', 'level3', 'level4']\n",
            "      level1  level2  level3  level4\n",
            "1247       0       0       1       0\n",
            "99         1       0       0       0\n",
            "1215       0       0       1       0\n",
            "1604       0       0       0       1\n",
            "1566       0       0       0       1\n",
            "language: fr\n",
            "Word Counts: 19507\n",
            "Nrows: 1854\n",
            "1854 train sequences\n",
            "train sequence lengths:\n",
            "\tmean : 166\n",
            "\t95percentile : 368\n",
            "\t99percentile : 525\n",
            "x_train shape: (1854,150)\n",
            "y_train shape: (1854, 4)\n",
            "Is Multi-Label? False\n",
            "206 test sequences\n",
            "test sequence lengths:\n",
            "\tmean : 156\n",
            "\t95percentile : 330\n",
            "\t99percentile : 409\n",
            "x_test shape: (206,150)\n",
            "y_test shape: (206, 4)\n",
            "Is Multi-Label? False\n",
            "compiling word ID features...\n",
            "maxlen is 150\n",
            "done.\n",
            "-------------------------------------------------------run 0\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 1.7986 - accuracy: 0.2864 - val_loss: 1.3827 - val_accuracy: 0.2476\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.8131 - accuracy: 0.2686 - val_loss: 1.3816 - val_accuracy: 0.3010\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.8087 - accuracy: 0.2605 - val_loss: 1.3809 - val_accuracy: 0.3058\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.7519 - accuracy: 0.2918 - val_loss: 1.3803 - val_accuracy: 0.3107\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.7348 - accuracy: 0.2945 - val_loss: 1.3792 - val_accuracy: 0.3010\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.7495 - accuracy: 0.2799 - val_loss: 1.3782 - val_accuracy: 0.3107\n",
            "Epoch 7/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.7180 - accuracy: 0.2923 - val_loss: 1.3764 - val_accuracy: 0.3204\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6938 - accuracy: 0.2934 - val_loss: 1.3746 - val_accuracy: 0.3204\n",
            "Epoch 9/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6653 - accuracy: 0.2950 - val_loss: 1.3710 - val_accuracy: 0.3350\n",
            "Epoch 10/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6826 - accuracy: 0.2994 - val_loss: 1.3672 - val_accuracy: 0.3544\n",
            "Epoch 11/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6444 - accuracy: 0.3252 - val_loss: 1.3629 - val_accuracy: 0.3641\n",
            "Epoch 12/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6496 - accuracy: 0.3053 - val_loss: 1.3588 - val_accuracy: 0.3398\n",
            "Epoch 13/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6220 - accuracy: 0.3042 - val_loss: 1.3548 - val_accuracy: 0.3641\n",
            "Epoch 14/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6106 - accuracy: 0.3301 - val_loss: 1.3512 - val_accuracy: 0.3592\n",
            "Epoch 15/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5568 - accuracy: 0.3231 - val_loss: 1.3479 - val_accuracy: 0.3641\n",
            "Epoch 16/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5863 - accuracy: 0.3026 - val_loss: 1.3454 - val_accuracy: 0.3592\n",
            "Epoch 17/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5364 - accuracy: 0.3328 - val_loss: 1.3433 - val_accuracy: 0.3786\n",
            "Epoch 18/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5916 - accuracy: 0.3004 - val_loss: 1.3415 - val_accuracy: 0.3738\n",
            "Epoch 19/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5323 - accuracy: 0.3501 - val_loss: 1.3401 - val_accuracy: 0.3641\n",
            "Epoch 20/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5111 - accuracy: 0.3501 - val_loss: 1.3389 - val_accuracy: 0.3883\n",
            "Epoch 21/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5184 - accuracy: 0.3457 - val_loss: 1.3368 - val_accuracy: 0.3932\n",
            "Epoch 22/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4892 - accuracy: 0.3544 - val_loss: 1.3368 - val_accuracy: 0.3932\n",
            "Epoch 23/1024\n",
            "51/58 [=========================>....] - ETA: 0s - loss: 1.5367 - accuracy: 0.3358\n",
            "Epoch 00023: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5289 - accuracy: 0.3371 - val_loss: 1.3369 - val_accuracy: 0.3981\n",
            "Epoch 24/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5218 - accuracy: 0.3296 - val_loss: 1.3355 - val_accuracy: 0.3932\n",
            "Epoch 25/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5109 - accuracy: 0.3576 - val_loss: 1.3354 - val_accuracy: 0.4078\n",
            "Epoch 26/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4917 - accuracy: 0.3538 - val_loss: 1.3340 - val_accuracy: 0.4029\n",
            "Epoch 27/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4844 - accuracy: 0.3549 - val_loss: 1.3329 - val_accuracy: 0.4126\n",
            "Epoch 28/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4949 - accuracy: 0.3490 - val_loss: 1.3328 - val_accuracy: 0.4126\n",
            "Epoch 29/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4618 - accuracy: 0.3592 - val_loss: 1.3332 - val_accuracy: 0.3883\n",
            "Epoch 30/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4687 - accuracy: 0.3554 - val_loss: 1.3318 - val_accuracy: 0.4078\n",
            "Epoch 31/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4692 - accuracy: 0.3603 - val_loss: 1.3320 - val_accuracy: 0.3981\n",
            "Epoch 32/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5077 - accuracy: 0.3376 - val_loss: 1.3303 - val_accuracy: 0.3932\n",
            "Epoch 33/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4713 - accuracy: 0.3463 - val_loss: 1.3299 - val_accuracy: 0.3883\n",
            "Epoch 34/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4648 - accuracy: 0.3673 - val_loss: 1.3285 - val_accuracy: 0.3786\n",
            "Epoch 35/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4527 - accuracy: 0.3522 - val_loss: 1.3284 - val_accuracy: 0.3786\n",
            "Epoch 36/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4396 - accuracy: 0.3592 - val_loss: 1.3288 - val_accuracy: 0.3932\n",
            "Epoch 37/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4455 - accuracy: 0.3679 - val_loss: 1.3275 - val_accuracy: 0.4029\n",
            "Epoch 38/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4195 - accuracy: 0.3706 - val_loss: 1.3263 - val_accuracy: 0.3981\n",
            "Epoch 39/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4315 - accuracy: 0.3641 - val_loss: 1.3259 - val_accuracy: 0.4078\n",
            "Epoch 40/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4308 - accuracy: 0.3625 - val_loss: 1.3257 - val_accuracy: 0.4126\n",
            "Epoch 41/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4121 - accuracy: 0.3754 - val_loss: 1.3252 - val_accuracy: 0.4126\n",
            "Epoch 42/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4205 - accuracy: 0.3883 - val_loss: 1.3241 - val_accuracy: 0.3981\n",
            "Epoch 43/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3978 - accuracy: 0.3797 - val_loss: 1.3230 - val_accuracy: 0.4223\n",
            "Epoch 44/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4317 - accuracy: 0.3603 - val_loss: 1.3215 - val_accuracy: 0.4029\n",
            "Epoch 45/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3870 - accuracy: 0.3803 - val_loss: 1.3204 - val_accuracy: 0.4126\n",
            "Epoch 46/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3727 - accuracy: 0.3997 - val_loss: 1.3202 - val_accuracy: 0.4078\n",
            "Epoch 47/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4038 - accuracy: 0.3759 - val_loss: 1.3179 - val_accuracy: 0.4175\n",
            "Epoch 48/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4188 - accuracy: 0.3646 - val_loss: 1.3169 - val_accuracy: 0.4272\n",
            "Epoch 49/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3835 - accuracy: 0.3857 - val_loss: 1.3162 - val_accuracy: 0.4272\n",
            "Epoch 50/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3899 - accuracy: 0.3706 - val_loss: 1.3150 - val_accuracy: 0.4369\n",
            "Epoch 51/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3424 - accuracy: 0.4051 - val_loss: 1.3142 - val_accuracy: 0.4175\n",
            "Epoch 52/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3985 - accuracy: 0.3894 - val_loss: 1.3132 - val_accuracy: 0.4175\n",
            "Epoch 53/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3680 - accuracy: 0.3937 - val_loss: 1.3130 - val_accuracy: 0.4272\n",
            "Epoch 54/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3730 - accuracy: 0.3948 - val_loss: 1.3130 - val_accuracy: 0.4320\n",
            "Epoch 55/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3726 - accuracy: 0.4072 - val_loss: 1.3120 - val_accuracy: 0.4223\n",
            "Epoch 56/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3666 - accuracy: 0.4072 - val_loss: 1.3096 - val_accuracy: 0.4223\n",
            "Epoch 57/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3557 - accuracy: 0.4083 - val_loss: 1.3099 - val_accuracy: 0.4272\n",
            "Epoch 58/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3519 - accuracy: 0.4186 - val_loss: 1.3078 - val_accuracy: 0.4320\n",
            "Epoch 59/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3164 - accuracy: 0.4099 - val_loss: 1.3068 - val_accuracy: 0.4223\n",
            "Epoch 60/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3493 - accuracy: 0.4035 - val_loss: 1.3048 - val_accuracy: 0.4369\n",
            "Epoch 61/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3415 - accuracy: 0.4121 - val_loss: 1.3040 - val_accuracy: 0.4417\n",
            "Epoch 62/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3179 - accuracy: 0.4002 - val_loss: 1.3047 - val_accuracy: 0.4515\n",
            "Epoch 63/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3367 - accuracy: 0.4056 - val_loss: 1.3021 - val_accuracy: 0.4417\n",
            "Epoch 64/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3282 - accuracy: 0.4110 - val_loss: 1.3011 - val_accuracy: 0.4466\n",
            "Epoch 65/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2948 - accuracy: 0.4180 - val_loss: 1.3009 - val_accuracy: 0.4417\n",
            "Epoch 66/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3290 - accuracy: 0.4153 - val_loss: 1.2984 - val_accuracy: 0.4515\n",
            "Epoch 67/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3300 - accuracy: 0.4029 - val_loss: 1.2981 - val_accuracy: 0.4417\n",
            "Epoch 68/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2989 - accuracy: 0.4234 - val_loss: 1.2964 - val_accuracy: 0.4466\n",
            "Epoch 69/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2919 - accuracy: 0.4342 - val_loss: 1.2959 - val_accuracy: 0.4515\n",
            "Epoch 70/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3182 - accuracy: 0.4277 - val_loss: 1.2950 - val_accuracy: 0.4563\n",
            "Epoch 71/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2718 - accuracy: 0.4461 - val_loss: 1.2945 - val_accuracy: 0.4417\n",
            "Epoch 72/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2947 - accuracy: 0.4288 - val_loss: 1.2932 - val_accuracy: 0.4369\n",
            "Epoch 73/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3028 - accuracy: 0.4326 - val_loss: 1.2917 - val_accuracy: 0.4515\n",
            "Epoch 74/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2705 - accuracy: 0.4574 - val_loss: 1.2907 - val_accuracy: 0.4563\n",
            "Epoch 75/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3083 - accuracy: 0.4153 - val_loss: 1.2901 - val_accuracy: 0.4515\n",
            "Epoch 76/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2570 - accuracy: 0.4391 - val_loss: 1.2884 - val_accuracy: 0.4515\n",
            "Epoch 77/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2903 - accuracy: 0.4266 - val_loss: 1.2872 - val_accuracy: 0.4563\n",
            "Epoch 78/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2665 - accuracy: 0.4396 - val_loss: 1.2846 - val_accuracy: 0.4660\n",
            "Epoch 79/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2467 - accuracy: 0.4525 - val_loss: 1.2835 - val_accuracy: 0.4563\n",
            "Epoch 80/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2262 - accuracy: 0.4622 - val_loss: 1.2818 - val_accuracy: 0.4466\n",
            "Epoch 81/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2760 - accuracy: 0.4385 - val_loss: 1.2817 - val_accuracy: 0.4563\n",
            "Epoch 82/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2675 - accuracy: 0.4326 - val_loss: 1.2816 - val_accuracy: 0.4612\n",
            "Epoch 83/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2545 - accuracy: 0.4504 - val_loss: 1.2793 - val_accuracy: 0.4515\n",
            "Epoch 84/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2234 - accuracy: 0.4720 - val_loss: 1.2777 - val_accuracy: 0.4466\n",
            "Epoch 85/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2476 - accuracy: 0.4536 - val_loss: 1.2770 - val_accuracy: 0.4466\n",
            "Epoch 86/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2269 - accuracy: 0.4633 - val_loss: 1.2761 - val_accuracy: 0.4612\n",
            "Epoch 87/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2258 - accuracy: 0.4622 - val_loss: 1.2751 - val_accuracy: 0.4612\n",
            "Epoch 88/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2153 - accuracy: 0.4779 - val_loss: 1.2736 - val_accuracy: 0.4563\n",
            "Epoch 89/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2215 - accuracy: 0.4773 - val_loss: 1.2737 - val_accuracy: 0.4515\n",
            "Epoch 90/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1825 - accuracy: 0.4930 - val_loss: 1.2710 - val_accuracy: 0.4612\n",
            "Epoch 91/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2260 - accuracy: 0.4693 - val_loss: 1.2701 - val_accuracy: 0.4709\n",
            "Epoch 92/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1872 - accuracy: 0.4671 - val_loss: 1.2688 - val_accuracy: 0.4709\n",
            "Epoch 93/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2117 - accuracy: 0.4617 - val_loss: 1.2672 - val_accuracy: 0.4709\n",
            "Epoch 94/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1782 - accuracy: 0.4800 - val_loss: 1.2650 - val_accuracy: 0.4660\n",
            "Epoch 95/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2099 - accuracy: 0.4795 - val_loss: 1.2631 - val_accuracy: 0.4660\n",
            "Epoch 96/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1694 - accuracy: 0.4898 - val_loss: 1.2620 - val_accuracy: 0.4660\n",
            "Epoch 97/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1862 - accuracy: 0.4741 - val_loss: 1.2603 - val_accuracy: 0.4757\n",
            "Epoch 98/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1596 - accuracy: 0.5054 - val_loss: 1.2576 - val_accuracy: 0.4806\n",
            "Epoch 99/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1457 - accuracy: 0.5065 - val_loss: 1.2552 - val_accuracy: 0.4903\n",
            "Epoch 100/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1746 - accuracy: 0.4887 - val_loss: 1.2543 - val_accuracy: 0.4757\n",
            "Epoch 101/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1782 - accuracy: 0.4752 - val_loss: 1.2521 - val_accuracy: 0.4854\n",
            "Epoch 102/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1481 - accuracy: 0.5032 - val_loss: 1.2505 - val_accuracy: 0.4854\n",
            "Epoch 103/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1299 - accuracy: 0.5183 - val_loss: 1.2488 - val_accuracy: 0.4854\n",
            "Epoch 104/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1321 - accuracy: 0.5232 - val_loss: 1.2473 - val_accuracy: 0.5000\n",
            "Epoch 105/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1504 - accuracy: 0.5108 - val_loss: 1.2456 - val_accuracy: 0.5000\n",
            "Epoch 106/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1469 - accuracy: 0.5129 - val_loss: 1.2443 - val_accuracy: 0.4903\n",
            "Epoch 107/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1299 - accuracy: 0.5156 - val_loss: 1.2421 - val_accuracy: 0.4854\n",
            "Epoch 108/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1024 - accuracy: 0.5448 - val_loss: 1.2412 - val_accuracy: 0.5049\n",
            "Epoch 109/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1335 - accuracy: 0.5076 - val_loss: 1.2389 - val_accuracy: 0.5049\n",
            "Epoch 110/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1096 - accuracy: 0.5167 - val_loss: 1.2359 - val_accuracy: 0.5097\n",
            "Epoch 111/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1018 - accuracy: 0.5356 - val_loss: 1.2355 - val_accuracy: 0.4951\n",
            "Epoch 112/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1229 - accuracy: 0.5254 - val_loss: 1.2343 - val_accuracy: 0.5000\n",
            "Epoch 113/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1021 - accuracy: 0.5318 - val_loss: 1.2324 - val_accuracy: 0.4951\n",
            "Epoch 114/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0838 - accuracy: 0.5415 - val_loss: 1.2329 - val_accuracy: 0.4951\n",
            "Epoch 115/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1188 - accuracy: 0.5178 - val_loss: 1.2313 - val_accuracy: 0.5049\n",
            "Epoch 116/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0755 - accuracy: 0.5458 - val_loss: 1.2295 - val_accuracy: 0.5000\n",
            "Epoch 117/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0656 - accuracy: 0.5469 - val_loss: 1.2278 - val_accuracy: 0.5049\n",
            "Epoch 118/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0959 - accuracy: 0.5394 - val_loss: 1.2258 - val_accuracy: 0.5000\n",
            "Epoch 119/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1164 - accuracy: 0.5173 - val_loss: 1.2254 - val_accuracy: 0.4951\n",
            "Epoch 120/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0575 - accuracy: 0.5496 - val_loss: 1.2217 - val_accuracy: 0.4951\n",
            "Epoch 121/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0476 - accuracy: 0.5680 - val_loss: 1.2203 - val_accuracy: 0.4951\n",
            "Epoch 122/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0568 - accuracy: 0.5620 - val_loss: 1.2174 - val_accuracy: 0.5097\n",
            "Epoch 123/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0625 - accuracy: 0.5588 - val_loss: 1.2157 - val_accuracy: 0.5097\n",
            "Epoch 124/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0582 - accuracy: 0.5512 - val_loss: 1.2125 - val_accuracy: 0.5291\n",
            "Epoch 125/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0234 - accuracy: 0.5674 - val_loss: 1.2111 - val_accuracy: 0.5146\n",
            "Epoch 126/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0485 - accuracy: 0.5642 - val_loss: 1.2110 - val_accuracy: 0.5097\n",
            "Epoch 127/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0476 - accuracy: 0.5739 - val_loss: 1.2075 - val_accuracy: 0.5097\n",
            "Epoch 128/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0661 - accuracy: 0.5588 - val_loss: 1.2066 - val_accuracy: 0.5049\n",
            "Epoch 129/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0068 - accuracy: 0.5831 - val_loss: 1.2039 - val_accuracy: 0.5097\n",
            "Epoch 130/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0252 - accuracy: 0.5804 - val_loss: 1.2009 - val_accuracy: 0.5146\n",
            "Epoch 131/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0207 - accuracy: 0.5631 - val_loss: 1.1987 - val_accuracy: 0.5340\n",
            "Epoch 132/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0167 - accuracy: 0.5669 - val_loss: 1.1961 - val_accuracy: 0.5291\n",
            "Epoch 133/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0187 - accuracy: 0.5847 - val_loss: 1.1935 - val_accuracy: 0.5340\n",
            "Epoch 134/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.9915 - accuracy: 0.5852 - val_loss: 1.1941 - val_accuracy: 0.5291\n",
            "Epoch 135/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0051 - accuracy: 0.5874 - val_loss: 1.1925 - val_accuracy: 0.5340\n",
            "Epoch 136/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9767 - accuracy: 0.6187 - val_loss: 1.1906 - val_accuracy: 0.5340\n",
            "Epoch 137/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9831 - accuracy: 0.6019 - val_loss: 1.1888 - val_accuracy: 0.5243\n",
            "Epoch 138/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0118 - accuracy: 0.5885 - val_loss: 1.1865 - val_accuracy: 0.5291\n",
            "Epoch 139/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.9982 - accuracy: 0.5766 - val_loss: 1.1850 - val_accuracy: 0.5194\n",
            "Epoch 140/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9726 - accuracy: 0.5895 - val_loss: 1.1833 - val_accuracy: 0.5340\n",
            "Epoch 141/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.9772 - accuracy: 0.5960 - val_loss: 1.1822 - val_accuracy: 0.5291\n",
            "Epoch 142/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9819 - accuracy: 0.5922 - val_loss: 1.1790 - val_accuracy: 0.5388\n",
            "Epoch 143/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.9441 - accuracy: 0.6214 - val_loss: 1.1767 - val_accuracy: 0.5388\n",
            "Epoch 144/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9498 - accuracy: 0.6106 - val_loss: 1.1746 - val_accuracy: 0.5388\n",
            "Epoch 145/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9536 - accuracy: 0.6143 - val_loss: 1.1740 - val_accuracy: 0.5437\n",
            "Epoch 146/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9388 - accuracy: 0.6251 - val_loss: 1.1730 - val_accuracy: 0.5388\n",
            "Epoch 147/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.9366 - accuracy: 0.6160 - val_loss: 1.1713 - val_accuracy: 0.5340\n",
            "Epoch 148/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9188 - accuracy: 0.6257 - val_loss: 1.1691 - val_accuracy: 0.5437\n",
            "Epoch 149/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9374 - accuracy: 0.6052 - val_loss: 1.1680 - val_accuracy: 0.5388\n",
            "Epoch 150/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9187 - accuracy: 0.6365 - val_loss: 1.1648 - val_accuracy: 0.5340\n",
            "Epoch 151/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9356 - accuracy: 0.6170 - val_loss: 1.1613 - val_accuracy: 0.5340\n",
            "Epoch 152/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9388 - accuracy: 0.6192 - val_loss: 1.1603 - val_accuracy: 0.5437\n",
            "Epoch 153/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8885 - accuracy: 0.6456 - val_loss: 1.1599 - val_accuracy: 0.5388\n",
            "Epoch 154/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.8969 - accuracy: 0.6451 - val_loss: 1.1574 - val_accuracy: 0.5340\n",
            "Epoch 155/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.9313 - accuracy: 0.6176 - val_loss: 1.1546 - val_accuracy: 0.5340\n",
            "Epoch 156/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8840 - accuracy: 0.6499 - val_loss: 1.1527 - val_accuracy: 0.5340\n",
            "Epoch 157/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.8793 - accuracy: 0.6413 - val_loss: 1.1490 - val_accuracy: 0.5388\n",
            "Epoch 158/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8812 - accuracy: 0.6359 - val_loss: 1.1465 - val_accuracy: 0.5388\n",
            "Epoch 159/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9090 - accuracy: 0.6343 - val_loss: 1.1443 - val_accuracy: 0.5437\n",
            "Epoch 160/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8734 - accuracy: 0.6543 - val_loss: 1.1437 - val_accuracy: 0.5388\n",
            "Epoch 161/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8436 - accuracy: 0.6597 - val_loss: 1.1416 - val_accuracy: 0.5437\n",
            "Epoch 162/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8597 - accuracy: 0.6624 - val_loss: 1.1375 - val_accuracy: 0.5437\n",
            "Epoch 163/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.8433 - accuracy: 0.6753 - val_loss: 1.1366 - val_accuracy: 0.5437\n",
            "Epoch 164/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8702 - accuracy: 0.6446 - val_loss: 1.1342 - val_accuracy: 0.5437\n",
            "Epoch 165/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8626 - accuracy: 0.6656 - val_loss: 1.1308 - val_accuracy: 0.5485\n",
            "Epoch 166/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8538 - accuracy: 0.6591 - val_loss: 1.1288 - val_accuracy: 0.5485\n",
            "Epoch 167/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8361 - accuracy: 0.6715 - val_loss: 1.1280 - val_accuracy: 0.5437\n",
            "Epoch 168/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.8180 - accuracy: 0.6807 - val_loss: 1.1268 - val_accuracy: 0.5485\n",
            "Epoch 169/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8248 - accuracy: 0.6812 - val_loss: 1.1249 - val_accuracy: 0.5485\n",
            "Epoch 170/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8057 - accuracy: 0.6764 - val_loss: 1.1217 - val_accuracy: 0.5485\n",
            "Epoch 171/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8392 - accuracy: 0.6775 - val_loss: 1.1175 - val_accuracy: 0.5534\n",
            "Epoch 172/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8163 - accuracy: 0.6802 - val_loss: 1.1150 - val_accuracy: 0.5534\n",
            "Epoch 173/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8200 - accuracy: 0.6742 - val_loss: 1.1134 - val_accuracy: 0.5485\n",
            "Epoch 174/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8079 - accuracy: 0.6969 - val_loss: 1.1108 - val_accuracy: 0.5485\n",
            "Epoch 175/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8078 - accuracy: 0.6796 - val_loss: 1.1094 - val_accuracy: 0.5583\n",
            "Epoch 176/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7946 - accuracy: 0.6953 - val_loss: 1.1074 - val_accuracy: 0.5583\n",
            "Epoch 177/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7852 - accuracy: 0.6963 - val_loss: 1.1068 - val_accuracy: 0.5534\n",
            "Epoch 178/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7917 - accuracy: 0.6990 - val_loss: 1.1053 - val_accuracy: 0.5631\n",
            "Epoch 179/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7444 - accuracy: 0.7184 - val_loss: 1.1021 - val_accuracy: 0.5485\n",
            "Epoch 180/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7667 - accuracy: 0.7093 - val_loss: 1.0981 - val_accuracy: 0.5534\n",
            "Epoch 181/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7763 - accuracy: 0.7001 - val_loss: 1.0966 - val_accuracy: 0.5728\n",
            "Epoch 182/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7563 - accuracy: 0.7077 - val_loss: 1.0941 - val_accuracy: 0.5680\n",
            "Epoch 183/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7823 - accuracy: 0.6958 - val_loss: 1.0914 - val_accuracy: 0.5680\n",
            "Epoch 184/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7164 - accuracy: 0.7249 - val_loss: 1.0902 - val_accuracy: 0.5631\n",
            "Epoch 185/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7544 - accuracy: 0.6974 - val_loss: 1.0883 - val_accuracy: 0.5631\n",
            "Epoch 186/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7257 - accuracy: 0.7179 - val_loss: 1.0879 - val_accuracy: 0.5631\n",
            "Epoch 187/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7389 - accuracy: 0.7179 - val_loss: 1.0850 - val_accuracy: 0.5583\n",
            "Epoch 188/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7249 - accuracy: 0.7179 - val_loss: 1.0829 - val_accuracy: 0.5631\n",
            "Epoch 189/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7668 - accuracy: 0.7023 - val_loss: 1.0808 - val_accuracy: 0.5631\n",
            "Epoch 190/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7205 - accuracy: 0.7255 - val_loss: 1.0790 - val_accuracy: 0.5583\n",
            "Epoch 191/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7239 - accuracy: 0.7082 - val_loss: 1.0766 - val_accuracy: 0.5631\n",
            "Epoch 192/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.7124 - accuracy: 0.7287 - val_loss: 1.0748 - val_accuracy: 0.5631\n",
            "Epoch 193/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7123 - accuracy: 0.7206 - val_loss: 1.0716 - val_accuracy: 0.5631\n",
            "Epoch 194/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.7005 - accuracy: 0.7335 - val_loss: 1.0703 - val_accuracy: 0.5631\n",
            "Epoch 195/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7030 - accuracy: 0.7287 - val_loss: 1.0690 - val_accuracy: 0.5777\n",
            "Epoch 196/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7170 - accuracy: 0.7319 - val_loss: 1.0673 - val_accuracy: 0.5777\n",
            "Epoch 197/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6882 - accuracy: 0.7362 - val_loss: 1.0650 - val_accuracy: 0.5728\n",
            "Epoch 198/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6966 - accuracy: 0.7384 - val_loss: 1.0640 - val_accuracy: 0.5777\n",
            "Epoch 199/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6919 - accuracy: 0.7395 - val_loss: 1.0603 - val_accuracy: 0.5777\n",
            "Epoch 200/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.7219 - accuracy: 0.7147 - val_loss: 1.0589 - val_accuracy: 0.5680\n",
            "Epoch 201/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6996 - accuracy: 0.7335 - val_loss: 1.0556 - val_accuracy: 0.5777\n",
            "Epoch 202/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6557 - accuracy: 0.7546 - val_loss: 1.0538 - val_accuracy: 0.5874\n",
            "Epoch 203/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6720 - accuracy: 0.7470 - val_loss: 1.0553 - val_accuracy: 0.5874\n",
            "Epoch 204/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6371 - accuracy: 0.7611 - val_loss: 1.0536 - val_accuracy: 0.5825\n",
            "Epoch 205/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6557 - accuracy: 0.7492 - val_loss: 1.0505 - val_accuracy: 0.5825\n",
            "Epoch 206/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6576 - accuracy: 0.7535 - val_loss: 1.0496 - val_accuracy: 0.5874\n",
            "Epoch 207/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6621 - accuracy: 0.7535 - val_loss: 1.0478 - val_accuracy: 0.5825\n",
            "Epoch 208/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6589 - accuracy: 0.7562 - val_loss: 1.0461 - val_accuracy: 0.5825\n",
            "Epoch 209/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6327 - accuracy: 0.7638 - val_loss: 1.0444 - val_accuracy: 0.5922\n",
            "Epoch 210/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6670 - accuracy: 0.7546 - val_loss: 1.0437 - val_accuracy: 0.6019\n",
            "Epoch 211/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6365 - accuracy: 0.7578 - val_loss: 1.0418 - val_accuracy: 0.5971\n",
            "Epoch 212/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6437 - accuracy: 0.7611 - val_loss: 1.0406 - val_accuracy: 0.5922\n",
            "Epoch 213/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6145 - accuracy: 0.7729 - val_loss: 1.0371 - val_accuracy: 0.5874\n",
            "Epoch 214/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6327 - accuracy: 0.7638 - val_loss: 1.0359 - val_accuracy: 0.5971\n",
            "Epoch 215/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6152 - accuracy: 0.7519 - val_loss: 1.0357 - val_accuracy: 0.5971\n",
            "Epoch 216/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5724 - accuracy: 0.7875 - val_loss: 1.0348 - val_accuracy: 0.6019\n",
            "Epoch 217/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6008 - accuracy: 0.7875 - val_loss: 1.0325 - val_accuracy: 0.5874\n",
            "Epoch 218/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6102 - accuracy: 0.7745 - val_loss: 1.0304 - val_accuracy: 0.5922\n",
            "Epoch 219/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6074 - accuracy: 0.7767 - val_loss: 1.0285 - val_accuracy: 0.5971\n",
            "Epoch 220/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5909 - accuracy: 0.7799 - val_loss: 1.0271 - val_accuracy: 0.5971\n",
            "Epoch 221/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6274 - accuracy: 0.7621 - val_loss: 1.0254 - val_accuracy: 0.6019\n",
            "Epoch 222/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5811 - accuracy: 0.7789 - val_loss: 1.0233 - val_accuracy: 0.6019\n",
            "Epoch 223/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5812 - accuracy: 0.7853 - val_loss: 1.0211 - val_accuracy: 0.6019\n",
            "Epoch 224/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5749 - accuracy: 0.7816 - val_loss: 1.0210 - val_accuracy: 0.6068\n",
            "Epoch 225/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5898 - accuracy: 0.7886 - val_loss: 1.0194 - val_accuracy: 0.6019\n",
            "Epoch 226/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5680 - accuracy: 0.7940 - val_loss: 1.0186 - val_accuracy: 0.6019\n",
            "Epoch 227/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5558 - accuracy: 0.8010 - val_loss: 1.0166 - val_accuracy: 0.6068\n",
            "Epoch 228/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5823 - accuracy: 0.7665 - val_loss: 1.0149 - val_accuracy: 0.5971\n",
            "Epoch 229/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5478 - accuracy: 0.8074 - val_loss: 1.0117 - val_accuracy: 0.5971\n",
            "Epoch 230/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5779 - accuracy: 0.7896 - val_loss: 1.0102 - val_accuracy: 0.5922\n",
            "Epoch 231/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5429 - accuracy: 0.8031 - val_loss: 1.0101 - val_accuracy: 0.5971\n",
            "Epoch 232/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5466 - accuracy: 0.8042 - val_loss: 1.0096 - val_accuracy: 0.6019\n",
            "Epoch 233/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5424 - accuracy: 0.8020 - val_loss: 1.0091 - val_accuracy: 0.6068\n",
            "Epoch 234/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5359 - accuracy: 0.8015 - val_loss: 1.0073 - val_accuracy: 0.5971\n",
            "Epoch 235/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5477 - accuracy: 0.7972 - val_loss: 1.0051 - val_accuracy: 0.6068\n",
            "Epoch 236/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5558 - accuracy: 0.7896 - val_loss: 1.0042 - val_accuracy: 0.5922\n",
            "Epoch 237/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5234 - accuracy: 0.8112 - val_loss: 1.0031 - val_accuracy: 0.6068\n",
            "Epoch 238/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5422 - accuracy: 0.7891 - val_loss: 0.9983 - val_accuracy: 0.5922\n",
            "Epoch 239/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5371 - accuracy: 0.7956 - val_loss: 0.9991 - val_accuracy: 0.5971\n",
            "Epoch 240/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5042 - accuracy: 0.8177 - val_loss: 0.9977 - val_accuracy: 0.6019\n",
            "Epoch 241/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5191 - accuracy: 0.8112 - val_loss: 0.9966 - val_accuracy: 0.5971\n",
            "Epoch 242/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5218 - accuracy: 0.8101 - val_loss: 0.9967 - val_accuracy: 0.5922\n",
            "Epoch 243/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4972 - accuracy: 0.8139 - val_loss: 0.9940 - val_accuracy: 0.6068\n",
            "Epoch 244/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5091 - accuracy: 0.8215 - val_loss: 0.9928 - val_accuracy: 0.6019\n",
            "Epoch 245/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5002 - accuracy: 0.8182 - val_loss: 0.9923 - val_accuracy: 0.5971\n",
            "Epoch 246/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4879 - accuracy: 0.8252 - val_loss: 0.9916 - val_accuracy: 0.6019\n",
            "Epoch 247/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4867 - accuracy: 0.8333 - val_loss: 0.9891 - val_accuracy: 0.5922\n",
            "Epoch 248/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4750 - accuracy: 0.8290 - val_loss: 0.9877 - val_accuracy: 0.6117\n",
            "Epoch 249/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4830 - accuracy: 0.8296 - val_loss: 0.9858 - val_accuracy: 0.6117\n",
            "Epoch 250/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4808 - accuracy: 0.8209 - val_loss: 0.9861 - val_accuracy: 0.6214\n",
            "Epoch 251/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5017 - accuracy: 0.8096 - val_loss: 0.9849 - val_accuracy: 0.6068\n",
            "Epoch 252/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5076 - accuracy: 0.8112 - val_loss: 0.9828 - val_accuracy: 0.6068\n",
            "Epoch 253/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4781 - accuracy: 0.8193 - val_loss: 0.9814 - val_accuracy: 0.6019\n",
            "Epoch 254/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4698 - accuracy: 0.8269 - val_loss: 0.9804 - val_accuracy: 0.5971\n",
            "Epoch 255/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4749 - accuracy: 0.8279 - val_loss: 0.9787 - val_accuracy: 0.5971\n",
            "Epoch 256/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4600 - accuracy: 0.8403 - val_loss: 0.9768 - val_accuracy: 0.6068\n",
            "Epoch 257/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4597 - accuracy: 0.8420 - val_loss: 0.9756 - val_accuracy: 0.6068\n",
            "Epoch 258/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4726 - accuracy: 0.8360 - val_loss: 0.9753 - val_accuracy: 0.6117\n",
            "Epoch 259/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4468 - accuracy: 0.8274 - val_loss: 0.9750 - val_accuracy: 0.6117\n",
            "Epoch 260/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4613 - accuracy: 0.8252 - val_loss: 0.9740 - val_accuracy: 0.6165\n",
            "Epoch 261/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4407 - accuracy: 0.8382 - val_loss: 0.9739 - val_accuracy: 0.6214\n",
            "Epoch 262/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4454 - accuracy: 0.8506 - val_loss: 0.9732 - val_accuracy: 0.6214\n",
            "Epoch 263/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4390 - accuracy: 0.8403 - val_loss: 0.9713 - val_accuracy: 0.6214\n",
            "Epoch 264/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4313 - accuracy: 0.8495 - val_loss: 0.9723 - val_accuracy: 0.6068\n",
            "Epoch 265/1024\n",
            "55/58 [===========================>..] - ETA: 0s - loss: 0.4545 - accuracy: 0.8375\n",
            "Epoch 00265: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4570 - accuracy: 0.8350 - val_loss: 0.9714 - val_accuracy: 0.6068\n",
            "Epoch 266/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4307 - accuracy: 0.8425 - val_loss: 0.9696 - val_accuracy: 0.6117\n",
            "Epoch 267/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4311 - accuracy: 0.8549 - val_loss: 0.9696 - val_accuracy: 0.6214\n",
            "Epoch 268/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4344 - accuracy: 0.8447 - val_loss: 0.9693 - val_accuracy: 0.6165\n",
            "Epoch 269/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4558 - accuracy: 0.8398 - val_loss: 0.9680 - val_accuracy: 0.6165\n",
            "Epoch 270/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4389 - accuracy: 0.8376 - val_loss: 0.9678 - val_accuracy: 0.6214\n",
            "Epoch 271/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4606 - accuracy: 0.8328 - val_loss: 0.9680 - val_accuracy: 0.6262\n",
            "Epoch 272/1024\n",
            "53/58 [==========================>...] - ETA: 0s - loss: 0.4206 - accuracy: 0.8455\n",
            "Epoch 00272: Reducing Max LR on Plateau: new max lr will be 1.25e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4214 - accuracy: 0.8468 - val_loss: 0.9680 - val_accuracy: 0.6262\n",
            "Epoch 273/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4093 - accuracy: 0.8517 - val_loss: 0.9679 - val_accuracy: 0.6165\n",
            "Epoch 274/1024\n",
            "53/58 [==========================>...] - ETA: 0s - loss: 0.4120 - accuracy: 0.8614\n",
            "Epoch 00274: Reducing Max LR on Plateau: new max lr will be 6.25e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4125 - accuracy: 0.8581 - val_loss: 0.9679 - val_accuracy: 0.6117\n",
            "Epoch 275/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.3825 - accuracy: 0.8623Restoring model weights from the end of the best epoch: 270.\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3834 - accuracy: 0.8614 - val_loss: 0.9686 - val_accuracy: 0.6214\n",
            "Epoch 275: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "run 0 CORPUSNAME ljl class_names ['level1', 'level2', 'level3', 'level4']\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.44      0.33      0.38        24\n",
            "      level2       0.56      0.62      0.59        61\n",
            "      level3       0.70      0.69      0.69        70\n",
            "      level4       0.67      0.67      0.67        51\n",
            "\n",
            "    accuracy                           0.62       206\n",
            "   macro avg       0.59      0.58      0.58       206\n",
            "weighted avg       0.62      0.62      0.62       206\n",
            "\n",
            "-------------------------------------------------------run 1\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4402 - accuracy: 0.8393 - val_loss: 0.9684 - val_accuracy: 0.6165\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4117 - accuracy: 0.8447 - val_loss: 0.9659 - val_accuracy: 0.6214\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4082 - accuracy: 0.8598 - val_loss: 0.9638 - val_accuracy: 0.6165\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3843 - accuracy: 0.8668 - val_loss: 0.9620 - val_accuracy: 0.6214\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3924 - accuracy: 0.8495 - val_loss: 0.9607 - val_accuracy: 0.6165\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4083 - accuracy: 0.8533 - val_loss: 0.9579 - val_accuracy: 0.6165\n",
            "Epoch 7/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3996 - accuracy: 0.8479 - val_loss: 0.9567 - val_accuracy: 0.6359\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3793 - accuracy: 0.8727 - val_loss: 0.9552 - val_accuracy: 0.6359\n",
            "Epoch 9/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3949 - accuracy: 0.8608 - val_loss: 0.9540 - val_accuracy: 0.6262\n",
            "Epoch 10/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4169 - accuracy: 0.8538 - val_loss: 0.9535 - val_accuracy: 0.6262\n",
            "Epoch 11/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3707 - accuracy: 0.8662 - val_loss: 0.9526 - val_accuracy: 0.6359\n",
            "Epoch 12/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3519 - accuracy: 0.8776 - val_loss: 0.9503 - val_accuracy: 0.6262\n",
            "Epoch 13/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3655 - accuracy: 0.8738 - val_loss: 0.9487 - val_accuracy: 0.6262\n",
            "Epoch 14/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3627 - accuracy: 0.8727 - val_loss: 0.9474 - val_accuracy: 0.6214\n",
            "Epoch 15/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3777 - accuracy: 0.8673 - val_loss: 0.9473 - val_accuracy: 0.6068\n",
            "Epoch 16/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3476 - accuracy: 0.8770 - val_loss: 0.9471 - val_accuracy: 0.6068\n",
            "Epoch 17/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3427 - accuracy: 0.8862 - val_loss: 0.9469 - val_accuracy: 0.6068\n",
            "Epoch 18/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3362 - accuracy: 0.8857 - val_loss: 0.9469 - val_accuracy: 0.6068\n",
            "Epoch 19/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3384 - accuracy: 0.8813 - val_loss: 0.9459 - val_accuracy: 0.6117\n",
            "Epoch 20/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3290 - accuracy: 0.8813 - val_loss: 0.9452 - val_accuracy: 0.6165\n",
            "Epoch 21/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3540 - accuracy: 0.8819 - val_loss: 0.9449 - val_accuracy: 0.6165\n",
            "Epoch 22/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3296 - accuracy: 0.8819 - val_loss: 0.9464 - val_accuracy: 0.6165\n",
            "Epoch 23/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3219 - accuracy: 0.8959 - val_loss: 0.9446 - val_accuracy: 0.6068\n",
            "Epoch 24/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3192 - accuracy: 0.8910 - val_loss: 0.9445 - val_accuracy: 0.6165\n",
            "Epoch 25/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3224 - accuracy: 0.8873 - val_loss: 0.9435 - val_accuracy: 0.6165\n",
            "Epoch 26/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3282 - accuracy: 0.8867 - val_loss: 0.9413 - val_accuracy: 0.6165\n",
            "Epoch 27/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3078 - accuracy: 0.8959 - val_loss: 0.9418 - val_accuracy: 0.6165\n",
            "Epoch 28/1024\n",
            "58/58 [==============================] - ETA: 0s - loss: 0.3110 - accuracy: 0.8889\n",
            "Epoch 00028: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3110 - accuracy: 0.8889 - val_loss: 0.9418 - val_accuracy: 0.6214\n",
            "Epoch 29/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3267 - accuracy: 0.8867 - val_loss: 0.9425 - val_accuracy: 0.6117\n",
            "Epoch 30/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2944 - accuracy: 0.8986 - val_loss: 0.9408 - val_accuracy: 0.6165\n",
            "Epoch 31/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2836 - accuracy: 0.8970 - val_loss: 0.9410 - val_accuracy: 0.6165\n",
            "Epoch 32/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.2887 - accuracy: 0.9115\n",
            "Epoch 00032: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2849 - accuracy: 0.9126 - val_loss: 0.9409 - val_accuracy: 0.6214\n",
            "Epoch 33/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2978 - accuracy: 0.8900 - val_loss: 0.9414 - val_accuracy: 0.6214\n",
            "Epoch 34/1024\n",
            "53/58 [==========================>...] - ETA: 0s - loss: 0.2996 - accuracy: 0.8974\n",
            "Epoch 00034: Reducing Max LR on Plateau: new max lr will be 1.25e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2962 - accuracy: 0.9008 - val_loss: 0.9423 - val_accuracy: 0.6214\n",
            "Epoch 35/1024\n",
            "52/58 [=========================>....] - ETA: 0s - loss: 0.2823 - accuracy: 0.9008Restoring model weights from the end of the best epoch: 30.\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2832 - accuracy: 0.9002 - val_loss: 0.9412 - val_accuracy: 0.6262\n",
            "Epoch 35: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "run 1 CORPUSNAME ljl class_names ['level1', 'level2', 'level3', 'level4']\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.38      0.38      0.38        24\n",
            "      level2       0.61      0.59      0.60        61\n",
            "      level3       0.67      0.70      0.69        70\n",
            "      level4       0.66      0.65      0.65        51\n",
            "\n",
            "    accuracy                           0.62       206\n",
            "   macro avg       0.58      0.58      0.58       206\n",
            "weighted avg       0.62      0.62      0.62       206\n",
            "\n",
            "-------------------------------------------------------run 2\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2755 - accuracy: 0.9088 - val_loss: 0.9417 - val_accuracy: 0.6165\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2900 - accuracy: 0.8981 - val_loss: 0.9420 - val_accuracy: 0.6117\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2685 - accuracy: 0.9105 - val_loss: 0.9405 - val_accuracy: 0.6117\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2988 - accuracy: 0.8975 - val_loss: 0.9401 - val_accuracy: 0.6214\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2692 - accuracy: 0.9045 - val_loss: 0.9391 - val_accuracy: 0.6117\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2789 - accuracy: 0.9056 - val_loss: 0.9403 - val_accuracy: 0.6165\n",
            "Epoch 7/1024\n",
            "51/58 [=========================>....] - ETA: 0s - loss: 0.2857 - accuracy: 0.9032\n",
            "Epoch 00007: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2838 - accuracy: 0.9040 - val_loss: 0.9408 - val_accuracy: 0.6117\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2776 - accuracy: 0.9056 - val_loss: 0.9401 - val_accuracy: 0.6165\n",
            "Epoch 9/1024\n",
            "51/58 [=========================>....] - ETA: 0s - loss: 0.2410 - accuracy: 0.9173\n",
            "Epoch 00009: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2399 - accuracy: 0.9175 - val_loss: 0.9400 - val_accuracy: 0.6165\n",
            "Epoch 10/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.2563 - accuracy: 0.9155Restoring model weights from the end of the best epoch: 5.\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2600 - accuracy: 0.9142 - val_loss: 0.9407 - val_accuracy: 0.6165\n",
            "Epoch 10: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "run 2 CORPUSNAME ljl class_names ['level1', 'level2', 'level3', 'level4']\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.38      0.38      0.38        24\n",
            "      level2       0.60      0.59      0.60        61\n",
            "      level3       0.68      0.69      0.68        70\n",
            "      level4       0.65      0.65      0.65        51\n",
            "\n",
            "    accuracy                           0.61       206\n",
            "   macro avg       0.57      0.57      0.57       206\n",
            "weighted avg       0.61      0.61      0.61       206\n",
            "\n",
            "-------------------------------------------------------run 3\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2863 - accuracy: 0.9051 - val_loss: 0.9401 - val_accuracy: 0.6117\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2626 - accuracy: 0.9153 - val_loss: 0.9427 - val_accuracy: 0.6214\n",
            "Epoch 3/1024\n",
            "53/58 [==========================>...] - ETA: 0s - loss: 0.2632 - accuracy: 0.9110\n",
            "Epoch 00003: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2621 - accuracy: 0.9126 - val_loss: 0.9412 - val_accuracy: 0.6165\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2656 - accuracy: 0.9099 - val_loss: 0.9409 - val_accuracy: 0.6165\n",
            "Epoch 5/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.2675 - accuracy: 0.9138\n",
            "Epoch 00005: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2702 - accuracy: 0.9121 - val_loss: 0.9412 - val_accuracy: 0.6214\n",
            "Epoch 6/1024\n",
            "55/58 [===========================>..] - ETA: 0s - loss: 0.2606 - accuracy: 0.9114Restoring model weights from the end of the best epoch: 1.\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2596 - accuracy: 0.9105 - val_loss: 0.9412 - val_accuracy: 0.6214\n",
            "Epoch 6: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "run 3 CORPUSNAME ljl class_names ['level1', 'level2', 'level3', 'level4']\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.38      0.38      0.38        24\n",
            "      level2       0.60      0.59      0.60        61\n",
            "      level3       0.68      0.69      0.68        70\n",
            "      level4       0.65      0.65      0.65        51\n",
            "\n",
            "    accuracy                           0.61       206\n",
            "   macro avg       0.57      0.57      0.57       206\n",
            "weighted avg       0.61      0.61      0.61       206\n",
            "\n",
            "-------------------------------------------------------run 4\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2812 - accuracy: 0.9018 - val_loss: 0.9395 - val_accuracy: 0.6117\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2812 - accuracy: 0.8997 - val_loss: 0.9395 - val_accuracy: 0.6117\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2736 - accuracy: 0.8986 - val_loss: 0.9407 - val_accuracy: 0.6214\n",
            "Epoch 4/1024\n",
            "52/58 [=========================>....] - ETA: 0s - loss: 0.2554 - accuracy: 0.9117\n",
            "Epoch 00004: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2487 - accuracy: 0.9153 - val_loss: 0.9414 - val_accuracy: 0.6214\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2559 - accuracy: 0.9164 - val_loss: 0.9392 - val_accuracy: 0.6262\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2375 - accuracy: 0.9196 - val_loss: 0.9384 - val_accuracy: 0.6262\n",
            "Epoch 7/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2404 - accuracy: 0.9175 - val_loss: 0.9382 - val_accuracy: 0.6311\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2499 - accuracy: 0.9148 - val_loss: 0.9379 - val_accuracy: 0.6311\n",
            "Epoch 9/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2560 - accuracy: 0.9137 - val_loss: 0.9370 - val_accuracy: 0.6262\n",
            "Epoch 10/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2507 - accuracy: 0.9110 - val_loss: 0.9398 - val_accuracy: 0.6214\n",
            "Epoch 11/1024\n",
            "55/58 [===========================>..] - ETA: 0s - loss: 0.2309 - accuracy: 0.9256\n",
            "Epoch 00011: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2329 - accuracy: 0.9250 - val_loss: 0.9388 - val_accuracy: 0.6214\n",
            "Epoch 12/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2368 - accuracy: 0.9196 - val_loss: 0.9384 - val_accuracy: 0.6311\n",
            "Epoch 13/1024\n",
            "53/58 [==========================>...] - ETA: 0s - loss: 0.2558 - accuracy: 0.9157\n",
            "Epoch 00013: Reducing Max LR on Plateau: new max lr will be 1.25e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2562 - accuracy: 0.9148 - val_loss: 0.9380 - val_accuracy: 0.6311\n",
            "Epoch 14/1024\n",
            "53/58 [==========================>...] - ETA: 0s - loss: 0.2481 - accuracy: 0.9210Restoring model weights from the end of the best epoch: 9.\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2479 - accuracy: 0.9218 - val_loss: 0.9395 - val_accuracy: 0.6262\n",
            "Epoch 14: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "run 4 CORPUSNAME ljl class_names ['level1', 'level2', 'level3', 'level4']\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.39      0.38      0.38        24\n",
            "      level2       0.61      0.61      0.61        61\n",
            "      level3       0.70      0.69      0.69        70\n",
            "      level4       0.66      0.69      0.67        51\n",
            "\n",
            "    accuracy                           0.63       206\n",
            "   macro avg       0.59      0.59      0.59       206\n",
            "weighted avg       0.63      0.63      0.63       206\n",
            "\n",
            "-------------------------------------------------------------\n",
            "total run 4\n",
            "CORPUSNAME ljl\n",
            "{   'accuracy': 0.62,\n",
            "    'class_names': ['level1', 'level2', 'level3', 'level4'],\n",
            "    'data_name': 'ljl',\n",
            "    'false_neg': array([ 76, 122, 109,  87]),\n",
            "    'false_pos': array([ 69, 125, 112,  88]),\n",
            "    'fmeasure': array([0.38, 0.6 , 0.69, 0.66]),\n",
            "    'macro_avg_fmeasure': 0.58,\n",
            "    'macro_avg_precision': 0.58,\n",
            "    'macro_avg_recall': 0.58,\n",
            "    'precision': array([0.39, 0.59, 0.68, 0.66]),\n",
            "    'recall': array([0.37, 0.6 , 0.69, 0.66]),\n",
            "    'support': array([120, 305, 350, 255]),\n",
            "    'total_support': 1030,\n",
            "    'true_pos': array([ 44, 183, 241, 168]),\n",
            "    'weighted_average_fmeasure': 0.62,\n",
            "    'weighted_average_precision': 0.62,\n",
            "    'weighted_average_recall': 0.62}\n",
            "(ljl) &\\multicolumn{3}{|c|}{level1}&\\multicolumn{3}{|c|}{level2}&\\multicolumn{3}{|c|}{level3}&\\multicolumn{3}{|c|}{level4}\\\\\n",
            "&P&\tR&\tF1&\tP&\tR&\tF1&\tP&\tR&\tF1&\tP&\tR&\tF1&\tAcc.&\tMacro avg.\\\\\n",
            "\t&0.39\t&0.37\t&0.38\t&0.59\t&0.6\t&0.6\t&0.68\t&0.69\t&0.69\t&0.66\t&0.66\t&0.66\t&0.62\t&0.58\\\\\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {},
          "execution_count": 192
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###CamemBERT"
      ],
      "metadata": {
        "id": "X3SSrkCnLrgP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This takes multiple hours without having enabled the GPU, remember to do this before:    \n",
        "Edit -> Notebook Settings -> Hardware Accelerator : GPU"
      ],
      "metadata": {
        "id": "Lb-3SwrVEWgY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "bert.demo_doBert(\"ljl\") #Can pass \"ljl\", \"bibebook.com\", \"JeLisLibre\", or \"all\" as a parameter\n",
        "#Takes around 15 minutes for the ljl corpus on GPU (default parameter)"
      ],
      "metadata": {
        "id": "mRRUtxGQwFwC",
        "outputId": "087e1a5f-1549-4879-896f-dbc3786a0894",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------------------\n",
            "len_train 1854\n",
            "CORPUS_NAME ljl MODEL_NAME camembert-base class_names ['level1', 'level2', 'level3', 'level4']\n",
            "--> getTransformer\n",
            "preprocessing train...\n",
            "language: fr\n",
            "train sequence lengths:\n",
            "\tmean : 195\n",
            "\t95percentile : 424\n",
            "\t99percentile : 608\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              ""
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Is Multi-Label? False\n",
            "preprocessing test...\n",
            "language: fr\n",
            "test sequence lengths:\n",
            "\tmean : 178\n",
            "\t95percentile : 389\n",
            "\t99percentile : 495\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              ""
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------run 0\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "309/309 [==============================] - 78s 203ms/step - loss: 1.2980 - accuracy: 0.3490 - val_loss: 1.2239 - val_accuracy: 0.4223\n",
            "Epoch 2/1024\n",
            "309/309 [==============================] - 61s 196ms/step - loss: 1.1461 - accuracy: 0.4612 - val_loss: 0.9781 - val_accuracy: 0.5194\n",
            "Epoch 3/1024\n",
            "309/309 [==============================] - 62s 200ms/step - loss: 0.8985 - accuracy: 0.6073 - val_loss: 0.8142 - val_accuracy: 0.6408\n",
            "Epoch 4/1024\n",
            "309/309 [==============================] - 62s 199ms/step - loss: 0.6846 - accuracy: 0.7427 - val_loss: 0.7512 - val_accuracy: 0.6796\n",
            "Epoch 5/1024\n",
            "309/309 [==============================] - 62s 200ms/step - loss: 0.4619 - accuracy: 0.8457 - val_loss: 0.6100 - val_accuracy: 0.7330\n",
            "Epoch 6/1024\n",
            "309/309 [==============================] - 61s 198ms/step - loss: 0.3022 - accuracy: 0.9018 - val_loss: 0.6384 - val_accuracy: 0.7767\n",
            "Epoch 7/1024\n",
            "309/309 [==============================] - ETA: 0s - loss: 0.2083 - accuracy: 0.9261\n",
            "Epoch 00007: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "309/309 [==============================] - 62s 199ms/step - loss: 0.2083 - accuracy: 0.9261 - val_loss: 0.6966 - val_accuracy: 0.7670\n",
            "Epoch 8/1024\n",
            "309/309 [==============================] - 61s 198ms/step - loss: 0.1005 - accuracy: 0.9730 - val_loss: 0.6774 - val_accuracy: 0.7864\n",
            "Epoch 9/1024\n",
            "309/309 [==============================] - ETA: 0s - loss: 0.0627 - accuracy: 0.9806\n",
            "Epoch 00009: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "309/309 [==============================] - 61s 198ms/step - loss: 0.0627 - accuracy: 0.9806 - val_loss: 0.8805 - val_accuracy: 0.7670\n",
            "Epoch 10/1024\n",
            "309/309 [==============================] - ETA: 0s - loss: 0.0276 - accuracy: 0.9951Restoring model weights from the end of the best epoch: 5.\n",
            "309/309 [==============================] - 62s 199ms/step - loss: 0.0276 - accuracy: 0.9951 - val_loss: 0.8688 - val_accuracy: 0.7816\n",
            "Epoch 10: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "MODEL_NAME camembert-base run 0 CORPUSNAME ljl class_names ['level1', 'level2', 'level3', 'level4']\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.70      0.28      0.40        25\n",
            "      level2       0.66      0.77      0.71        65\n",
            "      level3       0.72      0.83      0.77        69\n",
            "      level4       0.90      0.79      0.84        47\n",
            "\n",
            "    accuracy                           0.73       206\n",
            "   macro avg       0.75      0.67      0.68       206\n",
            "weighted avg       0.74      0.73      0.72       206\n",
            "\n",
            "-------------------------------------------------------run 1\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "309/309 [==============================] - 62s 201ms/step - loss: 0.3678 - accuracy: 0.8743 - val_loss: 0.6336 - val_accuracy: 0.7767\n",
            "Epoch 2/1024\n",
            "309/309 [==============================] - 62s 200ms/step - loss: 0.3689 - accuracy: 0.8571 - val_loss: 0.6499 - val_accuracy: 0.7718\n",
            "Epoch 3/1024\n",
            "309/309 [==============================] - ETA: 0s - loss: 0.2545 - accuracy: 0.9024\n",
            "Epoch 00003: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "309/309 [==============================] - 62s 199ms/step - loss: 0.2545 - accuracy: 0.9024 - val_loss: 0.7610 - val_accuracy: 0.7621\n",
            "Epoch 4/1024\n",
            "309/309 [==============================] - 62s 199ms/step - loss: 0.1663 - accuracy: 0.9455 - val_loss: 0.7390 - val_accuracy: 0.7816\n",
            "Epoch 5/1024\n",
            "309/309 [==============================] - ETA: 0s - loss: 0.1257 - accuracy: 0.9563\n",
            "Epoch 00005: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "309/309 [==============================] - 62s 198ms/step - loss: 0.1257 - accuracy: 0.9563 - val_loss: 0.8130 - val_accuracy: 0.7913\n",
            "Epoch 6/1024\n",
            "208/309 [===================>..........] - ETA: 19s - loss: 0.0975 - accuracy: 0.9647"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction to the library"
      ],
      "metadata": {
        "id": "BJDUhalJXG-V"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eO5yJj1MLyFO"
      },
      "source": [
        "## Importing data for the examples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 85,
      "metadata": {
        "id": "fOKIaEKfLyFP"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "with open(os.path.join(os.getcwd(),\"readability\",\"data\",\"tokens_split.pkl\"), \"rb\") as file:\n",
        "    corpus = pickle.load(file)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## How to use"
      ],
      "metadata": {
        "id": "XTrGwhry70p2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### The first main component of the library\n",
        "\n",
        "The class that takes care of calling the processes developped for handling various text readability tasks is called the \"ReadabilityProcessor\".\n",
        "\n",
        "When initializing it, it loads external resources that may be needed such as NLP processors, language models, or dataframes containing data such as word lists.\n",
        "\n",
        "Each measure is enabled by default, but can be excluded, alongside their dependencies, on a case-by-case basis.  \n",
        "As of July 22th 2022, the following measures are available:\n",
        "\n",
        "Traditional scores :\n",
        "* gfi, ari, fre, fkgl, smog, rel\n",
        "\n",
        "Measures related to perplexity:\n",
        "* pppl (pseudo-perplexity)\n",
        "\n",
        "Measures related to text diversity:\n",
        "* ttr, ntr (text/noun token ratio)\n",
        "\n",
        "Measures linked with word lists:\n",
        "* old20, pld20 (Orthographic/Phonemic Levenshtein Distance 20)\n",
        "* dubois_buyse_ratio\n",
        "\n",
        "Measures related to text cohesion:\n",
        "* entity_density, average_entity_word_length\n",
        "* cosine_similarity_tfidf, cosine_similarity_LDA\n",
        "* referring_entity_ratio, average_length_reference_chain (Uses coreference chains)\n",
        "\n",
        "For an quick explanation on a measure's origin, and how it works, please note that the help function can be used on the functions, which share the same name as the measures themselves.  \n",
        "For instance, try help(readability_processor.gfi)\n",
        "\n",
        "Since the module is documented, it can also be appropriate to call help() on an instance in order to view everything."
      ],
      "metadata": {
        "id": "Th_yOOjM70oF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### The second main component of the library\n",
        "\n",
        "The ParsedText class, and its extension ParsedCollection can be created from a readability processor when supplied with a collection.\n",
        "\n",
        "They're not only an interface between texts and the readability processor, these are also used to store the measures obtained from the readability processor, alongside various common statistics, which can be re-used across several of the processor's functions in order to speed up the process.\n",
        "\n",
        "They're currently composed of four attributes:\n",
        "* content: Storing a text, or a collection of text\n",
        "* readability_processor: a shared instance of the readability processor\n",
        "* statistics: A dictionary storing various common statistics\n",
        "* scores: A dictionary storing measures obtained from the readability processor\n",
        "\n",
        "Since the modules are documented, it can also be appropriate to call help() on an instance in order to view everything."
      ],
      "metadata": {
        "id": "8QRlsUOY70mG"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Apwsw5v2LyFP"
      },
      "source": [
        "## Example one : Using the library for a text\n",
        "\n",
        "Texts can be strings, but it is preferred to prepare them beforehand as tokenized sentences.\n",
        "If using spacy, something like this can be used :  \n",
        "new_text = [[token.text for token in sent] for sent in spacy(text).sents]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hgaAcEHLLyFS"
      },
      "source": [
        "A text is processed when calling the parse function of a ReadabilityProcessor instance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "metadata": {
        "id": "-fNjDrEgLyFT"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import spacy\n",
        "processed_text = readability_processor.parse(corpus['level1'][0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bCznBc_8LyFU"
      },
      "source": [
        "Common scores can be accessed by using the corresponding function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "metadata": {
        "id": "XEJK_MYOLyFU",
        "outputId": "131e5217-a16b-4781-b906-7be85eb276e9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "61.52380952380953"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ],
      "source": [
        "gfi = processed_text.gfi()\n",
        "gfi"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZLsDXGXrLyFV"
      },
      "source": [
        "More conveniently, a list each available score can be obtained by using .show_scores()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {
        "id": "EqHKydKmLyFW",
        "outputId": "ac45f4bd-7e72-4581-fc06-0ddd124dc79e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['gfi',\n",
              " 'ari',\n",
              " 'fre',\n",
              " 'fkgl',\n",
              " 'smog',\n",
              " 'rel',\n",
              " 'pppl',\n",
              " 'ttr',\n",
              " 'ntr',\n",
              " 'dubois_buyse_ratio',\n",
              " 'old20',\n",
              " 'pld20',\n",
              " 'cosine_similarity_tfidf',\n",
              " 'entity_density',\n",
              " 'referring_entity_ratio',\n",
              " 'average_entity_word_length',\n",
              " 'average_length_reference_chain',\n",
              " 'cosine_similarity_LDA']"
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ],
      "source": [
        "processed_text.show_available_scores()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "processed_text.show_scores(force=False)"
      ],
      "metadata": {
        "id": "IvEHQkK384sG",
        "outputId": "6e36dcde-309c-4c0c-b159-a981f9cd3b96",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 145
        }
      },
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        gfi        ari        fre     fkgl      smog        rel       pppl  \\\n",
              "0  61.52381  21.503161  52.633986  8.63882  21.86996  71.403333  25.510916   \n",
              "\n",
              "        ttr       ntr  dubois_buyse_ratio     old20     pld20  \\\n",
              "0  0.605128  0.612245            0.641026  2.541743  2.064141   \n",
              "\n",
              "   cosine_similarity_tfidf  entity_density referring_entity_ratio  \\\n",
              "0                 0.070205             0.0                   None   \n",
              "\n",
              "  average_entity_word_length average_length_reference_chain  \\\n",
              "0                       None                           None   \n",
              "\n",
              "  cosine_similarity_LDA  \n",
              "0                  None  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f1478b00-e8e8-46d5-9c75-0f0aaf6033b0\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>gfi</th>\n",
              "      <th>ari</th>\n",
              "      <th>fre</th>\n",
              "      <th>fkgl</th>\n",
              "      <th>smog</th>\n",
              "      <th>rel</th>\n",
              "      <th>pppl</th>\n",
              "      <th>ttr</th>\n",
              "      <th>ntr</th>\n",
              "      <th>dubois_buyse_ratio</th>\n",
              "      <th>old20</th>\n",
              "      <th>pld20</th>\n",
              "      <th>cosine_similarity_tfidf</th>\n",
              "      <th>entity_density</th>\n",
              "      <th>referring_entity_ratio</th>\n",
              "      <th>average_entity_word_length</th>\n",
              "      <th>average_length_reference_chain</th>\n",
              "      <th>cosine_similarity_LDA</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>61.52381</td>\n",
              "      <td>21.503161</td>\n",
              "      <td>52.633986</td>\n",
              "      <td>8.63882</td>\n",
              "      <td>21.86996</td>\n",
              "      <td>71.403333</td>\n",
              "      <td>25.510916</td>\n",
              "      <td>0.605128</td>\n",
              "      <td>0.612245</td>\n",
              "      <td>0.641026</td>\n",
              "      <td>2.541743</td>\n",
              "      <td>2.064141</td>\n",
              "      <td>0.070205</td>\n",
              "      <td>0.0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f1478b00-e8e8-46d5-9c75-0f0aaf6033b0')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f1478b00-e8e8-46d5-9c75-0f0aaf6033b0 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f1478b00-e8e8-46d5-9c75-0f0aaf6033b0');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2RsaWHmrLyFW"
      },
      "source": [
        "Processed texts also incorporate some common statistics for analyzing texts, these can be viewed from an dictionary attribute called \"statistics\", or by calling the show_statistics() method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 98,
      "metadata": {
        "id": "3iNUbMcOLyFW",
        "outputId": "a32686c6-b972-4f9e-c84f-7ccf0b407397",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "totalWords = 230\n",
            "totalLongWords = 30\n",
            "totalSentences = 21\n",
            "totalCharacters = 837\n",
            "totalSyllables = 389\n",
            "nbPolysyllables = 226\n",
            "vocabulary = 125 words\n"
          ]
        }
      ],
      "source": [
        "processed_text.show_statistics()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xeG9xrv1LyFX"
      },
      "source": [
        "## Example two : Using the library for a corpus\n",
        "\n",
        "Currently, a corpus will be recognized by the library if using the following two structures:\n",
        "type(corpus) = dict[class][text]\n",
        "type(corpus) = list(list(text))  \n",
        "As mentioned earlier, text can be a string, or a list of sentences, or a list of sentences split into tokens."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 99,
      "metadata": {
        "id": "ja15sx79LyFX"
      },
      "outputs": [],
      "source": [
        "processed_corpus = readability_processor.parseCollection(corpus)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X-dg0de3LyFY"
      },
      "source": [
        "ParsedCollection instances are similar to ParsedText instances, and possess many functions with the same name that do the same thing, but applied over a corpus."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {
        "id": "TKII3iNjLyFY",
        "outputId": "1d2ba67e-117b-4ae4-e40e-ac4623032b80",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "level1------------------\n",
            "totalWords = 38976\n",
            "totalLongWords = 5182\n",
            "totalSentences = 4880\n",
            "totalCharacters = 141592\n",
            "totalSyllables = 58965\n",
            "nbPolysyllables = 27367\n",
            "vocabulary = 4836 words\n",
            "totalTexts = 240\n",
            "meanSentences = 20.3\n",
            "meanTokens = 162.4\n",
            "level2------------------\n",
            "totalWords = 128019\n",
            "totalLongWords = 20547\n",
            "totalSentences = 13049\n",
            "totalCharacters = 487685\n",
            "totalSyllables = 205889\n",
            "nbPolysyllables = 102106\n",
            "vocabulary = 10903 words\n",
            "totalTexts = 628\n",
            "meanSentences = 20.8\n",
            "meanTokens = 203.9\n",
            "level3------------------\n",
            "totalWords = 124901\n",
            "totalLongWords = 22224\n",
            "totalSentences = 10354\n",
            "totalCharacters = 491007\n",
            "totalSyllables = 207672\n",
            "nbPolysyllables = 107141\n",
            "vocabulary = 11953 words\n",
            "totalTexts = 670\n",
            "meanSentences = 15.5\n",
            "meanTokens = 186.4\n",
            "level4------------------\n",
            "totalWords = 101165\n",
            "totalLongWords = 19550\n",
            "totalSentences = 7743\n",
            "totalCharacters = 410227\n",
            "totalSyllables = 173298\n",
            "nbPolysyllables = 92325\n",
            "vocabulary = 11410 words\n",
            "totalTexts = 522\n",
            "meanSentences = 14.8\n",
            "meanTokens = 193.8\n"
          ]
        }
      ],
      "source": [
        "processed_corpus.show_statistics()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {
        "id": "aC4P0miCLyFZ",
        "outputId": "e8b35e7d-4b0c-4667-99cf-f15d3e7736a7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'level1': 45.13251776706827,\n",
              " 'level2': 59.36149618067255,\n",
              " 'level3': 60.12379485721084,\n",
              " 'level4': 57.573795900895234}"
            ]
          },
          "metadata": {},
          "execution_count": 101
        }
      ],
      "source": [
        "processed_corpus.gfi()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ParsedText instances can be accessed from a ParsedCollection instance:"
      ],
      "metadata": {
        "id": "4C_PY0rF-5Ly"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "processed_corpus.content['level1'][0].gfi() # Is also 61.52380952380953, as seen in the previous section"
      ],
      "metadata": {
        "id": "XBpKtfOYfUgG",
        "outputId": "6ce83843-ef78-4afb-a340-1d37c697eda3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "61.52380952380953"
            ]
          },
          "metadata": {},
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FeZFhG7cLyFa"
      },
      "source": [
        "r.show_scores behaves differently, instead of giving the scores for each text, it returns a dataframe showing the mean values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {
        "id": "vod9d0HBLyFa",
        "outputId": "a68a2525-9795-40f5-dae6-0a143cc0a074",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        }
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-109-6f7f1b771981>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprocessed_corpus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow_scores\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mforce\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcorrelation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"pearson\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/content/READI-LREC22/readability/parsed_collection/parsed_collection.py\u001b[0m in \u001b[0;36mshow_scores\u001b[0;34m(self, force, correlation)\u001b[0m\n\u001b[1;32m    165\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mforce\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    166\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mscore_name\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscores\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 167\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscores\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mscore_name\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcall_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mscore_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mforce\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    168\u001b[0m         \u001b[0mdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    169\u001b[0m         \u001b[0mscore_names\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/parsed_collection/parsed_collection.py\u001b[0m in \u001b[0;36mcall_score\u001b[0;34m(self, score_name, arguments, force, iterable_arguments)\u001b[0m\n\u001b[1;32m    144\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0miterable_arguments\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    145\u001b[0m                     \u001b[0;32mfor\u001b[0m \u001b[0mtext\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m                         \u001b[0mmoy\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mtext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcall_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mscore_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0marguments\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mforce\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m                     \u001b[0;32mfor\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtext\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/parsed_text/parsed_text.py\u001b[0m in \u001b[0;36mcall_score\u001b[0;34m(self, score_name, arguments, force)\u001b[0m\n\u001b[1;32m    104\u001b[0m                 \u001b[0;31m#print(\"WARNING: defaulting to default arguments :\", arguments)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 106\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscores\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mscore_name\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marguments\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    107\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscores\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mscore_name\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    108\u001b[0m         \u001b[0;31m# If function is unavailable, return None to indicate so.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/readability.py\u001b[0m in \u001b[0;36mperplexity\u001b[0;34m(self, content)\u001b[0m\n\u001b[1;32m    345\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"measure 'pppl' cannot be calculated, please try ReadabilityProcessor.load('pppl') and try again.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    346\u001b[0m         \u001b[0;31m#print(\"Please be patient, pseudo-perplexity takes a lot of time to calculate.\")\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 347\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mperplexity\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPPPL_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdependencies\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"GPT2_LM\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    348\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    349\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mstub_rsrs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/stats/perplexity.py\u001b[0m in \u001b[0;36mPPPL_score\u001b[0;34m(GPT2_LM, content)\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0mtokenize_input\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mGPT2_LM\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"tokenizer\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0mtensor_input\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtokenize_input\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mGPT2_LM\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"max_length\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m     \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mGPT2_LM\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"model\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensor_input\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtensor_input\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     22\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetach\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1128\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1129\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1130\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1131\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1132\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/modeling_gpt2.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, labels, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m    960\u001b[0m             \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    961\u001b[0m             \u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_hidden_states\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 962\u001b[0;31m             \u001b[0mreturn_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreturn_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    963\u001b[0m         )\n\u001b[1;32m    964\u001b[0m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtransformer_outputs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1128\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1129\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1130\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1131\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1132\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/modeling_gpt2.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[1;32m    799\u001b[0m                     \u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencoder_attention_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    800\u001b[0m                     \u001b[0muse_cache\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_cache\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 801\u001b[0;31m                     \u001b[0moutput_attentions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moutput_attentions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    802\u001b[0m                 )\n\u001b[1;32m    803\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1128\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1129\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1130\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1131\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1132\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/modeling_gpt2.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states, layer_past, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, use_cache, output_attentions)\u001b[0m\n\u001b[1;32m    353\u001b[0m         \u001b[0mresidual\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    354\u001b[0m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mln_2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 355\u001b[0;31m         \u001b[0mfeed_forward_hidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmlp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    356\u001b[0m         \u001b[0;31m# residual connection\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    357\u001b[0m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mresidual\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mfeed_forward_hidden_states\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1128\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1129\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1130\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1131\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1132\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/modeling_gpt2.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, hidden_states)\u001b[0m\n\u001b[1;32m    282\u001b[0m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mc_fc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    283\u001b[0m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 284\u001b[0;31m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mc_proj\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    285\u001b[0m         \u001b[0mhidden_states\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdropout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden_states\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    286\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mhidden_states\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1128\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1129\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1130\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1131\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1132\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m   1599\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1600\u001b[0m         \u001b[0msize_out\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnf\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1601\u001b[0;31m         \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maddmm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mview\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1602\u001b[0m         \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mview\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msize_out\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1603\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "processed_corpus.show_scores(force=True,correlation=\"pearson\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "processed_corpus_bb.show_scores()(force=True,correlation=\"pearson\")"
      ],
      "metadata": {
        "id": "it4lzVHIwJvQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "processed_corpus_jll.show_scores(force=True,correlation=\"pearson\")"
      ],
      "metadata": {
        "id": "Ee42YZAL_CfF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Example three, Machine Learning and Deep Learning applications\n",
        "\n",
        "machine learning and deep learning applications can be used with the corpus' data to help develop NLP solutions.\n",
        "\n",
        "Keep in mind that at the moment, these can only be used with a ParsedCollection instance, and not a ParsedText instance."
      ],
      "metadata": {
        "id": "JAU11v1dAxbe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "SVM_metrics = processed_corpus.classify_corpus_SVM()"
      ],
      "metadata": {
        "id": "fKDvnylkBBr2",
        "outputId": "ac0cc3b3-e0a5-4b25-c932-715e89bf19b0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Now using Support Vector Machine to attempt to classify corpus\n",
            "cross-validation result for 5 runs = 0.47524271844660193\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.45      0.42      0.44       240\n",
            "      level2       0.46      0.60      0.52       628\n",
            "      level3       0.47      0.49      0.48       670\n",
            "      level4       0.53      0.33      0.41       522\n",
            "\n",
            "    accuracy                           0.48      2060\n",
            "   macro avg       0.48      0.46      0.46      2060\n",
            "weighted avg       0.48      0.48      0.47      2060\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "MLP_metrics = processed_corpus.classify_corpus_MLP()"
      ],
      "metadata": {
        "id": "seqWhiPPBe_X",
        "outputId": "9ed2915b-807a-4ebb-a780-68b034ab9956",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Now using Multilayer Perceptron to attempt to classify corpus\n",
            "cross-validation result for 5 runs = 0.479126213592233\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.45      0.47      0.46       240\n",
            "      level2       0.47      0.61      0.53       628\n",
            "      level3       0.47      0.49      0.48       670\n",
            "      level4       0.56      0.31      0.40       522\n",
            "\n",
            "    accuracy                           0.48      2060\n",
            "   macro avg       0.49      0.47      0.47      2060\n",
            "weighted avg       0.49      0.48      0.47      2060\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fasttext_metrics = processed_corpus.classify_corpus_fasttext()"
      ],
      "metadata": {
        "id": "AuczEDKVBi-c",
        "outputId": "535e3e7d-c097-487d-8b5e-e46024e8a0d4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "language: fr\n",
            "Word Counts: 19792\n",
            "Nrows: 1854\n",
            "1854 train sequences\n",
            "train sequence lengths:\n",
            "\tmean : 191\n",
            "\t95percentile : 412\n",
            "\t99percentile : 581\n",
            "x_train shape: (1854,150)\n",
            "y_train shape: (1854, 4)\n",
            "Is Multi-Label? False\n",
            "206 test sequences\n",
            "test sequence lengths:\n",
            "\tmean : 186\n",
            "\t95percentile : 445\n",
            "\t99percentile : 582\n",
            "x_test shape: (206,150)\n",
            "y_test shape: (206, 4)\n",
            "task: text classification\n",
            "Is Multi-Label? False\n",
            "compiling word ID features...\n",
            "maxlen is 150\n",
            "done.\n",
            "-------------------------------------------------------run 0\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 5s 10ms/step - loss: 1.7763 - accuracy: 0.2789 - val_loss: 1.3771 - val_accuracy: 0.3058\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.7381 - accuracy: 0.2524 - val_loss: 1.3761 - val_accuracy: 0.3058\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.7725 - accuracy: 0.2503 - val_loss: 1.3749 - val_accuracy: 0.3301\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.7126 - accuracy: 0.2675 - val_loss: 1.3741 - val_accuracy: 0.3204\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6521 - accuracy: 0.2772 - val_loss: 1.3733 - val_accuracy: 0.3641\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6794 - accuracy: 0.2805 - val_loss: 1.3719 - val_accuracy: 0.3447\n",
            "Epoch 7/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6561 - accuracy: 0.2869 - val_loss: 1.3698 - val_accuracy: 0.3447\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6359 - accuracy: 0.2826 - val_loss: 1.3668 - val_accuracy: 0.3786\n",
            "Epoch 9/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5912 - accuracy: 0.3004 - val_loss: 1.3625 - val_accuracy: 0.3932\n",
            "Epoch 10/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6539 - accuracy: 0.2859 - val_loss: 1.3562 - val_accuracy: 0.3981\n",
            "Epoch 11/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6159 - accuracy: 0.2950 - val_loss: 1.3500 - val_accuracy: 0.4175\n",
            "Epoch 12/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.6038 - accuracy: 0.3010 - val_loss: 1.3452 - val_accuracy: 0.4223\n",
            "Epoch 13/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5995 - accuracy: 0.2940 - val_loss: 1.3398 - val_accuracy: 0.4369\n",
            "Epoch 14/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5481 - accuracy: 0.3236 - val_loss: 1.3357 - val_accuracy: 0.4320\n",
            "Epoch 15/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5744 - accuracy: 0.3026 - val_loss: 1.3321 - val_accuracy: 0.4272\n",
            "Epoch 16/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5186 - accuracy: 0.3296 - val_loss: 1.3289 - val_accuracy: 0.4175\n",
            "Epoch 17/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5395 - accuracy: 0.3047 - val_loss: 1.3260 - val_accuracy: 0.4320\n",
            "Epoch 18/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5143 - accuracy: 0.3215 - val_loss: 1.3238 - val_accuracy: 0.4563\n",
            "Epoch 19/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5348 - accuracy: 0.3182 - val_loss: 1.3199 - val_accuracy: 0.4660\n",
            "Epoch 20/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4932 - accuracy: 0.3350 - val_loss: 1.3175 - val_accuracy: 0.4660\n",
            "Epoch 21/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4915 - accuracy: 0.3258 - val_loss: 1.3169 - val_accuracy: 0.4757\n",
            "Epoch 22/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.5151 - accuracy: 0.3376 - val_loss: 1.3157 - val_accuracy: 0.4806\n",
            "Epoch 23/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4434 - accuracy: 0.3533 - val_loss: 1.3110 - val_accuracy: 0.4757\n",
            "Epoch 24/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4914 - accuracy: 0.3387 - val_loss: 1.3076 - val_accuracy: 0.4709\n",
            "Epoch 25/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4559 - accuracy: 0.3436 - val_loss: 1.3049 - val_accuracy: 0.4709\n",
            "Epoch 26/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4833 - accuracy: 0.3317 - val_loss: 1.3029 - val_accuracy: 0.4563\n",
            "Epoch 27/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4615 - accuracy: 0.3355 - val_loss: 1.3001 - val_accuracy: 0.4806\n",
            "Epoch 28/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4287 - accuracy: 0.3625 - val_loss: 1.2987 - val_accuracy: 0.4951\n",
            "Epoch 29/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4134 - accuracy: 0.3554 - val_loss: 1.2958 - val_accuracy: 0.4951\n",
            "Epoch 30/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3963 - accuracy: 0.3743 - val_loss: 1.2952 - val_accuracy: 0.5000\n",
            "Epoch 31/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4195 - accuracy: 0.3533 - val_loss: 1.2925 - val_accuracy: 0.5049\n",
            "Epoch 32/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3538 - accuracy: 0.3738 - val_loss: 1.2897 - val_accuracy: 0.5146\n",
            "Epoch 33/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3844 - accuracy: 0.3668 - val_loss: 1.2877 - val_accuracy: 0.5097\n",
            "Epoch 34/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3795 - accuracy: 0.3781 - val_loss: 1.2850 - val_accuracy: 0.5097\n",
            "Epoch 35/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3680 - accuracy: 0.3786 - val_loss: 1.2821 - val_accuracy: 0.5194\n",
            "Epoch 36/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3486 - accuracy: 0.3873 - val_loss: 1.2813 - val_accuracy: 0.5146\n",
            "Epoch 37/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3746 - accuracy: 0.3851 - val_loss: 1.2765 - val_accuracy: 0.5146\n",
            "Epoch 38/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3518 - accuracy: 0.3943 - val_loss: 1.2734 - val_accuracy: 0.5194\n",
            "Epoch 39/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3547 - accuracy: 0.3916 - val_loss: 1.2715 - val_accuracy: 0.5291\n",
            "Epoch 40/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3664 - accuracy: 0.3781 - val_loss: 1.2680 - val_accuracy: 0.5291\n",
            "Epoch 41/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3441 - accuracy: 0.3786 - val_loss: 1.2654 - val_accuracy: 0.5146\n",
            "Epoch 42/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3337 - accuracy: 0.4040 - val_loss: 1.2626 - val_accuracy: 0.5243\n",
            "Epoch 43/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3070 - accuracy: 0.4088 - val_loss: 1.2596 - val_accuracy: 0.5340\n",
            "Epoch 44/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3079 - accuracy: 0.4008 - val_loss: 1.2579 - val_accuracy: 0.5243\n",
            "Epoch 45/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3284 - accuracy: 0.3889 - val_loss: 1.2553 - val_accuracy: 0.5340\n",
            "Epoch 46/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3386 - accuracy: 0.3991 - val_loss: 1.2513 - val_accuracy: 0.5485\n",
            "Epoch 47/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2821 - accuracy: 0.4094 - val_loss: 1.2495 - val_accuracy: 0.5485\n",
            "Epoch 48/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2970 - accuracy: 0.4061 - val_loss: 1.2468 - val_accuracy: 0.5437\n",
            "Epoch 49/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2894 - accuracy: 0.4229 - val_loss: 1.2440 - val_accuracy: 0.5485\n",
            "Epoch 50/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2089 - accuracy: 0.4579 - val_loss: 1.2416 - val_accuracy: 0.5485\n",
            "Epoch 51/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2418 - accuracy: 0.4315 - val_loss: 1.2375 - val_accuracy: 0.5583\n",
            "Epoch 52/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2476 - accuracy: 0.4277 - val_loss: 1.2332 - val_accuracy: 0.5583\n",
            "Epoch 53/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2380 - accuracy: 0.4455 - val_loss: 1.2319 - val_accuracy: 0.5534\n",
            "Epoch 54/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2240 - accuracy: 0.4612 - val_loss: 1.2299 - val_accuracy: 0.5728\n",
            "Epoch 55/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2061 - accuracy: 0.4601 - val_loss: 1.2272 - val_accuracy: 0.5631\n",
            "Epoch 56/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2045 - accuracy: 0.4671 - val_loss: 1.2245 - val_accuracy: 0.5583\n",
            "Epoch 57/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2154 - accuracy: 0.4509 - val_loss: 1.2178 - val_accuracy: 0.5583\n",
            "Epoch 58/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.2015 - accuracy: 0.4676 - val_loss: 1.2155 - val_accuracy: 0.5680\n",
            "Epoch 59/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1744 - accuracy: 0.4833 - val_loss: 1.2120 - val_accuracy: 0.5874\n",
            "Epoch 60/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1540 - accuracy: 0.5005 - val_loss: 1.2082 - val_accuracy: 0.5825\n",
            "Epoch 61/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1951 - accuracy: 0.4633 - val_loss: 1.2060 - val_accuracy: 0.5971\n",
            "Epoch 62/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1755 - accuracy: 0.4671 - val_loss: 1.2017 - val_accuracy: 0.6019\n",
            "Epoch 63/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1619 - accuracy: 0.5000 - val_loss: 1.1979 - val_accuracy: 0.6019\n",
            "Epoch 64/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1381 - accuracy: 0.4968 - val_loss: 1.1933 - val_accuracy: 0.6068\n",
            "Epoch 65/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1535 - accuracy: 0.5027 - val_loss: 1.1883 - val_accuracy: 0.6019\n",
            "Epoch 66/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1163 - accuracy: 0.4984 - val_loss: 1.1837 - val_accuracy: 0.6019\n",
            "Epoch 67/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.1268 - accuracy: 0.5092 - val_loss: 1.1803 - val_accuracy: 0.6019\n",
            "Epoch 68/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0846 - accuracy: 0.5324 - val_loss: 1.1783 - val_accuracy: 0.6165\n",
            "Epoch 69/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0713 - accuracy: 0.5372 - val_loss: 1.1764 - val_accuracy: 0.6117\n",
            "Epoch 70/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0875 - accuracy: 0.5254 - val_loss: 1.1693 - val_accuracy: 0.6117\n",
            "Epoch 71/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0845 - accuracy: 0.5313 - val_loss: 1.1655 - val_accuracy: 0.6117\n",
            "Epoch 72/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0691 - accuracy: 0.5221 - val_loss: 1.1622 - val_accuracy: 0.6165\n",
            "Epoch 73/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.0567 - accuracy: 0.5448 - val_loss: 1.1590 - val_accuracy: 0.6117\n",
            "Epoch 74/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0543 - accuracy: 0.5340 - val_loss: 1.1539 - val_accuracy: 0.6117\n",
            "Epoch 75/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0345 - accuracy: 0.5383 - val_loss: 1.1511 - val_accuracy: 0.6165\n",
            "Epoch 76/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0458 - accuracy: 0.5502 - val_loss: 1.1477 - val_accuracy: 0.6165\n",
            "Epoch 77/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0015 - accuracy: 0.5739 - val_loss: 1.1408 - val_accuracy: 0.6262\n",
            "Epoch 78/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0155 - accuracy: 0.5707 - val_loss: 1.1365 - val_accuracy: 0.6359\n",
            "Epoch 79/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0035 - accuracy: 0.5820 - val_loss: 1.1335 - val_accuracy: 0.6262\n",
            "Epoch 80/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0094 - accuracy: 0.5631 - val_loss: 1.1290 - val_accuracy: 0.6214\n",
            "Epoch 81/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9998 - accuracy: 0.5717 - val_loss: 1.1240 - val_accuracy: 0.6359\n",
            "Epoch 82/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9825 - accuracy: 0.5831 - val_loss: 1.1200 - val_accuracy: 0.6359\n",
            "Epoch 83/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9855 - accuracy: 0.5723 - val_loss: 1.1162 - val_accuracy: 0.6359\n",
            "Epoch 84/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9742 - accuracy: 0.5814 - val_loss: 1.1112 - val_accuracy: 0.6408\n",
            "Epoch 85/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.9756 - accuracy: 0.5906 - val_loss: 1.1071 - val_accuracy: 0.6408\n",
            "Epoch 86/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9480 - accuracy: 0.6079 - val_loss: 1.1031 - val_accuracy: 0.6408\n",
            "Epoch 87/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.9420 - accuracy: 0.6106 - val_loss: 1.0982 - val_accuracy: 0.6408\n",
            "Epoch 88/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9186 - accuracy: 0.6219 - val_loss: 1.0957 - val_accuracy: 0.6408\n",
            "Epoch 89/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9200 - accuracy: 0.6332 - val_loss: 1.0902 - val_accuracy: 0.6262\n",
            "Epoch 90/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9327 - accuracy: 0.6079 - val_loss: 1.0854 - val_accuracy: 0.6311\n",
            "Epoch 91/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8965 - accuracy: 0.6338 - val_loss: 1.0791 - val_accuracy: 0.6262\n",
            "Epoch 92/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8861 - accuracy: 0.6397 - val_loss: 1.0746 - val_accuracy: 0.6359\n",
            "Epoch 93/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8527 - accuracy: 0.6559 - val_loss: 1.0709 - val_accuracy: 0.6505\n",
            "Epoch 94/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8894 - accuracy: 0.6381 - val_loss: 1.0664 - val_accuracy: 0.6602\n",
            "Epoch 95/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8560 - accuracy: 0.6532 - val_loss: 1.0637 - val_accuracy: 0.6602\n",
            "Epoch 96/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.8352 - accuracy: 0.6645 - val_loss: 1.0581 - val_accuracy: 0.6553\n",
            "Epoch 97/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8605 - accuracy: 0.6483 - val_loss: 1.0553 - val_accuracy: 0.6456\n",
            "Epoch 98/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8337 - accuracy: 0.6710 - val_loss: 1.0490 - val_accuracy: 0.6505\n",
            "Epoch 99/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8400 - accuracy: 0.6478 - val_loss: 1.0456 - val_accuracy: 0.6505\n",
            "Epoch 100/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8071 - accuracy: 0.6785 - val_loss: 1.0407 - val_accuracy: 0.6699\n",
            "Epoch 101/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8093 - accuracy: 0.6710 - val_loss: 1.0342 - val_accuracy: 0.6748\n",
            "Epoch 102/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.8080 - accuracy: 0.6721 - val_loss: 1.0308 - val_accuracy: 0.6650\n",
            "Epoch 103/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7952 - accuracy: 0.6640 - val_loss: 1.0265 - val_accuracy: 0.6699\n",
            "Epoch 104/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7749 - accuracy: 0.6915 - val_loss: 1.0218 - val_accuracy: 0.6748\n",
            "Epoch 105/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.7828 - accuracy: 0.7033 - val_loss: 1.0169 - val_accuracy: 0.6748\n",
            "Epoch 106/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7509 - accuracy: 0.7033 - val_loss: 1.0134 - val_accuracy: 0.6699\n",
            "Epoch 107/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7345 - accuracy: 0.7077 - val_loss: 1.0059 - val_accuracy: 0.6699\n",
            "Epoch 108/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7334 - accuracy: 0.7157 - val_loss: 1.0047 - val_accuracy: 0.6553\n",
            "Epoch 109/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.7440 - accuracy: 0.7120 - val_loss: 1.0014 - val_accuracy: 0.6699\n",
            "Epoch 110/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.7007 - accuracy: 0.7282 - val_loss: 0.9947 - val_accuracy: 0.6699\n",
            "Epoch 111/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6893 - accuracy: 0.7287 - val_loss: 0.9897 - val_accuracy: 0.6650\n",
            "Epoch 112/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6934 - accuracy: 0.7276 - val_loss: 0.9855 - val_accuracy: 0.6602\n",
            "Epoch 113/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6994 - accuracy: 0.7292 - val_loss: 0.9812 - val_accuracy: 0.6650\n",
            "Epoch 114/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6727 - accuracy: 0.7298 - val_loss: 0.9788 - val_accuracy: 0.6699\n",
            "Epoch 115/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6725 - accuracy: 0.7400 - val_loss: 0.9722 - val_accuracy: 0.6699\n",
            "Epoch 116/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6878 - accuracy: 0.7298 - val_loss: 0.9691 - val_accuracy: 0.6699\n",
            "Epoch 117/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6739 - accuracy: 0.7362 - val_loss: 0.9627 - val_accuracy: 0.6602\n",
            "Epoch 118/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6722 - accuracy: 0.7335 - val_loss: 0.9591 - val_accuracy: 0.6650\n",
            "Epoch 119/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6439 - accuracy: 0.7454 - val_loss: 0.9560 - val_accuracy: 0.6650\n",
            "Epoch 120/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6267 - accuracy: 0.7718 - val_loss: 0.9524 - val_accuracy: 0.6650\n",
            "Epoch 121/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6378 - accuracy: 0.7454 - val_loss: 0.9479 - val_accuracy: 0.6699\n",
            "Epoch 122/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6437 - accuracy: 0.7470 - val_loss: 0.9436 - val_accuracy: 0.6650\n",
            "Epoch 123/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6258 - accuracy: 0.7594 - val_loss: 0.9389 - val_accuracy: 0.6650\n",
            "Epoch 124/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6018 - accuracy: 0.7816 - val_loss: 0.9366 - val_accuracy: 0.6602\n",
            "Epoch 125/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.6122 - accuracy: 0.7665 - val_loss: 0.9331 - val_accuracy: 0.6505\n",
            "Epoch 126/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5885 - accuracy: 0.7751 - val_loss: 0.9276 - val_accuracy: 0.6650\n",
            "Epoch 127/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5810 - accuracy: 0.7799 - val_loss: 0.9235 - val_accuracy: 0.6699\n",
            "Epoch 128/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5462 - accuracy: 0.7864 - val_loss: 0.9201 - val_accuracy: 0.6796\n",
            "Epoch 129/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5643 - accuracy: 0.7929 - val_loss: 0.9159 - val_accuracy: 0.6602\n",
            "Epoch 130/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5707 - accuracy: 0.7810 - val_loss: 0.9125 - val_accuracy: 0.6456\n",
            "Epoch 131/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5540 - accuracy: 0.7843 - val_loss: 0.9094 - val_accuracy: 0.6456\n",
            "Epoch 132/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5364 - accuracy: 0.7999 - val_loss: 0.9052 - val_accuracy: 0.6553\n",
            "Epoch 133/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5439 - accuracy: 0.7988 - val_loss: 0.8999 - val_accuracy: 0.6602\n",
            "Epoch 134/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5243 - accuracy: 0.8020 - val_loss: 0.8978 - val_accuracy: 0.6553\n",
            "Epoch 135/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5243 - accuracy: 0.8015 - val_loss: 0.8933 - val_accuracy: 0.6796\n",
            "Epoch 136/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5222 - accuracy: 0.8037 - val_loss: 0.8910 - val_accuracy: 0.6650\n",
            "Epoch 137/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4976 - accuracy: 0.8123 - val_loss: 0.8873 - val_accuracy: 0.6553\n",
            "Epoch 138/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5203 - accuracy: 0.8053 - val_loss: 0.8845 - val_accuracy: 0.6553\n",
            "Epoch 139/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4907 - accuracy: 0.8193 - val_loss: 0.8817 - val_accuracy: 0.6650\n",
            "Epoch 140/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4814 - accuracy: 0.8312 - val_loss: 0.8793 - val_accuracy: 0.6699\n",
            "Epoch 141/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.5241 - accuracy: 0.8053 - val_loss: 0.8752 - val_accuracy: 0.6699\n",
            "Epoch 142/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4626 - accuracy: 0.8263 - val_loss: 0.8734 - val_accuracy: 0.6748\n",
            "Epoch 143/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4891 - accuracy: 0.8285 - val_loss: 0.8698 - val_accuracy: 0.6796\n",
            "Epoch 144/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4741 - accuracy: 0.8182 - val_loss: 0.8641 - val_accuracy: 0.6796\n",
            "Epoch 145/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4525 - accuracy: 0.8312 - val_loss: 0.8614 - val_accuracy: 0.6796\n",
            "Epoch 146/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4708 - accuracy: 0.8279 - val_loss: 0.8572 - val_accuracy: 0.6845\n",
            "Epoch 147/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4496 - accuracy: 0.8355 - val_loss: 0.8560 - val_accuracy: 0.6893\n",
            "Epoch 148/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4647 - accuracy: 0.8333 - val_loss: 0.8506 - val_accuracy: 0.6845\n",
            "Epoch 149/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4609 - accuracy: 0.8312 - val_loss: 0.8496 - val_accuracy: 0.6845\n",
            "Epoch 150/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4442 - accuracy: 0.8430 - val_loss: 0.8463 - val_accuracy: 0.6893\n",
            "Epoch 151/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4151 - accuracy: 0.8571 - val_loss: 0.8418 - val_accuracy: 0.6893\n",
            "Epoch 152/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4403 - accuracy: 0.8355 - val_loss: 0.8403 - val_accuracy: 0.6893\n",
            "Epoch 153/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4458 - accuracy: 0.8360 - val_loss: 0.8374 - val_accuracy: 0.6893\n",
            "Epoch 154/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4075 - accuracy: 0.8538 - val_loss: 0.8365 - val_accuracy: 0.6796\n",
            "Epoch 155/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4194 - accuracy: 0.8484 - val_loss: 0.8351 - val_accuracy: 0.6845\n",
            "Epoch 156/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4043 - accuracy: 0.8576 - val_loss: 0.8332 - val_accuracy: 0.6845\n",
            "Epoch 157/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3895 - accuracy: 0.8592 - val_loss: 0.8294 - val_accuracy: 0.6796\n",
            "Epoch 158/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3862 - accuracy: 0.8695 - val_loss: 0.8273 - val_accuracy: 0.6942\n",
            "Epoch 159/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3967 - accuracy: 0.8668 - val_loss: 0.8261 - val_accuracy: 0.6990\n",
            "Epoch 160/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3818 - accuracy: 0.8657 - val_loss: 0.8221 - val_accuracy: 0.6990\n",
            "Epoch 161/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3542 - accuracy: 0.8646 - val_loss: 0.8213 - val_accuracy: 0.6990\n",
            "Epoch 162/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.4115 - accuracy: 0.8457 - val_loss: 0.8208 - val_accuracy: 0.6893\n",
            "Epoch 163/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3825 - accuracy: 0.8641 - val_loss: 0.8170 - val_accuracy: 0.6845\n",
            "Epoch 164/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3713 - accuracy: 0.8625 - val_loss: 0.8146 - val_accuracy: 0.6893\n",
            "Epoch 165/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3601 - accuracy: 0.8689 - val_loss: 0.8132 - val_accuracy: 0.6796\n",
            "Epoch 166/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3469 - accuracy: 0.8808 - val_loss: 0.8132 - val_accuracy: 0.6893\n",
            "Epoch 167/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3395 - accuracy: 0.8738 - val_loss: 0.8111 - val_accuracy: 0.6893\n",
            "Epoch 168/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3505 - accuracy: 0.8749 - val_loss: 0.8093 - val_accuracy: 0.6942\n",
            "Epoch 169/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3520 - accuracy: 0.8749 - val_loss: 0.8075 - val_accuracy: 0.6942\n",
            "Epoch 170/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3449 - accuracy: 0.8743 - val_loss: 0.8055 - val_accuracy: 0.6893\n",
            "Epoch 171/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3493 - accuracy: 0.8765 - val_loss: 0.8037 - val_accuracy: 0.6942\n",
            "Epoch 172/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3407 - accuracy: 0.8813 - val_loss: 0.8015 - val_accuracy: 0.6942\n",
            "Epoch 173/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3416 - accuracy: 0.8759 - val_loss: 0.7997 - val_accuracy: 0.6942\n",
            "Epoch 174/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3189 - accuracy: 0.8835 - val_loss: 0.7998 - val_accuracy: 0.6893\n",
            "Epoch 175/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3207 - accuracy: 0.8873 - val_loss: 0.7995 - val_accuracy: 0.6893\n",
            "Epoch 176/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3451 - accuracy: 0.8803 - val_loss: 0.7964 - val_accuracy: 0.6893\n",
            "Epoch 177/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3051 - accuracy: 0.9013 - val_loss: 0.7936 - val_accuracy: 0.6845\n",
            "Epoch 178/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2946 - accuracy: 0.8878 - val_loss: 0.7931 - val_accuracy: 0.6845\n",
            "Epoch 179/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3158 - accuracy: 0.8889 - val_loss: 0.7928 - val_accuracy: 0.6942\n",
            "Epoch 180/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2959 - accuracy: 0.9013 - val_loss: 0.7900 - val_accuracy: 0.6845\n",
            "Epoch 181/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3205 - accuracy: 0.8900 - val_loss: 0.7879 - val_accuracy: 0.6942\n",
            "Epoch 182/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2883 - accuracy: 0.8991 - val_loss: 0.7856 - val_accuracy: 0.6942\n",
            "Epoch 183/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.3029 - accuracy: 0.8921 - val_loss: 0.7833 - val_accuracy: 0.6942\n",
            "Epoch 184/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2901 - accuracy: 0.8997 - val_loss: 0.7822 - val_accuracy: 0.6893\n",
            "Epoch 185/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2919 - accuracy: 0.8970 - val_loss: 0.7825 - val_accuracy: 0.6845\n",
            "Epoch 186/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2768 - accuracy: 0.9029 - val_loss: 0.7820 - val_accuracy: 0.6845\n",
            "Epoch 187/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2736 - accuracy: 0.8970 - val_loss: 0.7816 - val_accuracy: 0.6942\n",
            "Epoch 188/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2775 - accuracy: 0.9132 - val_loss: 0.7807 - val_accuracy: 0.6942\n",
            "Epoch 189/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2751 - accuracy: 0.9061 - val_loss: 0.7790 - val_accuracy: 0.6942\n",
            "Epoch 190/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2831 - accuracy: 0.8997 - val_loss: 0.7773 - val_accuracy: 0.6942\n",
            "Epoch 191/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2655 - accuracy: 0.9056 - val_loss: 0.7747 - val_accuracy: 0.6893\n",
            "Epoch 192/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2767 - accuracy: 0.9072 - val_loss: 0.7736 - val_accuracy: 0.6942\n",
            "Epoch 193/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2708 - accuracy: 0.9051 - val_loss: 0.7739 - val_accuracy: 0.6893\n",
            "Epoch 194/1024\n",
            "58/58 [==============================] - ETA: 0s - loss: 0.2650 - accuracy: 0.9121\n",
            "Epoch 00194: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2650 - accuracy: 0.9121 - val_loss: 0.7743 - val_accuracy: 0.6893\n",
            "Epoch 195/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2403 - accuracy: 0.9175 - val_loss: 0.7736 - val_accuracy: 0.6893\n",
            "Epoch 196/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2367 - accuracy: 0.9191 - val_loss: 0.7722 - val_accuracy: 0.6845\n",
            "Epoch 197/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2723 - accuracy: 0.9061 - val_loss: 0.7721 - val_accuracy: 0.6942\n",
            "Epoch 198/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2504 - accuracy: 0.9175 - val_loss: 0.7711 - val_accuracy: 0.6893\n",
            "Epoch 199/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2616 - accuracy: 0.9110 - val_loss: 0.7716 - val_accuracy: 0.6893\n",
            "Epoch 200/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2432 - accuracy: 0.9239 - val_loss: 0.7702 - val_accuracy: 0.6942\n",
            "Epoch 201/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2425 - accuracy: 0.9218 - val_loss: 0.7693 - val_accuracy: 0.6845\n",
            "Epoch 202/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2387 - accuracy: 0.9110 - val_loss: 0.7701 - val_accuracy: 0.6942\n",
            "Epoch 203/1024\n",
            "56/58 [===========================>..] - ETA: 0s - loss: 0.2402 - accuracy: 0.9191\n",
            "Epoch 00203: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2392 - accuracy: 0.9196 - val_loss: 0.7696 - val_accuracy: 0.6990\n",
            "Epoch 204/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2447 - accuracy: 0.9153 - val_loss: 0.7695 - val_accuracy: 0.6942\n",
            "Epoch 205/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2485 - accuracy: 0.9202 - val_loss: 0.7680 - val_accuracy: 0.6990\n",
            "Epoch 206/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2360 - accuracy: 0.9180 - val_loss: 0.7691 - val_accuracy: 0.6845\n",
            "Epoch 207/1024\n",
            "56/58 [===========================>..] - ETA: 0s - loss: 0.2313 - accuracy: 0.9269\n",
            "Epoch 00207: Reducing Max LR on Plateau: new max lr will be 1.25e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2362 - accuracy: 0.9256 - val_loss: 0.7689 - val_accuracy: 0.6893\n",
            "Epoch 208/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2174 - accuracy: 0.9234 - val_loss: 0.7676 - val_accuracy: 0.6942\n",
            "Epoch 209/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2157 - accuracy: 0.9320 - val_loss: 0.7677 - val_accuracy: 0.6942\n",
            "Epoch 210/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2443 - accuracy: 0.9218 - val_loss: 0.7650 - val_accuracy: 0.6990\n",
            "Epoch 211/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2460 - accuracy: 0.9137 - val_loss: 0.7670 - val_accuracy: 0.6990\n",
            "Epoch 212/1024\n",
            "56/58 [===========================>..] - ETA: 0s - loss: 0.2141 - accuracy: 0.9342\n",
            "Epoch 00212: Reducing Max LR on Plateau: new max lr will be 6.25e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2149 - accuracy: 0.9342 - val_loss: 0.7675 - val_accuracy: 0.6942\n",
            "Epoch 213/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2436 - accuracy: 0.9137 - val_loss: 0.7688 - val_accuracy: 0.6893\n",
            "Epoch 214/1024\n",
            "55/58 [===========================>..] - ETA: 0s - loss: 0.2535 - accuracy: 0.9159\n",
            "Epoch 00214: Reducing Max LR on Plateau: new max lr will be 3.125e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2521 - accuracy: 0.9164 - val_loss: 0.7688 - val_accuracy: 0.6942\n",
            "Epoch 215/1024\n",
            "57/58 [============================>.] - ETA: 0s - loss: 0.2265 - accuracy: 0.9216Restoring model weights from the end of the best epoch: 210.\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 0.2268 - accuracy: 0.9213 - val_loss: 0.7685 - val_accuracy: 0.6942\n",
            "Epoch 215: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.43      0.47      0.45        19\n",
            "      level2       0.73      0.70      0.72        63\n",
            "      level3       0.65      0.73      0.69        64\n",
            "      level4       0.83      0.73      0.78        60\n",
            "\n",
            "    accuracy                           0.70       206\n",
            "   macro avg       0.66      0.66      0.66       206\n",
            "weighted avg       0.71      0.70      0.70       206\n",
            "\n",
            "-------------------------------------------------------run 1\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.7730 - accuracy: 0.2654 - val_loss: 1.3730 - val_accuracy: 0.3058\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.7563 - accuracy: 0.2540 - val_loss: 1.3692 - val_accuracy: 0.3204\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6911 - accuracy: 0.2772 - val_loss: 1.3683 - val_accuracy: 0.3495\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6032 - accuracy: 0.3037 - val_loss: 1.3671 - val_accuracy: 0.3592\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6076 - accuracy: 0.2967 - val_loss: 1.3662 - val_accuracy: 0.3544\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6093 - accuracy: 0.2810 - val_loss: 1.3636 - val_accuracy: 0.3592\n",
            "Epoch 7/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6112 - accuracy: 0.2896 - val_loss: 1.3597 - val_accuracy: 0.3592\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5783 - accuracy: 0.3026 - val_loss: 1.3543 - val_accuracy: 0.4029\n",
            "Epoch 9/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5700 - accuracy: 0.2886 - val_loss: 1.3470 - val_accuracy: 0.3981\n",
            "Epoch 10/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5549 - accuracy: 0.3204 - val_loss: 1.3390 - val_accuracy: 0.4029\n",
            "Epoch 11/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5169 - accuracy: 0.3252 - val_loss: 1.3306 - val_accuracy: 0.4175\n",
            "Epoch 12/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5038 - accuracy: 0.3091 - val_loss: 1.3233 - val_accuracy: 0.4417\n",
            "Epoch 13/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4883 - accuracy: 0.3350 - val_loss: 1.3164 - val_accuracy: 0.4612\n",
            "Epoch 14/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4742 - accuracy: 0.3457 - val_loss: 1.3115 - val_accuracy: 0.4612\n",
            "Epoch 15/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4935 - accuracy: 0.3323 - val_loss: 1.3078 - val_accuracy: 0.4660\n",
            "Epoch 16/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5000 - accuracy: 0.3209 - val_loss: 1.3040 - val_accuracy: 0.4854\n",
            "Epoch 17/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4715 - accuracy: 0.3296 - val_loss: 1.2995 - val_accuracy: 0.4757\n",
            "Epoch 18/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4634 - accuracy: 0.3549 - val_loss: 1.2978 - val_accuracy: 0.4709\n",
            "Epoch 19/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4421 - accuracy: 0.3382 - val_loss: 1.2974 - val_accuracy: 0.4854\n",
            "Epoch 20/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4369 - accuracy: 0.3457 - val_loss: 1.2926 - val_accuracy: 0.4951\n",
            "Epoch 21/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4633 - accuracy: 0.3387 - val_loss: 1.2885 - val_accuracy: 0.5000\n",
            "Epoch 22/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4312 - accuracy: 0.3511 - val_loss: 1.2874 - val_accuracy: 0.4903\n",
            "Epoch 23/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4033 - accuracy: 0.3560 - val_loss: 1.2846 - val_accuracy: 0.4951\n",
            "Epoch 24/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.4076 - accuracy: 0.3517 - val_loss: 1.2830 - val_accuracy: 0.4854\n",
            "Epoch 25/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3966 - accuracy: 0.3603 - val_loss: 1.2810 - val_accuracy: 0.5194\n",
            "Epoch 26/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3723 - accuracy: 0.3770 - val_loss: 1.2794 - val_accuracy: 0.5243\n",
            "Epoch 27/1024\n",
            "58/58 [==============================] - 0s 7ms/step - loss: 1.3877 - accuracy: 0.3662 - val_loss: 1.2754 - val_accuracy: 0.4951\n",
            "Epoch 28/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3630 - accuracy: 0.3813 - val_loss: 1.2724 - val_accuracy: 0.5340\n",
            "Epoch 29/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3611 - accuracy: 0.3781 - val_loss: 1.2713 - val_accuracy: 0.5291\n",
            "Epoch 30/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3559 - accuracy: 0.4040 - val_loss: 1.2694 - val_accuracy: 0.5243\n",
            "Epoch 31/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3558 - accuracy: 0.3851 - val_loss: 1.2659 - val_accuracy: 0.5485\n",
            "Epoch 32/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3453 - accuracy: 0.3846 - val_loss: 1.2634 - val_accuracy: 0.5388\n",
            "Epoch 33/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3297 - accuracy: 0.3867 - val_loss: 1.2624 - val_accuracy: 0.5631\n",
            "Epoch 34/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3274 - accuracy: 0.3835 - val_loss: 1.2605 - val_accuracy: 0.5680\n",
            "Epoch 35/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3173 - accuracy: 0.3927 - val_loss: 1.2571 - val_accuracy: 0.5728\n",
            "Epoch 36/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3259 - accuracy: 0.4051 - val_loss: 1.2538 - val_accuracy: 0.5728\n",
            "Epoch 37/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2742 - accuracy: 0.4110 - val_loss: 1.2528 - val_accuracy: 0.5534\n",
            "Epoch 38/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2752 - accuracy: 0.4310 - val_loss: 1.2493 - val_accuracy: 0.5680\n",
            "Epoch 39/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2769 - accuracy: 0.4186 - val_loss: 1.2495 - val_accuracy: 0.5534\n",
            "Epoch 40/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2836 - accuracy: 0.4094 - val_loss: 1.2450 - val_accuracy: 0.5825\n",
            "Epoch 41/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2745 - accuracy: 0.4137 - val_loss: 1.2398 - val_accuracy: 0.5728\n",
            "Epoch 42/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2788 - accuracy: 0.4293 - val_loss: 1.2401 - val_accuracy: 0.5631\n",
            "Epoch 43/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2757 - accuracy: 0.4180 - val_loss: 1.2382 - val_accuracy: 0.5680\n",
            "Epoch 44/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2548 - accuracy: 0.4385 - val_loss: 1.2349 - val_accuracy: 0.5922\n",
            "Epoch 45/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2233 - accuracy: 0.4407 - val_loss: 1.2332 - val_accuracy: 0.5874\n",
            "Epoch 46/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2188 - accuracy: 0.4482 - val_loss: 1.2308 - val_accuracy: 0.5874\n",
            "Epoch 47/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2340 - accuracy: 0.4536 - val_loss: 1.2272 - val_accuracy: 0.5777\n",
            "Epoch 48/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2260 - accuracy: 0.4574 - val_loss: 1.2250 - val_accuracy: 0.5777\n",
            "Epoch 49/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1995 - accuracy: 0.4703 - val_loss: 1.2227 - val_accuracy: 0.6019\n",
            "Epoch 50/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2433 - accuracy: 0.4434 - val_loss: 1.2203 - val_accuracy: 0.5922\n",
            "Epoch 51/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2062 - accuracy: 0.4639 - val_loss: 1.2170 - val_accuracy: 0.5971\n",
            "Epoch 52/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2005 - accuracy: 0.4628 - val_loss: 1.2123 - val_accuracy: 0.5922\n",
            "Epoch 53/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1712 - accuracy: 0.4741 - val_loss: 1.2082 - val_accuracy: 0.5971\n",
            "Epoch 54/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1597 - accuracy: 0.4951 - val_loss: 1.2056 - val_accuracy: 0.5971\n",
            "Epoch 55/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1204 - accuracy: 0.4995 - val_loss: 1.2028 - val_accuracy: 0.6068\n",
            "Epoch 56/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1560 - accuracy: 0.5000 - val_loss: 1.1990 - val_accuracy: 0.6068\n",
            "Epoch 57/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1231 - accuracy: 0.4935 - val_loss: 1.1962 - val_accuracy: 0.6019\n",
            "Epoch 58/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1453 - accuracy: 0.5027 - val_loss: 1.1919 - val_accuracy: 0.6165\n",
            "Epoch 59/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1515 - accuracy: 0.4978 - val_loss: 1.1875 - val_accuracy: 0.6311\n",
            "Epoch 60/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1130 - accuracy: 0.4984 - val_loss: 1.1869 - val_accuracy: 0.6311\n",
            "Epoch 61/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1100 - accuracy: 0.5000 - val_loss: 1.1828 - val_accuracy: 0.6117\n",
            "Epoch 62/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1260 - accuracy: 0.5016 - val_loss: 1.1780 - val_accuracy: 0.6214\n",
            "Epoch 63/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0900 - accuracy: 0.5189 - val_loss: 1.1739 - val_accuracy: 0.6262\n",
            "Epoch 64/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0710 - accuracy: 0.5329 - val_loss: 1.1710 - val_accuracy: 0.6456\n",
            "Epoch 65/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0566 - accuracy: 0.5378 - val_loss: 1.1674 - val_accuracy: 0.6311\n",
            "Epoch 66/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0628 - accuracy: 0.5345 - val_loss: 1.1611 - val_accuracy: 0.6262\n",
            "Epoch 67/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0243 - accuracy: 0.5545 - val_loss: 1.1594 - val_accuracy: 0.6311\n",
            "Epoch 68/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0370 - accuracy: 0.5599 - val_loss: 1.1535 - val_accuracy: 0.6311\n",
            "Epoch 69/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0498 - accuracy: 0.5561 - val_loss: 1.1500 - val_accuracy: 0.6262\n",
            "Epoch 70/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0372 - accuracy: 0.5534 - val_loss: 1.1430 - val_accuracy: 0.6262\n",
            "Epoch 71/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0226 - accuracy: 0.5653 - val_loss: 1.1396 - val_accuracy: 0.6311\n",
            "Epoch 72/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0498 - accuracy: 0.5572 - val_loss: 1.1355 - val_accuracy: 0.6311\n",
            "Epoch 73/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9891 - accuracy: 0.5847 - val_loss: 1.1323 - val_accuracy: 0.6311\n",
            "Epoch 74/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0140 - accuracy: 0.5793 - val_loss: 1.1277 - val_accuracy: 0.6311\n",
            "Epoch 75/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9619 - accuracy: 0.5976 - val_loss: 1.1249 - val_accuracy: 0.6262\n",
            "Epoch 76/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9794 - accuracy: 0.5933 - val_loss: 1.1211 - val_accuracy: 0.6505\n",
            "Epoch 77/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9658 - accuracy: 0.5933 - val_loss: 1.1153 - val_accuracy: 0.6699\n",
            "Epoch 78/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9579 - accuracy: 0.5976 - val_loss: 1.1116 - val_accuracy: 0.6650\n",
            "Epoch 79/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9304 - accuracy: 0.6208 - val_loss: 1.1069 - val_accuracy: 0.6505\n",
            "Epoch 80/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9488 - accuracy: 0.5944 - val_loss: 1.1016 - val_accuracy: 0.6456\n",
            "Epoch 81/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9106 - accuracy: 0.6294 - val_loss: 1.0969 - val_accuracy: 0.6505\n",
            "Epoch 82/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9485 - accuracy: 0.6197 - val_loss: 1.0935 - val_accuracy: 0.6553\n",
            "Epoch 83/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9155 - accuracy: 0.6160 - val_loss: 1.0873 - val_accuracy: 0.6602\n",
            "Epoch 84/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9150 - accuracy: 0.6343 - val_loss: 1.0834 - val_accuracy: 0.6505\n",
            "Epoch 85/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9048 - accuracy: 0.6294 - val_loss: 1.0803 - val_accuracy: 0.6553\n",
            "Epoch 86/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8873 - accuracy: 0.6483 - val_loss: 1.0760 - val_accuracy: 0.6553\n",
            "Epoch 87/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8535 - accuracy: 0.6537 - val_loss: 1.0721 - val_accuracy: 0.6359\n",
            "Epoch 88/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8616 - accuracy: 0.6543 - val_loss: 1.0650 - val_accuracy: 0.6456\n",
            "Epoch 89/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8657 - accuracy: 0.6408 - val_loss: 1.0600 - val_accuracy: 0.6456\n",
            "Epoch 90/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8256 - accuracy: 0.6726 - val_loss: 1.0565 - val_accuracy: 0.6505\n",
            "Epoch 91/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8254 - accuracy: 0.6650 - val_loss: 1.0523 - val_accuracy: 0.6553\n",
            "Epoch 92/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8316 - accuracy: 0.6677 - val_loss: 1.0485 - val_accuracy: 0.6650\n",
            "Epoch 93/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7901 - accuracy: 0.6953 - val_loss: 1.0426 - val_accuracy: 0.6650\n",
            "Epoch 94/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8290 - accuracy: 0.6661 - val_loss: 1.0388 - val_accuracy: 0.6553\n",
            "Epoch 95/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8143 - accuracy: 0.6866 - val_loss: 1.0344 - val_accuracy: 0.6650\n",
            "Epoch 96/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 0.7847 - accuracy: 0.6888 - val_loss: 1.0307 - val_accuracy: 0.6602\n",
            "Epoch 97/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7672 - accuracy: 0.7055 - val_loss: 1.0261 - val_accuracy: 0.6650\n",
            "Epoch 98/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7568 - accuracy: 0.7017 - val_loss: 1.0206 - val_accuracy: 0.6650\n",
            "Epoch 99/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7802 - accuracy: 0.6780 - val_loss: 1.0165 - val_accuracy: 0.6553\n",
            "Epoch 100/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7786 - accuracy: 0.6839 - val_loss: 1.0114 - val_accuracy: 0.6602\n",
            "Epoch 101/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7492 - accuracy: 0.7044 - val_loss: 1.0063 - val_accuracy: 0.6505\n",
            "Epoch 102/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7220 - accuracy: 0.7217 - val_loss: 1.0026 - val_accuracy: 0.6796\n",
            "Epoch 103/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6981 - accuracy: 0.7325 - val_loss: 0.9985 - val_accuracy: 0.6699\n",
            "Epoch 104/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6999 - accuracy: 0.7298 - val_loss: 0.9969 - val_accuracy: 0.6845\n",
            "Epoch 105/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7126 - accuracy: 0.7184 - val_loss: 0.9920 - val_accuracy: 0.6845\n",
            "Epoch 106/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6847 - accuracy: 0.7373 - val_loss: 0.9875 - val_accuracy: 0.6796\n",
            "Epoch 107/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6654 - accuracy: 0.7406 - val_loss: 0.9830 - val_accuracy: 0.6796\n",
            "Epoch 108/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6802 - accuracy: 0.7449 - val_loss: 0.9785 - val_accuracy: 0.6699\n",
            "Epoch 109/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6487 - accuracy: 0.7492 - val_loss: 0.9745 - val_accuracy: 0.6699\n",
            "Epoch 110/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6927 - accuracy: 0.7346 - val_loss: 0.9682 - val_accuracy: 0.6748\n",
            "Epoch 111/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6339 - accuracy: 0.7665 - val_loss: 0.9626 - val_accuracy: 0.6650\n",
            "Epoch 112/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6249 - accuracy: 0.7648 - val_loss: 0.9589 - val_accuracy: 0.6699\n",
            "Epoch 113/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6560 - accuracy: 0.7416 - val_loss: 0.9532 - val_accuracy: 0.6650\n",
            "Epoch 114/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6227 - accuracy: 0.7594 - val_loss: 0.9512 - val_accuracy: 0.6748\n",
            "Epoch 115/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6518 - accuracy: 0.7460 - val_loss: 0.9450 - val_accuracy: 0.6748\n",
            "Epoch 116/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6307 - accuracy: 0.7589 - val_loss: 0.9426 - val_accuracy: 0.6602\n",
            "Epoch 117/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6015 - accuracy: 0.7762 - val_loss: 0.9387 - val_accuracy: 0.6650\n",
            "Epoch 118/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6186 - accuracy: 0.7659 - val_loss: 0.9354 - val_accuracy: 0.6699\n",
            "Epoch 119/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5732 - accuracy: 0.7864 - val_loss: 0.9300 - val_accuracy: 0.6748\n",
            "Epoch 120/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5859 - accuracy: 0.7805 - val_loss: 0.9257 - val_accuracy: 0.6748\n",
            "Epoch 121/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5646 - accuracy: 0.7896 - val_loss: 0.9208 - val_accuracy: 0.6748\n",
            "Epoch 122/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5718 - accuracy: 0.7810 - val_loss: 0.9174 - val_accuracy: 0.6845\n",
            "Epoch 123/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5679 - accuracy: 0.7794 - val_loss: 0.9158 - val_accuracy: 0.6650\n",
            "Epoch 124/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5557 - accuracy: 0.7956 - val_loss: 0.9113 - val_accuracy: 0.6748\n",
            "Epoch 125/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5561 - accuracy: 0.7999 - val_loss: 0.9081 - val_accuracy: 0.6748\n",
            "Epoch 126/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5241 - accuracy: 0.8047 - val_loss: 0.9034 - val_accuracy: 0.6650\n",
            "Epoch 127/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5555 - accuracy: 0.7940 - val_loss: 0.9003 - val_accuracy: 0.6748\n",
            "Epoch 128/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5498 - accuracy: 0.7923 - val_loss: 0.8975 - val_accuracy: 0.6748\n",
            "Epoch 129/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5068 - accuracy: 0.8188 - val_loss: 0.8943 - val_accuracy: 0.6699\n",
            "Epoch 130/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5207 - accuracy: 0.8101 - val_loss: 0.8918 - val_accuracy: 0.6699\n",
            "Epoch 131/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4968 - accuracy: 0.8198 - val_loss: 0.8875 - val_accuracy: 0.6650\n",
            "Epoch 132/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5045 - accuracy: 0.8069 - val_loss: 0.8837 - val_accuracy: 0.6699\n",
            "Epoch 133/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4953 - accuracy: 0.8193 - val_loss: 0.8810 - val_accuracy: 0.6699\n",
            "Epoch 134/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5167 - accuracy: 0.8047 - val_loss: 0.8783 - val_accuracy: 0.6748\n",
            "Epoch 135/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4713 - accuracy: 0.8376 - val_loss: 0.8755 - val_accuracy: 0.6796\n",
            "Epoch 136/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4853 - accuracy: 0.8252 - val_loss: 0.8720 - val_accuracy: 0.6748\n",
            "Epoch 137/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4703 - accuracy: 0.8269 - val_loss: 0.8691 - val_accuracy: 0.6796\n",
            "Epoch 138/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4881 - accuracy: 0.8155 - val_loss: 0.8654 - val_accuracy: 0.6845\n",
            "Epoch 139/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4607 - accuracy: 0.8323 - val_loss: 0.8621 - val_accuracy: 0.6845\n",
            "Epoch 140/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4231 - accuracy: 0.8533 - val_loss: 0.8586 - val_accuracy: 0.6845\n",
            "Epoch 141/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4471 - accuracy: 0.8328 - val_loss: 0.8554 - val_accuracy: 0.6796\n",
            "Epoch 142/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4420 - accuracy: 0.8420 - val_loss: 0.8516 - val_accuracy: 0.6796\n",
            "Epoch 143/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4053 - accuracy: 0.8587 - val_loss: 0.8515 - val_accuracy: 0.6796\n",
            "Epoch 144/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4115 - accuracy: 0.8581 - val_loss: 0.8494 - val_accuracy: 0.6796\n",
            "Epoch 145/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4399 - accuracy: 0.8333 - val_loss: 0.8472 - val_accuracy: 0.6893\n",
            "Epoch 146/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3968 - accuracy: 0.8646 - val_loss: 0.8468 - val_accuracy: 0.6845\n",
            "Epoch 147/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3919 - accuracy: 0.8722 - val_loss: 0.8452 - val_accuracy: 0.6796\n",
            "Epoch 148/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3928 - accuracy: 0.8581 - val_loss: 0.8428 - val_accuracy: 0.6796\n",
            "Epoch 149/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4118 - accuracy: 0.8522 - val_loss: 0.8397 - val_accuracy: 0.6845\n",
            "Epoch 150/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4096 - accuracy: 0.8517 - val_loss: 0.8366 - val_accuracy: 0.6845\n",
            "Epoch 151/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4054 - accuracy: 0.8581 - val_loss: 0.8338 - val_accuracy: 0.6942\n",
            "Epoch 152/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4009 - accuracy: 0.8565 - val_loss: 0.8321 - val_accuracy: 0.6845\n",
            "Epoch 153/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3766 - accuracy: 0.8732 - val_loss: 0.8286 - val_accuracy: 0.6942\n",
            "Epoch 154/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3827 - accuracy: 0.8684 - val_loss: 0.8263 - val_accuracy: 0.7039\n",
            "Epoch 155/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3723 - accuracy: 0.8700 - val_loss: 0.8269 - val_accuracy: 0.7039\n",
            "Epoch 156/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3659 - accuracy: 0.8689 - val_loss: 0.8246 - val_accuracy: 0.7039\n",
            "Epoch 157/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3606 - accuracy: 0.8770 - val_loss: 0.8214 - val_accuracy: 0.7039\n",
            "Epoch 158/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3585 - accuracy: 0.8727 - val_loss: 0.8180 - val_accuracy: 0.6942\n",
            "Epoch 159/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3373 - accuracy: 0.8786 - val_loss: 0.8150 - val_accuracy: 0.7039\n",
            "Epoch 160/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3397 - accuracy: 0.8840 - val_loss: 0.8140 - val_accuracy: 0.6990\n",
            "Epoch 161/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3556 - accuracy: 0.8727 - val_loss: 0.8122 - val_accuracy: 0.7039\n",
            "Epoch 162/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3321 - accuracy: 0.8808 - val_loss: 0.8095 - val_accuracy: 0.7087\n",
            "Epoch 163/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3239 - accuracy: 0.8932 - val_loss: 0.8089 - val_accuracy: 0.7039\n",
            "Epoch 164/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3385 - accuracy: 0.8851 - val_loss: 0.8082 - val_accuracy: 0.7039\n",
            "Epoch 165/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3378 - accuracy: 0.8792 - val_loss: 0.8067 - val_accuracy: 0.7087\n",
            "Epoch 166/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3241 - accuracy: 0.8813 - val_loss: 0.8032 - val_accuracy: 0.6942\n",
            "Epoch 167/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3237 - accuracy: 0.8824 - val_loss: 0.8017 - val_accuracy: 0.6942\n",
            "Epoch 168/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2991 - accuracy: 0.9024 - val_loss: 0.7999 - val_accuracy: 0.7136\n",
            "Epoch 169/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3025 - accuracy: 0.8867 - val_loss: 0.7984 - val_accuracy: 0.7136\n",
            "Epoch 170/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3282 - accuracy: 0.8873 - val_loss: 0.7976 - val_accuracy: 0.7087\n",
            "Epoch 171/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3272 - accuracy: 0.8824 - val_loss: 0.7978 - val_accuracy: 0.7136\n",
            "Epoch 172/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2972 - accuracy: 0.8997 - val_loss: 0.7960 - val_accuracy: 0.7136\n",
            "Epoch 173/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2936 - accuracy: 0.8964 - val_loss: 0.7930 - val_accuracy: 0.7136\n",
            "Epoch 174/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3097 - accuracy: 0.8900 - val_loss: 0.7910 - val_accuracy: 0.7087\n",
            "Epoch 175/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2934 - accuracy: 0.8986 - val_loss: 0.7905 - val_accuracy: 0.7087\n",
            "Epoch 176/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2983 - accuracy: 0.8921 - val_loss: 0.7893 - val_accuracy: 0.7087\n",
            "Epoch 177/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2875 - accuracy: 0.8959 - val_loss: 0.7871 - val_accuracy: 0.7136\n",
            "Epoch 178/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3027 - accuracy: 0.8894 - val_loss: 0.7837 - val_accuracy: 0.7136\n",
            "Epoch 179/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2882 - accuracy: 0.8975 - val_loss: 0.7825 - val_accuracy: 0.7136\n",
            "Epoch 180/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2594 - accuracy: 0.9094 - val_loss: 0.7820 - val_accuracy: 0.7136\n",
            "Epoch 181/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2712 - accuracy: 0.9013 - val_loss: 0.7813 - val_accuracy: 0.7184\n",
            "Epoch 182/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2814 - accuracy: 0.9035 - val_loss: 0.7820 - val_accuracy: 0.7087\n",
            "Epoch 183/1024\n",
            "58/58 [==============================] - ETA: 0s - loss: 0.2449 - accuracy: 0.9159\n",
            "Epoch 00183: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2449 - accuracy: 0.9159 - val_loss: 0.7825 - val_accuracy: 0.6990\n",
            "Epoch 184/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2546 - accuracy: 0.9110 - val_loss: 0.7806 - val_accuracy: 0.7039\n",
            "Epoch 185/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2470 - accuracy: 0.9148 - val_loss: 0.7819 - val_accuracy: 0.7087\n",
            "Epoch 186/1024\n",
            "51/58 [=========================>....] - ETA: 0s - loss: 0.2812 - accuracy: 0.9013\n",
            "Epoch 00186: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2779 - accuracy: 0.9018 - val_loss: 0.7810 - val_accuracy: 0.7136\n",
            "Epoch 187/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2392 - accuracy: 0.9121 - val_loss: 0.7822 - val_accuracy: 0.7136\n",
            "Epoch 188/1024\n",
            "56/58 [===========================>..] - ETA: 0s - loss: 0.2446 - accuracy: 0.9213\n",
            "Epoch 00188: Reducing Max LR on Plateau: new max lr will be 1.25e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2445 - accuracy: 0.9213 - val_loss: 0.7808 - val_accuracy: 0.7136\n",
            "Epoch 189/1024\n",
            "52/58 [=========================>....] - ETA: 0s - loss: 0.2639 - accuracy: 0.9038Restoring model weights from the end of the best epoch: 184.\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2685 - accuracy: 0.9029 - val_loss: 0.7814 - val_accuracy: 0.7136\n",
            "Epoch 189: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.47      0.47      0.47        19\n",
            "      level2       0.73      0.70      0.72        63\n",
            "      level3       0.69      0.73      0.71        64\n",
            "      level4       0.76      0.75      0.76        60\n",
            "\n",
            "    accuracy                           0.70       206\n",
            "   macro avg       0.67      0.66      0.66       206\n",
            "weighted avg       0.70      0.70      0.70       206\n",
            "\n",
            "-------------------------------------------------------run 2\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.7733 - accuracy: 0.2540 - val_loss: 1.3735 - val_accuracy: 0.3058\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.7485 - accuracy: 0.2718 - val_loss: 1.3703 - val_accuracy: 0.3155\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6888 - accuracy: 0.2718 - val_loss: 1.3691 - val_accuracy: 0.3544\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6374 - accuracy: 0.2891 - val_loss: 1.3679 - val_accuracy: 0.3107\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6767 - accuracy: 0.2853 - val_loss: 1.3672 - val_accuracy: 0.3301\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5909 - accuracy: 0.3155 - val_loss: 1.3649 - val_accuracy: 0.3301\n",
            "Epoch 7/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5710 - accuracy: 0.3118 - val_loss: 1.3608 - val_accuracy: 0.3641\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5487 - accuracy: 0.3096 - val_loss: 1.3559 - val_accuracy: 0.3544\n",
            "Epoch 9/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5254 - accuracy: 0.3112 - val_loss: 1.3494 - val_accuracy: 0.3786\n",
            "Epoch 10/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5689 - accuracy: 0.2999 - val_loss: 1.3418 - val_accuracy: 0.3883\n",
            "Epoch 11/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5235 - accuracy: 0.3161 - val_loss: 1.3333 - val_accuracy: 0.4126\n",
            "Epoch 12/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.5179 - accuracy: 0.3155 - val_loss: 1.3240 - val_accuracy: 0.4320\n",
            "Epoch 13/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5153 - accuracy: 0.3269 - val_loss: 1.3172 - val_accuracy: 0.4272\n",
            "Epoch 14/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4944 - accuracy: 0.3139 - val_loss: 1.3132 - val_accuracy: 0.4417\n",
            "Epoch 15/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4934 - accuracy: 0.3242 - val_loss: 1.3084 - val_accuracy: 0.4612\n",
            "Epoch 16/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4753 - accuracy: 0.3269 - val_loss: 1.3049 - val_accuracy: 0.4757\n",
            "Epoch 17/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4456 - accuracy: 0.3614 - val_loss: 1.3032 - val_accuracy: 0.4854\n",
            "Epoch 18/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4615 - accuracy: 0.3495 - val_loss: 1.3001 - val_accuracy: 0.4757\n",
            "Epoch 19/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4451 - accuracy: 0.3382 - val_loss: 1.2973 - val_accuracy: 0.4854\n",
            "Epoch 20/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4509 - accuracy: 0.3614 - val_loss: 1.2941 - val_accuracy: 0.4757\n",
            "Epoch 21/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3900 - accuracy: 0.3797 - val_loss: 1.2934 - val_accuracy: 0.4903\n",
            "Epoch 22/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3738 - accuracy: 0.3786 - val_loss: 1.2915 - val_accuracy: 0.4806\n",
            "Epoch 23/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3636 - accuracy: 0.3803 - val_loss: 1.2870 - val_accuracy: 0.4951\n",
            "Epoch 24/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3968 - accuracy: 0.3625 - val_loss: 1.2836 - val_accuracy: 0.4951\n",
            "Epoch 25/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3825 - accuracy: 0.3700 - val_loss: 1.2827 - val_accuracy: 0.5049\n",
            "Epoch 26/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3860 - accuracy: 0.3759 - val_loss: 1.2810 - val_accuracy: 0.5097\n",
            "Epoch 27/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3812 - accuracy: 0.3770 - val_loss: 1.2784 - val_accuracy: 0.5243\n",
            "Epoch 28/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3410 - accuracy: 0.3905 - val_loss: 1.2762 - val_accuracy: 0.5097\n",
            "Epoch 29/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3617 - accuracy: 0.3797 - val_loss: 1.2740 - val_accuracy: 0.5000\n",
            "Epoch 30/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3484 - accuracy: 0.3732 - val_loss: 1.2714 - val_accuracy: 0.5097\n",
            "Epoch 31/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3061 - accuracy: 0.4099 - val_loss: 1.2694 - val_accuracy: 0.5000\n",
            "Epoch 32/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3371 - accuracy: 0.3883 - val_loss: 1.2664 - val_accuracy: 0.4951\n",
            "Epoch 33/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3163 - accuracy: 0.3964 - val_loss: 1.2622 - val_accuracy: 0.5097\n",
            "Epoch 34/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3229 - accuracy: 0.3964 - val_loss: 1.2602 - val_accuracy: 0.5243\n",
            "Epoch 35/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3177 - accuracy: 0.4072 - val_loss: 1.2585 - val_accuracy: 0.5194\n",
            "Epoch 36/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2797 - accuracy: 0.4229 - val_loss: 1.2545 - val_accuracy: 0.5243\n",
            "Epoch 37/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3048 - accuracy: 0.4159 - val_loss: 1.2513 - val_accuracy: 0.5291\n",
            "Epoch 38/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2967 - accuracy: 0.4239 - val_loss: 1.2466 - val_accuracy: 0.5437\n",
            "Epoch 39/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2839 - accuracy: 0.4164 - val_loss: 1.2437 - val_accuracy: 0.5485\n",
            "Epoch 40/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2596 - accuracy: 0.4423 - val_loss: 1.2425 - val_accuracy: 0.5388\n",
            "Epoch 41/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2469 - accuracy: 0.4439 - val_loss: 1.2392 - val_accuracy: 0.5340\n",
            "Epoch 42/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2387 - accuracy: 0.4563 - val_loss: 1.2383 - val_accuracy: 0.5437\n",
            "Epoch 43/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2575 - accuracy: 0.4364 - val_loss: 1.2343 - val_accuracy: 0.5388\n",
            "Epoch 44/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2451 - accuracy: 0.4304 - val_loss: 1.2302 - val_accuracy: 0.5485\n",
            "Epoch 45/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2547 - accuracy: 0.4353 - val_loss: 1.2263 - val_accuracy: 0.5583\n",
            "Epoch 46/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2292 - accuracy: 0.4412 - val_loss: 1.2246 - val_accuracy: 0.5583\n",
            "Epoch 47/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2234 - accuracy: 0.4471 - val_loss: 1.2223 - val_accuracy: 0.5680\n",
            "Epoch 48/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1990 - accuracy: 0.4622 - val_loss: 1.2184 - val_accuracy: 0.5825\n",
            "Epoch 49/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 1.2073 - accuracy: 0.4720 - val_loss: 1.2163 - val_accuracy: 0.5728\n",
            "Epoch 50/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1904 - accuracy: 0.4660 - val_loss: 1.2122 - val_accuracy: 0.5680\n",
            "Epoch 51/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1723 - accuracy: 0.4844 - val_loss: 1.2110 - val_accuracy: 0.5825\n",
            "Epoch 52/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1683 - accuracy: 0.4822 - val_loss: 1.2069 - val_accuracy: 0.5777\n",
            "Epoch 53/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.1446 - accuracy: 0.4941 - val_loss: 1.2003 - val_accuracy: 0.5825\n",
            "Epoch 54/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1684 - accuracy: 0.4827 - val_loss: 1.1981 - val_accuracy: 0.5971\n",
            "Epoch 55/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1540 - accuracy: 0.4838 - val_loss: 1.1949 - val_accuracy: 0.5971\n",
            "Epoch 56/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1298 - accuracy: 0.4984 - val_loss: 1.1908 - val_accuracy: 0.5922\n",
            "Epoch 57/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1213 - accuracy: 0.5119 - val_loss: 1.1863 - val_accuracy: 0.6019\n",
            "Epoch 58/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1159 - accuracy: 0.5183 - val_loss: 1.1838 - val_accuracy: 0.5874\n",
            "Epoch 59/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1126 - accuracy: 0.5124 - val_loss: 1.1790 - val_accuracy: 0.5971\n",
            "Epoch 60/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1293 - accuracy: 0.5059 - val_loss: 1.1775 - val_accuracy: 0.5971\n",
            "Epoch 61/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0851 - accuracy: 0.5275 - val_loss: 1.1730 - val_accuracy: 0.5922\n",
            "Epoch 62/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0750 - accuracy: 0.5210 - val_loss: 1.1686 - val_accuracy: 0.5971\n",
            "Epoch 63/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0706 - accuracy: 0.5259 - val_loss: 1.1625 - val_accuracy: 0.6019\n",
            "Epoch 64/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0783 - accuracy: 0.5286 - val_loss: 1.1596 - val_accuracy: 0.6165\n",
            "Epoch 65/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0696 - accuracy: 0.5415 - val_loss: 1.1554 - val_accuracy: 0.6214\n",
            "Epoch 66/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.0418 - accuracy: 0.5534 - val_loss: 1.1505 - val_accuracy: 0.6262\n",
            "Epoch 67/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0470 - accuracy: 0.5588 - val_loss: 1.1478 - val_accuracy: 0.6214\n",
            "Epoch 68/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0195 - accuracy: 0.5545 - val_loss: 1.1423 - val_accuracy: 0.6165\n",
            "Epoch 69/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0200 - accuracy: 0.5636 - val_loss: 1.1377 - val_accuracy: 0.6117\n",
            "Epoch 70/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0074 - accuracy: 0.5663 - val_loss: 1.1322 - val_accuracy: 0.6311\n",
            "Epoch 71/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0101 - accuracy: 0.5545 - val_loss: 1.1267 - val_accuracy: 0.6311\n",
            "Epoch 72/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0051 - accuracy: 0.5653 - val_loss: 1.1210 - val_accuracy: 0.6214\n",
            "Epoch 73/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9946 - accuracy: 0.5734 - val_loss: 1.1154 - val_accuracy: 0.6311\n",
            "Epoch 74/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9806 - accuracy: 0.5739 - val_loss: 1.1104 - val_accuracy: 0.6262\n",
            "Epoch 75/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9756 - accuracy: 0.5831 - val_loss: 1.1063 - val_accuracy: 0.6262\n",
            "Epoch 76/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9637 - accuracy: 0.5955 - val_loss: 1.1026 - val_accuracy: 0.6262\n",
            "Epoch 77/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9695 - accuracy: 0.5992 - val_loss: 1.0969 - val_accuracy: 0.6214\n",
            "Epoch 78/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9191 - accuracy: 0.6160 - val_loss: 1.0941 - val_accuracy: 0.6262\n",
            "Epoch 79/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9695 - accuracy: 0.5868 - val_loss: 1.0871 - val_accuracy: 0.6408\n",
            "Epoch 80/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.9196 - accuracy: 0.6316 - val_loss: 1.0824 - val_accuracy: 0.6311\n",
            "Epoch 81/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9374 - accuracy: 0.6133 - val_loss: 1.0789 - val_accuracy: 0.6359\n",
            "Epoch 82/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8951 - accuracy: 0.6170 - val_loss: 1.0731 - val_accuracy: 0.6408\n",
            "Epoch 83/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9119 - accuracy: 0.6208 - val_loss: 1.0682 - val_accuracy: 0.6408\n",
            "Epoch 84/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8835 - accuracy: 0.6321 - val_loss: 1.0652 - val_accuracy: 0.6456\n",
            "Epoch 85/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8534 - accuracy: 0.6424 - val_loss: 1.0607 - val_accuracy: 0.6602\n",
            "Epoch 86/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8684 - accuracy: 0.6478 - val_loss: 1.0541 - val_accuracy: 0.6553\n",
            "Epoch 87/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8128 - accuracy: 0.6823 - val_loss: 1.0486 - val_accuracy: 0.6796\n",
            "Epoch 88/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8539 - accuracy: 0.6548 - val_loss: 1.0453 - val_accuracy: 0.6748\n",
            "Epoch 89/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.8271 - accuracy: 0.6683 - val_loss: 1.0397 - val_accuracy: 0.6602\n",
            "Epoch 90/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8523 - accuracy: 0.6543 - val_loss: 1.0369 - val_accuracy: 0.6505\n",
            "Epoch 91/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8443 - accuracy: 0.6618 - val_loss: 1.0308 - val_accuracy: 0.6553\n",
            "Epoch 92/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8278 - accuracy: 0.6629 - val_loss: 1.0230 - val_accuracy: 0.6699\n",
            "Epoch 93/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8039 - accuracy: 0.6769 - val_loss: 1.0196 - val_accuracy: 0.6699\n",
            "Epoch 94/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7917 - accuracy: 0.6866 - val_loss: 1.0129 - val_accuracy: 0.6699\n",
            "Epoch 95/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7666 - accuracy: 0.7152 - val_loss: 1.0085 - val_accuracy: 0.6699\n",
            "Epoch 96/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7851 - accuracy: 0.6807 - val_loss: 1.0054 - val_accuracy: 0.6699\n",
            "Epoch 97/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7831 - accuracy: 0.6888 - val_loss: 0.9997 - val_accuracy: 0.6796\n",
            "Epoch 98/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7372 - accuracy: 0.7131 - val_loss: 0.9943 - val_accuracy: 0.6699\n",
            "Epoch 99/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7382 - accuracy: 0.7087 - val_loss: 0.9910 - val_accuracy: 0.6748\n",
            "Epoch 100/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7661 - accuracy: 0.6893 - val_loss: 0.9845 - val_accuracy: 0.6699\n",
            "Epoch 101/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7311 - accuracy: 0.7157 - val_loss: 0.9824 - val_accuracy: 0.6748\n",
            "Epoch 102/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7213 - accuracy: 0.7125 - val_loss: 0.9802 - val_accuracy: 0.6748\n",
            "Epoch 103/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7280 - accuracy: 0.7109 - val_loss: 0.9752 - val_accuracy: 0.6748\n",
            "Epoch 104/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7301 - accuracy: 0.7190 - val_loss: 0.9696 - val_accuracy: 0.6748\n",
            "Epoch 105/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7143 - accuracy: 0.7244 - val_loss: 0.9644 - val_accuracy: 0.6748\n",
            "Epoch 106/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6944 - accuracy: 0.7335 - val_loss: 0.9623 - val_accuracy: 0.6796\n",
            "Epoch 107/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7008 - accuracy: 0.7265 - val_loss: 0.9556 - val_accuracy: 0.6893\n",
            "Epoch 108/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6701 - accuracy: 0.7438 - val_loss: 0.9518 - val_accuracy: 0.6748\n",
            "Epoch 109/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6766 - accuracy: 0.7352 - val_loss: 0.9493 - val_accuracy: 0.6650\n",
            "Epoch 110/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6451 - accuracy: 0.7481 - val_loss: 0.9449 - val_accuracy: 0.6796\n",
            "Epoch 111/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.6631 - accuracy: 0.7481 - val_loss: 0.9401 - val_accuracy: 0.6893\n",
            "Epoch 112/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6448 - accuracy: 0.7589 - val_loss: 0.9345 - val_accuracy: 0.6796\n",
            "Epoch 113/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6401 - accuracy: 0.7513 - val_loss: 0.9295 - val_accuracy: 0.6845\n",
            "Epoch 114/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6490 - accuracy: 0.7557 - val_loss: 0.9256 - val_accuracy: 0.6796\n",
            "Epoch 115/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6194 - accuracy: 0.7632 - val_loss: 0.9224 - val_accuracy: 0.6796\n",
            "Epoch 116/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6074 - accuracy: 0.7767 - val_loss: 0.9171 - val_accuracy: 0.6845\n",
            "Epoch 117/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5990 - accuracy: 0.7691 - val_loss: 0.9128 - val_accuracy: 0.6796\n",
            "Epoch 118/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5995 - accuracy: 0.7681 - val_loss: 0.9081 - val_accuracy: 0.6845\n",
            "Epoch 119/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.6130 - accuracy: 0.7546 - val_loss: 0.9026 - val_accuracy: 0.6893\n",
            "Epoch 120/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5596 - accuracy: 0.7859 - val_loss: 0.9015 - val_accuracy: 0.6893\n",
            "Epoch 121/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5717 - accuracy: 0.7821 - val_loss: 0.8968 - val_accuracy: 0.6942\n",
            "Epoch 122/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5571 - accuracy: 0.7929 - val_loss: 0.8942 - val_accuracy: 0.6942\n",
            "Epoch 123/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5806 - accuracy: 0.7826 - val_loss: 0.8945 - val_accuracy: 0.6845\n",
            "Epoch 124/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5334 - accuracy: 0.7967 - val_loss: 0.8908 - val_accuracy: 0.6893\n",
            "Epoch 125/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5663 - accuracy: 0.7864 - val_loss: 0.8867 - val_accuracy: 0.6893\n",
            "Epoch 126/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5212 - accuracy: 0.8064 - val_loss: 0.8807 - val_accuracy: 0.6893\n",
            "Epoch 127/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5356 - accuracy: 0.7999 - val_loss: 0.8769 - val_accuracy: 0.6893\n",
            "Epoch 128/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5136 - accuracy: 0.8096 - val_loss: 0.8731 - val_accuracy: 0.6942\n",
            "Epoch 129/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4912 - accuracy: 0.8220 - val_loss: 0.8721 - val_accuracy: 0.6796\n",
            "Epoch 130/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4969 - accuracy: 0.8204 - val_loss: 0.8676 - val_accuracy: 0.6845\n",
            "Epoch 131/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5067 - accuracy: 0.8166 - val_loss: 0.8636 - val_accuracy: 0.6893\n",
            "Epoch 132/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4974 - accuracy: 0.8209 - val_loss: 0.8597 - val_accuracy: 0.6893\n",
            "Epoch 133/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4826 - accuracy: 0.8220 - val_loss: 0.8544 - val_accuracy: 0.6893\n",
            "Epoch 134/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4932 - accuracy: 0.8182 - val_loss: 0.8510 - val_accuracy: 0.6942\n",
            "Epoch 135/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4734 - accuracy: 0.8296 - val_loss: 0.8495 - val_accuracy: 0.6893\n",
            "Epoch 136/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4961 - accuracy: 0.8128 - val_loss: 0.8480 - val_accuracy: 0.6893\n",
            "Epoch 137/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4502 - accuracy: 0.8366 - val_loss: 0.8473 - val_accuracy: 0.6845\n",
            "Epoch 138/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4700 - accuracy: 0.8366 - val_loss: 0.8455 - val_accuracy: 0.6796\n",
            "Epoch 139/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4515 - accuracy: 0.8376 - val_loss: 0.8417 - val_accuracy: 0.6845\n",
            "Epoch 140/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4455 - accuracy: 0.8366 - val_loss: 0.8413 - val_accuracy: 0.6748\n",
            "Epoch 141/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4764 - accuracy: 0.8166 - val_loss: 0.8382 - val_accuracy: 0.6796\n",
            "Epoch 142/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4172 - accuracy: 0.8522 - val_loss: 0.8331 - val_accuracy: 0.6748\n",
            "Epoch 143/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4223 - accuracy: 0.8554 - val_loss: 0.8320 - val_accuracy: 0.6796\n",
            "Epoch 144/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4207 - accuracy: 0.8479 - val_loss: 0.8276 - val_accuracy: 0.6845\n",
            "Epoch 145/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4255 - accuracy: 0.8420 - val_loss: 0.8268 - val_accuracy: 0.6748\n",
            "Epoch 146/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4195 - accuracy: 0.8484 - val_loss: 0.8208 - val_accuracy: 0.6748\n",
            "Epoch 147/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4066 - accuracy: 0.8581 - val_loss: 0.8196 - val_accuracy: 0.6748\n",
            "Epoch 148/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4000 - accuracy: 0.8549 - val_loss: 0.8159 - val_accuracy: 0.6699\n",
            "Epoch 149/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4025 - accuracy: 0.8457 - val_loss: 0.8116 - val_accuracy: 0.6796\n",
            "Epoch 150/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3938 - accuracy: 0.8625 - val_loss: 0.8096 - val_accuracy: 0.6845\n",
            "Epoch 151/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3892 - accuracy: 0.8544 - val_loss: 0.8068 - val_accuracy: 0.6796\n",
            "Epoch 152/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3957 - accuracy: 0.8625 - val_loss: 0.8052 - val_accuracy: 0.6699\n",
            "Epoch 153/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3555 - accuracy: 0.8743 - val_loss: 0.8047 - val_accuracy: 0.6699\n",
            "Epoch 154/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3728 - accuracy: 0.8587 - val_loss: 0.8027 - val_accuracy: 0.6699\n",
            "Epoch 155/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3806 - accuracy: 0.8749 - val_loss: 0.8032 - val_accuracy: 0.6796\n",
            "Epoch 156/1024\n",
            "58/58 [==============================] - 1s 8ms/step - loss: 0.3953 - accuracy: 0.8533 - val_loss: 0.8016 - val_accuracy: 0.6748\n",
            "Epoch 157/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3851 - accuracy: 0.8652 - val_loss: 0.8028 - val_accuracy: 0.6748\n",
            "Epoch 158/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3547 - accuracy: 0.8700 - val_loss: 0.7992 - val_accuracy: 0.6699\n",
            "Epoch 159/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3728 - accuracy: 0.8695 - val_loss: 0.7975 - val_accuracy: 0.6699\n",
            "Epoch 160/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3612 - accuracy: 0.8700 - val_loss: 0.7944 - val_accuracy: 0.6699\n",
            "Epoch 161/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3517 - accuracy: 0.8754 - val_loss: 0.7940 - val_accuracy: 0.6748\n",
            "Epoch 162/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3439 - accuracy: 0.8830 - val_loss: 0.7924 - val_accuracy: 0.6845\n",
            "Epoch 163/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3273 - accuracy: 0.8889 - val_loss: 0.7896 - val_accuracy: 0.6796\n",
            "Epoch 164/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3479 - accuracy: 0.8689 - val_loss: 0.7909 - val_accuracy: 0.6748\n",
            "Epoch 165/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3229 - accuracy: 0.8857 - val_loss: 0.7878 - val_accuracy: 0.6893\n",
            "Epoch 166/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3210 - accuracy: 0.8883 - val_loss: 0.7856 - val_accuracy: 0.6845\n",
            "Epoch 167/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3212 - accuracy: 0.8803 - val_loss: 0.7838 - val_accuracy: 0.6845\n",
            "Epoch 168/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3205 - accuracy: 0.8910 - val_loss: 0.7839 - val_accuracy: 0.6942\n",
            "Epoch 169/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3170 - accuracy: 0.8900 - val_loss: 0.7829 - val_accuracy: 0.6942\n",
            "Epoch 170/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3179 - accuracy: 0.8776 - val_loss: 0.7815 - val_accuracy: 0.6893\n",
            "Epoch 171/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3017 - accuracy: 0.8981 - val_loss: 0.7781 - val_accuracy: 0.6845\n",
            "Epoch 172/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3104 - accuracy: 0.8867 - val_loss: 0.7759 - val_accuracy: 0.6990\n",
            "Epoch 173/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2959 - accuracy: 0.8975 - val_loss: 0.7750 - val_accuracy: 0.6990\n",
            "Epoch 174/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2866 - accuracy: 0.9013 - val_loss: 0.7741 - val_accuracy: 0.6893\n",
            "Epoch 175/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2880 - accuracy: 0.9013 - val_loss: 0.7729 - val_accuracy: 0.6942\n",
            "Epoch 176/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3085 - accuracy: 0.8975 - val_loss: 0.7705 - val_accuracy: 0.6942\n",
            "Epoch 177/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2907 - accuracy: 0.8975 - val_loss: 0.7674 - val_accuracy: 0.6942\n",
            "Epoch 178/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.2749 - accuracy: 0.9061 - val_loss: 0.7679 - val_accuracy: 0.6893\n",
            "Epoch 179/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3003 - accuracy: 0.8921 - val_loss: 0.7672 - val_accuracy: 0.6893\n",
            "Epoch 180/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2612 - accuracy: 0.9126 - val_loss: 0.7675 - val_accuracy: 0.6845\n",
            "Epoch 181/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2848 - accuracy: 0.9024 - val_loss: 0.7643 - val_accuracy: 0.6845\n",
            "Epoch 182/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2662 - accuracy: 0.9078 - val_loss: 0.7641 - val_accuracy: 0.6893\n",
            "Epoch 183/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2608 - accuracy: 0.9094 - val_loss: 0.7609 - val_accuracy: 0.6796\n",
            "Epoch 184/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.2809 - accuracy: 0.8975 - val_loss: 0.7604 - val_accuracy: 0.6796\n",
            "Epoch 185/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2655 - accuracy: 0.9088 - val_loss: 0.7578 - val_accuracy: 0.6893\n",
            "Epoch 186/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2439 - accuracy: 0.9148 - val_loss: 0.7569 - val_accuracy: 0.6845\n",
            "Epoch 187/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2840 - accuracy: 0.9013 - val_loss: 0.7557 - val_accuracy: 0.6893\n",
            "Epoch 188/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2461 - accuracy: 0.9191 - val_loss: 0.7553 - val_accuracy: 0.6893\n",
            "Epoch 189/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.2664 - accuracy: 0.9083 - val_loss: 0.7542 - val_accuracy: 0.6845\n",
            "Epoch 190/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2492 - accuracy: 0.9094 - val_loss: 0.7519 - val_accuracy: 0.6893\n",
            "Epoch 191/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2595 - accuracy: 0.9035 - val_loss: 0.7507 - val_accuracy: 0.6942\n",
            "Epoch 192/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.2512 - accuracy: 0.9110 - val_loss: 0.7506 - val_accuracy: 0.6942\n",
            "Epoch 193/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2502 - accuracy: 0.9137 - val_loss: 0.7497 - val_accuracy: 0.6990\n",
            "Epoch 194/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2382 - accuracy: 0.9234 - val_loss: 0.7502 - val_accuracy: 0.6990\n",
            "Epoch 195/1024\n",
            "52/58 [=========================>....] - ETA: 0s - loss: 0.2452 - accuracy: 0.9183\n",
            "Epoch 00195: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2417 - accuracy: 0.9213 - val_loss: 0.7509 - val_accuracy: 0.6990\n",
            "Epoch 196/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2315 - accuracy: 0.9234 - val_loss: 0.7510 - val_accuracy: 0.6990\n",
            "Epoch 197/1024\n",
            "56/58 [===========================>..] - ETA: 0s - loss: 0.2254 - accuracy: 0.9280\n",
            "Epoch 00197: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2246 - accuracy: 0.9283 - val_loss: 0.7508 - val_accuracy: 0.6990\n",
            "Epoch 198/1024\n",
            "57/58 [============================>.] - ETA: 0s - loss: 0.2513 - accuracy: 0.9172Restoring model weights from the end of the best epoch: 193.\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2512 - accuracy: 0.9175 - val_loss: 0.7521 - val_accuracy: 0.6942\n",
            "Epoch 198: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.46      0.58      0.51        19\n",
            "      level2       0.72      0.68      0.70        63\n",
            "      level3       0.67      0.75      0.71        64\n",
            "      level4       0.84      0.70      0.76        60\n",
            "\n",
            "    accuracy                           0.70       206\n",
            "   macro avg       0.67      0.68      0.67       206\n",
            "weighted avg       0.71      0.70      0.70       206\n",
            "\n",
            "-------------------------------------------------------run 3\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.7333 - accuracy: 0.2616 - val_loss: 1.3740 - val_accuracy: 0.3058\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.7155 - accuracy: 0.2762 - val_loss: 1.3695 - val_accuracy: 0.3204\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 1.6924 - accuracy: 0.2843 - val_loss: 1.3685 - val_accuracy: 0.3398\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.6819 - accuracy: 0.2794 - val_loss: 1.3679 - val_accuracy: 0.3495\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 1.6473 - accuracy: 0.2945 - val_loss: 1.3669 - val_accuracy: 0.3252\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6170 - accuracy: 0.2983 - val_loss: 1.3644 - val_accuracy: 0.3544\n",
            "Epoch 7/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5893 - accuracy: 0.3037 - val_loss: 1.3590 - val_accuracy: 0.3592\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5496 - accuracy: 0.3177 - val_loss: 1.3536 - val_accuracy: 0.3738\n",
            "Epoch 9/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5283 - accuracy: 0.3306 - val_loss: 1.3467 - val_accuracy: 0.3932\n",
            "Epoch 10/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.5793 - accuracy: 0.2988 - val_loss: 1.3385 - val_accuracy: 0.4029\n",
            "Epoch 11/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5023 - accuracy: 0.3236 - val_loss: 1.3288 - val_accuracy: 0.4029\n",
            "Epoch 12/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.5159 - accuracy: 0.3350 - val_loss: 1.3192 - val_accuracy: 0.4417\n",
            "Epoch 13/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5168 - accuracy: 0.3166 - val_loss: 1.3117 - val_accuracy: 0.4466\n",
            "Epoch 14/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4937 - accuracy: 0.3350 - val_loss: 1.3069 - val_accuracy: 0.4515\n",
            "Epoch 15/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4612 - accuracy: 0.3333 - val_loss: 1.3010 - val_accuracy: 0.4709\n",
            "Epoch 16/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4479 - accuracy: 0.3457 - val_loss: 1.2981 - val_accuracy: 0.4660\n",
            "Epoch 17/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.4676 - accuracy: 0.3366 - val_loss: 1.2938 - val_accuracy: 0.4612\n",
            "Epoch 18/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.5051 - accuracy: 0.3252 - val_loss: 1.2917 - val_accuracy: 0.4757\n",
            "Epoch 19/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4728 - accuracy: 0.3414 - val_loss: 1.2886 - val_accuracy: 0.4563\n",
            "Epoch 20/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4140 - accuracy: 0.3635 - val_loss: 1.2858 - val_accuracy: 0.4612\n",
            "Epoch 21/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4255 - accuracy: 0.3565 - val_loss: 1.2835 - val_accuracy: 0.4709\n",
            "Epoch 22/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4384 - accuracy: 0.3495 - val_loss: 1.2831 - val_accuracy: 0.4903\n",
            "Epoch 23/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3898 - accuracy: 0.3673 - val_loss: 1.2803 - val_accuracy: 0.4757\n",
            "Epoch 24/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3830 - accuracy: 0.3630 - val_loss: 1.2779 - val_accuracy: 0.4951\n",
            "Epoch 25/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3966 - accuracy: 0.3684 - val_loss: 1.2760 - val_accuracy: 0.4854\n",
            "Epoch 26/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3581 - accuracy: 0.3695 - val_loss: 1.2735 - val_accuracy: 0.4951\n",
            "Epoch 27/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3936 - accuracy: 0.3824 - val_loss: 1.2720 - val_accuracy: 0.4951\n",
            "Epoch 28/1024\n",
            "58/58 [==============================] - 1s 8ms/step - loss: 1.3437 - accuracy: 0.3754 - val_loss: 1.2679 - val_accuracy: 0.5049\n",
            "Epoch 29/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.3360 - accuracy: 0.3851 - val_loss: 1.2650 - val_accuracy: 0.5000\n",
            "Epoch 30/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3334 - accuracy: 0.3916 - val_loss: 1.2623 - val_accuracy: 0.5194\n",
            "Epoch 31/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3483 - accuracy: 0.3954 - val_loss: 1.2599 - val_accuracy: 0.5146\n",
            "Epoch 32/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3190 - accuracy: 0.3937 - val_loss: 1.2577 - val_accuracy: 0.5000\n",
            "Epoch 33/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3184 - accuracy: 0.3862 - val_loss: 1.2551 - val_accuracy: 0.5000\n",
            "Epoch 34/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.3108 - accuracy: 0.4078 - val_loss: 1.2533 - val_accuracy: 0.5097\n",
            "Epoch 35/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3061 - accuracy: 0.4137 - val_loss: 1.2504 - val_accuracy: 0.5049\n",
            "Epoch 36/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3017 - accuracy: 0.4126 - val_loss: 1.2485 - val_accuracy: 0.5194\n",
            "Epoch 37/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2894 - accuracy: 0.4088 - val_loss: 1.2472 - val_accuracy: 0.5194\n",
            "Epoch 38/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2719 - accuracy: 0.4304 - val_loss: 1.2448 - val_accuracy: 0.5097\n",
            "Epoch 39/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2728 - accuracy: 0.4342 - val_loss: 1.2402 - val_accuracy: 0.5146\n",
            "Epoch 40/1024\n",
            "58/58 [==============================] - 1s 8ms/step - loss: 1.2693 - accuracy: 0.4132 - val_loss: 1.2383 - val_accuracy: 0.5146\n",
            "Epoch 41/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2569 - accuracy: 0.4417 - val_loss: 1.2349 - val_accuracy: 0.5146\n",
            "Epoch 42/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2239 - accuracy: 0.4434 - val_loss: 1.2313 - val_accuracy: 0.5243\n",
            "Epoch 43/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2503 - accuracy: 0.4272 - val_loss: 1.2277 - val_accuracy: 0.5388\n",
            "Epoch 44/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2297 - accuracy: 0.4558 - val_loss: 1.2243 - val_accuracy: 0.5437\n",
            "Epoch 45/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1965 - accuracy: 0.4639 - val_loss: 1.2199 - val_accuracy: 0.5485\n",
            "Epoch 46/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2325 - accuracy: 0.4471 - val_loss: 1.2157 - val_accuracy: 0.5437\n",
            "Epoch 47/1024\n",
            "58/58 [==============================] - 1s 8ms/step - loss: 1.2048 - accuracy: 0.4552 - val_loss: 1.2141 - val_accuracy: 0.5583\n",
            "Epoch 48/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1809 - accuracy: 0.4790 - val_loss: 1.2114 - val_accuracy: 0.5631\n",
            "Epoch 49/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1573 - accuracy: 0.4779 - val_loss: 1.2089 - val_accuracy: 0.5534\n",
            "Epoch 50/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1716 - accuracy: 0.4871 - val_loss: 1.2050 - val_accuracy: 0.5680\n",
            "Epoch 51/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1725 - accuracy: 0.4784 - val_loss: 1.2011 - val_accuracy: 0.5631\n",
            "Epoch 52/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1684 - accuracy: 0.4822 - val_loss: 1.1968 - val_accuracy: 0.5728\n",
            "Epoch 53/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1469 - accuracy: 0.4838 - val_loss: 1.1926 - val_accuracy: 0.5728\n",
            "Epoch 54/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1680 - accuracy: 0.4811 - val_loss: 1.1893 - val_accuracy: 0.5728\n",
            "Epoch 55/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.1387 - accuracy: 0.4951 - val_loss: 1.1838 - val_accuracy: 0.5777\n",
            "Epoch 56/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1307 - accuracy: 0.5151 - val_loss: 1.1816 - val_accuracy: 0.5825\n",
            "Epoch 57/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1559 - accuracy: 0.5011 - val_loss: 1.1772 - val_accuracy: 0.5680\n",
            "Epoch 58/1024\n",
            "58/58 [==============================] - 1s 8ms/step - loss: 1.1049 - accuracy: 0.5065 - val_loss: 1.1746 - val_accuracy: 0.5777\n",
            "Epoch 59/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.1027 - accuracy: 0.5124 - val_loss: 1.1697 - val_accuracy: 0.5728\n",
            "Epoch 60/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0893 - accuracy: 0.5254 - val_loss: 1.1675 - val_accuracy: 0.5825\n",
            "Epoch 61/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0890 - accuracy: 0.5178 - val_loss: 1.1654 - val_accuracy: 0.5728\n",
            "Epoch 62/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.0799 - accuracy: 0.5442 - val_loss: 1.1599 - val_accuracy: 0.5777\n",
            "Epoch 63/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0792 - accuracy: 0.5351 - val_loss: 1.1550 - val_accuracy: 0.5825\n",
            "Epoch 64/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0733 - accuracy: 0.5280 - val_loss: 1.1522 - val_accuracy: 0.5825\n",
            "Epoch 65/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0486 - accuracy: 0.5485 - val_loss: 1.1470 - val_accuracy: 0.5971\n",
            "Epoch 66/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0365 - accuracy: 0.5685 - val_loss: 1.1436 - val_accuracy: 0.5971\n",
            "Epoch 67/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0486 - accuracy: 0.5502 - val_loss: 1.1379 - val_accuracy: 0.5922\n",
            "Epoch 68/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0444 - accuracy: 0.5604 - val_loss: 1.1361 - val_accuracy: 0.5922\n",
            "Epoch 69/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0063 - accuracy: 0.5744 - val_loss: 1.1314 - val_accuracy: 0.6068\n",
            "Epoch 70/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.9926 - accuracy: 0.5858 - val_loss: 1.1266 - val_accuracy: 0.6068\n",
            "Epoch 71/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0161 - accuracy: 0.5647 - val_loss: 1.1241 - val_accuracy: 0.6019\n",
            "Epoch 72/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.9999 - accuracy: 0.5723 - val_loss: 1.1192 - val_accuracy: 0.5971\n",
            "Epoch 73/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.9747 - accuracy: 0.5960 - val_loss: 1.1132 - val_accuracy: 0.6068\n",
            "Epoch 74/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0021 - accuracy: 0.5814 - val_loss: 1.1085 - val_accuracy: 0.6068\n",
            "Epoch 75/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9865 - accuracy: 0.5696 - val_loss: 1.1043 - val_accuracy: 0.6019\n",
            "Epoch 76/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.9554 - accuracy: 0.5933 - val_loss: 1.1002 - val_accuracy: 0.6019\n",
            "Epoch 77/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9557 - accuracy: 0.5976 - val_loss: 1.0964 - val_accuracy: 0.5971\n",
            "Epoch 78/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9394 - accuracy: 0.6138 - val_loss: 1.0909 - val_accuracy: 0.6117\n",
            "Epoch 79/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9012 - accuracy: 0.6143 - val_loss: 1.0883 - val_accuracy: 0.6068\n",
            "Epoch 80/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.9408 - accuracy: 0.6079 - val_loss: 1.0870 - val_accuracy: 0.6117\n",
            "Epoch 81/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8913 - accuracy: 0.6397 - val_loss: 1.0801 - val_accuracy: 0.6068\n",
            "Epoch 82/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8899 - accuracy: 0.6381 - val_loss: 1.0769 - val_accuracy: 0.6068\n",
            "Epoch 83/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8791 - accuracy: 0.6521 - val_loss: 1.0725 - val_accuracy: 0.5971\n",
            "Epoch 84/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8905 - accuracy: 0.6375 - val_loss: 1.0654 - val_accuracy: 0.6068\n",
            "Epoch 85/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9040 - accuracy: 0.6268 - val_loss: 1.0596 - val_accuracy: 0.6408\n",
            "Epoch 86/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8649 - accuracy: 0.6446 - val_loss: 1.0571 - val_accuracy: 0.6214\n",
            "Epoch 87/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8585 - accuracy: 0.6516 - val_loss: 1.0518 - val_accuracy: 0.6068\n",
            "Epoch 88/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8583 - accuracy: 0.6435 - val_loss: 1.0462 - val_accuracy: 0.6068\n",
            "Epoch 89/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.8313 - accuracy: 0.6715 - val_loss: 1.0414 - val_accuracy: 0.6117\n",
            "Epoch 90/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.8025 - accuracy: 0.6866 - val_loss: 1.0382 - val_accuracy: 0.6117\n",
            "Epoch 91/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8317 - accuracy: 0.6683 - val_loss: 1.0315 - val_accuracy: 0.6165\n",
            "Epoch 92/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8137 - accuracy: 0.6731 - val_loss: 1.0249 - val_accuracy: 0.6165\n",
            "Epoch 93/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7933 - accuracy: 0.6780 - val_loss: 1.0178 - val_accuracy: 0.6359\n",
            "Epoch 94/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7739 - accuracy: 0.6942 - val_loss: 1.0122 - val_accuracy: 0.6262\n",
            "Epoch 95/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7872 - accuracy: 0.6942 - val_loss: 1.0086 - val_accuracy: 0.6456\n",
            "Epoch 96/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.7972 - accuracy: 0.6812 - val_loss: 1.0042 - val_accuracy: 0.6311\n",
            "Epoch 97/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7659 - accuracy: 0.6920 - val_loss: 0.9989 - val_accuracy: 0.6262\n",
            "Epoch 98/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7573 - accuracy: 0.6915 - val_loss: 0.9936 - val_accuracy: 0.6456\n",
            "Epoch 99/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7470 - accuracy: 0.6990 - val_loss: 0.9916 - val_accuracy: 0.6359\n",
            "Epoch 100/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7263 - accuracy: 0.7152 - val_loss: 0.9866 - val_accuracy: 0.6408\n",
            "Epoch 101/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7482 - accuracy: 0.7050 - val_loss: 0.9836 - val_accuracy: 0.6359\n",
            "Epoch 102/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.7267 - accuracy: 0.7152 - val_loss: 0.9810 - val_accuracy: 0.6359\n",
            "Epoch 103/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7120 - accuracy: 0.7157 - val_loss: 0.9787 - val_accuracy: 0.6408\n",
            "Epoch 104/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7301 - accuracy: 0.7109 - val_loss: 0.9739 - val_accuracy: 0.6262\n",
            "Epoch 105/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.7067 - accuracy: 0.7330 - val_loss: 0.9694 - val_accuracy: 0.6456\n",
            "Epoch 106/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6885 - accuracy: 0.7341 - val_loss: 0.9649 - val_accuracy: 0.6456\n",
            "Epoch 107/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6792 - accuracy: 0.7400 - val_loss: 0.9621 - val_accuracy: 0.6505\n",
            "Epoch 108/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6666 - accuracy: 0.7319 - val_loss: 0.9571 - val_accuracy: 0.6505\n",
            "Epoch 109/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6699 - accuracy: 0.7400 - val_loss: 0.9529 - val_accuracy: 0.6553\n",
            "Epoch 110/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6513 - accuracy: 0.7503 - val_loss: 0.9493 - val_accuracy: 0.6553\n",
            "Epoch 111/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6647 - accuracy: 0.7465 - val_loss: 0.9473 - val_accuracy: 0.6602\n",
            "Epoch 112/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.6455 - accuracy: 0.7546 - val_loss: 0.9436 - val_accuracy: 0.6553\n",
            "Epoch 113/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6423 - accuracy: 0.7524 - val_loss: 0.9385 - val_accuracy: 0.6553\n",
            "Epoch 114/1024\n",
            "58/58 [==============================] - 1s 8ms/step - loss: 0.6514 - accuracy: 0.7487 - val_loss: 0.9348 - val_accuracy: 0.6505\n",
            "Epoch 115/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.6238 - accuracy: 0.7584 - val_loss: 0.9280 - val_accuracy: 0.6602\n",
            "Epoch 116/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6079 - accuracy: 0.7638 - val_loss: 0.9265 - val_accuracy: 0.6650\n",
            "Epoch 117/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5889 - accuracy: 0.7659 - val_loss: 0.9199 - val_accuracy: 0.6699\n",
            "Epoch 118/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6145 - accuracy: 0.7751 - val_loss: 0.9177 - val_accuracy: 0.6796\n",
            "Epoch 119/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5747 - accuracy: 0.7891 - val_loss: 0.9117 - val_accuracy: 0.6796\n",
            "Epoch 120/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5732 - accuracy: 0.7859 - val_loss: 0.9089 - val_accuracy: 0.6748\n",
            "Epoch 121/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5571 - accuracy: 0.7902 - val_loss: 0.9076 - val_accuracy: 0.6796\n",
            "Epoch 122/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5873 - accuracy: 0.7697 - val_loss: 0.9020 - val_accuracy: 0.6602\n",
            "Epoch 123/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5459 - accuracy: 0.7994 - val_loss: 0.8985 - val_accuracy: 0.6602\n",
            "Epoch 124/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5469 - accuracy: 0.7875 - val_loss: 0.8978 - val_accuracy: 0.6650\n",
            "Epoch 125/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5496 - accuracy: 0.7961 - val_loss: 0.8921 - val_accuracy: 0.6748\n",
            "Epoch 126/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.5422 - accuracy: 0.8004 - val_loss: 0.8888 - val_accuracy: 0.6748\n",
            "Epoch 127/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5328 - accuracy: 0.8020 - val_loss: 0.8863 - val_accuracy: 0.6845\n",
            "Epoch 128/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5443 - accuracy: 0.8047 - val_loss: 0.8830 - val_accuracy: 0.6796\n",
            "Epoch 129/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.5401 - accuracy: 0.7972 - val_loss: 0.8797 - val_accuracy: 0.6748\n",
            "Epoch 130/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.5111 - accuracy: 0.8031 - val_loss: 0.8771 - val_accuracy: 0.6699\n",
            "Epoch 131/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.5142 - accuracy: 0.8064 - val_loss: 0.8744 - val_accuracy: 0.6650\n",
            "Epoch 132/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5077 - accuracy: 0.7977 - val_loss: 0.8727 - val_accuracy: 0.6650\n",
            "Epoch 133/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4905 - accuracy: 0.8220 - val_loss: 0.8701 - val_accuracy: 0.6602\n",
            "Epoch 134/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4693 - accuracy: 0.8258 - val_loss: 0.8677 - val_accuracy: 0.6699\n",
            "Epoch 135/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4677 - accuracy: 0.8258 - val_loss: 0.8647 - val_accuracy: 0.6845\n",
            "Epoch 136/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.4701 - accuracy: 0.8236 - val_loss: 0.8622 - val_accuracy: 0.6845\n",
            "Epoch 137/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4545 - accuracy: 0.8323 - val_loss: 0.8601 - val_accuracy: 0.6796\n",
            "Epoch 138/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4596 - accuracy: 0.8258 - val_loss: 0.8576 - val_accuracy: 0.6699\n",
            "Epoch 139/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4506 - accuracy: 0.8409 - val_loss: 0.8537 - val_accuracy: 0.6796\n",
            "Epoch 140/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4456 - accuracy: 0.8376 - val_loss: 0.8523 - val_accuracy: 0.6748\n",
            "Epoch 141/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4224 - accuracy: 0.8414 - val_loss: 0.8502 - val_accuracy: 0.6796\n",
            "Epoch 142/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4161 - accuracy: 0.8501 - val_loss: 0.8476 - val_accuracy: 0.6796\n",
            "Epoch 143/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4464 - accuracy: 0.8355 - val_loss: 0.8456 - val_accuracy: 0.6893\n",
            "Epoch 144/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4371 - accuracy: 0.8339 - val_loss: 0.8430 - val_accuracy: 0.6845\n",
            "Epoch 145/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4047 - accuracy: 0.8522 - val_loss: 0.8401 - val_accuracy: 0.6893\n",
            "Epoch 146/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4293 - accuracy: 0.8376 - val_loss: 0.8360 - val_accuracy: 0.6796\n",
            "Epoch 147/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4163 - accuracy: 0.8495 - val_loss: 0.8338 - val_accuracy: 0.6893\n",
            "Epoch 148/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3901 - accuracy: 0.8581 - val_loss: 0.8298 - val_accuracy: 0.6845\n",
            "Epoch 149/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4083 - accuracy: 0.8506 - val_loss: 0.8294 - val_accuracy: 0.6942\n",
            "Epoch 150/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3955 - accuracy: 0.8581 - val_loss: 0.8269 - val_accuracy: 0.6893\n",
            "Epoch 151/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4116 - accuracy: 0.8554 - val_loss: 0.8243 - val_accuracy: 0.6893\n",
            "Epoch 152/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3779 - accuracy: 0.8711 - val_loss: 0.8243 - val_accuracy: 0.6942\n",
            "Epoch 153/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3795 - accuracy: 0.8759 - val_loss: 0.8205 - val_accuracy: 0.6893\n",
            "Epoch 154/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3848 - accuracy: 0.8689 - val_loss: 0.8164 - val_accuracy: 0.6845\n",
            "Epoch 155/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3597 - accuracy: 0.8695 - val_loss: 0.8158 - val_accuracy: 0.6942\n",
            "Epoch 156/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3780 - accuracy: 0.8635 - val_loss: 0.8141 - val_accuracy: 0.6893\n",
            "Epoch 157/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3749 - accuracy: 0.8635 - val_loss: 0.8111 - val_accuracy: 0.6990\n",
            "Epoch 158/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3691 - accuracy: 0.8646 - val_loss: 0.8087 - val_accuracy: 0.6942\n",
            "Epoch 159/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3377 - accuracy: 0.8797 - val_loss: 0.8088 - val_accuracy: 0.6990\n",
            "Epoch 160/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3738 - accuracy: 0.8528 - val_loss: 0.8067 - val_accuracy: 0.6990\n",
            "Epoch 161/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3430 - accuracy: 0.8770 - val_loss: 0.8042 - val_accuracy: 0.6990\n",
            "Epoch 162/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3359 - accuracy: 0.8819 - val_loss: 0.8029 - val_accuracy: 0.6942\n",
            "Epoch 163/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.3613 - accuracy: 0.8657 - val_loss: 0.8016 - val_accuracy: 0.6942\n",
            "Epoch 164/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3142 - accuracy: 0.8921 - val_loss: 0.8013 - val_accuracy: 0.6990\n",
            "Epoch 165/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3317 - accuracy: 0.8910 - val_loss: 0.7979 - val_accuracy: 0.7039\n",
            "Epoch 166/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3245 - accuracy: 0.8776 - val_loss: 0.7967 - val_accuracy: 0.7039\n",
            "Epoch 167/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3139 - accuracy: 0.8883 - val_loss: 0.7964 - val_accuracy: 0.6990\n",
            "Epoch 168/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3461 - accuracy: 0.8700 - val_loss: 0.7970 - val_accuracy: 0.6893\n",
            "Epoch 169/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3077 - accuracy: 0.8851 - val_loss: 0.7952 - val_accuracy: 0.6990\n",
            "Epoch 170/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3199 - accuracy: 0.8921 - val_loss: 0.7930 - val_accuracy: 0.6893\n",
            "Epoch 171/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3119 - accuracy: 0.8916 - val_loss: 0.7918 - val_accuracy: 0.6893\n",
            "Epoch 172/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3009 - accuracy: 0.8997 - val_loss: 0.7919 - val_accuracy: 0.6893\n",
            "Epoch 173/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3164 - accuracy: 0.8813 - val_loss: 0.7886 - val_accuracy: 0.6893\n",
            "Epoch 174/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3064 - accuracy: 0.8932 - val_loss: 0.7853 - val_accuracy: 0.6942\n",
            "Epoch 175/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3328 - accuracy: 0.8727 - val_loss: 0.7869 - val_accuracy: 0.7039\n",
            "Epoch 176/1024\n",
            "52/58 [=========================>....] - ETA: 0s - loss: 0.2896 - accuracy: 0.8966\n",
            "Epoch 00176: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2880 - accuracy: 0.8975 - val_loss: 0.7890 - val_accuracy: 0.6942\n",
            "Epoch 177/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2672 - accuracy: 0.9115 - val_loss: 0.7886 - val_accuracy: 0.6845\n",
            "Epoch 178/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.2892 - accuracy: 0.8993\n",
            "Epoch 00178: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2851 - accuracy: 0.9002 - val_loss: 0.7858 - val_accuracy: 0.6942\n",
            "Epoch 179/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2798 - accuracy: 0.9094 - val_loss: 0.7847 - val_accuracy: 0.6893\n",
            "Epoch 180/1024\n",
            "58/58 [==============================] - 1s 8ms/step - loss: 0.2683 - accuracy: 0.9056 - val_loss: 0.7868 - val_accuracy: 0.6748\n",
            "Epoch 181/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.2805 - accuracy: 0.8970\n",
            "Epoch 00181: Reducing Max LR on Plateau: new max lr will be 1.25e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2856 - accuracy: 0.8937 - val_loss: 0.7859 - val_accuracy: 0.6796\n",
            "Epoch 182/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2793 - accuracy: 0.9051 - val_loss: 0.7851 - val_accuracy: 0.6845\n",
            "Epoch 183/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2832 - accuracy: 0.8932 - val_loss: 0.7840 - val_accuracy: 0.6893\n",
            "Epoch 184/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2585 - accuracy: 0.9164 - val_loss: 0.7833 - val_accuracy: 0.6942\n",
            "Epoch 185/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2811 - accuracy: 0.8970 - val_loss: 0.7840 - val_accuracy: 0.6893\n",
            "Epoch 186/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2699 - accuracy: 0.9121 - val_loss: 0.7830 - val_accuracy: 0.6942\n",
            "Epoch 187/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2696 - accuracy: 0.9078 - val_loss: 0.7831 - val_accuracy: 0.6893\n",
            "Epoch 188/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.2805 - accuracy: 0.9018 - val_loss: 0.7817 - val_accuracy: 0.6942\n",
            "Epoch 189/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2819 - accuracy: 0.8997 - val_loss: 0.7821 - val_accuracy: 0.6893\n",
            "Epoch 190/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2591 - accuracy: 0.9153 - val_loss: 0.7813 - val_accuracy: 0.6942\n",
            "Epoch 191/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2640 - accuracy: 0.9088 - val_loss: 0.7807 - val_accuracy: 0.6893\n",
            "Epoch 192/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.2855 - accuracy: 0.8975 - val_loss: 0.7791 - val_accuracy: 0.6893\n",
            "Epoch 193/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2777 - accuracy: 0.9002 - val_loss: 0.7802 - val_accuracy: 0.6893\n",
            "Epoch 194/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2899 - accuracy: 0.9083 - val_loss: 0.7787 - val_accuracy: 0.6893\n",
            "Epoch 195/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2738 - accuracy: 0.9056 - val_loss: 0.7783 - val_accuracy: 0.6893\n",
            "Epoch 196/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2575 - accuracy: 0.9126 - val_loss: 0.7786 - val_accuracy: 0.6942\n",
            "Epoch 197/1024\n",
            "55/58 [===========================>..] - ETA: 0s - loss: 0.2494 - accuracy: 0.9142\n",
            "Epoch 00197: Reducing Max LR on Plateau: new max lr will be 6.25e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2496 - accuracy: 0.9142 - val_loss: 0.7784 - val_accuracy: 0.6942\n",
            "Epoch 198/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2614 - accuracy: 0.9061 - val_loss: 0.7796 - val_accuracy: 0.6942\n",
            "Epoch 199/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2858 - accuracy: 0.9002 - val_loss: 0.7775 - val_accuracy: 0.6942\n",
            "Epoch 200/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2774 - accuracy: 0.8970 - val_loss: 0.7787 - val_accuracy: 0.6990\n",
            "Epoch 201/1024\n",
            "56/58 [===========================>..] - ETA: 0s - loss: 0.2480 - accuracy: 0.9196\n",
            "Epoch 00201: Reducing Max LR on Plateau: new max lr will be 3.125e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2494 - accuracy: 0.9207 - val_loss: 0.7790 - val_accuracy: 0.6942\n",
            "Epoch 202/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2755 - accuracy: 0.9061 - val_loss: 0.7800 - val_accuracy: 0.6942\n",
            "Epoch 203/1024\n",
            "56/58 [===========================>..] - ETA: 0s - loss: 0.2751 - accuracy: 0.9007\n",
            "Epoch 00203: Reducing Max LR on Plateau: new max lr will be 1.5625e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2710 - accuracy: 0.9024 - val_loss: 0.7799 - val_accuracy: 0.6942\n",
            "Epoch 204/1024\n",
            "58/58 [==============================] - ETA: 0s - loss: 0.2886 - accuracy: 0.9002Restoring model weights from the end of the best epoch: 199.\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2886 - accuracy: 0.9002 - val_loss: 0.7807 - val_accuracy: 0.6942\n",
            "Epoch 204: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.61      0.58      0.59        19\n",
            "      level2       0.71      0.75      0.73        63\n",
            "      level3       0.64      0.69      0.66        64\n",
            "      level4       0.77      0.68      0.73        60\n",
            "\n",
            "    accuracy                           0.69       206\n",
            "   macro avg       0.68      0.67      0.68       206\n",
            "weighted avg       0.70      0.69      0.69       206\n",
            "\n",
            "-------------------------------------------------------run 4\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.7539 - accuracy: 0.2573 - val_loss: 1.3733 - val_accuracy: 0.3058\n",
            "Epoch 2/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.7491 - accuracy: 0.2789 - val_loss: 1.3703 - val_accuracy: 0.3155\n",
            "Epoch 3/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.7176 - accuracy: 0.2686 - val_loss: 1.3679 - val_accuracy: 0.3350\n",
            "Epoch 4/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.6628 - accuracy: 0.2762 - val_loss: 1.3677 - val_accuracy: 0.3301\n",
            "Epoch 5/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.6252 - accuracy: 0.3026 - val_loss: 1.3673 - val_accuracy: 0.3155\n",
            "Epoch 6/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.6675 - accuracy: 0.2864 - val_loss: 1.3654 - val_accuracy: 0.3350\n",
            "Epoch 7/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5874 - accuracy: 0.3026 - val_loss: 1.3613 - val_accuracy: 0.3689\n",
            "Epoch 8/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5796 - accuracy: 0.2999 - val_loss: 1.3557 - val_accuracy: 0.3738\n",
            "Epoch 9/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5426 - accuracy: 0.3058 - val_loss: 1.3497 - val_accuracy: 0.4126\n",
            "Epoch 10/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5217 - accuracy: 0.3355 - val_loss: 1.3421 - val_accuracy: 0.4223\n",
            "Epoch 11/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5276 - accuracy: 0.3112 - val_loss: 1.3348 - val_accuracy: 0.4320\n",
            "Epoch 12/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5115 - accuracy: 0.3231 - val_loss: 1.3291 - val_accuracy: 0.4320\n",
            "Epoch 13/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.5146 - accuracy: 0.3290 - val_loss: 1.3215 - val_accuracy: 0.4612\n",
            "Epoch 14/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4972 - accuracy: 0.3182 - val_loss: 1.3160 - val_accuracy: 0.4612\n",
            "Epoch 15/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4674 - accuracy: 0.3452 - val_loss: 1.3124 - val_accuracy: 0.4806\n",
            "Epoch 16/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4974 - accuracy: 0.3215 - val_loss: 1.3076 - val_accuracy: 0.4806\n",
            "Epoch 17/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4462 - accuracy: 0.3614 - val_loss: 1.3038 - val_accuracy: 0.4903\n",
            "Epoch 18/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4447 - accuracy: 0.3646 - val_loss: 1.3041 - val_accuracy: 0.5000\n",
            "Epoch 19/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.4023 - accuracy: 0.3738 - val_loss: 1.3015 - val_accuracy: 0.5049\n",
            "Epoch 20/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4401 - accuracy: 0.3554 - val_loss: 1.3013 - val_accuracy: 0.5000\n",
            "Epoch 21/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4302 - accuracy: 0.3538 - val_loss: 1.2966 - val_accuracy: 0.5097\n",
            "Epoch 22/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4368 - accuracy: 0.3323 - val_loss: 1.2914 - val_accuracy: 0.5243\n",
            "Epoch 23/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4201 - accuracy: 0.3625 - val_loss: 1.2897 - val_accuracy: 0.5388\n",
            "Epoch 24/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3879 - accuracy: 0.3576 - val_loss: 1.2866 - val_accuracy: 0.5388\n",
            "Epoch 25/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.4045 - accuracy: 0.3571 - val_loss: 1.2859 - val_accuracy: 0.5388\n",
            "Epoch 26/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.4028 - accuracy: 0.3554 - val_loss: 1.2838 - val_accuracy: 0.5437\n",
            "Epoch 27/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.3547 - accuracy: 0.3824 - val_loss: 1.2828 - val_accuracy: 0.5146\n",
            "Epoch 28/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3589 - accuracy: 0.3862 - val_loss: 1.2808 - val_accuracy: 0.5194\n",
            "Epoch 29/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3523 - accuracy: 0.3867 - val_loss: 1.2773 - val_accuracy: 0.5340\n",
            "Epoch 30/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3140 - accuracy: 0.3981 - val_loss: 1.2750 - val_accuracy: 0.5291\n",
            "Epoch 31/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.3381 - accuracy: 0.3916 - val_loss: 1.2708 - val_accuracy: 0.5291\n",
            "Epoch 32/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3664 - accuracy: 0.3776 - val_loss: 1.2738 - val_accuracy: 0.5243\n",
            "Epoch 33/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3339 - accuracy: 0.3873 - val_loss: 1.2701 - val_accuracy: 0.5243\n",
            "Epoch 34/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3450 - accuracy: 0.3878 - val_loss: 1.2673 - val_accuracy: 0.5291\n",
            "Epoch 35/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3196 - accuracy: 0.3959 - val_loss: 1.2650 - val_accuracy: 0.5291\n",
            "Epoch 36/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.3126 - accuracy: 0.4072 - val_loss: 1.2614 - val_accuracy: 0.5437\n",
            "Epoch 37/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2975 - accuracy: 0.4169 - val_loss: 1.2582 - val_accuracy: 0.5340\n",
            "Epoch 38/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2983 - accuracy: 0.4083 - val_loss: 1.2556 - val_accuracy: 0.5243\n",
            "Epoch 39/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2691 - accuracy: 0.4213 - val_loss: 1.2516 - val_accuracy: 0.5388\n",
            "Epoch 40/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2672 - accuracy: 0.4310 - val_loss: 1.2474 - val_accuracy: 0.5388\n",
            "Epoch 41/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2462 - accuracy: 0.4164 - val_loss: 1.2452 - val_accuracy: 0.5534\n",
            "Epoch 42/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2804 - accuracy: 0.4180 - val_loss: 1.2430 - val_accuracy: 0.5485\n",
            "Epoch 43/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2456 - accuracy: 0.4374 - val_loss: 1.2410 - val_accuracy: 0.5631\n",
            "Epoch 44/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2595 - accuracy: 0.4266 - val_loss: 1.2370 - val_accuracy: 0.5728\n",
            "Epoch 45/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2050 - accuracy: 0.4617 - val_loss: 1.2346 - val_accuracy: 0.5874\n",
            "Epoch 46/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.2221 - accuracy: 0.4520 - val_loss: 1.2331 - val_accuracy: 0.5825\n",
            "Epoch 47/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1806 - accuracy: 0.4698 - val_loss: 1.2287 - val_accuracy: 0.5728\n",
            "Epoch 48/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2056 - accuracy: 0.4439 - val_loss: 1.2244 - val_accuracy: 0.5680\n",
            "Epoch 49/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2037 - accuracy: 0.4423 - val_loss: 1.2242 - val_accuracy: 0.5680\n",
            "Epoch 50/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.2050 - accuracy: 0.4595 - val_loss: 1.2220 - val_accuracy: 0.5777\n",
            "Epoch 51/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1796 - accuracy: 0.4741 - val_loss: 1.2186 - val_accuracy: 0.5825\n",
            "Epoch 52/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1573 - accuracy: 0.4817 - val_loss: 1.2139 - val_accuracy: 0.5728\n",
            "Epoch 53/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1710 - accuracy: 0.4730 - val_loss: 1.2090 - val_accuracy: 0.5680\n",
            "Epoch 54/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1693 - accuracy: 0.4779 - val_loss: 1.2054 - val_accuracy: 0.5777\n",
            "Epoch 55/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1682 - accuracy: 0.4779 - val_loss: 1.2007 - val_accuracy: 0.5825\n",
            "Epoch 56/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1427 - accuracy: 0.4892 - val_loss: 1.1977 - val_accuracy: 0.5825\n",
            "Epoch 57/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.1437 - accuracy: 0.4941 - val_loss: 1.1965 - val_accuracy: 0.5825\n",
            "Epoch 58/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1341 - accuracy: 0.4806 - val_loss: 1.1929 - val_accuracy: 0.5825\n",
            "Epoch 59/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1133 - accuracy: 0.4946 - val_loss: 1.1893 - val_accuracy: 0.5825\n",
            "Epoch 60/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.1093 - accuracy: 0.5189 - val_loss: 1.1848 - val_accuracy: 0.5971\n",
            "Epoch 61/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 1.1018 - accuracy: 0.4995 - val_loss: 1.1785 - val_accuracy: 0.5922\n",
            "Epoch 62/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0806 - accuracy: 0.5254 - val_loss: 1.1753 - val_accuracy: 0.5874\n",
            "Epoch 63/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 1.0644 - accuracy: 0.5367 - val_loss: 1.1696 - val_accuracy: 0.5971\n",
            "Epoch 64/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0590 - accuracy: 0.5502 - val_loss: 1.1661 - val_accuracy: 0.6165\n",
            "Epoch 65/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0783 - accuracy: 0.5280 - val_loss: 1.1653 - val_accuracy: 0.6117\n",
            "Epoch 66/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0619 - accuracy: 0.5243 - val_loss: 1.1601 - val_accuracy: 0.6117\n",
            "Epoch 67/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0582 - accuracy: 0.5415 - val_loss: 1.1549 - val_accuracy: 0.6165\n",
            "Epoch 68/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0378 - accuracy: 0.5458 - val_loss: 1.1508 - val_accuracy: 0.6214\n",
            "Epoch 69/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0093 - accuracy: 0.5734 - val_loss: 1.1474 - val_accuracy: 0.6311\n",
            "Epoch 70/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0312 - accuracy: 0.5631 - val_loss: 1.1432 - val_accuracy: 0.6214\n",
            "Epoch 71/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0126 - accuracy: 0.5550 - val_loss: 1.1357 - val_accuracy: 0.6311\n",
            "Epoch 72/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 1.0224 - accuracy: 0.5572 - val_loss: 1.1339 - val_accuracy: 0.6408\n",
            "Epoch 73/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9904 - accuracy: 0.5728 - val_loss: 1.1295 - val_accuracy: 0.6408\n",
            "Epoch 74/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9722 - accuracy: 0.5949 - val_loss: 1.1267 - val_accuracy: 0.6408\n",
            "Epoch 75/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9863 - accuracy: 0.5771 - val_loss: 1.1209 - val_accuracy: 0.6456\n",
            "Epoch 76/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9514 - accuracy: 0.6030 - val_loss: 1.1156 - val_accuracy: 0.6456\n",
            "Epoch 77/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9370 - accuracy: 0.6090 - val_loss: 1.1117 - val_accuracy: 0.6456\n",
            "Epoch 78/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9769 - accuracy: 0.5836 - val_loss: 1.1047 - val_accuracy: 0.6456\n",
            "Epoch 79/1024\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.9330 - accuracy: 0.6268 - val_loss: 1.0998 - val_accuracy: 0.6553\n",
            "Epoch 80/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9470 - accuracy: 0.6160 - val_loss: 1.0935 - val_accuracy: 0.6553\n",
            "Epoch 81/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9275 - accuracy: 0.6084 - val_loss: 1.0905 - val_accuracy: 0.6553\n",
            "Epoch 82/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8920 - accuracy: 0.6381 - val_loss: 1.0874 - val_accuracy: 0.6553\n",
            "Epoch 83/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.9173 - accuracy: 0.6073 - val_loss: 1.0833 - val_accuracy: 0.6650\n",
            "Epoch 84/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8945 - accuracy: 0.6241 - val_loss: 1.0788 - val_accuracy: 0.6602\n",
            "Epoch 85/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8796 - accuracy: 0.6440 - val_loss: 1.0729 - val_accuracy: 0.6602\n",
            "Epoch 86/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8855 - accuracy: 0.6365 - val_loss: 1.0687 - val_accuracy: 0.6553\n",
            "Epoch 87/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8605 - accuracy: 0.6537 - val_loss: 1.0648 - val_accuracy: 0.6602\n",
            "Epoch 88/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8680 - accuracy: 0.6440 - val_loss: 1.0608 - val_accuracy: 0.6602\n",
            "Epoch 89/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8410 - accuracy: 0.6467 - val_loss: 1.0559 - val_accuracy: 0.6650\n",
            "Epoch 90/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8496 - accuracy: 0.6526 - val_loss: 1.0496 - val_accuracy: 0.6602\n",
            "Epoch 91/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8313 - accuracy: 0.6499 - val_loss: 1.0435 - val_accuracy: 0.6650\n",
            "Epoch 92/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.8005 - accuracy: 0.6893 - val_loss: 1.0379 - val_accuracy: 0.6699\n",
            "Epoch 93/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7966 - accuracy: 0.6942 - val_loss: 1.0322 - val_accuracy: 0.6699\n",
            "Epoch 94/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.8063 - accuracy: 0.6742 - val_loss: 1.0268 - val_accuracy: 0.6650\n",
            "Epoch 95/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7869 - accuracy: 0.6996 - val_loss: 1.0223 - val_accuracy: 0.6699\n",
            "Epoch 96/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7614 - accuracy: 0.6963 - val_loss: 1.0179 - val_accuracy: 0.6748\n",
            "Epoch 97/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7665 - accuracy: 0.6931 - val_loss: 1.0130 - val_accuracy: 0.6699\n",
            "Epoch 98/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7498 - accuracy: 0.7050 - val_loss: 1.0081 - val_accuracy: 0.6699\n",
            "Epoch 99/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7519 - accuracy: 0.7152 - val_loss: 1.0047 - val_accuracy: 0.6699\n",
            "Epoch 100/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7397 - accuracy: 0.7055 - val_loss: 1.0010 - val_accuracy: 0.6748\n",
            "Epoch 101/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7517 - accuracy: 0.6947 - val_loss: 0.9949 - val_accuracy: 0.6699\n",
            "Epoch 102/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7374 - accuracy: 0.7131 - val_loss: 0.9902 - val_accuracy: 0.6748\n",
            "Epoch 103/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.7151 - accuracy: 0.7217 - val_loss: 0.9862 - val_accuracy: 0.6748\n",
            "Epoch 104/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.7050 - accuracy: 0.7244 - val_loss: 0.9816 - val_accuracy: 0.6699\n",
            "Epoch 105/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6961 - accuracy: 0.7206 - val_loss: 0.9766 - val_accuracy: 0.6748\n",
            "Epoch 106/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6910 - accuracy: 0.7292 - val_loss: 0.9711 - val_accuracy: 0.6748\n",
            "Epoch 107/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6900 - accuracy: 0.7298 - val_loss: 0.9676 - val_accuracy: 0.6748\n",
            "Epoch 108/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6503 - accuracy: 0.7503 - val_loss: 0.9630 - val_accuracy: 0.6748\n",
            "Epoch 109/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6440 - accuracy: 0.7659 - val_loss: 0.9617 - val_accuracy: 0.6699\n",
            "Epoch 110/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6576 - accuracy: 0.7530 - val_loss: 0.9591 - val_accuracy: 0.6699\n",
            "Epoch 111/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6283 - accuracy: 0.7616 - val_loss: 0.9531 - val_accuracy: 0.6748\n",
            "Epoch 112/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6276 - accuracy: 0.7638 - val_loss: 0.9491 - val_accuracy: 0.6699\n",
            "Epoch 113/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6549 - accuracy: 0.7438 - val_loss: 0.9453 - val_accuracy: 0.6699\n",
            "Epoch 114/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6206 - accuracy: 0.7697 - val_loss: 0.9441 - val_accuracy: 0.6699\n",
            "Epoch 115/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.6102 - accuracy: 0.7638 - val_loss: 0.9377 - val_accuracy: 0.6845\n",
            "Epoch 116/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5959 - accuracy: 0.7665 - val_loss: 0.9326 - val_accuracy: 0.6893\n",
            "Epoch 117/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.6193 - accuracy: 0.7697 - val_loss: 0.9289 - val_accuracy: 0.6796\n",
            "Epoch 118/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5926 - accuracy: 0.7767 - val_loss: 0.9228 - val_accuracy: 0.6893\n",
            "Epoch 119/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5852 - accuracy: 0.7724 - val_loss: 0.9202 - val_accuracy: 0.6893\n",
            "Epoch 120/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5730 - accuracy: 0.7853 - val_loss: 0.9148 - val_accuracy: 0.6942\n",
            "Epoch 121/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5692 - accuracy: 0.7859 - val_loss: 0.9117 - val_accuracy: 0.6942\n",
            "Epoch 122/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5506 - accuracy: 0.8031 - val_loss: 0.9076 - val_accuracy: 0.6942\n",
            "Epoch 123/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5627 - accuracy: 0.7907 - val_loss: 0.9021 - val_accuracy: 0.6990\n",
            "Epoch 124/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5287 - accuracy: 0.8015 - val_loss: 0.8987 - val_accuracy: 0.6942\n",
            "Epoch 125/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.5273 - accuracy: 0.7956 - val_loss: 0.8957 - val_accuracy: 0.6942\n",
            "Epoch 126/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5350 - accuracy: 0.7988 - val_loss: 0.8915 - val_accuracy: 0.6942\n",
            "Epoch 127/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5117 - accuracy: 0.8236 - val_loss: 0.8880 - val_accuracy: 0.7039\n",
            "Epoch 128/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5171 - accuracy: 0.8101 - val_loss: 0.8849 - val_accuracy: 0.7039\n",
            "Epoch 129/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.5164 - accuracy: 0.8134 - val_loss: 0.8826 - val_accuracy: 0.6845\n",
            "Epoch 130/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4950 - accuracy: 0.8231 - val_loss: 0.8773 - val_accuracy: 0.6845\n",
            "Epoch 131/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4781 - accuracy: 0.8204 - val_loss: 0.8744 - val_accuracy: 0.6942\n",
            "Epoch 132/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4786 - accuracy: 0.8274 - val_loss: 0.8725 - val_accuracy: 0.6893\n",
            "Epoch 133/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4918 - accuracy: 0.8182 - val_loss: 0.8694 - val_accuracy: 0.6942\n",
            "Epoch 134/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4934 - accuracy: 0.8091 - val_loss: 0.8650 - val_accuracy: 0.6990\n",
            "Epoch 135/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4694 - accuracy: 0.8209 - val_loss: 0.8630 - val_accuracy: 0.6942\n",
            "Epoch 136/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4613 - accuracy: 0.8301 - val_loss: 0.8582 - val_accuracy: 0.6990\n",
            "Epoch 137/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4439 - accuracy: 0.8409 - val_loss: 0.8558 - val_accuracy: 0.6990\n",
            "Epoch 138/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4746 - accuracy: 0.8258 - val_loss: 0.8535 - val_accuracy: 0.6942\n",
            "Epoch 139/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4542 - accuracy: 0.8350 - val_loss: 0.8489 - val_accuracy: 0.6845\n",
            "Epoch 140/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4483 - accuracy: 0.8274 - val_loss: 0.8448 - val_accuracy: 0.6942\n",
            "Epoch 141/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4548 - accuracy: 0.8317 - val_loss: 0.8424 - val_accuracy: 0.6893\n",
            "Epoch 142/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4488 - accuracy: 0.8398 - val_loss: 0.8398 - val_accuracy: 0.6893\n",
            "Epoch 143/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.4149 - accuracy: 0.8528 - val_loss: 0.8360 - val_accuracy: 0.6893\n",
            "Epoch 144/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4383 - accuracy: 0.8344 - val_loss: 0.8345 - val_accuracy: 0.7039\n",
            "Epoch 145/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4319 - accuracy: 0.8436 - val_loss: 0.8324 - val_accuracy: 0.6942\n",
            "Epoch 146/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4211 - accuracy: 0.8457 - val_loss: 0.8310 - val_accuracy: 0.6893\n",
            "Epoch 147/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.4637 - accuracy: 0.8193 - val_loss: 0.8272 - val_accuracy: 0.6990\n",
            "Epoch 148/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3991 - accuracy: 0.8587 - val_loss: 0.8249 - val_accuracy: 0.6990\n",
            "Epoch 149/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3879 - accuracy: 0.8576 - val_loss: 0.8239 - val_accuracy: 0.6990\n",
            "Epoch 150/1024\n",
            "58/58 [==============================] - 1s 8ms/step - loss: 0.3646 - accuracy: 0.8662 - val_loss: 0.8219 - val_accuracy: 0.6942\n",
            "Epoch 151/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3998 - accuracy: 0.8603 - val_loss: 0.8175 - val_accuracy: 0.6893\n",
            "Epoch 152/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3580 - accuracy: 0.8770 - val_loss: 0.8173 - val_accuracy: 0.6942\n",
            "Epoch 153/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3751 - accuracy: 0.8625 - val_loss: 0.8136 - val_accuracy: 0.6990\n",
            "Epoch 154/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 0.3709 - accuracy: 0.8716 - val_loss: 0.8135 - val_accuracy: 0.6942\n",
            "Epoch 155/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3759 - accuracy: 0.8630 - val_loss: 0.8122 - val_accuracy: 0.6942\n",
            "Epoch 156/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3762 - accuracy: 0.8727 - val_loss: 0.8084 - val_accuracy: 0.6893\n",
            "Epoch 157/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3466 - accuracy: 0.8824 - val_loss: 0.8078 - val_accuracy: 0.6990\n",
            "Epoch 158/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3355 - accuracy: 0.8754 - val_loss: 0.8071 - val_accuracy: 0.6990\n",
            "Epoch 159/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3707 - accuracy: 0.8657 - val_loss: 0.8066 - val_accuracy: 0.6990\n",
            "Epoch 160/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3681 - accuracy: 0.8630 - val_loss: 0.8039 - val_accuracy: 0.6990\n",
            "Epoch 161/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3351 - accuracy: 0.8797 - val_loss: 0.8037 - val_accuracy: 0.6990\n",
            "Epoch 162/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3484 - accuracy: 0.8695 - val_loss: 0.8018 - val_accuracy: 0.6990\n",
            "Epoch 163/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.3318 - accuracy: 0.8781 - val_loss: 0.7993 - val_accuracy: 0.6990\n",
            "Epoch 164/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3098 - accuracy: 0.8981 - val_loss: 0.7979 - val_accuracy: 0.6990\n",
            "Epoch 165/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3339 - accuracy: 0.8738 - val_loss: 0.7937 - val_accuracy: 0.6893\n",
            "Epoch 166/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3124 - accuracy: 0.8894 - val_loss: 0.7930 - val_accuracy: 0.6893\n",
            "Epoch 167/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3183 - accuracy: 0.8889 - val_loss: 0.7926 - val_accuracy: 0.6942\n",
            "Epoch 168/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3083 - accuracy: 0.8948 - val_loss: 0.7907 - val_accuracy: 0.6942\n",
            "Epoch 169/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3065 - accuracy: 0.8916 - val_loss: 0.7887 - val_accuracy: 0.6942\n",
            "Epoch 170/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3030 - accuracy: 0.8900 - val_loss: 0.7873 - val_accuracy: 0.6942\n",
            "Epoch 171/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2866 - accuracy: 0.9056 - val_loss: 0.7863 - val_accuracy: 0.6942\n",
            "Epoch 172/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 0.3036 - accuracy: 0.8975 - val_loss: 0.7846 - val_accuracy: 0.6893\n",
            "Epoch 173/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2995 - accuracy: 0.8975 - val_loss: 0.7830 - val_accuracy: 0.6893\n",
            "Epoch 174/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2954 - accuracy: 0.8991 - val_loss: 0.7845 - val_accuracy: 0.6893\n",
            "Epoch 175/1024\n",
            "58/58 [==============================] - ETA: 0s - loss: 0.2879 - accuracy: 0.9018\n",
            "Epoch 00175: Reducing Max LR on Plateau: new max lr will be 5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2879 - accuracy: 0.9018 - val_loss: 0.7842 - val_accuracy: 0.6893\n",
            "Epoch 176/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2803 - accuracy: 0.9099 - val_loss: 0.7820 - val_accuracy: 0.6942\n",
            "Epoch 177/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.3040 - accuracy: 0.8943 - val_loss: 0.7814 - val_accuracy: 0.6893\n",
            "Epoch 178/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2937 - accuracy: 0.8970 - val_loss: 0.7816 - val_accuracy: 0.6893\n",
            "Epoch 179/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 0.2664 - accuracy: 0.9115 - val_loss: 0.7794 - val_accuracy: 0.6893\n",
            "Epoch 180/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2873 - accuracy: 0.9029 - val_loss: 0.7781 - val_accuracy: 0.6942\n",
            "Epoch 181/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2764 - accuracy: 0.9061 - val_loss: 0.7764 - val_accuracy: 0.6942\n",
            "Epoch 182/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2745 - accuracy: 0.9013 - val_loss: 0.7776 - val_accuracy: 0.6942\n",
            "Epoch 183/1024\n",
            "55/58 [===========================>..] - ETA: 0s - loss: 0.2861 - accuracy: 0.9057\n",
            "Epoch 00183: Reducing Max LR on Plateau: new max lr will be 2.5e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 9ms/step - loss: 0.2912 - accuracy: 0.9018 - val_loss: 0.7775 - val_accuracy: 0.6893\n",
            "Epoch 184/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2567 - accuracy: 0.9175 - val_loss: 0.7783 - val_accuracy: 0.6893\n",
            "Epoch 185/1024\n",
            "53/58 [==========================>...] - ETA: 0s - loss: 0.2734 - accuracy: 0.9057\n",
            "Epoch 00185: Reducing Max LR on Plateau: new max lr will be 1.25e-05 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2769 - accuracy: 0.9072 - val_loss: 0.7781 - val_accuracy: 0.6893\n",
            "Epoch 186/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2773 - accuracy: 0.8981 - val_loss: 0.7760 - val_accuracy: 0.6942\n",
            "Epoch 187/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2476 - accuracy: 0.9169 - val_loss: 0.7765 - val_accuracy: 0.6990\n",
            "Epoch 188/1024\n",
            "58/58 [==============================] - ETA: 0s - loss: 0.2662 - accuracy: 0.9067\n",
            "Epoch 00188: Reducing Max LR on Plateau: new max lr will be 6.25e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2662 - accuracy: 0.9067 - val_loss: 0.7763 - val_accuracy: 0.6893\n",
            "Epoch 189/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2604 - accuracy: 0.9202 - val_loss: 0.7757 - val_accuracy: 0.6942\n",
            "Epoch 190/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2810 - accuracy: 0.9040 - val_loss: 0.7762 - val_accuracy: 0.6942\n",
            "Epoch 191/1024\n",
            "53/58 [==========================>...] - ETA: 0s - loss: 0.2776 - accuracy: 0.9057\n",
            "Epoch 00191: Reducing Max LR on Plateau: new max lr will be 3.125e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2754 - accuracy: 0.9061 - val_loss: 0.7759 - val_accuracy: 0.6942\n",
            "Epoch 192/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2798 - accuracy: 0.9018 - val_loss: 0.7761 - val_accuracy: 0.6990\n",
            "Epoch 193/1024\n",
            "55/58 [===========================>..] - ETA: 0s - loss: 0.2765 - accuracy: 0.9062\n",
            "Epoch 00193: Reducing Max LR on Plateau: new max lr will be 1.5625e-06 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2746 - accuracy: 0.9078 - val_loss: 0.7758 - val_accuracy: 0.6893\n",
            "Epoch 194/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2518 - accuracy: 0.9164 - val_loss: 0.7743 - val_accuracy: 0.6942\n",
            "Epoch 195/1024\n",
            "58/58 [==============================] - 1s 10ms/step - loss: 0.2728 - accuracy: 0.9105 - val_loss: 0.7743 - val_accuracy: 0.6942\n",
            "Epoch 196/1024\n",
            "57/58 [============================>.] - ETA: 0s - loss: 0.2667 - accuracy: 0.9035\n",
            "Epoch 00196: Reducing Max LR on Plateau: new max lr will be 7.8125e-07 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 11ms/step - loss: 0.2690 - accuracy: 0.9013 - val_loss: 0.7752 - val_accuracy: 0.6942\n",
            "Epoch 197/1024\n",
            "58/58 [==============================] - 1s 11ms/step - loss: 0.2662 - accuracy: 0.8997 - val_loss: 0.7744 - val_accuracy: 0.6942\n",
            "Epoch 198/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2689 - accuracy: 0.9072 - val_loss: 0.7739 - val_accuracy: 0.6942\n",
            "Epoch 199/1024\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2631 - accuracy: 0.9105 - val_loss: 0.7747 - val_accuracy: 0.6942\n",
            "Epoch 200/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.2580 - accuracy: 0.9109\n",
            "Epoch 00200: Reducing Max LR on Plateau: new max lr will be 3.90625e-07 (if not early_stopping).\n",
            "58/58 [==============================] - 0s 8ms/step - loss: 0.2608 - accuracy: 0.9094 - val_loss: 0.7752 - val_accuracy: 0.6990\n",
            "Epoch 201/1024\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2715 - accuracy: 0.9040 - val_loss: 0.7751 - val_accuracy: 0.6942\n",
            "Epoch 202/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.2517 - accuracy: 0.9149\n",
            "Epoch 00202: Reducing Max LR on Plateau: new max lr will be 1.953125e-07 (if not early_stopping).\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2552 - accuracy: 0.9142 - val_loss: 0.7755 - val_accuracy: 0.6942\n",
            "Epoch 203/1024\n",
            "54/58 [==========================>...] - ETA: 0s - loss: 0.2759 - accuracy: 0.9028Restoring model weights from the end of the best epoch: 198.\n",
            "58/58 [==============================] - 1s 9ms/step - loss: 0.2761 - accuracy: 0.9035 - val_loss: 0.7750 - val_accuracy: 0.6942\n",
            "Epoch 203: early stopping\n",
            "Weights from best epoch have been loaded into model.\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "      level1       0.47      0.47      0.47        19\n",
            "      level2       0.75      0.68      0.72        63\n",
            "      level3       0.66      0.69      0.67        64\n",
            "      level4       0.75      0.78      0.76        60\n",
            "\n",
            "    accuracy                           0.69       206\n",
            "   macro avg       0.66      0.66      0.66       206\n",
            "weighted avg       0.70      0.69      0.69       206\n",
            "\n",
            "{   'accuracy': 0.7,\n",
            "    'class_names': ['level1', 'level2', 'level3', 'level4'],\n",
            "    'data_name': '',\n",
            "    'false_neg': array([46, 94, 90, 81]),\n",
            "    'false_pos': array([ 52,  82, 118,  59]),\n",
            "    'fmeasure': array([0.5 , 0.72, 0.69, 0.76]),\n",
            "    'macro_avg_fmeasure': 0.67,\n",
            "    'macro_avg_precision': 0.67,\n",
            "    'macro_avg_recall': 0.67,\n",
            "    'precision': array([0.49, 0.73, 0.66, 0.79]),\n",
            "    'recall': array([0.52, 0.7 , 0.72, 0.73]),\n",
            "    'support': array([ 95, 315, 320, 300]),\n",
            "    'total_support': 1030,\n",
            "    'true_pos': array([ 49, 221, 230, 219]),\n",
            "    'weighted_average_fmeasure': 0.7,\n",
            "    'weighted_average_precision': 0.7,\n",
            "    'weighted_average_recall': 0.7}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "BERT_metrics = processed_corpus.classify_corpus_BERT()"
      ],
      "metadata": {
        "id": "Y0FZshCYBU3N",
        "outputId": "98ab1e8f-7b2d-410a-ecef-a33b82d6dc40",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "b178215104fd46b6ac9b7b60a8532f23",
            "cf91e623288542ff86f9b267366a5022",
            "948aa44833fc4c0cada2f87dc0d73885",
            "82614e8a452340648c1cfa4ec040d733",
            "806085bbe57441b8bf82e1533deb4adc",
            "1ae3eb412d454da08f2b3dc46bbeb92d",
            "5231b9f1ac9d41f8b13a80e7876a203a",
            "02f787eed0cc4f5b94147b002a24c553",
            "37246e73e82c418fa73e60e68d4f94bc",
            "15dcab77b4b54c58ad8a0a0bf08e5111",
            "71b3cadd6ac448ff894d415cfa2bb787",
            "fcf4816e88e141e6844a9fa677b3d6dd",
            "d6dc3bae80994995ba9434cd8091277d",
            "5a541661022147c3839e6c19586cd32e",
            "726673fdda0f4b0792d907d90bd9fbd8",
            "81cb478ad74b421ab221067fe0569207",
            "065105b426584b639d9b023f103e01cd",
            "ad99d9661da54a5aa2c1b124c4fb5fd4",
            "df188f058e944160967e4a2a091b1345",
            "3f7104e72b75460b9e11e8c3af79cc2f",
            "1b4c141b87f045599c4dbbbb8974b67b",
            "8ff90fffe67c40ba903a24573be4a292",
            "8eb73b24258e47249ae183997273fb79",
            "83bf6947859440358772224659130d39",
            "94d7b80591f449e588571c4f8d60b460",
            "a247da6a034d44aeaf2ef44a6c77a6fc",
            "b56fdf420f9c4949a6f49a65625b0c1e",
            "0f792d40dd7a47c88220a3a2ad6c3d20",
            "8c42db6beb2b4d8d8f91438d79544d80",
            "e98047af09104d35a2c3d341d64f1da1",
            "80a9cdc397c946108e929db910143820",
            "547cf098d28441198fd2dded170b8c22",
            "a6190406ba6c4b3b8d9bb034c0d5980f",
            "9f5222f2a4a3460bb2aad10de35178bd",
            "7d61fdd551fd40f7afc4e0fadc8454e8",
            "248aba563fcc427ea95a7c7b289a27fa",
            "5d204caeebcb4ef981bc3cd47cfb97c2",
            "1be2fe3a8cb94edfb6960ac7499fb5a5",
            "3b99ea8b09804c90be982c7eed1a7ccc",
            "8f8f544a73324b4bbea27ffebee8a74f",
            "026018ed729e46b7b1079bf191ce789c",
            "97440e68b05148e8bfceb77f4e9abdab",
            "882fe6ca3e69489da1d6d188c851c22a",
            "13353603d34549a2ac088b38141d2384"
          ]
        }
      },
      "execution_count": 114,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading transformer model..\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/508 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b178215104fd46b6ac9b7b60a8532f23"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "preprocessing train...\n",
            "language: fr\n",
            "train sequence lengths:\n",
            "\tmean : 192\n",
            "\t95percentile : 420\n",
            "\t99percentile : 594\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/811k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "fcf4816e88e141e6844a9fa677b3d6dd"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.40M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "8eb73b24258e47249ae183997273fb79"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              ""
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Is Multi-Label? False\n",
            "preprocessing test...\n",
            "language: fr\n",
            "test sequence lengths:\n",
            "\tmean : 174\n",
            "\t95percentile : 380\n",
            "\t99percentile : 496\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              ""
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/543M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9f5222f2a4a3460bb2aad10de35178bd"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------------------------------------------run 0\n",
            "early_stopping automatically enabled at patience=5\n",
            "reduce_on_plateau automatically enabled at patience=2\n",
            "\n",
            "\n",
            "begin training using triangular learning rate policy with max lr of 0.0001...\n",
            "Epoch 1/1024\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "ResourceExhaustedError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mResourceExhaustedError\u001b[0m                    Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-114-ea7751aaeff1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mBERT_metrics\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprocessed_corpus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclassify_corpus_BERT\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/content/READI-LREC22/readability/parsed_collection/parsed_collection.py\u001b[0m in \u001b[0;36mclassify_corpus_BERT\u001b[0;34m(self, model_name)\u001b[0m\n\u001b[1;32m    411\u001b[0m         \u001b[0;34m:\u001b[0m\u001b[0;32mreturn\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mClassification\u001b[0m \u001b[0mtask\u001b[0m \u001b[0mmetrics\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mdetailed\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute_evaluation_metrics\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    412\u001b[0m         \"\"\"\n\u001b[0;32m--> 413\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadability_processor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclassify_corpus_BERT\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    414\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/READI-LREC22/readability/readability.py\u001b[0m in \u001b[0;36mclassify_corpus_BERT\u001b[0;34m(self, collection, model_name)\u001b[0m\n\u001b[1;32m    670\u001b[0m         \"\"\"\n\u001b[1;32m    671\u001b[0m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbert\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclassify_corpus_BERT\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 672\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcollection\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/content/READI-LREC22/readability/models/bert.py\u001b[0m in \u001b[0;36mclassify_corpus_BERT\u001b[0;34m(corpus, model_name, percent_train)\u001b[0m\n\u001b[1;32m     79\u001b[0m             \u001b[0mlearner\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_weights\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minit_weights\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     80\u001b[0m         \u001b[0;31m# train\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 81\u001b[0;31m         \u001b[0mlearner\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautofit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0.0001\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     82\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     83\u001b[0m         \u001b[0;31m# validate\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ktrain/core.py\u001b[0m in \u001b[0;36mautofit\u001b[0;34m(self, lr, epochs, early_stopping, reduce_on_plateau, reduce_factor, cycle_momentum, max_momentum, min_momentum, monitor, checkpoint_folder, class_weight, callbacks, steps_per_epoch, verbose)\u001b[0m\n\u001b[1;32m   1231\u001b[0m             \u001b[0mclass_weight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mclass_weight\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1232\u001b[0m             \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkcallbacks\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1233\u001b[0;31m             \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1234\u001b[0m         )\n\u001b[1;32m   1235\u001b[0m         \u001b[0mhist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhistory\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"lr\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhistory\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"lr\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/ktrain/core.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, lr, n_cycles, cycle_len, cycle_mult, lr_decay, checkpoint_folder, early_stopping, class_weight, callbacks, steps_per_epoch, verbose)\u001b[0m\n\u001b[1;32m   1646\u001b[0m                 \u001b[0mshuffle\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1647\u001b[0m                 \u001b[0mclass_weight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mclass_weight\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1648\u001b[0;31m                 \u001b[0mcallbacks\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkcallbacks\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1649\u001b[0m             )\n\u001b[1;32m   1650\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0msgdr\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     65\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# pylint: disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m       \u001b[0mfiltered_tb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_process_traceback_frames\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__traceback__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m       \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfiltered_tb\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     68\u001b[0m     \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m       \u001b[0;32mdel\u001b[0m \u001b[0mfiltered_tb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 55\u001b[0;31m                                         inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m     56\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mResourceExhaustedError\u001b[0m: Graph execution error:\n\nDetected at node 'tf_camembert_for_sequence_classification/roberta/encoder/layer_._4/intermediate/Gelu/Erf' defined at (most recent call last):\n    File \"/usr/lib/python3.7/runpy.py\", line 193, in _run_module_as_main\n      \"__main__\", mod_spec)\n    File \"/usr/lib/python3.7/runpy.py\", line 85, in _run_code\n      exec(code, run_globals)\n    File \"/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py\", line 16, in <module>\n      app.launch_new_instance()\n    File \"/usr/local/lib/python3.7/dist-packages/traitlets/config/application.py\", line 846, in launch_instance\n      app.start()\n    File \"/usr/local/lib/python3.7/dist-packages/ipykernel/kernelapp.py\", line 499, in start\n      self.io_loop.start()\n    File \"/usr/local/lib/python3.7/dist-packages/tornado/platform/asyncio.py\", line 132, in start\n      self.asyncio_loop.run_forever()\n    File \"/usr/lib/python3.7/asyncio/base_events.py\", line 541, in run_forever\n      self._run_once()\n    File \"/usr/lib/python3.7/asyncio/base_events.py\", line 1786, in _run_once\n      handle._run()\n    File \"/usr/lib/python3.7/asyncio/events.py\", line 88, in _run\n      self._context.run(self._callback, *self._args)\n    File \"/usr/local/lib/python3.7/dist-packages/tornado/ioloop.py\", line 758, in _run_callback\n      ret = callback()\n    File \"/usr/local/lib/python3.7/dist-packages/tornado/stack_context.py\", line 300, in null_wrapper\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py\", line 661, in <lambda>\n      self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))\n    File \"/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py\", line 577, in _handle_events\n      self._handle_recv()\n    File \"/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py\", line 606, in _handle_recv\n      self._run_callback(callback, msg)\n    File \"/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py\", line 556, in _run_callback\n      callback(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/tornado/stack_context.py\", line 300, in null_wrapper\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\n      return self.dispatch_shell(stream, msg)\n    File \"/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py\", line 233, in dispatch_shell\n      handler(stream, idents, msg)\n    File \"/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py\", line 399, in execute_request\n      user_expressions, allow_stdin)\n    File \"/usr/local/lib/python3.7/dist-packages/ipykernel/ipkernel.py\", line 208, in do_execute\n      res = shell.run_cell(code, store_history=store_history, silent=silent)\n    File \"/usr/local/lib/python3.7/dist-packages/ipykernel/zmqshell.py\", line 537, in run_cell\n      return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py\", line 2718, in run_cell\n      interactivity=interactivity, compiler=compiler, result=result)\n    File \"/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py\", line 2822, in run_ast_nodes\n      if self.run_code(code, result):\n    File \"/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py\", line 2882, in run_code\n      exec(code_obj, self.user_global_ns, self.user_ns)\n    File \"<ipython-input-114-ea7751aaeff1>\", line 1, in <module>\n      BERT_metrics = processed_corpus.classify_corpus_BERT()\n    File \"/content/READI-LREC22/readability/parsed_collection/parsed_collection.py\", line 413, in classify_corpus_BERT\n      return self.readability_processor.classify_corpus_BERT(self, model_name)\n    File \"/content/READI-LREC22/readability/readability.py\", line 672, in classify_corpus_BERT\n      return func(collection, model_name)\n    File \"/content/READI-LREC22/readability/models/bert.py\", line 81, in classify_corpus_BERT\n      learner.autofit(0.0001)\n    File \"/usr/local/lib/python3.7/dist-packages/ktrain/core.py\", line 1233, in autofit\n      steps_per_epoch=steps_per_epoch,\n    File \"/usr/local/lib/python3.7/dist-packages/ktrain/core.py\", line 1648, in fit\n      callbacks=kcallbacks,\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/training.py\", line 1384, in fit\n      tmp_logs = self.train_function(iterator)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/training.py\", line 1021, in train_function\n      return step_function(self, iterator)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/training.py\", line 1010, in step_function\n      outputs = model.distribute_strategy.run(run_step, args=(data,))\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/training.py\", line 1000, in run_step\n      outputs = model.train_step(data)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/training.py\", line 859, in train_step\n      y_pred = self(x, training=True)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py\", line 1096, in __call__\n      outputs = call_fn(inputs, *args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 92, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py\", line 1015, in call\n      outputs = self.roberta(\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py\", line 1096, in __call__\n      outputs = call_fn(inputs, *args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 92, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py\", line 558, in call\n      encoder_outputs = self.encoder(\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py\", line 1096, in __call__\n      outputs = call_fn(inputs, *args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 92, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py\", line 414, in call\n      for i, layer_module in enumerate(self.layer):\n    File \"/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py\", line 418, in call\n      layer_outputs = layer_module(\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py\", line 1096, in __call__\n      outputs = call_fn(inputs, *args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 92, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py\", line 385, in call\n      intermediate_output = self.intermediate(hidden_states=attention_output)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 64, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py\", line 1096, in __call__\n      outputs = call_fn(inputs, *args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py\", line 92, in error_handler\n      return fn(*args, **kwargs)\n    File \"/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py\", line 336, in call\n      hidden_states = self.intermediate_act_fn(hidden_states)\n    File \"/usr/local/lib/python3.7/dist-packages/keras/activations.py\", line 351, in gelu\n      return tf.nn.gelu(x, approximate)\nNode: 'tf_camembert_for_sequence_classification/roberta/encoder/layer_._4/intermediate/Gelu/Erf'\nfailed to allocate memory\n\t [[{{node tf_camembert_for_sequence_classification/roberta/encoder/layer_._4/intermediate/Gelu/Erf}}]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.\n [Op:__inference_train_function_673626]"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "interpreter": {
      "hash": "0d8382c64f832ccbfeaab2a5edd229bc484a8f789774f43ccfafc2e4aa4ca7e5"
    },
    "kernelspec": {
      "display_name": "Python 3.9.12 ('base')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12"
    },
    "orig_nbformat": 4,
    "colab": {
      "name": "readi_reproduction.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "accelerator": "GPU",
    "gpuClass": "standard",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "b178215104fd46b6ac9b7b60a8532f23": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cf91e623288542ff86f9b267366a5022",
              "IPY_MODEL_948aa44833fc4c0cada2f87dc0d73885",
              "IPY_MODEL_82614e8a452340648c1cfa4ec040d733"
            ],
            "layout": "IPY_MODEL_806085bbe57441b8bf82e1533deb4adc"
          }
        },
        "cf91e623288542ff86f9b267366a5022": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1ae3eb412d454da08f2b3dc46bbeb92d",
            "placeholder": "​",
            "style": "IPY_MODEL_5231b9f1ac9d41f8b13a80e7876a203a",
            "value": "Downloading: 100%"
          }
        },
        "948aa44833fc4c0cada2f87dc0d73885": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_02f787eed0cc4f5b94147b002a24c553",
            "max": 508,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_37246e73e82c418fa73e60e68d4f94bc",
            "value": 508
          }
        },
        "82614e8a452340648c1cfa4ec040d733": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_15dcab77b4b54c58ad8a0a0bf08e5111",
            "placeholder": "​",
            "style": "IPY_MODEL_71b3cadd6ac448ff894d415cfa2bb787",
            "value": " 508/508 [00:00&lt;00:00, 10.4kB/s]"
          }
        },
        "806085bbe57441b8bf82e1533deb4adc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1ae3eb412d454da08f2b3dc46bbeb92d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5231b9f1ac9d41f8b13a80e7876a203a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "02f787eed0cc4f5b94147b002a24c553": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "37246e73e82c418fa73e60e68d4f94bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "15dcab77b4b54c58ad8a0a0bf08e5111": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "71b3cadd6ac448ff894d415cfa2bb787": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fcf4816e88e141e6844a9fa677b3d6dd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d6dc3bae80994995ba9434cd8091277d",
              "IPY_MODEL_5a541661022147c3839e6c19586cd32e",
              "IPY_MODEL_726673fdda0f4b0792d907d90bd9fbd8"
            ],
            "layout": "IPY_MODEL_81cb478ad74b421ab221067fe0569207"
          }
        },
        "d6dc3bae80994995ba9434cd8091277d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_065105b426584b639d9b023f103e01cd",
            "placeholder": "​",
            "style": "IPY_MODEL_ad99d9661da54a5aa2c1b124c4fb5fd4",
            "value": "Downloading: 100%"
          }
        },
        "5a541661022147c3839e6c19586cd32e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_df188f058e944160967e4a2a091b1345",
            "max": 810912,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3f7104e72b75460b9e11e8c3af79cc2f",
            "value": 810912
          }
        },
        "726673fdda0f4b0792d907d90bd9fbd8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1b4c141b87f045599c4dbbbb8974b67b",
            "placeholder": "​",
            "style": "IPY_MODEL_8ff90fffe67c40ba903a24573be4a292",
            "value": " 811k/811k [00:01&lt;00:00, 717kB/s]"
          }
        },
        "81cb478ad74b421ab221067fe0569207": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "065105b426584b639d9b023f103e01cd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ad99d9661da54a5aa2c1b124c4fb5fd4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "df188f058e944160967e4a2a091b1345": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3f7104e72b75460b9e11e8c3af79cc2f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1b4c141b87f045599c4dbbbb8974b67b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8ff90fffe67c40ba903a24573be4a292": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8eb73b24258e47249ae183997273fb79": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_83bf6947859440358772224659130d39",
              "IPY_MODEL_94d7b80591f449e588571c4f8d60b460",
              "IPY_MODEL_a247da6a034d44aeaf2ef44a6c77a6fc"
            ],
            "layout": "IPY_MODEL_b56fdf420f9c4949a6f49a65625b0c1e"
          }
        },
        "83bf6947859440358772224659130d39": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0f792d40dd7a47c88220a3a2ad6c3d20",
            "placeholder": "​",
            "style": "IPY_MODEL_8c42db6beb2b4d8d8f91438d79544d80",
            "value": "Downloading: 100%"
          }
        },
        "94d7b80591f449e588571c4f8d60b460": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e98047af09104d35a2c3d341d64f1da1",
            "max": 1395301,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_80a9cdc397c946108e929db910143820",
            "value": 1395301
          }
        },
        "a247da6a034d44aeaf2ef44a6c77a6fc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_547cf098d28441198fd2dded170b8c22",
            "placeholder": "​",
            "style": "IPY_MODEL_a6190406ba6c4b3b8d9bb034c0d5980f",
            "value": " 1.40M/1.40M [00:01&lt;00:00, 1.11MB/s]"
          }
        },
        "b56fdf420f9c4949a6f49a65625b0c1e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0f792d40dd7a47c88220a3a2ad6c3d20": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8c42db6beb2b4d8d8f91438d79544d80": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e98047af09104d35a2c3d341d64f1da1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "80a9cdc397c946108e929db910143820": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "547cf098d28441198fd2dded170b8c22": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a6190406ba6c4b3b8d9bb034c0d5980f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9f5222f2a4a3460bb2aad10de35178bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_7d61fdd551fd40f7afc4e0fadc8454e8",
              "IPY_MODEL_248aba563fcc427ea95a7c7b289a27fa",
              "IPY_MODEL_5d204caeebcb4ef981bc3cd47cfb97c2"
            ],
            "layout": "IPY_MODEL_1be2fe3a8cb94edfb6960ac7499fb5a5"
          }
        },
        "7d61fdd551fd40f7afc4e0fadc8454e8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3b99ea8b09804c90be982c7eed1a7ccc",
            "placeholder": "​",
            "style": "IPY_MODEL_8f8f544a73324b4bbea27ffebee8a74f",
            "value": "Downloading: 100%"
          }
        },
        "248aba563fcc427ea95a7c7b289a27fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_026018ed729e46b7b1079bf191ce789c",
            "max": 542808764,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_97440e68b05148e8bfceb77f4e9abdab",
            "value": 542808764
          }
        },
        "5d204caeebcb4ef981bc3cd47cfb97c2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_882fe6ca3e69489da1d6d188c851c22a",
            "placeholder": "​",
            "style": "IPY_MODEL_13353603d34549a2ac088b38141d2384",
            "value": " 543M/543M [00:09&lt;00:00, 60.9MB/s]"
          }
        },
        "1be2fe3a8cb94edfb6960ac7499fb5a5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3b99ea8b09804c90be982c7eed1a7ccc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8f8f544a73324b4bbea27ffebee8a74f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "026018ed729e46b7b1079bf191ce789c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "97440e68b05148e8bfceb77f4e9abdab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "882fe6ca3e69489da1d6d188c851c22a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "13353603d34549a2ac088b38141d2384": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}