chapter05_fundamentals-of-ml.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dJlvY6QDTk2x"
      },
      "source": [
        "이 노트북은 [케라스 창시자에게 배우는 딥러닝 2판](https://tensorflow.blog/kerasdl2/)의 예제 코드를 담고 있습니다.\n",
        "\n",
        "<table align=\"left\">\n",
        "    <tr>\n",
        "        <td>\n",
        "            <a href=\"https://colab.research.google.com/github/rickiepark/deep-learning-with-python-2nd/blob/main/chapter05_fundamentals-of-ml.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
        "        </td>\n",
        "    </tr>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mPvlk_EMTk24"
      },
      "source": [
        "# 머신 러닝의 기본 요소"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QIwdb7e7Tk25"
      },
      "source": [
        "## 일반화: 머신 러닝의 목표"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tPMwQAH7Tk29"
      },
      "source": [
        "### 과소적합과 과대적합"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pGWJKdoUTk2-"
      },
      "source": [
        "#### 잡음 섞인 훈련 데이터"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6px_zXDbTk2_"
      },
      "source": [
        "#### 불확실한 특성"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QgElP-miTk3A"
      },
      "source": [
        "#### 드문 특성과 가짜 상관관계"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VfRNGJB8Tk3B"
      },
      "source": [
        "**MNIST에 백색 잡음 픽셀과 0 픽셀을 추가하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:01:24.254476Z",
          "iopub.status.busy": "2022-05-21T01:01:24.253958Z",
          "iopub.status.idle": "2022-05-21T01:01:27.373051Z",
          "shell.execute_reply": "2022-05-21T01:01:27.372074Z"
        },
        "id": "KBz18RQVTk3C",
        "outputId": "c02f5e8b-f202-474a-f3f0-3ff584b536c9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11490434/11490434 [==============================] - 0s 0us/step\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.datasets import mnist\n",
        "import numpy as np\n",
        "\n",
        "(train_images, train_labels), _ = mnist.load_data()\n",
        "train_images = train_images.reshape((60000, 28 * 28))\n",
        "train_images = train_images.astype(\"float32\") / 255\n",
        "\n",
        "train_images_with_noise_channels = np.concatenate(\n",
        "    [train_images, np.random.random((len(train_images), 784))], axis=1)\n",
        "\n",
        "train_images_with_zeros_channels = np.concatenate(\n",
        "    [train_images, np.zeros((len(train_images), 784))], axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:01:27.377455Z",
          "iopub.status.busy": "2022-05-21T01:01:27.376945Z",
          "iopub.status.idle": "2022-05-21T01:01:27.388938Z",
          "shell.execute_reply": "2022-05-21T01:01:27.387917Z"
        },
        "id": "8cUOncJ1Tk3F",
        "outputId": "2a3dee9a-4216-44c0-b42f-af46e217d344",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(60000, 1568)"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "train_images_with_noise_channels.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eh9dzVI3Tk3I"
      },
      "source": [
        "**백색 잡음과 0을 추가한 MNIST 데이터에서 모델 훈련하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:01:27.392783Z",
          "iopub.status.busy": "2022-05-21T01:01:27.392444Z",
          "iopub.status.idle": "2022-05-21T01:02:25.502977Z",
          "shell.execute_reply": "2022-05-21T01:02:25.502078Z"
        },
        "id": "H8BUR0xETk3J",
        "outputId": "e11fe53b-adf6-48c0-9551-8bb08ad99200",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "375/375 [==============================] - 6s 7ms/step - loss: 0.6131 - accuracy: 0.8128 - val_loss: 0.3282 - val_accuracy: 0.8984\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 2s 6ms/step - loss: 0.2433 - accuracy: 0.9243 - val_loss: 0.2150 - val_accuracy: 0.9346\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 0.1586 - accuracy: 0.9513 - val_loss: 0.1745 - val_accuracy: 0.9510\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.1150 - accuracy: 0.9638 - val_loss: 0.1363 - val_accuracy: 0.9581\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0823 - accuracy: 0.9743 - val_loss: 0.1341 - val_accuracy: 0.9607\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0616 - accuracy: 0.9806 - val_loss: 0.1170 - val_accuracy: 0.9669\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0461 - accuracy: 0.9850 - val_loss: 0.1313 - val_accuracy: 0.9647\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0338 - accuracy: 0.9887 - val_loss: 0.1462 - val_accuracy: 0.9610\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0268 - accuracy: 0.9922 - val_loss: 0.1416 - val_accuracy: 0.9647\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0217 - accuracy: 0.9931 - val_loss: 0.1378 - val_accuracy: 0.9696\n",
            "Epoch 1/10\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 0.2858 - accuracy: 0.9164 - val_loss: 0.1457 - val_accuracy: 0.9584\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.1204 - accuracy: 0.9652 - val_loss: 0.1181 - val_accuracy: 0.9632\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0788 - accuracy: 0.9768 - val_loss: 0.0958 - val_accuracy: 0.9715\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0576 - accuracy: 0.9828 - val_loss: 0.0830 - val_accuracy: 0.9760\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0425 - accuracy: 0.9871 - val_loss: 0.0830 - val_accuracy: 0.9772\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0313 - accuracy: 0.9910 - val_loss: 0.0930 - val_accuracy: 0.9737\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0249 - accuracy: 0.9925 - val_loss: 0.0769 - val_accuracy: 0.9797\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0184 - accuracy: 0.9946 - val_loss: 0.0844 - val_accuracy: 0.9797\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0142 - accuracy: 0.9958 - val_loss: 0.0825 - val_accuracy: 0.9789\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0106 - accuracy: 0.9969 - val_loss: 0.0830 - val_accuracy: 0.9814\n"
          ]
        }
      ],
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "def get_model():\n",
        "    model = keras.Sequential([\n",
        "        layers.Dense(512, activation=\"relu\"),\n",
        "        layers.Dense(10, activation=\"softmax\")\n",
        "    ])\n",
        "    model.compile(optimizer=\"rmsprop\",\n",
        "                  loss=\"sparse_categorical_crossentropy\",\n",
        "                  metrics=[\"accuracy\"])\n",
        "    return model\n",
        "\n",
        "model = get_model()\n",
        "history_noise = model.fit(\n",
        "    train_images_with_noise_channels, train_labels,\n",
        "    epochs=10,\n",
        "    batch_size=128,\n",
        "    validation_split=0.2)\n",
        "\n",
        "model = get_model()\n",
        "history_zeros = model.fit(\n",
        "    train_images_with_zeros_channels, train_labels,\n",
        "    epochs=10,\n",
        "    batch_size=128,\n",
        "    validation_split=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vx3EJaPaTk3K"
      },
      "source": [
        "**검증 정확도 비교 그래프 그리기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:02:25.507068Z",
          "iopub.status.busy": "2022-05-21T01:02:25.506610Z",
          "iopub.status.idle": "2022-05-21T01:02:26.352048Z",
          "shell.execute_reply": "2022-05-21T01:02:26.351089Z"
        },
        "id": "FZnn2Uw4Tk3L",
        "outputId": "b24d4950-75b0-4347-e3e0-82387afa14f1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUZfbA8e+hV0EBkSLNQk1CQpEiRdG1wIogRcSCiK591dVV1lVR17XAKqK4/AQBRRTEgqxiAQRBQSUgQUV6kaoBpbeU8/vjvZNMwiQZkkxuyvk8zzyZuW3OvZm5Z95y3yuqijHGGJNZKb8DMMYYUzhZgjDGGBOSJQhjjDEhWYIwxhgTkiUIY4wxIVmCMMYYE5IliAgQkX+JyG4R2eW97iMiW0XkoIjE+hhXROIQkQbeNkvn1zZzeL/JIvKvgnivkyEim0XkIr/jyI3g2EXkHyIyIZxlc/E+XURkTW7jNAXLEkQueF+QI95JMfB42ZvXAPgb0EJVz/BWGQXcqapVVPX7PLyvisjZeQg9X+LITFV/8baZkl/bNP5R1X+r6rD82Fbmz6yqLlLVpvmxbRN5ZfwOoAj7s6rODTG9AbBHVX8LmtYQ+KlgwspWYYnDmGJFRMqoarLfceQ3K0HkI6/YPQeo65Uq3haRg0BpIEFENnjL1RWR90QkUUQ2icjdQdso7RXxN4jIARFZJiJnishCb5EEb9sDQ7x/KRH5p4hsEZHfROQNEakmIuVDxRFifRWRW0VknYjsFZGxIiLZbdub18hbt4z3eoiIbPTi3yQig4PeY6iI/Cwif4jIZyLSMJvjeb6ILPZi2SoiQ4JmnyoiH3vv8a2InBW03ove8vu949claN4IEXnHi/+AiPwkIm2D5m8WkftFZKWI7BOR6SJSIWh+LxFZ4cW0WESis4i9vYjEezH8KiLPZ7OfN4vIehH5XURmiUjdcP4nmbZR1yvVnhY0LVZcVWdZETlLRL4QkT3etKkiUj2LeEaIyJtBr6/z/u97ROThEPu5xIttp4i8LCLlvHknfGZFpLuIbAtav7mILPDW/0lErgiaN9nb35D/5xBxzxCRXd7/baGItAyaV1FE/uPtxz4R+UpEKnrzQn7OvLiGBW1jiIh8lel/c4eIrAPWedOy++xl9d0eKyL/ybQvs0Tk3qz2tcCoqj1O8gFsBi7KYl53YFumaQqc7T0vBSwDHgXKAU2AjcAl3vwHgB+ApoAAMUCNzNvJ4r2HAuu9bVYB3gemhIoji/UV+AiojisJJQKX5rRtoJG3bhmgMrAfaOrNqwO09J739rbR3Fv2n8DiLGJpCBwABgFlgRpAa2/eZGAP0N7bzlRgWtC613rLl8FV9+0CKnjzRgBHgctxCfNp4JtM/9vvgLrAacDPwK3evFjgN+A8b90bvOXLZ/5cAEuA67znVYAOWeznhcBuIA4oD7wELAznfxJiW18ANwe9HgmM856fDVzsvUctYCEwOtRn2jtGb3rPWwAHga7eus8DyUHLtgE6eMe6kXe87snqM0fQ98P7v64H/oH7Llzo/c8Dn51s/89ZfP6renGOBlYEzRsLLADqef+7Tt5y2X3OFgDDgrYxBPgq077N8T4nFcP47IX8bnv7twMo5S1XEzgM1Pb9XOd3AEXx4X2ZDgJ7gx43Z/4CZPogBRLEecAvmeYPByZ5z9cAvbN435xO8POA24NeNwWSgDJhrq/A+UGv3wEeymnbnJgg9gJXBb40Qet8AtwU9LqU90VoGCKW4cAHWcQ5GZgQ9PpyYHU2+/UHEOM9HwHMDZrXAjiS6X97bdDr50g/yf4XeDLTttcA3YLWDZw4FwKPAzVz+Cy9BjwX9LqKd1wb5fQ/CbGtYcAX3nMBtgJds1j2SuD7TPsdKkE8SsbkWxk4TtY/kO4J/r9l/syRMUF0wZ1ASwXNfxsYkZv/c6Y4qnvvXc37nB0JfAZO4nO2gJwTxIU5xBH82cvuu/0zcLH3/E5gdjj7GemHVTHl3pWqWj3oMT7M9RriqqD2Bh64X1C1vflnAiGrgMJQF9gS9HoL7qRdO/TiIe0Ken4Yd8IKe9uqeggYCNwK7PSqB5p5sxsCLwbt9++4E1m9EHHkdByyihOviuhnryphL+4kUTObdSuIVz2Ww7YbAn/L9L87E3dsMrsJOBdYLSJLRaRXFvuR4biq6kHcr+bgY5LlvmbyHtBRROrgfvGnAosARKS2iEwTke0ish94k4zHJCt1cYkmEN8hLz687Z4rIh95VTv7gX+Hud20batqatC0LeRi373qm2e86pv9uISHF0tNoAKhP095+b5B0LHx4sjus5fde72OK33g/Z2Sh5jyjSWIgrcV2JQpuVRV1cuD5mdZz5qDHbiTWEADXHXAr7kP9+S3raqfqerFuOql1UAgeW4F/pJp3yuq6uIQ75er4+DV+f4dGACcqqrVgX24RJRXW4GnMsVfSVXfzrygqq5T1UHA6cCzwLsiUjnENjMcV2+ZGsD2kw1OVf8APscl6Gtwv/zVm/1v3C/eKFU9BXcSCueY7MSd2ALxVfLiC/gv7n98jrfdf4S5XXD7fqaIBJ+HGpCLfcftb2/gItxJuVEgZFwV3lFCf56y+5wdAioFvT4jxDKB4xvOZy+793oT6C0iMbgq2JlZLFegLEEUvO+AAyLyoNdwVlpEWolIO2/+BOBJETlHnGgRCXwhf8W1AWTlbeBeEWksIlVwJ4Xpmj+9K8LatvdLtbd3ojuGq4oL/EIcBwwPNB6Ka0Dvn8X7TQUuEpEBIlJGRGqISOsw4qyKS1yJQBkReRQ45ST3NSvjgVtF5Dzvf1NZRHqKSNXMC4rItSJSy/t1vNebnJp5OdxxvVFEWotIedxx/VZVN+cyxreA64F+3vOAqrj/xT4RqYerDw/Hu0AvryG3HPAEGc8bVXFtTge9kuJtmdbP7jP7La5U8HdxDendgT8D08KMLVhV3OdtD+6k/u/ADO9/MBF4XlxjfmkR6egd7+w+ZyuAviJSSVxX3ZvCiCG7z16W321V3QYsxZUc3lPVI7k4BvnOEkTu/U8yXgfxQTgrqbtWoBfQGtiE+3UzAferB1wj4Du4X4L7cXXUFb15I4DXveqNASE2PxH3AVvobfsocFcu9i2UcLddCrgP9+vwd6Ab3klDVT/A/Zqe5lUD/AhcFurNVPUXXJ3z37ztrMA16uXkM+BTYC2uuuIomaoBcktV44GbgZdxdcvrcfXSoVwK/CSu99iLwNWhvvTquko/gqse2on7hXl1HsKcBZwD7FLVhKDpj+MawvcBH+M6GeRIVX8C7sAlm524/d4WtMj9uF/vB3AJdHqmTYwgi8+sqh7HJYTLcN+DV4DrVXV1OLFl8gbu/70dWAV8k2n+/bgG4qW4z9OzuLaP7D5nL+DaW37FVQFNzSGGnD572X238d4jikJSvQQg6SVQY4wxfhGRrriqpoZaSE7MVoIwxhifiUhZ4K+4XluFIjmAJQhjjPGViDTHtVPVwV2/UWhYFZMxxpiQrARhjDEmpGIzWF/NmjW1UaNGfodhjDFFyrJly3araq1Q84pNgmjUqBHx8fF+h2GMMUWKiGzJap5VMRljjAnJEoQxxpiQLEEYY4wJyRKEMcaYkCxBGGOMCckShDHGmJAsQRhjjAnJEoQxxhRBhw9H/j0imiBE5FIRWSMi60XkoRDzG4rIPBFZKSILRKR+0LznROQn7/Z9Y0QkP+4IZowxRc7PP8Prr8P998Mll0CdOtCuXc7r5VXErqQWkdLAWOBi3A1GlorILFVdFbTYKOANVX1dRC4EngauE5FOQGcg2lvuK9yNZxZEKl5jjPFTaips2gQrV8IPP8C6dfDGGyACzzzjnpcvDy1awJ/+BLGxkY8pkkNttAfWq+pGABGZhrtnbHCCaIG7+xjAfNLvw6q4m4yXw93PtSz5c19lY4zx3W+/uSTQqRNUrAhjx8KDD8KhQ26+CDRpAn/8AaedBg8/DMOHw9lnQ5kCHCApkm9Vj4y329sGnJdpmQSgL+6WjH2AqiJSQ1WXiMh83C0OBXhZVX+OYKzGmBysWQPTpp1Y9/3ss+7v++/Dt99mnFeuHDz5pHv+9tuwYkXG+VWrwj//6Z5PnuyqUoLVrAkPeHfP/r//g40b3fNSpdy6jRrBNde4aStXgipUq+Yep5wCpUvndm/z188/w/jxLimsXOkSBMCSJdChAzRvDsOGQVSUe7RsCZUrp69/7rn+xB2x+0GISD/gUlUd5r2+DjhPVe8MWqYu7v6+jXH3Or4KaAXUxCWNgd6ic4C/q+qiTO9xC3ALQIMGDdps2ZLlmFPGmFxISXG/ZkuVcr9gn33WVXMEO+LdafvOO+G11zLOq1IFEhPd8xtugHfeyTj/jDNctQrAVVfB7NkZ5599tjupgqtWWbQoPa6kJFcP/913blps7IkJ6NJL4ZNP3POrr4Z9+9ITSLVqEBfnpgPMmeMSWrVqUL36ySWZlBTYsMHFGvx47jm48kr48ksXS8uWLgFER7u/553nEp2fRGSZqrYNOS+CCaIjMEJVL/FeDwdQ1aezWL4KsFpV64vIA0AFVX3Sm/cocFRVn8vq/dq2bas2mqsx+WPPHpg4EV55BV5+GXr2hN273Ymwdm2/o3OOHXPJqXp193rxYti1yyWBwKN+fbjpJjf/6qtdCWTv3vT5V17pSkXgEsL+/RnfY8gQmDTJlUxiY6FSpYwJpn9/uOgi114Q+JUv4hJbVBTcdRd07+6OGxSeEk2w7BJEJKuYlgLniEhjYDtwNXBNpsBqAr+raiowHJjozfoFuFlEnsZVMXWjkN2Kz5ji6PvvXUJ46y04ehS6dXO/osFV9xQm5ctnLM106pT98oFEECxw4gb44ouMyWPfPmjWzM1LTnYn/X37XPLcuNE9r1LFJYgmTVxCbdXKlRIqVcr4PoUxMYQjorccFZHLcSf20sBEVX1KRJ4A4lV1llcN9TSuUXohcIeqHvN6QL0CdPXmfaqq94V+F8dKEMbkjqr71ZuS4k50u3fDddfBHXe4X8GmePOliqmgWYIw5uTs2OEafj/80DUuly8PS5fCOeekV9uY4s+vKiZjipxjx9zf8uVdT5O5c13VSseO/jcm5gdV+PprV4303nuu1HDZZa7apG7dgrn4yhQdliBMsXbokOtFs3u3+5uY6HqutGoFmzfD3XenT//tNzhwAKZMgWuvhbVrYfBgt53SpV0jZdeurrdO48a+7lauffMNdOniSgh33w233ebq1o0JxRKEKTJU3Qk8JQVOPdV1c5wyJf0EH3j06wdDh8L27a4XS2bPPOMSRKlSsHUr1KrlTvi1arlHtHf9flyc67++davrXrlwobugKdAr5n//c90yu3Z1J91Q7+W3jRvhv/91F2M98YTrcz91KvTunbGfvTGhWIIwhc7Ro1Chgnt+xRWwbZv7dZ+YCMePw1/+AuPGuRP8sGEucVSsmH6CD1QT1aoFTz+dPj3wqFPHzW/QwPXayUqlSq4XS7NmcPHFbtqxY66vPMD69e5kO26ce924sUsW48dD2bL5f1zClZrq+vS//DJ8/LE7TkOGuHki6ReWGZMTa6Q2vlN1V5d+8IF71K2bfnFTr17upBZ8gm/b1vUtB9iyxbUR+PVrODnZxb5woStl7Nzp+uODq77Zs8eVLrp2dT2CShXA+MkPPugu0Dr9dJdM//IXqFcv8u9riiZrpDaF1tix8PzzripEBDp3dhdlBXz0UfbrN2wY2fhyUqaMq4qKi4N77sk4r1w5V+c/Y4Z7Xb26q54aNcq9Tk3Nn4Tx00/pVV9t2sD117tqsn79Trzq2ZiTYQnCFJjjx93FSB9+CCNHuouMjh2Dpk3hoYdcdVJhuUo3P7z4onts2ZLehnHGGW7e8ePuV310dHoJo0OHEy+wykpyMsya5aqR5s93iSAuziWIli3dw5i8siomE1GHD7uG3A8+cPXh+/a56qA5c1zX0ZJq714YMcIljYQEV5ooU8Y1KA8b5hLn4cOuMT4zVWjd2lVtNWgAt9/uSg+F7UpnUzRYFZMpULt3u5NbgwauIbd/f3fyuuoq6NPHDU0QaIQuqapXh9He4DH79rl2i4ULXSkAXEmrZ0/XbtG1K7RvDz/+6HpgibirnE8/Hf7856I7jIMp/KwEYfLF1q0wc6Yb8nnhQlcPHhjkbPFiN2plQY5jX9StX++Gx1640A0JfeiQq5Jbtsy/oZ9N8WRDbZiI6tvXVSGBu9tVnz6u1BAT429cxUVSEqxa5brRBgbOMya/WBWTyRepqRAf75LBV1/BggWueqNbN1dC6NPHft1GQtmylmyNPyxBmBytWuXuCzBzprs6uXRpdx3Cnj2uHvyvf/U7QmNMJBTAZTumqDlyxHVFXbfOvd661Y11364dvP56+iB2p5/ub5zGmMiyEoRJs38//Otf7qKrw4fdvYKffBJ69HDDXNjYPcaULJYgDABvvuluDr9rlxvJ9Prr04ezKFPGeiAZUxLZ194A7h4BDRu6qqX27f2OxhhTGFgbRAn1229w883pA8s9/7x7bsnBGBNgCaKESUpy4wOdey5Mnpw+3HXFigUz0qgxJv+ouu/wnDmR2b5VMZUg8+fDXXe50T//9CeXKJo18zsqY8zJ+uknmDYNpk93vQ1btHDT8psliBJk6VLXO2nmTDdyqojfERljwrV2rUsI06e7ZFCqlOtIcv/9bjSDSLChNoqxw4fdjWNatnRDXxw/7q6GLukD5RlTVGzalJ4UVqxw084/HwYOdPf7CAwfnxc21EYJo+oGzbvvPvjlF3dz+v7902+VaYwpvLZudTeZmj4dvvvOTTvvPNeRpH//gr33uSWIYmbVKtfO8MUXbqjo+fPTr2cwxhROu3alJ4Wvv3bTYmPh2WdhwABo1MifuCxBFDOrVrleDS+/7O5FbBe4GVM47d4N773nGpu//NKV/Fu1cqMXDBwI55zjd4SWIIq8lBR334WjR+HOO91NeXr0CH0nMlP0/PYbjBsHEya4QRKbNj3xUa+edTgoKv74w42GPH06zJvnvr9Nm8Ijj7ik0KKF3xFmZAmiCFuyxFUnLVsGl1zi7jImYsmhOPjhB3fHualT3e1HL7kETjsN1qxxQ60fOpS+bOXK7rqWzInj3HPdTYaMv/bvd/cPnzYNPv/cXYvUuLEb2mbgQDeUe2FN8JYgiqBdu+DBB+GNN6BuXXcSGTSo8H7ITHhSU+GTT1ximDvXXbx4441uOPXg61VUYccOlyyCH99+636ZBndMrFfPrZs5eTRoYBdGRtKhQ/DRR+7/MXu2S/Jnnuk6jAwcCG3bFo3vq3VzLYKWLoUuXeDee+Hhh+1XYlF36JBL9i++6E709eq56sKbb4YaNU5uW0eOuNuVZk4ea9a4e18HVKjg6rhDVVlVq5a/+1dSHDniEvz06S45HD4Mdeq4nkcDB0KHDoUzKVs312Lg009dYnjkEXdfhq1boVYtv6MyebFtm+tM8Oqrrm66bVtXGuzf391FLjcqVnS916KiMk5Xde0ZmZNGQoKrE09JSV+2du3QiaNxY+v0kNnx467aaPp0N9DlgQNQs6YbDXngQPdDrnRpv6PMPStBFHIbNrjrGWbNcl/S5cuhUiW/ozJ58d138MILrlujqrsK9p57oFMnf6odjh+HjRtdwli9OmMC2bMnfbmyZV2p4/zzXUeICy90J8OSZvNmVwU4b5774bZ3r2v369vXJYULLihaiTS7EoQliELq4EF4+mkYNcpd4PbII+4kYhe7FU3Jye6X+ujRbtTcU06BYcNcJwO/+riHY8+ejAnjxx9h0SLX8Aqur36PHnDRRe7XcnH88bJ7t7ueKJAUNmxw0+vUcWOa9e8PF19cdL+bliCKoK1boXlz96vkmWdcY7Qpevbuhddeg5degi1boEkT1+h8441Qtarf0eVOcjLEx6efMBcvdqWQcuWgY0eXLC66yFWZFaVf0gGHD7skOG+e28cVK1xJr2pVVzoIJMTmzYtGQ3NOLEEUEStWwFtvuasnRVxvpfwYa8UUvA0bXKPzpEmuNNitm+tU0KtX0a6TDuXwYdf1du7cjCfUU05xV/EX9hNqVgmvbFlX7VfUE15OLEEUYjt3uqsp33nHfclq1HAf1oYN/Y7MnCxVd0Xs6NGuzahMGbj6alc1GBfnd3QFJ7sqmUCy6NGjYMcUCqYKP/+cXkJYsCBjlVkgvvPPLxn3YbcEUcioul9S8+e7D2LgEvsBA1z3RrvQrWg5dsz1YnnhBffruWZNuPVWuP12d1Is6TZtcifjwCMx0U1v2jT913n37lC9euRi2LYtPSHMm+d+mIGr8gvEcMEFJbPR3RJEIbBrlxthdcYM16D1j3+4ovmoUW7Y3sJ2ib3JWWKiGwbjlVfc/7dlS1daGDzYdTc1J0pNdQ3dgeqohQvddSClSrkqnEAJo1OnvA1Lv3evKxkE3mfNGje9Vi33HoFH48b5sltFmiUIH02Y4Pq2L1zovhzNm7uTyC23+B2Zya0ff3TVSG++6UoPl13m/qcXX1w469gLs+PH3RXggV/233zjrsmoUMFV8QSqe2Jjs2+7OXrUtR0EthMf775vlStD167ppYRWrQrnxWp+sgRRgH77zSWDfv3c6169XB/z/v1dFVLLlv7GZ3InNRU++8xVI82Z40oI11/veiQ1b+53dMXHgQPu+xP45f/jj276qae66y4CJYwmTdyoxYFqo6++ckmidGl3xXJgufPOK7rdTwuKbwlCRC4FXgRKAxNU9ZlM8xsCE4FawO/Ataq6zZvXAJgAnAkocLmqbs7qvfxMEImJrvronXdcsTY11dV51qvnerBUrmy/LIuqgwddSeHFF91FZHXrunaiW245+WEwzMnbtcvd22TePJeYt25108uXd6U3cKWCQAmha9ei233YL74kCBEpDawFLga2AUuBQaq6KmiZGcBHqvq6iFwI3Kiq13nzFgBPqeocEakCpKrq4azez68EMXOmG2I7NdWNnjlggHu0amVJoShRdUl95Uo3/ERCgnu+dq3737Zt67qp5mUYDJM3qm6cqXnzXC+k885zpQrrCp43fo3F1B5Yr6obvSCmAb2BVUHLtADu857PB2Z6y7YAyqjqHABVPRjBOMO2e7e7Gvadd1xD5JAhrjg7fLhLClFRlhSKgiNH3E3fMyeDP/5IX6ZxYzcM84AB7mpZv4bBMOlE3FAfheFGOiVFJBNEPWBr0OttwHmZlkkA+uKqofoAVUWkBnAusFdE3gcaA3OBh1Q1JXhlEbkFuAWgQYMGkdgHVN2VsO+844q6KSnuAxpo6DrjDPjXvyLy1iaPVGH79oxJICEhvVQArvovOtolgpgY9zwqyl3kZUxJ5/d1gfcDL4vIEGAhsB1IwcXVBYgFfgGmA0OA14JXVtVXgVfBVTHlV1B79rhB8QK9UsaNc0MlP/igq2IozDf4KKkylwpWrnSP339PXya4VBBIBk2aWK8WY7ISyQSxHdfAHFDfm5ZGVXfgShB47QxXqepeEdkGrAiqnpoJdCBTgshPe/a49oR33nF1nGXLusbnKlVc75XTTrOkUBgElwqCk8GaNRlLBVFRLplHR7tkYKUCY05eJBPEUuAcEWmMSwxXA9cELyAiNYHfVTUVGI7r0RRYt7qI1FLVROBCIGIt0K+/7kbWTE52vygfeMCdXAKX2VtvFX8cOQKrVp2YDDKXCqKj00t2ViowJv9ELEGoarKI3Al8huvmOlFVfxKRJ4B4VZ0FdAeeFhHFVTHd4a2bIiL3A/NERIBlwPhIxdq+Pfztb67qITbWSgp+++YbeOIJV3KzUoEx/rEL5UyhsWQJPP64Sww1a7pSXdu2LhlYqcCYyLBbjppCLXNieO45uO02u9e2MX6zBGF8s3ixSwyff26JwZjCyBKEKXCLF8OIEW7ohFq1YORIlxhKwtj7xhQlliBMgfn6a1dimDMHTj/dEoMxhZ0lCBNxX3/tSgxz57rEMGqUu6GOJQZjCjdLECZivvrKJYZ581xi+M9/XGKoVMnvyIwx4bAEYfLdokWuKskSgzFFm/UsN/lm0SJ3o5auXd2NXp5/3t2P+L77LDkYUxRZgjB5tnChG5e/a1c3YN7zz7u76N17ryUGY4oyq2Iyufbll64qaf58qF3b3Y7zllssKRhTXFgJwpy0L7+ECy6A7t3dnb1eeMGVGO65x5KDMcWJJQgTtgULXFLo3t3dn3n0aEsMxhRnliBMjgKJ4YIL3N3YXnzRJYa//hUqVvQ7OmNMpFiCMCGpuraFbt0yJoYNG+Duuy0xGFMSWIIwGQQSQ/furmfSunUwZowlBmNKIksQJoOHH86YGDZuhLvussRgTElk3VxNmldfhaefhptugpdfhgoV/I7IGOMnSxAGgE8/hdtvh8sug3HjoIx9Mowp8ayKyZCQ4O7H3aoVTJ9uycEY41iCKOG2b4eePeGUU+Djj6FqVb8jMsYUFvZbsQQ7cAB69YJ9+9zQ3PXq+R2RMaYwsQRRQiUnw8CB8MMP8NFHEBPjd0TGmMImxyomEfmziFhVVDGi6rqufvIJvPIKXHqp3xEZYwqjcE78A4F1IvKciDSLdEAm8v7zH9dT6e9/d6OvGmNMKDkmCFW9FogFNgCTRWSJiNwiItacWQS99x488AD07++ueTDGmKyEVXWkqvuBd4FpQB2gD7BcRO6KYGwmn33zDVx7LXTsCK+/DqWs4tAYk41w2iCuEJEPgAVAWaC9ql4GxAB/i2x4Jr9s3AhXXAF168KHH9rQGcaYnIXTi+kq4AVVXRg8UVUPi8hNkQnL5Kfff4fLL4eUFNcwXauW3xEZY4qCcBLECGBn4IWIVARqq+pmVZ0XqcBM/jh2DPr0gU2bYO5cOPdcvyMyxhQV4dRCzwBSg16neNNMIacKw4bBwoUwaRJ06eJ3RMaYoiScBFFGVY8HXnjPy0UuJJNfRoyAN9+Ef/0LrrnG72iMMUVNOAkiUUSuCLwQkd7A7siFZPLD5MnwxBMwdCj84x9+R2OMKYrCaYO4FZgqIi8DAmwFro9oVCZP5s2Dm2+Giy5yF8SJ+B2RMaYoyjFBqOoGoIOIVPFeH4x4VCbXVq2Cq66Cpk3h3ZmSjPEAACAASURBVHehbFm/IzLGFFVhDdYnIj2BlkAF8X6OquoTEYzL5MKuXa47a8WKbujuatX8jsgYU5TlmCBEZBxQCbgAmAD0A76LcFzmJB06BH/+MyQmul5LDRv6HZExpqgLp5G6k6peD/yhqo8DHQHrTV+IpKTA4MGwfDlMmwZt2vgdkTGmOAgnQRz1/h4WkbpAEm48JlNI3H+/Gz5j9GhXijDGmPwQThvE/0SkOjASWA4oMD6iUZmwvfSSSwz33OPu8WCMMfkl2xKEd6Ogeaq6V1XfAxoCzVT10XA2LiKXisgaEVkvIg+FmN9QROaJyEoRWSAi9TPNP0VEtnldbE0ms2a5xNC7N4wa5Xc0xpjiJtsEoaqpwNig18dUdV84GxaR0t66lwEtgEEi0iLTYqOAN1Q1GngCyHyHgieBhZgTLFsGgwZBXBxMnQqlS/sdkTGmuAmnDWKeiFwlctKXW7UH1qvqRm94jmlA70zLtAC+8J7PD54vIm2A2sDnJ/m+xd6WLdCrlxuV9X//g8qV/Y7IGFMchZMg/oIbnO+YiOwXkQMisj+M9erhrroO2OZNC5YA9PWe9wGqikgNr2rrP8D92b2Bd2e7eBGJT0xMDCOkom/fPujZE44cgdmz4Ywz/I7IGFNchXPL0aqqWkpVy6nqKd7rU/Lp/e8HuonI90A3YDtutNjbgdmqui2H2F5V1baq2rZWCbjJQVIS9OsHa9bA++9Di8wVdsYYk4/CuVCua6jpmW8gFMJ24Myg1/W9acHb2IFXgvCG8rhKVfeKSEegi4jcDlQByonIQVU9oaG7pFCFW29193SYPBkuvNDviIwxxV043VwfCHpeAde2sAzI6RS1FDhHRBrjEsPVQIZBp0WkJvC71xg+HJgIoKqDg5YZArQtyckB4N//hokT4dFH4YYb/I7GGFMShDNYX4ZLr0TkTGB0GOsli8idwGdAaWCiqv4kIk8A8ao6C+gOPC0iiuutdMfJ70Lx99Zb8M9/wrXXuns8GGNMQRBVPbkVXG+mn1S1UNWAt23bVuPj4/0OI98tWuSG7e7YET77DMqX9zsiY0xxIiLLVLVtqHnhtEG8hLt6GlyjdmvcFdUmwtasgSuvhMaN4YMPLDkYYwpWOG0QwT/Lk4G3VfXrCMVjPImJbuju0qVdd9ZTT/U7ImNMSRNOgngXOKqqKeCukBaRSqp6OLKhlVxHjrjhM3bsgPnzoUkTvyMyxpREYV1JDVQMel0RmBuZcExqquul9M038Oab0KGD3xEZY0qqcBJEheDbjHrPK0UupJJt+HCYMQNGjnS3DjXGGL+EkyAOiUhc4IU3RtKRyIVUcv3f/8Fzz8Ftt8F99/kdjTGmpAunDeIeYIaI7AAEOAMYGNGoSqAVK+COO1zD9JgxcNJDIxpjTD4L50K5pSLSDGjqTVqjqkmRDavkeeUV14116lQoE07aNsaYCMuxiklE7gAqq+qPqvojUMUbI8nkk4MH4e23YeBAqF7d72iMMcYJpw3iZlXdG3ihqn8AN0cupJJn2jSXJG62o2qMKUTCSRClg28W5N0prlzkQip5JkxwQ3dbl1ZjTGESToL4FJguIj1EpAfwNvBJZMMqOX74Ab791pUerGHaGFOYhNMc+iBwC3Cr93olrieTyQcTJkC5cm6kVmOMKUzCuaNcKvAtsBl3L4gLgZ8jG1bJcPQoTJkCfftCzZp+R2OMMRllWYIQkXOBQd5jNzAdQFUvKJjQir/334c//oBhw/yOxBhjTpRdFdNqYBHQS1XXA4jIvQUSVQkxfrwbiO8CS7nGmEIouyqmvsBOYL6IjPcaqK0ZNZ+sWwcLFsBNN0GpcLoKGGNMAcvy1KSqM1X1aqAZMB835MbpIvJfEflTQQVYXL32mrvXw5AhfkdijDGhhdNIfUhV3/LuTV0f+B7Xs8nkUlISTJ4MPXtC3bp+R2OMMaGdVOWGqv6hqq+qao9IBVQSfPQR/PqrXTltjCncrPbbBxMmuJLDpZf6HYkxxmTNEkQB27oVPv0Uhg61UVuNMYWbJYgCNmmSu63o0KF+R2KMMdmzBFGAUlJc76WLL4bGjf2OxhhjsmcJogDNnQu//GJXThtjigZLEAVo/Hg35lLv3n5HYowxObMEUUB++w0+/BCuv97dWtQYYwo7SxAF5PXXITnZqpeMMUWHJYgCoOqufejcGZo39zsaY4wJjyWIArBoEaxda1dOG2OKFksQBWDCBDjlFOjXz+9IjDEmfJYgIuyPP2DGDBg8GCpX9jsaY4wJnyWICHvrLXdrUWucNsYUNZYgIkjVXfsQF+cexhhTlFiCiKBlyyAhwRqnjTFFkyWICBo/HipVgkGD/I7EGGNOniWICDl40LU/DBgA1ar5HY0xxpw8SxAR8s47LklY47QxpqiKaIIQkUtFZI2IrBeRh0LMbygi80RkpYgsEJH63vTWIrJERH7y5g2MZJyRMGGCu2q6Uye/IzHGmNyJWIIQkdLAWOAyoAUwSERaZFpsFPCGqkYDTwBPe9MPA9erakvgUmC0iFSPVKz57aefYMkSV3oQ8TsaY4zJnUiWINoD61V1o6oeB6YBmQe6bgF84T2fH5ivqmtVdZ33fAfwG1ArgrHmq/HjoWxZN3KrMcYUVZFMEPWArUGvt3nTgiUAfb3nfYCqIlIjeAERaQ+UAzZEKM58dfQoTJkCffq4ez8YY0xR5Xcj9f1ANxH5HugGbAdSAjNFpA4wBbhRVVMzrywit4hIvIjEJyYmFlTM2frgA/j9d7v2wRhT9EUyQWwHzgx6Xd+blkZVd6hqX1WNBR72pu0FEJFTgI+Bh1X1m1BvoKqvqmpbVW1bq1bhqIGaMMHdb/rCC/2OxBhj8iaSCWIpcI6INBaRcsDVwKzgBUSkpogEYhgOTPSmlwM+wDVgvxvBGPPVhg3wxRdw001Qyu+ymTHG5FHETmOqmgzcCXwG/Ay8o6o/icgTInKFt1h3YI2IrAVqA0950wcAXYEhIrLCe7SOVKz55bXXXGIYMsTvSIwxJu9EVf2OIV+0bdtW4+PjfXv/pCRo0ADatYNZs3Je3hhjCgMRWaaqbUPNs4qQfDJ7NuzaZVdOG2OKD0sQ+WT8eKhTBy6/3O9IjDEmf1iCyAfbtsEnn8CNN0KZMn5HY4wx+cMSRD6YNAlSU13vJWOMKS4sQeRRaqrrvdSjBzRp4nc0xhiTfyxB5NHcubBli105bYwpfixB5NGECVCjBlx5pd+RGGNM/rIEkQeJiTBzphu1tXx5v6Mxxpj8ZQkiD954w10gZ9c+GGOKI0sQuaTqrn3o1AlaZL4NkjHGFAPWaz+Xvv4a1qyBiRP9jsQYYyLDShC5NH48VK0KAwb4HYkxxkSGJYhc2LsXZsyAa66BypX9jsYYYyLDEkQuvPUWHDli1z4YY4o3SxC5MGECtG4NcXF+R2KMMZFjCeIkLVsG33/vSg8ifkdjjDGRYwniJE2YABUruvYHY4wpzixBnIRDh2DqVOjfH6pX9zsaY4yJLLsO4iTMmAEHDtiV08GSkpLYtm0bR48e9TsUY0w2KlSoQP369SlbtmzY61iCOAnjx0PTpnD++X5HUnhs27aNqlWr0qhRI8QaZYwplFSVPXv2sG3bNho3bhz2elbFFKZVq2DxYld6sPNguqNHj1KjRg1LDsYUYiJCjRo1TrqkbwkiTBMmQNmybuRWk5ElB2MKv9x8Ty1BhOHYMTdya+/ecPrpfkdjjDEFwxJEGGbOhD177MrpwuiCCy7gs88+yzBt9OjR3HbbbVmu0717d+Lj4wG4/PLL2bt37wnLjBgxglGjRmX73jNnzmTVqlVprx999FHmzp17MuGXWIHjvnfvXl555ZW06QsWLKBXr175/n7x8fHcfffd+b5dCO+zEklVqlSJ2LYtQYRh/Hho2BAuusjvSExmgwYNYtq0aRmmTZs2jUGDBoW1/uzZs6meyz7LmRPEE088wUVF7EOSkpLiy/sGjnvmBBEpbdu2ZcyYMRF/n+LGEkQONm6EefPgppuglB2tbN1zD3Tvnr+Pe+7J/j379evHxx9/zPHjxwHYvHkzO3bsoEuXLtx22220bduWli1b8thjj4Vcv1GjRuzevRuAp556inPPPZfzzz+fNWvWpC0zfvx42rVrR0xMDFdddRWHDx9m8eLFzJo1iwceeIDWrVuzYcMGhgwZwrvvvgvAvHnziI2NJSoqiqFDh3Ls2LG093vssceIi4sjKiqK1atXnxDT5s2b6dKlC3FxccTFxbF48eK0ec8++yxRUVHExMTw0EMPAbB+/XouuugiYmJiiIuLY8OGDSf8Er/zzjuZPHlyWgwPPvggcXFxzJgxI+T+Afz666/06dOHmJgYYmJiWLx4MY8++iijR49O2+7DDz/Miy++mCH+kSNHpp2M7733Xi688EIAvvjiCwYPHpzhuD/00ENs2LCB1q1b88ADDwBw8OBB+vXrR7NmzRg8eDCqesIx6t69Ow8++CDt27fn3HPPZdGiRYDrNHHjjTcSFRVFbGws8+fPBzKWTL788ktat25N69atiY2N5cCBA2lxt2vXjujo6Cw/L59++ilxcXHExMTQo0ePtOmrVq2ie/fuNGnSJEMiuvLKK2nTpg0tW7bk1VdfTZtepUoVHn74YWJiYujQoQO//vorAEOGDOHuu++mU6dONGnSJO3zFE58O3fupGvXrrRu3ZpWrVqlHZO8sFNeDl57zSWGG2/0OxITymmnnUb79u355JNPAFd6GDBgACLCU089RXx8PCtXruTLL79k5cqVWW5n2bJlTJs2jRUrVjB79myWLl2aNq9v374sXbqUhIQEmjdvzmuvvUanTp244oorGDlyJCtWrOCss85KW/7o0aMMGTKE6dOn88MPP5CcnMx///vftPk1a9Zk+fLl3HbbbSGrJk4//XTmzJnD8uXLmT59elrVyCeffMKHH37It99+S0JCAn//+98BGDx4MHfccQcJCQksXryYOnXq5HjcatSowfLly7n66qtD7h/A3XffTbdu3UhISGD58uW0bNmSoUOH8sYbbwCQmprKtGnTuPbaazNsu0uXLmknp/j4eA4ePEhSUhKLFi2ia9euGZZ95plnOOuss1ixYgUjR44E4Pvvv2f06NGsWrWKjRs38vXXX4fch+TkZL777jtGjx7N448/DsDYsWMREX744QfefvttbrjhhhN67owaNYqxY8eyYsUKFi1aRMWKFfn8889Zt24d3333HStWrGDZsmUsXLgww3qJiYncfPPNvPfeeyQkJDBjxoy0eatXr+azzz7ju+++4/HHHycpKQmAiRMnsmzZMuLj4xkzZgx79uwB4NChQ3To0IGEhAS6du3K+PHj07a1c+dOvvrqKz766KO0HwHhxPfWW29xySWXsGLFChISEmjdunXI43Yy7DqIbCQnw6RJcNllUL++39EUfkE/LAtUoJqpd+/eTJs2Le0E98477/Dqq6+SnJzMzp07WbVqFdHR0SG3sWjRIvr06UOlSpUAuOKKK9Lm/fjjj/zzn/9k7969HDx4kEsuuSTbeNasWUPjxo0599xzAbjhhhsYO3Ys93jFob59+wLQpk0b3n///RPWT0pK4s4772TFihWULl2atWvXAjB37lxuvPHGtBhPO+00Dhw4wPbt2+nTpw/gLoYKx8CBA3Pcvy+++CItGZQuXZpq1apRrVo1atSowffff8+vv/5KbGwsNWrUyLDtNm3asGzZMvbv30/58uWJi4sjPj6eRYsWhVXN0759e+p7X7jWrVuzefNmzg9x8VHwcdy8eTMAX331FXfddRcAzZo1o2HDhmnHL6Bz587cd999DB48mL59+1K/fn0+//xzPv/8c2JjYwFXilm3bl2GhPbNN9/QtWvXtOsITjvttLR5PXv2pHz58pQvX57TTz+dX3/9lfr16zNmzBg++OADALZu3cq6deuoUaMG5cqVSyvRtGnThjlz5qRt68orr6RUqVK0aNEirWQRTnzt2rVj6NChJCUlceWVV1qCiLTZs2HnTrtyurDr3bs39957L8uXL+fw4cO0adOGTZs2MWrUKJYuXcqpp57KkCFDcn2195AhQ5g5cyYxMTFMnjyZBQsW5Cne8uXLA+6km5ycfML8F154gdq1a5OQkEBqamrYJ/1gZcqUITU1Ne115n2vHHQjk5Pdv2HDhjF58mR27drF0KFDT5hftmxZGjduzOTJk+nUqRPR0dHMnz+f9evX07x58xxjDxwfyPoYBS+X3TKhPPTQQ/Ts2ZPZs2fTuXNnPvvsM1SV4cOH85e//CXs7eQU84IFC5g7dy5LliyhUqVKdO/ePe3/ULZs2bRup5njD95WoHotnPi6du3KwoUL+fjjjxkyZAj33Xcf1+exX75VMWVjwgQ44wzo2dPvSEx2qlSpwgUXXMDQoUPTGqf3799P5cqVqVatGr/++mtaFVRWunbtysyZMzly5AgHDhzgf//7X9q8AwcOUKdOHZKSkpg6dWra9KpVq6bVXwdr2rQpmzdvZv369QBMmTKFbt26hb0/+/bto06dOpQqVYopU6akNSRffPHFTJo0Ka2N4Pfff6dq1arUr1+fmTNnAnDs2DEOHz5Mw4YNWbVqFceOHWPv3r3Mmzcvy/fLav969OiRVjWWkpLCvn37AOjTpw+ffvopS5cuzbI01aVLF0aNGkXXrl3p0qUL48aNIzY29oS++Fkdw9zq0qVL2j6sXbuWX375haZNm2ZYZsOGDURFRfHggw/Srl07Vq9ezSWXXMLEiRM5ePAgANu3b+e3337LsF6HDh1YuHAhmzZtAtzxz86+ffs49dRTqVSpEqtXr+abb77J9X6FE9+WLVuoXbs2N998M8OGDWP58uW5fr8ASxBZ2L4dPv4YhgxxF8iZwm3QoEEkJCSkJYiYmBhiY2Np1qwZ11xzDZ07d852/bi4OAYOHEhMTAyXXXYZ7dq1S5v35JNPct5559G5c2eaNWuWNv3qq69m5MiRxMbGsmHDhrTpFSpUYNKkSfTv35+oqChKlSrFrbfeGva+3H777bz++uvExMSwevXqtF/7l156KVdccQVt27aldevWae0XU6ZMYcyYMURHR9OpUyd27drFmWeeyYABA2jVqhUDBgxIq5oIJav9e/HFF5k/fz5RUVG0adMmrcdWuXLluOCCCxgwYAClS5cOuc0uXbqwc+dOOnbsSO3atalQoQJdunQ5YbkaNWrQuXNnWrVqldZInRe33347qampREVFMXDgQCZPnpzhFzm4btCtWrUiOjqasmXLctlll/GnP/2Ja665ho4dOxIVFUW/fv1OSFy1atXi1VdfpW/fvsTExGSopgvl0ksvJTk5mebNm/PQQw/RoUOHXO9XOPEtWLAg7XM/ffp0/vrXv+b6/QIkVA+Boqht27Ya6NueH556Cv75T1i3Ds4+O982W+z8/PPPYVUbmOIjNTU1rQfUOeec43c45iSE+r6KyDJVbRtqeStBhJCa6novXXCBJQdjgq1atYqzzz6bHj16WHIoAayROoQvvoBNm1wpwhiTrkWLFmzcuNHvMEwBsRJECOPHw2mngddz0BhjSiRLEJns3g0ffADXXQe56F1ojDHFhiWITN54A5KS7NoHY4yxBBFE1V370KEDtGrldzTGGOMvSxBBFi+Gn3+20kNRYsN9F00FPdx3JAV/ngpapI9XRBOEiFwqImtEZL2IPBRifkMRmSciK0VkgYjUD5p3g4is8x43RDLOgAkToEoVyOH6F1OI2HDfeVNShvsOOJkhOUwEE4SIlAbGApcBLYBBItIi02KjgDdUNRp4AnjaW/c04DHgPKA98JiInBqpWAH27YPp02HQIJckTO6EGrI78P0/fDj0fG8UanbvPnFeTmy475I33PeOHTvShutu3bo1pUuXZsuWLSQmJnLVVVfRrl072rVrlzYK7IgRI7juuuvo3Lkz1113HZs3b+bCCy8kOjqaHj168MsvvwAwY8YMWrVqRUxMzAmjzmZ3/APrZh56PKv/44IFC+jevXvIfczq83Ho0CGGDh1K+/btiY2N5cMPPzwhtqyGMc8TVY3IA+gIfBb0ejgwPNMyPwFnes8F2O89HwT8X9By/wcMyu792rRpo3nx3/+qgup33+VpMyXOqlWrMrzu1u3Ex9ixbt6hQ6HnT5rk5icmnjgvHD179tSZM2eqqurTTz+tf/vb31RVdc+ePaqqmpycrN26ddOEhAQvxm66dOlSVVVt2LChJiYmanx8vLZq1UoPHTqk+/bt07POOktHjhypqqq7d+9Oe6+HH35Yx4wZo6qqN9xwg86YMSNtXuD1kSNHtH79+rpmzRpVVb3uuuv0hRdeSHu/wPpjx47Vm2666YT9OXTokB45ckRVVdeuXauBz/bs2bO1Y8eOeujQoQz71759e33//fdVVfXIkSN66NAhnT9/vvbs2TNtm3fccYdO8g50w4YN9dlnn02bl9X+DRgwIC3u5ORk3bt3r27atEljY2NVVTUlJUWbNGmSYX1V1SVLlmi/fv1UVfX888/Xdu3a6fHjx3XEiBE6bty4DMd906ZN2rJly7R158+fr6eccopu3bpVU1JStEOHDrpo0aITjlHAyy+/rP3791dV1UGDBqUtu2XLFm3WrJmqqj722GMaFxenhw8fVlXVXr166eTJk1VV9bXXXtPevXurqmqrVq1027Ztqqr6xx9/nPBeWR3/bt266X333aeqqh9//LH26NFDVbP+P2a3j1l9PoYPH65TpkxJi+2cc87RgwcPZvg/9+rVS7/66itVVT1w4IAmJSWdsA+Zv6+qqkC8ZnFejeSFcvWArUGvt+FKBMESgL7Ai0AfoKqI1Mhi3XqZ30BEbgFuAWjQoEGegh0/HqKjoW3IC85NuLIbCLRSpezn16yZ/fys2HDfJXO476+//prx48fz1VdfpR2f4Cq//fv3pw1ud8UVV1CxYkUAlixZknbcr7vuurT7anTu3JkhQ4YwYMCAtP9RsFDHPyDU0ONZ/R9z2sdQn4/PP/+cWbNmpbWLHT16NK3kExBqGPO88vtK6vuBl0VkCLAQ2A6EXSmqqq8Cr4Ibiym3QSxf7h4vvQSZBps0RYAN932i4j7c986dO7npppuYNWtW2j2ZU1NT+eabb0Ier+D9zcq4ceP49ttv+fjjj9OSXObkl1PMwfFm93/Mbh9DbUtVee+9904YmTZwvwgIPYx58OCLuRHJRurtwJlBr+t709Ko6g5V7auqscDD3rS94aybnyZMcBfFedWjpoix4b5L1nDfSUlJ9O/fn2effTatlAZuxNOXXnop7fWKFStCrt+pU6e0jg1Tp05NG2V2w4YNnHfeeTzxxBPUqlWLrVu3Zlgv1PHPTlb/x9y45JJLeOmll9LaKr7//vsTlgk1jHleRTJBLAXOEZHGIlIOuBqYFbyAiNQUkUAMw4GJ3vPPgD+JyKle4/SfvGn57vBhmDoV+vWDUyPaDG4iyYb7LjnDfS9evJj4+Hgee+yxtEbZHTt2MGbMGOLj44mOjqZFixaMGzcu5PovvfQSkyZNIjo6milTpqQ1sj/wwANERUXRqlUrOnXqRExMTIb1sjr+Wcnq/5gbjzzyCElJSURHR9OyZUseeeSRE5YJNYx5nmXVOJEfD+ByYC2wAXjYm/YEcIX3vB+wzltmAlA+aN2hwHrvcWNO75XbRurt21Wvvlp14cJcrV7ihWr0MsVbSkqKxsTE6Nq1a/0OxZykwtRIjarOBmZnmvZo0PN3gXezWHci6SWKiKlbF95+O9LvYkzxsGrVKnr16kWfPn1suO8SwO9GamNMEWLDfZcsNtSGyTMtJnclNKY4y8331BKEyZMKFSqwZ88eSxLGFGKqyp49e066y7RVMZk8qV+/Ptu2bSMxMdHvUIwx2ahQocJJXzxnCcLkSeCiKGNM8WNVTMYYY0KyBGGMMSYkSxDGGGNCkuLS+0REEoEtfseRRzWB3X4HUYjY8cjIjkc6OxYZ5eV4NFTVWqFmFJsEURyISLyq2oDjHjseGdnxSGfHIqNIHQ+rYjLGGBOSJQhjjDEhWYIoXF71O4BCxo5HRnY80tmxyCgix8PaIIwxxoRkJQhjjDEhWYIwxhgTkiWIQkBEzhSR+SKySkR+EpG/+h2T30SktIh8LyIf+R2L30Skuoi8KyKrReRnEenod0x+EpF7ve/JjyLytoic3BClRZyITBSR30Tkx6Bpp4nIHBFZ5/3NlxsoW4IoHJKBv6lqC6ADcIeItPA5Jr/9FfjZ7yAKiReBT1W1GRBDCT4uIlIPuBtoq6qtgNK4+92XJJOBSzNNewiYp6rnAPO813lmCaIQUNWdqrrce34AdwKo529U/hGR+kBP3H3KSzQRqQZ0BV4DUNXjqrrX36h8VwaoKCJlgErADp/jKVCquhD4PdPk3sDr3vPXgSvz470sQRQyItIIiAW+9TcSX40G/g6k+h1IIdAYSAQmeVVuE0Skst9B+UVVtwOjgF+AncA+Vf3c36gKhdqqutN7vguonR8btQRRiIhIFeA94B5V3e93PH4QkV7Ab6q6zO9YCokyQBzwX1WNBQ6RT9UHRZFXt94blzjrApVF5Fp/oypc1F27kC/XL1iCKCREpCwuOUxV1ff9jsdHnYErRGQzMA24UETe9DckX20DtqlqoET5Li5hlFQXAZtUNVFVk4D3gU4+x1QY/CoidQC8v7/lx0YtQRQCIiK4OuafVfV5v+Pxk6oOV9X6qtoI1/j4haqW2F+IqroL2CoiTb1JPYBVPobkt1+ADiJSyfve9KAEN9oHmQXc4D2/AfgwPzZqCaJw6Axch/u1vMJ7XO53UKbQuAuYKiIrgdbAv32OxzdeSepdYDnwA+4cVqKG3RCRt4ElQFMR2SYiNwHPABeLyDpcKeuZfHkvG2rDGGNMKFaCMMYYE5IlCGOMMSFZgjDGGBOSJQhjCCthhgAAAeFJREFUjDEhWYIwxhgTkiUIY3IgIilB3Y9XiEi+XcksIo2CR+U0pjAp43cAxhQBR1S1td9BGFPQrARhTC6JyGYReU5EfhCR70TkbG96IxH5QkRWisg8EWngTa8tIh+ISIL3CAwRUVpExnv3OPhcRCp6y9/t3SNkpYhM82k3TQlmCcKYnFXMVMU0MGjePlWNAl7GjUIL8BLwuqpGA1OBMd70McCXqhqDG0/pJ2/6OcBYVW0J7AWu8qY/BMR627k1UjtnTFbsSmpjciAiB1W1Sojpm4ELVXWjN9jiLlWtISK7gTqqmuRN36mqNUUkEaivqseCttEImOPd6AUReRAoq6r/EpFPgYPATGCmqh6M8K4ak4GVIIzJG83i+ck4FvQ8hfS2wZ7AWFxpY6l3gxxjCowlCGPyZmDQ3yXe88Wk3wZzMLDIez4PuA3S7rldLauNikgp4ExVnQ88CFQDTijFGBNJ9ovEmJxVFJEVQa8/VdVAV9dTvVFWjwGDvGl34e4A9wDubnA3etP/Crzqjb6ZgksWOwmtNPCml0QEGGO3GjUFzdogjMklrw2iraru9jsWYyLBqpiMMcaEZCUIY4wxIVkJwhhjTEiWIIwxxoRkCcIYY0xIliCMMcaEZAnCGGNMSP8PIqq53JVgOjcAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "val_acc_noise = history_noise.history[\"val_accuracy\"]\n",
        "val_acc_zeros = history_zeros.history[\"val_accuracy\"]\n",
        "epochs = range(1, 11)\n",
        "plt.plot(epochs, val_acc_noise, \"b-\",\n",
        "         label=\"Validation accuracy with noise channels\")\n",
        "plt.plot(epochs, val_acc_zeros, \"b--\",\n",
        "         label=\"Validation accuracy with zeros channels\")\n",
        "plt.title(\"Effect of noise channels on validation accuracy\")\n",
        "plt.xlabel(\"Epochs\")\n",
        "plt.ylabel(\"Accuracy\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ve-7nJARTk3N"
      },
      "source": [
        "### 딥러닝에서 일반화의 본질"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mMxFZdiQTk3O"
      },
      "source": [
        "**랜덤하게 섞은 레이블로 MNIST 모델 훈련하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:02:26.358294Z",
          "iopub.status.busy": "2022-05-21T01:02:26.357787Z",
          "iopub.status.idle": "2022-05-21T01:05:33.171731Z",
          "shell.execute_reply": "2022-05-21T01:05:33.170427Z"
        },
        "id": "8ocMHIMMTk3O",
        "outputId": "0316c007-217d-45cd-8241-dcdb647bee16",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/100\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 2.3186 - accuracy: 0.1031 - val_loss: 2.3072 - val_accuracy: 0.1043\n",
            "Epoch 2/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.2998 - accuracy: 0.1152 - val_loss: 2.3135 - val_accuracy: 0.1005\n",
            "Epoch 3/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.2899 - accuracy: 0.1271 - val_loss: 2.3189 - val_accuracy: 0.1002\n",
            "Epoch 4/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.2763 - accuracy: 0.1399 - val_loss: 2.3280 - val_accuracy: 0.1031\n",
            "Epoch 5/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.2591 - accuracy: 0.1542 - val_loss: 2.3431 - val_accuracy: 0.0995\n",
            "Epoch 6/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.2381 - accuracy: 0.1685 - val_loss: 2.3538 - val_accuracy: 0.1006\n",
            "Epoch 7/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.2124 - accuracy: 0.1839 - val_loss: 2.3768 - val_accuracy: 0.1002\n",
            "Epoch 8/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.1854 - accuracy: 0.2006 - val_loss: 2.3873 - val_accuracy: 0.1012\n",
            "Epoch 9/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.1549 - accuracy: 0.2142 - val_loss: 2.4122 - val_accuracy: 0.0979\n",
            "Epoch 10/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.1223 - accuracy: 0.2318 - val_loss: 2.4398 - val_accuracy: 0.0990\n",
            "Epoch 11/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.0893 - accuracy: 0.2485 - val_loss: 2.4616 - val_accuracy: 0.0998\n",
            "Epoch 12/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.0555 - accuracy: 0.2647 - val_loss: 2.4841 - val_accuracy: 0.0973\n",
            "Epoch 13/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.0202 - accuracy: 0.2788 - val_loss: 2.5276 - val_accuracy: 0.1006\n",
            "Epoch 14/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.9841 - accuracy: 0.2942 - val_loss: 2.5473 - val_accuracy: 0.0988\n",
            "Epoch 15/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.9489 - accuracy: 0.3109 - val_loss: 2.5792 - val_accuracy: 0.0996\n",
            "Epoch 16/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.9130 - accuracy: 0.3245 - val_loss: 2.6166 - val_accuracy: 0.0995\n",
            "Epoch 17/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.8770 - accuracy: 0.3402 - val_loss: 2.6580 - val_accuracy: 0.0980\n",
            "Epoch 18/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.8433 - accuracy: 0.3550 - val_loss: 2.6808 - val_accuracy: 0.0995\n",
            "Epoch 19/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.8088 - accuracy: 0.3680 - val_loss: 2.7338 - val_accuracy: 0.0979\n",
            "Epoch 20/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.7760 - accuracy: 0.3830 - val_loss: 2.7485 - val_accuracy: 0.0988\n",
            "Epoch 21/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.7443 - accuracy: 0.3923 - val_loss: 2.8009 - val_accuracy: 0.1004\n",
            "Epoch 22/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.7121 - accuracy: 0.4042 - val_loss: 2.8423 - val_accuracy: 0.0993\n",
            "Epoch 23/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.6805 - accuracy: 0.4171 - val_loss: 2.8660 - val_accuracy: 0.1023\n",
            "Epoch 24/100\n",
            "375/375 [==============================] - 2s 5ms/step - loss: 1.6496 - accuracy: 0.4316 - val_loss: 2.9057 - val_accuracy: 0.0972\n",
            "Epoch 25/100\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 1.6222 - accuracy: 0.4423 - val_loss: 2.9563 - val_accuracy: 0.0954\n",
            "Epoch 26/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.5917 - accuracy: 0.4545 - val_loss: 2.9968 - val_accuracy: 0.0995\n",
            "Epoch 27/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.5636 - accuracy: 0.4605 - val_loss: 3.0348 - val_accuracy: 0.1004\n",
            "Epoch 28/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.5354 - accuracy: 0.4737 - val_loss: 3.0866 - val_accuracy: 0.0952\n",
            "Epoch 29/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.5103 - accuracy: 0.4828 - val_loss: 3.1236 - val_accuracy: 0.0982\n",
            "Epoch 30/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.4831 - accuracy: 0.4900 - val_loss: 3.1906 - val_accuracy: 0.0980\n",
            "Epoch 31/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.4577 - accuracy: 0.5029 - val_loss: 3.2379 - val_accuracy: 0.0943\n",
            "Epoch 32/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.4322 - accuracy: 0.5096 - val_loss: 3.2618 - val_accuracy: 0.0991\n",
            "Epoch 33/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.4090 - accuracy: 0.5195 - val_loss: 3.3252 - val_accuracy: 0.0960\n",
            "Epoch 34/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.3839 - accuracy: 0.5294 - val_loss: 3.3417 - val_accuracy: 0.0966\n",
            "Epoch 35/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.3598 - accuracy: 0.5364 - val_loss: 3.4162 - val_accuracy: 0.0969\n",
            "Epoch 36/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.3375 - accuracy: 0.5445 - val_loss: 3.4589 - val_accuracy: 0.0967\n",
            "Epoch 37/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.3136 - accuracy: 0.5531 - val_loss: 3.5233 - val_accuracy: 0.0997\n",
            "Epoch 38/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.2943 - accuracy: 0.5593 - val_loss: 3.5791 - val_accuracy: 0.0983\n",
            "Epoch 39/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.2713 - accuracy: 0.5673 - val_loss: 3.6012 - val_accuracy: 0.0960\n",
            "Epoch 40/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.2496 - accuracy: 0.5776 - val_loss: 3.6670 - val_accuracy: 0.0997\n",
            "Epoch 41/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.2283 - accuracy: 0.5854 - val_loss: 3.7607 - val_accuracy: 0.1000\n",
            "Epoch 42/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.2087 - accuracy: 0.5919 - val_loss: 3.7635 - val_accuracy: 0.1016\n",
            "Epoch 43/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.1901 - accuracy: 0.5993 - val_loss: 3.8218 - val_accuracy: 0.0969\n",
            "Epoch 44/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.1692 - accuracy: 0.6067 - val_loss: 3.8766 - val_accuracy: 0.0982\n",
            "Epoch 45/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.1501 - accuracy: 0.6154 - val_loss: 3.9646 - val_accuracy: 0.0979\n",
            "Epoch 46/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.1351 - accuracy: 0.6165 - val_loss: 3.9600 - val_accuracy: 0.0982\n",
            "Epoch 47/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.1132 - accuracy: 0.6260 - val_loss: 4.0421 - val_accuracy: 0.0989\n",
            "Epoch 48/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 1.0965 - accuracy: 0.6302 - val_loss: 4.0925 - val_accuracy: 0.0962\n",
            "Epoch 49/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.0788 - accuracy: 0.6388 - val_loss: 4.1590 - val_accuracy: 0.0976\n",
            "Epoch 50/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.0611 - accuracy: 0.6461 - val_loss: 4.2004 - val_accuracy: 0.0961\n",
            "Epoch 51/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.0452 - accuracy: 0.6500 - val_loss: 4.3015 - val_accuracy: 0.0936\n",
            "Epoch 52/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.0287 - accuracy: 0.6556 - val_loss: 4.3338 - val_accuracy: 0.1009\n",
            "Epoch 53/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 1.0127 - accuracy: 0.6629 - val_loss: 4.3611 - val_accuracy: 0.1011\n",
            "Epoch 54/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.9988 - accuracy: 0.6665 - val_loss: 4.4318 - val_accuracy: 0.0972\n",
            "Epoch 55/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.9826 - accuracy: 0.6719 - val_loss: 4.4911 - val_accuracy: 0.0979\n",
            "Epoch 56/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.9643 - accuracy: 0.6779 - val_loss: 4.5594 - val_accuracy: 0.0988\n",
            "Epoch 57/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.9509 - accuracy: 0.6826 - val_loss: 4.5985 - val_accuracy: 0.0988\n",
            "Epoch 58/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.9366 - accuracy: 0.6877 - val_loss: 4.6551 - val_accuracy: 0.1031\n",
            "Epoch 59/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.9263 - accuracy: 0.6910 - val_loss: 4.7360 - val_accuracy: 0.0979\n",
            "Epoch 60/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.9067 - accuracy: 0.6985 - val_loss: 4.7859 - val_accuracy: 0.0980\n",
            "Epoch 61/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.8946 - accuracy: 0.7025 - val_loss: 4.8275 - val_accuracy: 0.0989\n",
            "Epoch 62/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.8781 - accuracy: 0.7095 - val_loss: 4.9280 - val_accuracy: 0.0955\n",
            "Epoch 63/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.8655 - accuracy: 0.7144 - val_loss: 4.9603 - val_accuracy: 0.0980\n",
            "Epoch 64/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.8543 - accuracy: 0.7177 - val_loss: 5.0255 - val_accuracy: 0.0976\n",
            "Epoch 65/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.8408 - accuracy: 0.7217 - val_loss: 5.0733 - val_accuracy: 0.0986\n",
            "Epoch 66/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.8266 - accuracy: 0.7276 - val_loss: 5.1761 - val_accuracy: 0.0940\n",
            "Epoch 67/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.8154 - accuracy: 0.7306 - val_loss: 5.2076 - val_accuracy: 0.0965\n",
            "Epoch 68/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.8016 - accuracy: 0.7348 - val_loss: 5.2610 - val_accuracy: 0.0971\n",
            "Epoch 69/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.7899 - accuracy: 0.7410 - val_loss: 5.3232 - val_accuracy: 0.0960\n",
            "Epoch 70/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.7794 - accuracy: 0.7421 - val_loss: 5.3920 - val_accuracy: 0.1001\n",
            "Epoch 71/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.7674 - accuracy: 0.7476 - val_loss: 5.4507 - val_accuracy: 0.0969\n",
            "Epoch 72/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.7548 - accuracy: 0.7519 - val_loss: 5.5362 - val_accuracy: 0.0983\n",
            "Epoch 73/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.7438 - accuracy: 0.7580 - val_loss: 5.5741 - val_accuracy: 0.0978\n",
            "Epoch 74/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.7345 - accuracy: 0.7598 - val_loss: 5.6117 - val_accuracy: 0.0986\n",
            "Epoch 75/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.7224 - accuracy: 0.7666 - val_loss: 5.6968 - val_accuracy: 0.0982\n",
            "Epoch 76/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.7116 - accuracy: 0.7681 - val_loss: 5.7686 - val_accuracy: 0.0978\n",
            "Epoch 77/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.7010 - accuracy: 0.7710 - val_loss: 5.8179 - val_accuracy: 0.0957\n",
            "Epoch 78/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.6926 - accuracy: 0.7730 - val_loss: 5.9001 - val_accuracy: 0.0996\n",
            "Epoch 79/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.6785 - accuracy: 0.7779 - val_loss: 6.0065 - val_accuracy: 0.0981\n",
            "Epoch 80/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.6697 - accuracy: 0.7811 - val_loss: 5.9748 - val_accuracy: 0.0962\n",
            "Epoch 81/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.6605 - accuracy: 0.7837 - val_loss: 6.1038 - val_accuracy: 0.0993\n",
            "Epoch 82/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.6519 - accuracy: 0.7861 - val_loss: 6.1116 - val_accuracy: 0.0982\n",
            "Epoch 83/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.6421 - accuracy: 0.7935 - val_loss: 6.2254 - val_accuracy: 0.0973\n",
            "Epoch 84/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.6318 - accuracy: 0.7950 - val_loss: 6.2705 - val_accuracy: 0.0985\n",
            "Epoch 85/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.6226 - accuracy: 0.7975 - val_loss: 6.3630 - val_accuracy: 0.0984\n",
            "Epoch 86/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.6150 - accuracy: 0.7994 - val_loss: 6.4845 - val_accuracy: 0.0974\n",
            "Epoch 87/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.6059 - accuracy: 0.8046 - val_loss: 6.4846 - val_accuracy: 0.0990\n",
            "Epoch 88/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.5958 - accuracy: 0.8067 - val_loss: 6.5508 - val_accuracy: 0.0988\n",
            "Epoch 89/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.5899 - accuracy: 0.8086 - val_loss: 6.6126 - val_accuracy: 0.0976\n",
            "Epoch 90/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.5804 - accuracy: 0.8112 - val_loss: 6.6667 - val_accuracy: 0.0968\n",
            "Epoch 91/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.5712 - accuracy: 0.8164 - val_loss: 6.7257 - val_accuracy: 0.0975\n",
            "Epoch 92/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.5629 - accuracy: 0.8184 - val_loss: 6.8342 - val_accuracy: 0.0983\n",
            "Epoch 93/100\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 0.5571 - accuracy: 0.8215 - val_loss: 6.9250 - val_accuracy: 0.0980\n",
            "Epoch 94/100\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 0.5470 - accuracy: 0.8248 - val_loss: 6.9499 - val_accuracy: 0.0957\n",
            "Epoch 95/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.5406 - accuracy: 0.8264 - val_loss: 7.0343 - val_accuracy: 0.0937\n",
            "Epoch 96/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.5331 - accuracy: 0.8278 - val_loss: 7.1203 - val_accuracy: 0.0983\n",
            "Epoch 97/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.5236 - accuracy: 0.8311 - val_loss: 7.1420 - val_accuracy: 0.0961\n",
            "Epoch 98/100\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.5185 - accuracy: 0.8343 - val_loss: 7.2705 - val_accuracy: 0.0988\n",
            "Epoch 99/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.5110 - accuracy: 0.8358 - val_loss: 7.2819 - val_accuracy: 0.0998\n",
            "Epoch 100/100\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.5029 - accuracy: 0.8384 - val_loss: 7.3875 - val_accuracy: 0.0981\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f70b409f150>"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "(train_images, train_labels), _ = mnist.load_data()\n",
        "train_images = train_images.reshape((60000, 28 * 28))\n",
        "train_images = train_images.astype(\"float32\") / 255\n",
        "\n",
        "random_train_labels = train_labels[:]\n",
        "np.random.shuffle(random_train_labels)\n",
        "\n",
        "model = keras.Sequential([\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "model.fit(train_images, random_train_labels,\n",
        "          epochs=100,\n",
        "          batch_size=128,\n",
        "          validation_split=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oXAyqCrXTk3P"
      },
      "source": [
        "#### 매니폴드 가설"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L0Uvbr43Tk3P"
      },
      "source": [
        "#### 일반화의 원천인 보간"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pHMR5mNZTk3P"
      },
      "source": [
        "#### 딥러닝이 작동하는 이유"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eb0YYZR4Tk3P"
      },
      "source": [
        "#### 가장 중요한 훈련 데이터"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PunfvZ7uTk3Q"
      },
      "source": [
        "## 머신 러닝 모델 평가"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5P4VJfMFTk3Q"
      },
      "source": [
        "### 훈련, 검증, 테스트 세트"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mrwB851-Tk3Q"
      },
      "source": [
        "#### 단순 홀드아웃 검증"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1sEzzYjjTk3R"
      },
      "source": [
        "#### K-겹 교차 검증"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XYrpFDZQTk3R"
      },
      "source": [
        "#### 셔플링을 사용한 반복 K-겹 교차 검증"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kngFjE0QTk3R"
      },
      "source": [
        "### 상식 수준의 기준점을 넘기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "czziT5WVTk3R"
      },
      "source": [
        "### 모델 평가에 대해 유념해야 할 점"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p6nofQfTTk3R"
      },
      "source": [
        "## 훈련 성능 향상하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XtWaoT4BTk3S"
      },
      "source": [
        "### 경사 하강법의 핵심 파라미터 튜닝하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pRY-dxWTTk3S"
      },
      "source": [
        "**잘못된 높은 학습률로 MNIST 모델 훈련하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:05:33.177460Z",
          "iopub.status.busy": "2022-05-21T01:05:33.177117Z",
          "iopub.status.idle": "2022-05-21T01:05:52.585061Z",
          "shell.execute_reply": "2022-05-21T01:05:52.584241Z"
        },
        "id": "ZqvcFMPRTk3T",
        "outputId": "18e0b507-88da-465a-f1c8-073a2760fb67",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 1072.8800 - accuracy: 0.4609 - val_loss: 8.7493 - val_accuracy: 0.3795\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 4.0622 - accuracy: 0.3121 - val_loss: 2.1783 - val_accuracy: 0.3994\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 4.2331 - accuracy: 0.3183 - val_loss: 2.0610 - val_accuracy: 0.3086\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.9689 - accuracy: 0.3133 - val_loss: 4.8494 - val_accuracy: 0.3406\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.5279 - accuracy: 0.2779 - val_loss: 2.1086 - val_accuracy: 0.2690\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.6724 - accuracy: 0.3167 - val_loss: 2.4861 - val_accuracy: 0.3088\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.6682 - accuracy: 0.2937 - val_loss: 2.5677 - val_accuracy: 0.3142\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.3338 - accuracy: 0.3061 - val_loss: 2.0164 - val_accuracy: 0.3242\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 2.6984 - accuracy: 0.3266 - val_loss: 3.2928 - val_accuracy: 0.3170\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 2.3395 - accuracy: 0.3095 - val_loss: 2.5336 - val_accuracy: 0.3151\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f709e681b50>"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "(train_images, train_labels), _ = mnist.load_data()\n",
        "train_images = train_images.reshape((60000, 28 * 28))\n",
        "train_images = train_images.astype(\"float32\") / 255\n",
        "\n",
        "model = keras.Sequential([\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(optimizer=keras.optimizers.RMSprop(1.),\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "model.fit(train_images, train_labels,\n",
        "          epochs=10,\n",
        "          batch_size=128,\n",
        "          validation_split=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77iXsoAYTk3U"
      },
      "source": [
        "**같은 모델을 적절한 학습률로 훈련하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:05:52.588549Z",
          "iopub.status.busy": "2022-05-21T01:05:52.588260Z",
          "iopub.status.idle": "2022-05-21T01:06:11.526008Z",
          "shell.execute_reply": "2022-05-21T01:06:11.525112Z"
        },
        "id": "TQAqlYm0Tk3U",
        "outputId": "7b26b962-1097-426b-a3b0-16213cc0194b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 0.3588 - accuracy: 0.9128 - val_loss: 0.2375 - val_accuracy: 0.9398\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.1431 - accuracy: 0.9638 - val_loss: 0.1824 - val_accuracy: 0.9566\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.1178 - accuracy: 0.9721 - val_loss: 0.1684 - val_accuracy: 0.9709\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0987 - accuracy: 0.9786 - val_loss: 0.1908 - val_accuracy: 0.9693\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0854 - accuracy: 0.9819 - val_loss: 0.2492 - val_accuracy: 0.9664\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0858 - accuracy: 0.9835 - val_loss: 0.2023 - val_accuracy: 0.9728\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0713 - accuracy: 0.9859 - val_loss: 0.2328 - val_accuracy: 0.9728\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0629 - accuracy: 0.9884 - val_loss: 0.2934 - val_accuracy: 0.9693\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0601 - accuracy: 0.9896 - val_loss: 0.3093 - val_accuracy: 0.9735\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0609 - accuracy: 0.9899 - val_loss: 0.3793 - val_accuracy: 0.9707\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f71100bba50>"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(10, activation=\"softmax\")\n",
        "])\n",
        "model.compile(optimizer=keras.optimizers.RMSprop(1e-2),\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "model.fit(train_images, train_labels,\n",
        "          epochs=10,\n",
        "          batch_size=128,\n",
        "          validation_split=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9IVJeVeOTk3V"
      },
      "source": [
        "### 구조에 대해 더 나은 가정하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Br1F2jcwTk3V"
      },
      "source": [
        "### 모델 용량 늘리기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ubvdcv_-Tk3V"
      },
      "source": [
        "**MNIST 데이터를 사용한 간단한 로지스틱 회귀 모델**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:06:11.530131Z",
          "iopub.status.busy": "2022-05-21T01:06:11.529560Z",
          "iopub.status.idle": "2022-05-21T01:06:25.955536Z",
          "shell.execute_reply": "2022-05-21T01:06:25.954654Z"
        },
        "id": "xl5ejzSRTk3W",
        "outputId": "515b5c38-0fc7-41cb-a8ab-a217997afff3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 0.6689 - accuracy: 0.8348 - val_loss: 0.3602 - val_accuracy: 0.9028\n",
            "Epoch 2/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.3514 - accuracy: 0.9035 - val_loss: 0.3079 - val_accuracy: 0.9149\n",
            "Epoch 3/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.3153 - accuracy: 0.9124 - val_loss: 0.2902 - val_accuracy: 0.9187\n",
            "Epoch 4/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2990 - accuracy: 0.9168 - val_loss: 0.2811 - val_accuracy: 0.9221\n",
            "Epoch 5/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2896 - accuracy: 0.9187 - val_loss: 0.2752 - val_accuracy: 0.9232\n",
            "Epoch 6/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2830 - accuracy: 0.9209 - val_loss: 0.2722 - val_accuracy: 0.9245\n",
            "Epoch 7/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2781 - accuracy: 0.9222 - val_loss: 0.2692 - val_accuracy: 0.9254\n",
            "Epoch 8/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2744 - accuracy: 0.9240 - val_loss: 0.2689 - val_accuracy: 0.9266\n",
            "Epoch 9/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2714 - accuracy: 0.9251 - val_loss: 0.2658 - val_accuracy: 0.9282\n",
            "Epoch 10/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2687 - accuracy: 0.9254 - val_loss: 0.2647 - val_accuracy: 0.9276\n",
            "Epoch 11/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2667 - accuracy: 0.9257 - val_loss: 0.2660 - val_accuracy: 0.9268\n",
            "Epoch 12/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2649 - accuracy: 0.9268 - val_loss: 0.2627 - val_accuracy: 0.9283\n",
            "Epoch 13/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2632 - accuracy: 0.9272 - val_loss: 0.2630 - val_accuracy: 0.9283\n",
            "Epoch 14/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2618 - accuracy: 0.9280 - val_loss: 0.2619 - val_accuracy: 0.9296\n",
            "Epoch 15/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2606 - accuracy: 0.9281 - val_loss: 0.2616 - val_accuracy: 0.9298\n",
            "Epoch 16/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2593 - accuracy: 0.9293 - val_loss: 0.2607 - val_accuracy: 0.9293\n",
            "Epoch 17/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2584 - accuracy: 0.9287 - val_loss: 0.2628 - val_accuracy: 0.9296\n",
            "Epoch 18/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2573 - accuracy: 0.9297 - val_loss: 0.2608 - val_accuracy: 0.9307\n",
            "Epoch 19/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2563 - accuracy: 0.9294 - val_loss: 0.2615 - val_accuracy: 0.9305\n",
            "Epoch 20/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.2556 - accuracy: 0.9307 - val_loss: 0.2614 - val_accuracy: 0.9309\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([layers.Dense(10, activation=\"softmax\")])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_small_model = model.fit(\n",
        "    train_images, train_labels,\n",
        "    epochs=20,\n",
        "    batch_size=128,\n",
        "    validation_split=0.2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:06:25.959685Z",
          "iopub.status.busy": "2022-05-21T01:06:25.959399Z",
          "iopub.status.idle": "2022-05-21T01:06:26.277516Z",
          "shell.execute_reply": "2022-05-21T01:06:26.276506Z"
        },
        "id": "-yiFhvT4Tk3W",
        "outputId": "7421f121-24fa-40c4-9a7f-994733415b8f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5hU5fn/8fcNLCBVBESqIMIiiBQXG1iDUcSAXYgxYtefqInRaBKDqNGvsUvExBKxRMQWDVEMMQpiiQoSiihINYCI1ADS4f798ZyB2XF2d3bZ2bPl87quuXbmtLnnzNlzz1POeczdERERSVUt7gBERKR8UoIQEZG0lCBERCQtJQgREUlLCUJERNJSghARkbSqfIIws9+Z2Uoz+yZ6fbqZLTazDWbWI8a4Co0jmn5AlmPobWZzo/c6zcyamdkkM1tvZveZ2a/N7IkMtvMnM/ttNmMtC2Y2xMzez3DZp8zsd9mOqTSY2ZtmdkHccZQWMzvOzJYkvZ5lZsdlsmwJ3isrx7aZDTezv5T2dourRtwBZJuZLQKaATuSJj/l7kPNrA3wC2B/d/82mncvMNTd/7aH7+tAB3efV8JNFBqHu9crcXCZuw142N0fAoj+EVYCDbwYF9C4+xWlEUz0T/4Xd29VGtuTwN37JZ6b2RDgEnfvE19Epcvdu5TGdtLtm9I6tsurSp8gIj9y93+lmd4GWJWUHAD2B2aVTViFKg9xpMawP/B5cZKDiFRg7l6pH8AioG+a6X2BTcBOYAPwfPTXge+A+dFyLYBXgBXAQuCapG1UB34NzAfWA58CrYFJSdvZAJyb5v2rATcDXwHfAs8ADYFa6eJIs74DB0bPnwJGAm9EcXwMtI/mGfBA9B7rgJnAwdG8iYRfRIltDgHej57Pj/bNpqT9sw3YGr3uCwwn/KJPrN8H+BBYCywGhiTF97uk5U4FpkXLfQgckvJ9XQ/MAP4HvADUBuqmfF8bgBZp9stTwCPAm9EyHwD7AQ8Ca4DZQI+k5Q+K9sNaQjIckDSvMTA22m+fALcn9k80vxPwFrAamAOckxLH79J9d9H8S4Evou/rc6BnNP0mdh9PnwOnp3w/HwAPR/tmNvCDpPkXJm1zAXB5ynsOjPb7uug9Tk4+DqJ9sZlQ2t4Q7ZNewHKgetJ2zgCmF/C5GhKO5RWEY/tmoFry8UUoHa8h/D/1K2A7NwIvp0x7CBhR1GcFjgOWpDsHAHtF382aaP/ekLJs2v2fbt8UcGxfCsyLjomxJB2jhP/ZK4C50b4dCVgBn384+f+3BhCOz7XR93VQyr5aGsU8J3FMAIcBU6Lvezlwf7HPn6V9Qi5vDwpIEOkOpKQvMXHirUY46Q8DagIHRAfjSdH8Gwgn3FzCibgb0Dh1OwW890XRgXQAUA/4K/BsujgKWD81QayKDogawHPAmGjeSdFn2DuK8SCgeTRvIgUkiHT7Ls0/w66DmFC6WA8MBnIIJ9fuqesBPQjJ6nBCgr0gep9aSe/5CSEx70M4CVxR0PeVZr88RagGO5SQWN4hnIh+Gr3f74AJ0bI50Xfw6+j7PSH6DLnR/DHAi4TkdDDhnzCRQOsSkuCF0T7vEb1v53T7KiXGs6Nt9Yq+kwMJ1ZyJeS0Ix965hB8JzZO+n+3Az6PYzyUkin2i+f2B9tE2jwU2sjvxHBYte2K07ZZAp9TjIPUYiKZ9TtKJHHgV+EUBn+0Z4G9AfaAt8CVwcdK2txFOotWBK4GvSXOSJBxPG4H60evqwDLgiAw+a77jhPwJ4i7gPcKx1Rr4LGXZovZ/6r7Z9T0Tjp+VQE/CD70/AJNS/mdfJ/wvtiEk0ZML2I/D2f2/1TGK48Toe/8l4bitSTj3LCZKRNE+T/w4/DdwfvS8XmLfFedRVRqpXzOztUmPSzNcrxfQ1N1vc/et7r4AeBwYFM2/BLjZ3ed4MN3dV2W47fMIGX2Bu28AfgUMMrOSVvu96u6fuPt2QoLoHk3fRvhn7UT4R/zC3ZeV8D0K82PgX+7+vLtvc/dV7j4tzXKXAY+6+8fuvsPdnwa2AEckLTPC3b9299XA35M+S6ZedfdP3X0z4WS22d2fcfcdhBJJotH/CMI/zl3R9/sO4R94sJlVB84Ehrn7d+7+GfB00nucCixy91Huvt3d/0MoaZ6dQXyXAHe7++TouJnn7l8BuPtL0Wff6e4vEH5tHpa07rfAg9E+foHwi7F/tO4b7j4/2ua7wD+Bo6P1LgaedPe3om0vdffZGe7Pp4GfAJjZPoQfHaNTF4r22SDgV+6+3t0XAfcB5yct9pW7Px59F08DzQlthPlE+2MqcHo06QRgo7t/lMFnLcw5wB3uvtrdFwMjUt63qP1fmPMI+3iqu28h/E8faWZtk5a5y93Xuvt/gQlkdmyfC7wRfXfbCCWwvYCjCCWaWkBnM8tx90XuPj9abxtwoJk1cfcNiX1XHFUlQZzm7nsnPR7PcL39gRbJyYXwazNxQLcmFEdLogWhCJ7wFeGX6Pf+WTL0TdLzjYQTH9FJ72FCcfZbM3vMzBqU8D0Kk+m+2B/4Rco+bU3YHwlpP0sxLE96vinN68T2WgCL3X1n0vyvCL+umxK+j8Up85I/x+Epn+M8QnVWUQrcV2b2UzOblrTNg4EmSYss9egnYVJMLaJ1+5nZR2a2Olr3lKR19+RY/QvwIzOrSzjBvlfAj4wmhF+4qcd1y6TXu75bd98YPS3o+x1NKJFC+AGyKykV8VkL04KCv9NM9n9R2961veiH3yoK+Pxkfmynbndn9BlaeugE8zNCieNbMxtjZon/pYsJpY/ZZjbZzE7N8HPsUlUSREktBhamJJf67n5K0vz2Jdz214STTEIbQvXB8vSLl5y7j3D3Q4HOhAPmhmjWd0CdpEUzObkVJNN9sZjwCy55n9Zx9+czWLe0G8e/BlqbWfL/QRtC9c8KwvfROmVewmLg3ZTPUc/dr8zgfdPuKzPbn1BCHUqoqtybUAViSYu1NLPk122Ar82sFqEEcy/QLFp3XNK6mX4/39vH7r6UUF1xBqE08GwB664k/GpNPa6XZvC+6bwEHGdmrQglidEAGXzWwiyjgO80g/1f1PGX7386SqiNKfnnL2i7RvgMSwHcfbSHnlX7RzH+Ppo+190HA/tG016OYsqYEkThPgHWm9mNZraXmVU3s4PNrFc0/wngdjPrYMEhZtY4mrec0L5QkOeBn5tZOzOrB9wJvBBVEZUaM+tlZoebWQ4hIWwmNPRCaLA8w8zqmNmBhF8cJfUc0NfMzjGzGmbW2MzSFZ8fB66IYjIzq2tm/c2sfgbvsRxobGYN9yDOZB8TfsX90sxyom60PyK03+wgtAsNj/ZPZ0J7ScLrQEczOz9aNyfa1wdl8L5PANeb2aHRPjgwOjnVJfyDrwAwswsJv2CT7QtcE73f2YQ2pXGE+uha0brbzawf8MOk9f4MXGhmPzCzambW0sw6pYltOdDKzGqmTH+GUPfdNdov3xPtsxeBO8ysfvSZriOUQIrN3VcQ2kdGEX6ofRHNKuqzFuZF4Fdm1ihKPFcnzStq/xe0bxKeJ+zj7lESuxP4OKpq2xMvAv2j7y6H0DV/C/ChmeWa2QnR+21md0cOzOwnZtY0KnGsjba1M832C1RVEsTfLVzslXi8mslK0QF/KqGecCHhF9IThJ4aAPcTvrx/EnoK/JlQNwihyPd0VFQ9J83mnyT8EpsUbXsz+Q/W0tKAcFJeQyimrgLuieY9QOiVtJxQH/xcSd8kqlM9hXDwriYkn25plptCaKR8OIppHqHxL5P3mE34J1wQ7dcWRa1TxPa2EhJCP8J3+wjw06S6+aGEKoBvCI2Ro5LWXU84KQ0i/ML7hvArrVYG7/sScAfhF/F64DVCQ/PnhDr7fxO+k66EXkvJPgY6RPHeAZzlob1nPXAN4XhcQ6iSGZv0np8QGtQfIDRWv0v+X/oJ7xB6y3xjZiuTpr8aLf9qUtVQOlcTfogsIPRYGk041ktqNKHH3K7qpaI+axFuJfwfLCT83+4qDWWw/wvaN4n1/wX8llC6WUYosQ1KXa643H0OoQ3oD4Tv/UeErvtbCcfbXdH0bwg/IH4VrXoyMMvMNhB6gA1y903FeW/LX50pIuWVxXwRm5nNJ3QnTXdNkVRCVaUEISJ7wMzOJFS/vBN3LFJ2qsqV1CJSQmY2kdDB4fyUHl9SyamKSURE0lIVk4iIpFVpqpiaNGnibdu2jTsMEZEK5dNPP13p7k3Tzas0CaJt27ZMmTIl7jBERCoUM/uqoHmqYhIRkbSUIEREJC0lCBERSavStEGISNnbtm0bS5YsYfPmzXGHIkWoXbs2rVq1IicnJ+N1lCBEpMSWLFlC/fr1adu2LflvMivlibuzatUqlixZQrt27TJeT1VMIlJimzdvpnHjxkoO5ZyZ0bhx42KX9LKaIMzsZDObY2bzzOymNPOvMLOZ0QAd70e3VE7MO8TM/m1ms6JlamczVhEpGSWHiqEk31PWEoSF4QdHEm6l3JkwjGPnlMVGu3tXd+8O3E24fTYWht38C2Es4i6EMWa3ZStWERH5vmyWIA4D5nkYc3krYQD4gckLuPu6pJeJwTog3Gd/hrtPj5ZbFY3NUOq2bYMTT4RHH83G1kUkm44//njGjx+fb9qDDz7IlVcWPLDfcccdt+ui2lNOOYW1a9d+b5nhw4dz7733Fvrer732Gp9//vmu18OGDeNf/9rzO6FPnDiRU08t9uigWZHNBNGS/GO/LiH/2KwAmNlV0X3m7yYMAgJhWEw3s/FmNtXMfpnuDczsMjObYmZTVqxYUaIgc3Lgs8/go2IP5y0icRs8eDBjxozJN23MmDEMHjy4gDXyGzduHHvvvXeJ3js1Qdx222307du3RNsqr2JvpHb3ke7eHrgRuDmaXAPoQxgEvg9wupn9IM26j7l7nrvnNW2a9lYiGenUCebMKfHqIhKTs846izfeeIOtW7cCsGjRIr7++muOPvporrzySvLy8ujSpQu33HJL2vXbtm3LypVhcLg77riDjh070qdPH+YknRAef/xxevXqRbdu3TjzzDPZuHEjH374IWPHjuWGG26ge/fuzJ8/nyFDhvDyyy8D8Pbbb9OjRw+6du3KRRddxJYtW3a93y233ELPnj3p2rUrs2fP/n5QSVavXs1pp53GIYccwhFHHMGMGTMAePfdd+nevTvdu3enR48erF+/nmXLlnHMMcfQvXt3Dj74YN57770927lkN0EsJf/g4K0ofPDuMcBp0fMlwCR3XxkNbzgO6JmVKAkJYvZs0J3PRfbMccd9//HII2Hexo3p5z/1VJi/cuX35xVln3324bDDDuPNN98EQunhnHPOwcy44447mDJlCjNmzODdd9/ddXJN59NPP2XMmDFMmzaNcePGMXny5F3zzjjjDCZPnsz06dM56KCD+POf/8xRRx3FgAEDuOeee5g2bRrt27fftfzmzZsZMmQIL7zwAjNnzmT79u388Y9/3DW/SZMmTJ06lSuvvLLIaqxbbrmFHj16MGPGDO68805++tOfAnDvvfcycuRIpk2bxnvvvcdee+3F6NGjOemkk5g2bRrTp0+ne/d0Q8IXTzYTxGSgg5m1iwb5HkTKuLFm1iHpZX9gbvR8PNA1Giy+BnAs8DlZkpsLa9aEA1REKpbkaqbk6qUXX3yRnj170qNHD2bNmpWvOijVe++9x+mnn06dOnVo0KABAwYM2DXvs88+4+ijj6Zr164899xzzJo1q9B45syZQ7t27ejYsSMAF1xwAZMmTdo1/4wzzgDg0EMPZdGiRYVu6/333+f8888H4IQTTmDVqlWsW7eO3r17c9111zFixAjWrl1LjRo16NWrF6NGjWL48OHMnDmT+vXrF7rtTGTtQjl3325mQwkn++rAk+4+y8xuA6a4+1hgqJn1JfRQWgNcEK27xszuJyQZB8a5+xvZirVHD+jbF9avhz2oqRKp8iZOLHhenTqFz2/SpPD5BRk4cCA///nPmTp1Khs3buTQQw9l4cKF3HvvvUyePJlGjRoxZMiQEl/tPWTIEF577TW6devGU089xcSSBJmkVq1aAFSvXp3t27eXaBs33XQT/fv3Z9y4cfTu3Zvx48dzzDHHMGnSJN544w2GDBnCddddt6vEUVJZbYNw93Hu3tHd27v7HdG0YVFywN2vdfcu7t7d3Y9391lJ6/4lmnewu6dtpC4txx4Lb70FBxyQzXcRkWyoV68exx9/PBdddNGu0sO6deuoW7cuDRs2ZPny5buqoApyzDHH8Nprr7Fp0ybWr1/P3//+913z1q9fT/Pmzdm2bRvPPffcrun169dn/fr139tWbm4uixYtYt68eQA8++yzHHvssSX6bEcfffSu95w4cSJNmjShQYMGzJ8/n65du3LjjTfSq1cvZs+ezVdffUWzZs249NJLueSSS5g6dWqJ3jOZbrWRxB10zY9IxTN48GBOP/30XVVN3bp1o0ePHnTq1InWrVvTu3fvQtfv2bMn5557Lt26dWPfffelV69eu+bdfvvtHH744TRt2pTDDz98V1IYNGgQl156KSNGjNjVOA3hnkejRo3i7LPPZvv27fTq1YsrrriiRJ9r+PDhXHTRRRxyyCHUqVOHp59+GghdeSdMmEC1atXo0qUL/fr1Y8yYMdxzzz3k5ORQr149nnnmmRK9Z7JKMyZ1Xl6e78mAQQMHQo0a8MorpRiUSCX3xRdfcNBBB8UdhmQo3fdlZp+6e1665WPv5lpe1KwJM2fGHYWISPmhBBHp1AkWLICoO7WISJWnBBHJzYUdO2D+/LgjEalYKks1dWVXku9JCSLSqVP4W8SFjSKSpHbt2qxatUpJopxLjAdRu3bxboqtXkyR3FwYMgSaN487EpGKo1WrVixZsoSS3gtNyk5iRLniUIKI1K8Po0bFHYVIxZKTk1OsEcqkYlEVUxJ30A8hEZFACSLJtddCx466aZ+ICChB5NO+Paxdq1KEiAgoQeSTmxv+amwIEREliHzU1VVEZDcliCRt2kDt2ipBiIiAurnmU60a3H03dO0adyQiIvFTgkhx9dVxRyAiUj6oiinFhg3w0UewbVvckYiIxEsJIsXf/gZHHglz5xa9rIhIZaYEkSLR1VU9mUSkqlOCSKFrIUREAiWIFPXrQ8uWShAiIkoQaeTmqopJRETdXNO49VaoXj3uKERE4qUEkUafPnFHICISP1UxpbF+Pbz0EixaFHckIiLxUYJIY/VqOOcc+Oc/445ERCQ+ShBptG4Ne+2lnkwiUrUpQaRRrVoYWU49mUSkKlOCKEBurkoQIlK1KUEUoFMnWLgQtmyJOxIRkXhkNUGY2clmNsfM5pnZTWnmX2FmM81smpm9b2adU+a3MbMNZnZ9NuNM5/LLww37cnLK+p1FRMqHrF0HYWbVgZHAicASYLKZjXX3z5MWG+3uf4qWHwDcD5ycNP9+4M1sxViYFi3ieFcRkfIjmyWIw4B57r7A3bcCY4CByQu4+7qkl3UBT7wws9OAhcCsLMZYIHd46CF4M5b0JCISv2wmiJbA4qTXS6Jp+ZjZVWY2H7gbuCaaVg+4Ebi1sDcws8vMbIqZTVmxYkWpBR62DffeC88/X6qbFRGpMGJvpHb3ke7enpAQbo4mDwcecPcNRaz7mLvnuXte06ZNSz029WQSkaosm/diWgq0TnrdKppWkDHAH6PnhwNnmdndwN7ATjPb7O4PZyXSAnTqBH/5S6huMivLdxYRiV82E8RkoIOZtSMkhkHAj5MXMLMO7p4Y3LM/MBfA3Y9OWmY4sKGskwOEEsT//gfLl8N++5X1u4uIxCtrCcLdt5vZUGA8UB140t1nmdltwBR3HwsMNbO+wDZgDXBBtuIpidzcUHJYuFAJQkSqHnP3opeqAPLy8nzKlCmlus2tW2HHjnBfJhGRysjMPnX3vHTzNB5EIWrWjDsCEZH4xN6Lqbx78EEYPjzuKEREyp4SRBE+/hiefTbuKEREyp4SRBFyc0Mj9ebNcUciIlK2lCCKkJsbroOYNy/uSEREypYSRBE6dQp/NXiQiFQ1ShBF6NgR9t0XNhR60w8RkcpH3VyLULduuJJaRKSqUQlCRETSUoLIwKhR0Lt3aKwWEakqlCAy8N138OGH8M03cUciIlJ2lCAyoJ5MIlIVKUFkIDc3/NXgQSJSlShBZKBlS6hTRyUIEalalCAyUK0aDBwIzZvHHYmISNnRdRAZGj067ghERMqWShDFpK6uIlJVKEFk6K23oHFj+OyzuCMRESkbShAZatoUVq9WQ7WIVB1KEBnq0CH8VVdXEakqlCAyVLcutGmjEoSIVB1KEMWQm6sShIhUHermWgxnnw1ffx13FCIiZUMJohguvTTuCEREyo6qmIpp48Zwd1cRkcpOCaIYli0LjdXPPBN3JCIi2acEUQz77RcShBqqRaQqUIIoBrPQk0ldXUWkKlCCKCZ1dRWRqkIJopg6dYKvvgqN1SIilZm6uRZT//6w996wc2fckYiIZFdWSxBmdrKZzTGzeWZ2U5r5V5jZTDObZmbvm1nnaPqJZvZpNO9TMzshm3EWx6GHwjXXQL16cUciIpJdWUsQZlYdGAn0AzoDgxMJIMlod+/q7t2Bu4H7o+krgR+5e1fgAuDZbMVZEvPmhYeISGWWzRLEYcA8d1/g7luBMcDA5AXcfV3Sy7qAR9P/4+6Jm1rMAvYys1pZjLVY+vaF4cPjjkJEJLuy2QbRElic9HoJcHjqQmZ2FXAdUBNIV5V0JjDV3bekWfcy4DKANm3alELImVFXVxGpCmLvxeTuI929PXAjcHPyPDPrAvweuLyAdR9z9zx3z2vatGn2g40kurpq+FERqcyymSCWAq2TXreKphVkDHBa4oWZtQJeBX7q7vOzEmEJdeoEGzbozq4iUrllM0FMBjqYWTszqwkMAsYmL2BmHZJe9gfmRtP3Bt4AbnL3D7IYY4nk5oa/qmYSkcosawnC3bcDQ4HxwBfAi+4+y8xuM7MB0WJDzWyWmU0jtENckJgOHAgMi7rATjOzfbMVa3Edeii88gp06xZ3JCIi2WNeSSrS8/LyfMqUKXGHISJSoZjZp+6el25e7I3UFdWnn8Lrr8cdhYhI9ihBlND998PQoXFHISKSPUoQJaSb9olIZacEUUKJnkxz58Ybh4hItihBlJC6uopIZZdRgjCzumZWLXre0cwGmFlOdkMr3zp0CCPMafAgEamsMr0X0yTgaDNrBPyTcBHcucB52QqsvKtTJ/RkOvDAuCMREcmOTKuYzN03AmcAj7j72UCX7IVVMfToAfXrxx2FiEh2ZJwgzOxIQonhjWha9eyEVHFMngzDhummfSJSOWWaIH4G/Ap4NbpdxgHAhOyFVTFMmQK33w5LC7sFoYhIBZVRG4S7vwu8CxA1Vq9092uyGVhFkOjJNGcOtGoVbywiIqUt015Mo82sgZnVBT4DPjezG7IbWvnXqVP4q66uIlIZZVrF1DkaHvQ04E2gHXB+1qKqIJo3h3r11NVVRCqnTBNETnTdw2nAWHffRjR+dFVmFqqZFi2KOxIRkdKX6XUQjwKLgOnAJDPbH1iXraAqknfeUVdXEamcMm2kHgGMSJr0lZkdn52QKpYGDeKOQEQkOzJtpG5oZveb2ZTocR9QN8uxVQizZ8MFF6gdQkQqn0zbIJ4E1gPnRI91wKhsBVWRbN0KzzwD//lP3JGIiJSuTBNEe3e/xd0XRI9bgQOyGVhFkZsLDRvCq6/GHYmISOnKNEFsMrM+iRdm1hvYlJ2QKpZateDyy+Hll2HBgrijEREpPZkmiCuAkWa2yMwWAQ8Dl2ctqgrm2muhenV48MG4IxERKT2Z9mKaDnQzswbR63Vm9jNgRjaDqyhatIDrroOWLeOORESk9GR6HQQQEkPSy+sA/WaO3HVX3BGIiJSuPRly1Eotikpi2zYYPRo2b447EhGRPbcnCaLK32oj1QcfwHnnwbPPxh2JiMieKzRBmNl6M1uX5rEeaFFGMVYYxx4bRpm77z7YuTPuaERE9kyhCcLd67t7gzSP+u5erPaLqsAMbrghXFX9+utxRyMismf2pIpJ0jj7bGjTBu65J+5IRET2jBJEKatRA37+c1izJjxERCoqJYgsuOoqmDkTGjWKOxIRkZJTgsiCnJzQHrFuHaxcGXc0IiIlk9UEYWYnm9kcM5tnZjelmX+Fmc00s2lm9r6ZdU6a96tovTlmdlI248yGzZuhQwe45Za4IxERKZmsJQgzqw6MBPoBnYHByQkgMtrdu7p7d+Bu4P5o3c7AIKALcDLwSLS9CqN2bTj1VBg1SqUIEamYslmCOAyYF90efCswBhiYvEDKrTvqsvviu4HAGHff4u4LgXnR9iqUX/wCNm2CRx6JOxIRkeLLZoJoCSxOer0kmpaPmV1lZvMJJYhrirnuZYlR7lasWFFqgZeWzp2hf394+OGQKEREKpLYG6ndfaS7twduBG4u5rqPuXueu+c1bdo0OwHuoRtugBUrYPz4uCMRESmebF4NvRRonfS6VTStIGOAP5Zw3XLrmGNCl9eDD447EhGR4slmCWIy0MHM2plZTUKj89jkBcysQ9LL/sDc6PlYYJCZ1TKzdkAH4JMsxpo1ZruTw44d8cYiIlIcWStBuPt2MxsKjAeqA0+6+ywzuw2Y4u5jgaFm1hfYBqwBLojWnWVmLwKfA9uBq9y9Qp9eb7453O11woS4IxERyUxWb7jn7uOAcSnThiU9v7aQde8A7shedGWrWTOYOBE+/BCOOiruaEREihZ7I3VVceGF4dYb994bdyQiIplRgigj9erB//t/8Npr8OWXcUcjIlI0JYgydPXV4T5N998fdyQiIkXToD9lqFkzePppOPzwuCMRESmaEkQZGzQo7ghERDKjKqYYTJsWEsXGjXFHIiJSMCWIGKxfDy+8AE89FXckIiIFU4KIQZ8+oR3i/vt1dbWIlF9KEDEwg+uvh/nzQ7dXEZHySAkiJqefDgccAPfcA+5FLy8iUtbUiykm1avD8OHw3/+GaqYa+iZEpJzRaSlG558fdwQiIgVTFVPMtm+HF1+EuXOLXlZEpCwpQcRs9Wq46CIYPBg2bIg7GhGR3RpAlSQAABMwSURBVJQgYrbvvuGaiGnT4OyzYdu2uCMSEQmUIMqB/v3hT3+Cf/wDLrtMvZpEpHxQgignLrkEhg0LV1d/8EHc0YiIKEGUK8OHhxHn+vSJOxIRESWIcsUMjjwyPJ8wAd54I954RKRqU4Ioh9zht7+Fc86BTz6JOxoRqaqUIMohM3jlFdhvv9CAPW9e3BGJSFWkBFFONWsGb74ZShMnnwzffht3RCJS1ShBlGMdO8Lrr8PXX4dusCIiZUn3YirnjjgitEN07hx3JCJS1agEUQEcfDBUqwYLF8Ktt+pCOhEpG0oQFciYMeFaidtvjzsSEakKVMVUgdx0E8yZA7fcAq1ahZv8iYhkixJEBWIGjz8Oy5aFezY1bw79+sUdlYhUVqpiqmBycuDll+GQQ+D3v1d7hIhkj0oQFVD9+uEaibp1Q6lCRCQbslqCMLOTzWyOmc0zs5vSzL/OzD43sxlm9raZ7Z80724zm2VmX5jZCDOdCpM1awb16oVBhq68UhfSiUjpy1qCMLPqwEigH9AZGGxmqb35/wPkufshwMvA3dG6RwG9gUOAg4FewLHZirUimz073CL80EPhvffijkZEKpNsliAOA+a5+wJ33wqMAQYmL+DuE9x9Y/TyI6BVYhZQG6gJ1AJygOVZjLXCyssL40fUrg3HHRe6wO7YEXdUIlIZZDNBtAQWJ71eEk0ryMXAmwDu/m9gArAseox39y+yFGeF17MnTJ0axrUeNgyuuSbuiESkMigXjdRm9hMgj6gaycwOBA5id4niLTM72t3fS1nvMuAygDZt2pRdwOVQ/frw7LPQt+/uMSV27gxXYIuIlEQ2Tx9LgdZJr1tF0/Ixs77Ab4AB7r4lmnw68JG7b3D3DYSSxZGp67r7Y+6e5+55TZs2LfUPUNGYwZAhkJsbur+edx5cfz1s3Rp3ZCJSEWUzQUwGOphZOzOrCQwCxiYvYGY9gEcJySG5H85/gWPNrIaZ5RBKFqpiKoYdO6BxY7jvPujdG+bPjzsiEalospYg3H07MBQYTzi5v+jus8zsNjMbEC12D1APeMnMpplZIoG8DMwHZgLTgenu/vdsxVoZ1agBDz8cBh6aNw969IAXXog7KhGpSMwryaW4eXl5PmXKlLjDKJe++io0YH/5JcydC40axR2RiJQXZvapu+elm6cmzCpg//3h3Xdh0qSQHHbu1DCmIlI0JYgqIidn96BDI0ZA167w6KO6l5OIFEwJogoaPBiOOQauuALOPRfWro07IhEpj5QgqqBmzcLN/n7/e3j11dCA/fHHcUclIuWNEkQVVa0a/PKX4f5NNWrsvlZi/nz46CNVPYmIEkSVd8QR4YZ/ffqE1w89FK7EPugg+L//gyVL4o1PROKjBCFUr757XInf/Q6eeAKaNoVf/xratIGzz443PhGJhxKE5NOgAVx8cah6mjsXbr55d+8n9/D6ww9VBSVSFehCOcnYokXQpQts3AgdOsAFF8D554dShohUTLpQTkpF27bwzTcwahS0aBFKE23bwttvxx2ZiGSDEoQUS/364Y6xEyfCggVw661w1FFh3oMPwk9+AuPHw/btcUYpIqVBCUJKrF07+O1vYa+9wuvvvoM33oCTT4bWreEXv4Dp0+ONUURKTglCSs1vfhOqoF55JXSf/cMfQk+ohJUr44tNRIqvXIwoJ5VHrVpwxhnhsWoVrF4dpv/3v3DAAWHc7PPPD/Pr1481VBEpgkoQkjWNG4feTgC1a4cSxsKFoQ2jWbMw4t3ChbGGKCKFUDdXKVPu8O9/h/GzX3kFZs4MyWLkyNC4Xa9eeNStCw0bwvDhYb0PPoCvvw7TE/MbNNidgESkZArr5qoqJilTZqHX01FHhRHvqlcP09etC9VQ330HGzaEv9Wr704Qf/jD90fE228/WLYsPL/rLqhZM1RfaXhykdKhEoSUW+67bwGyfHlo5E4kjw0bwvQBA8Jy/fqFEkhOTph20UVw0km7E5CIpKcShFRIieQAoRqqWbOCl/vHP2DWLHjySXjmmVB9dcMNcPfdZROrSGWkRmqpNLp0gfvug6VL4eWX4cILw/QPP4QTToDnnoNNm+KNUaQiUYKQSqdmTTjzzHDLcghdbRctCld5t2gBV10FU6fGGqJIhaAEIZXeqafCvHnhnlGnnAJ//jOceOLuQZJ27IgvtoUL4aWXdMW5lE9qg5AqoVq1UM10wgmh99Rnn4WSxs6d0K1buGivXbtw88F27eCww8L00vC//8EXX+R/jBoFTZqE9pJET60jjwylm7POChccisRNvZikSvvuu3BX2hkzwq/5//43lCiuuy60Z2zaBLm5uxNHIon07p3/Ggz30NMqkQBOPTXcBv2550LVVkLNmtCxI7z4YqgCW7wYVqyASZPgkUfCGBwHHxziSW6kF8kW9WISKUDduvDAA7tfb98eGrlzcsLrjRvh+OND8njnnTDPHe6/H37+8zCG94knwpo1sHbt7u3ss09IEIcdFoZu7dw5JIR27cIY4AmtW4dHz55wzTWhGmzlypAcduyASy+Fc86BH/4wlIJEypJKECLFsGVL+NXfsGG4IG/ePBg2LFzVnUgCBx0ELVvueQlgzhw49thQMmnfHq68MtympHHjUvkoIkDhJQglCJFybOtW+Otfw61I3n8/3NPqo49Kr31ERFVMIhVUzZowaFB4zJgBzz8f2igAHn00NGafe+7uMTmS7dwZ2lCSrz7fe+9Q9bV1a2gHSczbe+/QNtKlS6geEwGVIEQqrOOPDyP7NWoUqrU2bID+/eHOO0M7Sc2a3x/Z75pr4KGHYPPm9Enl5pvh9ttDe8rQoaEhvmPH8OjQIVSlVQbLl8OECaFdaeLE0L7zxz+GfZp8i5eqQCUIkUronXfg3Xfh8cfh229h331D2weEE9wtt4QkkbgDbr164UQPoeTx5ZdhWp06YeyOL78MPbQgDPz0wQcwenQ4YSaMGhXaQRYvDqWZRPLYd99Q/VWnTvlsTF+zJpSmWrQIXZy7dg3TGzQI7Tw7d4bPAOFz3X57GLsk8SjoNi+VnUoQIlKgzZtDT60vvwyPAQNCaeVvf4PTTvv+8hMnhhPuiy/C5ZeHpFGr1u6/L7wAnTqFoWn/9KcwvXbtUK3VqlW4yWLjxqH3WE7O7t5kxbVhA7z3Xkii77wD//lPaOQfOTL0DnvggRBnjx75e5VBuOnjiBFh/fXrw7SDDgq3qW/YMJTKUtcpjvXrw12Ikx+HHAI/+EGo+nvssbD9GjXC569RI7Q5HXxw+D4mTMg/LycH8vJKfmPK2EoQZnYy8BBQHXjC3e9KmX8dcAmwHVgBXOTuX0Xz2gBPAK0BB05x90XZjFdE8qtdO7RLdOmSf/rAgeFX+dy5IXGsWRNOXu3bh/nt2oVbr2/eHHp+Jf7Wrh3mf/dd6DK8ZUv4Zb9yZThx/vjHYf4DD4TxzvfbL3QDbtUq/L3zzlBKWbo0nOhbtAgnyU2bwu1UErdX6do1vK5ZM1yAOHx4qH6DcCK9/vqCP/NJJ4XH9u3hliwTJ4YbQTZsGOaff35oDzr++FC6OPbY/LeYf+utMHZJcgLo3RuuvRa2bUtfTffLX4YE8d13cPXV359/++0hQXz7bbgbQKpNm7Jz5+KslSDMrDrwJXAisASYDAx298+Tljke+NjdN5rZlcBx7n5uNG8icIe7v2Vm9YCd7r6xoPdTCUKkYlu3LlR5VasWemy99RYsWRKqsxYvDtVeK1eGE+Hll4df2tWqhSSyalW4Mn3x4lC99tJLoVRy1FHp21r2xKOPwmuvhRLGd9+FaWedFd4TQjzLl4fnDRqE14MGwa23hmkjRoR2o+bNdz8aNAiJbufOcO+w7dtDMtm+PTwaNQqfb8uWUBpKTE8sc9JJJa/ai6Wbq5kdCQx395Oi178CcPf/K2D5HsDD7t7bzDoDj7l7n0zfTwlCpOqYMiWcKBcvDkmkUaNwG5V+/cquDWTbtt0ljAYNQhUWhLgSiaFu3bKJZU/EVcXUElic9HoJcHghy18MvBk97wisNbO/Au2AfwE3uXu+26qZ2WXAZQBt2rQppbBFpLzLywuPOOXkwOGHh0eyHj3iiScbykV/AzP7CZAH3BNNqgEcDVwP9AIOAIakrufuj7l7nrvnNdU4kyIipSqbCWIpoYE5oVU0LR8z6wv8Bhjg7luiyUuAae6+wN23A68BPbMYq4iIpMhmgpgMdDCzdmZWExgEjE1eIGp3eJSQHL5NWXdvM0sUC04APkdERMpM1hJE9Mt/KDAe+AJ40d1nmdltZjYgWuweoB7wkplNM7Ox0bo7CNVLb5vZTMCAx7MVq4iIfJ8ulBMRqcIK68VULhqpRUSk/FGCEBGRtJQgREQkrUrTBmFmK4Cv4o6jEE2AlXEHUQjFt2cU355RfHtmT+Lb393TXkhWaRJEeWdmUwpqCCoPFN+eUXx7RvHtmWzFpyomERFJSwlCRETSUoIoO4/FHUARFN+eUXx7RvHtmazEpzYIERFJSyUIERFJSwlCRETSUoIoJWbW2swmmNnnZjbLzK5Ns8xxZva/6MaE08xsWAxxLjKzmdH7f+/mVRaMMLN5ZjbDzMrsNutmlpu0b6aZ2Toz+1nKMmW6D83sSTP71sw+S5q2j5m9ZWZzo7+NClj3gmiZuWZ2QRnGd4+ZzY6+v1fNbO8C1i30WMhifMPNbGnSd5hmlOUwpr2ZzYmOxZvKML4XkmJbZGbTCli3LPZf2vNKmR2D7q5HKTyA5kDP6Hl9wnjcnVOWOQ54PeY4FwFNCpl/CmFkPwOOIIwZHkec1YFvCBfxxLYPgWMIY5F8ljTtbsIIhwA3Ab9Ps94+wILob6PoeaMyiu+HQI3o+e/TxZfJsZDF+IYD12fw/c8nDBZWE5ie+v+UrfhS5t8HDItx/6U9r5TVMagSRClx92XuPjV6vp5wi/OW8UZVIgOBZzz4iDAuR/MY4vgBMN/dY7063t0nAatTJg8Eno6ePw2clmbVk4C33H21u68B3gJOLov43P2fHm63D/ARYbCuWBSw/zJxGDDPw6BhW4ExhP1eqgqLz8wMOAd4vrTfN1OFnFfK5BhUgsgCM2sL9AA+TjP7SDObbmZvmlmXMg0scOCfZvZpNKZ3qnRjiceR6AZR8D9m3Puwmbsvi55/AzRLs0x52Y8XsXus91RFHQvZNDSqAnuygOqR8rD/jgaWu/vcAuaX6f5LOa+UyTGoBFHKzKwe8ArwM3dflzJ7KqHKpBvwB8JQqmWtj7v3BPoBV5nZMTHEUCgLIxAOAF5KM7s87MNdPJTly2VfcTP7DbAdeK6AReI6Fv4ItAe6A8sI1Tjl0WAKLz2U2f4r7LySzWNQCaIUmVkO4Ut8zt3/mjrf3de5+4bo+Tggx8yalGWM7r40+vst8CqhKJ8so7HEs6wfMNXdl6fOKA/7EFieqHaL/n6bZplY96OZDQFOBc6LTiDfk8GxkBXuvtzdd7j7TsJIkeneN+79VwM4A3ihoGXKav8VcF4pk2NQCaKURPWVfwa+cPf7C1hmv2g5zOwwwv5fVYYx1jWz+onnhMbMz1IWGwv8NOrNdATwv6SibFkp8Jdb3PswMhZI9Ai5APhbmmXGAz80s0ZRFcoPo2lZZ2YnA78kjPW+sYBlMjkWshVfcpvW6QW8b5Fj2mdZX2C2uy9JN7Os9l8h55WyOQaz2QJflR5AH0IxbwYwLXqcAlwBXBEtMxSYReiR8RFwVBnHeED03tOjOH4TTU+O0YCRhB4kM4G8Mo6xLuGE3zBpWmz7kJColgHbCHW4FwONgbeBucC/gH2iZfOAJ5LWvQiYFz0uLMP45hHqnhPH4Z+iZVsA4wo7FsoovmejY2sG4UTXPDW+6PUphF4788syvmj6U4ljLmnZOPZfQeeVMjkGdasNERFJS1VMIiKSlhKEiIikpQQhIiJpKUGIiEhaShAiIpKWEoRIEcxsh+W/y2yp3VnUzNom30lUpDypEXcAIhXAJnfvHncQImVNJQiREorGA7g7GhPgEzM7MJre1szeiW5G97aZtYmmN7MwPsP06HFUtKnqZvZ4dL//f5rZXtHy10TjAMwwszExfUypwpQgRIq2V0oV07lJ8/7n7l2Bh4EHo2l/AJ5290MIN8obEU0fAbzr4UaDPQlX4AJ0AEa6exdgLXBmNP0moEe0nSuy9eFECqIrqUWKYGYb3L1emumLgBPcfUF0Q7Vv3L2xma0k3D5iWzR9mbs3MbMVQCt335K0jbaEe/Z3iF7fCOS4++/M7B/ABsIda1/z6CaFImVFJQiRPeMFPC+OLUnPd7C7bbA/4b5YPYHJ0R1GRcqMEoTInjk36e+/o+cfEu4+CnAe8F70/G3gSgAzq25mDQvaqJlVA1q7+wTgRqAh8L1SjEg26ReJSNH2svwD1//D3RNdXRuZ2QxCKWBwNO1qYJSZ3QCsAC6Mpl8LPGZmFxNKClcS7iSaTnXgL1ESMWCEu68ttU8kkgG1QYiUUNQGkefuK+OORSQbVMUkIiJpqQQhIiJpqQQhIiJpKUGIiEhaShAiIpKWEoSIiKSlBCEiImn9f8Rw64lgQKngAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "val_loss = history_small_model.history[\"val_loss\"]\n",
        "epochs = range(1, 21)\n",
        "plt.plot(epochs, val_loss, \"b--\",\n",
        "         label=\"Validation loss\")\n",
        "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
        "plt.xlabel(\"Epochs\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:06:26.281161Z",
          "iopub.status.busy": "2022-05-21T01:06:26.280863Z",
          "iopub.status.idle": "2022-05-21T01:06:48.235589Z",
          "shell.execute_reply": "2022-05-21T01:06:48.234644Z"
        },
        "id": "TYz3NAseTk3X",
        "outputId": "0e274062-e4dc-41d8-fc90-cb3b9083812d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "375/375 [==============================] - 2s 4ms/step - loss: 0.3696 - accuracy: 0.8978 - val_loss: 0.2014 - val_accuracy: 0.9406\n",
            "Epoch 2/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.1610 - accuracy: 0.9518 - val_loss: 0.1392 - val_accuracy: 0.9598\n",
            "Epoch 3/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.1113 - accuracy: 0.9665 - val_loss: 0.1105 - val_accuracy: 0.9683\n",
            "Epoch 4/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0863 - accuracy: 0.9740 - val_loss: 0.0984 - val_accuracy: 0.9702\n",
            "Epoch 5/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0679 - accuracy: 0.9789 - val_loss: 0.0969 - val_accuracy: 0.9709\n",
            "Epoch 6/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0562 - accuracy: 0.9827 - val_loss: 0.0968 - val_accuracy: 0.9732\n",
            "Epoch 7/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0465 - accuracy: 0.9860 - val_loss: 0.0923 - val_accuracy: 0.9727\n",
            "Epoch 8/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0397 - accuracy: 0.9874 - val_loss: 0.0930 - val_accuracy: 0.9734\n",
            "Epoch 9/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0327 - accuracy: 0.9899 - val_loss: 0.0981 - val_accuracy: 0.9734\n",
            "Epoch 10/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0279 - accuracy: 0.9918 - val_loss: 0.1013 - val_accuracy: 0.9740\n",
            "Epoch 11/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0232 - accuracy: 0.9933 - val_loss: 0.1002 - val_accuracy: 0.9762\n",
            "Epoch 12/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0202 - accuracy: 0.9935 - val_loss: 0.1096 - val_accuracy: 0.9732\n",
            "Epoch 13/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0169 - accuracy: 0.9945 - val_loss: 0.1043 - val_accuracy: 0.9747\n",
            "Epoch 14/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0143 - accuracy: 0.9959 - val_loss: 0.1211 - val_accuracy: 0.9737\n",
            "Epoch 15/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0126 - accuracy: 0.9959 - val_loss: 0.1186 - val_accuracy: 0.9743\n",
            "Epoch 16/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0101 - accuracy: 0.9969 - val_loss: 0.1180 - val_accuracy: 0.9772\n",
            "Epoch 17/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0093 - accuracy: 0.9974 - val_loss: 0.1481 - val_accuracy: 0.9736\n",
            "Epoch 18/20\n",
            "375/375 [==============================] - 1s 4ms/step - loss: 0.0072 - accuracy: 0.9977 - val_loss: 0.1356 - val_accuracy: 0.9764\n",
            "Epoch 19/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0064 - accuracy: 0.9981 - val_loss: 0.1418 - val_accuracy: 0.9753\n",
            "Epoch 20/20\n",
            "375/375 [==============================] - 1s 3ms/step - loss: 0.0062 - accuracy: 0.9979 - val_loss: 0.1474 - val_accuracy: 0.9760\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(96, activation=\"relu\"),\n",
        "    layers.Dense(96, activation=\"relu\"),\n",
        "    layers.Dense(10, activation=\"softmax\"),\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"sparse_categorical_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_large_model = model.fit(\n",
        "    train_images, train_labels,\n",
        "    epochs=20,\n",
        "    batch_size=128,\n",
        "    validation_split=0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xsjm0jXuTk3Y"
      },
      "source": [
        "## 일반화 성능 향상하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SCOzA1LKTk3Y"
      },
      "source": [
        "### 데이터셋 큐레이션"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cDNJQp2yTk3Y"
      },
      "source": [
        "### 특성 공학"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BA30Ys7CTk3Y"
      },
      "source": [
        "### 조기 종료 사용하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FS0RwwIpTk3Z"
      },
      "source": [
        "### 모델 규제하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g1lVn3NcTk3Z"
      },
      "source": [
        "#### 네트워크 크기 축소"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3r8_V04Tk3Z"
      },
      "source": [
        "**원본 모델**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:06:48.240327Z",
          "iopub.status.busy": "2022-05-21T01:06:48.239635Z",
          "iopub.status.idle": "2022-05-21T01:07:01.599335Z",
          "shell.execute_reply": "2022-05-21T01:07:01.598428Z"
        },
        "id": "yZ__XxRqTk3Z",
        "outputId": "42070f91-b5a8-4806-c5a3-b6fb087605bc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17464789/17464789 [==============================] - 0s 0us/step\n",
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 41ms/step - loss: 0.5118 - accuracy: 0.7942 - val_loss: 0.3966 - val_accuracy: 0.8645\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.3088 - accuracy: 0.9041 - val_loss: 0.3057 - val_accuracy: 0.8865\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2241 - accuracy: 0.9275 - val_loss: 0.2775 - val_accuracy: 0.8937\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.1776 - accuracy: 0.9411 - val_loss: 0.2884 - val_accuracy: 0.8850\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1439 - accuracy: 0.9547 - val_loss: 0.2798 - val_accuracy: 0.8879\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.1197 - accuracy: 0.9633 - val_loss: 0.2933 - val_accuracy: 0.8857\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1027 - accuracy: 0.9682 - val_loss: 0.3094 - val_accuracy: 0.8853\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0843 - accuracy: 0.9758 - val_loss: 0.3293 - val_accuracy: 0.8847\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0710 - accuracy: 0.9796 - val_loss: 0.3508 - val_accuracy: 0.8812\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.0615 - accuracy: 0.9833 - val_loss: 0.3765 - val_accuracy: 0.8793\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0520 - accuracy: 0.9863 - val_loss: 0.3998 - val_accuracy: 0.8776\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0408 - accuracy: 0.9904 - val_loss: 0.4280 - val_accuracy: 0.8760\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0333 - accuracy: 0.9925 - val_loss: 0.4632 - val_accuracy: 0.8739\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0281 - accuracy: 0.9938 - val_loss: 0.4990 - val_accuracy: 0.8722\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.0225 - accuracy: 0.9953 - val_loss: 0.5676 - val_accuracy: 0.8653\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0190 - accuracy: 0.9959 - val_loss: 0.6024 - val_accuracy: 0.8654\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0139 - accuracy: 0.9981 - val_loss: 0.6486 - val_accuracy: 0.8631\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0118 - accuracy: 0.9985 - val_loss: 0.6424 - val_accuracy: 0.8643\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0120 - accuracy: 0.9976 - val_loss: 0.6860 - val_accuracy: 0.8648\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.0057 - accuracy: 0.9995 - val_loss: 0.7044 - val_accuracy: 0.8640\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.datasets import imdb\n",
        "(train_data, train_labels), _ = imdb.load_data(num_words=10000)\n",
        "\n",
        "def vectorize_sequences(sequences, dimension=10000):\n",
        "    results = np.zeros((len(sequences), dimension))\n",
        "    for i, sequence in enumerate(sequences):\n",
        "        results[i, sequence] = 1.\n",
        "    return results\n",
        "train_data = vectorize_sequences(train_data)\n",
        "\n",
        "model = keras.Sequential([\n",
        "    layers.Dense(16, activation=\"relu\"),\n",
        "    layers.Dense(16, activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_original = model.fit(train_data, train_labels,\n",
        "                             epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ur2zKeT6Tk3a"
      },
      "source": [
        "**작은 용량의 모델**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:07:01.603485Z",
          "iopub.status.busy": "2022-05-21T01:07:01.603168Z",
          "iopub.status.idle": "2022-05-21T01:07:09.910749Z",
          "shell.execute_reply": "2022-05-21T01:07:09.909893Z"
        },
        "id": "J07S8kRUTk3a",
        "outputId": "c0a63b10-4be7-4862-b259-50a1ba0abdd4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 36ms/step - loss: 0.5793 - accuracy: 0.7719 - val_loss: 0.4977 - val_accuracy: 0.8150\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 24ms/step - loss: 0.4241 - accuracy: 0.8781 - val_loss: 0.4017 - val_accuracy: 0.8668\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.3374 - accuracy: 0.9001 - val_loss: 0.3441 - val_accuracy: 0.8817\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2784 - accuracy: 0.9167 - val_loss: 0.3107 - val_accuracy: 0.8882\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2377 - accuracy: 0.9285 - val_loss: 0.2951 - val_accuracy: 0.8892\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.2075 - accuracy: 0.9373 - val_loss: 0.2814 - val_accuracy: 0.8924\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 23ms/step - loss: 0.1825 - accuracy: 0.9467 - val_loss: 0.2786 - val_accuracy: 0.8889\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1631 - accuracy: 0.9521 - val_loss: 0.2756 - val_accuracy: 0.8891\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1456 - accuracy: 0.9582 - val_loss: 0.2774 - val_accuracy: 0.8886\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1303 - accuracy: 0.9637 - val_loss: 0.2830 - val_accuracy: 0.8887\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1174 - accuracy: 0.9675 - val_loss: 0.2885 - val_accuracy: 0.8861\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 25ms/step - loss: 0.1062 - accuracy: 0.9708 - val_loss: 0.2932 - val_accuracy: 0.8859\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 23ms/step - loss: 0.0961 - accuracy: 0.9741 - val_loss: 0.3011 - val_accuracy: 0.8853\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0866 - accuracy: 0.9771 - val_loss: 0.3206 - val_accuracy: 0.8807\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0783 - accuracy: 0.9801 - val_loss: 0.3235 - val_accuracy: 0.8825\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0710 - accuracy: 0.9825 - val_loss: 0.3366 - val_accuracy: 0.8816\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0635 - accuracy: 0.9853 - val_loss: 0.3488 - val_accuracy: 0.8801\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0577 - accuracy: 0.9868 - val_loss: 0.3650 - val_accuracy: 0.8785\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.0515 - accuracy: 0.9883 - val_loss: 0.3816 - val_accuracy: 0.8788\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.0461 - accuracy: 0.9895 - val_loss: 0.3973 - val_accuracy: 0.8761\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(4, activation=\"relu\"),\n",
        "    layers.Dense(4, activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_smaller_model = model.fit(\n",
        "    train_data, train_labels,\n",
        "    epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k6040fZeTk3a"
      },
      "source": [
        "**큰 용량의 모델**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:07:09.914619Z",
          "iopub.status.busy": "2022-05-21T01:07:09.914070Z",
          "iopub.status.idle": "2022-05-21T01:07:55.440909Z",
          "shell.execute_reply": "2022-05-21T01:07:55.439855Z"
        },
        "id": "gRyaH_CkTk3b",
        "outputId": "926ff59e-c18a-439c-957d-c7f1b60584ca",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 42ms/step - loss: 0.5673 - accuracy: 0.7789 - val_loss: 0.3003 - val_accuracy: 0.8852\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 27ms/step - loss: 0.2481 - accuracy: 0.9052 - val_loss: 0.3609 - val_accuracy: 0.8541\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 27ms/step - loss: 0.1528 - accuracy: 0.9397 - val_loss: 0.3480 - val_accuracy: 0.8729\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 0.1118 - accuracy: 0.9672 - val_loss: 0.3435 - val_accuracy: 0.8910\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 0.0219 - accuracy: 0.9935 - val_loss: 1.6897 - val_accuracy: 0.6846\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 0.0350 - accuracy: 0.9893 - val_loss: 0.5067 - val_accuracy: 0.8879\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 8.4210e-04 - accuracy: 0.9999 - val_loss: 0.6132 - val_accuracy: 0.8886\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 9.0304e-04 - accuracy: 0.9999 - val_loss: 0.7161 - val_accuracy: 0.8876\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 28ms/step - loss: 2.9179e-05 - accuracy: 1.0000 - val_loss: 0.8425 - val_accuracy: 0.8869\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 5.3758e-06 - accuracy: 1.0000 - val_loss: 0.9504 - val_accuracy: 0.8880\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 1.2510e-06 - accuracy: 1.0000 - val_loss: 1.0460 - val_accuracy: 0.8871\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 3.7285e-07 - accuracy: 1.0000 - val_loss: 1.1332 - val_accuracy: 0.8859\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 1.2854e-07 - accuracy: 1.0000 - val_loss: 1.1943 - val_accuracy: 0.8863\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 5.4076e-08 - accuracy: 1.0000 - val_loss: 1.2503 - val_accuracy: 0.8865\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 25ms/step - loss: 2.9129e-08 - accuracy: 1.0000 - val_loss: 1.2856 - val_accuracy: 0.8864\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 1.9333e-08 - accuracy: 1.0000 - val_loss: 1.3063 - val_accuracy: 0.8870\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 1.4389e-08 - accuracy: 1.0000 - val_loss: 1.3258 - val_accuracy: 0.8868\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 25ms/step - loss: 1.1431e-08 - accuracy: 1.0000 - val_loss: 1.3392 - val_accuracy: 0.8870\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 9.5314e-09 - accuracy: 1.0000 - val_loss: 1.3522 - val_accuracy: 0.8869\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 28ms/step - loss: 8.1679e-09 - accuracy: 1.0000 - val_loss: 1.3632 - val_accuracy: 0.8864\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(512, activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_larger_model = model.fit(\n",
        "    train_data, train_labels,\n",
        "    epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HkMwsNWiTk3b"
      },
      "source": [
        "#### 가중치 규제 추가하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "slmc7NwkTk3b"
      },
      "source": [
        "**모델에 L2 가중치 추가하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:07:55.445676Z",
          "iopub.status.busy": "2022-05-21T01:07:55.444826Z",
          "iopub.status.idle": "2022-05-21T01:08:04.225182Z",
          "shell.execute_reply": "2022-05-21T01:08:04.224432Z"
        },
        "id": "BbKqHmXtTk3b",
        "outputId": "89b3595e-57f9-4e11-f373-b7d9ae79f9ba",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 39ms/step - loss: 0.6008 - accuracy: 0.7869 - val_loss: 0.5094 - val_accuracy: 0.8202\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.4126 - accuracy: 0.8913 - val_loss: 0.4147 - val_accuracy: 0.8704\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.3432 - accuracy: 0.9120 - val_loss: 0.3858 - val_accuracy: 0.8798\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.3126 - accuracy: 0.9197 - val_loss: 0.3618 - val_accuracy: 0.8865\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2898 - accuracy: 0.9306 - val_loss: 0.3603 - val_accuracy: 0.8866\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2761 - accuracy: 0.9340 - val_loss: 0.4112 - val_accuracy: 0.8648\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2647 - accuracy: 0.9390 - val_loss: 0.3612 - val_accuracy: 0.8852\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2570 - accuracy: 0.9409 - val_loss: 0.3638 - val_accuracy: 0.8843\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2502 - accuracy: 0.9425 - val_loss: 0.3681 - val_accuracy: 0.8832\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2431 - accuracy: 0.9456 - val_loss: 0.3712 - val_accuracy: 0.8828\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2415 - accuracy: 0.9454 - val_loss: 0.3795 - val_accuracy: 0.8803\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2322 - accuracy: 0.9491 - val_loss: 0.3876 - val_accuracy: 0.8774\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.2328 - accuracy: 0.9482 - val_loss: 0.4040 - val_accuracy: 0.8753\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.2279 - accuracy: 0.9502 - val_loss: 0.3943 - val_accuracy: 0.8767\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2261 - accuracy: 0.9488 - val_loss: 0.3987 - val_accuracy: 0.8774\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2204 - accuracy: 0.9521 - val_loss: 0.3924 - val_accuracy: 0.8781\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.2176 - accuracy: 0.9525 - val_loss: 0.3966 - val_accuracy: 0.8770\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.2153 - accuracy: 0.9532 - val_loss: 0.4018 - val_accuracy: 0.8781\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 24ms/step - loss: 0.2141 - accuracy: 0.9545 - val_loss: 0.4348 - val_accuracy: 0.8689\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.2179 - accuracy: 0.9533 - val_loss: 0.4065 - val_accuracy: 0.8767\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras import regularizers\n",
        "model = keras.Sequential([\n",
        "    layers.Dense(16,\n",
        "                 kernel_regularizer=regularizers.l2(0.002),\n",
        "                 activation=\"relu\"),\n",
        "    layers.Dense(16,\n",
        "                 kernel_regularizer=regularizers.l2(0.002),\n",
        "                 activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_l2_reg = model.fit(\n",
        "    train_data, train_labels,\n",
        "    epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "766AbsD9Tk3c"
      },
      "source": [
        "**케라스에서 사용할 수 있는 가중치 규제**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:08:04.229728Z",
          "iopub.status.busy": "2022-05-21T01:08:04.228866Z",
          "iopub.status.idle": "2022-05-21T01:08:04.237099Z",
          "shell.execute_reply": "2022-05-21T01:08:04.235990Z"
        },
        "id": "tT6wJynITk3c",
        "outputId": "65db72bb-52b7-4506-ef29-2c744b4ab897",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.regularizers.L1L2 at 0x7f70020eee10>"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "from tensorflow.keras import regularizers\n",
        "regularizers.l1(0.001)\n",
        "regularizers.l1_l2(l1=0.001, l2=0.001)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ultTwtPKTk3c"
      },
      "source": [
        "#### 드롭아웃 추가"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MWW1i75kTk3c"
      },
      "source": [
        "**IMDB 모델에 드롭아웃 추가하기**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2022-05-21T01:08:04.241161Z",
          "iopub.status.busy": "2022-05-21T01:08:04.240762Z",
          "iopub.status.idle": "2022-05-21T01:08:13.113623Z",
          "shell.execute_reply": "2022-05-21T01:08:13.112554Z"
        },
        "id": "K6C938cRTk3d",
        "outputId": "cab61d67-bb61-424e-f606-f01b4aec4aec",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "30/30 [==============================] - 2s 37ms/step - loss: 0.6394 - accuracy: 0.6262 - val_loss: 0.5249 - val_accuracy: 0.8433\n",
            "Epoch 2/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.5247 - accuracy: 0.7531 - val_loss: 0.4134 - val_accuracy: 0.8722\n",
            "Epoch 3/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.4367 - accuracy: 0.8188 - val_loss: 0.3441 - val_accuracy: 0.8766\n",
            "Epoch 4/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.3715 - accuracy: 0.8570 - val_loss: 0.3002 - val_accuracy: 0.8856\n",
            "Epoch 5/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.3236 - accuracy: 0.8821 - val_loss: 0.2817 - val_accuracy: 0.8929\n",
            "Epoch 6/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2814 - accuracy: 0.8973 - val_loss: 0.2889 - val_accuracy: 0.8803\n",
            "Epoch 7/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.2499 - accuracy: 0.9121 - val_loss: 0.2747 - val_accuracy: 0.8940\n",
            "Epoch 8/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 0.2197 - accuracy: 0.9281 - val_loss: 0.2865 - val_accuracy: 0.8903\n",
            "Epoch 9/20\n",
            "30/30 [==============================] - 1s 29ms/step - loss: 0.1983 - accuracy: 0.9359 - val_loss: 0.3085 - val_accuracy: 0.8901\n",
            "Epoch 10/20\n",
            "30/30 [==============================] - 1s 30ms/step - loss: 0.1809 - accuracy: 0.9456 - val_loss: 0.3147 - val_accuracy: 0.8894\n",
            "Epoch 11/20\n",
            "30/30 [==============================] - 1s 26ms/step - loss: 0.1619 - accuracy: 0.9508 - val_loss: 0.3341 - val_accuracy: 0.8890\n",
            "Epoch 12/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1424 - accuracy: 0.9531 - val_loss: 0.3532 - val_accuracy: 0.8870\n",
            "Epoch 13/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1310 - accuracy: 0.9593 - val_loss: 0.3662 - val_accuracy: 0.8887\n",
            "Epoch 14/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1216 - accuracy: 0.9631 - val_loss: 0.3979 - val_accuracy: 0.8886\n",
            "Epoch 15/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1117 - accuracy: 0.9663 - val_loss: 0.4015 - val_accuracy: 0.8880\n",
            "Epoch 16/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.1022 - accuracy: 0.9685 - val_loss: 0.4369 - val_accuracy: 0.8886\n",
            "Epoch 17/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0999 - accuracy: 0.9681 - val_loss: 0.4553 - val_accuracy: 0.8875\n",
            "Epoch 18/20\n",
            "30/30 [==============================] - 1s 22ms/step - loss: 0.0901 - accuracy: 0.9707 - val_loss: 0.4918 - val_accuracy: 0.8888\n",
            "Epoch 19/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.0849 - accuracy: 0.9719 - val_loss: 0.5229 - val_accuracy: 0.8871\n",
            "Epoch 20/20\n",
            "30/30 [==============================] - 1s 21ms/step - loss: 0.0871 - accuracy: 0.9707 - val_loss: 0.5527 - val_accuracy: 0.8845\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    layers.Dense(16, activation=\"relu\"),\n",
        "    layers.Dropout(0.5),\n",
        "    layers.Dense(16, activation=\"relu\"),\n",
        "    layers.Dropout(0.5),\n",
        "    layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "model.compile(optimizer=\"rmsprop\",\n",
        "              loss=\"binary_crossentropy\",\n",
        "              metrics=[\"accuracy\"])\n",
        "history_dropout = model.fit(\n",
        "    train_data, train_labels,\n",
        "    epochs=20, batch_size=512, validation_split=0.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JuocOmx8Tk3d"
      },
      "source": [
        "## 요약"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "chapter05_fundamentals-of-ml.i",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "default:Python",
      "language": "python",
      "name": "conda-env-default-py"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.10"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}