chapter11_part03_transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5PUseM_N42in"
   },
   "source": [
    "이 노트북은 [케라스 창시자에게 배우는 딥러닝 2판](https://tensorflow.blog/kerasdl2/)의 예제 코드를 담고 있습니다.\n",
    "\n",
    "<table align=\"left\">\n",
    "    <tr>\n",
    "        <td>\n",
    "            <a href=\"https://colab.research.google.com/github/rickiepark/deep-learning-with-python-2nd/blob/main/chapter11_part03_transformer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NhOmioaR42iq"
   },
   "source": [
    "## 트랜스포머 아키텍처"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xWSmzK5142ir"
   },
   "source": [
    "### 셀프 어텐션 이해하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2XdzrCS-42ir"
   },
   "source": [
    "#### 일반화된 셀프 어텐션: 쿼리-키-값 모델"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qo8cK-c_42ir"
   },
   "source": [
    "### 멀티 헤드 어텐션"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F0HqiH6C42ir"
   },
   "source": [
    "### 트랜스포머 인코더"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LjYMcwAl42is"
   },
   "source": [
    "**Getting the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "i7VVn1yl42is",
    "outputId": "c45bb997-c586-4664-dbac-b2d16c589ab8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: cannot remove 'aclImdb': No such file or directory\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 80.2M  100 80.2M    0     0  56.3M      0  0:00:01  0:00:01 --:--:-- 56.3M\n"
     ]
    }
   ],
   "source": [
    "!rm -r aclImdb\n",
    "!curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
    "!tar -xf aclImdb_v1.tar.gz\n",
    "!rm -r aclImdb/train/unsup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D8aiOAjJ42iu"
   },
   "source": [
    "**Preparing the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7C8nBL2242iu",
    "outputId": "2003846d-67d1-4c86-d31d-6ca1941deb02"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 20000 files belonging to 2 classes.\n",
      "Found 5000 files belonging to 2 classes.\n",
      "Found 25000 files belonging to 2 classes.\n"
     ]
    }
   ],
   "source": [
    "import os, pathlib, shutil, random\n",
    "from tensorflow import keras\n",
    "batch_size = 32\n",
    "base_dir = pathlib.Path(\"aclImdb\")\n",
    "val_dir = base_dir / \"val\"\n",
    "train_dir = base_dir / \"train\"\n",
    "for category in (\"neg\", \"pos\"):\n",
    "    os.makedirs(val_dir / category)\n",
    "    files = os.listdir(train_dir / category)\n",
    "    random.Random(1337).shuffle(files)\n",
    "    num_val_samples = int(0.2 * len(files))\n",
    "    val_files = files[-num_val_samples:]\n",
    "    for fname in val_files:\n",
    "        shutil.move(train_dir / category / fname,\n",
    "                    val_dir / category / fname)\n",
    "\n",
    "train_ds = keras.utils.text_dataset_from_directory(\n",
    "    \"aclImdb/train\", batch_size=batch_size\n",
    ")\n",
    "val_ds = keras.utils.text_dataset_from_directory(\n",
    "    \"aclImdb/val\", batch_size=batch_size\n",
    ")\n",
    "test_ds = keras.utils.text_dataset_from_directory(\n",
    "    \"aclImdb/test\", batch_size=batch_size\n",
    ")\n",
    "text_only_train_ds = train_ds.map(lambda x, y: x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e0MsIvhr42iv"
   },
   "source": [
    "**Vectorizing the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "--rJ5PYw42iv"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import layers\n",
    "\n",
    "max_length = 600\n",
    "max_tokens = 20000\n",
    "text_vectorization = layers.TextVectorization(\n",
    "    max_tokens=max_tokens,\n",
    "    output_mode=\"int\",\n",
    "    output_sequence_length=max_length,\n",
    ")\n",
    "text_vectorization.adapt(text_only_train_ds)\n",
    "\n",
    "int_train_ds = train_ds.map(\n",
    "    lambda x, y: (text_vectorization(x), y),\n",
    "    num_parallel_calls=4)\n",
    "int_val_ds = val_ds.map(\n",
    "    lambda x, y: (text_vectorization(x), y),\n",
    "    num_parallel_calls=4)\n",
    "int_test_ds = test_ds.map(\n",
    "    lambda x, y: (text_vectorization(x), y),\n",
    "    num_parallel_calls=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uZrJUai142iv"
   },
   "source": [
    "**`Layer` 층을 상속하여 구현한 트랜스포머 인코더**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "924jztFe42iv"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "class TransformerEncoder(layers.Layer):\n",
    "    def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.embed_dim = embed_dim\n",
    "        self.dense_dim = dense_dim\n",
    "        self.num_heads = num_heads\n",
    "        self.attention = layers.MultiHeadAttention(\n",
    "            num_heads=num_heads, key_dim=embed_dim)\n",
    "        self.dense_proj = keras.Sequential(\n",
    "            [layers.Dense(dense_dim, activation=\"relu\"),\n",
    "             layers.Dense(embed_dim),]\n",
    "        )\n",
    "        self.layernorm_1 = layers.LayerNormalization()\n",
    "        self.layernorm_2 = layers.LayerNormalization()\n",
    "\n",
    "    def call(self, inputs, mask=None):\n",
    "        if mask is not None:\n",
    "            mask = mask[:, tf.newaxis, :]\n",
    "        attention_output = self.attention(\n",
    "            inputs, inputs, attention_mask=mask)\n",
    "        proj_input = self.layernorm_1(inputs + attention_output)\n",
    "        proj_output = self.dense_proj(proj_input)\n",
    "        return self.layernorm_2(proj_input + proj_output)\n",
    "\n",
    "    def get_config(self):\n",
    "        config = super().get_config()\n",
    "        config.update({\n",
    "            \"embed_dim\": self.embed_dim,\n",
    "            \"num_heads\": self.num_heads,\n",
    "            \"dense_dim\": self.dense_dim,\n",
    "        })\n",
    "        return config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8yaSXFTO42iw"
   },
   "source": [
    "**트랜스포머 인코더를 사용하여 텍스트 분류하기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-Nk9lx8A42iw",
    "outputId": "4cb0e25b-b812-4abe-81e0-3accfd33ecdc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, None)]            0         \n",
      "                                                                 \n",
      " embedding (Embedding)       (None, None, 256)         5120000   \n",
      "                                                                 \n",
      " transformer_encoder (Transf  (None, None, 256)        543776    \n",
      " ormerEncoder)                                                   \n",
      "                                                                 \n",
      " global_max_pooling1d (Globa  (None, 256)              0         \n",
      " lMaxPooling1D)                                                  \n",
      "                                                                 \n",
      " dropout (Dropout)           (None, 256)               0         \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 1)                 257       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 5,664,033\n",
      "Trainable params: 5,664,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "vocab_size = 20000\n",
    "embed_dim = 256\n",
    "num_heads = 2\n",
    "dense_dim = 32\n",
    "\n",
    "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
    "x = layers.Embedding(vocab_size, embed_dim)(inputs)\n",
    "x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
    "x = layers.GlobalMaxPooling1D()(x)\n",
    "x = layers.Dropout(0.5)(x)\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "model = keras.Model(inputs, outputs)\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7nXp84h_42iw"
   },
   "source": [
    "**트랜스포머 인코더 기반 모델 훈련하고 평가하기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "E3H26gQC42ix",
    "outputId": "90c88a70-1365-4ecc-c13b-59b05a085808"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "625/625 [==============================] - 48s 66ms/step - loss: 0.4855 - accuracy: 0.7738 - val_loss: 0.3135 - val_accuracy: 0.8680\n",
      "Epoch 2/20\n",
      "625/625 [==============================] - 42s 67ms/step - loss: 0.3048 - accuracy: 0.8709 - val_loss: 0.2706 - val_accuracy: 0.8930\n",
      "Epoch 3/20\n",
      "625/625 [==============================] - 43s 68ms/step - loss: 0.2277 - accuracy: 0.9084 - val_loss: 0.2634 - val_accuracy: 0.8948\n",
      "Epoch 4/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.1791 - accuracy: 0.9312 - val_loss: 0.2659 - val_accuracy: 0.8926\n",
      "Epoch 5/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.1477 - accuracy: 0.9435 - val_loss: 0.2681 - val_accuracy: 0.8942\n",
      "Epoch 6/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.1288 - accuracy: 0.9512 - val_loss: 0.2794 - val_accuracy: 0.8916\n",
      "Epoch 7/20\n",
      "625/625 [==============================] - 44s 70ms/step - loss: 0.1064 - accuracy: 0.9596 - val_loss: 0.3904 - val_accuracy: 0.8830\n",
      "Epoch 8/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0865 - accuracy: 0.9672 - val_loss: 0.5325 - val_accuracy: 0.8700\n",
      "Epoch 9/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0709 - accuracy: 0.9736 - val_loss: 0.4413 - val_accuracy: 0.8848\n",
      "Epoch 10/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0558 - accuracy: 0.9795 - val_loss: 0.4035 - val_accuracy: 0.8670\n",
      "Epoch 11/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0422 - accuracy: 0.9850 - val_loss: 0.5172 - val_accuracy: 0.8662\n",
      "Epoch 12/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0359 - accuracy: 0.9885 - val_loss: 0.6248 - val_accuracy: 0.8366\n",
      "Epoch 13/20\n",
      "625/625 [==============================] - 44s 70ms/step - loss: 0.0264 - accuracy: 0.9908 - val_loss: 0.6386 - val_accuracy: 0.8624\n",
      "Epoch 14/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0231 - accuracy: 0.9927 - val_loss: 0.6820 - val_accuracy: 0.8662\n",
      "Epoch 15/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0204 - accuracy: 0.9931 - val_loss: 0.8993 - val_accuracy: 0.8546\n",
      "Epoch 16/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0181 - accuracy: 0.9945 - val_loss: 0.7890 - val_accuracy: 0.8634\n",
      "Epoch 17/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0187 - accuracy: 0.9953 - val_loss: 0.9705 - val_accuracy: 0.8466\n",
      "Epoch 18/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0159 - accuracy: 0.9956 - val_loss: 0.9134 - val_accuracy: 0.8686\n",
      "Epoch 19/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0158 - accuracy: 0.9956 - val_loss: 1.0816 - val_accuracy: 0.8534\n",
      "Epoch 20/20\n",
      "625/625 [==============================] - 43s 69ms/step - loss: 0.0150 - accuracy: 0.9959 - val_loss: 1.2647 - val_accuracy: 0.8536\n",
      "782/782 [==============================] - 20s 25ms/step - loss: 0.2915 - accuracy: 0.8802\n",
      "테스트 정확도: 0.880\n"
     ]
    }
   ],
   "source": [
    "callbacks = [\n",
    "    keras.callbacks.ModelCheckpoint(\"transformer_encoder.keras\",\n",
    "                                    save_best_only=True)\n",
    "]\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)\n",
    "model = keras.models.load_model(\n",
    "    \"transformer_encoder.keras\",\n",
    "    custom_objects={\"TransformerEncoder\": TransformerEncoder})\n",
    "print(f\"테스트 정확도: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ffb0_42j42ix"
   },
   "source": [
    "#### 위치 인코딩을 사용해 위치 정보 주입하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c3eCKYxO42ix"
   },
   "source": [
    "**서브클래싱으로 위치 임베딩 구현하기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "7ngZXecb42ix"
   },
   "outputs": [],
   "source": [
    "class PositionalEmbedding(layers.Layer):\n",
    "    def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.token_embeddings = layers.Embedding(\n",
    "            input_dim=input_dim, output_dim=output_dim)\n",
    "        self.position_embeddings = layers.Embedding(\n",
    "            input_dim=sequence_length, output_dim=output_dim)\n",
    "        self.sequence_length = sequence_length\n",
    "        self.input_dim = input_dim\n",
    "        self.output_dim = output_dim\n",
    "\n",
    "    def call(self, inputs):\n",
    "        length = tf.shape(inputs)[-1]\n",
    "        positions = tf.range(start=0, limit=length, delta=1)\n",
    "        embedded_tokens = self.token_embeddings(inputs)\n",
    "        embedded_positions = self.position_embeddings(positions)\n",
    "        return embedded_tokens + embedded_positions\n",
    "\n",
    "    def compute_mask(self, inputs, mask=None):\n",
    "        return tf.math.not_equal(inputs, 0)\n",
    "\n",
    "    def get_config(self):\n",
    "        config = super().get_config()\n",
    "        config.update({\n",
    "            \"output_dim\": self.output_dim,\n",
    "            \"sequence_length\": self.sequence_length,\n",
    "            \"input_dim\": self.input_dim,\n",
    "        })\n",
    "        return config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YAOg6EBF42ix"
   },
   "source": [
    "#### 텍스트 분류 트랜스포머"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9RXcbNDa42iy"
   },
   "source": [
    "**트랜스포머 인코더와 위치 임베딩 합치기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QmzXmO7042iy",
    "outputId": "2350656d-0de3-4505-8866-80f44f3a695e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_2 (InputLayer)        [(None, None)]            0         \n",
      "                                                                 \n",
      " positional_embedding (Posit  (None, None, 256)        5273600   \n",
      " ionalEmbedding)                                                 \n",
      "                                                                 \n",
      " transformer_encoder_1 (Tran  (None, None, 256)        543776    \n",
      " sformerEncoder)                                                 \n",
      "                                                                 \n",
      " global_max_pooling1d_1 (Glo  (None, 256)              0         \n",
      " balMaxPooling1D)                                                \n",
      "                                                                 \n",
      " dropout_1 (Dropout)         (None, 256)               0         \n",
      "                                                                 \n",
      " dense_7 (Dense)             (None, 1)                 257       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 5,817,633\n",
      "Trainable params: 5,817,633\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "625/625 [==============================] - 46s 71ms/step - loss: 0.4821 - accuracy: 0.7800 - val_loss: 0.3220 - val_accuracy: 0.8610\n",
      "Epoch 2/20\n",
      "625/625 [==============================] - 44s 70ms/step - loss: 0.2353 - accuracy: 0.9071 - val_loss: 0.2663 - val_accuracy: 0.8934\n",
      "Epoch 3/20\n",
      "625/625 [==============================] - 45s 71ms/step - loss: 0.1764 - accuracy: 0.9341 - val_loss: 0.2732 - val_accuracy: 0.8904\n",
      "Epoch 4/20\n",
      "625/625 [==============================] - 44s 71ms/step - loss: 0.1461 - accuracy: 0.9474 - val_loss: 0.3096 - val_accuracy: 0.8866\n",
      "Epoch 5/20\n",
      "625/625 [==============================] - 44s 69ms/step - loss: 0.1248 - accuracy: 0.9548 - val_loss: 0.3062 - val_accuracy: 0.8882\n",
      "Epoch 6/20\n",
      "625/625 [==============================] - 44s 71ms/step - loss: 0.1098 - accuracy: 0.9609 - val_loss: 0.3035 - val_accuracy: 0.8850\n",
      "Epoch 7/20\n",
      "625/625 [==============================] - 44s 71ms/step - loss: 0.0947 - accuracy: 0.9677 - val_loss: 0.3925 - val_accuracy: 0.8792\n",
      "Epoch 8/20\n",
      "625/625 [==============================] - 44s 71ms/step - loss: 0.0817 - accuracy: 0.9726 - val_loss: 0.4082 - val_accuracy: 0.8786\n",
      "Epoch 9/20\n",
      "625/625 [==============================] - 44s 71ms/step - loss: 0.0736 - accuracy: 0.9745 - val_loss: 0.3837 - val_accuracy: 0.8810\n",
      "Epoch 10/20\n",
      "625/625 [==============================] - 44s 70ms/step - loss: 0.0640 - accuracy: 0.9793 - val_loss: 0.3908 - val_accuracy: 0.8762\n",
      "Epoch 11/20\n",
      "625/625 [==============================] - 44s 70ms/step - loss: 0.0548 - accuracy: 0.9814 - val_loss: 0.4751 - val_accuracy: 0.8774\n",
      "Epoch 12/20\n",
      "625/625 [==============================] - 45s 71ms/step - loss: 0.0484 - accuracy: 0.9833 - val_loss: 0.3969 - val_accuracy: 0.8768\n",
      "Epoch 13/20\n",
      "625/625 [==============================] - 44s 71ms/step - loss: 0.0410 - accuracy: 0.9865 - val_loss: 0.4613 - val_accuracy: 0.8750\n",
      "Epoch 14/20\n",
      "625/625 [==============================] - 45s 72ms/step - loss: 0.0359 - accuracy: 0.9883 - val_loss: 0.4811 - val_accuracy: 0.8682\n",
      "Epoch 15/20\n",
      "625/625 [==============================] - 44s 71ms/step - loss: 0.0309 - accuracy: 0.9899 - val_loss: 0.5404 - val_accuracy: 0.8664\n",
      "Epoch 16/20\n",
      "625/625 [==============================] - 44s 71ms/step - loss: 0.0289 - accuracy: 0.9902 - val_loss: 0.4783 - val_accuracy: 0.8710\n",
      "Epoch 17/20\n",
      "625/625 [==============================] - 44s 70ms/step - loss: 0.0229 - accuracy: 0.9934 - val_loss: 0.6532 - val_accuracy: 0.8606\n",
      "Epoch 18/20\n",
      "625/625 [==============================] - 45s 72ms/step - loss: 0.0193 - accuracy: 0.9936 - val_loss: 0.6286 - val_accuracy: 0.8738\n",
      "Epoch 19/20\n",
      "625/625 [==============================] - 44s 70ms/step - loss: 0.0210 - accuracy: 0.9938 - val_loss: 0.6121 - val_accuracy: 0.8676\n",
      "Epoch 20/20\n",
      "625/625 [==============================] - 44s 70ms/step - loss: 0.0157 - accuracy: 0.9955 - val_loss: 0.6550 - val_accuracy: 0.8760\n",
      "782/782 [==============================] - 22s 28ms/step - loss: 0.2935 - accuracy: 0.8796\n",
      "테스트 정확도: 0.880\n"
     ]
    }
   ],
   "source": [
    "vocab_size = 20000\n",
    "sequence_length = 600\n",
    "embed_dim = 256\n",
    "num_heads = 2\n",
    "dense_dim = 32\n",
    "\n",
    "inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
    "x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\n",
    "x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
    "x = layers.GlobalMaxPooling1D()(x)\n",
    "x = layers.Dropout(0.5)(x)\n",
    "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "model = keras.Model(inputs, outputs)\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "callbacks = [\n",
    "    keras.callbacks.ModelCheckpoint(\"full_transformer_encoder.keras\",\n",
    "                                    save_best_only=True)\n",
    "]\n",
    "model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)\n",
    "model = keras.models.load_model(\n",
    "    \"full_transformer_encoder.keras\",\n",
    "    custom_objects={\"TransformerEncoder\": TransformerEncoder,\n",
    "                    \"PositionalEmbedding\": PositionalEmbedding})\n",
    "print(f\"테스트 정확도: {model.evaluate(int_test_ds)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T17kNue242iy"
   },
   "source": [
    "### BoW 모델 대신 언제 시퀀스 모델을 사용하나요?"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "chapter11_part03_transformer.i",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "default:Python",
   "language": "python",
   "name": "conda-env-default-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}