chapter12_part03_neural-style-transfer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "이 노트북은 [케라스 창시자에게 배우는 딥러닝 2판](https://tensorflow.blog/kerasdl2/)의 예제 코드를 담고 있습니다.\n",
    "\n",
    "<table align=\"left\">\n",
    "    <tr>\n",
    "        <td>\n",
    "            <a href=\"https://colab.research.google.com/github/rickiepark/deep-learning-with-python-2nd/blob/main/chapter12_part03_neural-style-transfer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "## 뉴럴 스타일 트랜스퍼"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### 콘텐츠 손실"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### 스타일 손실"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### 케라스로 뉴럴 스타일 트랜스퍼 구현하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**스타일 이미지와 콘텐츠 이미지 준비하기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "base_image_path = keras.utils.get_file(\n",
    "    \"sf.jpg\", origin=\"https://img-datasets.s3.amazonaws.com/sf.jpg\")\n",
    "style_reference_image_path = keras.utils.get_file(\n",
    "    \"starry_night.jpg\", origin=\"https://img-datasets.s3.amazonaws.com/starry_night.jpg\")\n",
    "\n",
    "original_width, original_height = keras.utils.load_img(base_image_path).size\n",
    "img_height = 400\n",
    "img_width = round(original_width * img_height / original_height)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**유틸리티 함수**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def preprocess_image(image_path):\n",
    "    img = keras.utils.load_img(\n",
    "        image_path, target_size=(img_height, img_width))\n",
    "    img = keras.utils.img_to_array(img)\n",
    "    img = np.expand_dims(img, axis=0)\n",
    "    img = keras.applications.vgg19.preprocess_input(img)\n",
    "    return img\n",
    "\n",
    "def deprocess_image(img):\n",
    "    img = img.reshape((img_height, img_width, 3))\n",
    "    img[:, :, 0] += 103.939\n",
    "    img[:, :, 1] += 116.779\n",
    "    img[:, :, 2] += 123.68\n",
    "    img = img[:, :, ::-1]\n",
    "    img = np.clip(img, 0, 255).astype(\"uint8\")\n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**사전 훈련된 VGG19 모델을 사용해 특성 추출기 만들기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-06-02 01:43:54.544851: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "model = keras.applications.vgg19.VGG19(weights=\"imagenet\", include_top=False)\n",
    "\n",
    "outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])\n",
    "feature_extractor = keras.Model(inputs=model.inputs, outputs=outputs_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**콘텐츠 손실**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "def content_loss(base_img, combination_img):\n",
    "    return tf.reduce_sum(tf.square(combination_img - base_img))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**스타일 손실**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "def gram_matrix(x):\n",
    "    x = tf.transpose(x, (2, 0, 1))\n",
    "    features = tf.reshape(x, (tf.shape(x)[0], -1))\n",
    "    gram = tf.matmul(features, tf.transpose(features))\n",
    "    return gram\n",
    "\n",
    "def style_loss(style_img, combination_img):\n",
    "    S = gram_matrix(style_img)\n",
    "    C = gram_matrix(combination_img)\n",
    "    channels = 3\n",
    "    size = img_height * img_width\n",
    "    return tf.reduce_sum(tf.square(S - C)) / (4.0 * (channels ** 2) * (size ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**총 변위 손실**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "def total_variation_loss(x):\n",
    "    a = tf.square(\n",
    "        x[:, : img_height - 1, : img_width - 1, :] - x[:, 1:, : img_width - 1, :]\n",
    "    )\n",
    "    b = tf.square(\n",
    "        x[:, : img_height - 1, : img_width - 1, :] - x[:, : img_height - 1, 1:, :]\n",
    "    )\n",
    "    return tf.reduce_sum(tf.pow(a + b, 1.25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**최소화할 최종 손실 정의하기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [],
   "source": [
    "style_layer_names = [\n",
    "    \"block1_conv1\",\n",
    "    \"block2_conv1\",\n",
    "    \"block3_conv1\",\n",
    "    \"block4_conv1\",\n",
    "    \"block5_conv1\",\n",
    "]\n",
    "content_layer_name = \"block5_conv2\"\n",
    "total_variation_weight = 1e-6\n",
    "style_weight = 1e-6\n",
    "content_weight = 2.5e-8\n",
    "\n",
    "def compute_loss(combination_image, base_image, style_reference_image):\n",
    "    input_tensor = tf.concat(\n",
    "        [base_image, style_reference_image, combination_image], axis=0\n",
    "    )\n",
    "    features = feature_extractor(input_tensor)\n",
    "    loss = tf.zeros(shape=())\n",
    "    layer_features = features[content_layer_name]\n",
    "    base_image_features = layer_features[0, :, :, :]\n",
    "    combination_features = layer_features[2, :, :, :]\n",
    "    loss = loss + content_weight * content_loss(\n",
    "        base_image_features, combination_features\n",
    "    )\n",
    "    for layer_name in style_layer_names:\n",
    "        layer_features = features[layer_name]\n",
    "        style_reference_features = layer_features[1, :, :, :]\n",
    "        combination_features = layer_features[2, :, :, :]\n",
    "        style_loss_value = style_loss(\n",
    "          style_reference_features, combination_features)\n",
    "        loss += (style_weight / len(style_layer_names)) * style_loss_value\n",
    "\n",
    "    loss += total_variation_weight * total_variation_loss(combination_image)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "**경사 하강법 단계 설정하기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab_type": "code"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100번째 반복: loss=8137.27\n",
      "200번째 반복: loss=6652.48\n",
      "300번째 반복: loss=6063.02\n",
      "400번째 반복: loss=5742.32\n",
      "500번째 반복: loss=5535.75\n",
      "600번째 반복: loss=5389.79\n",
      "700번째 반복: loss=5280.41\n",
      "800번째 반복: loss=5194.97\n",
      "900번째 반복: loss=5125.85\n",
      "1000번째 반복: loss=5068.76\n",
      "1100번째 반복: loss=5020.89\n",
      "1200번째 반복: loss=4980.00\n",
      "1300번째 반복: loss=4944.65\n",
      "1400번째 반복: loss=4913.70\n",
      "1500번째 반복: loss=4886.41\n",
      "1600번째 반복: loss=4862.26\n",
      "1700번째 반복: loss=4840.58\n",
      "1800번째 반복: loss=4821.15\n",
      "1900번째 반복: loss=4803.64\n",
      "2000번째 반복: loss=4787.83\n",
      "2100번째 반복: loss=4773.46\n",
      "2800번째 반복: loss=4700.88\n",
      "2900번째 반복: loss=4693.30\n",
      "3000번째 반복: loss=4686.26\n",
      "3100번째 반복: loss=4679.70\n",
      "3200번째 반복: loss=4673.58\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "@tf.function\n",
    "def compute_loss_and_grads(combination_image, base_image, style_reference_image):\n",
    "    with tf.GradientTape() as tape:\n",
    "        loss = compute_loss(combination_image, base_image, style_reference_image)\n",
    "    grads = tape.gradient(loss, combination_image)\n",
    "    return loss, grads\n",
    "\n",
    "optimizer = keras.optimizers.SGD(\n",
    "    keras.optimizers.schedules.ExponentialDecay(\n",
    "        initial_learning_rate=100.0, decay_steps=100, decay_rate=0.96\n",
    "    )\n",
    ")\n",
    "\n",
    "base_image = preprocess_image(base_image_path)\n",
    "style_reference_image = preprocess_image(style_reference_image_path)\n",
    "combination_image = tf.Variable(preprocess_image(base_image_path))\n",
    "\n",
    "iterations = 4000\n",
    "for i in range(1, iterations + 1):\n",
    "    loss, grads = compute_loss_and_grads(\n",
    "        combination_image, base_image, style_reference_image\n",
    "    )\n",
    "    optimizer.apply_gradients([(grads, combination_image)])\n",
    "    if i % 100 == 0:\n",
    "        print(f\"{i}번째 반복: loss={loss:.2f}\")\n",
    "        img = deprocess_image(combination_image.numpy())\n",
    "        fname = f\"combination_image_at_iteration_{i}.png\"\n",
    "        keras.utils.save_img(fname, img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![combination_image_at_iteration_4000.png](combination_image_at_iteration_4000.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text"
   },
   "source": [
    "### 정리"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "chapter12_part03_neural-style-transfer.i",
   "private_outputs": false,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "default:Python",
   "language": "python",
   "name": "conda-env-default-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}