fastai_NLP/LanguageModeling.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "# 多行输出\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Language Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH=32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据\n",
    "- 先在小数据集或者重采样数据集上进行模型的测试是一个策略"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/lab/.fastai/data/imdb_sample')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/lab/.fastai/data/imdb_sample/texts.csv'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb_sample/data_lm')]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB_SAMPLE)\n",
    "path\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>negative</td>\n",
       "      <td>Un-bleeping-believable! Meg Ryan doesn't even ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>This is a extremely well-made film. The acting...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>negative</td>\n",
       "      <td>Every once in a long while a movie will come a...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>positive</td>\n",
       "      <td>Name just says it all. I watched this movie wi...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>negative</td>\n",
       "      <td>This movie succeeds at being one of the most u...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      label                                               text  is_valid\n",
       "0  negative  Un-bleeping-believable! Meg Ryan doesn't even ...     False\n",
       "1  positive  This is a extremely well-made film. The acting...     False\n",
       "2  negative  Every once in a long while a movie will come a...     False\n",
       "3  positive  Name just says it all. I watched this movie wi...     False\n",
       "4  negative  This movie succeeds at being one of the most u...     False"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv(path/r'texts.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_lm = TextDataBunch.from_csv(path, 'texts.csv')  # 随机分割训练集和测试集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面这一步可能是耗时的。图像可以(几乎)直接输入到模型中，因为它们只是一个很大的像素值数组，这些像素值在0到1之间浮动。文本是由单词组成的，我们不能将数学函数直接应用于它们。我们首先要把它们转换成数字。\n",
    "\n",
    "这需要两个不同的步骤:标记化和数字化（向量化）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 先保存 避免每次都计算"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_lm.save('data_lm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, 'data_lm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj raising xxmaj victor xxmaj vargas : a xxmaj review \\n \\n  xxmaj you know , xxmaj raising xxmaj victor xxmaj vargas is like sticking your hands into a big , xxunk bowl of xxunk . xxmaj it 's warm and gooey , but you 're not sure if it feels right . xxmaj try as i might , no matter how warm and gooey xxmaj raising xxmaj</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj now that xxmaj che(2008 ) has finished its relatively short xxmaj australian cinema run ( extremely limited xxunk screen in xxmaj sydney , after xxunk ) , i can xxunk join both xxunk of \" xxmaj at xxmaj the xxmaj movies \" in taking xxmaj steven xxmaj soderbergh to task . \\n \\n  xxmaj it 's usually satisfying to watch a film director change his style /</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj this film sat on my xxmaj tivo for weeks before i watched it . i xxunk a self - indulgent xxunk flick about relationships gone bad . i was wrong ; this was an xxunk xxunk into the screwed - up xxunk of xxmaj new xxmaj yorkers . \\n \\n  xxmaj the format is the same as xxmaj max xxmaj xxunk ' \" xxmaj la xxmaj ronde</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj many neglect that this is n't just a classic due to the fact that it 's the first xxup 3d game , or even the first xxunk - up . xxmaj it 's also one of the first stealth games , one of the xxunk definitely the first ) truly claustrophobic games , and just a pretty well - rounded gaming experience in general . xxmaj with graphics</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos i really wanted to love this show . i truly , honestly did . \\n \\n  xxmaj for the first time , gay viewers get their own version of the \" xxmaj the xxmaj bachelor \" . xxmaj with the help of his obligatory \" hag \" xxmaj xxunk , xxmaj james , a good looking , well - to - do thirty - something has the chance</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一旦我们从文本中提取了标记，我们就通过创建使用的所有单词的列表将其转换为整数。我们**只保留两次**出现在列表中的单词，**最大词汇量为60,000**((默认情况下)，*并替换那些没有被未知标记扣掉的单词*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxeos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " 'the']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.vocab.itos[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.vocab.stoi['the']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = data_lm.train_ds[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text xxbos xxmaj having xxmaj just \" xxmaj welcomed xxmaj home \" my xxunk xxup xxunk old daughter from a year in xxmaj iraq , xxmaj camp xxmaj xxunk medical support unit , i felt compelled to get this xxup dvd . i wanted to hear other returning vets feelings in order to attempt to better understand her mentality on arrival and not waiting until after something bad happened . xxmaj regardless on your take on the war and peace this movie serves as a great start for all xxmaj americans to begin the healing of our returning vets emotional void . xxmaj the paramount statement of the entire movie is \" xxmaj take xxmaj action \" on the problem . xxmaj incredibly emotional movie . i would highly recommend this movie to the vet the vets entire mature family and ask that they follow through with a plan to listen comfort help the returning xxmaj gulf xxmaj war xxmaj xxunk xxmaj freedom vets . \n",
       " \n",
       "  xxmaj fast forward nearly one year later & xxmaj my daughter has seen this xxup dvd . xxmaj took account of her emotions and actually has made a commitment to re - up for another 6 years . xxmaj her take on her time spent in the sand is that she did some good . xxmaj local xxmaj xxunk children got first rate medical treatment for various common xxunk not ordinarily able to afford free with an xxunk and xxunk . xxmaj her look over her shoulder at her xxmaj iraq tour was . \" xxmaj we changed some hearts and minds back there \" xxmaj great xxup dvd you have to keep an open mind and see all sides"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x # 原始的文本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   2,    5,  293,    5,   56,   22,    5, 6105,    5,  441])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.data[:10] # 用于计算的数字 其单词的坐标"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos xxmaj having xxmaj just \" xxmaj welcomed xxmaj home'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'xxbos xxmaj having xxmaj just \" xxmaj welcomed xxmaj home'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(data_lm.vocab.itos[i] for i in x.data[:10].tolist())\n",
    "' '.join(str(x).split(' ')[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- data block API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = (TextList.from_csv(path, 'texts.csv', cols='text')\n",
    "                .split_from_df(col=2)\n",
    "                .label_from_df(cols=0)\n",
    "                .databunch(bs=BATCH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用全部的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/lab/.fastai/data/imdb')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/lab/.fastai/data/imdb/tmp_lm'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/test'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/tmp_clas'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/unsup'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/train'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/imdb.vocab'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/models'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/imdb_textlist_class'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/data_lm'),\n",
       " PosixPath('/home/lab/.fastai/data/imdb/README')]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = untar_data(URLs.IMDB)\n",
    "path\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- unsup 是没有标签的数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用预训练的语言模型\n",
    "\n",
    "- 在[The WikiText Long Term Dependency Language Modeling Dataset](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/)数据上训练\n",
    "- 预测下一个单词是什么（多分类），预测下一个词的分布"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建语言模型数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH = 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 这里使用所有的数据-训练集和测试集 训练我们的语言模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_lm = (TextList.from_folder(path)\n",
    "#            #Inputs: all the text files in path\n",
    "#             .filter_by_folder(include=['train', 'test']) \n",
    "#            #We may have other temp folders that contain text files so we only keep what's in train and test\n",
    "#             .split_by_rand_pct(0.1)\n",
    "#            #We randomly split and keep 10% (10,000 reviews) for validation\n",
    "#             .label_for_lm()           \n",
    "#            #We want to do a language model so we label accordingly, 自己就是自己的标签\n",
    "#             .databunch(bs=BATCH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 保存数据集，避免重复性工作（标记化和数值化）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_lm.save('data_lm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, 'data_lm', bs=BATCH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 字典词数\n",
    "- 很多单词映射为unk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(49852, 49852)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_lm.vocab.itos), len(data_lm.vocab.stoi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>to do movies such as pretty in pink and sixteen xxunk is fun watching her a little xxunk not seen by many people .. i do n't even think the average movie fan has ever heard of this xxunk when i first came xxunk it i was glad i xxunk gave this movie 10 out of 10 stars as i think it is a fun movie the whole family can</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>my interpretation was that this film was taking the mickey out all the silly prejudices and innuendo of small town gossip and national tabloid sensationalism . i loved that the film was championing the cause that a person 's sexuality is xxup not determined by their hobbies , idiosyncrasies , fashion sense or whatever . xxmaj and then the ending goes and re - enforces all the gossip and stereotypes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>knock it up a notch to the excellent category but i did talk to some viewers who felt that it was an excellent film . xxmaj louhimies said that back in xxmaj finland people either loved this movie or hated it and said a lot of people in theaters walked out on it . i 'm sure some objected to some of the violence , swearing , drug and alcohol</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>( \" xxmaj sorte xxmaj nula \" ) , or so he believes with that music on the car radio , and the dubious talk by his best friend and company associate who is taking him to parts unknown through a desert road . xxmaj alberto wished simply to be left alone , to take a flight abroad next day , with ",
       "  well , someone we 're left</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>to connect this film to xxmaj antonioni , i think is a bit of a stretch , the character most likely to be seen in one of xxmaj michaelangelo 's movies is xxmaj francoise xxmaj moitessier de xxmaj xxunk . xxmaj it 's funny on the main imdb page , he is n't even listed as playing \" xxmaj himself \" which is probably a function of his lengthy name</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 构建模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4 id=\"RNNLearner.save_encoder\" class=\"doc_header\"><code>save_encoder</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/learner.py#L57\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#RNNLearner-save_encoder-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4><blockquote><p><code>save_encoder</code>(<strong><code>name</code></strong>:<code>str</code>)</p>\n",
       "</blockquote>\n",
       "<div class=\"collapse\" id=\"RNNLearner-save_encoder-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#RNNLearner-save_encoder-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>save_encoder</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div><p>Save the encoder to <code>name</code> inside the model directory.</p>\n",
       "<p><a href=\"https://docs.fast.ai/text.learner.html#RNNLearner.save_encoder\" target=\"_blank\" rel=\"noreferrer noopener\">Show in docs</a></p>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc(learn_lm.save_encoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- fastai 模型的第1层保存的是编码器部分，第二部分保存的是解码器部分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AWD_LSTM(\n",
       "  (encoder): Embedding(49852, 400, padding_idx=1)\n",
       "  (encoder_dp): EmbeddingDropout(\n",
       "    (emb): Embedding(49852, 400, padding_idx=1)\n",
       "  )\n",
       "  (rnns): ModuleList(\n",
       "    (0): WeightDropout(\n",
       "      (module): LSTM(400, 1150, batch_first=True)\n",
       "    )\n",
       "    (1): WeightDropout(\n",
       "      (module): LSTM(1150, 1150, batch_first=True)\n",
       "    )\n",
       "    (2): WeightDropout(\n",
       "      (module): LSTM(1150, 400, batch_first=True)\n",
       "    )\n",
       "  )\n",
       "  (input_dp): RNNDropout()\n",
       "  (hidden_dps): ModuleList(\n",
       "    (0): RNNDropout()\n",
       "    (1): RNNDropout()\n",
       "    (2): RNNDropout()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder = get_model(learn_lm.model)[0]\n",
    "encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- wiki 的字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "wiki_itos = pickle.load(open(Config().model_path()/'wt103-1/itos_wt103.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk', 'xxpad', 'xxmaj', 'the', ',', '.', 'of', 'and', 'in', 'to']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_itos[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AWD_LSTM(\n",
       "  (encoder): Embedding(49852, 400, padding_idx=1)\n",
       "  (encoder_dp): EmbeddingDropout(\n",
       "    (emb): Embedding(49852, 400, padding_idx=1)\n",
       "  )\n",
       "  (rnns): ModuleList(\n",
       "    (0): WeightDropout(\n",
       "      (module): LSTM(400, 1150, batch_first=True)\n",
       "    )\n",
       "    (1): WeightDropout(\n",
       "      (module): LSTM(1150, 1150, batch_first=True)\n",
       "    )\n",
       "    (2): WeightDropout(\n",
       "      (module): LSTM(1150, 400, batch_first=True)\n",
       "    )\n",
       "  )\n",
       "  (input_dp): RNNDropout()\n",
       "  (hidden_dps): ModuleList(\n",
       "    (0): RNNDropout()\n",
       "    (1): RNNDropout()\n",
       "    (2): RNNDropout()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "awd_enc = learn_lm.model[0]\n",
    "awd_enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearDecoder(\n",
       "  (decoder): Linear(in_features=400, out_features=49852, bias=True)\n",
       "  (output_dp): RNNDropout()\n",
       ")"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dec = learn_lm.model[1]\n",
    "dec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(49852, 400, padding_idx=1)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1519,  0.0584, -0.2037,  ..., -0.0608, -0.0194, -0.2690],\n",
       "        [ 0.2069, -0.0384,  0.1234,  ..., -0.0258, -0.0756, -0.1575],\n",
       "        [-0.9274,  0.0241,  0.1056,  ..., -0.1405, -0.1731,  0.0738],\n",
       "        ...,\n",
       "        [ 0.3072,  0.0193,  0.1959,  ...,  0.1347,  0.0935, -0.1177],\n",
       "        [ 0.2356, -0.0249,  0.1637,  ..., -0.0054, -0.0643, -0.1600],\n",
       "        [ 0.2356, -0.0249,  0.1637,  ..., -0.0054, -0.0643, -0.1600]],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([49852, 400])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc = awd_enc.encoder # 编码器的embedding层\n",
    "enc\n",
    "enc.weight.data\n",
    "enc.weight.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMDB 和 WIKI 的词汇是存在差别的，但是对于迁移学习这是没有任何问题的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60001, 49852)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(wiki_itos), len(data_lm.vocab.itos) # wiki 词汇， IMDB 词汇"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 找不50个IMDB中有的词，但是wiki中没有"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "i, unks = 0, []\n",
    "while len(unks) < 50:\n",
    "    if data_lm.vocab.itos[i] not in wiki_itos: unks.append((i, data_lm.vocab.itos[i]))\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(2865, '---'),\n",
       " (2890, 'fulci'),\n",
       " (2987, ':)'),\n",
       " (2990, 'stanwyck'),\n",
       " (3382, 'matthau'),\n",
       " (3402, 'dialogs'),\n",
       " (3493, \"don't\"),\n",
       " (3575, 'anyways'),\n",
       " (3655, 'mst3'),\n",
       " (3858, 'drivel'),\n",
       " (3883, 'porno'),\n",
       " (3905, 'stinker'),\n",
       " (3962, 'unwatchable')]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unks[12:25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"don't\""
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.vocab.itos[3493]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17445"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diff = set.difference(set(data_lm.vocab.itos), set(wiki_itos))\n",
    "len(diff)\n",
    "# diff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 只在 imdb 中出现而不在 wiki 中出现的词汇的权重会被随机初始化为相同的权重，我们会在训练中，学习到这些词的准确权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(False, True)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"30-something\" in wiki_itos, \"30-something\" in data_lm.vocab.itos\n",
    "\"linklater\" in wiki_itos, \"30-something\" in data_lm.vocab.itos\n",
    "# 30-something 在 IMDB中但是不在 wiki中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28229"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm.vocab.stoi[\"30-something\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- enc 是一个 embedding 层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(49852, 400, padding_idx=1)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 2.3563e-01, -2.4887e-02,  1.6367e-01, -1.5576e-01, -3.9991e-02,\n",
       "        -1.1837e+00,  1.3771e-02,  2.6963e+00, -2.4460e-01, -3.0679e-01,\n",
       "         4.9507e-02, -9.3249e-02, -1.2190e-01,  7.3875e-02,  8.2177e-02,\n",
       "         1.7709e-01, -4.6640e-02, -1.6259e-01,  1.3047e-01, -7.7981e-02,\n",
       "         1.9621e-01, -2.2501e-01, -8.7846e-02, -7.1112e-02,  2.4189e-01,\n",
       "        -9.3553e-02, -1.2042e-02,  1.6407e-01, -6.7238e-02, -3.6608e-01,\n",
       "        -4.9779e-02, -2.6294e-01, -5.8337e-02,  2.9274e-01,  3.0474e-02,\n",
       "        -4.3713e-01,  9.1709e-02, -7.4387e-02,  1.3249e-01, -2.5243e-02,\n",
       "         1.0777e-01,  5.1872e-02, -1.1154e-01,  3.9594e-02,  1.6266e-01,\n",
       "        -1.2775e-01,  3.4357e-02, -3.5956e-02, -3.2496e-01, -1.6485e-01,\n",
       "        -3.0641e-02, -7.6774e-02,  1.4561e-01,  1.1531e-01,  1.8984e+00,\n",
       "         1.5799e-01,  8.4391e-02, -1.3433e-01,  1.1566e-01, -1.2106e-01,\n",
       "         1.0962e+00,  2.9572e-01,  2.3983e-01,  1.1480e-01,  6.6237e-03,\n",
       "        -1.3348e-01,  6.8766e-02,  1.5964e-01, -8.7224e-02, -2.3924e-01,\n",
       "        -1.7510e-01,  2.0747e-01, -1.6984e-01, -2.3177e-03, -4.9442e-01,\n",
       "        -1.6792e-01,  1.3460e-01, -1.8338e-02, -2.6355e-01, -1.6453e-01,\n",
       "         4.1224e-02, -1.8444e-01, -3.0363e-02, -6.3805e-02, -4.0107e-02,\n",
       "        -2.5481e-01,  3.1321e-02,  1.9193e+00, -2.1612e-01, -6.2593e-02,\n",
       "        -1.5077e-03,  5.2033e-03,  8.3699e-02,  4.0290e-02,  2.8803e-01,\n",
       "        -2.0666e-01,  1.1335e-01, -9.5299e-02,  2.2939e-01,  4.0757e-01,\n",
       "        -3.3981e-01, -9.0904e-02, -1.7730e-01, -5.3028e-02, -1.9618e-01,\n",
       "        -1.2492e-02, -1.0680e-01, -1.5958e-01, -5.0954e-02,  8.5235e-02,\n",
       "        -1.1794e-01,  3.0984e-01,  9.6872e-02,  3.1385e-01, -4.9641e-01,\n",
       "         5.1671e-03, -1.9114e-01,  1.5572e-01, -7.9885e-02,  1.3268e-01,\n",
       "         9.0588e-01,  6.7491e-02,  2.2338e-01, -1.0066e-01,  4.6575e-01,\n",
       "        -1.2312e-01,  9.3612e-02,  1.3096e-01, -3.3108e-01, -1.7744e-01,\n",
       "         1.1523e+00, -1.1713e+00, -3.3524e-02,  1.5911e-01, -1.0232e-01,\n",
       "        -1.4164e-01, -3.1028e-01, -2.4289e-01,  4.4483e-02, -7.8837e-04,\n",
       "        -2.5294e-01, -1.4119e-01,  7.1791e-03, -7.7787e-02,  6.7305e-02,\n",
       "         1.1571e-01, -9.7474e-02, -1.0530e+00, -1.5756e-01,  1.9618e-01,\n",
       "        -1.4641e-01, -2.1020e-01, -6.0207e-02,  2.0658e+00, -5.8721e-02,\n",
       "         1.3172e+00, -1.1151e-01, -6.5037e-02,  2.6665e-01, -1.5142e-01,\n",
       "        -1.2816e-02,  2.9121e-01,  1.5877e-01, -2.1064e-01, -2.3437e-01,\n",
       "         1.2074e-03,  1.3788e-01,  8.4911e-02,  9.8629e-03,  1.5704e-01,\n",
       "         1.1210e-01, -3.4607e-01,  2.2828e-02,  1.6462e-01, -5.3240e-02,\n",
       "         7.4044e-03,  3.4998e-02,  3.2578e-02, -1.2031e-01, -7.6079e-03,\n",
       "        -6.2595e-02,  1.6175e-01, -1.2981e-01,  2.4086e-01,  6.1456e-01,\n",
       "        -1.1655e-01,  1.2812e-01,  7.0912e-03,  6.9810e-02,  1.2587e-02,\n",
       "        -3.8313e-01,  3.9995e-01,  4.6975e-02, -3.8577e-02,  4.7613e-03,\n",
       "        -8.6680e-02,  8.2375e-02,  1.8488e-01,  6.3544e-02,  9.2193e+00,\n",
       "        -1.6143e-01,  9.3521e-02,  1.4227e-01, -1.1530e-02, -1.0175e-02,\n",
       "         1.7004e-02,  9.6651e-02,  7.3341e-01, -8.3059e-02, -8.0967e-02,\n",
       "        -2.4441e-01, -2.1112e-01,  1.3093e-01, -2.7853e-02, -3.1662e-02,\n",
       "         3.0409e-02,  2.4478e-02,  1.3748e-01,  5.7066e-01,  1.0895e-01,\n",
       "         1.0938e-02, -8.3650e-01, -1.2638e-01,  1.2183e-01, -1.4355e-01,\n",
       "        -4.6536e-02, -2.8628e-01, -1.8231e-01, -1.2930e-01,  1.6428e-01,\n",
       "         1.3582e-02, -1.5254e-01,  2.9710e-01,  3.5783e-02,  2.2965e-02,\n",
       "        -1.3139e-01,  6.8671e-02, -7.0601e-02,  4.3105e-03, -7.3840e-02,\n",
       "        -1.4461e-01, -1.2314e-01,  8.2990e-02,  1.5063e-01,  4.9478e-02,\n",
       "        -1.0543e-01, -1.4868e-01, -6.2989e-02, -2.4564e-01,  1.2984e-01,\n",
       "        -9.8305e-02,  9.6116e-03,  1.8233e-01,  1.1220e-01,  1.3147e-01,\n",
       "        -1.0651e-01, -1.9505e-03,  7.7677e-03, -1.5431e+00,  5.4617e-02,\n",
       "         1.4792e-01,  9.1974e-02, -5.9575e-02,  9.0482e-02, -4.0548e-02,\n",
       "         2.7410e-02,  1.6957e-01,  1.7686e-01, -2.4234e-01,  5.2125e-03,\n",
       "        -4.4122e-02,  4.4026e-02, -7.9261e-02, -1.2375e-01,  3.3830e-01,\n",
       "        -1.0975e-01, -4.1963e-02,  6.1868e-02,  3.8249e-02, -3.4647e-01,\n",
       "         6.9477e-02, -2.7612e-02, -9.5724e-02,  3.5617e-02,  3.0908e-01,\n",
       "         1.5625e-02, -1.5706e-02, -1.8647e-02, -1.3613e+00,  3.9733e-02,\n",
       "        -1.4360e-02, -5.6631e-01,  1.4364e-02, -2.0767e-01,  1.0025e-01,\n",
       "         5.8167e-01, -1.2165e-01, -9.4708e-02,  8.2568e-02,  1.6126e-01,\n",
       "         9.0681e-04,  2.1125e-01, -1.5409e-01, -1.5994e-01, -1.4560e-01,\n",
       "         8.7726e-02,  2.9405e-01,  2.7593e-02,  1.7728e-01, -4.0951e-02,\n",
       "         7.2310e-02, -1.1495e-01,  3.9496e-02, -1.3501e-01, -5.7911e-02,\n",
       "        -1.2620e-01,  2.1860e-01,  5.4778e-02,  5.2384e-02,  1.2734e-01,\n",
       "        -3.0588e-02, -2.3094e-01, -1.8980e-01, -2.1621e-02, -1.8698e-01,\n",
       "         5.8853e-01, -4.9644e-02, -1.7897e-01,  7.8744e-02,  1.8341e-01,\n",
       "        -1.0343e-01, -3.8085e-02, -1.4170e+00, -1.2232e-01,  6.6738e-02,\n",
       "        -1.5288e-01,  3.8547e-02,  1.4078e+00,  9.0353e-01, -6.9630e-02,\n",
       "         4.7199e-02,  1.0277e-01, -1.1472e-01, -3.2788e-02,  1.1692e-01,\n",
       "         8.2559e-02, -7.4094e-02, -1.7668e-01, -1.9571e-01, -1.2744e-01,\n",
       "         4.8258e-02, -6.5957e-02, -1.0730e-01, -1.7549e-01,  5.2153e-03,\n",
       "        -4.1777e-02, -2.4510e-02,  5.3595e-02, -5.0036e-02, -3.6234e-01,\n",
       "        -1.4430e-01,  1.5534e-01,  1.4770e-01,  2.0897e-03,  2.1379e-01,\n",
       "         1.8835e-02, -5.7863e-02,  9.5830e-02,  2.3745e-02, -1.2344e-01,\n",
       "        -1.3102e-01,  8.9221e-02,  4.6883e-02,  5.0474e-02, -4.1123e-02,\n",
       "         2.2810e+00,  5.2746e-01,  4.3884e-02,  1.1691e+00, -4.7088e-01,\n",
       "        -1.0998e-01,  5.3738e-02,  5.3674e-01, -1.0251e-01, -4.4597e-02,\n",
       "         1.1504e-01,  4.9237e-02,  9.1344e-02, -1.7354e-01,  9.9623e-02,\n",
       "        -3.6215e-02,  3.1805e-02,  8.1981e-02,  4.9603e-02, -9.5985e-01,\n",
       "        -1.3482e-01,  1.3472e-01, -5.4147e-03, -6.4292e-02, -1.5996e-01],\n",
       "       device='cuda:0', grad_fn=<SelectBackward>)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "tensor([ 2.3563e-01, -2.4887e-02,  1.6367e-01, -1.5576e-01, -3.9991e-02,\n",
       "        -1.1837e+00,  1.3771e-02,  2.6963e+00, -2.4460e-01, -3.0679e-01,\n",
       "         4.9507e-02, -9.3249e-02, -1.2190e-01,  7.3875e-02,  8.2177e-02,\n",
       "         1.7709e-01, -4.6640e-02, -1.6259e-01,  1.3047e-01, -7.7981e-02,\n",
       "         1.9621e-01, -2.2501e-01, -8.7846e-02, -7.1112e-02,  2.4189e-01,\n",
       "        -9.3553e-02, -1.2042e-02,  1.6407e-01, -6.7238e-02, -3.6608e-01,\n",
       "        -4.9779e-02, -2.6294e-01, -5.8337e-02,  2.9274e-01,  3.0474e-02,\n",
       "        -4.3713e-01,  9.1709e-02, -7.4387e-02,  1.3249e-01, -2.5243e-02,\n",
       "         1.0777e-01,  5.1872e-02, -1.1154e-01,  3.9594e-02,  1.6266e-01,\n",
       "        -1.2775e-01,  3.4357e-02, -3.5956e-02, -3.2496e-01, -1.6485e-01,\n",
       "        -3.0641e-02, -7.6774e-02,  1.4561e-01,  1.1531e-01,  1.8984e+00,\n",
       "         1.5799e-01,  8.4391e-02, -1.3433e-01,  1.1566e-01, -1.2106e-01,\n",
       "         1.0962e+00,  2.9572e-01,  2.3983e-01,  1.1480e-01,  6.6237e-03,\n",
       "        -1.3348e-01,  6.8766e-02,  1.5964e-01, -8.7224e-02, -2.3924e-01,\n",
       "        -1.7510e-01,  2.0747e-01, -1.6984e-01, -2.3177e-03, -4.9442e-01,\n",
       "        -1.6792e-01,  1.3460e-01, -1.8338e-02, -2.6355e-01, -1.6453e-01,\n",
       "         4.1224e-02, -1.8444e-01, -3.0363e-02, -6.3805e-02, -4.0107e-02,\n",
       "        -2.5481e-01,  3.1321e-02,  1.9193e+00, -2.1612e-01, -6.2593e-02,\n",
       "        -1.5077e-03,  5.2033e-03,  8.3699e-02,  4.0290e-02,  2.8803e-01,\n",
       "        -2.0666e-01,  1.1335e-01, -9.5299e-02,  2.2939e-01,  4.0757e-01,\n",
       "        -3.3981e-01, -9.0904e-02, -1.7730e-01, -5.3028e-02, -1.9618e-01,\n",
       "        -1.2492e-02, -1.0680e-01, -1.5958e-01, -5.0954e-02,  8.5235e-02,\n",
       "        -1.1794e-01,  3.0984e-01,  9.6872e-02,  3.1385e-01, -4.9641e-01,\n",
       "         5.1671e-03, -1.9114e-01,  1.5572e-01, -7.9885e-02,  1.3268e-01,\n",
       "         9.0588e-01,  6.7491e-02,  2.2338e-01, -1.0066e-01,  4.6575e-01,\n",
       "        -1.2312e-01,  9.3612e-02,  1.3096e-01, -3.3108e-01, -1.7744e-01,\n",
       "         1.1523e+00, -1.1713e+00, -3.3524e-02,  1.5911e-01, -1.0232e-01,\n",
       "        -1.4164e-01, -3.1028e-01, -2.4289e-01,  4.4483e-02, -7.8837e-04,\n",
       "        -2.5294e-01, -1.4119e-01,  7.1791e-03, -7.7787e-02,  6.7305e-02,\n",
       "         1.1571e-01, -9.7474e-02, -1.0530e+00, -1.5756e-01,  1.9618e-01,\n",
       "        -1.4641e-01, -2.1020e-01, -6.0207e-02,  2.0658e+00, -5.8721e-02,\n",
       "         1.3172e+00, -1.1151e-01, -6.5037e-02,  2.6665e-01, -1.5142e-01,\n",
       "        -1.2816e-02,  2.9121e-01,  1.5877e-01, -2.1064e-01, -2.3437e-01,\n",
       "         1.2074e-03,  1.3788e-01,  8.4911e-02,  9.8629e-03,  1.5704e-01,\n",
       "         1.1210e-01, -3.4607e-01,  2.2828e-02,  1.6462e-01, -5.3240e-02,\n",
       "         7.4044e-03,  3.4998e-02,  3.2578e-02, -1.2031e-01, -7.6079e-03,\n",
       "        -6.2595e-02,  1.6175e-01, -1.2981e-01,  2.4086e-01,  6.1456e-01,\n",
       "        -1.1655e-01,  1.2812e-01,  7.0912e-03,  6.9810e-02,  1.2587e-02,\n",
       "        -3.8313e-01,  3.9995e-01,  4.6975e-02, -3.8577e-02,  4.7613e-03,\n",
       "        -8.6680e-02,  8.2375e-02,  1.8488e-01,  6.3544e-02,  9.2193e+00,\n",
       "        -1.6143e-01,  9.3521e-02,  1.4227e-01, -1.1530e-02, -1.0175e-02,\n",
       "         1.7004e-02,  9.6651e-02,  7.3341e-01, -8.3059e-02, -8.0967e-02,\n",
       "        -2.4441e-01, -2.1112e-01,  1.3093e-01, -2.7853e-02, -3.1662e-02,\n",
       "         3.0409e-02,  2.4478e-02,  1.3748e-01,  5.7066e-01,  1.0895e-01,\n",
       "         1.0938e-02, -8.3650e-01, -1.2638e-01,  1.2183e-01, -1.4355e-01,\n",
       "        -4.6536e-02, -2.8628e-01, -1.8231e-01, -1.2930e-01,  1.6428e-01,\n",
       "         1.3582e-02, -1.5254e-01,  2.9710e-01,  3.5783e-02,  2.2965e-02,\n",
       "        -1.3139e-01,  6.8671e-02, -7.0601e-02,  4.3105e-03, -7.3840e-02,\n",
       "        -1.4461e-01, -1.2314e-01,  8.2990e-02,  1.5063e-01,  4.9478e-02,\n",
       "        -1.0543e-01, -1.4868e-01, -6.2989e-02, -2.4564e-01,  1.2984e-01,\n",
       "        -9.8305e-02,  9.6116e-03,  1.8233e-01,  1.1220e-01,  1.3147e-01,\n",
       "        -1.0651e-01, -1.9505e-03,  7.7677e-03, -1.5431e+00,  5.4617e-02,\n",
       "         1.4792e-01,  9.1974e-02, -5.9575e-02,  9.0482e-02, -4.0548e-02,\n",
       "         2.7410e-02,  1.6957e-01,  1.7686e-01, -2.4234e-01,  5.2125e-03,\n",
       "        -4.4122e-02,  4.4026e-02, -7.9261e-02, -1.2375e-01,  3.3830e-01,\n",
       "        -1.0975e-01, -4.1963e-02,  6.1868e-02,  3.8249e-02, -3.4647e-01,\n",
       "         6.9477e-02, -2.7612e-02, -9.5724e-02,  3.5617e-02,  3.0908e-01,\n",
       "         1.5625e-02, -1.5706e-02, -1.8647e-02, -1.3613e+00,  3.9733e-02,\n",
       "        -1.4360e-02, -5.6631e-01,  1.4364e-02, -2.0767e-01,  1.0025e-01,\n",
       "         5.8167e-01, -1.2165e-01, -9.4708e-02,  8.2568e-02,  1.6126e-01,\n",
       "         9.0681e-04,  2.1125e-01, -1.5409e-01, -1.5994e-01, -1.4560e-01,\n",
       "         8.7726e-02,  2.9405e-01,  2.7593e-02,  1.7728e-01, -4.0951e-02,\n",
       "         7.2310e-02, -1.1495e-01,  3.9496e-02, -1.3501e-01, -5.7911e-02,\n",
       "        -1.2620e-01,  2.1860e-01,  5.4778e-02,  5.2384e-02,  1.2734e-01,\n",
       "        -3.0588e-02, -2.3094e-01, -1.8980e-01, -2.1621e-02, -1.8698e-01,\n",
       "         5.8853e-01, -4.9644e-02, -1.7897e-01,  7.8744e-02,  1.8341e-01,\n",
       "        -1.0343e-01, -3.8085e-02, -1.4170e+00, -1.2232e-01,  6.6738e-02,\n",
       "        -1.5288e-01,  3.8547e-02,  1.4078e+00,  9.0353e-01, -6.9630e-02,\n",
       "         4.7199e-02,  1.0277e-01, -1.1472e-01, -3.2788e-02,  1.1692e-01,\n",
       "         8.2559e-02, -7.4094e-02, -1.7668e-01, -1.9571e-01, -1.2744e-01,\n",
       "         4.8258e-02, -6.5957e-02, -1.0730e-01, -1.7549e-01,  5.2153e-03,\n",
       "        -4.1777e-02, -2.4510e-02,  5.3595e-02, -5.0036e-02, -3.6234e-01,\n",
       "        -1.4430e-01,  1.5534e-01,  1.4770e-01,  2.0897e-03,  2.1379e-01,\n",
       "         1.8835e-02, -5.7863e-02,  9.5830e-02,  2.3745e-02, -1.2344e-01,\n",
       "        -1.3102e-01,  8.9221e-02,  4.6883e-02,  5.0474e-02, -4.1123e-02,\n",
       "         2.2810e+00,  5.2746e-01,  4.3884e-02,  1.1691e+00, -4.7088e-01,\n",
       "        -1.0998e-01,  5.3738e-02,  5.3674e-01, -1.0251e-01, -4.4597e-02,\n",
       "         1.1504e-01,  4.9237e-02,  9.1344e-02, -1.7354e-01,  9.9623e-02,\n",
       "        -3.6215e-02,  3.1805e-02,  8.1981e-02,  4.9603e-02, -9.5985e-01,\n",
       "        -1.3482e-01,  1.3472e-01, -5.4147e-03, -6.4292e-02, -1.5996e-01],\n",
       "       device='cuda:0', grad_fn=<SelectBackward>)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc.weight[data_lm.vocab.stoi[\"30-something\"]]\n",
    "enc.weight[data_lm.vocab.stoi[\"linklater\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[data_lm.vocab.stoi[\"30-something\"]], enc.weight[data_lm.vocab.stoi[\"linklater\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 对于都有的词汇，使用预训练的权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"house\" in wiki_itos, \"30-something\" in data_lm.vocab.itos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[data_lm.vocab.stoi[\"30-something\"]], enc.weight[data_lm.vocab.stoi[\"house\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成假的评论，使用语言模型预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = \"The color of the sky is\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The color of the sky is a mixture of Christian and Protestant Values . The sky is distinctive in that the sky consists of dark - coloured images of the Sun , and the stars are based on the Holy\n",
      "\n",
      "The color of the sky is European , the Caribbean , the Andes , and the Mediterranean . \n",
      " \n",
      "  = = = Internal = = = \n",
      " \n",
      "  The United States has a long history of about 70 million\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The color of the sky is a mixture of Italian and Greek and European words . The Italian used a second red , Spanish , as Italian . The other Italian word for \" Italian \"\n",
      "\n",
      "The color of the sky is Throat - Henry , a French Canadian using a Latin name , and has a \" French \" accent . He also wears a French , Australian , and\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4 id=\"LanguageLearner.predict\" class=\"doc_header\"><code>predict</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/learner.py#L116\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#LanguageLearner-predict-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4><blockquote><p><code>predict</code>(<strong><code>text</code></strong>:<code>str</code>, <strong><code>n_words</code></strong>:<code>int</code>=<strong><em><code>1</code></em></strong>, <strong><code>no_unk</code></strong>:<code>bool</code>=<strong><em><code>True</code></em></strong>, <strong><code>temperature</code></strong>:<code>float</code>=<strong><em><code>1.0</code></em></strong>, <strong><code>min_p</code></strong>:<code>float</code>=<strong><em><code>None</code></em></strong>, <strong><code>sep</code></strong>:<code>str</code>=<strong><em><code>' '</code></em></strong>, <strong><code>decoder</code></strong>=<strong><em><code>'decode_spec_tokens'</code></em></strong>)</p>\n",
       "</blockquote>\n",
       "<div class=\"collapse\" id=\"LanguageLearner-predict-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#LanguageLearner-predict-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>No tests found for <code>predict</code>. To contribute a test please refer to <a href=\"/dev/test.html\">this guide</a> and <a href=\"https://forums.fast.ai/t/improving-expanding-functional-tests/32929\">this discussion</a>.</p></div></div><p>Return the <code>n_words</code> that come after <code>text</code>.</p>\n",
       "<p><a href=\"https://docs.fast.ai/text.learner.html#LanguageLearner.predict\" target=\"_blank\" rel=\"noreferrer noopener\">Show in docs</a></p>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc(LanguageLearner.predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`temperature` 控制生成文本的随机性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The color of the sky is a reference to the American Civil War . The American Civil War was the first war in Europe . The United States Army was the first to use\n",
      "The color of the sky is a combination of English and French , and the French language is French . The French language is French , and the French language is French . The French\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.1) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 精调模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8XXWd//HXJ1uzNlvTdE1b2tJSlgoNBabI4lIFlUVQURkFYSrKiKM/Hf0N/pwZ/Tno6Di4dxgVV8RBqAIiFBFkp7SldKdbuqRt9qTZ1/uZP+5NCGm2NndN3s/H4z5y7znfe8/n3ib33e/5nvM95u6IiIgAJMW6ABERiR8KBRER6aNQEBGRPgoFERHpo1AQEZE+CgUREemjUBARkT4KBRER6aNQEBGRPimxLuBETZkyxefOnRvrMkREEsqGDRtq3L1opHYJFwpz585l/fr1sS5DRCShmNmB0bTT7iMREemjUBARkT4KBRER6aNQEBGRPhENBTPbb2ZbzGyTmQ05Omxm55pZj5ldG8l6RERkeNE4+uhSd68ZaqWZJQPfAB6LQi0iIjKMeNh99CngfqAq1oWIiEx0kQ4FB9aa2QYzWzVwpZnNBK4GVke4DhGRhHbnn3fxzO7qiG8n0qGwwt3PAS4DbjWziwasvxP4grv3DPciZrbKzNab2frq6sh/KCIi8aQn4Hz3id28XFYX8W1FNBTc/UjoZxWwBlg+oEkpcK+Z7QeuBX5oZlcN8jp3uXupu5cWFY14lraIyLhS19JJwGFKzqSIbytiA81mlgUkuXtT6P5K4Cv927j7vH7tfwY87O6/j1RNIiKJqLqpA4Ci7AQOBaAYWGNmvdu5x90fNbNbANxd4wgiIqNQ0xwKhUTuKbj7PmDpIMsHDQN3vyFStYiIJLLensKUKPQU4uGQVBERGUZ1FHsKCgURkThX3dRBZloyWZMif76xQkFEJM7VNHdEpZcACgURkbhX3dQRlfEEUCiIiMS96qaOqByOCgoFEZG4V63dRyIiAtDZHaChtUuhICIiUNsSvXMUQKEgIhLX+qa4UE9BREQUCiIi0iea8x6BQkFEJK719hQKs9Kisj2FgohIHKtu6mByegrpqclR2Z5CQUQkjkXzHAVQKIiIxLWaps6oHY4KCgURkbimnoKIiPSpblIoiIgI0NbZQ3NHt0JBREReP0dBYwoiIkJVlM9mBoWCiEjc6pviQj0FERGpDu0+mqqegoiI1DR1YAYFUZriAhQKIiJxq7q5g4LMNFKSo/dVHdEtmdl+M9tiZpvMbP0g6680s829683swkjWIyKSSKJ9jgJAShS2cam71wyx7gngQXd3MzsL+B9gcRRqEhGJe7EIhZjuPnL3Znf30MMswIdrLyIykdQ0d0T1HAWIfCg4sNbMNpjZqsEamNnVZrYT+CPwsSHarArtXlpfXV0dwXJFROKDu4/LnsIKdz8HuAy41cwuGtjA3de4+2LgKuCrg72Iu9/l7qXuXlpUVBTZikVE4kBTRzcd3YGonqMAEQ4Fdz8S+lkFrAGWD9P2aWC+mU2JZE0iIokg2tdm7hWxUDCzLDPL6b0PrAS2DmizwMwsdP8cIA2ojVRNIiKJoqYp+vMeQWSPPioG1oS+81OAe9z9UTO7BcDdVwPXAB8xsy6gDfhAv4FnEZEJq/ds5nFzSKq77wOWDrJ8db/73wC+EakaREQS1bjbfSQiIievuqmD5CQjLyM1qttVKIiIxKHgOQppJCVZVLerUBARiUOxOEcBFAoiInGpOgZnM4NCQUQkLh2ub2NGXkbUt6tQEBGJM80d3dS3djErX6EgIjLhlde3AjA7PzPq21YoiIjEmfK6NgD1FEREpF9PoUA9BRGRCe9QfRsZqckURvHazL0UCiIicaa8vpVZ+RmE5o6LKoWCiEicOVTXFpPxBFAoiIjEnfL61piMJ4BCQUQkrhxr66KxvVs9BRERie05CqBQEBGJK4f6zlFQKIiITHi9PQXtPhIREcrr28ielEJeZnQvrtNLoSAiEkdieY4CKBREROJK8ByF2IwngEJBRCRuuHtfTyFWFAoiInGiobWLls6emJ24BgoFEZG4cSjGRx4BpETyxc1sP9AE9ADd7l46YP2HgS+EHjYDn3D3VyNZk4hIvCqvD56jEKsT1yDCoRByqbvXDLGuDLjY3evN7DLgLuC8KNQkIhJ3DtWFegoF47SnMBJ3f77fwxeBWbGqRUQk1srr28jNSGVyemzOUYDIjyk4sNbMNpjZqhHa3gT8KcL1iIjErUMxPvIIIt9TWOHuR8xsKvC4me1096cHNjKzSwmGwoWDvUgoUFYBlJSURLJeEZGYKa9vY0FRdkxriGhPwd2PhH5WAWuA5QPbmNlZwI+BK929dojXucvdS929tKioKJIli4jERDycowARDAUzyzKznN77wEpg64A2JcADwN+6+65I1SIiEu9qmjtp7wrE9BwFiOzuo2JgTWj+jhTgHnd/1MxuAXD31cCXgULgh6F2xx22KiIyEcTDOQoQwVBw933A0kGWr+53/2bg5kjVICKSKPrOUYhxT0FnNIuIxIHecxRm5o3TMQURERm98vo2CrPSyJoU09PHFAoiIvFgf01LzHcdgUJBRCQu7K5q5tTi2J6jAAoFEZGYq2/ppKa5g4VTc2JdikJBRCTWdlc1A7BQPQUREdld1QTAwmL1FEREJrzdlc1kpSUzIzc91qUoFEREYm13VRMLinMIzewQUwoFEZEY21XZzMKpsR9PAIWCiEhMNbR2Ut3UEReHo4JCQUQkpvb0HnkUB4ejgkJBRCSmdlUGQ2GBdh+JiMjuqiYy05JjPhFeL4WCiEgM7alqZsHUbJKSYn/kESgURERialdlU9yMJ4BCQUQkZo61dVHZ2BEX01v0UiiIiMTInt7pLeJkkBkUCiIiMbM7dOTRqXEw51EvhYKISIzsrmomIzV+jjyCUYaCmc03s0mh+5eY2W1mlhfZ0kRExrddlU1xdeQRjL6ncD/QY2YLgJ8A84B7IlaViMgEsKcqfuY86jXaUAi4ezdwNXCnu38GmB65skRExrem9i6OHmtnQRwdeQSjD4UuM/sg8FHg4dCy1JGeZGb7zWyLmW0ys/WDrF9sZi+YWYeZfW70ZYuIJLbeq62dGkfnKACkjLLdjcAtwNfcvczM5gG/GuVzL3X3miHW1QG3AVeN8rVERMaFPZXxcwnO/kYVCu6+neCXN2aWD+S4+9fHunF3rwKqzOxdY30tEZFEsquyifTUJGblZ8a6lDcY7dFHT5nZZDMrAF4F7jazb4/iqQ6sNbMNZrZqLIWKiIwnOyoaWVScQ3IcHXkEox9TyHX3RuC9wN3uvgx42yiet8LdzwEuA241s4tOpkgzW2Vm681sfXV19cm8hIhI3HB3dhxtYvG0ybEu5TijDYUUM5sOvJ/XB5pH5O5HQj+rgDXA8hOuMPj8u9y91N1Li4qKTuYlRETiRnVTB3UtnSyeHl+DzDD6UPgK8Biw191fNrNTgN3DPcHMsswsp/c+sBLYOpZiRUTGgx0VwTmPTpsefz2F0Q403wfc1+/xPuCaEZ5WDKwxs97t3OPuj5rZLaHXWG1m04D1wGQgYGb/ACwJ7aoSERmXdhwNfsWdFoe7j0YVCmY2C/gesILg4PGzwKfdvXyo54SCY+kgy1f3u18BzDrBmkVEEtrOo43MyE0nN3PE072ibrS7j+4GHgRmADOBh0LLRETkBO2saGJxHO46gtGHQpG73+3u3aHbzwCN+IqInKCO7h72VDWzeFr8DTLD6EOhxsyuN7Pk0O16oDaShYmIjEd7q1roDnhcDjLD6EPhYwQPR60AjgLXEpz6QkRETkDfIHMcHo4KowwFdz/o7le4e5G7T3X3qwieyCYiIidgZ0UjaSlJzC3MinUpgxrLldc+G7YqREQmiJ0VTSwqziElOT4vfDmWquJrwg4RkQSw42hj3A4yw9hCwcNWhYjIBFDd1EFNc2fcDjLDCCevmVkTg3/5GxA/V5oWEUkAvYPM8TjnUa9hQ8Hd47dyEZEEs7Mifqe36BWfIx0iIuPQjqNNTJucTn5WWqxLGZJCQUQkSnYcbYzrXUegUBARiYrO7gB7q5vjepAZFAoiIlGxt7qZrh6P68NRQaEgIhIV24/0Tm+hnoKIyIS3/kAdOekpzC/KjnUpw1IoiIhEwbqyOs6dW0ByUnxPBqFQEBGJsJrmDvZWt7B8XkGsSxmRQkFEJMJeLqsDUCiIiAi8VFZHRmoyZ8zIjXUpI1IoiIhE2LqyOs6Zk0daSvx/5cZ/hSIiCexYWxc7KhpZPrcw1qWMikJBRCSCNhyowz0xxhMgwqFgZvvNbIuZbTKz9YOsNzP7rpntMbPNZnZOJOsREYm2l8rqSE02zi7Ji3UpozLs1Nlhcqm71wyx7jJgYeh2HvCj0E8RkXFhXVkdS2flkZ6aHOtSRiXWu4+uBH7hQS8CeWY2PcY1iYiERWtnN1vKj3Fuguw6gsiHggNrzWyDma0aZP1M4FC/x+WhZSIiCW/TwQa6A54w4wkQ+d1HK9z9iJlNBR43s53u/nS/9YOd733c5T9DgbIKoKSkJDKVioiE2UtldSQZLJuTH+tSRi2iPQV3PxL6WQWsAZYPaFIOzO73eBZwZJDXucvdS929tKioKFLlioiE1bqyOpbMmMzk9NRYlzJqEQsFM8sys5ze+8BKYOuAZg8CHwkdhXQ+cMzdj0aqJhGRaOnsDrDxYH3CnJ/QK5K7j4qBNWbWu5173P1RM7sFwN1XA48AlwN7gFbgxgjWIyISNa8crKejO5BQ4wkQwVBw933A0kGWr+5334FbI1WDiEisPL69krTkJFYsSKyeQqwPSRURGXfcnbXbK/mbBYXkJNB4AigURETCbmdFEwfrWnnH6dNiXcoJUyiIiITZ2m2VmMHbTiuOdSknTKEgIhJmj22rYFlJPkU5k2JdyglTKIiIhNGhula2H21k5emJ10sAhYKISFit3V4JkJDjCaBQEBEJq7XbKlg8LYc5hVmxLuWkKBRERMKktrmDl/fXsXJJYu46AoWCiEjYPLGzioDDygTddQQKBRGRsFm7rYKZeRmcPmNyrEs5aQoFEZEwaO7o5pndNaw8vZjQnG8JSaEgIhIGj26toKM7wLvOTOyLRyoURETC4IGN5cwpzEyoC+oMRqEgIjJGhxvaeGFfLe89e1ZC7zoChYKIyJj9/pXDuMPVZyf+JeYVCiIiY+DuPLCxnOVzCygpzIx1OWOmUBARGYPN5cfYW93Ce89J/F4CKBRERMbkgY3lTEpJ4vKzEvuoo14KBRGRk9TZHeDBV4+w8vRpTE6wK6wNRaEgInKSnnqtivrWrnGz6wgUCiIiJ+2BjYeZkj2JNy+YEutSwkahICJyEmqbO3hiZyVXvWkGKcnj56t0/LwTEZEoun9jOV09znXLZ8e6lLBSKIiInCB35951hyidk8+CqTmxLiesIh4KZpZsZq+Y2cODrMs3szVmttnM1pnZGZGuR0RkrF4qq2NfTQsfXF4S61LCLho9hU8DO4ZY90/AJnc/C/gI8J0o1CMiMib3rjtITnoKlyf4jKiDiWgomNks4F3Aj4dosgR4AsDddwJzzSxxr2MnIuNeQ2snj2yt4OqzZ5KRlhzrcsIu0j2FO4F/BAJDrH8VeC+AmS0H5gCzBjYys1Vmtt7M1ldXV0eqVhGRET2w8TCd3QGuO3f87TqCCIaCmb0bqHL3DcM0+zqQb2abgE8BrwDdAxu5+13uXurupUVFRZEpWERkBO7OvS8fZOnsPJYk8CU3h5MSwddeAVxhZpcD6cBkM/uVu1/f28DdG4EbASw4CXlZ6CYiEnc2HqxnV2Uz37jmzFiXEjER6ym4+/9191nuPhe4DvhL/0AAMLM8M0sLPbwZeDoUFCIicefXLx4kKy2Zd581I9alREwkewqDMrNbANx9NXAa8Asz6wG2AzdFux4RkdF4YW8tD7xymJsvnEfWpKh/dUZNVN6Zuz8FPBW6v7rf8heAhdGoQUTkZDV3dPP5373K3MJMPrvy1FiXE1HjN+5ERMLka3/czpGGNu675QIy08b316amuRARGcaTO6v4zbpDrLpoPsvmFMS6nIgb35GX4KqbOvjdhnIe3nyEpvZuHMcdUpOTOKckn4tOncKbFxZRkJU28ouJyAlraO3kC/dvZlFxDp95+8TY061QiAPuTmNbNzUtHdQ2d3L0WBuPbDnKEzuq6A44y+bks3BqNmaGAS2d3Tyxs5L7N5ZjBmfNyuOKpTO4YukMinImxfrtiIwb//rQdupaOvnpDecyKWX8nb08GIVCjL1ysJ5//N1mdlc1v2F5YVYaH7twHu8vnc2CqdnHPa8n4Gw5fIynd1WzdnsFX314O//2yA4uXDCF95fO5rIzppGUZNF6GyLjztO7qlnzymFue8sCzpiZG+tyosbcPdY1nJDS0lJfv359rMt4g0N1rTyxo5L3Lps16uu0tnf1cOefd3PX03uZNjmdG1bMZWpOOoXZaRRkpbFwag5pKaMf8tlT1cSaVw7z+1eOcLihjTNn5nL7u07j/FMKT/ZtiUxYbZ09rLzzr6QmJ/HIbW8mPTXxewlmtsHdS0dsp1AY3Hef2M1j2yro6gnQ1eN0dgeYPzWbtywq4i2LiykpzGRL+THuemYfj2w5Sk/AWTo7j1/etHzEYNhc3sD/+Z9X2V3VzHXnzub2d51GTpgu+h0IOH949TDffPQ1jhxr5+1Lirn+/DmkpySRkmwkJyWxqDhnXE7kJRIud/xpB//1133cu+r8cfMfK4XCGPxx81FuvWcjZ5fkMW1yOqnJSSQnGa+WN7CvugWAaZPTqWhsJ3tSCh86r4QFU7O5fc0WTp+Ryy+GCIb+vYOpOel8/ZozuWTR1Ii8h/auHn7ybBk/emovzR1vnE7qlClZ/Orm85iRlxGRbYsksm1HjnHF95/jfctm8fVrzop1OWGjUDhJhxvauOzOpzmlKJv7brmA1AHXXt1f08JfdlbxUlkty+bkc93ykr4AWLutgk/+eiNnzDw+GDYcqOcff/cqe6tb+EDpbG5/92mj3tU0FvUtneyqbKIn4HQHnJrmDv75D9uYnJHKr28+j7lTsiJew3hS1djOn3dUYRYM11OKspmSnUZbVw9lNS3sq27hQG0LDa1dNLV309jeRcCdc0ryWbFgCkumTx5xrKe9q4dNhxp4aV8drxyq56xZefzdm+eFrTcpQ+sJOFf/8DmONLTzxGcvJjdz/HzmCoUB1pXV8eU/bOWiU4u4aGERpXPzj9tP2BNwPvjfL7Lt8DEe+fSbmVN44l+Yj22r4NZfb2TelCxm5mdwrK2LY21dlNW0MCM3gzveeyYXnRrbmV63Hj7G3/7kJVKSk/jVTeexaNr4upxgOLV39VBe38ozu2t4ZMtR1h+oZ+CfTFZaMi2dPW9YlpGazOSMFCanp9IdcMpqgj3MvMxUlkwPzq7ZHXACobDuDgTo7gneP1jbSmdPADOYV5jFvpoWCrLS+NRbFvCh80omzFEwsfDTZ8v4ysPb+d4Hz+Y9S8fX/EYKhQFe2lfLnX/ezfoDdXT1OOmpSVxwSiHvWTqDladPI3tSCj94cg/ffOw1vvW+pVy77LjLOoza2m0VfPvxXaQmJ5GbkUpuZirzp2Sx6uL5ZMfJnCm7K5u4/icv0dEd4K2Li2lo7aS+tZPmjm7+Zv4UPri8JOxh4e7srW7hpbJaXtpXx+byBjLTUiiePImpOenMys/gmmWzRtyt5e7sqWqmoa2LyempTM5IIXtSCrXNneypamZPdTP7a1ro7A6QlGQkGaSl9J7bUcSU7OMP2+3qCbCrsomth4+xufwYuyubOVjXSkVje1+bRcU5XH7mdC47cxoZqcnsrW5mX3ULB+taKcxK45SibE4pymJuYdZxYzZVje08v7eW5/bUsKe6mWQzkpOMlGQjyYzU5CRSQo9n5mVw3rxCzp1bQG5mKpvLG/j6n3by/N5aZuVn8E+Xn8ZlZ0wjOLGwhMuRhjbe/u2/cu68Au6+4dxx9/kqFIbQ0tHNS2W1PL2rhse3V3K4oY1JKUlcfGoRf9lZxTvPmMb3Pnj2uPuFGMzB2lY+ec8G6po7ycsMHvWUnGQ8v7eGrh7nnJI8rl02m5KCTCZnpJCbkUpBVtqod2P0BJydFY28XFbHuv11rCurp6a5A4CpOZM4pySfzp4AVU3tVDZ2UNPcQbIZV509k1sunt93KG5jexd7qprZUn6sL1BqWzqH3faU7DTSU5Nxh4A7zR3dNLV3B8/rmJnLsjkFNLZ3UXGsnaPH2jhU30Znd/BaUDnpKSyelkNJQRYlBZmUFGZw5sy8QQ8NjhZ355ndNdzxp53sONrIhQum8C9XLBl3F42PpVW/WM/Tu6t5/DMXM7sgM9blhJ1CYRQCAWfjwXr+sOkIf9xylKxJyTz8qTeTmzF+9iOejNrmDh7YeJjfvHywb2C9v7mFmSydncfSWXksLM4mIzWZSSnJTEpNouJYOxsO1LPxYD2bDjbQFBrknpmXwblz8znvlELOP6WQuYWZxwVveX0rP36mjHtfPkhHd4CzZuZy9Fg7VU0dfW1m5mVw3ikFnD+vkOl56TS2BffbN7Z1kZ+Zxvyp2Swoyj5uX3Ag4Gw70siTr1Xx1GtVbD3cSGF2GtNy05mem86s/EzOmJnLWTNzKSnIjNtzPLp7Atyz7iDfeuw1Wjt7uHHFXD5ywdxx+SUWTWu3VbDqlxv4wjsX84lL5se6nIhQKJyg7p4A3QEfF8cjh0twd08zdS3BcZHGti4qGtvZXN7ApkMNVDZ2DPq8JINF0yZzTkkepXPzWT6vkJkncKRTbXMHP3t+Py/tq2N2QSYLpmazYGo2p03PYVa+vvwg+Bl987HX+O36Q7jD0lm5vPusGbxtSTEz8tI17nACWjq6efu3/0pOeioP33bhcQeXjBcKBYm4imPtHKhtoaM7ELr1kJeRxtLZuTpSJkoO1bXyyJajPLz5KFsOH+tbnjMphcLsNCZnpJIUGr9INuPsOXl8buWicfvFdzL+/8Pb+fGzZdz/iQvG9YR3ow2F+Bj1lIQ0LTedabnpsS5jQptdkMnHL57Pxy+ez4HaFl7YW0tNcwc1zZ3UtnTS2BY8JDbgTntXgP/66z52Hm3iBx8+J24OeoilrYeP8dPnyvjQeSXjOhBOhH4rRMaJOYVZIx5Gfe+6g9z++628f/UL3H3juRRPnrihHgg4X/r9Vgqy0vjCOxbHupy4oT6kyARy3fISfvLRUg7UtnD1D55j7bYKjjS0kWi7kcPht+sPselQA/90+Wnj6iS1sVJPQWSCuWTRVH778Qv42M9eZtUvNwDBw3AXFefwnqUzuG757HE/UF3X0sk3Ht3J8nkFXH32zFiXE1cUCiIT0Bkzc3nq85ew9XAjr1U08lplE5sONfDPD25j9V/38vdvWcD7ls0+oZl6E8k3/rST5vZuvnrlGRPinKQToVAQmaAy01JYPq+A5fOCA6zuznN7avmPx1/j9jVb+eGTe1k+r4CZeRnMzM9gVn4GcwqymJGXTkoCH7204UA9v11/iI9fdIqmeBmEQkFEADAzLlw4hRULCnlqVzU/fbaMdWV1VDS20xN4fcwhJcmYXZDJouIc3lc6i0sWTSU5Tk/2G6irJ8CXfr+V6bnp3PbWiXF5zROlUBCRNzAzLl00lUtD07p39wSoaGynvL6Ng7Wt7K9t4UBtKy/vr+PRbRXMzMvgw+eX8IHS2RQOMq9UvOjqCXDbb15hx9FGVl+/jCwdkjuoiJ+8ZmbJwHrgsLu/e8C6XOBXQAnBgPqWu9893Ovp5DWR+NDVE+Dx7ZX88oUDvLCvluxJKXzxssV8aHlJ3E0T0tkd4O/v2cja7ZX8v3cv4aYL58W6pKgb7clr0dgx+GlgxxDrbgW2u/tS4BLgP8wsLQo1icgYpSYncfmZ0/nNqvNZ+5mLWDo7ly/9fivX/feLfVOFx4OO7h4++etgIPzLeyZmIJyIiIaCmc0C3gX8eIgmDuRYcPg/G6gDuodoKyJx6tTiHH5103n8+zVnseNoI++882m+9dhrlNe3Rr0Wd6e6qYPn99Tws+fK+OhP1/HnHZV89crTuWGFAmEkEd19ZGa/A+4AcoDPDbL7KAd4EFgcavMBd//jIK+zClgFUFJSsuzAgQMRq1lExqaysZ1/fWgbf9paAcCK+VN4X+ksVi6ZFvFrgz/5WhVffWg7+/r1VPIyU/niOxdz3fKSiG473sV8Qjwzezdwubt/0swuYfBQuBZYAXwWmA88Dix198ahXldjCiKJoby+lfs3HOa+DYcorw9et+SC+YV9g9glheGb8ba8vpWvPLSdtdsrmV+UxYfPm8OiaTksLM6mKHuSzkUgPkLhDuBvCe4OSgcmAw+4+/X92vwR+Lq7PxN6/Bfgi+6+bqjXVSiIJJZAwHmxrJY/bw9ey6L3f/GLinN4xxnTuOyMaSyeloOZ0d0ToKqpg9bObuYXZQ/7Ze7ubDl8jD9uPsrPX9iPYXzqrQu4+cJTxu1Jd2MR81AYUMwlDN5T+BFQ6e7/YmbFwEaCPYWaoV5LoSCS2PbXtPDEzioe21bBy/vrcIcZuel0B5ya5g56T4mYXZDBlUtnctXZM1gwNYf2rh4O1LZSVtPMurJ6HttWweGGNpKTjHecXszt71pyQtftmGjiNhTM7BYAd19tZjOAnwHTASPYa/jVcK+lUBAZP6qbOnh8eyXP7akha1Iy03IzmDY5nSSDR7ZW8OzuagIOU7InUdvSQe/XVVpKEhctnMI7Tp/G204rJj9LBy2OJK5CIZwUCiITR3VTBw9vPsLWw43MLshg3pQsTpmSzfypWWSm6eSzE6GL7IhIwivKmcSNOow0qjQaIyIifRQKIiLSR6EgIiJ9FAoiItJHoSAiIn0UCiIi0kehICIifRQKIiLSJ+HOaDazY8DuQVblAseGWTZwfe/jwdpMAYacf2kEg9UxmvUj1T/w8WD3VX981A8n/x5Gqn+4NsPVO/DxeKy///14qH+4Ovs/jtZ30Bx3Lxrx2e6eUDfgrtEu779s4Prex4O1AdaHu76x1j8nFiqNAAAH80lEQVTc+xn4XlR/bOsfy3sYqf4TeQ8Trf5w/A6Fs/7h6hzmc4/438BIt0TcffTQCSx/aJj1D42izckY6TVOtv6Bjwe7r/rHf/3DtRmu3oGPx2P9o93+cMJZ/8Bl8fIdNKyE230UDWa23kcxcVS8Uv2xl+jvQfXHVizrT8SeQjTcFesCxkj1x16ivwfVH1sxq189BRER6aOegoiI9Bn3oWBmPzWzKjPbehLPXWZmW8xsj5l91/pdMNbM3m9m281sm5ndE96q31BD2Os3sxvMrNrMNoVuN4e/8r4aIvL5h9Zfa2ZuZhHb9xqhz/+W0PJNZvasmS0Jf+V9NUSi/s+Gfvc3m9kTZjYn/JW/oY5IvIeLzGyjmXWb2bXxVPMQr/dRM9sdun203/J5ZvZSaPlvzWzsl6Aby6FLiXADLgLOAbaexHPXARcQvFTon4DLQssXAq8A+aHHUxOs/huA7yfq5x9alwM8DbwIlCZS/cDkfm2uAB5NsPovBTJD9z8B/DbRfoeAucBZwC+Aa+OlZuApYO6AZQXAvtDP/ND93u+e/wGuC91fDXxirLWP+56Cuz8N1PVfZmbzzexRM9tgZs+Y2eKBzzOz6QT/eF/w4Cf+C+Cq0Oq/A37g7vWhbVQlWP1RE8H6vwr8O9AewfIjUr+7N/ZrmgVEbGAvQvU/6e6toaYvArMiVX8E38N+d98MBOKp5iG8A3jc3etC3zmPA+8M9XreAvwu1O7nhOFvfNyHwhDuAj7l7suAzwE/HKTNTKC83+Py0DKAU4FTzew5M3vRzN4Z0WqPN9b6Aa4Jdf9/Z2azI1fqoMZUv5mdDcx294cjXegQxvz5m9mtZraXYLDdFsFaBxOO359eNxH8H3i0hfM9RMtoah7MTOBQv8e976MQaHD37gHLx2TCXaPZzLKBvwHu67eLetJgTQdZ1vs/uhSCu5AuIfi/pGfM7Ax3bwhvtYMUFZ76HwJ+4+4dZnYLwf9hvCXctQ5mrPWbWRLwnwR3gUVdmD5/3P0HwA/M7EPAl4CPDtI+7MJVf+i1rgdKgYvDWeNIwvkeomW4ms3sRuDToWULgEfMrBMoc/erGfp9ROT9TbhQINg7anD3N/VfaGbJwIbQwweBH/HGbvEs4Ejofjnwort3AWVm9hrBkHg5koWHjLl+d6/tt/y/gW9ErNrjjbX+HOAM4KnQH9c04EEzu8Ld10e4dgjP709/94baRktY6jeztwG3Axe7e0dEKz5euP8NomHQmgHc/W7gbgAzewq4wd3392tSTvA/oL1mERx7qAHyzCwl1FsIz/sL9wBLPN4IDipt7ff4eeB9ofsGLB3ieS8D5/P6INXloeXvBH4euj+FYNeuMIHqn96vzdUEAy5hPv8BbZ4iggPNEfr8F/Zr8x7GMM9NjOo/G9jb/31E+hap3yHgZ0RgoPlka2bogeYygoPM+aH7BaF19/HGgeZPjrnuaP2jxuoG/AY4CnQRTNybgHnAo8CrwHbgy0M8txTYGvoD+D6vn+xnwLdDz93S+4+SQPXfAWwLPf9JYHEi1T+gzVNE9uijSHz+3wl9/ptCn//pCVb/n4HKUP2bgAcjVX8E38O5oddqAWqBbfFQM4OEQmj5x4A9oduN/ZafQvAIqz0EA2LSWGvXGc0iItJnoh59JCIig1AoiIhIH4WCiIj0USiIiEgfhYKIiPRRKMi4YGbNUd7ej8M1u6mZ9VhwxtStZvaQmeWN0D7PzD4Zjm2LDKRDUmVcMLNmd88O4+v1niUacf1rN7OfA7vc/WvDtJ8LPOzuZ0SjPplY1FOQccvMiszsfjN7OXRbEVq+3MyeN7NXQj8XhZbfYGb3mdlDwFozu8TMngpNGrjTzH4dmpmS0PLS0P1mM/uamb0amiCxOLR8fujxy2b2lVH2Zl7g9Yn/si14vYKNFrwmwJWhNl8H5od6F98Mtf18aDubzexfw/gxygSjUJDx7DvAf7r7ucA1wI9Dy3cCF7n72cCXgX/r95wLgI+6e+8EgWcD/wAsIXj26IpBtpNFcKqQpQSv8fB3/bb/ndD2R5yTJjR3z1sJztsDwWnBr3b3cwhew+A/QqH0RWCvu7/J3T9vZisJzr21HHgTsMzMLhppeyKDmYgT4snE8TZgSb9ZKSebWQ6QC/zczBYSnFUytd9zHnf3/vPgr3P3cgAz20RwPptnB2ynE+idxnsD8PbQ/Qt4fX77e4BvDVFnRr/X3kBwvnwITqfyb6Ev+ADBHkTxIM9fGbq9EnqcTTAknh5ieyJDUijIeJYEXODubf0Xmtn3gCfd/erQ/vmn+q1uGfAa/WcA7WHwv5kuf31wbqg2w2lz9zeZWS7BcLkV+C7wYaAIWObuXWa2H0gf5PkG3OHu/3WC2xU5jnYfyXi2Fvj73gdm1jttcS5wOHT/hghu/0WCu60ArhupsbsfI3jBnc+ZWSrBOqtCgXAp0Hst5CaCU4j3egz4WGjOfsxspplNDdN7kAlGoSDjRaaZlfe7fZbgF2xpaPB1O3BLqO2/A3eY2XNAcgRr+gfgs2a2DpgOHBvpCe7+CsFZNK8Dfk2w/vUEew07Q21qgedCh7B+093XEtw99YKZbSF4ecacQTcgMgIdkioSIWaWSXDXkJvZdcAH3f3KkZ4nEksaUxCJnGXA90NHDDUQnBNfJK6ppyAiIn00piAiIn0UCiIi0kehICIifRQKIiLSR6EgIiJ9FAoiItLnfwHeuMamhGkCcAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot(skip_end=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 14:14 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.260913</td>\n",
       "      <td>5.668831</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>14:14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(1, 1e-2, moms=(0.8, 0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7663"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 保存权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save('fit_1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 这一步适应性训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[data_lm.vocab.stoi[\"30-something\"], :], \n",
    "            enc.weight[data_lm.vocab.stoi[\"linklater\"], :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.load('fit_1');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 16:43 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.917771</td>\n",
       "      <td>5.118861</td>\n",
       "      <td>0.185714</td>\n",
       "      <td>16:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(1, 1e-3, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm = None\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 重新加载模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.load('fine_tuned');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 现在看一下只有 IMDB 中有的某些的词的权重\n",
    "- 训练之后就不一样了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 3.2072e-03, -1.6049e-01,  2.1455e-01, -6.2848e-02, -2.8184e-01,\n",
       "        -9.3318e-01, -1.0888e-01,  2.1016e+00, -4.3233e-01, -8.8225e-03,\n",
       "         1.4522e-01, -2.9050e-01, -1.9969e-01, -9.7504e-02,  1.1139e-01,\n",
       "        -2.4093e-01, -3.9123e-01, -1.7941e-01, -1.4594e-01,  3.5768e-01,\n",
       "         3.1340e-02, -1.2044e+00, -3.4958e-01, -8.0277e-02, -8.3552e-03,\n",
       "        -4.3774e-02,  8.2727e-02,  1.1909e-01,  1.0847e-01, -4.1513e-01,\n",
       "        -6.0598e-02, -1.0957e-01,  1.0951e-01, -4.2390e-02,  8.6893e-02,\n",
       "        -5.6217e-01,  8.1836e-02, -3.1742e-02,  1.0327e-01, -2.5945e-01,\n",
       "         2.8270e-02, -2.5572e-01,  3.9597e-02,  2.4161e-01,  3.7453e-01,\n",
       "         4.4306e-02, -1.4005e-01,  1.8597e-01, -3.8311e-01, -2.4049e-01,\n",
       "         1.5460e-01, -2.1910e-01,  3.0859e-02,  4.8729e-01,  1.5921e+00,\n",
       "         1.5239e-01, -3.4949e-02, -3.6023e-01,  1.0635e-01,  3.2286e-02,\n",
       "        -3.7845e-01,  8.3109e-02,  1.6261e-01,  2.8154e-01,  3.8619e-01,\n",
       "        -2.0305e-01, -1.4512e-01,  1.1038e-01, -3.5582e-01, -1.9876e-01,\n",
       "        -5.3879e-01, -1.4806e-01, -2.1337e-01,  3.4775e-01, -4.2924e-01,\n",
       "        -1.1430e-01, -5.5299e-02, -2.1030e-01, -3.0929e-01, -5.1003e-01,\n",
       "         8.7944e-02, -3.2874e-01, -2.8365e-01, -1.3081e-01,  1.9118e-01,\n",
       "        -2.7696e-01,  2.0294e-01,  1.3704e+00, -2.0966e-01,  4.8365e-02,\n",
       "         4.2256e-02, -5.6682e-01,  1.2348e-01, -6.4662e-02,  1.5671e-01,\n",
       "        -3.7555e-01,  2.4129e-01, -4.8201e-01,  9.7320e-02,  5.6699e-01,\n",
       "        -1.5622e-01, -3.4294e-01, -1.6296e-02, -3.0570e-01, -6.3216e-01,\n",
       "        -1.1362e-01,  1.8251e-02, -2.0633e-01, -4.0456e-01, -4.0568e-01,\n",
       "        -2.6845e-02,  4.0977e-01,  5.6334e-02,  2.9079e-01,  6.8994e-01,\n",
       "        -5.6448e-02,  2.5892e-02, -2.9972e-01, -3.5513e-01, -5.0623e-02,\n",
       "         7.8032e-01,  5.2905e-01,  6.2244e-01,  1.7565e-01,  4.3115e-01,\n",
       "         2.0677e-02,  6.8202e-02,  7.4512e-03, -2.1387e-01, -2.4766e-01,\n",
       "         2.6501e-01, -1.8106e-01, -1.5098e-01,  9.9215e-02,  2.7061e-01,\n",
       "         1.1871e-02, -8.1747e-01, -1.0925e-01,  8.9148e-02, -6.4667e-03,\n",
       "         5.5533e-01,  6.0048e-02,  1.7300e-01, -9.7943e-02,  4.8628e-02,\n",
       "         6.4111e-01, -1.1639e-01,  1.4264e-01, -3.7794e-01,  4.0327e-01,\n",
       "        -6.5348e-02, -5.6119e-01, -1.7663e-01,  1.1106e+00, -1.1166e-01,\n",
       "         7.4152e-02, -3.3279e-01,  8.8858e-02,  3.2363e-01, -3.2095e-01,\n",
       "         2.9635e-01,  4.0026e-01,  7.7452e-02,  3.3269e-03,  6.7458e-03,\n",
       "         3.4015e-01,  9.0730e-01,  3.3794e-01,  2.2707e-02,  2.9353e-01,\n",
       "         7.8215e-02, -4.4989e-01,  3.8187e-02,  2.6789e-01, -3.1380e-01,\n",
       "        -2.4873e-01,  2.9518e-01, -8.9570e-02, -1.9902e-01,  1.7157e-02,\n",
       "        -4.8244e-01, -3.2615e-02, -1.9876e-01,  2.9867e-01,  8.6943e-01,\n",
       "        -1.5601e-01,  5.5195e-01,  5.0381e-02,  3.3495e-01,  1.8484e-01,\n",
       "        -3.8948e-01,  3.6178e-01,  1.0290e-01, -1.3010e-01,  3.5800e-01,\n",
       "         1.9129e-01, -2.0529e-01,  5.0182e-01, -4.0331e-02,  8.8757e+00,\n",
       "        -5.9882e-02, -2.2947e-01,  2.0299e-01, -1.5453e-01, -1.2900e-01,\n",
       "         2.2644e-01,  1.2496e-01,  5.6830e-01, -1.8268e-01,  9.3942e-02,\n",
       "        -4.1212e-01, -9.5684e-01, -3.8017e-01, -2.1725e-01,  2.6683e-01,\n",
       "        -1.9907e-01,  2.7064e-01,  1.1694e-01,  3.5839e-01,  2.1850e-01,\n",
       "        -3.7102e-01, -1.4626e-01, -2.5134e-01,  7.4154e-02, -2.5009e-01,\n",
       "        -3.2561e-01, -2.2738e-01, -3.7088e-01,  6.0383e-01,  2.5604e-01,\n",
       "        -6.3643e-01, -2.5005e-01,  4.3555e-01,  2.8744e-02,  1.9224e-01,\n",
       "        -1.0021e-01, -1.1240e-01,  2.8212e-01, -4.3475e-01, -8.3729e-02,\n",
       "         6.3341e-02, -1.3068e-01, -1.1026e-01,  5.9544e-01, -2.5056e-01,\n",
       "        -5.8367e-02, -2.8420e-01, -6.0897e-02, -1.9172e-01,  1.2294e-01,\n",
       "         5.1019e-02,  5.0826e-03,  3.5157e-01,  1.5534e-01,  3.3241e-02,\n",
       "        -1.3837e-01,  1.8443e-01,  2.9667e-01, -6.9218e-01, -1.1258e-01,\n",
       "         5.8610e-02,  2.7609e-01, -4.4075e-01, -5.0866e-01, -5.0395e-02,\n",
       "         8.1926e-02,  4.2362e-02,  3.0205e-02, -5.5013e-01,  1.7376e-01,\n",
       "        -1.4639e-01, -1.2295e-01, -6.2993e-02, -2.5340e-01,  9.6575e-01,\n",
       "        -2.9015e-02, -1.4661e-01,  1.9746e-01, -2.0859e-02, -7.5505e-01,\n",
       "         4.1256e-01, -2.9339e-01, -4.4570e-02,  1.3843e-01,  5.8493e-01,\n",
       "         2.1994e-01, -3.3715e-01,  1.4294e-01, -1.1739e+00, -4.6497e-02,\n",
       "        -2.6216e-01, -1.2824e-01,  5.7833e-02, -2.0968e-01,  9.1114e-02,\n",
       "         6.1353e-01, -2.1772e-01,  6.0257e-02, -6.2656e-02,  4.2045e-01,\n",
       "         5.1398e-02,  5.3682e-01,  1.4036e-02, -1.3292e-01,  8.0947e-02,\n",
       "         1.0566e-01,  3.1596e-02, -4.0706e-01,  3.3408e-02, -4.3693e-01,\n",
       "        -1.6066e-01, -1.9122e-02, -1.7751e-01, -5.9775e-01,  1.4602e-01,\n",
       "        -2.5772e-02,  5.1349e-01,  1.0483e-01,  2.2570e-01, -4.7489e-01,\n",
       "         4.6471e-01, -3.5066e-01, -5.3258e-01,  2.6321e-01, -1.9204e-01,\n",
       "        -7.6083e-02, -3.5986e-01, -2.3723e-01,  3.3098e-01,  2.6599e-01,\n",
       "        -5.5280e-02, -4.2651e-01, -1.5426e+00,  3.7679e-02, -1.3529e-01,\n",
       "        -2.2923e-01,  1.6272e-01, -3.9520e-02,  7.4568e-01,  3.0796e-01,\n",
       "        -1.1327e-01,  3.0942e-01,  1.0860e-02, -5.6625e-02,  2.7232e-01,\n",
       "        -6.9728e-03, -1.6894e-01,  9.4825e-02,  8.7606e-02,  1.0178e-02,\n",
       "         1.0923e-01, -4.0916e-01, -1.4897e-01, -2.3481e-01,  2.6810e-01,\n",
       "        -2.6776e-01,  7.7929e-03,  8.7749e-02,  1.7263e-01, -3.6510e-01,\n",
       "        -6.0855e-02,  1.5435e-01,  4.6148e-02, -4.5243e-01, -2.5102e-01,\n",
       "         1.1618e-01, -4.6618e-01,  3.9574e-01,  2.3393e-01, -2.2546e-01,\n",
       "        -3.0398e-01,  1.6105e-01,  1.7953e-01,  5.1444e-02,  1.0394e-01,\n",
       "         2.1791e+00,  3.3454e-02,  5.2646e-01,  3.0997e-01, -3.6243e-01,\n",
       "        -6.3585e-02, -4.9616e-03,  6.0955e-01,  1.8524e-01, -2.4877e-01,\n",
       "         6.6345e-01, -4.4104e-02, -6.1766e-02, -1.5054e-01,  8.3021e-02,\n",
       "        -4.4084e-02, -3.8747e-02,  4.3642e-02,  4.4176e-01, -1.4935e+00,\n",
       "        -2.0664e-01,  6.6437e-02,  2.7769e-02, -2.3057e-01,  4.1281e-02],\n",
       "       device='cuda:0', grad_fn=<SelectBackward>)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "tensor([-2.8410e-01, -1.8796e-01,  4.4240e-01, -4.2873e-02, -1.2539e-01,\n",
       "        -1.1355e+00, -1.4613e-01,  2.2823e+00, -3.4737e-01, -1.5362e-01,\n",
       "         1.3069e-01, -3.0548e-01, -4.7165e-01,  1.9453e-01, -5.0568e-03,\n",
       "        -6.0487e-02, -6.3183e-02,  1.8900e-02,  5.2805e-02,  2.5666e-01,\n",
       "         5.8136e-01, -6.8515e-01, -5.9119e-02,  7.8001e-03, -1.7621e-01,\n",
       "         1.7742e-01,  2.3140e-01,  1.7526e-01, -6.6958e-03, -5.5932e-01,\n",
       "        -2.6108e-01, -1.1224e-01,  1.3781e-01,  5.2185e-02, -1.0861e-02,\n",
       "        -8.6108e-01,  2.8564e-01, -7.3393e-02,  2.9471e-01, -4.2485e-01,\n",
       "        -5.1629e-02, -5.6315e-01,  2.6824e-01, -1.5991e-02,  9.3768e-02,\n",
       "         1.9217e-01, -2.1691e-01, -1.1009e-01, -3.7561e-01, -3.5039e-01,\n",
       "         1.2531e-01, -2.2624e-01, -2.5802e-01,  4.4876e-01,  1.8530e+00,\n",
       "         6.5083e-01, -3.6856e-01, -7.3601e-02,  1.0223e-01,  1.7379e-01,\n",
       "        -1.9559e-01, -1.0485e-01,  1.7183e-01,  1.0943e-01, -8.1200e-03,\n",
       "         4.7157e-02,  2.2152e-01,  3.3878e-01, -8.8590e-01, -6.6900e-02,\n",
       "        -6.2824e-01, -2.7627e-02, -6.1724e-01,  3.9871e-01, -5.7253e-01,\n",
       "        -1.9158e-01, -1.0822e-01, -3.0273e-01, -4.0532e-01, -4.8808e-01,\n",
       "        -2.1560e-01, -5.5545e-01, -2.4717e-02, -2.1521e-01, -1.5785e-03,\n",
       "        -1.8136e-01,  2.8467e-01,  1.3490e+00, -2.4802e-01, -4.8331e-02,\n",
       "         2.0568e-02,  1.3431e-01, -1.6624e-01, -2.9725e-03,  3.8115e-01,\n",
       "        -1.5479e-01,  5.2534e-02, -1.9676e-01, -4.3543e-02,  7.1940e-01,\n",
       "        -2.7793e-01, -1.2500e-01, -1.2166e-01, -2.1813e-01, -2.3993e-01,\n",
       "        -1.1590e-01,  2.4179e-01, -7.4378e-02, -1.4798e-01, -4.1731e-01,\n",
       "        -1.5961e-01,  2.3532e-01,  3.2890e-01,  2.3144e-01,  8.2273e-01,\n",
       "         1.5512e-01, -2.8808e-01,  6.8657e-02, -1.2321e-01,  2.2262e-03,\n",
       "         9.2312e-01,  3.5150e-01,  4.7032e-01,  4.0830e-01,  1.5860e-01,\n",
       "         3.0736e-02,  2.4376e-01,  3.2240e-01, -3.9955e-01, -3.9860e-01,\n",
       "         2.5719e-01, -5.7429e-01, -1.2209e-01,  8.5918e-01,  3.8260e-01,\n",
       "        -9.9970e-02, -1.0898e+00, -3.4944e-01, -1.4341e-01,  1.5101e-03,\n",
       "         5.5787e-01,  3.6318e-01,  6.1475e-02, -4.2635e-01,  2.0925e-01,\n",
       "         8.2191e-01, -7.1197e-01,  2.2355e-01, -4.0029e-01,  1.4997e-01,\n",
       "        -7.0186e-01, -1.7296e-01, -2.7346e-01,  2.5176e-01,  2.3784e-01,\n",
       "         6.5719e-01, -3.7346e-01, -1.0494e-01,  1.1830e-01, -2.9759e-02,\n",
       "         3.3359e-01,  2.8145e-01,  3.4801e-01,  2.2596e-02,  2.3512e-03,\n",
       "         3.6920e-01,  5.5980e-01,  1.3984e-01,  1.8084e-01, -9.7945e-02,\n",
       "         1.1761e-01, -1.8338e-01, -1.8169e-01,  3.7646e-01, -2.9441e-01,\n",
       "        -1.8420e-01, -7.5501e-02,  1.3482e-01, -5.4801e-01, -7.3966e-02,\n",
       "        -2.7229e-01,  6.7325e-02, -1.0520e-01,  3.3520e-01,  6.1277e-01,\n",
       "        -2.3914e-01,  3.2733e-01,  8.5604e-03,  8.3634e-02,  2.8925e-02,\n",
       "        -3.7579e-01,  4.1374e-01, -1.9406e-01, -4.1681e-01,  9.2406e-02,\n",
       "        -1.5869e-01, -4.0148e-01,  1.1211e+00, -1.0736e-01,  7.2367e+00,\n",
       "        -2.7321e-01, -2.0823e-01,  8.4814e-02,  1.4035e-01, -1.1856e-01,\n",
       "         1.6187e-01,  6.7617e-01,  3.5006e-01, -1.3847e-01,  1.1139e-01,\n",
       "         3.1352e-02, -1.2064e+00, -3.5382e-02,  1.7106e-01,  1.8934e-01,\n",
       "         4.4297e-02,  5.1899e-01, -2.2949e-01,  5.7160e-02,  2.6484e-01,\n",
       "        -4.5649e-02, -3.8650e-01, -2.9634e-01, -1.0247e-01, -3.8327e-01,\n",
       "        -2.4686e-01, -3.6578e-01, -4.4170e-02,  1.7244e-01, -5.4352e-02,\n",
       "        -1.0271e+00,  9.7718e-02,  5.8638e-01, -2.7264e-01, -1.9690e-01,\n",
       "        -5.1144e-02, -5.1291e-02,  3.2031e-01, -7.6261e-01, -1.3990e-01,\n",
       "        -1.2354e-01,  1.9556e-02, -2.4576e-01,  4.4885e-01,  1.0648e-01,\n",
       "        -4.2025e-01,  1.6266e-01, -2.6699e-01, -4.8763e-01,  4.8752e-01,\n",
       "        -1.8274e-01, -1.3313e-01,  2.6738e-01,  6.0671e-02,  7.3673e-03,\n",
       "        -2.3561e-01, -1.2789e-01,  3.7196e-01, -1.2162e+00,  2.1617e-01,\n",
       "         1.8818e-01,  3.0653e-02,  8.2175e-02, -7.8372e-01, -2.0378e-01,\n",
       "        -2.4649e-01, -3.5442e-01,  8.9984e-02, -3.9282e-01,  3.8246e-01,\n",
       "        -6.5304e-01,  2.0555e-02, -7.1869e-01, -3.4911e-01,  1.1765e+00,\n",
       "        -1.6891e-01, -1.3634e-01,  1.1095e-01,  1.8429e-01, -9.7468e-01,\n",
       "         6.5331e-01, -1.5775e-01,  1.6140e-01,  5.7514e-02,  7.0419e-01,\n",
       "        -3.0371e-01, -3.3138e-01, -1.8356e-01, -1.5634e+00, -3.3345e-01,\n",
       "        -2.0439e-01,  3.8797e-02, -2.1169e-01, -5.0553e-01, -8.1121e-02,\n",
       "         5.5980e-01, -4.9569e-01,  1.4824e-01,  1.4086e-01,  1.0681e-01,\n",
       "         6.8477e-03,  9.3167e-01,  1.5246e-01,  2.5944e-01,  1.3474e-02,\n",
       "         3.7308e-01,  3.7578e-01, -2.6785e-01, -1.6664e-01, -6.4617e-01,\n",
       "        -7.6377e-02, -1.5058e-01, -7.4865e-02, -3.9274e-01, -1.6629e-01,\n",
       "        -2.5767e-01,  2.0508e-01, -1.8796e-01,  3.2464e-01, -1.7722e-01,\n",
       "         2.5431e-01,  1.4741e-01, -3.4853e-01,  2.3387e-01, -8.7161e-05,\n",
       "        -8.7632e-01, -5.1200e-02, -1.6130e-01, -1.1157e-01,  5.2701e-01,\n",
       "        -9.9872e-02, -6.3589e-01, -2.0251e+00,  1.4272e-01, -2.7880e-01,\n",
       "         2.0807e-01,  2.0852e-01,  4.2798e-01,  8.0418e-01,  3.3886e-01,\n",
       "        -3.2953e-01,  1.7258e-01,  2.1137e-01,  1.3464e-01, -9.6772e-02,\n",
       "         2.0936e-01, -1.3584e-01,  4.0104e-02, -2.0041e-02,  2.2589e-02,\n",
       "         4.5968e-02, -4.7307e-01,  3.3104e-01,  8.5960e-02,  1.1722e-01,\n",
       "        -1.3776e-01, -5.3053e-02,  8.7054e-02, -2.7274e-01, -3.2108e-01,\n",
       "        -8.7048e-03,  4.2871e-01,  4.2416e-01, -5.4685e-01, -5.0112e-01,\n",
       "        -3.1476e-02, -5.6220e-01,  1.0262e-01,  1.4496e-02, -3.2813e-01,\n",
       "         3.2563e-01,  9.6348e-01,  8.1113e-02,  2.8098e-01,  5.5310e-01,\n",
       "         1.7733e+00,  1.2737e-01,  2.6959e-02,  7.0505e-01, -7.8179e-02,\n",
       "         2.0829e-01, -9.1146e-02,  9.8538e-01, -3.2410e-01, -5.4431e-01,\n",
       "         6.1956e-01,  2.0331e-01, -1.1844e-01, -6.0821e-01, -1.3395e-01,\n",
       "         5.0338e-01,  3.4244e-01, -2.1357e-01,  2.3819e-01, -1.6635e+00,\n",
       "         5.8559e-03,  4.0923e-01,  1.0143e-01,  2.5649e-02, -1.8601e-01],\n",
       "       device='cuda:0', grad_fn=<SelectBackward>)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc.weight[data_lm.vocab.stoi[\"30-something\"]]\n",
    "enc.weight[data_lm.vocab.stoi[\"linklater\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(enc.weight[data_lm.vocab.stoi[\"30-something\"], :], \n",
    "            enc.weight[data_lm.vocab.stoi[\"linklater\"], :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 使用训练好的语言模型生成假的评论"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = \"i liked this movie because\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i liked this movie because it was one of my favorite movies of all time . i was in a high school , i was a teenager . It was a new life that i was earned at the time of the Film\n",
      "\n",
      "i liked this movie because i am a huge fan of Prom Night , and for those of Blair Witch Project who have the better performances of the three . \n",
      " \n",
      "  The plot of the movie is about the\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i liked this movie because it was a great movie . It was a great movie . It was a great movie . It was a great movie . It was a great movie . It was a great movie\n",
      "\n",
      "i liked this movie because it was a great movie . It was a great movie , and i loved it . It was a great movie , and i loved it . It was a great movie . It was\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.1) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存编码器\n",
    "- fastai 的第1层保存的是模型的编码器部分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**语言模型涉及到一些伦理问题**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_lm = None\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分类器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.IMDB)\n",
    "BATCH = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)\n",
    "#              #grab all the text files in path\n",
    "#              .split_by_folder(valid='test')\n",
    "#              #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)\n",
    "#              .label_from_folder(classes=['neg', 'pos'])\n",
    "#              #label them all with their folders\n",
    "#              .databunch(bs=BATCH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 保存预处理数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_clas.save('imdb_textlist_class')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 加载预处理数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, 'imdb_textlist_class', bs=BATCH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj in xxup nyc , seaman xxmaj michael o'hara ( xxmaj orson xxmaj welles ) rescues xxmaj elsa xxmaj bannister ( xxmaj rita xxmaj hayworth ) from a mugging &amp; rape as she takes a horse &amp; carriage through xxmaj central xxmaj park -and lives to regret it . xxmaj xxunk - haired xxmaj hayworth 's a platinum blonde in this one ; as dazzling as fresh - fallen</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj after reading the previous comments , i 'm just glad that i was n't the only person left confused , especially by the last 20 minutes . xxmaj john xxmaj carradine is shown twice walking down into a grave and pulling the lid shut after him . i anxiously awaited some kind of explanation for this odd behavior ... naturally i assumed he had something to do with</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj in his 1966 film \" xxmaj blow xxmaj up \" , xxmaj antonioni had his hero question truth against a backdrop of xxmaj british youth protesters . xxmaj by setting such questions against a fabric of hippie youth movements , xxmaj antonioni questioned , intentionally or not , the effectiveness of these organisations . xxmaj how can you fight for a cause when what you think is true</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>xxbos xxmaj this movie was extremely poorly conceived from every angle except technological . i stood and watched everyone xxunk out of the theater , their faces drained like their lives flashed before their eyes -- eyes wandering at their neighbor , wondering if it was just them . i mean , how could the movie really be bad . xxmaj xxunk admit it , it 's a classic case</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_clas.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 构建分类模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SequentialRNN(\n",
       "  (0): MultiBatchEncoder(\n",
       "    (module): AWD_LSTM(\n",
       "      (encoder): Embedding(49852, 400, padding_idx=1)\n",
       "      (encoder_dp): EmbeddingDropout(\n",
       "        (emb): Embedding(49852, 400, padding_idx=1)\n",
       "      )\n",
       "      (rnns): ModuleList(\n",
       "        (0): WeightDropout(\n",
       "          (module): LSTM(400, 1150, batch_first=True)\n",
       "        )\n",
       "        (1): WeightDropout(\n",
       "          (module): LSTM(1150, 1150, batch_first=True)\n",
       "        )\n",
       "        (2): WeightDropout(\n",
       "          (module): LSTM(1150, 400, batch_first=True)\n",
       "        )\n",
       "      )\n",
       "      (input_dp): RNNDropout()\n",
       "      (hidden_dps): ModuleList(\n",
       "        (0): RNNDropout()\n",
       "        (1): RNNDropout()\n",
       "        (2): RNNDropout()\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (1): PoolingLinearClassifier(\n",
       "    (layers): Sequential(\n",
       "      (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (1): Dropout(p=0.2)\n",
       "      (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "      (3): ReLU(inplace)\n",
       "      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (5): Dropout(p=0.1)\n",
       "      (6): Linear(in_features=50, out_features=2, bias=True)\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultiBatchEncoder(\n",
       "  (module): AWD_LSTM(\n",
       "    (encoder): Embedding(49852, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(49852, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(400, 1150, batch_first=True)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(1150, 1150, batch_first=True)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(1150, 400, batch_first=True)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 编码器\n",
    "enc = learn_c.model[0]\n",
    "enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PoolingLinearClassifier(\n",
       "  (layers): Sequential(\n",
       "    (0): BatchNorm1d(1200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (1): Dropout(p=0.2)\n",
       "    (2): Linear(in_features=1200, out_features=50, bias=True)\n",
       "    (3): ReLU(inplace)\n",
       "    (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (5): Dropout(p=0.1)\n",
       "    (6): Linear(in_features=50, out_features=2, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 解码器\n",
    "dec = learn_c.model[1]\n",
    "dec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4 id=\"text_classifier_learner\" class=\"doc_header\"><code>text_classifier_learner</code><a href=\"https://github.com/fastai/fastai/blob/master/fastai/text/learner.py#L286\" class=\"source_link\" style=\"float:right\">[source]</a><a class=\"source_link\" data-toggle=\"collapse\" data-target=\"#text_classifier_learner-pytest\" style=\"float:right; padding-right:10px\">[test]</a></h4><blockquote><p><code>text_classifier_learner</code>(<strong><code>data</code></strong>:<a href=\"https://docs.fast.ai/basic_data.html#DataBunch\"><code>DataBunch</code></a>, <strong><code>arch</code></strong>:<code>Callable</code>, <strong><code>bptt</code></strong>:<code>int</code>=<strong><em><code>70</code></em></strong>, <strong><code>max_len</code></strong>:<code>int</code>=<strong><em><code>1400</code></em></strong>, <strong><code>config</code></strong>:<code>dict</code>=<strong><em><code>None</code></em></strong>, <strong><code>pretrained</code></strong>:<code>bool</code>=<strong><em><code>True</code></em></strong>, <strong><code>drop_mult</code></strong>:<code>float</code>=<strong><em><code>1.0</code></em></strong>, <strong><code>lin_ftrs</code></strong>:<code>Collection</code>[<code>int</code>]=<strong><em><code>None</code></em></strong>, <strong><code>ps</code></strong>:<code>Collection</code>[<code>float</code>]=<strong><em><code>None</code></em></strong>, <strong>**<code>learn_kwargs</code></strong>) → <code>TextClassifierLearner</code></p>\n",
       "</blockquote>\n",
       "<div class=\"collapse\" id=\"text_classifier_learner-pytest\"><div class=\"card card-body pytest_card\"><a type=\"button\" data-toggle=\"collapse\" data-target=\"#text_classifier_learner-pytest\" class=\"close\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;</span></a><p>Tests found for <code>text_classifier_learner</code>:</p><ul><li><code>pytest -sv tests/test_text_train.py::test_classifier</code> <a href=\"https://github.com/fastai/fastai/blob/master/tests/test_text_train.py#L100\" class=\"source_link\" style=\"float:right\">[source]</a></li><li><code>pytest -sv tests/test_text_train.py::test_order_preds</code> <a href=\"https://github.com/fastai/fastai/blob/master/tests/test_text_train.py#L139\" class=\"source_link\" style=\"float:right\">[source]</a></li></ul><p>To run tests please refer to this <a href=\"/dev/test.html#quick-guide\">guide</a>.</p></div></div><p>Create a <a href=\"https://docs.fast.ai/basic_train.html#Learner\"><code>Learner</code></a> with a text classifier from <code>data</code> and <code>arch</code>.</p>\n",
       "<p><a href=\"https://docs.fast.ai/text.learner.html#text_classifier_learner\" target=\"_blank\" rel=\"noreferrer noopener\">Show in docs</a></p>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc(text_classifier_learner)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 使用从语言模型学习到的embedding向量-编码器（加载完成后悔freeze）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VNX5+PHPk51ACAHCFiAECEhA1sgmIuKGG2jVClbF2mqxolVbvz/aWr9dvrYqVVuVSnGldcUFRYvgUhERUQKyhCWyQ8IWSNhC9jy/P+YCYxySQGZyZybP+/WaV+aee+69z2GGPLnn3HuuqCrGGGPM6YpwOwBjjDGhzRKJMcaYerFEYowxpl4skRhjjKkXSyTGGGPqxRKJMcaYerFEYowxpl4skRhjjKkXSyTGGGPqJcrtABpC69attUuXLm6HYYwxIWXZsmX7VDW5tnqNIpF06dKFrKwst8MwxpiQIiLb6lLPuraMMcbUiyUSY4wx9WKJxBhjTL1YIjHGGFMvlkiMMcbUiyUSY4wx9WKJxBhjTL1YIjHGmDC093AJU+evZ3P+kYAfyxKJMcaEoY17jjDt003sPlQS8GNZIjHGmDCUW1gMQKek+IAfK6CJRETGiEiOiGwUkSk+1t8nIiucV7aIVIpIy5q2FZGWIvKRiGxwfiYFsg3GGBOKcguPEiHQLjEu4McKWCIRkUhgGnAJkAFMEJEM7zqqOlVV+6tqf+DXwGeqWlDLtlOAT1Q1HfjEWTbGGOMlt7CY9olNiI4MfMdTII8wGNioqptVtQx4DRhXQ/0JwKt12HYcMNN5PxO40u+RG2NMiMstLCYlqUmDHCuQiSQF2OG1nOuUfY+IxANjgLfqsG1bVd0F4Pxsc5J93iYiWSKSlZ+ff9qNMMaYUJRbeJSOYZBIxEeZnqTuFcAXqlpwGtv6pKozVDVTVTOTk2udTt8YY8JGWUUVuw+V0LEBBtohsIkkF+jktdwR2HmSuuM50a1V27Z7RKQ9gPNzr1+iNcaYMLH7YAlVSlickSwF0kUkTURi8CSLOdUriUgicC7wbh23nQNMdN5PrLadMcY0ermFR4GGSyQBe0KiqlaIyGRgPhAJPK+qa0RkkrN+ulP1KuBDVS2qbVtn9UPALBH5CbAduDZQbTDGmFDUkPeQQIAftauqc4G51cqmV1t+EXixLts65fuB8/0ZpzHGhJOGvIcE7M52Y4wJO7kHimnXPK5B7iEBSyTGGBN2cguLG+yKLbBEYowxYSevsLjBBtrBEokxxoSV8soqdh20RGKMMeY0nbiHxLq2jDHGnIYdDXwPCVgiMQ2koKjs+E1SxpjAOXYPSUOekQT0PhLTOBwuKedP76/lv+vzyUxNYlTPZM7tmUxCXDQfrtnNnJU7+XzDPqpUuXpgR351Uc8Gu77dmMYmt7C4Qe8hAUskpp6+3LSfX72xkl0Hizm/V1tW5h5g3prdAERHCuWVSkqLJtx6Tlcqq6qYuXgb/1m1i1tHduVnI7vSNNa+gsb4U27hUdo1jyMmquE6nOx/cQ1yC49y4Gg5fVIS3Q4l6BSXVfLohzk898UWUlvG8+btwxnYOQlV5ds9R/js273kHy7l4t7tGNg5iYgIz4TONw7twsPz1/PEJxt4M2sHj13Xn6FdW7ncGmPCR0PfQwKWSGo0dX4O76/axaRzu3Ln6HTioiPdDilgDpeUs2bnIQZ0bkFs1MnbebSsgpeWbGPGws3sO1LGDUM785tLexEf4/kqiQg92yXQs12Cz+07t4pn2vUD+fHwAu57cxUTnlnCpHO7cc8FPU7rL6iKyireWbGTZxZupqisgjOcY/dom4Aq5B0oJrewmF0HPf3GTWOjaBYTRdPYKBSlrKKKsooqKqqUwWktubxvexLiok85DmOCRV5hMUPSWjboMS2R1OCPY/sQHRnBtE83MX/NHh65pi8DO4fPI+KrqpQlW/bzZlYuH2Tvpri8ktbNYrlxaCo3DO1Mq2axgOeX9eZ9RXy0dg/PLdpCQVEZI7q35hcXpHNWl9P7wmZ2acn7d47gT++v5ekFm1i0YR9TLjmDxCbRxEVHEhsVQXxMJE1jo4iNikDkxCNqVJXSiire+SaPfyzYxPaCo/Rq35z+nVqQs/swn+bkU1l14vE1LZvG0KFFHBEibC84SlFpBUdKKogQISYqgpioCCqrlNnf5PHH99ZyWd/2/DCzEz3aNqNpbFSDTTNhTH1VVB57DknDXbEFIKqn9LyokJSZmalZWVmnvf2CnL385u3V7DpUwi/OT+fuC3r4MbrAKCgq47Wl2xnXP4WUFt//Uq3YcYC7Xv2G7QVHSYiN4or+HRiS1pLZ3+SxICefmKgIzu2RzJ5DJazffZiyiioARvVM5s7R6QxK9V9CnZe9mylvr+LA0XKf6yMjhPiYSCJEKKuoorSikmN5om/HRO4cnc4FvdocTzalFZVszi8iOjKCDi3ijp8t1URVWbHjALOydjBnxU6KyiqPr4uNiqBV0xge/MGZnNfT5wM5jQkKOwqOcs4jn/Lw1Wdy3Vmd670/EVmmqpm11rNEUjeHS8r57exs5qzcyRuThp32X+KBpqrMWbmTP7y3loKiMtokxPL8zWd9Z5xn8aZ93Dozi5bNYvjVRT25uHe773Tbbdx7mOe/2MpnOfl0aR1PRvvmZHRoTt+OLeiW3CwgcRcUlbEq9wClFVWeV3klxeWVHCmt4Gip56eqEuucrcRERtCvUwvOSW/9nbMVfygqreDTnL3sPVTqOXspreC/6/ey+1AJcyaPIK11U78ezxh/+XLTfiY8s4SXfzqEs7u3rvf+LJF48UciAc/4wAWPfkaL+Bjeu3MEkRH+/QVWX7mFR7n/nWwW5OTTv1MLJp3bjT+9v5bCo2VM+9FAzuvZho/X7uHnrywntWU8L/10CG2b22W4dZFbeJQrnlxEckIss39+tl1tZoLSG1k7uO/NVXx23yhSW9X/D566JhLr/D0F8TFR/PayDNbuOsQrX293OxzAcyo7c/FWJj7/NaMf/YyvtxTwwOUZvHX7cMb0acfsnw8nrXVTfjozi9/MXs2kl5ZxRrsEXv/ZMEsip6BjUjxPTBjAxr1H+J+3VtEY/gAzoSe3sBgRaJ/YsGMkAU0kIjJGRHJEZKOITDlJnVEiskJE1ojIZ05ZT6fs2OuQiNztrPu9iOR5rbs0kG2o7tIz2zGsayv+Oj+HgqKyhjz0cTsPFPOPBRu5+PGFnPPIp/zvnDVsLzjKDUNSmX/3SG4ZkXb8bKlN8zhm/WwY56S35pWvtjMwNYmXfzqElk1jXIk9lJ2Tnsx9F5/Bf1bt4tnPt7gdjjHfk1tY3OD3kEAAr9oSkUhgGnAhkAssFZE5qrrWq04L4B/AGFXdLiJtAFQ1B+jvtZ88YLbX7h9X1b8GKvaaiAh/GNebS/7+OX/9MIc/X3Vmgx177updvPzVNhZv2o8qDEpN4v7LejH6jDZ0rWHsomlsFM/elMmnOfmck946rC9jDrRJ53ZlVe4B/vLBOvKPlHLL2Wl2l74JGrmFRxv8ii0I7OW/g4GNqroZQEReA8YBa73qXA+8rarbAVR1r4/9nA9sUtVtAYz1lPRom8DEYV14YfEWrh/cuUFuWJz26Uamzs+hc8t47hqdzg8GppxSH2hUZAQXZrQNYISNg4gw9dp+xEZF8Oznm3nhiy2M65/CbSO70qOt73tnjGkouYXFDG7ge0ggsIkkBdjhtZwLDKlWpwcQLSILgATg76r6r2p1xgOvViubLCI3AVnAL1W10G9R19HdF6YzZ2UeP/jHYhLiooiLjiQuOoJz0pO5/7JeRPnx3oOnF2xi6vwcruzfgUd/2D/oBvkbm2axUfxt/AB+eVFPnlu0hdeX7uCt5bn85pJe3Dqyq9vhmUbKrXtIILBjJL5+21UfoYwCBgGXARcDvxOR4zdpiEgMMBZ4w2ubp4FueLq+dgGP+jy4yG0ikiUiWfn5+afdiJNpHhfNCzcP5sZhqYzp046hXVvRuWU8Ly7eyi9eX0F5ZZVfjjP9s008PG894yyJBJ1OLeP5/djeLJ4ymkv7tOfBuev489x1VFXZQLxpeLsOllBZpT7vGwu0QJ6R5AKdvJY7Ajt91NmnqkVAkYgsBPoB3zrrLwGWq+qeYxt4vxeRZ4D3fR1cVWcAM8Bz+W/9muLbmR0TObPjd7u1ZizcxJ/nrqeisoonJwys16DXjIWbeOiD9VzRrwOPXtvPkkiQSmoawxMTBtCqWYwzdUwpD1/d1+6INw3qYLHnhl43LqQJ5Dd9KZAuImnOmcV4YE61Ou8C54hIlIjE4+n6Wue1fgLVurVEpL3X4lVAtt8jr4fbRnbjf6/IYP6aPfz85WWUVlTWvlE1qsrU+ev589z1XNa3PY//sJ9fu8qM/0VGCH8Y25t7L+zB28vzuPVfWZSUn/pnb8zpOvZ9c+NimoCdkahqhYhMBuYDkcDzqrpGRCY566er6joRmQesAqqAZ1U1G8BJLBcCP6u260dEpD+ebrKtPta77sdnpxEVGcHv3slm9F8/I7NLEv07tWBA5ySSE2IpLa88fgd3l1bxtIg/8RdERWUV97+TzWtLdzBhcCf+78oz7UwkRIgId52fTqtmMfx2djYPvJvNw1f39fud98b4UlLu6U4Pq0QCoKpzgbnVyqZXW54KTPWx7VHge/OLq+qNfg4zIG4cmkpys1je+SaPJZv38+6K6r16HtGRwrk9khnbP4WR6a35nzdX8eHaPdw5ujv3XtjDfgmFoB8NSWX3wRKe/O9G+nVqwY+GpLodkmkEip0zkibhlkgauzF92jGmTzsAdh0sZuWOAxwqriA2OoLYqAgiIyJYurWAOSt28vG6vRzLGb+/IoObz05zMXJTX3df0IPVeQf5/Zw19GrfPKxmjTbB6UTXVsN3g1siaSDtE5v4nLbgwoy2TBlzBl9vLeCjtXsY2rWV3e8RBiIjhL9d15+xT33B7S8t4707R9AmwW5cNIHj5hiJjeAGgYgIYWjXVvzu8gxLImGkRXwM028YxMHicia//I0NvpuAKnEe9RDrwhmJJRJjAiijQ3MevrovS7cVcO30L9l5oNjtkEyYKilzb4zEEokxATaufwrP3JjJln1FjH1qEVlbC9wOyYQh69oyJsxdkNGW2T8fTrPYKCY8s4RXg+QxBCZ8lFRUEhkhrtwIa4nEmAaS3jaBd+8YwdCurfj126uZl73b7ZBMGCkpr3KlWwsskRjToBLjo3lu4ln0SWnOlLdXsedQidshmTBRXF7pyqW/YInEmAYXExXB38cPoKS8kl/OWmmTPBq/KCmvJDbKzkiMaTS6JTfjgct7s2jjPp7/wp62aOqvtLzKzkiMaWwmDO7EhRlteWReDmt2HnQ7HBPiissraRJjZyTGNCoiwsNX9yUxPppfvLbi+DTgxpyOkvJK4qxry5jGp2XTGP5+XX+27S/i5he+5khphdshmRBVUl7pyj0kYInEGNcN796aJycMZFXuQW55cSnFZTaVijl1JTZGYkzjNqZPOx6/rj9ZWwvsoVjmtNgZiTGGsf068Mg1/Vi0cR+TX1mOql0WbOrOEokxBoBrBnXkt5f24uN1e/l8wz63wzEhpKQiTLu2RGSMiOSIyEYRmXKSOqNEZIWIrBGRz7zKt4rIamddlld5SxH5SEQ2OD/tiUEmrNw0PJXkhFieXWT3l5i6KymvDL8pUkQkEpgGXAJkABNEJKNanRbAP4CxqtobuLbabs5T1f6qmulVNgX4RFXTgU+cZWPCRmxUJBOHpbLw23xydh92OxwTAlTVmSIlzBIJMBjYqKqbVbUMeA0YV63O9cDbqrodQFX31mG/44CZzvuZwJV+iteYoPGjIanERUfwvJ2VmDooq6xC1Z0p5CGwiSQF2OG1nOuUeesBJInIAhFZJiI3ea1T4EOn/Dav8raqugvA+dnG18FF5DYRyRKRrPz8/Ho3xpiGlNQ0hmsGdWT2N3nkHy51OxwT5ErKnacjRoXfGIn4KKt+GUoUMAi4DLgY+J2I9HDWna2qA/F0jd0hIiNP5eCqOkNVM1U1Mzk5+RRDN8Z9t5ydRlllFf9ess3tUEyQO3a5eDhOkZILdPJa7gjs9FFnnqoWqeo+YCHQD0BVdzo/9wKz8XSVAewRkfYAzs+6dIcZE3K6Jjfjgl5teGnJNruvxNTo+NMRw3CKlKVAuoikiUgMMB6YU63Ou8A5IhIlIvHAEGCdiDQVkQQAEWkKXARkO9vMASY67yc6+zAmLP1kRFcKisqY/U2e26GYIHasayvsxkhUtQKYDMwH1gGzVHWNiEwSkUlOnXXAPGAV8DXwrKpmA22BRSKy0in/j6rOc3b9EHChiGwALnSWjQlLQ7u2pE9Kc579fLM9t8Sc1InntbszRhIVyJ2r6lxgbrWy6dWWpwJTq5Vtxuni8rHP/cD5/o3UmOAkItw2sht3vfoN89bs5tIz27sdkglCxcfGSMLtjMQY4x+Xndmerq2b8sQnG+ysxPh07Iwk1hKJMcaXyAhh8ujurN99mI/X7XE7HBOEToyRhN/lv8YYPxnbrwOpreJ54r8bbDJH8z2lFda1ZYypRVRkBHeM6k523iE+zbEr3s13HXuGTdhdtWWM8a+rBqaQ0qIJT3yy0c5KzHecuGrLEokxpgbRkRHccV53Vuw4YFPMm+8oqbAxEmNMHV09KIUOiXH87eNv7azEHHe8aysM72w3xvhZbFQkd56fzvLtB3hnhd3tbjxKKiqJiYogIsLXFIeBZ4nEmBBzXWYn+nVqwYP/WcfB4nK3wzFBoLS8ijiXZv4FSyTGhJyICOHBK/tQUFTGox/muB2OCQJuPq8dLJEYE5L6pCRy07Au/HvJNlbnHnQ7HOOy4vJK16aQB0skxoSsey/qQetmsfz2ndVU2tQpjVpJeaVrA+1gicSYkNU8Lpr7L+vFqtyDvPL1drfDMS4qKa9y7dJfsERiTEgb268DZ3dvxdR56ykoKnM7HOOSkvJK1yZsBEskxoQ0EeH3V/SmqKySxz/61u1wjEtKyitdm2cLLJEYE/LS2yZww5DOvPzVNnJ2H3Y7HOOCsO7aEpExIpIjIhtFZMpJ6owSkRUiskZEPnPKOonIpyKyzin/hVf934tInrPNChG5NJBtMCYU3H1BDxLiovm//6y1O94boZKKML38V0QigWnAJUAGMEFEMqrVaQH8Axirqr2Ba51VFcAvVbUXMBS4o9q2j6tqf+f1nScwGtMYJTWN4e4L0vl8wz4+WWezAzc2xWXh27U1GNioqptVtQx4DRhXrc71wNuquh1AVfc6P3ep6nLn/WE8z3xPCWCsxoS8G4am0r1NMx6cu44yZxI/0ziE8w2JKcAOr+Vcvp8MegBJIrJARJaJyE3VdyIiXYABwFdexZNFZJWIPC8iSf4N25jQFB0Zwf2X9WLLviJmLt7qdjimAZVUVBEbpmMkvmYPq955GwUMAi4DLgZ+JyI9ju9ApBnwFnC3qh5yip8GugH9gV3Aoz4PLnKbiGSJSFZ+fn69GmJMqBjVsw2jeibzxCcb2Hek1O1wTAOoqlLKKqrC9obEXKCT13JHYKePOvNUtUhV9wELgX4AIhKNJ4m8rKpvH9tAVfeoaqWqVgHP4OlC+x5VnaGqmaqamZyc7LdGGRPs7r8sg+LySh790C4HbgxKjj1mN0ynSFkKpItImojEAOOBOdXqvAucIyJRIhIPDAHWiYgAzwHrVPUx7w1EpL3X4lVAdsBaYEwI6t6mGTcOS+X1pdtZu/NQ7RuYkFZS7jzUKhxn/1XVCmAyMB/PYPksVV0jIpNEZJJTZx0wD1gFfA08q6rZwNnAjcBoH5f5PiIiq0VkFXAecE+g2mBMqLr7/B4kNonmj++vscuBw5zbj9kFzxhFwDiX5s6tVja92vJUYGq1skX4HmNBVW/0c5jGhJ3E+Gjuvagnv3snm/lrdjOmT/vaNzIhKRgSid3ZbkyYmnBWJ3q2TeDBueuO/7Ix4afYEokxJlCiIiN44IoMdhQU89yiLW6HYwLk+BhJmF7+a4xx2dndWzOmdzv+/vEGlm8vdDscEwCloXJGIiLdRCTWeT9KRO5ypjcxxgS5h64+k3aJcUz69zL2HipxOxzjZ8e6tkJhipS3gEoR6Y7nstw04JWARWWM8ZsW8THMuGkQh0squP3l5TZ9Spg50bUV/Imkyrmc9yrgb6p6D2CXgRgTIs5o15yp1/Zl2bZC/vDeGrfDMX504qot90Yq6nr5b7mITAAmAlc4ZdGBCckYEwiX9+1Adt4hpn+2ib4dE7nurM5uh2T84Nid7aFwRvJjYBjwoKpuEZE04KXAhWWMCYT7Lu7J0K4teXhejl0SHCaKy0IkkajqWlW9S1VfdWbbTVDVhwIcmzHGzyIjhLsv6EFBURmzv8lzOxzjB6UVIXL5rzPNe3MRaQmsBF4Qkcdq284YE3yGpLWkT0pznv18M1VVNn1KqCspr0QEYiKDPJEAic407j8AXlDVQcAFgQvLGBMoIsKt53RlU34Rn31rj1gIdSXllcRFReKZ69YddU0kUc6suz8E3g9gPMaYBnDpme1pnxjHM59vdjsUU0/F5ZWuTiEPdU8kf8Qzi+8mVV0qIl2BDYELyxgTSNGREdw8vAuLN+1nzc6Dbodj6qGkvMrVKeSh7oPtb6hqX1W93VnerKpXBzY0Y0wgjR/cmaYxkTz3uc3DFcrcfl471H2wvaOIzBaRvSKyR0TeEpGOgQ7OGBM4iU2i+eFZnZizcie7D9rUKaGqpLwqNBIJ8AKepxt2AFKA95wyY0wIu+XsNKpUefyjb+0KrhDlOSMJga4tIFlVX1DVCuf1ImAPQjcmxHVqGc/E4V14PWsHE1/4mv1HSt0OyZyikOnaAvaJyA0iEum8bgD217aRiIwRkRwR2SgiU05SZ5TzKN01IvJZbduKSEsR+UhENjg/k+rYBmOMDw9cnsGfrzqTr7YUcNkTi1i6tcDtkMwpKKkInURyC55Lf3cDu4Br8EybclIiEglMAy4BMoAJIpJRrU4L4B/AWFXtDVxbh22nAJ+oajrwibNsjDlNIsL1Qzoz++fDiYuOYPyMJby9PNftsEwdFZdVujqFPNT9qq3tqjpWVZNVtY2qXonn5sSaDAY2Old4lQGvAeOq1bkeeFtVtzvH2VuHbccBM533M4Er69IGY0zNendI5L07R9CnQ3Me//hbVG3MJBSUlFcRGyJjJL7cW8v6FGCH13KuU+atB5DkTMGyTERuqsO2bVV1F4Dzs42vg4vIbSKSJSJZ+fl2964xdZEQF81Nw7qwo6DYnqgYIkpDqGvLl9rux/e1vvqfOFHAIOAy4GLgdyLSo47b1khVZ6hqpqpmJifbdQHG1NXFfdoRFx1hkzqGCM8NiaGbSGr7xZ4LdPJa7gjs9FFnnqoWqeo+YCHQr5Zt9zjTteD83Isxxm+axUZxUUY73l+1y56mGAI8U6QEcdeWiBwWkUM+Xofx3FNSk6VAuoikiUgMMB7PvSje3gXOEZEoEYkHhgDratl2Dp4HbOH8fLeObTXG1NFVA1I4cLTcJnUMcuWVVVRWqetnJDU+IVFVE053x6paISKT8czRFQk8r6prRGSSs366qq4TkXnAKqAKeFZVswF8bevs+iFgloj8BNiOc6WXMcZ/RqS3plXTGN75Jo8LM9q6HY45iROP2Q3iRFJfqjoXmFutbHq15anA1Lps65TvB873b6TGGG/RkRFc0a8Dr3y9nUMl5TSPsydrB6OScuehViEy+68xppG5ckAKZRVVzFu92+1QzEkcPyMJhdl/jTGNT7+OiaS1bmpXbwWxYOnaskRijPFJRLiyfwpLtuxn54Fit8MxPhzv2rJEYowJVlcO6IAqdlYSpIqdM5KQmCLFGNM4pbZqyjnprZmxcDOFRWVuh2OqOdG1ZWMkxpggdv9lGRwpreCxj751OxRTjY2RGGNCQs92CdwwpDMvf7WNdbsOuR2O8VJScWyMxM5IjDFB7p4Le5DYJJo/vLfGZgUOIiVldkZijAkRLeJjuPeinizZXMAH2XZfSbAoqbBEYowJIdcP7swZ7RJ48D/rjvfNG3fZGIkxJqRERgi/H9ubvAPFPLNws9vhGLzuI7E7240xoWJo11ZclNHWLgcOEsXllURHClGRlkiMMSHkVxf35EhZBU9/tsntUBq9kvJK16eQB0skxphT1KNtAlcNSGHm4q3sPljidjiNmud57ZZIjDEh6J4LelClyt8/2eB2KI1aSRA8HREskRhjTkOnlvFcP7gzs7J2sGVfkdvhNFqNomtLRMaISI6IbBSRKT7WjxKRgyKywnk94JT39Cpb4Tze925n3e9FJM9r3aWBbIMxxrfJo9OJiYywqVNcVFJe6fqlvxDAJySKSCQwDbgQyAWWisgcVV1brernqnq5d4Gq5gD9vfaTB8z2qvK4qv41ULEbY2qXnBDLLSO6MO3TTUw6tyu9OyS6HVKjU1Je5fr0KBDYM5LBwEZV3ayqZcBrwLjT2M/5wCZV3ebX6Iwx9XbbyG40j4viyU82uh1Ko1QcJGckgUwkKcAOr+Vcp6y6YSKyUkQ+EJHePtaPB16tVjZZRFaJyPMikuSneI0xpyixSTQTh3dh/trdbNx7xO1wGp1g6doKZCIRH2XVZ3tbDqSqaj/gSeCd7+xAJAYYC7zhVfw00A1P19cu4FGfBxe5TUSyRCQrPz//9FpgjKnVzcO7EBsVwT/tvpIGV1pRFfaJJBfo5LXcEdjpXUFVD6nqEef9XCBaRFp7VbkEWK6qe7y22aOqlapaBTyDpwvte1R1hqpmqmpmcnKyf1pkjPmeVs1iGX9WZ95ZkWeP5A2gqirl0Q9zvnPm57lqK7zHSJYC6SKS5pxZjAfmeFcQkXYiIs77wU48+72qTKBat5aItPdavArIDkDsxphT8NNz0lCFZz/f4nYoYWvd7kM8+d+N3PnqN5RXeubYKi6vpElMGJ+RqGoFMBmYD6wDZqnqGhGZJCKTnGrXANkishJ4AhivzsMORCQezxVfb1fb9SMislpEVgHnAfcEqg3GmLrpmBTP2P4dePXr7TYHV4As31YIwLpdh5i+wNMThnxEAAAWC0lEQVSNGCxjJAG7/BeOd1fNrVY23ev9U8BTJ9n2KNDKR/mNfg7TGOMHk87txtvL83hx8VbuubCH2+GEnWXbCmmTEMvgtJY88d8NXNS7nefy3zDv2jLGNCI92iZwYUZbZn65laLSCrfDCTvLthcyKDWJP4ztTUJcNPe8vgKAuHDu2jLGND63j+rGgaPl/OK1Few9bBM6+sveQyXsKChmUGoSrZrF8oexvVm76xBA+E+RYoxpXAZ2TuK3l/Zi4YZ8LnxsIW8uy7VnvPvBMmd8ZGCq57a5y/u256KMtoD7T0cESyTGGD+7dWRX5t51DultmvGrN1Yy8YWlLMjZy9Ey6+46Xcu2FRITFUEfZxoaEeH/ruzDkLSW9O3o/tQ0AR1sN8Y0Tt3bNGPWz4bx0lfbePiD9Sz8Np/oSGFApyRGpLfmlhFpNIu1Xz91tWx7If06JhLjNbDepnkcr/9smItRnWCfpDEmICIihJuGdeHaQZ1YurWALzbtY/HG/Tz+8bfk7DnMUxMG4NxGZmpQUl5Jdt5BbhmR5nYoJ2WJxBgTUE1iIhnZI5mRPTwzTEz7dCNT5+dwUUZbxvX3Nf2e8Zadd5DySmVQ5+CdVtDGSIwxDepnI7syKDWJ372TbVOq1EH1gfZgZInEGNOgoiIjeOyH/aioUu57cyVVVXZVV02WbSskrXVTWjeLdTuUk7JEYoxpcKmtmnL/ZRl8sXE/M7/c6nY4QUtVWbatkIFB3K0FNkZijHHJhMGd+HjdHv7ywXo+WL2bssoqyiuriIoQnpgwgNRWTd0O0XXb9h9lf1EZg4K4WwvsjMQY4xIR4aGrz2R0zzZERgjNm0TTPjGOdbsO88IXW90OLygcGx8J9kRiZyTGGNe0SYhj+o2DvlN2z+sreGtZLvdd3JOmjfxek2XbC0mIjSK9TTO3Q6mRnZEYY4LKDUNTOVxawbsrdtZeOcwt31bIgNQkIiKC+34bSyTGmKAysHMLMto3519fbm3U83QdLiknZ8/hoL5/5BhLJMaYoCIi3DgslfW7D7N8e6Hb4bhmdd5BVKFfJ/fn0qqNJRJjTNAZ178DCbFR/PvLbW6H4ppVuQcB6NexhcuR1C6giURExohIjohsFJEpPtaPEpGDIrLCeT3gtW6r80jdFSKS5VXeUkQ+EpENzs/gP+8zxpyS+Jgorh7Ukbmrd7P/SKnb4bhiVe4BOreMJ6lpjNuh1CpgiUREIoFpwCVABjBBRDJ8VP1cVfs7rz9WW3eeU57pVTYF+ERV04FPnGVjTJi5YWhnyiqrmJWV63Yorli542BQTBFfF4E8IxkMbFTVzapaBrwGjPPDfscBM533M4Er/bBPY0yQ6d4mgWFdW/HyV9uobGTTqOw7UkregeKQ6NaCwCaSFGCH13KuU1bdMBFZKSIfiEhvr3IFPhSRZSJym1d5W1XdBeD8bOPr4CJym4hkiUhWfn5+/VpijHHFTcNSyS0s5k/vr21UV3Ctyj0AQL9Olkh8Xfhc/ZuwHEhV1X7Ak8A7XuvOVtWBeLrG7hCRkadycFWdoaqZqpqZnJx8KpsaY4LEmD7t+MmINF5cvJXfvZvdaCZ4XLnjIBECfVKaux1KnQQykeQCnbyWOwLfucNIVQ+p6hHn/VwgWkRaO8s7nZ97gdl4usoA9ohIewDn594AtsEY4yIR4f7LejHp3G68tGQ7v357daPo5lqZe4D0NgnEx4TGnf2BTCRLgXQRSRORGGA8MMe7goi0E+cRaSIy2Ilnv4g0FZEEp7wpcBGQ7Ww2B5jovJ8IvBvANhhjXCYi/L8xPblrdHdez9rBfW+sDOtkoqqsyg2dgXYI4FxbqlohIpOB+UAk8LyqrhGRSc766cA1wO0iUgEUA+NVVUWkLTDbyTFRwCuqOs/Z9UPALBH5CbAduDZQbTDGBAcR4d6LenqeZfLRt3RNbsrk0eluhxUQuYXFFBSVhcz4CAR40kanu2putbLpXu+fAp7ysd1moN9J9rkfON+/kRpjQsGdo7uzKf8Ij330LYPTWjE4raXbIfldKN2IeIzd2W6MCRkiwoNXnUnnlvHc9eo3FBSVuR2S363MPUBMZAQ92yW4HUqdWSIxxoSUZrFRPHX9QAqKyrh31oqwu5Jr5Y4D9OrQnJio0Pn1HDqRGmOMo09KIvdf3osFOfk8u2iz2+H4TWWVkp13kP4hNNAOlkiMMSHqxqGpXNKnHY/Myzl+A1+o25x/hKKySvqG0PgIWCIxxoQoz6N6+5KcEMs9r6+guKzS7ZDqbcWOY3e02xmJMcY0iMQm0fz12n5syi/i4Xnr3Q6n3lblHqRZbBRdWwf3o3Wrs0RijAlpZ3dvzc3Du/Di4q18viG059VblXuAM1MSg/7RutVZIjHGhLwpl5xBt+Sm3PfGKg4eLXc7nNNSXFbJul2H6Rti3VpgicQYEwbioiP523UD2HeklPvfzQ7JmYKXbi2grLKK4d1aux3KKbNEYowJC2d2TOTuC9J5b+VOJr/yDUWlFW6HdEoWbdxHTGQEg7uE3t36oTG1pDHG1MEd53UnOjKCh+et59s9h/nnjYPomhwaA9eLNuxjUGoSTWIi3Q7llNkZiTEmbIgIPzu3G//+yRD2HSll3FNf8PHaPW6HVat9R0pZu+sQI9JDr1sLLJEYY8LQ2d1b896dI+jSuik//VcW0z7dGNTjJos37QdgRHdLJMYYEzQ6JsXzxqRhXNm/A1Pn53D36ysoKQ/OmxYXbcgnsUk0fVJC74otsDESY0wYi4uO5PHr+pPeNoGp83PYuv8oz9w4iDbN49wO7ThVZdGGfQzv1orIELt/5Bg7IzHGhDUR4Y7zuvPPGwexYc9hxk37gm/3HHY7rOO27Cti58GSkB0fgQAnEhEZIyI5IrJRRKb4WD9KRA6KyArn9YBT3klEPhWRdSKyRkR+4bXN70Ukz2ubSwPZBmNMeLi4dzvemDSMyirlmqcXs3RrgdshAZ7LfiF0x0cggIlERCKBacAlQAYwQUQyfFT9XFX7O68/OmUVwC9VtRcwFLij2raPe20z93t7NMYYH3p3SOSt24fTulksNzz7FR+u2e12SCzasI9OLZuQ2qqp26GctkCekQwGNqrqZlUtA14DxtVlQ1XdparLnfeHgXVASsAiNcY0Gp1aegbhz2jfnEkvLePVr7e7FktFZRVfbtof0mcjENhEkgLs8FrOxXcyGCYiK0XkAxHpXX2liHQBBgBfeRVPFpFVIvK8iCT5MWZjTCPQqlksr946hHPSk/n126t55St3ksmqvIMcLq1gRPdkV47vL4FMJL4uP6h+IfdyIFVV+wFPAu98ZwcizYC3gLtV9ZBT/DTQDegP7AIe9XlwkdtEJEtEsvLzQ3tGUGOM/8XHRDHjpkGc1zOZ376zmjeX5TZ4DIs27EMEhndr1eDH9qdAJpJcoJPXckdgp3cFVT2kqkec93OBaBFpDSAi0XiSyMuq+rbXNntUtVJVq4Bn8HShfY+qzlDVTFXNTE4O7WxvjAmM2KhInr5hEGd3a83/vLmSd1fkNejxF23YR58OiSQ1jWnQ4/pbIBPJUiBdRNJEJAYYD8zxriAi7UREnPeDnXj2O2XPAetU9bFq27T3WrwKyA5gG4wxYS4uOpJnbsrkrC4tuXfWSuZl72qQ476+dDtfby3gooy2DXK8QApYIlHVCmAyMB/PYPksVV0jIpNEZJJT7RogW0RWAk8A49Uzj8HZwI3AaB+X+T4iIqtFZBVwHnBPoNpgjGkcmsRE8vzNZ9GvYyL3vL6SvAPFAT3eZ9/m85vZ2YzskcykUd0CeqyGIME8/4y/ZGZmalZWltthGGOCXG7hUS547DNGn9GGf/xoUECOkZ13kOv++SWprZoya9IwmsUG7wQjIrJMVTNrq2d3thtjjKNjUjx3jOrO3NW7WbRhn9/3n3egmFteXEpik2he+PFZQZ1EToUlEmOM8XLryK6ktorngTnZlFVU+W2/uw4Wc9NzX1FcVskLPx5M2yCa76u+LJEYY4yXuOhI/veKDDbnF/H8F1v8ss9N+Ue45ukv2XOolGcnZtKzXYJf9hssLJEYY0w1o89oywW92vDEJxvYdbB+A++rcg9w7fQvKSmv5LXbhjKka2jfM+KLJRJjjPHhgct7U1GlPPDuGiqrTu+ipC827mPCjCXEx0Ty5u3DQ/Z5I7WxRGKMMT50bhXP/1zck4/W7uHeWSuoqDy18ZJ3V+Rx8wtf0zEpnrduH05a69CdlLE24XHJgDHGBMBPz+lKWWUVj8zLobS8iicmDCAmyvP3d1FpBe+t9EzWMbZ/B+JjPL9OVZV/LtzMQx+sZ0haS2bclElik2jX2tAQLJEYY0wNfj6qO7FRkfzp/bVMemkZv7qoJ7OydvDWslwOl1YA8Oe565gwpDM3DEnluUVbeHHxVi7v255Hf9iP2KhIl1sQeJZIjDGmFj8ZkUZsVAT3v5PNf9fvJTpSuOzM9tw4rAuqygtfbOWZhZv552ebAbj1nDR+fUkvIkL00bmnyhKJMcbUwQ1DU2nVNIYt+4u4dlAnkhNij6/L7NKSvAPFvPLVNjolxTN+cGcXI214lkiMMaaOLjmz/UnXpbRown0Xn9GA0QQPu2rLGGNMvVgiMcYYUy+WSIwxxtSLJRJjjDH1YonEGGNMvVgiMcYYUy+WSIwxxtSLJRJjjDH10iie2S4i+cC2asWJwMFayryXa3vfGjjdZ3P6iqWudWprR13aFOxtqL7s5mdR0/qaYq5pOZS+T9WXa/o+gTufRSh9n2qqEwyfRaqqJtdaS1Ub5QuYUVuZ93Jt74Esf8ZS1zq1taMubQr2NgTTZ1HT+ppirmk5lL5Pdfz39y5r8M8ilL5PofJZ1PZqzF1b79Wh7L1TfO/PWOpap7Z21KVNwd6G6stufhY1ra8p5pqWQ+n7VH05UN+nuuwnHL5PNdUJps+iRo2ia6shiEiWqma6HUd9hEMbIDzaEQ5tgPBoRzi0AQLbjsZ8RuJvM9wOwA/CoQ0QHu0IhzZAeLQjHNoAAWyHnZEYY4ypFzsjMcYYUy+WSHwQkedFZK+IZJ/GtoNEZLWIbBSRJ0REvNb9UETWisgaEXnFv1F/Lw6/t0FEbhaRfBFZ4bx+6v/IvxdLQD4LZ/01IqIiEtD+7wB9FpOc8hUiskhEMvwf+XfiCEQb7nX+P6wSkU9EJNX/kX8vlkC0Y6SILBeRChG5xv9RHz/+acd+kv1NFJENzmuiV3maiHzllL8uIjG17ixQl4OF8gsYCQwEsk9j26+BYYAAHwCXOOXpwDdAkrPcJgTbcDPwVKh/Fs66BGAhsATIDLU2AM296owF5oVgG84D4p33twOvh+L3CegC9AX+BVwTbLEDC4Au1cpaApudn0nO+2O/m2YB453304HbazuGnZH4oKoLgQLvMhHpJiLzRGSZiHwuIt97FJqItMfzH/xL9XwK/wKudFbfCkxT1ULnGHtDsA0NLoDt+BPwCFASwPCBwLRBVQ95VW0KBHSwM0Bt+FRVjzpVlwAdA9kG55iBaMdWVV0FVAVj7CdxMfCRqhY4v5M+AsY4Z1mjgTedejOpw/9/SyR1NwO4U1UHAb8C/uGjTgqQ67Wc65QB9AB6iMgXIrJERMYENFrf6tsGgKudrog3RaRT4EKtUb3aISIDgE6q+n6gA61BvT8LEblDRDbhSYh3BTDWk/HH9+mYn+D5K98N/mxHQ6tL7L6kADu8lo+1pxVwQFUrqpXXyJ7ZXgci0gwYDrzh1c0e66uqj7JjfylG4eneGoXnL6/PRaSPqh7wb7S++akN7wGvqmqpiEzC89fKaH/HWpP6tkNEIoDH8XTTucJPnwWqOg2YJiLXA/cDE33UDwh/tcHZ1w1AJnCuP2OsC3+2o6HVFLuI/Bj4hVPWHZgrImXAFlW9ipO357TaaYmkbiLwZOn+3oUiEgkscxbnAE/z3dPzjsBO530usERVy4EtIpKDJ7EsDWTgXurdBlXd71X+DPBwwKI9ufq2IwHoAyxw/vO1A+aIyFhVzQpw7Mf44/vk7TWnbkPySxtE5ALgt8C5qloa0Ih98/dn0ZB8xg6gqi8ALwCIyALgZlXd6lUlF88ftcd0xDOWsg9oISJRzllJ3doZqIGhUH/hGUDL9lpeDFzrvBeg30m2WwoM5cSA3KVO+RhgpvO+NZ7TylYh1ob2XnWuwpMYQ+6zqFZnAQEebA/QZ5HuVecKAjiPUgDbMADY5N2WUP4+AS8SwMH2042dkw+2b8Ez0J7kvG/prHuD7w62/7zWuBryAwyVF/AqsAsox5O5fwKkAfOAlcBa4IGTbJsJZDv/QZ7ixE2fAjzmbLv62AcVYm34C7DG2f5T4IxQ/Cyq1VlA4K/aCsRn8Xfns1jhfBa9Q7ANHwN7nDasAOaE4vcJOMvZVxGwH1gTTLHjI5E45bcAG53Xj73Ku+K5Qm0jnqQSW1tsdme7McaYerGrtowxxtSLJRJjjDH1YonEGGNMvVgiMcYYUy+WSIwxxtSLJRLTKInIkQY+3rP+mqFXRCrFM+tvtoi8JyItaqnfQkR+7o9jG+OLXf5rGiUROaKqzfy4v2N3Agecd+wiMhP4VlUfrKF+F+B9Ve3TEPGZxsfOSIxxiEiyiLwlIkud19lO+WARWSwi3zg/ezrlN4vIGyLyHvChiIwSkQXOhJbrReRlZzZVnPJM5/0REXlQRFY6E3i2dcq7OctLReSPdTxr+pITk1E2E89zPZaL57kZ45w6DwHdnLOYqU7d+5zjrBKRP/jxn9E0QpZIjDnh78DjqnoWcDXwrFO+HhipqgOAB4A/e20zDJioqscmrxwA3A1k4LlD+Gwfx2mKZ3qZfnieiXKr1/H/7hy/1vmNnPmgzsczFxR4psS/SlUH4nnWx6NOIpsCbFLV/qp6n4hchGeet8FAf2CQiIys7XjGnIxN2mjMCRcAGV4zqTYXkQQgEZgpIul4ZkKN9trmI1X1fkbE16qaCyAiK/DMjbSo2nHKgGNT2C8DLnTeD+PEsx9eAf56kjibeO17GZ5nSYBnGp4/O0mhCs+ZSlsf21/kvL5xlpvhSSwLT3I8Y2pkicSYEyKAYapa7F0oIk8Cn6rqVc54wwKv1UXV9uE9g20lvv+PleuJwcmT1alJsar2F5FEPAnpDuAJ4EdAMjBIVctFZCsQ52N7Af6iqv88xeMa45N1bRlzwofA5GMLInJseu5EIM95f3MAj78ET5cawPjaKqvqQTwPtPqViETjiXOvk0TOA449A/0wnunzj5kP3OI8zwIRSRGRNn5qg2mELJGYxipeRHK9Xvfi+aWc6QxArwUmOXUfAf4iIl8AkQGM6W7gXhH5GmgPHKxtA1X9Bs/Mr+OBl/HEn4Xn7GS9U2c/8IVzufBUVf0QT9fZlyKyGs9jVRN8HsCYOrDLf40JEiISj6fbSkVkPDBBVcfVtp0xbrMxEmOCxyDgKedKqwN4nhdhTNCzMxJjjDH1YmMkxhhj6sUSiTHGmHqxRGKMMaZeLJEYY4ypF0skxhhj6sUSiTHGmHr5/8f0QznSf3WzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.lr_find()\n",
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 12:26 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.360218</td>\n",
       "      <td>0.269857</td>\n",
       "      <td>0.888720</td>\n",
       "      <td>04:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.343186</td>\n",
       "      <td>0.255186</td>\n",
       "      <td>0.898080</td>\n",
       "      <td>04:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.325663</td>\n",
       "      <td>0.251580</td>\n",
       "      <td>0.898920</td>\n",
       "      <td>03:59</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(3, 5e-3, moms=(0.8, 0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('fine1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "549"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c = None\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load('fine1');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, 'imdb_textlist_class', bs=BATCH//2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.data = data_clas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 1:06:36 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.292029</td>\n",
       "      <td>0.212996</td>\n",
       "      <td>0.919720</td>\n",
       "      <td>21:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.272948</td>\n",
       "      <td>0.196127</td>\n",
       "      <td>0.927080</td>\n",
       "      <td>22:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.199419</td>\n",
       "      <td>0.175645</td>\n",
       "      <td>0.933280</td>\n",
       "      <td>22:20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(3, 5e-4, moms=(0.8, 0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save('fine2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Category pos, tensor(1), tensor([0.0021, 0.9979]))"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.predict(\"I really loved that movie, it was awesome!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Category neg, tensor(0), tensor([0.9127, 0.0873]))"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.predict(\"I really hated that movie!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 逐层 unfreeze 会更有效"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "374"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c = None\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}