47-text-classification-Emojify.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://akhavanpour.ir/notebook/images/srttu.gif\" alt=\"SRTTU\" style=\"width: 150px;\"/>\n",
    "\n",
    "[![Azure Notebooks](https://notebooks.azure.com/launch.png)](https://notebooks.azure.com/import/gh/Alireza-Akhavan/class.vision)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">طبقه بندی متن</div>\n",
    "# Emojify! \n",
    "\n",
    "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
    "کدها  با تغییرات برگرفته از کورس Sequence Models پروفسور Andrew NG است.\n",
    "</div>\n",
    "\n",
    "[https://www.coursera.org/learn/nlp-sequence-models](https://www.coursera.org/learn/nlp-sequence-models)\n",
    "\n",
    "\n",
    "\n",
    "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">در این نوت بوک میخواهید برای جملات دلخواه یک emoji مرتبط به صورت خودکار بگذاریم!\n",
    "در واقع یک طبقه بندی ساده 5 کلاسه است که هر جمله را به یک ایموجی نسبت می‌دهد.\n",
    "</div>\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">لود کتابخانه‌های مورد استفاده</div>\n",
    "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
    "برای اجرای این نوت‌بوک باید  کتابخانه ی emoji را نصب کنید.\n",
    "بدین منظور به اینترنت متصل شود و در ترمینال  دستورات زیر را بنویسید:\n",
    "</div>\n",
    "<p>pip install emoji </p>\n",
    "\n",
    "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
    "میتوانید به جای pip از کلمه ی conda استفاده کنید. (اگر از آناکوندا استفاده میکنید.)\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import keras\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Baseline model: Emojifier-V1\n",
    "\n",
    "### 1.1 - Dataset EMOJISET\n",
    "\n",
    "Let's start by building a simple baseline classifier. \n",
    "\n",
    "You have a tiny dataset (X, Y) where:\n",
    "- X contains 127 sentences (strings)\n",
    "- Y contains a integer label between 0 and 4 corresponding to an emoji for each sentence\n",
    "\n",
    "<img src=\"images/data_set.png\" style=\"width:700px;height:300px;\">\n",
    "<caption><center> **Figure 1**: EMOJISET - a classification problem with 5 classes. A few examples of sentences are given here. </center></caption>\n",
    "\n",
    "Let's load the dataset using the code below. We split the dataset between training (127 examples) and testing (56 examples)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\"> تابع کمکی برای خواند مجموعه داده\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "def read_csv(filename):\n",
    "    phrase = []\n",
    "    emoji = []\n",
    "\n",
    "    with open (filename) as csvDataFile:\n",
    "        csvReader = csv.reader(csvDataFile)\n",
    "\n",
    "        for row in csvReader:\n",
    "            phrase.append(row[0])\n",
    "            emoji.append(row[1])\n",
    "\n",
    "    X = np.asarray(phrase)\n",
    "    Y = np.asarray(emoji, dtype=int)\n",
    "\n",
    "    return X, Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\"> مجموعه داده‌ی Emoji\n",
    "</div>\n",
    "\n",
    "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">مجموعه داده را میتوانید از مسیر زیر دانلود کنید.</div>\n",
    "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">  \n",
    "\n",
    "[http://dataset.class.vision/NLP/emoji.zip](http://dataset.class.vision/NLP/emoji.zip)\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, Y_train = read_csv('D:/dataset/NLP/emoji/train_emoji.csv')\n",
    "X_test, Y_test = read_csv('D:/dataset/NLP/emoji/tesss.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">طول بزرگترین جمله</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "maxLen = len(max(X_train, key=len).split())\n",
    "maxLen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\"> تابع کمکی تبدیل label ها به Emoji\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import emoji \n",
    "\n",
    "emoji_dictionary = {\"0\": \"\\u2764\\uFE0F\",    # :heart: prints a black instead of red heart depending on the font\n",
    "                    \"1\": \":baseball:\",\n",
    "                    \"2\": \":smile:\",\n",
    "                    \"3\": \":disappointed:\",\n",
    "                    \"4\": \":fork_and_knife:\"}\n",
    "\n",
    "def label_to_emoji(label):\n",
    "    \"\"\"\n",
    "    Converts a label (int or string) into the corresponding emoji code (string) ready to be printed\n",
    "    \"\"\"\n",
    "    return emoji.emojize(emoji_dictionary[str(label)], use_aliases=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I am proud of your achievements 😄\n"
     ]
    }
   ],
   "source": [
    "index = 1\n",
    "print(X_train[index], label_to_emoji(Y_train[index]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 - Overview of the Emojifier-V1\n",
    "\n",
    "\n",
    "<center>\n",
    "<img src=\"images/image_1.png\" style=\"width:900px;height:300px;\">\n",
    "<caption><center> **Figure 2**: Baseline model (Emojifier-V1).</center></caption>\n",
    "</center>\n",
    "\n",
    "The input of the model is a string corresponding to a sentence (e.g. \"I love you). In the code, the output will be a probability vector of shape (1,5), that you then pass in an argmax layer to extract the index of the most likely emoji output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">   تبدیل Labelها به بردار One-Hot\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_oh_train =  keras.utils.to_categorical(Y_train,  5)\n",
    "Y_oh_test =  keras.utils.to_categorical(Y_test, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 is converted into one hot [1. 0. 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "index = 50\n",
    "print(Y_train[index], \"is converted into one hot\", Y_oh_train[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 - Implementing Emojifier-V1\n",
    "\n",
    "As shown in Figure (2), the first step is to convert an input sentence into the word vector representation, which then get averaged together. Similar to the previous exercise, we will use pretrained 50-dimensional GloVe embeddings. Run the following cell to load the `word_to_vec_map`, which contains all the vector representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\"> تابع کمکی برای خواندن embedding از پیش آموزش داده شده.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_glove_vecs(glove_file):\n",
    "    with open(glove_file, encoding=\"utf8\") as f:\n",
    "        words = set()\n",
    "        word_to_vec_map = {}\n",
    "        for line in f:\n",
    "            line = line.strip().split()\n",
    "            curr_word = line[0]\n",
    "            words.add(curr_word)\n",
    "            word_to_vec_map[curr_word] = np.array(line[1:], dtype=np.float64)\n",
    "        \n",
    "        i = 1\n",
    "        words_to_index = {}\n",
    "        index_to_words = {}\n",
    "        for w in sorted(words):\n",
    "            words_to_index[w] = i\n",
    "            index_to_words[i] = w\n",
    "            i = i + 1\n",
    "    return words_to_index, index_to_words, word_to_vec_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_to_index, index_to_word, word_to_vec_map = read_glove_vecs('D:/data/glove.6B.50d.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You've loaded:\n",
    "- `word_to_index`: dictionary mapping from words to their indices in the vocabulary (400,001 words, with the valid indices ranging from 0 to 400,000)\n",
    "- `index_to_word`: dictionary mapping from indices to their corresponding words in the vocabulary\n",
    "- `word_to_vec_map`: dictionary mapping words to their GloVe vector representation.(`embeddings_index` in perivious notbook)\n",
    "\n",
    "Run the following cell to check if it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the index of ali in the vocabulary is 51314\n",
      "the 113317th word in the vocabulary is cucumber\n"
     ]
    }
   ],
   "source": [
    "word = \"ali\"\n",
    "index = 113317\n",
    "print(\"the index of\", word, \"in the vocabulary is\", word_to_index[word])\n",
    "print(\"the\", str(index) + \"th word in the vocabulary is\", index_to_word[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.71587 ,  0.7874  ,  0.71305 , -0.089955,  1.366   , -1.3149  ,\n",
       "        0.7309  ,  0.79725 ,  0.47211 ,  0.53347 ,  0.37542 , -0.10256 ,\n",
       "       -1.0003  , -0.31226 ,  0.26217 ,  0.92426 ,  0.43014 , -0.015593,\n",
       "        0.4149  ,  0.88286 ,  0.10869 ,  0.95213 ,  1.1807  ,  0.06445 ,\n",
       "       -0.05814 , -1.797   , -0.18432 , -0.41754 , -0.73625 ,  1.1607  ,\n",
       "        1.5932  , -0.70268 , -0.61621 ,  0.47118 ,  0.95046 ,  0.35206 ,\n",
       "        0.6072  ,  0.59339 , -0.47091 ,  1.4916  ,  0.27146 ,  1.8252  ,\n",
       "       -1.2073  , -0.80058 ,  0.52558 , -0.33346 , -1.4102  , -0.21514 ,\n",
       "        0.12945 , -0.69603 ])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_to_vec_map[\"ali\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">   تبدیل جمله به میانگین Embeddingهای کلمات آن\n",
    "</div>\n",
    "<div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">  \n",
    "هر جمله را به کلمات تشکیل دهنده آن و سپس هر کلمه را به embedding و در نهایت این بردارهای embedding را میانگین خواهیم گرفت.\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence_to_avg(sentence, word_to_vec_map):\n",
    "    \n",
    "    # Split sentence into list of lower case words\n",
    "    words = sentence.lower().split()\n",
    "\n",
    "    # Initialize the average word vector, should have the same shape as your word vectors.\n",
    "    avg = np.zeros((50,))\n",
    "    \n",
    "    # average the word vectors. You can loop over the words in the list \"words\".\n",
    "    for w in words:\n",
    "        avg += word_to_vec_map[w]\n",
    "    avg = avg / len(words)\n",
    "    \n",
    "   \n",
    "    return avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "avg =  [-0.008005    0.56370833 -0.50427333  0.258865    0.55131103  0.03104983\n",
      " -0.21013718  0.16893933 -0.09590267  0.141784   -0.15708967  0.18525867\n",
      "  0.6495785   0.38371117  0.21102167  0.11301667  0.02613967  0.26037767\n",
      "  0.05820667 -0.01578167 -0.12078833 -0.02471267  0.4128455   0.5152061\n",
      "  0.38756167 -0.898661   -0.535145    0.33501167  0.68806933 -0.2156265\n",
      "  1.797155    0.10476933 -0.36775333  0.750785    0.10282583  0.348925\n",
      " -0.27262833  0.66768    -0.10706167 -0.283635    0.59580117  0.28747333\n",
      " -0.3366635   0.23393817  0.34349183  0.178405    0.1166155  -0.076433\n",
      "  0.1445417   0.09808667]\n"
     ]
    }
   ],
   "source": [
    "avg = sentence_to_avg(\"Morrocan couscous is my favorite dish\", word_to_vec_map)\n",
    "print(\"avg = \", avg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">   مدل\n",
    "</div>\n",
    "You now have all the pieces to finish implementing the `model()` function. After using `sentence_to_avg()` you need to pass the average through forward propagation, compute the cost, and then backpropagate to update the softmax's parameters. \n",
    "\n",
    "Assuming here that $Yoh$ (\"Y one hot\") is the one-hot encoding of the output labels, the equations you need to implement in the forward pass and to compute the cross-entropy cost are:\n",
    "$$ z^{(i)} = W . avg^{(i)} + b$$\n",
    "$$ a^{(i)} = softmax(z^{(i)})$$\n",
    "$$ \\mathcal{L}^{(i)} = - \\sum_{k = 0}^{n_y - 1} Yoh^{(i)}_k * log(a^{(i)}_k)$$\n",
    "\n",
    "It is possible to come up with a more efficient vectorized implementation. But since we are using a for-loop to convert the sentences one at a time into the avg^{(i)} representation anyway, let's not bother this time. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    \"\"\"Compute softmax values for each sets of scores in x.\"\"\"\n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum()\n",
    "\n",
    "def predict(X, Y, W, b, word_to_vec_map):\n",
    "    \"\"\"\n",
    "    Given X (sentences) and Y (emoji indices), predict emojis and compute the accuracy of your model over the given set.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- input data containing sentences, numpy array of shape (m, None)\n",
    "    Y -- labels, containing index of the label emoji, numpy array of shape (m, 1)\n",
    "    \n",
    "    Returns:\n",
    "    pred -- numpy array of shape (m, 1) with your predictions\n",
    "    \"\"\"\n",
    "    m = X.shape[0]\n",
    "    pred = np.zeros((m, 1))\n",
    "    \n",
    "    for j in range(m):                       # Loop over training examples\n",
    "        \n",
    "        # Split jth test example (sentence) into list of lower case words\n",
    "        words = X[j].lower().split()\n",
    "        \n",
    "        # Average words' vectors\n",
    "        avg = np.zeros((50,))\n",
    "        for w in words:\n",
    "            avg += word_to_vec_map[w]\n",
    "        avg = avg/len(words)\n",
    "\n",
    "        # Forward propagation\n",
    "        Z = np.dot(W, avg) + b\n",
    "        A = softmax(Z)\n",
    "        pred[j] = np.argmax(A)\n",
    "        \n",
    "    print(\"Accuracy: \"  + str(np.mean((pred[:] == Y.reshape(Y.shape[0],1)[:]))))\n",
    "    \n",
    "    return pred\n",
    "\n",
    "def model(X, Y, word_to_vec_map, learning_rate = 0.01, num_iterations = 401):\n",
    "    \"\"\"\n",
    "    Model to train word vector representations in numpy.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- input data, numpy array of sentences as strings, of shape (m, 1)\n",
    "    Y -- labels, numpy array of integers between 0 and 7, numpy-array of shape (m, 1)\n",
    "    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation\n",
    "    learning_rate -- learning_rate for the stochastic gradient descent algorithm\n",
    "    num_iterations -- number of iterations\n",
    "    \n",
    "    Returns:\n",
    "    pred -- vector of predictions, numpy-array of shape (m, 1)\n",
    "    W -- weight matrix of the softmax layer, of shape (n_y, n_h)\n",
    "    b -- bias of the softmax layer, of shape (n_y,)\n",
    "    \"\"\"\n",
    "    \n",
    "    np.random.seed(1)\n",
    "\n",
    "    # Define number of training examples\n",
    "    m = Y.shape[0]                          # number of training examples\n",
    "    n_y = 5                                 # number of classes  \n",
    "    n_h = 50                                # dimensions of the GloVe vectors \n",
    "    \n",
    "    # Initialize parameters using Xavier initialization\n",
    "    W = np.random.randn(n_y, n_h) / np.sqrt(n_h)\n",
    "    b = np.zeros((n_y,))\n",
    "    \n",
    "    # Convert Y to Y_onehot with n_y classes\n",
    "    Y_oh = keras.utils.to_categorical(Y, n_y) \n",
    "    \n",
    "    # Optimization loop\n",
    "    for t in range(num_iterations):                       # Loop over the number of iterations\n",
    "        for i in range(m):                                # Loop over the training examples\n",
    "            \n",
    "            # Average the word vectors of the words from the i'th training example\n",
    "            avg = sentence_to_avg(X[i], word_to_vec_map)\n",
    "\n",
    "            # Forward propagate the avg through the softmax layer\n",
    "            z = np.dot(W, avg) + b\n",
    "            a = softmax(z)\n",
    "\n",
    "            # Compute cost using the i'th training label's one hot representation and \"A\" (the output of the softmax)\n",
    "            cost = -np.sum(Y_oh[i] * np.log(a))\n",
    "            \n",
    "            # Compute gradients \n",
    "            dz = a - Y_oh[i]\n",
    "            dW = np.dot(dz.reshape(n_y,1), avg.reshape(1, n_h))\n",
    "            db = dz\n",
    "\n",
    "            # Update parameters with Stochastic Gradient Descent\n",
    "            W = W - learning_rate * dW\n",
    "            b = b - learning_rate * db\n",
    "        \n",
    "        if t % 100 == 0:\n",
    "            print(\"Epoch: \" + str(t) + \" --- cost = \" + str(cost))\n",
    "            pred = predict(X, Y, W, b, word_to_vec_map)\n",
    "\n",
    "    return pred, W, b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 --- cost = 1.9520498812810072\n",
      "Accuracy: 0.3484848484848485\n",
      "Epoch: 100 --- cost = 0.07971818726014807\n",
      "Accuracy: 0.9318181818181818\n",
      "Epoch: 200 --- cost = 0.04456369243681402\n",
      "Accuracy: 0.9545454545454546\n",
      "Epoch: 300 --- cost = 0.03432267378786059\n",
      "Accuracy: 0.9696969696969697\n",
      "Epoch: 400 --- cost = 0.02906976783312465\n",
      "Accuracy: 0.9772727272727273\n"
     ]
    }
   ],
   "source": [
    "pred, W, b = model(X_train, Y_train, word_to_vec_map)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3.]\n",
      " [2.]\n",
      " [3.]\n",
      " [0.]\n",
      " [4.]\n",
      " [0.]\n",
      " [3.]\n",
      " [2.]\n",
      " [3.]\n",
      " [1.]\n",
      " [3.]\n",
      " [3.]\n",
      " [1.]\n",
      " [3.]\n",
      " [2.]\n",
      " [3.]\n",
      " [2.]\n",
      " [3.]\n",
      " [1.]\n",
      " [2.]\n",
      " [3.]\n",
      " [0.]\n",
      " [2.]\n",
      " [2.]\n",
      " [2.]\n",
      " [1.]\n",
      " [4.]\n",
      " [3.]\n",
      " [3.]\n",
      " [4.]\n",
      " [0.]\n",
      " [3.]\n",
      " [4.]\n",
      " [2.]\n",
      " [0.]\n",
      " [3.]\n",
      " [2.]\n",
      " [2.]\n",
      " [3.]\n",
      " [4.]\n",
      " [2.]\n",
      " [2.]\n",
      " [0.]\n",
      " [2.]\n",
      " [3.]\n",
      " [0.]\n",
      " [3.]\n",
      " [2.]\n",
      " [4.]\n",
      " [3.]\n",
      " [0.]\n",
      " [3.]\n",
      " [3.]\n",
      " [3.]\n",
      " [4.]\n",
      " [2.]\n",
      " [1.]\n",
      " [1.]\n",
      " [1.]\n",
      " [2.]\n",
      " [3.]\n",
      " [1.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [3.]\n",
      " [4.]\n",
      " [4.]\n",
      " [2.]\n",
      " [2.]\n",
      " [1.]\n",
      " [2.]\n",
      " [0.]\n",
      " [3.]\n",
      " [2.]\n",
      " [2.]\n",
      " [0.]\n",
      " [0.]\n",
      " [3.]\n",
      " [1.]\n",
      " [2.]\n",
      " [1.]\n",
      " [2.]\n",
      " [2.]\n",
      " [4.]\n",
      " [3.]\n",
      " [3.]\n",
      " [2.]\n",
      " [4.]\n",
      " [0.]\n",
      " [0.]\n",
      " [3.]\n",
      " [3.]\n",
      " [3.]\n",
      " [3.]\n",
      " [2.]\n",
      " [0.]\n",
      " [1.]\n",
      " [2.]\n",
      " [3.]\n",
      " [0.]\n",
      " [2.]\n",
      " [2.]\n",
      " [2.]\n",
      " [3.]\n",
      " [2.]\n",
      " [2.]\n",
      " [2.]\n",
      " [4.]\n",
      " [1.]\n",
      " [1.]\n",
      " [3.]\n",
      " [3.]\n",
      " [4.]\n",
      " [1.]\n",
      " [2.]\n",
      " [1.]\n",
      " [1.]\n",
      " [3.]\n",
      " [1.]\n",
      " [0.]\n",
      " [4.]\n",
      " [0.]\n",
      " [3.]\n",
      " [3.]\n",
      " [4.]\n",
      " [4.]\n",
      " [1.]\n",
      " [4.]\n",
      " [3.]\n",
      " [0.]\n",
      " [2.]]\n"
     ]
    }
   ],
   "source": [
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 1.4 - Examining test set performance \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set:\n",
      "Accuracy: 0.9772727272727273\n",
      "Test set:\n",
      "Accuracy: 0.8571428571428571\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set:\")\n",
    "pred_train = predict(X_train, Y_train, W, b, word_to_vec_map)\n",
    "print('Test set:')\n",
    "pred_test = predict(X_test, Y_test, W, b, word_to_vec_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random guessing would have had 20% accuracy given that there are 5 classes. This is pretty good performance after training on only 127 examples. \n",
    "\n",
    "In the training set, the algorithm saw the sentence \"*I love you*\" with the label ❤️. You can check however that the word \"adore\" does not appear in the training set. Nonetheless, lets see what happens if you write \"*I adore you*.\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_predictions(X, pred):\n",
    "    print()\n",
    "    for i in range(X.shape[0]):\n",
    "        print(X[i], label_to_emoji(int(pred[i])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.8333333333333334\n",
      "\n",
      "i adore you ❤️\n",
      "i love you ❤️\n",
      "funny lol 😄\n",
      "lets play with a ball ⚾\n",
      "food is ready 🍴\n",
      "not feeling happy 😄\n"
     ]
    }
   ],
   "source": [
    "X_my_sentences = np.array([\"i adore you\", \"i love you\", \"funny lol\", \"lets play with a ball\", \"food is ready\", \"not feeling happy\"])\n",
    "Y_my_labels = np.array([[0], [0], [2], [1], [4],[3]])\n",
    "\n",
    "pred = predict(X_my_sentences, Y_my_labels , W, b, word_to_vec_map)\n",
    "print_predictions(X_my_sentences, pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazing! Because *adore* has a similar embedding as *love*, the algorithm has generalized correctly even to a word it has never seen before. Words such as *heart*, *dear*, *beloved* or *adore* have embedding vectors similar to *love*, and so might work too---feel free to modify the inputs above and try out a variety of input sentences. How well does it work?\n",
    "\n",
    "Note though that it doesn't get \"not feeling happy\" correct. This algorithm ignores word ordering, so is not good at understanding phrases like \"not happy.\" \n",
    "\n",
    "Printing the confusion matrix can also help understand which classes are more difficult for your model. A confusion matrix shows how often an example whose label is one class (\"actual\" class) is mislabeled by the algorithm with a different class (\"predicted\" class). \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(y_actu, y_pred, title='Confusion matrix', cmap=plt.cm.gray_r):\n",
    "    \n",
    "    df_confusion = pd.crosstab(y_actu, y_pred.reshape(y_pred.shape[0],), rownames=['Actual'], colnames=['Predicted'], margins=True)\n",
    "    \n",
    "    df_conf_norm = df_confusion / df_confusion.sum(axis=1)\n",
    "    \n",
    "    plt.matshow(df_confusion, cmap=cmap) # imshow\n",
    "    #plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(df_confusion.columns))\n",
    "    plt.xticks(tick_marks, df_confusion.columns, rotation=45)\n",
    "    plt.yticks(tick_marks, df_confusion.index)\n",
    "    #plt.tight_layout()\n",
    "    plt.ylabel(df_confusion.index.name)\n",
    "    plt.xlabel(df_confusion.columns.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(56,)\n",
      "           ❤️    ⚾    😄    😞   🍴\n",
      "Predicted  0.0  1.0  2.0  3.0  4.0  All\n",
      "Actual                                 \n",
      "0            6    0    0    1    0    7\n",
      "1            0    8    0    0    0    8\n",
      "2            2    0   16    0    0   18\n",
      "3            1    1    2   12    0   16\n",
      "4            0    0    1    0    6    7\n",
      "All          9    9   19   13    6   56\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQwAAAD3CAYAAADormr9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAGPlJREFUeJzt3Xu4ZFV95vHve/pGk24Gmm4Q6BZ6BLmEUUDS8ZGJwyUSQIIIJrEzOIjMA8ZhBkQjyFxCniSD0QQdEok2iqLIxaAIGkAJoQUMt25guDWkO4ihsbk0SLiE7p6Gd/7Y+0Bxci67zqldteuc9/M89Zzau3bt36o6Vb9ae62915JtIiKqGOh1ASKifyRhRERlSRgRUVkSRkRUloQREZUlYUREZUkYEVFZEkZEVJaEERGVTe91AeokaR9gI4DtVT0qw4DtV7sQZwkwA9hs+/a647XE7cl73Iu4kuQpfmr0pK1hSDoc+D7wMeCvJZ3QpbjvlfSHks6RtG2XksVvAFcD7wUulXSKpDldiNur97gncYGZZfyufG8kuY3bdd0oE7Yn1Q0QMAe4BjiqXPdOYA3w0Zpj/yrwU+B3gS8BPwHeBcyo8bXOAr4O/Ha5bh/geuCTwOzJ9B73+H+7G3AFsHO5PFBnvDJG5YQBrKi7PLYnXw3DhReBFcBWkmbYvg34IHCGpONrDL838CPbl9j+KPAd4FPAftD5X6bytW4EVgFvkzTH9j3AacARwEc6GW9I3K6/xz3+3z4B/Aw4R9Ii2692o6YhqdKtWyZdwmjxBHAIMBvA9grgQ8B/lbS4pph3ArMl7VHGPBe4BfiCpK1d3+HJvcC2wFskTbf9APD7wOmS3l5TTOjNe9zVuJL+naQrbb8AnA08Cvx5t5JGEkbNVL57ts8HtgS+JOnflL9Gt1B8uepquHoC2Ay8R9L8shx/BtwPnFxTTGxfC7wInArsXdY0VgLXUVTj64rb1fdY0rQexH2U4tDg8jJpnENxCFR70pDEwMBApVu3qDxW6muSdgfmUVRVX7X9SstjlwEvA7dR9AqdDvwH22s7FHvakHj7An9M8WVdbvs+SWeW5fpsB+LtCmwN3G97w5DHPgvMBTYAjwGfAA6w/WgH4v4yMB9YZfup1h6DOt9jSf8eWGz7m+XyTNubuhD3TbafKO/PAr4GzLJ9rKS5wKeBXYCzOvH+DmdgYMAzZsyotO2mTZtW2t6/jnK06vuEIekY4H8Dj5e3FcDXbT/fss1HgB2BtwNnl1X2icZ9q+1/KO9Ps/3K4JeoTBonU3yxDSwBjrZ93wRjHknxWp+hqM38ie37y1/Y/1ducxDwNuCtwBdtPziRmOU+Dwf+FHiEouv2JNuPD4nb0fe4/NXeEridopZ0nu0vlY9tMZgsa/rf7gE8CPwf4EHbF0j6JeALwALbR5dJ44+ArSjej80TjTvUwMCAZ86cWWnbjRs3JmGMRdIM4GKKD9NPJB1L0Wq+Efic7X8esv2sspFwonGPBL4NfM/275brBpPGQFlNnQ9sA/wKcKvtn04w5ruAC4Gltu+WdD6whe2PlI+/4XyPsi1jwh9iSQcCy4DjbN8h6UqKRPS3Q2tX5fYdeY9b9vcp4BWKhHC37c+PsF3H4kpaBFxG0VV9CLAOuJzi0PLjwJvLmsZWFLWOpzsRd6iBgQHPmjWr0rYbNmzoSsKYDG0YW1F0eQFcCfyAor98KRQnNEnar3x800SDlb80p1D0RGySdDFAmSymt3xpN9teXfaYTChZtPiM7bvL+38AzCury5RJ6lfKZAbFl6wTngROLpPFmyi6jk+R9GWKhkYkvaOT7/EQm4FFwEXAEknnSjqnjPuuOuLafgy4g6J36wiKw8uTgG8AXwEWSTrP9vN1JYtBafTsoLI6fC5wjKRfK7+stwD3AO+WNBs4APh5uf2Eq1O2X6LorryE4lyHLVqSxmaAsmfiOElbqHP/zduB75b7n0Zx/sXOFAkTSQuBPSgOyTryWsv9rLJ9Y7l4InC+7aMp2g2OkLQL8G46+B4PcRXwhO0bKF7b71Ec6kFRe+to3Jb/1xkUh5PzKWoYbwdWA/+LotHz/E7EG6MsjUsYfX1IAsXxLPCfKY7bL7Z9U7l+OXCi7X+sOf62FFX2l20fJ+ltFDWem20/VVPM6cAWwFW2D5F0HLAvxTH8C3XEHKEc1wKnDrbl1BRjR+BPgL+nOKflmxRtQpcAl9aQoAaTxkzgfwL/lqKmcabt70naDVhv+xedjjvUtGnTPHv27ErbvvTSS105JOn7a0lsb5D0LYpfg0+XDVYbgQUUXY11x39G0snA5yQ9TFFre3ddyaKMuRl4UdJjZfX8UOCEOpNFa69IuXwssB1Qa4Ky/XNJj1F8ef+L7e+XDbtr6kgWZUwDGyV9E7gZ+Avb3ysfW11HzJF0s8u0ir5PGAC2fyHpAoqW7ZMpuhWPs/1kl+Kvl3QvcDjwHtvr6oxX/gLOAH6t/HtI3R/kli7UWcBxFF2Yv1P3ay1dQFGbWlku/9hduEbH9sOSzgB2lrSl7X+pO+ZQ3TzcqGJSJAyAsm/+Rkk3FYv1f6AGSdqGonHs0Il2nVZRfnk3Sfoj4M4u/+q9SnFMf4zth7sRsGyEfGywltPN/y1wK3BMF+O9ptvtE1X0fRtGU7SeG9DFmFP+cutu6FXtYvr06Z47d26lbZ977rm0YfSTbieLMmaSRRf0IlkMaloNIwkjosGSMCKisiSMiKhE5dWqTdKs0tRA0klTIWbiTs64TTvTc9InDIprAKZCzMSdhHE7mTAkPSrpPkn3SFpRrpsn6XpJq8u/24y2j6mQMCL6Vg01jINs79PSBXsmcIPt3YAbyuWRy9MPPXPz5s3zokWLxvXcZ555hm233XZcz606eMlQTz/9NAsWLBjXcydiInEn8jlYv3498+fPH9dzJ1Kdnsjr3bRp/Be3jvcztXbtWp599tnKL3jmzJmu+r6uW7duzPMwJD0K7G97fcu6h4EDba+TtAPFoE+7j7SPvmj0XLRoEddcc03X4+60005dj9krmzd3fPyXSqZP781H8NFHH+16zKOOOqrt53S4fcLAj1SMMv5l28uA7QdP7y+Txnaj7aAvEkbEVNVGwpg/2C5RWlYmhFYHlBfzbQdcL+mhdsuThBHRYG10q64f65DE9uDYIU+pGDltCfCkpB1aDklGvco6jZ4RDdXJAXQk/ZKKcUgHR407lGLIwauBwflcjqcYsGhEqWFENFgH2zC2B64s9zcduMT2dZLuBL4t6UTgn4DfGm0nSRgRDdaphGH7EYphBoeuf4ZioONKkjAiGizXkkREZUkYEVFJEy8+S8KIaLCm1TB6kr4kHSbpYUlrVMw7GhHDmPJXq6qYhOeLFCNs7wUslbRXt8sR0Q+mfMKgOLtsje1HypG+LwPe14NyRDRaJ0/c6pReJIydgMdalteW6yJiiKYljF40eg736v7VtdXlqEYnwdS6ajSiVRo9ixpF6+AWCykn1G1le5nt/W3vP97xLCL63cDAQKVb18rTtUivuxPYTdJiSTOBD1JcABMRLZrYhtH1QxLbmyWdAvwQmAZcaPuBbpcjoh807ZCkJydu2b4G6P4QWhF9JgkjIipLwoiIypIwIqKSbjdoVpGEEdFguVo1IipLDSMiKkvCiIhK0oYREW1JwoiIypIwxmHGjBk9uWJ1zZo1XY8JsOuuu3Y9Zq/mOO2VXswlO54Jr5MwIqKSDAIcEW1JDSMiKkvCiIjKkjAiorIkjIioJCduRURbmpYwmtVnExFv0MlBgCVNk3S3pB+Uy4sl3S5ptaTLyzF2Ry/PBF9PRNSow4MAnwqsaln+U+DztncDfgGcONYOkjAiGqqTo4ZLWgi8F/hKuSzgYOCKcpOLgKPH2k/aMCIarINtGF8APgXMLZe3BZ6zPXiOfKUZCHs1e/uFkp6SdH8v4kf0izZqGPMlrWi5ndSyjyOBp2yvbN31MOHGvNilVzWMrwN/CXyjR/Ej+kIbNYz1tvcf4bEDgKMkHQFsAWxFUePYWtL0spYx7AyEQ/WkhmH7JuDZXsSO6BeDF59NtJfE9qdtL7S9C8VMg39n+z8CNwIfKDc7HrhqrDKl0TOiwWqeKvEM4HRJayjaNL461hMa2+jZOnv7m9/85h6XJqI3On3ilu3lwPLy/iPAknae39gaRuvs7QsWLOh1cSJ6YspPxhwR1eXUcEDSpcCtwO6S1koa8wyziKmmkydudUqvZm9f2ou4Ef2maTWMHJJENFjG9IyISjIeRkS0JQkjIipLwoiIypIwIqKyJIyIqCSNnhHRlnSrRkRlqWGMw6uvvsrLL7/c9bi9mEUd4Nprr+16zMMPP7zrMXvp3nvv7XrM8XyGkzAiopK0YUREW5IwIqKyJIyIqCwJIyIqGRwEuEmSMCIaLDWMiKgsCSMiKkvCiIjKkjAiopImnrjV9SZYSYsk3ShplaQHJJ3a7TJE9IuMGg6bgU/YvkvSXGClpOttP9iDskQ02pTvVrW9DlhX3n9B0ipgJyAJI2KIph2S9LQNQ9IuwL7A7b0sR0QTNbENo2cJQ9Ic4DvAabafH+bx1yZjXrRoUZdLF9EMTUsYvZoqcQZFsviW7e8Ot03rZMzz58/vbgEjGqJvGj0lfR/wSI/bPmo8AVW8uq8Cq2yfO559REwVTathjHZI8mc1xTwA+BBwn6R7ynVn2b6mpngRfalTF59J2gK4CZhF8Z2/wvYfSFoMXAbMA+4CPmR702j7GjFh2P7xhEs6/H5vAZqVNiMaqkM1jI3AwbZfLJsDbpF0LXA68Hnbl0n6EnAi8Fej7WjM9CVpN0lXSHpQ0iODt068iogYXSfaMFx4sVycUd4MHAxcUa6/CDh6rPJUqe98jSLrbAYOAr4BfLPC8yJigjrV6ClpWtkE8BRwPfCPwHO2N5ebrKU4H2pUVRLGbNs3ALL9M9tnU2SmiKhZGwljvqQVLbeTWvdj+xXb+wALgSXAnsOEG7GTY1CV8zA2SBoAVks6BXgc2K7C8yJiAtrsMl1ve/+xNrL9nKTlwDuBrSVNL2sZC4Gfj/X8KjWM04Atgf8GvIOih+P4Cs+LiAnqxCGJpAWSti7vzwZ+HVgF3Ah8oNzseOCqscozZg3D9p3l3ReBE8baPiI6p0MXn+0AXCRpGkUl4du2fyDpQeAySX8M3E1xftSoxkwYkm5kmGMb22nHiKhZJ7pVbd9Lcc3W0PWPULRnVFalDeOTLfe3AI6l6DGJiBr15cVntlcOWfUTSbWc1BURb9R3CUPSvJbFAYqGzzfVVqLhy8CMGTO6GRKAzZt7U5E68MADux7zjjvu6HpMgCVL2qoRd8zs2bO7HnM8X/6+SxjASoo2DFEcivyU4hTSiKhZPyaMPW1vaF0haVZN5YmIFk1LGFX6bP5+mHW3drogEfFGg1erVrl1y2jjYbyJ4tzy2ZL25fUrTLeiOJErImrWtBrGaIckvwF8mOKU0T/n9YTxPHBWvcWKCOijhGH7Ioqzw461/Z0ulikiSk1LGFUOft4xeB46gKRtylNJI6JGVa8j6WZSqZIwDrf93OCC7V8AR9RXpIgY1LSEUaVbdZqkWbY3wmtXu6VbNaILmnZIUiVhXAzcIOlr5fIJFMN5RUTN+m6qRNuflXQvxTX0Aq4Ddq67YBFTXV9efFZ6AngV+G2KU8PH3Wsy0pDn491fxGTWNwlD0luBDwJLgWeAyynG9TxogjGHHfLc9m0T3G/EpNM3CQN4CLgZ+E3bawAkfXyiAW2bYvQueOOQ5xExRNMSxmgtKsdSHIrcKOkCSYfQoQmIhg55bjuzt0cMo2ndqiMmDNtX2v4dYA9gOfBxYHtJfyXp0IkEHTrkuaS9h24j6aTBIdPXr18/kXARfakvT9yy/ZLtb9k+kuILfg9wZieClyeELQcOG+axzN4eU17TrlZtK5LtZ21/eSIDAI8w5PlD491fxGTWtBpG1W7VThp2yPMelCOi8ZrW6Nn1hDHSkOcR8Ub9fOJWRPRAEkZEVJaEERGV9d3FZxHRG2nDiIi2JGFERGVJGBFRWRJGRFTWtITRrCbYiHhNpy4+k7RI0o2SVkl6QNKp5fp5kq6XtLr8u81YZeqLGoYkpk/vi6L2rV7Nov7444/3JO6ee+7Z9ZjjmTG+Q92qm4FP2L5L0lxgpaTrKSYqu8H2ZySdSXFR6RmjlqcTpYmIenSihmF7ne27yvsvAKsopkF9H68P6H0RcPRY5cnPdkRD1XEehqRdKK7luh3Y3vY6KJKKpO3Gen4SRkSDtZEw5kta0bK8zPayIfuaQzGA92m2nx9PMkrCiGiwNr7U623vP8p+ZlAki2/Z/m65+klJO5S1ix0ohswcVdowIhqsQ70kAr4KrLJ9bstDVwPHl/ePB64aqzypYUQ0WIfaMA4APgTcVw6+DXAW8Bng25JOBP4J+K2xdpSEEdFQkjrSrWr7FkYe8f+QdvaVhBHRYE070zMJI6LBkjAiorIkjIiopIkD6PSsW7WcLvFuSZliIGIEmZfkdadSnNO+VQ/LENFoqWEAkhYC7wW+0ov4Ef2iaVMl9qqG8QXgU8DcHsWPaLy0YQCSjgSesr1yjO1em7396aef7lLpIpqlaW0YvTgkOQA4StKjwGXAwZIuHrpR6+ztCxYs6HYZIxphyicM25+2vdD2LsAHgb+zfVy3yxHRD5qWMHIeRkSDNa0No6cJw/ZyYHkvyxDRVE1s9EwNI6LBMrdqRFSWGkZEVJaEERGVpA0jItqShBERlSVhRERl6SWJiErShhERbUnCGIcNGzawatWqXheja+67776ux9xxxx27HhNg8eLFUypuu5IwIqKyJIyIqCwJIyIqSaNnRLQl3aoRUVlqGBFRWRJGRFSSNoyIaEvTEkazWlQi4g06NQiwpAslPSXp/pZ18yRdL2l1+XebsfaThBHRYB0cNfzrwGFD1p0J3GB7N+CGcnlUSRgRDSWpY1Ml2r4JeHbI6vcBF5X3LwKOHms/tSYMSe+XZEl7lMu7DFaJJB2YmdsjRlfzvCTb214HUP7dbqwn1F3DWArcQjFhUUS0qY2EMX9watHydlId5amtl0TSHIppEQ8CrgbOritWxGTVRu1hve3929z9k5J2sL1O0g7AU2M9oc4axtHAdbb/AXhW0n41xoqYlGo+JLkaOL68fzxw1VhPqDNhLKWYbJny79J2nqyW2duffXZoW03E5Fc1WVTsVr0UuBXYXdJaSScCnwHeI2k18J5yeVS1HJJI2hY4GNhbkoFpgIHzq+7D9jJgGcDee+/tOsoZ0XSdOnHL9kg/2Ie0s5+62jA+AHzD9smDKyT9GFhYU7yISalpV6vWVZqlwJVD1n0HOKumeBGTUs1tGG2rpYZh+8Bh1p0HnNeyvJzM3B4xolx8FhFtScKIiMqSMCKisiSMiKgsCSMiKhm8WrVJkjAiGiw1jIioLAkjIipLwoiISnLi1jg98MAD6/faa6+fjfPp84H1nSxPQ2MmbvPj7tzuE5IwxsH2gvE+V9KKcQwsMiG9iJm4kzNuEkZEVJZu1YioJG0YvbFsisRM3EkYt2kJo1n1nRqUI3dNipiSXpF0j6T7Jf21pC3HG7d1mgdJR0kacRIbSVtL+thIj48UV9LZkj5ZtUzt6sX/tttxmzYexqRPGJPMy7b3sb03sAn4aOuDKrT9P7V9te3RxnPcGhgxYUR9kjCiU24GdlUxOdQqSecDdwGLJB0q6VZJd5U1kTkAkg6T9JCkW4BjBnck6cOS/rK8v72kKyX93/L2LorBYd9S1m4+V273+5LulHSvpD9s2dd/l/SwpL8Fdu/auzFJNS1hTIU2jElH0nTgcOC6ctXuwAm2PyZpPvA/gF+3/ZKkM4DTJX0WuIBicOY1wOUj7P484Me23y9pGjCHYs7NvW3vU8Y/FNgNWAIIuFrSu4GXKCat2pfis3UXsLKzr37qyMVnMVGzJd1T3r8Z+CqwI/Az27eV698J7AX8pPzlmUkxvPwewE9trwaQdDEw3OxYBwP/CcD2K8A/61/P6n1oebu7XJ5DkUDmAlfa/pcyxtUTerXRuEbPJIz+8vLgr/yg8gP1Uusq4Pqhw8pL2odiqodOEHCO7S8PiXFaB2MEzUsYzarvRCfcBhwgaVcASVtKeivwELBY0lvK7Uaap+IG4PfK506TtBXwAkXtYdAPgY+0tI3sJGk74Cbg/ZJmS5oL/GaHX9uUUrX9Io2eMW62nwY+DFwq6V6KBLKH7Q0UhyB/UzZ6jnRtzqnAQZLuo2h/+GXbz1Ac4twv6XO2fwRcAtxabncFMNf2XRRtI/dQTCtxc20vdIpoWsKQnRpkRBPtt99+vvnmajl3zpw5K7txfUvaMCIarGltGEkYEQ2VbtWIaEtqGBFRWRJGRFTWtITRrAOkiHiDTnWrltcRPSxpjUa5MnksSRgRDdWpE7fKa4K+SHH90V7AUkl7jadMSRgRDdahGsYSYI3tR2xvAi4D3jee8qQNI6LBOtStuhPwWMvyWuBXx7OjJIyIhlq5cuUPy+EKqthC0oqW5WUtI4MNVwUZ1yneSRgRDWX7sA7tai2wqGV5IfDz8ewobRgRk9+dwG6SFkuaSTHI0bjGKkkNI2KSs71Z0ikUwxJMAy60/cB49pWrVSOishySRERlSRgRUVkSRkRUloQREZUlYUREZUkYEVFZEkZEVJaEERGV/X/OSvtg/TTzpgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 288x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "print(Y_test.shape)\n",
    "print('           '+ label_to_emoji(0)+ '    ' + label_to_emoji(1) + '    ' +  label_to_emoji(2)+ '    ' + label_to_emoji(3)+'   ' + label_to_emoji(4))\n",
    "print(pd.crosstab(Y_test, pred_test.reshape(56,), rownames=['Actual'], colnames=['Predicted'], margins=True))\n",
    "plot_confusion_matrix(Y_test, pred_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<font color='blue'>\n",
    "**What you should remember from this part**:\n",
    "- Even with a 127 training examples, you can get a reasonably good model for Emojifying. This is due to the generalization power word vectors gives you. \n",
    "- Emojify-V1 will perform poorly on sentences such as *\"This movie is not good and not enjoyable\"* because it doesn't understand combinations of words--it just averages all the words' embedding vectors together, without paying attention to the ordering of words. You will build a better algorithm in the next part. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Emojifier-V2: Using LSTMs in Keras: \n",
    "\n",
    "Let's build an LSTM model that takes as input word sequences. This model will be able to take word ordering into account. Emojifier-V2 will continue to use pre-trained word embeddings to represent words, but will feed them into an LSTM, whose job it is to predict the most appropriate emoji. \n",
    "\n",
    "Run the following cell to load the Keras packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.random.seed(0)\n",
    "from keras.models import Model\n",
    "from keras.layers import Dense, Input, Dropout, LSTM, Activation\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras.initializers import glorot_uniform\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 - Overview of the model\n",
    "\n",
    "Here is the Emojifier-v2 you will implement:\n",
    "\n",
    "<img src=\"images/emojifier-v2.png\" style=\"width:700px;height:400px;\"> <br>\n",
    "<caption><center> **Figure 3**: Emojifier-V2. A 2-layer LSTM sequence classifier. </center></caption>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Keras and mini-batching \n",
    "\n",
    "In this exercise, we want to train Keras using mini-batches. However, most deep learning frameworks require that all sequences in the same mini-batch have the same length. This is what allows vectorization to work: If you had a 3-word sentence and a 4-word sentence, then the computations needed for them are different (one takes 3 steps of an LSTM, one takes 4 steps) so it's just not possible to do them both at the same time.\n",
    "\n",
    "The common solution to this is to use padding. Specifically, set a maximum sequence length, and pad all sequences to the same length. For example, of the maximum sequence length is 20, we could pad every sentence with \"0\"s so that each input sentence is of length 20. Thus, a sentence \"i love you\" would be represented as $(e_{i}, e_{love}, e_{you}, \\vec{0}, \\vec{0}, \\ldots, \\vec{0})$. In this example, any sentences longer than 20 words would have to be truncated. One simple way to choose the maximum sequence length is to just pick the length of the longest sentence in the training set. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 - The Embedding layer\n",
    "\n",
    "In Keras, the embedding matrix is represented as a \"layer\", and maps positive integers (indices corresponding to words) into dense vectors of fixed size (the embedding vectors). It can be trained or initialized with a pretrained embedding. In this part, you will learn how to create an [Embedding()](https://keras.io/layers/embeddings/) layer in Keras, initialize it with the GloVe 50-dimensional vectors loaded earlier in the notebook. Because our training set is quite small, we will not update the word embeddings but will instead leave their values fixed. But in the code below, we'll show you how Keras allows you to either train or leave fixed this layer.  \n",
    "\n",
    "The `Embedding()` layer takes an integer matrix of size (batch size, max input length) as input. This corresponds to sentences converted into lists of indices (integers), as shown in the figure below.\n",
    "\n",
    "<img src=\"images/embedding1.png\" style=\"width:700px;height:250px;\">\n",
    "<caption><center> **Figure 4**: Embedding layer. This example shows the propagation of two examples through the embedding layer. Both have been zero-padded to a length of `max_len=5`. The final dimension of the representation is  `(2,max_len,50)` because the word embeddings we are using are 50 dimensional. </center></caption>\n",
    "\n",
    "The largest integer (i.e. word index) in the input should be no larger than the vocabulary size. The layer outputs an array of shape (batch size, max input length, dimension of word vectors).\n",
    "\n",
    "The first step is to convert all your training sentences into lists of indices, and then zero-pad all these lists so that their length is the length of the longest sentence. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">   تبدیل جمله به indexها \n",
    "</div>\n",
    "<div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">  \n",
    "این تابع طول تمام جمله ها را نیز یکسان میکند.\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def sentences_to_indices(X, word_to_index, max_len):\n",
    "    \"\"\"\n",
    "    Converts an array of sentences (strings) into an array of indices corresponding to words in the sentences.\n",
    "    The output shape should be such that it can be given to `Embedding()` (described in Figure 4). \n",
    "    \n",
    "    Arguments:\n",
    "    X -- array of sentences (strings), of shape (m, 1)\n",
    "    word_to_index -- a dictionary containing the each word mapped to its index\n",
    "    max_len -- maximum number of words in a sentence. You can assume every sentence in X is no longer than this. \n",
    "    \n",
    "    Returns:\n",
    "    X_indices -- array of indices corresponding to words in the sentences from X, of shape (m, max_len)\n",
    "    \"\"\"\n",
    "    \n",
    "    m = X.shape[0]                                   # number of training examples\n",
    "    \n",
    "    # Initialize X_indices as a numpy matrix of zeros and the correct shape (≈ 1 line)\n",
    "    X_indices = np.zeros((m, max_len))\n",
    "    \n",
    "    for i in range(m):                               # loop over training examples\n",
    "        \n",
    "        # Convert the ith training sentence in lower case and split is into words. You should get a list of words.\n",
    "        sentence_words =X[i].lower().split()\n",
    "\n",
    "        \n",
    "        # Loop over the words of sentence_words\n",
    "        for j, w in enumerate(sentence_words):\n",
    "            # Set the (i,j)th entry of X_indices to the index of the correct word.\n",
    "            X_indices[i, j] = word_to_index[w]\n",
    "\n",
    "        \n",
    "    return X_indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to check what `sentences_to_indices()` does, and check your results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1 = ['funny lol' 'lets play baseball' 'food is ready for you']\n",
      "X1_indices = [[155345. 225122.      0.      0.      0.]\n",
      " [220930. 286375.  69714.      0.      0.]\n",
      " [151204. 192973. 302254. 151349. 394475.]]\n"
     ]
    }
   ],
   "source": [
    "X1 = np.array([\"funny lol\", \"lets play baseball\", \"food is ready for you\"])\n",
    "X1_indices = sentences_to_indices(X1,word_to_index, max_len = 5)\n",
    "print(\"X1 =\", X1)\n",
    "print(\"X1_indices =\", X1_indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\">   تابعی برای ایجاد لایه Embedding و لود وزن های از پیش آموزش داده شده \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build the `Embedding()` layer in Keras, using pre-trained word vectors. After this layer is built, you will pass the output of `sentences_to_indices()` to it as an input, and the `Embedding()` layer will return the word embeddings for a sentence. \n",
    "\n",
    "\n",
    "1. Initialize the embedding matrix as a numpy array of zeroes with the correct shape.\n",
    "2. Fill in the embedding matrix with all the word embeddings extracted from `word_to_vec_map`.\n",
    "3. Define Keras embedding layer. Use [Embedding()](https://keras.io/layers/embeddings/). Be sure to make this layer non-trainable, by setting `trainable = False` when calling `Embedding()`. If you were to set `trainable = True`, then it will allow the optimization algorithm to modify the values of the word embeddings. \n",
    "4. Set the embedding weights to be equal to the embedding matrix "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Embedding\n",
    "def pretrained_embedding_layer(word_to_vec_map, word_to_index):\n",
    "    \"\"\"\n",
    "    Creates a Keras Embedding() layer and loads in pre-trained GloVe 50-dimensional vectors.\n",
    "    \n",
    "    Arguments:\n",
    "    word_to_vec_map -- dictionary mapping words to their GloVe vector representation.\n",
    "    word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)\n",
    "\n",
    "    Returns:\n",
    "    embedding_layer -- pretrained layer Keras instance\n",
    "    \"\"\"\n",
    "    \n",
    "    vocab_len = len(word_to_index) + 1                  # adding 1 to fit Keras embedding (requirement)\n",
    "    emb_dim = word_to_vec_map[\"cucumber\"].shape[0]      # define dimensionality of your GloVe word vectors (= 50)\n",
    "    \n",
    "    # Initialize the embedding matrix as a numpy array of zeros of shape (vocab_len, dimensions of word vectors = emb_dim)\n",
    "    emb_matrix = np.zeros((vocab_len, emb_dim))\n",
    "    \n",
    "    # Set each row \"index\" of the embedding matrix to be the word vector representation of the \"index\"th word of the vocabulary\n",
    "    for word, index in word_to_index.items():\n",
    "        emb_matrix[index, :] = word_to_vec_map[word]\n",
    "\n",
    "    # Define Keras embedding layer with the correct output/input sizes, make it trainable. Use Embedding(...). Make sure to set trainable=False. \n",
    "    embedding_layer = Embedding(vocab_len, emb_dim, trainable = False)\n",
    "\n",
    "    # Build the embedding layer, it is required before setting the weights of the embedding layer. Do not modify the \"None\".\n",
    "    embedding_layer.build((None,))\n",
    "    \n",
    "    # Set the weights of the embedding layer to the embedding matrix. Your layer is now pretrained.\n",
    "    embedding_layer.set_weights([emb_matrix])\n",
    "    \n",
    "    return embedding_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Building the Emojifier-V2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Input\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras.models import Model\n",
    "\n",
    "def Emojify_V2(input_shape, word_to_vec_map, word_to_index):\n",
    "    \"\"\"\n",
    "    Function creating the Emojify-v2 model's graph.\n",
    "    \n",
    "    Arguments:\n",
    "    input_shape -- shape of the input, usually (max_len,)\n",
    "    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation\n",
    "    word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)\n",
    "\n",
    "    Returns:\n",
    "    model -- a model instance in Keras\n",
    "    \"\"\"\n",
    "    \n",
    "    # Define sentence_indices as the input of the graph, it should be of shape input_shape and dtype 'int32' (as it contains indices).\n",
    "    sentence_indices = Input(input_shape, dtype = np.int32)\n",
    "    \n",
    "    # Create the embedding layer pretrained with GloVe Vectors (≈1 line)\n",
    "    embedding_layer = pretrained_embedding_layer(word_to_vec_map, word_to_index)\n",
    "    \n",
    "    # Propagate sentence_indices through your embedding layer, you get back the embeddings\n",
    "    embeddings = embedding_layer(sentence_indices)\n",
    "    \n",
    "    # Propagate the embeddings through an LSTM layer with 128-dimensional hidden state\n",
    "    # Be careful, the returned output should be a batch of sequences.\n",
    "    X = LSTM(128, return_sequences=True)(embeddings)\n",
    "    # Add dropout with a probability of 0.5\n",
    "    X = Dropout(0.5)(X)\n",
    "    # Propagate X trough another LSTM layer with 128-dimensional hidden state\n",
    "    # Be careful, the returned output should be a single hidden state, not a batch of sequences.\n",
    "    X = LSTM(128)(X)\n",
    "    # Add dropout with a probability of 0.5\n",
    "    X = Dropout(0.5)(X)\n",
    "    # Propagate X through a Dense layer with softmax activation to get back a batch of 5-dimensional vectors.\n",
    "    X = Dense(5, activation = 'softmax')(X)\n",
    "    # Add a softmax activation\n",
    "    X = Activation('softmax')(X)\n",
    "    \n",
    "    # Create Model instance which converts sentence_indices into X.\n",
    "    model = Model(sentence_indices, X)\n",
    "    \n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to create your model and check its summary. Because all sentences in the dataset are less than 10 words, we chose `max_len = 10`.  You should see your architecture, it uses \"20,223,927\" parameters, of which 20,000,050 (the word embeddings) are non-trainable, and the remaining 223,877 are. Because our vocabulary size has 400,001 words (with valid indices from 0 to 400,000) there are 400,001\\*50 = 20,000,050 non-trainable parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 10, 50)            20000050  \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 10, 128)           91648     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 10, 128)           0         \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 128)               131584    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 5)                 645       \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 5)                 0         \n",
      "=================================================================\n",
      "Total params: 20,223,927\n",
      "Trainable params: 223,877\n",
      "Non-trainable params: 20,000,050\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Emojify_V2((maxLen,), word_to_vec_map, word_to_index)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, after creating your model in Keras, you need to compile it and define what loss, optimizer and metrics your are want to use. Compile your model using `categorical_crossentropy` loss, `adam` optimizer and `['accuracy']` metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's time to train your model. Your Emojifier-V2 `model` takes as input an array of shape (`m`, `max_len`) and outputs probability vectors of shape (`m`, `number of classes`). We thus have to convert X_train (array of sentences as strings) to X_train_indices (array of sentences as list of word indices), and Y_train (labels as indices) to Y_train_oh (labels as one-hot vectors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_indices = sentences_to_indices(X_train, word_to_index, maxLen)\n",
    "Y_train_oh = keras.utils.to_categorical(Y_train, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "132/132 [==============================] - 2s 18ms/step - loss: 1.6072 - acc: 0.2045\n",
      "Epoch 2/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.5902 - acc: 0.2652\n",
      "Epoch 3/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.5738 - acc: 0.2727\n",
      "Epoch 4/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.5541 - acc: 0.3106\n",
      "Epoch 5/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.5373 - acc: 0.2879\n",
      "Epoch 6/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.5139 - acc: 0.3788\n",
      "Epoch 7/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.4485 - acc: 0.5985\n",
      "Epoch 8/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.3791 - acc: 0.6288\n",
      "Epoch 9/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.3555 - acc: 0.5682\n",
      "Epoch 10/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.3386 - acc: 0.6212\n",
      "Epoch 11/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.3254 - acc: 0.5985\n",
      "Epoch 12/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.2782 - acc: 0.6591\n",
      "Epoch 13/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.2132 - acc: 0.7500\n",
      "Epoch 14/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.1906 - acc: 0.7424\n",
      "Epoch 15/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.1485 - acc: 0.7803\n",
      "Epoch 16/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.1244 - acc: 0.7955\n",
      "Epoch 17/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0847 - acc: 0.8712\n",
      "Epoch 18/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.0822 - acc: 0.8409\n",
      "Epoch 19/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.0725 - acc: 0.8561\n",
      "Epoch 20/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0711 - acc: 0.8333\n",
      "Epoch 21/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0937 - acc: 0.8182\n",
      "Epoch 22/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.1135 - acc: 0.7879\n",
      "Epoch 23/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0901 - acc: 0.8258\n",
      "Epoch 24/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.0570 - acc: 0.8485\n",
      "Epoch 25/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.0444 - acc: 0.8712\n",
      "Epoch 26/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0076 - acc: 0.9167\n",
      "Epoch 27/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0178 - acc: 0.8939\n",
      "Epoch 28/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.1122 - acc: 0.7879\n",
      "Epoch 29/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0817 - acc: 0.8333\n",
      "Epoch 30/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0215 - acc: 0.8939\n",
      "Epoch 31/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.0272 - acc: 0.8788\n",
      "Epoch 32/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.0057 - acc: 0.9015\n",
      "Epoch 33/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0194 - acc: 0.8864\n",
      "Epoch 34/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0050 - acc: 0.9015\n",
      "Epoch 35/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0100 - acc: 0.9015\n",
      "Epoch 36/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.1691 - acc: 0.7424\n",
      "Epoch 37/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.2309 - acc: 0.6742\n",
      "Epoch 38/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.0858 - acc: 0.8182\n",
      "Epoch 39/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 1.0107 - acc: 0.9015\n",
      "Epoch 40/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 1.0232 - acc: 0.8864\n",
      "Epoch 41/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 0.9963 - acc: 0.9167\n",
      "Epoch 42/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 0.9957 - acc: 0.9167\n",
      "Epoch 43/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 0.9895 - acc: 0.9167\n",
      "Epoch 44/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 0.9913 - acc: 0.9167\n",
      "Epoch 45/50\n",
      "132/132 [==============================] - 0s 3ms/step - loss: 0.9866 - acc: 0.9167\n",
      "Epoch 46/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 0.9855 - acc: 0.9167\n",
      "Epoch 47/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 0.9880 - acc: 0.9242\n",
      "Epoch 48/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 0.9877 - acc: 0.9167\n",
      "Epoch 49/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 0.9846 - acc: 0.9242\n",
      "Epoch 50/50\n",
      "132/132 [==============================] - 0s 2ms/step - loss: 0.9881 - acc: 0.9242\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1263efb1518>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_indices, Y_train_oh, epochs = 50, batch_size = 32, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your model should perform close to **100% accuracy** on the training set. The exact accuracy you get may be a little different. Run the following cell to evaluate your model on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "56/56 [==============================] - 0s 7ms/step\n",
      "\n",
      "Test accuracy =  0.8035714370863778\n"
     ]
    }
   ],
   "source": [
    "X_test_indices = sentences_to_indices(X_test, word_to_index, max_len = maxLen)\n",
    "Y_test_oh = keras.utils.to_categorical(Y_test, 5)\n",
    "loss, acc = model.evaluate(X_test_indices, Y_test_oh)\n",
    "print()\n",
    "print(\"Test accuracy = \", acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get a test accuracy between 80% and 95%. Run the cell below to see the mislabelled examples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected emoji:😄 prediction: she got me a nice present\t❤️\n",
      "Expected emoji:😄 prediction: he is a good friend\t❤️\n",
      "Expected emoji:😞 prediction: This girl is messing with me\t❤️\n",
      "Expected emoji:🍴 prediction: any suggestions for dinner\t😄\n",
      "Expected emoji:😄 prediction: you brighten my day\t❤️\n",
      "Expected emoji:😞 prediction: she is a bully\t❤️\n",
      "Expected emoji:😞 prediction: My life is so boring\t❤️\n",
      "Expected emoji:😄 prediction: will you be my valentine\t😞\n",
      "Expected emoji:😄 prediction: What you did was awesome\t😞\n",
      "Expected emoji:😞 prediction: go away\t⚾\n",
      "Expected emoji:😞 prediction: yesterday we lost again\t⚾\n"
     ]
    }
   ],
   "source": [
    "# This code allows you to see the mislabelled examples\n",
    "C = 5\n",
    "y_test_oh = np.eye(C)[Y_test.reshape(-1)]\n",
    "X_test_indices = sentences_to_indices(X_test, word_to_index, maxLen)\n",
    "pred = model.predict(X_test_indices)\n",
    "for i in range(len(X_test)):\n",
    "    x = X_test_indices\n",
    "    num = np.argmax(pred[i])\n",
    "    if(num != Y_test[i]):\n",
    "        print('Expected emoji:'+ label_to_emoji(Y_test[i]) + ' prediction: '+ X_test[i] + label_to_emoji(num).strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can try it on your own example. Write your own sentence below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not feeling happy 😞\n"
     ]
    }
   ],
   "source": [
    "# Change the sentence below to see your prediction. Make sure all the words are in the Glove embeddings.  \n",
    "x_test = np.array(['not feeling happy'])\n",
    "X_test_indices = sentences_to_indices(x_test, word_to_index, maxLen)\n",
    "print(x_test[0] +' '+  label_to_emoji(np.argmax(model.predict(X_test_indices))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, Emojify-V1 model did not correctly label \"not feeling happy,\" but our implementation of Emojiy-V2 got it right. (Keras' outputs are slightly random each time, so you may not have obtained the same result.) The current model still isn't very robust at understanding negation (like \"not happy\") because the training set is small and so doesn't have a lot of examples of negation. But if the training set were larger, the LSTM model would be much better than the Emojify-V1 model at understanding such complex sentences. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Congratulations!\n",
    "\n",
    "You have completed this notebook! ❤️❤️❤️\n",
    "\n",
    "<font color='blue'>\n",
    "**What you should remember**:\n",
    "- If you have an NLP task where the training set is small, using word embeddings can help your algorithm significantly. Word embeddings allow your model to work on words in the test set that may not even have appeared in your training set. \n",
    "- Training sequence models in Keras (and in most other deep learning frameworks) requires a few important details:\n",
    "    - To use mini-batches, the sequences need to be padded so that all the examples in a mini-batch have the same length. \n",
    "    - An `Embedding()` layer can be initialized with pretrained values. These values can be either fixed or trained further on your dataset. If however your labeled dataset is small, it's usually not worth trying to train a large pre-trained set of embeddings.   \n",
    "    - `LSTM()` has a flag called `return_sequences` to decide if you would like to return every hidden states or only the last one. \n",
    "    - You can use `Dropout()` right after `LSTM()` to regularize your network. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "\n",
    "# 😀😀😀😀😀😀\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<div style=\"direction:rtl;text-align:right;font-family:B Lotus, B Nazanin, Tahoma\"> دانشگاه تربیت دبیر شهید رجایی<br>مباحث ویژه 2 - یادگیری عمیق پیشرفته<br>علیرضا اخوان پور<br>97-98<br>\n",
    "</div>\n",
    "<a href=\"https://www.srttu.edu/\">SRTTU.edu</a> - <a href=\"http://class.vision\">Class.Vision</a> - <a href=\"http://AkhavanPour.ir\">AkhavanPour.ir</a>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "nlp-sequence-models",
   "graded_item_id": "RNnEs",
   "launcher_item_id": "acNYU"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}