wordvecs_sentiment_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5bgPpghocFIa"
   },
   "source": [
    "# Word vectors: Sentiment classification \n",
    "\n",
    "We will use word vector representations to build an Emojifier. \n",
    "🤩 💫 🔥\n",
    "\n",
    "Have you ever wanted to make your text messages more expressive? Your emojifier app will help you do that. \n",
    "Rather than writing:\n",
    "> Congratulations on the promotion! Let's get coffee and talk. Love you!  \n",
    "\n",
    "The emojifier can automatically turn this into:\n",
    "> Congratulations on the promotion! 👍  Let's get coffee and talk. ☕️ Love you! ❤️\n",
    "\n",
    "We implement a model that inputs a sentence (such as \"Let's go see the baseball game tonight!\") and finds the most appropriate emoji to be used with this sentence (⚾️).\n",
    "\n",
    "### Using word vectors to improve emoji lookups\n",
    "* In many emoji interfaces, you need to remember that ❤️  is the \"heart\" symbol rather than the \"love\" symbol. \n",
    "    * In other words, you'll have to remember to type \"heart\" to find the desired emoji, and typing \"love\" won't bring up that symbol.\n",
    "* You can make a more flexible emoji interface by using word vectors.\n",
    "* When using word vectors, you'll see that even if your training set explicitly relates only a few words to a particular emoji, your algorithm will be able to generalize and associate additional words in the test set to the same emoji.\n",
    "    * This works even if those additional words don't even appear in the training set. \n",
    "    * This allows you to build an accurate classifier mapping from sentences to emojis, even using a small training set. \n",
    "\n",
    "### Two models\n",
    "1. We start with a baseline model (Emojifier-V1) using word embeddings.\n",
    "2. Then we build a more sophisticated model (Emojifier-V2) that further incorporates an LSTM. \n",
    "\n",
    "By the end of this notebook, you'll be able to:\n",
    "\n",
    "* Create an embedding layer in Keras with pre-trained word vectors\n",
    "* Explain the advantages and disadvantages of the GloVe algorithm\n",
    "* Describe how negative sampling learns word vectors more efficiently than other methods\n",
    "* Build a sentiment classifier using word embeddings\n",
    "* Build and train a more sophisticated classifier using an LSTM\n",
    "\n",
    "🏀 👑\n",
    "\n",
    "👆 😎\n",
    "\n",
    "(^^^ Emoji for \"skills\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HsztVBA8cFIg"
   },
   "source": [
    "<a name='0'></a>\n",
    "## Packages\n",
    "\n",
    "Run the following cell to load the packages we need. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "lMZ9xg8MFHZU"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from emo_utils import *\n",
    "import emoji\n",
    "import matplotlib.pyplot as plt\n",
    "from test_utils import *\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Av0PwZYscFIh"
   },
   "source": [
    "## 1 - Dataset EMOJISET\n",
    "\n",
    "Let's start by building a simple baseline classifier. \n",
    "\n",
    "You have a tiny dataset (X, Y) where:\n",
    "- X contains 127 sentences (strings).\n",
    "- Y contains an integer label between 0 and 4 corresponding to an emoji for each sentence.\n",
    "\n",
    "<img src=\"images/data_set.png\" style=\"width:700px;height:300px;\">\n",
    "<p><caption><center><font color='purple'><b>Figure 1</b>: EMOJISET - a classification problem with 5 classes. A few examples of sentences are given here. </center></caption></p>\n",
    "\n",
    "Load the dataset using the code below. The dataset is split between training (127 examples) and testing (56 examples)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "executionInfo": {
     "elapsed": 2671,
     "status": "ok",
     "timestamp": 1611738624467,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "OvuoZ8pWcFIi"
   },
   "outputs": [],
   "source": [
    "X_train, Y_train = read_csv('data/train_emoji.csv')\n",
    "X_test, Y_test = read_csv('data/tesss.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "executionInfo": {
     "elapsed": 2046,
     "status": "ok",
     "timestamp": 1611738634135,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "DjAuDbxrcFIi"
   },
   "outputs": [],
   "source": [
    "maxLen = len(max(X_train, key=len).split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EpbQIx7dcFIi"
   },
   "source": [
    "Run the following cell to print first 10 sentences from `X_train` and corresponding labels from `Y_train`. \n",
    "* Change `idx` to see different examples. \n",
    "* Due to the font used by iPython notebook, the heart emoji may be colored black rather than red."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2058,
     "status": "ok",
     "timestamp": 1611738637381,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "vE1Zd2SMcFIj",
    "outputId": "49f45ed1-8f2f-4ea8-da44-4acb41731287"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "never talk to me again 😞\n",
      "I am proud of your achievements 😄\n",
      "It is the worst day in my life 😞\n",
      "Miss you so much ❤️\n",
      "food is life 🍴\n",
      "I love you mum ❤️\n",
      "Stop saying bullshit 😞\n",
      "congratulations on your acceptance 😄\n",
      "The assignment is too long  😞\n",
      "I want to go play ⚾\n"
     ]
    }
   ],
   "source": [
    "for idx in range(10):\n",
    "    print(X_train[idx], label_to_emoji(Y_train[idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tS_N2pMpcFIk"
   },
   "source": [
    "## 2 - Baseline model: Emojifier-V1\n",
    "We will first implement a baseline model called \"Emojifier-v1\".  \n",
    "\n",
    "### 2.1 - Overview of the Emojifier-V1\n",
    "\n",
    "<center>\n",
    "<img src=\"images/image_1.png\" style=\"width:900px;height:300px;\">\n",
    "    <caption><center><font color='purple'><b>Figure 2</b>: Baseline model (Emojifier-V1).</center></caption>\n",
    "</center></font>\n",
    "\n",
    "\n",
    "#### Inputs and Outputs\n",
    "* The input of the model is a string corresponding to a sentence (e.g. \"I love you\"). \n",
    "* The output will be a probability vector of shape (1,5), (indicating that there are 5 emojis to choose from).\n",
    "* The (1,5) probability vector is passed to an argmax layer, which extracts the index of the emoji with the highest probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y6nloeF5cFIl"
   },
   "source": [
    "#### One-hot Encoding\n",
    "* To get your labels into a format suitable for training a softmax classifier, convert $Y$ from its current shape  $(m, 1)$ into a \"one-hot representation\" $(m, 5)$, \n",
    "    * Each row is a one-hot vector giving the label of one example.\n",
    "    * Here, `Y_oh` stands for \"Y-one-hot\" in the variable names `Y_oh_train` and `Y_oh_test`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "executionInfo": {
     "elapsed": 2619,
     "status": "ok",
     "timestamp": 1611738660835,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "RhRTRwVncFIm"
   },
   "outputs": [],
   "source": [
    "Y_oh_train = convert_to_one_hot(Y_train, C = 5)\n",
    "Y_oh_test = convert_to_one_hot(Y_test, C = 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2w3GRkw2cFIo"
   },
   "source": [
    "Now, see what `convert_to_one_hot()` did. Feel free to change `index` to print out different values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2838,
     "status": "ok",
     "timestamp": 1611738667164,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "GlHYeuBIcFIo",
    "outputId": "c2b73f6a-9a15-4728-a8b4-7ba38b5372ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence 'I missed you' has label index 0, which is emoji ❤️\n",
      "Label index 0 in one-hot encoding format is [1. 0. 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "idx = 50\n",
    "print(f\"Sentence '{X_train[50]}' has label index {Y_train[idx]}, which is emoji {label_to_emoji(Y_train[idx])}\", )\n",
    "print(f\"Label index {Y_train[idx]} in one-hot encoding format is {Y_oh_train[idx]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gbFECkqAcFIp"
   },
   "source": [
    "All the data is now ready to be fed into the Emojify-V1 model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KI8mJoafcFIp"
   },
   "source": [
    "### 2.2 - Averaging word vectors\n",
    "\n",
    "As shown in Figure 2 (above), the first step is to:\n",
    "* Convert each word in the input sentence into their word vector representations.\n",
    "* Take an average of the word vectors. \n",
    "\n",
    "We will use pre-trained 50-dimensional GloVe embeddings. In the `data` directory, extract `glove.6B.50d.txt.zip` archive so that you have `glove.6B.50d.txt` in the `data` directory.\n",
    "\n",
    "Run the following cell to load the `word_to_vec_map`, which contains all the vector representations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "executionInfo": {
     "elapsed": 8474,
     "status": "ok",
     "timestamp": 1611738705912,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "QXI3avt7cFIq"
   },
   "outputs": [],
   "source": [
    "word_to_index, index_to_word, word_to_vec_map = read_glove_vecs('data/glove.6B.50d.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9JM-0zg6cFIr"
   },
   "source": [
    "You've loaded:\n",
    "- `word_to_index`: dictionary mapping from words to their indices in the vocabulary \n",
    "    - (400,001 words, with the valid indices ranging from 0 to 400,000)\n",
    "- `index_to_word`: dictionary mapping from indices to their corresponding words in the vocabulary\n",
    "- `word_to_vec_map`: dictionary mapping words to their GloVe vector representation.\n",
    "\n",
    "Run the following cell to check if it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1164,
     "status": "ok",
     "timestamp": 1611738710682,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "RB2ZN6ajcFIr",
    "outputId": "3c733016-edf8-417d-f589-828c2a9d8a09"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the index of cucumber in the vocabulary is 113317\n",
      "the 289846th word in the vocabulary is potatos\n"
     ]
    }
   ],
   "source": [
    "word = \"cucumber\"\n",
    "idx = 289846\n",
    "print(\"the index of\", word, \"in the vocabulary is\", word_to_index[word])\n",
    "print(\"the\", str(idx) + \"th word in the vocabulary is\", index_to_word[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wg9QpkR5cFIs"
   },
   "source": [
    "Implement `sentence_to_avg()`. \n",
    "\n",
    "1. Convert every sentence to lower-case, then split the sentence into a list of words. \n",
    "    * `X.lower()` and `X.split()` might be useful. 😉\n",
    "2. For each word in the sentence, access its GloVe representation.\n",
    "    * Then take the average of all of these word vectors.\n",
    "    * You might use `numpy.zeros()`, which you can read more about [here]('https://numpy.org/doc/stable/reference/generated/numpy.zeros.html').\n",
    "    \n",
    "    \n",
    "#### Additional Hints\n",
    "* When creating the `avg` array of zeros, you'll want it to be a vector of the same shape as the other word vectors in the `word_to_vec_map`.  \n",
    "    * You can choose a word that exists in the `word_to_vec_map` and access its `.shape` field.\n",
    "    * Be careful not to hard-code the word that you access.  In other words, don't assume that if you see the word 'the' in the `word_to_vec_map` within this notebook, that this word will be in the `word_to_vec_map` when the function is being called by the automatic grader.\n",
    "\n",
    "**Hint**: you can use any one of the word vectors that you retrieved from the input `sentence` to find the shape of a word vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "executionInfo": {
     "elapsed": 1943,
     "status": "ok",
     "timestamp": 1611738728468,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "buYjsIBecFIs"
   },
   "outputs": [],
   "source": [
    "def sentence_to_avg(sentence, word_to_vec_map):\n",
    "    \"\"\"\n",
    "    Converts a sentence (string) into a list of words (strings). Extracts the GloVe representation of each word\n",
    "    and averages its value into a single vector encoding the meaning of the sentence.\n",
    "    \n",
    "    Arguments:\n",
    "    sentence -- string, one training example from X\n",
    "    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation\n",
    "    \n",
    "    Returns:\n",
    "    avg -- average vector encoding information about the sentence, numpy-array of shape (J,), where J can be any number\n",
    "    \"\"\"\n",
    "    \n",
    "    # Get a valid word contained in the word_to_vec_map. \n",
    "    any_word = list(word_to_vec_map.keys())[0]\n",
    "    \n",
    "    # Step 1: Split sentence into list of lower case words\n",
    "    words = sentence.lower().split()\n",
    "\n",
    "    # Initialize the average word vector, should have the same shape as your word vectors.\n",
    "    avg = np.zeros(word_to_vec_map[any_word].shape)\n",
    "    \n",
    "    # Initialize count to 0\n",
    "    count = 0\n",
    "    \n",
    "    # Step 2: average the word vectors. You can loop over the words in the list \"words\".\n",
    "    for w in words:\n",
    "        # Check that word exists in word_to_vec_map\n",
    "        if w in word_to_vec_map.keys():\n",
    "            avg += word_to_vec_map[w]\n",
    "            # Increment count\n",
    "            count +=1\n",
    "          \n",
    "    if count > 0:\n",
    "        # Get the average. But only if count > 0\n",
    "        avg /= count\n",
    "        \n",
    "    return avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1861,
     "status": "ok",
     "timestamp": 1611738734359,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "7OwW-r6ecFIt",
    "outputId": "7ed0ff55-10f4-4072-fdff-2edc216373fc",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "avg = \n",
      " [-0.008005    0.56370833 -0.50427333  0.258865    0.55131103  0.03104983\n",
      " -0.21013718  0.16893933 -0.09590267  0.141784   -0.15708967  0.18525867\n",
      "  0.6495785   0.38371117  0.21102167  0.11301667  0.02613967  0.26037767\n",
      "  0.05820667 -0.01578167 -0.12078833 -0.02471267  0.4128455   0.5152061\n",
      "  0.38756167 -0.898661   -0.535145    0.33501167  0.68806933 -0.2156265\n",
      "  1.797155    0.10476933 -0.36775333  0.750785    0.10282583  0.348925\n",
      " -0.27262833  0.66768    -0.10706167 -0.283635    0.59580117  0.28747333\n",
      " -0.3366635   0.23393817  0.34349183  0.178405    0.1166155  -0.076433\n",
      "  0.1445417   0.09808667]\n",
      "\u001b[92mAll tests passed!\n"
     ]
    }
   ],
   "source": [
    "# BEGIN UNIT TEST\n",
    "avg = sentence_to_avg(\"Morrocan couscous is my favorite dish\", word_to_vec_map)\n",
    "print(\"avg = \\n\", avg)\n",
    "\n",
    "def sentence_to_avg_test(target):\n",
    "    # Create a controlled word to vec map\n",
    "    word_to_vec_map = {'a': [3, 3], 'synonym_of_a': [3, 3], 'a_nw': [2, 4], 'a_s': [3, 2], \n",
    "                       'c': [-2, 1], 'c_n': [-2, 2],'c_ne': [-1, 2], 'c_e': [-1, 1], 'c_se': [-1, 0], \n",
    "                       'c_s': [-2, 0], 'c_sw': [-3, 0], 'c_w': [-3, 1], 'c_nw': [-3, 2]\n",
    "                      }\n",
    "    # Convert lists to np.arrays\n",
    "    for key in word_to_vec_map.keys():\n",
    "        word_to_vec_map[key] = np.array(word_to_vec_map[key])\n",
    "        \n",
    "    avg = target(\"a a_nw c_w a_s\", word_to_vec_map)\n",
    "    assert tuple(avg.shape) == tuple(word_to_vec_map['a'].shape),  \"Check the shape of your avg array\"  \n",
    "    assert np.allclose(avg, [1.25, 2.5]),  \"Check that you are finding the 4 words\"\n",
    "    avg = target(\"love a a_nw c_w a_s\", word_to_vec_map)\n",
    "    assert np.allclose(avg, [1.25, 2.5]), \"Divide by count, not len(words)\"\n",
    "    avg = target(\"love\", word_to_vec_map)\n",
    "    assert np.allclose(avg, [0, 0]), \"Average of no words must give an array of zeros\"\n",
    "    avg = target(\"c_se foo a a_nw c_w a_s deeplearning c_nw\", word_to_vec_map)\n",
    "    assert np.allclose(avg, [0.1666667, 2.0]), \"Debug the last example\"\n",
    "    \n",
    "    print(\"\\033[92mAll tests passed!\")\n",
    "    \n",
    "sentence_to_avg_test(sentence_to_avg)\n",
    "\n",
    "# END UNIT TEST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NPPv5gmucFIv"
   },
   "source": [
    "### 2.3 - Implementing the model\n",
    "\n",
    "You now have all the pieces to finish implementing the `model()` function. \n",
    "After using `sentence_to_avg()` you need to:\n",
    "* Pass the average through forward propagation\n",
    "* Compute the cost\n",
    "* Backpropagate to update the softmax parameters\n",
    "\n",
    "Implement the `model()` function described in Figure (2). \n",
    "\n",
    "* The equations you need to implement in the forward pass and to compute the cross-entropy cost are below:\n",
    "* The variable $Y_{oh}$ (\"Y one hot\") is the one-hot encoding of the output labels. \n",
    "\n",
    "$$ z^{(i)} = Wavg^{(i)} + b$$\n",
    "\n",
    "$$ a^{(i)} = softmax(z^{(i)})$$\n",
    "\n",
    "$$ \\mathcal{L}^{(i)} = - \\sum_{k = 0}^{n_y - 1} Y_{oh,k}^{(i)} * log(a^{(i)}_k)$$\n",
    "\n",
    "**Note**: It is possible to come up with a more efficient vectorized implementation. For now, just use nested for loops to better understand the algorithm, and for easier debugging.\n",
    "\n",
    "The function `softmax()` is provided, and has already been imported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "executionInfo": {
     "elapsed": 2322,
     "status": "ok",
     "timestamp": 1611738741724,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "O_BzrO-TcFIv"
   },
   "outputs": [],
   "source": [
    "def model(X, Y, word_to_vec_map, learning_rate = 0.01, num_iterations = 400):\n",
    "    \"\"\"\n",
    "    Model to train word vector representations in numpy.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- input data, numpy array of sentences as strings, of shape (m, 1)\n",
    "    Y -- labels, numpy array of integers between 0 and 7, numpy-array of shape (m, 1)\n",
    "    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation\n",
    "    learning_rate -- learning_rate for the stochastic gradient descent algorithm\n",
    "    num_iterations -- number of iterations\n",
    "    \n",
    "    Returns:\n",
    "    pred -- vector of predictions, numpy-array of shape (m, 1)\n",
    "    W -- weight matrix of the softmax layer, of shape (n_y, n_h)\n",
    "    b -- bias of the softmax layer, of shape (n_y,)\n",
    "    \"\"\"\n",
    "    \n",
    "    # Get a valid word contained in the word_to_vec_map \n",
    "    any_word = list(word_to_vec_map.keys())[0]\n",
    "        \n",
    "    # Initialize cost. It is needed during grading\n",
    "    cost = 0\n",
    "    \n",
    "    # Define number of training examples\n",
    "    m = Y.shape[0]                             # number of training examples\n",
    "    n_y = len(np.unique(Y))                    # number of classes  \n",
    "    n_h = word_to_vec_map[any_word].shape[0]   # dimensions of the GloVe vectors \n",
    "    \n",
    "    # Initialize parameters using Xavier initialization\n",
    "    W = np.random.randn(n_y, n_h) / np.sqrt(n_h)\n",
    "    b = np.zeros((n_y,))\n",
    "    \n",
    "    # Convert Y to Y_onehot with n_y classes\n",
    "    Y_oh = convert_to_one_hot(Y, C = n_y) \n",
    "    \n",
    "    # Optimization loop\n",
    "    for t in range(num_iterations): # Loop over the number of iterations\n",
    "        for i in range(m):          # Loop over the training examples\n",
    "            \n",
    "            # Average the word vectors of the words from the i'th training example\n",
    "            avg = sentence_to_avg(X[i], word_to_vec_map)\n",
    "\n",
    "            # Forward propagate the avg through the softmax layer. \n",
    "            # You can use np.dot() to perform the multiplication.\n",
    "            z = np.dot(W, avg) + b\n",
    "            a = softmax(z)\n",
    "\n",
    "            # Compute cost using the i'th training label's one hot representation and \"A\" (the output of the softmax)\n",
    "            cost = -np.sum(Y_oh[i] * np.log(a))\n",
    "            \n",
    "            # Compute gradients \n",
    "            dz = a - Y_oh[i]\n",
    "            dW = np.dot(dz.reshape(n_y,1), avg.reshape(1, n_h))\n",
    "            db = dz\n",
    "\n",
    "            # Update parameters with Stochastic Gradient Descent\n",
    "            W = W - learning_rate * dW\n",
    "            b = b - learning_rate * db\n",
    "        \n",
    "        if t % 100 == 0:\n",
    "            print(\"Epoch: \" + str(t) + \" --- cost = \" + str(cost))\n",
    "            pred = predict(X, Y, W, b, word_to_vec_map) #predict is defined in emo_utils.py\n",
    "\n",
    "    return pred, W, b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 --- cost = 0.05105772513207823\n",
      "Accuracy: 0.9166666666666666\n",
      "Epoch: 100 --- cost = 0.00970311068897676\n",
      "Accuracy: 1.0\n",
      "\u001b[92mAll tests passed!\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def model_test(target):\n",
    "    # Create a controlled word to vec map\n",
    "    word_to_vec_map = {'a': [3, 3], 'synonym_of_a': [3, 3], 'a_nw': [2, 4], 'a_s': [3, 2], 'a_n': [3, 4], \n",
    "                       'c': [-2, 1], 'c_n': [-2, 2],'c_ne': [-1, 2], 'c_e': [-1, 1], 'c_se': [-1, 0], \n",
    "                       'c_s': [-2, 0], 'c_sw': [-3, 0], 'c_w': [-3, 1], 'c_nw': [-3, 2]\n",
    "                      }\n",
    "    # Convert lists to np.arrays\n",
    "    for key in word_to_vec_map.keys():\n",
    "        word_to_vec_map[key] = np.array(word_to_vec_map[key])\n",
    "        \n",
    "    # Training set. Sentences composed of a_* words will be of class 0 and sentences composed of c_* words will be of class 1\n",
    "    X = np.asarray(['a a_s synonym_of_a a_n c_sw', 'a a_s a_n c_sw', 'a_s  a a_n', 'synonym_of_a a a_s a_n c_sw', \" a_s a_n\",\n",
    "                    \" a a_s a_n c \", \" a_n  a c c c_e\",\n",
    "                   'c c_nw c_n c c_ne', 'c_e c c_se c_s', 'c_nw c a_s c_e c_e', 'c_e a_nw c_sw', 'c_sw c c_ne c_ne'])\n",
    "    \n",
    "    Y = np.asarray([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1])\n",
    "    \n",
    "    np.random.seed(10)\n",
    "    pred, W, b = model(X, Y, word_to_vec_map, 0.0025, 110)\n",
    "    \n",
    "    assert W.shape == (2, 2), \"W must be of shape 2 x 2\"\n",
    "    assert np.allclose(pred.transpose(), Y), \"Model must give a perfect accuracy\"\n",
    "    assert np.allclose(b[0], -1 * b[1]), \"b should be symmetric in this example\"\n",
    "    \n",
    "    print(\"\\033[92mAll tests passed!\")\n",
    "    \n",
    "model_test(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KvdG05pgcFIw"
   },
   "source": [
    "Run the next cell to train your model and learn the softmax parameters (W, b). The training process will take about 5 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3817,
     "status": "ok",
     "timestamp": 1611738757775,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "umWTqRcpcFIw",
    "outputId": "7b74cb94-e98c-4936-98bc-693a3bb5f34e",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 --- cost = 1.9520498812810076\n",
      "Accuracy: 0.3484848484848485\n",
      "Epoch: 100 --- cost = 0.07971818726014807\n",
      "Accuracy: 0.9318181818181818\n",
      "Epoch: 200 --- cost = 0.04456369243681402\n",
      "Accuracy: 0.9545454545454546\n",
      "Epoch: 300 --- cost = 0.03432267378786059\n",
      "Accuracy: 0.9696969696969697\n",
      "[[3.]\n",
      " [2.]\n",
      " [3.]\n",
      " [0.]\n",
      " [4.]\n",
      " [0.]\n",
      " [3.]\n",
      " [2.]\n",
      " [3.]\n",
      " [1.]\n",
      " [3.]\n",
      " [3.]\n",
      " [1.]\n",
      " [3.]\n",
      " [2.]\n",
      " [3.]\n",
      " [2.]\n",
      " [3.]\n",
      " [1.]\n",
      " [2.]\n",
      " [3.]\n",
      " [0.]\n",
      " [2.]\n",
      " [2.]\n",
      " [2.]\n",
      " [1.]\n",
      " [4.]\n",
      " [3.]\n",
      " [3.]\n",
      " [4.]\n",
      " [0.]\n",
      " [3.]\n",
      " [4.]\n",
      " [2.]\n",
      " [0.]\n",
      " [3.]\n",
      " [2.]\n",
      " [2.]\n",
      " [3.]\n",
      " [4.]\n",
      " [2.]\n",
      " [2.]\n",
      " [0.]\n",
      " [2.]\n",
      " [3.]\n",
      " [0.]\n",
      " [3.]\n",
      " [2.]\n",
      " [4.]\n",
      " [3.]\n",
      " [0.]\n",
      " [3.]\n",
      " [3.]\n",
      " [3.]\n",
      " [4.]\n",
      " [2.]\n",
      " [1.]\n",
      " [1.]\n",
      " [1.]\n",
      " [2.]\n",
      " [3.]\n",
      " [1.]\n",
      " [0.]\n",
      " [0.]\n",
      " [0.]\n",
      " [3.]\n",
      " [4.]\n",
      " [4.]\n",
      " [2.]\n",
      " [2.]\n",
      " [1.]\n",
      " [2.]\n",
      " [0.]\n",
      " [3.]\n",
      " [2.]\n",
      " [2.]\n",
      " [0.]\n",
      " [3.]\n",
      " [3.]\n",
      " [1.]\n",
      " [2.]\n",
      " [1.]\n",
      " [2.]\n",
      " [2.]\n",
      " [4.]\n",
      " [3.]\n",
      " [3.]\n",
      " [2.]\n",
      " [4.]\n",
      " [0.]\n",
      " [0.]\n",
      " [3.]\n",
      " [3.]\n",
      " [3.]\n",
      " [3.]\n",
      " [2.]\n",
      " [0.]\n",
      " [1.]\n",
      " [2.]\n",
      " [3.]\n",
      " [0.]\n",
      " [2.]\n",
      " [2.]\n",
      " [2.]\n",
      " [3.]\n",
      " [2.]\n",
      " [2.]\n",
      " [2.]\n",
      " [4.]\n",
      " [1.]\n",
      " [1.]\n",
      " [3.]\n",
      " [3.]\n",
      " [4.]\n",
      " [1.]\n",
      " [2.]\n",
      " [1.]\n",
      " [1.]\n",
      " [3.]\n",
      " [1.]\n",
      " [0.]\n",
      " [4.]\n",
      " [0.]\n",
      " [3.]\n",
      " [3.]\n",
      " [4.]\n",
      " [4.]\n",
      " [1.]\n",
      " [4.]\n",
      " [3.]\n",
      " [0.]\n",
      " [2.]]\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "pred, W, b = model(X_train, Y_train, word_to_vec_map)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ygumNDIUcFIx"
   },
   "source": [
    "Your model has pretty high accuracy on the training set. Now see how it does on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O862gcUicFIx"
   },
   "source": [
    "### 2.4 - Examining test set performance \n",
    "\n",
    "The `predict` function used here is defined in `emo_utils.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3704,
     "status": "ok",
     "timestamp": 1611738776291,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "yhb6CzhrcFIx",
    "outputId": "08d07fd0-55c2-4eff-d570-2562deb0570b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set:\n",
      "Accuracy: 0.9772727272727273\n",
      "Test set:\n",
      "Accuracy: 0.8571428571428571\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set:\")\n",
    "pred_train = predict(X_train, Y_train, W, b, word_to_vec_map)\n",
    "print('Test set:')\n",
    "pred_test = predict(X_test, Y_test, W, b, word_to_vec_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hwmrm-aDcFIy"
   },
   "source": [
    "**Note**:\n",
    "* Random guessing would have had 20% accuracy, given that there are 5 classes. (1/5 = 20%).\n",
    "* This is pretty good performance after training on only 127 examples. \n",
    "\n",
    "\n",
    "#### The model matches emojis to relevant words.\n",
    "- In the training set, the algorithm saw the sentence \n",
    "  > I love you.\n",
    "\n",
    "  with the label ❤️. \n",
    "* The word \"adore\" does not appear in the training set. \n",
    "* Nonetheless, let's see what happens if you write \"I adore you.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1909,
     "status": "ok",
     "timestamp": 1611738785398,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "tvCl7fUvcFIz",
    "outputId": "a3913da3-85df-466d-dd8f-520245d495fc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.8333333333333334\n",
      "\n",
      "i adore you ❤️\n",
      "i love you ❤️\n",
      "funny lol 😄\n",
      "lets play with a ball ⚾\n",
      "food is ready 🍴\n",
      "not feeling happy 😄\n"
     ]
    }
   ],
   "source": [
    "X_my_sentences = np.array([\"i adore you\", \"i love you\", \"funny lol\", \"lets play with a ball\", \"food is ready\", \"not feeling happy\"])\n",
    "Y_my_labels = np.array([[0], [0], [2], [1], [4],[3]])\n",
    "\n",
    "pred = predict(X_my_sentences, Y_my_labels , W, b, word_to_vec_map)\n",
    "print_predictions(X_my_sentences, pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZyC-BGqKcFI0"
   },
   "source": [
    "* Because *adore* has a similar embedding as *love*, the algorithm has generalized correctly even to a word it has never seen before. \n",
    "* Words such as *heart*, *dear*, *beloved* or *adore* have embedding vectors similar to *love*. \n",
    "    * Feel free to modify the inputs above and try out a variety of input sentences. \n",
    "    * How well does it work?\n",
    "\n",
    "#### Word ordering is not considered in this model.\n",
    "* Note that the model doesn't get the following sentence correct:\n",
    "> not feeling happy \n",
    "\n",
    "* This algorithm ignores word ordering, so is not good at understanding phrases like \"not happy.\" \n",
    "\n",
    "#### Confusion matrix\n",
    "* Printing the confusion matrix can also help understand which classes are more difficult for your model. \n",
    "* A confusion matrix shows how often an example whose label is one class (\"actual\" class) is mislabeled by the algorithm with a different class (\"predicted\" class).\n",
    "\n",
    "Print the confusion matrix below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 433
    },
    "executionInfo": {
     "elapsed": 2061,
     "status": "ok",
     "timestamp": 1611738816883,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "Ab9aH9IQcFI1",
    "outputId": "d8cfc4cc-bbdc-487b-8efc-3d9a3cdbdd06"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(56,)\n",
      "           ❤️    ⚾    😄    😞   🍴\n",
      "Predicted  0.0  1.0  2.0  3.0  4.0  All\n",
      "Actual                                 \n",
      "0            6    0    0    1    0    7\n",
      "1            0    8    0    0    0    8\n",
      "2            2    0   16    0    0   18\n",
      "3            1    1    2   12    0   16\n",
      "4            0    0    1    0    6    7\n",
      "All          9    9   19   13    6   56\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQwAAAD2CAYAAAAj8rlYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAYVklEQVR4nO3de7hddX3n8ffn5EKigBISMEAwmSEKGarIE1Mr1YnSMlwcQCwWOrR5Ki3olBYvraLPM6NO25GpHeqlWglICXKTKSKUcm0kXBwEEkACRCQPptzCJSByGSAN+cwfa52yOZ6cs/Y5a++99jmf1/Ps5+y19trr+1vnnP3dv/Vb6/f7yTYREVUM9LoAEdE/kjAiorIkjIioLAkjIipLwoiIypIwIqKyJIyIqCwJIyIqm9rrAnSSpCXANGCL7Vt6VIYB21u7EKcnxzqZ4kqSJ/mdjhO2hiHpPwGXAYcBF0g6SdL2XYh7mKQvSvqSpJ27lCx6dayTKi4wvYzflc+NJLfxuKobZcL2hHoAArYDzgY+XK7bD7gW+FNgZgdj/yrwM+B3gG8BPwTeDUybSMc62eKWcRYC/wC8uVwe6FSslpiVEwawutPlsT3xahguvAysA94maXvbdwIfBw4FPtLB8PsC19g+3/ZHgYuBTwP7Q/3fTL061skWt/QY8C/AlyTNs721GzUNSZUe3TLhEkaLu4CdgX8vaarte4A/Az4p6e0dinkbMFPS3gC2TwNuAr4i6Y3u3OlJL451UsSV9CuSLrH9HPAFYAPwv7uVNJIwOkzlb8/2lcDzwMnAvuW30RrgKoqqbSc8BmwBflPS7LIcfw3cDZzYoZi9Otaux5U0pQdxN1CcGny3TBpfAtbThaQhiYGBgUqPblF5rtTXJL0VmAWsBrbafqXltb8CdgBeAh4CPgUcYHtDTbGnDIn3DuAvKP55V9leK+mUslx/VUO8vYA3AnfbfmnIax07Vkn/AZgNrLP9RBfj/jqwwPZ3yuVptv+1C3HfZPux8vl2wN8D29n+kKQdgM8C84HP1fW/NNTAwICnTZtWadvNmzevsb24E+Vo1fcJQ9JRwP8EHikfq4GzbT/bss37gLcBbwG+YfveGuK+xfZPy+dTbL8yeNmtTBonUnywDSwBjrS9dpwxP0BxrE9R1Gb+0vbdQz5EnTjWQ4D/BTxAcSnzBNuPlKcDWzoRt/zWfh1wC0Wt4Wu2v1W+NmMwWXboePcG7gW+Ctxr+wxJrwe+AsyxfWSZNP4c2JHi97FlvHGHGhgY8PTp0ytt+/LLLydhjEbSNOBcin+mH0r6EPAu4GXgy7Z/MWT7qXX8YcsP7kXA923/TrluMGkMlNXU2cBOwDuBm23/bJwx3w2cBRxr+w5J3wRm2P5I+fpr7veo8ViXAsuB42zfKukSig/mPw+NWWfclv19GngFeDtwh+2/2cZ2tcWVNA+4kOLS7YHARuC7FKeWnwD2LGsaO1LUOp6sI+5QAwMD3m677Spt+9JLL3UlYUyENowdKS55AVwCXE5xvfxYAEnvknRY+forv/z29pTfNCdRtMxvlnQuQJksprZ8gLbYvr+8YjKuZNHiVNt3lM8/D8wqq8uUSeqdZTKDGo619DhwYpks3kRx6fgkSacDvwdQxq3tdzzEFmAesAJYIuk0SV8q4/56J+Lafgi4leLq1qEUp5cnAOcAZwLzJH3N9rOdShaD0uhZo7IafhpwlKT3lB/Wm4A7gfeUH6Y9gdvL7cddnbL9AsXlu/Mprv3PaEkag9XztwPHSZqh+v6atwDfK/c/heJ+hDdTJEwk7QHsTXFKVsuxlvtZZ/u6cvF44Ju2jwR+BBxafhsvoMbf8RCXAo/ZXklxbB8D3lC+9qa647b8vT5DcTo5m6KG8XbgfuC/UzR6frOOeKOUpXEJo69PSaA4nwX+gOI89lzbN5Trr6P4Zvxph+PvTFFlf9H2cZLeRlHjuXFo42CNMacCM4BLbR8o6TjgHcAXypb8rpB0JfBJ2+s6GGM34C+B/0txT8t3KNqELrJ9TodiiqKW+t+Af0dR0zjF9vclLQQ22f55J2K3mjJlimfOnFlp2xdeeKErpyR935fE9kuSzqP4Nvhs2WD1MrAL8IsR31xP/KcknQh8WdJ9FLW293YqWZQxtwDPS3qorJ4fBPx+J5PFYINuy/KHKH7HHf3g2H5U0kMUH94/sv2PZUPn+g7GNPCypO8ANwJft/398rX7OxV3ON28ZFpF3ycMANs/l3QGRcv2iRSX2Y6z/XiX4m+SdBdwCPCbtjd2Ml75DTgNeE/588BO/yMPJovyNO844JPAbw9eeuywMyhqU2vK5euHNrZ2gu37JH0GeLOk19n+f52OOVQ3TzeqmBAJA8D2ZuA6STcUi53/hxokaSeKxrGDxnvptIryw7tZ0p8Dt3X5W28rxTn9Ubbv60bAshHyocFaTjf/tsDNwFFdjPdvut0+UUXft2E0Reu9AV2MOem7W3dDr2oXU6dO9Q477FBp22eeeSZtGP2k28mijJlk0QW9SBaDmlbDSMKIaLAkjIioLAkjIipR2Vu1SZpVmg6QdMJkiJm4EzNu0+70nPAJg6IPwGSImbgTMG6dCUPSBklrJd0paXW5bpakayXdX/7caaR9TIaEEdG3OlDDeJ/t/VouwZ4CrLS9EFhZLm+7PP1wZW7WrFmeN2/emN771FNPsfPOO4/pvVUHLxnqySefZM6cOWN673iMJ+54/g82bdrE7Nmzx/Te8VSnx3O8mzdvHnPcsf5PPfzwwzz99NOVD3j69Omu+nvduHHjqPdhSNoALLa9qWXdfcBS2xslzaUY9Omt29pHXzR6zps3jyuuuKLrcXffffeux+yVLVtqH/+lkqlTe/MvuGHDhq7HPPzww9t+T83tEwauUTHK+Om2lwO7DnZlKJPGLiPtoC8SRsRk1UbCmD3YLlFaXiaEVgeUnfl2Aa6V9JN2y5OEEdFgbVxW3TTaKYntR8ufT6gYOW0J8LikuS2nJCP2sk6jZ0RD1TmAjqTXqxiHdHDUuIMohhy8DFhWbraMYsCibUoNI6LBamzD2BW4pNzfVOB821dJug24SNLxwIPA0SPtJAkjosHqShi2H6AYZnDo+qcoBjquJAkjosHSlyQiKkvCiIhKmtj5LAkjosGaVsPoSfqSdLCk+yStVzHvaEQMY9L3VlUxCc83KEbYXgQcK2lRt8sR0Q8mfcKguLtsve0HypG+LwSO6EE5Ihqtzhu36tKLhLE78FDL8sPluogYomkJoxeNnsMd3S/1rS5HNToBJlev0YhWafQsahStg1vsATw6dCPby20vtr14rONZRPS7gYGBSo+uladrkV51G7BQ0gJJ04FjKDrARESLJrZhdP2UxPYWSScBVwNTgLNs39PtckT0g6adkvTkxi3bVwDdH0Iros8kYUREZUkYEVFZEkZEVNLtBs0qkjAiGiy9VSOistQwIqKyJIyIqCRtGBHRliSMiKgsCWMMpk2b1pMeq+vXr+96TIC99tqr6zF7Ncdpr/RiLtmxTHidhBERlWQQ4IhoS2oYEVFZEkZEVJaEERGVJWFERCW5cSsi2pKEERGVNe2yarNKExGvUecgwJKmSLpD0uXl8ixJ10q6v/y502j7SMKIaKgOjBp+MrCuZfkUYKXthcDKcnlESRgRDVZXwpC0B3AYcGbL6iOAFeXzFcCRo+2nV7O3nyXpCUl39yJ+RL9oI2HMlrS65XHCkF19Bfg0sLVl3a62NwKUP3cZrTy9avQ8G/hb4JwexY/oC22cbmyyvXgb+/gA8ITtNZKWjqc8vZqX5AZJ83sRO6Jf1Nj57ADgcEmHAjOAHSWdCzwuaa7tjZLmAk+MtqO0YUQ0WB1tGLY/a3sP2/Mppib9ge3jKKYoXVZutgy4dLTyNPY+jNbZ2/fcc88elyaiNzp849apwEWSjgceBI4e7Q2NTRi2lwPLARYvXtz+yCMRE0DdCcP2KmBV+fwp4MB23t/YhBERzbs1vFeXVS8AbgbeKunhskoUES06cOPWuPXqKsmxvYgb0W+aVsPIKUlEgzWt81kSRkRDZTyMiGhLEkZEVJaEERGVJWFERGVJGBFRSRo9I6ItuawaEZWlhjEGW7du5cUXX+x63F7Mog5w5ZVXdj3mIYcc0vWYvXTXXXd1PeZY/oeTMCKikrRhRERbkjAiorIkjIioLAkjIiqpcRDg2iRhRDRYahgRUVkSRkRUloQREZUlYUREJblxKyLa0rSE0fVrNpLmSbpO0jpJ90g6udtliOgXAwMDlR7d0osaxhbgU7Zvl7QDsEbStbbv7UFZIhqtaTWMricM2xuBjeXz5yStA3YHkjAiWqQNYwhJ84F3ALcM89q/TcY8b968rpYroimaljB6dt+ppO2Bi4GP23526Ou2l9tebHvx7Nmzu1/AiAbIVImApGkUyeI829/rRRki+kHTahjbTBiSvg54W6/b/pOxBFTxG/g2sM72aWPZR8Rk0G+dz1Z3KOYBwO8CayXdWa77nO0rOhQvom/VUcOQNAO4AdiO4jP/D7Y/L2kW8F1gPrAB+LDtn4+0r20mDNsrxl3S4fd7E9CselZEQ9V0SvIy8H7bz5fNATdJuhI4Clhp+1RJpwCnAJ8ZaUejtmFImlPuZBEwY3C97feP4wAiooI6EoZtA8+Xi9PKh4EjgKXl+hXAKkZJGFVOkM4D1gELgC9SVF1ua6/IETEWbVwlmS1pdcvjhCH7mVI2ATwBXGv7FmDX8r6owfujdhmtPFWukuxs+9uSTrZ9PXC9pOvbPfCIaE+bl0w32V68rRdtvwLsJ+mNwCWS9h1LmaokjH8tf26UdBjwKLDHWIJFRHvqvqxq+xlJq4CDgcclzbW9UdJcitrHiKqckvyFpDcAnwL+FDgT+MQ4yhwRFdXR+UzSnLJmgaSZwG8APwEuA5aVmy0DLh2tPKPWMGxfXj79BfC+0baPiPrUVMOYC6yQNIWiknCR7csl3QxcJOl44EHg6NF2VOUqyd8zzA1ctj/SdrEjorK6bvu2fRdFn62h658CDmxnX1XaMC5veT4D+CBFO0ZEdFjf3Bo+yPbFrcuSLgD+uWMlGoYkpk2b1s2QAGzZsqXrMQGWLl3a9Zi33npr12MCLFmypCdxZ86c2fWYY/nw913CGMZCYM+6CxIRv6zvEoak53htG8ZjjHI3WETUo+8Shu0dulGQiHitJvZWHbU0klZWWRcR9eubAXTKLrGvo7hHfSde7WG6I7BbF8oWMen10ynJicDHKZLDGl5NGM8C3+hwuSKCPkoYtr8KfFXSH9v+ehfLFBE0c9TwKi0qWwfvQweQtJOk/9rBMkVEqWltGFUSxh/afmZwoRzC6w87V6SIGNS0hFHlxq0BSSpH7aHswDK9s8WKCKBxl1WrJIyrKXq0fYviBq6PAld2tFQR0cg2jCoJ4zMUM5B9jOJKyR0U3WUjosOaljBGre/Y3gr8CHgAWEzRHXbdWANKmiHpVkk/VjF7+xfHuq+Iia5v2jAkvQU4BjgWeIpi/gJsj3cQnWGHPLf9o3HuN2LCaVoNY6RTkp8ANwL/2fZ6AEnjHppvhCHPI2KIpiWMkU5JPkTRM/U6SWdIOpCaJiDaxpDnQ7c5YXDI9E2bNtURNqKvVD0dacR9GLYvsf3bwN4UE5x8AthV0t9JOmg8QW2/Yns/itHHlww35Hlmb4+oZxDgWssz2ga2X7B9nu0PUHzA76SYUm3cyhvCVlEMeR4RQ/RNDWM4tp+2ffp4pkkcYcjziBiiaQljLEP0jdewQ573oBwRjdavN27ValtDnkfEL5v0CSMiqkvCiIjK+rHzWUT0QNowIqItSRgRUVkSRkRUloQREZUlYUREJWn0HCNJTJ3aF0XtW72aRf2RRx7pSdx99tmn6zHHMmN8HZdVJc0DzgHeBGwFltv+qqRZFOPczAc2AB8uB/nednnGXZqI6Jia+pJsAT5lex/gXcAfSVpE0Yl0pe2FwEoqdCpNwohoqLrGw7C90fbt5fPnKIbY3B04AlhRbrYCOHK0MqWeH9FgbbRhzJa0umV5ue3lw+xvPkVfrluAXW1vhCKpSNpltCBJGBEN1kbC2GR78Sj72h64GPi47WfH0qCaU5KIBqtrPIxywO2LgfNsf69c/bikueXrcymGzBxREkZEg9WRMFRs8G1gne3TWl66DFhWPl8GXDpaeXJKEtFQkurqrXoA8LvA2nLwbYDPAadSzGp4PPAgcPRoO0rCiGiwOm7csn0T2x7x/8B29pWEEdFgudMzIipLwoiIStKXJCLa0rSE0bPLquV0iXdIyhQDEduQeUledTLFPe079rAMEY3WtEGAe1IaSXsAhwFn9iJ+RD+oq/NZnXqVvr4CfJqib/6w1DJ7+5NPPtm9kkU0yKRPGJI+ADxhe81I27XO3j5nzpwulS6iWZqWMHrRhnEAcLikQ4EZwI6SzrV9XA/KEtFok/4qie3P2t7D9nzgGOAHSRYRw0sNIyIqyY1bQ9heBazqZRkimqxpl1VTw4hosNQwIqKyJIyIqCRtGBHRliSMiKgsCSMiKstVkoioJG0YEdGWJIwxeOmll1i3bl2vi9E1a9eu7XrM3XbbresxARYsWDCp4rYrCSMiKkvCiIjKkjAiopI0ekZEW3JZNSIqSw0jIipLwoiIStKGERFtScKIiMqaljCa1QQbEa9R1yDAks6S9ISku1vWzZJ0raT7y587jbafJIyIhpLEwMBApUcFZwMHD1l3CrDS9kJgZbk8oo4mDEkflGRJe5fL8wcznKSlmYg5YmR11TBs3wA8PWT1EcCK8vkK4MjR9tPpGsaxwE0U849ERJvaSBizB6cWLR8nVNj9rrY3ApQ/dxntDR1r9JS0PcUsZ+8DLgO+0KlYERNVG42em2wv7mRZoLM1jCOBq2z/FHha0v4djBUxIXV45rPHJc0t48wFnhjtDZ1MGMcCF5bPLyyXK2udvf3pp4eeekVMfFWTxTgSxmXAsvL5MuDS0d7QkVMSSTsD7wf2lWRgCmDgm1X3YXs5sBxg3333dSfKGdF0dd2HIekCYClFW8fDwOeBU4GLJB0PPAgcPdp+OtWG8VvAObZPHFwh6Xpgjw7Fi5iQ6uqtantbNfwD29lPp05JjgUuGbLuYuBzHYoXMSFNitnbbS8dZt3XgK+1LK8iEzFHbFM6n0VEW5IwIqKyJIyIqCwJIyIqS8KIiEoGe6s2SRJGRIOlhhERlSVhRERlSRgRUUlu3Bqje+65Z9OiRYv+ZYxvnw1sqrM8DY2ZuM2P++Z235CEMQa254z1vZJWd2NgkV7HTNyJGTcJIyIqy2XViKgkbRi9sXySxEzcCRi3aQmjWfWdDihH7powMSW9IulOSXdL+j+SXjfWuJLOlvRb5fMzJS0aYdulkt493GsjxZW0QdLsdspVVS/+tt2O27TxMCZ8wpiAXrS9n+19gc3AR1tflDRlLDu1/Qe27x1hk6XAsAkjOicJI+p0I7BX+e1/naTzgbWSpkj6sqTbJN0l6UQAFf5W0r2S/omWeSgkrZK0uHx+sKTbJf1Y0kpJ8ykS0yfK2s17JM2RdHEZ4zZJB5Tv3VnSNZLukHQ60Kw6dZ9pWsKYDG0YE5KkqcAhwFXlqiXAvrZ/pmISm1/Yfqek7YAfSroGeAfwVuBXgF2Be4Gzhux3DnAG8N5yX7NsPy3pW8Dztv+63O584G9s3yRpT+BqYB+KwWVvsv0/JB0GVJlQJ4aRzmdRh5mS7iyf3wh8m+JU4VbbPyvXHwS8bbB9AngDsBB4L3CB7VeARyX9YJj9vwu4YXBftrc1x8NvAItavt12lLRDGeOo8r3/JOnnYzzOoHmNnkkY/edF2/u1rij/qV5oXQX8se2rh2x3KMV0DyNRhW2gOJ39NdsvDlOWTAtRk6YljGbVd6IuVwMfkzQNQNJbJL0euAE4pmzjmEsxjeVQNwP/UdKC8r2zyvXPATu0bHcNcNLggqTBJHYD8F/KdYcAO9V2VJNM1faLNHrGeJ1J0T5xu6S7gdMpapOXAPcDa4G/A64f+kbbT1K0O3xP0o+B75Yv/SPwwcFGT+BPgMVlo+q9vHq15ovAeyXdTnFq9GCHjnFSaFrCkJ3aY0QT7b///r7xxhsrbbv99tuv6Ub/lrRhRDRY09owkjAiGiqXVSOiLalhRERlSRgRUVnTEkazTpAi4jXquqxa9g+6T9J6SaeMtTxJGBENVdeNWyp6MH+Dou/RIuBYjTCUwUiSMCIarKYaxhJgve0HbG8GLgSOGEt50oYR0WA1XVbdHXioZflh4FfHsqMkjIiGWrNmzdWqPlrZDEmrW5aXt4wMNlwVZEy3eCdhRDSU7YNr2tXDwLyW5T2AR8eyo7RhREx8twELJS2QNB04BrhsLDtKDSNigrO9RdJJFMMeTAHOsn3PWPaV3qoRUVlOSSKisiSMiKgsCSMiKkvCiIjKkjAiorIkjIioLAkjIipLwoiIyv4/+k/ihNCGPeMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# START SKIP FOR GRADING\n",
    "print(Y_test.shape)\n",
    "print('           '+ label_to_emoji(0)+ '    ' + label_to_emoji(1) + '    ' +  label_to_emoji(2)+ '    ' + label_to_emoji(3)+'   ' + label_to_emoji(4))\n",
    "print(pd.crosstab(Y_test, pred_test.reshape(56,), rownames=['Actual'], colnames=['Predicted'], margins=True))\n",
    "plot_confusion_matrix(Y_test, pred_test)\n",
    "# END SKIP FOR GRADING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zm2o8SQIcFI1"
   },
   "source": [
    "#### What you should remember:\n",
    "- Even with a mere 127 training examples, you can get a reasonably good model for sentiment classification. \n",
    "    - This is due to the generalization power word vectors gives you. \n",
    "- Emojify-V1 will perform poorly on sentences such as \"This movie is not good and not enjoyable\".\n",
    "    - It doesn't understand combinations of words.\n",
    "    - It just averages all the words' embedding vectors together, without considering the ordering of words. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BEeTqpjlcFI2"
   },
   "source": [
    "## 3 - Using LSTMs in Keras: Emojifier-V2\n",
    "\n",
    "We now build an LSTM model that takes word **sequences** as input. This model will be able to account for word ordering. \n",
    "\n",
    "Emojifier-V2 will continue to use pre-trained word embeddings to represent words. You'll feed word embeddings into an LSTM, and the LSTM will learn to predict the most appropriate emoji. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CPIihtFVFEbz"
   },
   "source": [
    "Run the following cell to load the Keras packages you'll need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "executionInfo": {
     "elapsed": 2501,
     "status": "ok",
     "timestamp": 1611738953388,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "uZ-fy9fYcFI3"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow\n",
    "np.random.seed(0)\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Dense, Input, Dropout, LSTM, Activation\n",
    "from tensorflow.keras.layers import Embedding\n",
    "from tensorflow.keras.preprocessing import sequence\n",
    "from tensorflow.keras.initializers import glorot_uniform\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P7LJvriXcFI3"
   },
   "source": [
    "### 3.1 - Model overview\n",
    "\n",
    "Here is the Emojifier-v2 you will implement:\n",
    "\n",
    "<img src=\"images/emojifier-v2.png\" style=\"width:700px;height:400px;\"> <br>\n",
    "<caption><center><font color='purple'><b>Figure 3</b>: Emojifier-V2. A 2-layer LSTM sequence classifier. </center></caption>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-3W3WTbpcFI3"
   },
   "source": [
    "### 3.2 - Keras and mini-batching \n",
    "\n",
    "We want to train a Keras model using mini-batches. However, most deep learning frameworks require that all sequences in the same mini-batch have the **same length**. \n",
    "\n",
    "This is what allows vectorization to work: If you had a 3-word sentence and a 4-word sentence, then the computations needed for them are different (one takes 3 steps of an LSTM, one takes 4 steps) so it's just not possible to do them both at the same time.\n",
    "    \n",
    "#### Padding handles sequences of varying lengths\n",
    "A common solution to handling sequences of **different length** is to use padding.  Specifically:\n",
    "* Set a maximum sequence length\n",
    "* Pad all sequences to have the same length \n",
    "    \n",
    "#### Example of padding:\n",
    "* Given a maximum sequence length of 20, you pad every sentence with \"0\"s so that each input sentence is of length 20. \n",
    "* Thus, the sentence \"I love you\" would be represented as $(e_{I}, e_{love}, e_{you}, \\vec{0}, \\vec{0}, \\ldots, \\vec{0})$. \n",
    "* Any sentences longer than 20 words would have to be truncated. \n",
    "* One way to choose the maximum sequence length is to just pick the length of the longest sentence in the training set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QuwbNWS0cFI4"
   },
   "source": [
    "### 3.3 - The embedding layer\n",
    "\n",
    "In Keras, the embedding matrix is represented as a \"layer.\"\n",
    "\n",
    "* The embedding matrix maps word indices to embedding vectors.\n",
    "    * The word indices are positive integers.\n",
    "    * The embedding vectors are dense vectors of fixed size.\n",
    "    * A \"dense\" vector is the opposite of a sparse vector. It means that most of its values are non-zero.  As a counter-example, a one-hot encoded vector is not \"dense.\"\n",
    "* The embedding matrix can be derived in two ways:\n",
    "    * Training a model to derive the embeddings from scratch. \n",
    "    * Using a pretrained embedding.\n",
    "    \n",
    "#### Updating pre-trained embeddings\n",
    "We create an [Embedding()](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding) layer in Keras.\n",
    "\n",
    "* You will initialize the embedding layer with GloVe 50-dimensional vectors. \n",
    "* In the code below, you'll observe how Keras allows you to either train or leave this layer fixed.  \n",
    "    * Because your training set is quite small, you'll leave the GloVe embeddings fixed instead of updating them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zhyVzuThcFI4"
   },
   "source": [
    "#### Inputs and outputs of the embedding layer\n",
    "\n",
    "* The `Embedding()` layer's input is an integer matrix of size **(batch size, max input length)**. \n",
    "    * This input corresponds to sentences converted into lists of indices (integers).\n",
    "    * The largest integer (the highest word index) in the input should be no larger than the vocabulary size.\n",
    "* The embedding layer outputs an array of shape (batch size, max input length, dimension of word vectors).\n",
    "\n",
    "* The figure shows the propagation of two example sentences through the embedding layer. \n",
    "    * Both examples have been zero-padded to a length of `max_len=5`.\n",
    "    * The word embeddings are 50 units in length.\n",
    "    * The final dimension of the representation is  `(2,max_len,50)`. \n",
    "\n",
    "<img src=\"images/embedding1.png\" style=\"width:700px;height:250px;\">\n",
    "<caption><center><font color='purple'><b>Figure 4</b>: Embedding layer</center></caption>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KnoTtNWBcFI5"
   },
   "source": [
    "Implement `sentences_to_indices`.\n",
    "\n",
    "This function processes an array of sentences X and returns inputs to the embedding layer:\n",
    "\n",
    "* Convert each training sentences into a list of indices (the indices correspond to each word in the sentence)\n",
    "* Zero-pad all these lists so that their length is the length of the longest sentence.\n",
    "    \n",
    "#### Additional hints:\n",
    "* You may have considered using the `enumerate()` function in the for loop, but please follow the starter code by initializing and incrementing `j` explicitly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2291,
     "status": "ok",
     "timestamp": 1611738965576,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "8cMm64iWcFI5",
    "outputId": "f82cb8b2-991c-44ae-9c81-7a5235edeea2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 I\n",
      "1 like\n",
      "2 learning\n"
     ]
    }
   ],
   "source": [
    "for idx, val in enumerate([\"I\", \"like\", \"learning\"]):\n",
    "    print(idx, val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "executionInfo": {
     "elapsed": 2881,
     "status": "ok",
     "timestamp": 1611738972334,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "Z0SixlIwcFI5"
   },
   "outputs": [],
   "source": [
    "def sentences_to_indices(X, word_to_index, max_len):\n",
    "    \"\"\"\n",
    "    Converts an array of sentences (strings) into an array of indices corresponding to words in the sentences.\n",
    "    The output shape should be such that it can be given to `Embedding()` (described in Figure 4). \n",
    "    \n",
    "    Arguments:\n",
    "    X -- array of sentences (strings), of shape (m, 1)\n",
    "    word_to_index -- a dictionary containing the each word mapped to its index\n",
    "    max_len -- maximum number of words in a sentence. You can assume every sentence in X is no longer than this. \n",
    "    \n",
    "    Returns:\n",
    "    X_indices -- array of indices corresponding to words in the sentences from X, of shape (m, max_len)\n",
    "    \"\"\"\n",
    "    \n",
    "    m = X.shape[0]                                   # number of training examples\n",
    "    \n",
    "    # Initialize X_indices as a numpy matrix of zeros and the correct shape\n",
    "    X_indices = np.zeros((m, max_len))\n",
    "    \n",
    "    for i in range(m):                               # loop over training examples\n",
    "        \n",
    "        # Convert the ith training sentence in lower case and split is into words. You should get a list of words.\n",
    "        sentence_words = X[i].lower().split()\n",
    "        \n",
    "        # Initialize j to 0\n",
    "        j = 0\n",
    "        \n",
    "        # Loop over the words of sentence_words\n",
    "\n",
    "        for w in sentence_words:\n",
    "            # if w exists in the word_to_index dictionary\n",
    "            if w in word_to_index.keys():\n",
    "                # Set the (i,j)th entry of X_indices to the index of the correct word.\n",
    "                X_indices[i, j] = word_to_index[w]\n",
    "                # Increment j to j + 1\n",
    "                j += 1\n",
    "                \n",
    "    return X_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 1. 3. 2.]\n",
      " [3. 7. 5. 4.]\n",
      " [4. 2. 6. 0.]]\n",
      "\u001b[92mAll tests passed!\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def sentences_to_indices_test(target):\n",
    "    \n",
    "    # Create a word_to_index dictionary\n",
    "    word_to_index = {}\n",
    "    for idx, val in enumerate([\"i\", \"like\", \"learning\", \"deep\", \"machine\", \"love\", \"smile\", '´0.=']):\n",
    "        word_to_index[val] = idx;\n",
    "       \n",
    "    max_len = 4\n",
    "    sentences = np.array([\"I like deep learning\", \"deep ´0.= love machine\", \"machine learning smile\"]);\n",
    "    indexes = target(sentences, word_to_index, max_len)\n",
    "    print(indexes)\n",
    "    \n",
    "    assert type(indexes) == np.ndarray, \"Wrong type. Use np arrays in the function\"\n",
    "    assert indexes.shape == (sentences.shape[0], max_len), \"Wrong shape of ouput matrix\"\n",
    "    assert np.allclose(indexes, [[0, 1, 3, 2],\n",
    "                                 [3, 7, 5, 4],\n",
    "                                 [4, 2, 6, 0]]), \"Wrong values. Debug with the given examples\"\n",
    "    \n",
    "    print(\"\\033[92mAll tests passed!\")\n",
    "    \n",
    "sentences_to_indices_test(sentences_to_indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "reyHmseecFI6"
   },
   "source": [
    "Run the following cell to check what `sentences_to_indices()` does, and take a look at your results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1992,
     "status": "ok",
     "timestamp": 1611738982161,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "oBL1PMOCcFI6",
    "outputId": "6781359c-bafd-4ab5-a477-f8a5a86ea219"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1 = ['funny lol' 'lets play baseball' 'food is ready for you']\n",
      "X1_indices =\n",
      " [[155345. 225122.      0.      0.      0.]\n",
      " [220930. 286375.  69714.      0.      0.]\n",
      " [151204. 192973. 302254. 151349. 394475.]]\n"
     ]
    }
   ],
   "source": [
    "X1 = np.array([\"funny lol\", \"lets play baseball\", \"food is ready for you\"])\n",
    "X1_indices = sentences_to_indices(X1, word_to_index, max_len=5)\n",
    "print(\"X1 =\", X1)\n",
    "print(\"X1_indices =\\n\", X1_indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0OJPAEM5cFI6"
   },
   "source": [
    "#### Build embedding Layer\n",
    "\n",
    "Now you'll build the `Embedding()` layer in Keras, using pre-trained word vectors. \n",
    "\n",
    "* The embedding layer takes as input a list of word indices.\n",
    "    * `sentences_to_indices()` creates these word indices.\n",
    "* The embedding layer will return the word embeddings for a sentence. \n",
    "\n",
    "Implement `pretrained_embedding_layer()` with these steps:\n",
    "\n",
    "1. Initialize the embedding matrix as a numpy array of zeros.\n",
    "    * The embedding matrix has a row for each unique word in the vocabulary.\n",
    "        * There is one additional row to handle \"unknown\" words.\n",
    "        * So vocab_size is the number of unique words plus one.\n",
    "    * Each row will store the vector representation of one word. \n",
    "        * For example, one row may be 50 positions long if using GloVe word vectors.\n",
    "    * In the code below, `emb_dim` represents the length of a word embedding.\n",
    "2. Fill in each row of the embedding matrix with the vector representation of a word\n",
    "    * Each word in `word_to_index` is a string.\n",
    "    * word_to_vec_map is a dictionary where the keys are strings and the values are the word vectors.\n",
    "3. Define the Keras embedding layer. \n",
    "    * Use [Embedding()](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding). \n",
    "    * The input dimension is equal to the vocabulary length (number of unique words plus one).\n",
    "    * The output dimension is equal to the number of positions in a word embedding.\n",
    "    * Make this layer's embeddings fixed.\n",
    "        * If you were to set `trainable = True`, then it will allow the optimization algorithm to modify the values of the word embeddings.\n",
    "        * In this case, you don't want the model to modify the word embeddings.\n",
    "4. Set the embedding weights to be equal to the embedding matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "executionInfo": {
     "elapsed": 2160,
     "status": "ok",
     "timestamp": 1611738992486,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "XBlEpiVkcFI7"
   },
   "outputs": [],
   "source": [
    "def pretrained_embedding_layer(word_to_vec_map, word_to_index):\n",
    "    \"\"\"\n",
    "    Creates a Keras Embedding() layer and loads in pre-trained GloVe 50-dimensional vectors.\n",
    "    \n",
    "    Arguments:\n",
    "    word_to_vec_map -- dictionary mapping words to their GloVe vector representation.\n",
    "    word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)\n",
    "\n",
    "    Returns:\n",
    "    embedding_layer -- pretrained layer Keras instance\n",
    "    \"\"\"\n",
    "    \n",
    "    vocab_size = len(word_to_index) + 1              # adding 1 to fit Keras embedding (requirement)\n",
    "    any_word = list(word_to_vec_map.keys())[0]\n",
    "    emb_dim = word_to_vec_map[any_word].shape[0]    # define dimensionality of your GloVe word vectors (= 50)\n",
    "      \n",
    "    # Step 1\n",
    "    # Initialize the embedding matrix as a numpy array of zeros.\n",
    "    # See instructions above to choose the correct shape.\n",
    "    emb_matrix = np.zeros((vocab_size, emb_dim))\n",
    "    \n",
    "    # Step 2\n",
    "    # Set each row \"idx\" of the embedding matrix to be \n",
    "    # the word vector representation of the idx'th word of the vocabulary\n",
    "    for word, idx in word_to_index.items():\n",
    "        emb_matrix[idx, :] = word_to_vec_map[word]\n",
    "\n",
    "    # Step 3\n",
    "    # Define Keras embedding layer with the correct input and output sizes\n",
    "    # Make it non-trainable.\n",
    "    embedding_layer = Embedding(vocab_size, emb_dim, trainable=False)\n",
    "\n",
    "    # Step 4 (already done for you; please do not modify)\n",
    "    # Build the embedding layer, it is required before setting the weights of the embedding layer. \n",
    "    embedding_layer.build((None,)) # Do not modify the \"None\".  This line of code is complete as-is.\n",
    "    \n",
    "    # Set the weights of the embedding layer to the embedding matrix. Your layer is now pretrained.\n",
    "    embedding_layer.set_weights([emb_matrix])\n",
    "    \n",
    "    return embedding_layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mAll tests passed!\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def pretrained_embedding_layer_test(target):\n",
    "    # Create a controlled word to vec map\n",
    "    word_to_vec_map = {'a': [3, 3], 'synonym_of_a': [3, 3], 'a_nw': [2, 4], 'a_s': [3, 2], 'a_n': [3, 4], \n",
    "                       'c': [-2, 1], 'c_n': [-2, 2],'c_ne': [-1, 2], 'c_e': [-1, 1], 'c_se': [-1, 0], \n",
    "                       'c_s': [-2, 0], 'c_sw': [-3, 0], 'c_w': [-3, 1], 'c_nw': [-3, 2]\n",
    "                      }\n",
    "    # Convert lists to np.arrays\n",
    "    for key in word_to_vec_map.keys():\n",
    "        word_to_vec_map[key] = np.array(word_to_vec_map[key])\n",
    "        \n",
    "    # Create a word_to_index dictionary\n",
    "    word_to_index = {}\n",
    "    for idx, val in enumerate(list(word_to_vec_map.keys())):\n",
    "        word_to_index[val] = idx;\n",
    "        \n",
    "    np.random.seed(1)\n",
    "    embedding_layer = target(word_to_vec_map, word_to_index)\n",
    "    \n",
    "    assert type(embedding_layer) == Embedding, \"Wrong type\"\n",
    "    assert embedding_layer.input_dim == len(list(word_to_vec_map.keys())) + 1, \"Wrong input shape\"\n",
    "    assert embedding_layer.output_dim == len(word_to_vec_map['a']), \"Wrong output shape\"\n",
    "    assert np.allclose(embedding_layer.get_weights(), \n",
    "                       [[[ 3, 3], [ 3, 3], [ 2, 4], [ 3, 2], [ 3, 4],\n",
    "                       [-2, 1], [-2, 2], [-1, 2], [-1, 1], [-1, 0],\n",
    "                       [-2, 0], [-3, 0], [-3, 1], [-3, 2], [ 0, 0]]]), \"Wrong vaulues\"\n",
    "    print(\"\\033[92mAll tests passed!\")\n",
    "       \n",
    "    \n",
    "pretrained_embedding_layer_test(pretrained_embedding_layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 6068,
     "status": "ok",
     "timestamp": 1611739002394,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "Gn4iGb0AcFI7",
    "outputId": "5bab44b5-bf23-4c6e-f891-429e8be34981"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weights[0][1][1] = 0.39031\n",
      "Input_dim 400001\n",
      "Output_dim 50\n"
     ]
    }
   ],
   "source": [
    "embedding_layer = pretrained_embedding_layer(word_to_vec_map, word_to_index)\n",
    "print(\"weights[0][1][1] =\", embedding_layer.get_weights()[0][1][1])\n",
    "print(\"Input_dim\", embedding_layer.input_dim)\n",
    "print(\"Output_dim\",embedding_layer.output_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uEsWnZ_2cFI7"
   },
   "source": [
    "### 3.4 - Building the model\n",
    "\n",
    "Now you're ready to build the Emojifier-V2 model, in which you feed the embedding layer's output to an LSTM network.\n",
    "\n",
    "<img src=\"images/emojifier-v2.png\" style=\"width:700px;height:400px;\"> <br>\n",
    "<p><caption><center><font color='purple'><b>Figure 3</b>: Emojifier-v2. A 2-layer LSTM sequence classifier. </center></caption></font></p>\n",
    "\n",
    "\n",
    "Implement `Emojify_V2()`.\n",
    "\n",
    "This function builds a Keras graph of the architecture shown in Figure (3). \n",
    "\n",
    "* The model takes as input an array of sentences of shape (`m, max_len`, ) defined by `input_shape`. \n",
    "* The model outputs a softmax probability vector of shape (`m, C = 5`). \n",
    "* You may need to use the following Keras layers:\n",
    "    * [Input()](https://www.tensorflow.org/api_docs/python/tf/keras/Input)\n",
    "        * Set the `shape` and `dtype` parameters.\n",
    "        * The inputs are integers, so you can specify the data type as a string, 'int32'.\n",
    "    * [LSTM()](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM)\n",
    "        * Set the `units` and `return_sequences` parameters.\n",
    "    * [Dropout()](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout)\n",
    "        * Set the `rate` parameter.\n",
    "    * [Dense()](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense)\n",
    "        * Set the `units`, \n",
    "        * Note that `Dense()` has an `activation` parameter.  For the purposes of passing the autograder, please do not set the activation within `Dense()`.  Use the separate `Activation` layer to do so.\n",
    "    * [Activation()](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Activation)\n",
    "        * You can pass in the activation of your choice as a lowercase string.\n",
    "    * [Model()](https://www.tensorflow.org/api_docs/python/tf/keras/Model)\n",
    "        * Set `inputs` and `outputs`.\n",
    "\n",
    "\n",
    "#### Additional hints\n",
    "* These Keras layers return an object, and you will feed in the outputs of the previous layer as the input arguments to that object.  The returned object can be created and called in the same line.\n",
    "\n",
    "```Python\n",
    "# How to use Keras layers in two lines of code\n",
    "dense_object = Dense(units = ...)\n",
    "X = dense_object(inputs)\n",
    "\n",
    "# How to use Keras layers in one line of code\n",
    "X = Dense(units = ...)(inputs)\n",
    "```\n",
    "\n",
    "* The `embedding_layer` that is returned by `pretrained_embedding_layer` is a layer object that can be called as a function, passing in a single argument (sentence indices).\n",
    "\n",
    "* Here is some sample code in case you're stuck: 😊\n",
    "```Python\n",
    "raw_inputs = Input(shape=(maxLen,), dtype='int32')\n",
    "preprocessed_inputs = ... # some pre-processing\n",
    "X = LSTM(units = ..., return_sequences= ...)(processed_inputs)\n",
    "X = Dropout(rate = ..., )(X)\n",
    "...\n",
    "X = Dense(units = ...)(X)\n",
    "X = Activation(...)(X)\n",
    "model = Model(inputs=..., outputs=...)\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "executionInfo": {
     "elapsed": 3214,
     "status": "ok",
     "timestamp": 1611739012958,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "Pb2ugsSUcFI7"
   },
   "outputs": [],
   "source": [
    "def Emojify_V2(input_shape, word_to_vec_map, word_to_index):\n",
    "    \"\"\"\n",
    "    Function creating the Emojify-v2 model's graph.\n",
    "    \n",
    "    Arguments:\n",
    "    input_shape -- shape of the input, usually (max_len,)\n",
    "    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation\n",
    "    word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)\n",
    "\n",
    "    Returns:\n",
    "    model -- a model instance in Keras\n",
    "    \"\"\"\n",
    "    \n",
    "    # Define sentence_indices as the input of the graph.\n",
    "    # It should be of shape input_shape and dtype 'int32' (as it contains indices, which are integers).\n",
    "    sentence_indices = Input(shape=input_shape, dtype='int32')\n",
    "    \n",
    "    # Create the embedding layer pretrained with GloVe Vectors\n",
    "    embedding_layer = pretrained_embedding_layer(word_to_vec_map, word_to_index)\n",
    "    \n",
    "    # Propagate sentence_indices through your embedding layer\n",
    "    # (See additional hints in the instructions).\n",
    "    embeddings = embedding_layer(sentence_indices)   \n",
    "    \n",
    "    # Propagate the embeddings through an LSTM layer with 128-dimensional hidden state\n",
    "    # The returned output should be a batch of sequences.\n",
    "    X = LSTM(units=128, return_sequences=True)(embeddings)\n",
    "    \n",
    "    # Add dropout with a probability of 0.5\n",
    "    X = Dropout(rate=0.5)(X)\n",
    "    \n",
    "    # Propagate X trough another LSTM layer with 128-dimensional hidden state\n",
    "    # The returned output should be a single hidden state, not a batch of sequences.\n",
    "    X = LSTM(units=128, return_sequences=False)(X)\n",
    "    \n",
    "    # Add dropout with a probability of 0.5\n",
    "    X = Dropout(rate=0.5)(X)\n",
    "    \n",
    "    # Propagate X through a Dense layer with 5 units\n",
    "    X = Dense(units=5)(X)\n",
    "    \n",
    "    # Add a softmax activation\n",
    "    X = Activation('softmax')(X)\n",
    "    \n",
    "    # Create Model instance which converts sentence_indices into X.\n",
    "    model = Model(inputs=sentence_indices, outputs=X)\n",
    "        \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mAll tests passed!\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# UNIT TEST\n",
    "def Emojify_V2_test(target):\n",
    "    # Create a controlled word to vec map\n",
    "    word_to_vec_map = {'a': [3, 3], 'synonym_of_a': [3, 3], 'a_nw': [2, 4], 'a_s': [3, 2], 'a_n': [3, 4], \n",
    "                       'c': [-2, 1], 'c_n': [-2, 2],'c_ne': [-1, 2], 'c_e': [-1, 1], 'c_se': [-1, 0], \n",
    "                       'c_s': [-2, 0], 'c_sw': [-3, 0], 'c_w': [-3, 1], 'c_nw': [-3, 2]\n",
    "                      }\n",
    "    # Convert lists to np.arrays\n",
    "    for key in word_to_vec_map.keys():\n",
    "        word_to_vec_map[key] = np.array(word_to_vec_map[key])\n",
    "        \n",
    "    # Create a word_to_index dictionary\n",
    "    word_to_index = {}\n",
    "    for idx, val in enumerate(list(word_to_vec_map.keys())):\n",
    "        word_to_index[val] = idx;\n",
    "        \n",
    "    maxLen = 4\n",
    "    model = target((maxLen,), word_to_vec_map, word_to_index)\n",
    "    \n",
    "    expectedModel = [['InputLayer', [(None, 4)], 0], ['Embedding', (None, 4, 2), 30], ['LSTM', (None, 4, 128), 67072, (None, 4, 2), 'tanh', True], ['Dropout', (None, 4, 128), 0, 0.5], ['LSTM', (None, 128), 131584, (None, 4, 128), 'tanh', False], ['Dropout', (None, 128), 0, 0.5], ['Dense', (None, 5), 645, 'linear'], ['Activation', (None, 5), 0]]\n",
    "    comparator(summary(model), expectedModel)\n",
    "    \n",
    "    \n",
    "Emojify_V2_test(Emojify_V2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-VamRAKtcFI8"
   },
   "source": [
    "Run the following cell to create your model and check its summary. \n",
    "\n",
    "* Because all sentences in the dataset are less than 10 words, `max_len = 10` was chosen.  \n",
    "* You should see that your architecture uses 20,223,927 parameters, of which 20,000,050 (the word embeddings) are non-trainable, with the remaining 223,877 being trainable. \n",
    "* Because your vocabulary size has 400,001 words (with valid indices from 0 to 400,000) there are 400,001\\*50 = 20,000,050 non-trainable parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3127,
     "status": "ok",
     "timestamp": 1611739019596,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "8fLhXJ9ucFI8",
    "outputId": "02d98359-a43b-4780-bb17-3d36e060aa86"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"functional_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_4 (InputLayer)         [(None, 10)]              0         \n",
      "_________________________________________________________________\n",
      "embedding_5 (Embedding)      (None, 10, 50)            20000050  \n",
      "_________________________________________________________________\n",
      "lstm_4 (LSTM)                (None, 10, 128)           91648     \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 10, 128)           0         \n",
      "_________________________________________________________________\n",
      "lstm_5 (LSTM)                (None, 128)               131584    \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 5)                 645       \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 5)                 0         \n",
      "=================================================================\n",
      "Total params: 20,223,927\n",
      "Trainable params: 223,877\n",
      "Non-trainable params: 20,000,050\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Emojify_V2((maxLen,), word_to_vec_map, word_to_index)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mKIsZqqicFI8"
   },
   "source": [
    "#### Compile the model \n",
    "\n",
    "You need to compile the model and define what loss, optimizer and metrics you want to use. Compile your model using `categorical_crossentropy` loss, `adam` optimizer and `['accuracy']` metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "executionInfo": {
     "elapsed": 2041,
     "status": "ok",
     "timestamp": 1611739024847,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "aMf79f45cFI9"
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mX6NORy7cFI9"
   },
   "source": [
    "### 3.5 - Train the model \n",
    "\n",
    "It's time to train your model. Your Emojifier-V2 `model` takes as input an array of shape (`m`, `max_len`) and outputs probability vectors of shape (`m`, `number of classes`). Thus, you have to convert X_train (array of sentences as strings) to X_train_indices (array of sentences as list of word indices), and Y_train (labels as indices) to Y_train_oh (labels as one-hot vectors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "executionInfo": {
     "elapsed": 2284,
     "status": "ok",
     "timestamp": 1611739029525,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "UgsBnWQqcFI-"
   },
   "outputs": [],
   "source": [
    "X_train_indices = sentences_to_indices(X_train, word_to_index, maxLen)\n",
    "Y_train_oh = convert_to_one_hot(Y_train, C = 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9fMyo0vqcFI_"
   },
   "source": [
    "Fit the Keras model on `X_train_indices` and `Y_train_oh`, using `epochs = 50` and `batch_size = 32`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "id": "LtFpvyJicFI_",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 1.5559 - accuracy: 0.2576\n",
      "Epoch 2/50\n",
      "5/5 [==============================] - 0s 39ms/step - loss: 1.5244 - accuracy: 0.3030\n",
      "Epoch 3/50\n",
      "5/5 [==============================] - 0s 23ms/step - loss: 1.4699 - accuracy: 0.3182\n",
      "Epoch 4/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 1.3883 - accuracy: 0.3864\n",
      "Epoch 5/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 1.2766 - accuracy: 0.5606\n",
      "Epoch 6/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 1.1662 - accuracy: 0.5606\n",
      "Epoch 7/50\n",
      "5/5 [==============================] - 0s 34ms/step - loss: 1.0390 - accuracy: 0.5909\n",
      "Epoch 8/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.9222 - accuracy: 0.7121\n",
      "Epoch 9/50\n",
      "5/5 [==============================] - 0s 33ms/step - loss: 0.8014 - accuracy: 0.6970\n",
      "Epoch 10/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.7658 - accuracy: 0.7197\n",
      "Epoch 11/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.6878 - accuracy: 0.7424\n",
      "Epoch 12/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.7110 - accuracy: 0.7955\n",
      "Epoch 13/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.6364 - accuracy: 0.7576\n",
      "Epoch 14/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.5553 - accuracy: 0.7803\n",
      "Epoch 15/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.4567 - accuracy: 0.8258\n",
      "Epoch 16/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.4188 - accuracy: 0.8333\n",
      "Epoch 17/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.3513 - accuracy: 0.8636\n",
      "Epoch 18/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.3166 - accuracy: 0.8939\n",
      "Epoch 19/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.3323 - accuracy: 0.9091\n",
      "Epoch 20/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.2392 - accuracy: 0.9091\n",
      "Epoch 21/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.2387 - accuracy: 0.9318\n",
      "Epoch 22/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.3398 - accuracy: 0.8712\n",
      "Epoch 23/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.3225 - accuracy: 0.8864\n",
      "Epoch 24/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.2381 - accuracy: 0.9242\n",
      "Epoch 25/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.2270 - accuracy: 0.9167\n",
      "Epoch 26/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.1973 - accuracy: 0.9318\n",
      "Epoch 27/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.2202 - accuracy: 0.9242\n",
      "Epoch 28/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.2004 - accuracy: 0.9318\n",
      "Epoch 29/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.1392 - accuracy: 0.9621\n",
      "Epoch 30/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.2060 - accuracy: 0.9242\n",
      "Epoch 31/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.1454 - accuracy: 0.9470\n",
      "Epoch 32/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.1507 - accuracy: 0.9394\n",
      "Epoch 33/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.0814 - accuracy: 0.9773\n",
      "Epoch 34/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.1150 - accuracy: 0.9470\n",
      "Epoch 35/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.2160 - accuracy: 0.9470\n",
      "Epoch 36/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.0992 - accuracy: 0.9697\n",
      "Epoch 37/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.0761 - accuracy: 0.9773\n",
      "Epoch 38/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.0544 - accuracy: 0.9924\n",
      "Epoch 39/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.0506 - accuracy: 0.9848\n",
      "Epoch 40/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.1778 - accuracy: 0.9697\n",
      "Epoch 41/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.1670 - accuracy: 0.9545\n",
      "Epoch 42/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.2193 - accuracy: 0.9470\n",
      "Epoch 43/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.1806 - accuracy: 0.9318\n",
      "Epoch 44/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.1629 - accuracy: 0.9545\n",
      "Epoch 45/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.0936 - accuracy: 0.9773\n",
      "Epoch 46/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.1464 - accuracy: 0.9545\n",
      "Epoch 47/50\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.2896 - accuracy: 0.9091\n",
      "Epoch 48/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.2263 - accuracy: 0.8939\n",
      "Epoch 49/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.1335 - accuracy: 0.9621\n",
      "Epoch 50/50\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.0990 - accuracy: 0.9697\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fc48be98c10>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_indices, Y_train_oh, epochs = 50, batch_size = 32, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gR2QJZAkcFJA"
   },
   "source": [
    "Your model should perform around **90% to 100% accuracy** on the training set. Exact model accuracy may vary.\n",
    "\n",
    "Run the following cell to evaluate your model on the test set: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2962,
     "status": "ok",
     "timestamp": 1611739058762,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "sIbcdVibcFJA",
    "outputId": "65b06e6c-415e-4038-c6fd-3839b3fd39f1",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2/2 [==============================] - 0s 2ms/step - loss: 0.3399 - accuracy: 0.8571\n",
      "\n",
      "Test accuracy =  0.8571428656578064\n"
     ]
    }
   ],
   "source": [
    "X_test_indices = sentences_to_indices(X_test, word_to_index, max_len = maxLen)\n",
    "Y_test_oh = convert_to_one_hot(Y_test, C = 5)\n",
    "loss, acc = model.evaluate(X_test_indices, Y_test_oh)\n",
    "print()\n",
    "print(\"Test accuracy = \", acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2d6y4sJbcFJA"
   },
   "source": [
    "You should get a test accuracy between 80% and 95%. Run the cell below to see the mislabelled examples: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1779,
     "status": "ok",
     "timestamp": 1611739122633,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "mjMyEGmYcFJC",
    "outputId": "0e87d217-d501-40d1-a6c7-71d567643d52"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected emoji:😄 prediction: he got a very nice raise\t❤️\n",
      "Expected emoji:😄 prediction: she got me a nice present\t❤️\n",
      "Expected emoji:😞 prediction: This girl is messing with me\t❤️\n",
      "Expected emoji:🍴 prediction: any suggestions for dinner\t😄\n",
      "Expected emoji:😄 prediction: you brighten my day\t❤️\n",
      "Expected emoji:😞 prediction: she is a bully\t❤️\n",
      "Expected emoji:😄 prediction: she said yes\t😞\n",
      "Expected emoji:🍴 prediction: I did not have breakfast 😞\n"
     ]
    }
   ],
   "source": [
    "# This code allows you to see the mislabelled examples\n",
    "C = 5\n",
    "y_test_oh = np.eye(C)[Y_test.reshape(-1)]\n",
    "X_test_indices = sentences_to_indices(X_test, word_to_index, maxLen)\n",
    "pred = model.predict(X_test_indices)\n",
    "for i in range(len(X_test)):\n",
    "    x = X_test_indices\n",
    "    num = np.argmax(pred[i])\n",
    "    if(num != Y_test[i]):\n",
    "        print('Expected emoji:'+ label_to_emoji(Y_test[i]) + ' prediction: '+ X_test[i] + label_to_emoji(num).strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UGg00oBRcFJD"
   },
   "source": [
    "Now you can try it on your own example! Write your own sentence below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1780,
     "status": "ok",
     "timestamp": 1611739204630,
     "user": {
      "displayName": "Mubsi K",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gip7OjOkdNkKxKDyWEQAq1o8ccGN_HrBTGdqjgQ=s64",
      "userId": "08094225471505108399"
     },
     "user_tz": -300
    },
    "id": "wEgCsIE7cFJE",
    "outputId": "28d6942b-ad6b-461e-9904-191430145d85"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I cannot play ⚾\n"
     ]
    }
   ],
   "source": [
    "# Change the sentence below to see your prediction. Make sure all the words are in the Glove embeddings.  \n",
    "x_test = np.array(['I cannot play'])\n",
    "X_test_indices = sentences_to_indices(x_test, word_to_index, maxLen)\n",
    "print(x_test[0] +' '+  label_to_emoji(np.argmax(model.predict(X_test_indices))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tT53ibgFcFJE"
   },
   "source": [
    "#### LSTM version accounts for word order\n",
    "* The Emojify-V1 model did not understand \"not feeling happy\" correctly, but your implementation of Emojify-V2 got it right. \n",
    "    * If it didn't, be aware that Keras' outputs are slightly random each time, so this is probably why. \n",
    "* The current model still isn't very robust at understanding negation (such as \"not happy\")\n",
    "    * This is because the training set is small and doesn't have a lot of examples of negation. \n",
    "    * If the training set were larger, the LSTM model would be much better than the Emojify-V1 model at understanding more complex sentences. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0ysWCkrcFEb7"
   },
   "source": [
    "By now, you've: \n",
    "\n",
    "* Created an embedding matrix\n",
    "* Observed how negative sampling learns word vectors more efficiently than other methods\n",
    "* Experienced the advantages and disadvantages of the GloVe algorithm\n",
    "* Built a sentiment classifier using word embeddings \n",
    "\n",
    "Cool! (or Emojified: 😎😎😎 ) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GgoBXYn3cFJE"
   },
   "source": [
    "#### Things to remember\n",
    "- If you have an NLP task where the training set is small, using word embeddings can help your algorithm significantly. \n",
    "- Word embeddings allow your model to work on words in the test set that may not even appear in the training set. \n",
    "- Training sequence models in Keras (and in most other deep learning frameworks) requires a few important details:\n",
    "    - To use mini-batches, the sequences need to be **padded** so that all the examples in a mini-batch have the **same length**. \n",
    "    - An `Embedding()` layer can be initialized with pretrained values. \n",
    "        - These values can be either fixed or trained further on your dataset. \n",
    "        - If however your labeled dataset is small, it's usually not worth trying to train a large pre-trained set of embeddings.   \n",
    "    - `LSTM()` has a flag called `return_sequences` to decide if you would like to return every hidden states or only the last one. \n",
    "    - You can use `Dropout()` right after `LSTM()` to regularize your network. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vYoEQMCVcFJG"
   },
   "source": [
    "<a name='3'></a>\n",
    "## 4 - Acknowledgments\n",
    "\n",
    "Thanks to Alison Darcy and the Woebot team for their advice on the creation of this assignment. \n",
    "* Woebot is a chatbot friend that is ready to speak with you 24/7. \n",
    "* Part of Woebot's technology uses word embeddings to understand the emotions of what you say. \n",
    "* You can chat with Woebot by going to http://woebot.io\n",
    "\n",
    "<img src=\"images/woebot.png\" style=\"width:600px;height:300px;\">"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "zhyVzuThcFI4",
    "LUSzrFkYcFJF"
   ],
   "name": "Solution_Emojify_v2a.ipynb",
   "provenance": []
  },
  "coursera": {
   "schema_names": [
    "DLSC5W2-A2"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}