rnn_sentiment_analysis-sp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis: RNN\n",
    "\n",
    "Examples:\n",
    "\n",
    "> \"I expected this movie to be much better.\" (Negative)<br> \"This movie was much better than I expected.\" (Positive)\n",
    "\n",
    "If we use a one-hot encoding or BoW for a sentiment analyses with these two sentences, their vectors and resulting output will be very similar because they have very similar words.\n",
    "\n",
    "One way to fix this is to consider the *order* of the words using RNN and LSTMs. With an RNN/LSTM architecture, we would be able to feed the one-hot-encoded word vectors one by one. \n",
    "\n",
    "The final output at each point is an encoding of a sentence (or *sequence*). With this encoding, we feed it to one or more dense layers, which will then get trained to proedict the sentiment of the review.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Switching gears - RNNs\n",
    "\n",
    "Now I will use Recurrent Neural Networks, and in particular LSTMs, to perform sentiment analysis in Keras. Conveniently, Keras has a built-in [IMDb movie reviews dataset](https://keras.io/datasets/#imdb-movie-reviews-sentiment-classification) that we can use, with the same vocabulary size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded dataset with 25000 training samples, 25000 test samples\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb  # import the built-in imdb dataset in Keras\n",
    "\n",
    "# Set the vocabulary size\n",
    "vocabulary_size = 5000\n",
    "\n",
    "# Load in training and test data (note the difference in convention compared to scikit-learn)\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=vocabulary_size)\n",
    "print(\"Loaded dataset with {} training samples, {} test samples\".format(len(X_train), len(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the built-in IMBb data set in keras, the positive and negative reviews were coded as 1 and 0 respectively. A review is stored as a sequence of integers, which represent *word ID*s preassigned by keras. 'imdb.get_word_index()' can be used to map the integers back to the original words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Review ---\n",
      "[1, 4, 2, 716, 4, 65, 7, 4, 689, 4367, 2, 2343, 4804, 2, 2, 2, 2, 2315, 2, 2, 2, 2, 4, 2, 628, 2, 37, 9, 150, 4, 2, 4069, 11, 2909, 4, 2, 847, 313, 6, 176, 2, 9, 2, 138, 9, 4434, 19, 4, 96, 183, 26, 4, 192, 15, 27, 2, 799, 2, 2, 588, 84, 11, 4, 3231, 152, 339, 2, 42, 4869, 2, 2, 345, 4804, 2, 142, 43, 218, 208, 54, 29, 853, 659, 46, 4, 882, 183, 80, 115, 30, 4, 172, 174, 10, 10, 1001, 398, 1001, 1055, 526, 34, 3717, 2, 2, 2, 17, 4, 2, 1094, 871, 64, 85, 22, 2030, 1109, 38, 230, 9, 4, 4324, 2, 251, 2, 1034, 195, 301, 14, 16, 31, 7, 4, 2, 8, 783, 2, 33, 4, 2945, 103, 465, 2, 42, 845, 45, 446, 11, 1895, 19, 184, 76, 32, 4, 2, 207, 110, 13, 197, 4, 2, 16, 601, 964, 2152, 595, 13, 258, 4, 1730, 66, 338, 55, 2, 4, 550, 728, 65, 1196, 8, 1839, 61, 1546, 42, 2, 61, 602, 120, 45, 2, 6, 320, 786, 99, 196, 2, 786, 2, 4, 225, 4, 373, 1009, 33, 4, 130, 63, 69, 72, 1104, 46, 1292, 225, 14, 66, 194, 2, 1703, 56, 8, 803, 1004, 6, 2, 155, 11, 4, 2, 3231, 45, 853, 2029, 8, 30, 6, 117, 430, 19, 6, 2, 9, 15, 66, 424, 8, 2337, 178, 9, 15, 66, 424, 8, 1465, 178, 9, 15, 66, 142, 15, 9, 424, 8, 28, 178, 662, 44, 12, 17, 4, 130, 898, 1686, 9, 6, 2, 267, 185, 430, 4, 118, 2, 277, 15, 4, 1188, 100, 216, 56, 19, 4, 357, 114, 2, 367, 45, 115, 93, 788, 121, 4, 2, 79, 32, 68, 278, 39, 8, 818, 162, 4165, 237, 600, 7, 98, 306, 8, 157, 549, 628, 11, 6, 2, 13, 824, 15, 4104, 76, 42, 138, 36, 774, 77, 1059, 159, 150, 4, 229, 497, 8, 1493, 11, 175, 251, 453, 19, 2, 189, 12, 43, 127, 6, 394, 292, 7, 2, 4, 107, 8, 4, 2826, 15, 1082, 1251, 9, 906, 42, 1134, 6, 66, 78, 22, 15, 13, 244, 2519, 8, 135, 233, 52, 44, 10, 10, 466, 112, 398, 526, 34, 4, 1572, 4413, 2, 1094, 225, 57, 599, 133, 225, 6, 227, 7, 541, 4323, 6, 171, 139, 7, 539, 2, 56, 11, 6, 3231, 21, 164, 25, 426, 81, 33, 344, 624, 19, 6, 4617, 7, 2, 2, 6, 2, 4, 22, 9, 1082, 629, 237, 45, 188, 6, 55, 655, 707, 2, 956, 225, 1456, 841, 42, 1310, 225, 6, 2493, 1467, 2, 2828, 21, 4, 2, 9, 364, 23, 4, 2228, 2407, 225, 24, 76, 133, 18, 4, 189, 2293, 10, 10, 814, 11, 2, 11, 2642, 14, 47, 15, 682, 364, 352, 168, 44, 12, 45, 24, 913, 93, 21, 247, 2441, 4, 116, 34, 35, 1859, 8, 72, 177, 9, 164, 8, 901, 344, 44, 13, 191, 135, 13, 126, 421, 233, 18, 259, 10, 10, 4, 2, 2, 4, 2, 3074, 7, 112, 199, 753, 357, 39, 63, 12, 115, 2, 763, 8, 15, 35, 3282, 1523, 65, 57, 599, 6, 1916, 277, 1730, 37, 25, 92, 202, 6, 2, 44, 25, 28, 6, 22, 15, 122, 24, 4171, 72, 33, 32]\n",
      "--- Label ---\n",
      "0\n",
      "[list([1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 2, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 2, 19, 178, 32])\n",
      " list([1, 194, 1153, 194, 2, 78, 228, 5, 6, 1463, 4369, 2, 134, 26, 4, 715, 8, 118, 1634, 14, 394, 20, 13, 119, 954, 189, 102, 5, 207, 110, 3103, 21, 14, 69, 188, 8, 30, 23, 7, 4, 249, 126, 93, 4, 114, 9, 2300, 1523, 5, 647, 4, 116, 9, 35, 2, 4, 229, 9, 340, 1322, 4, 118, 9, 4, 130, 4901, 19, 4, 1002, 5, 89, 29, 952, 46, 37, 4, 455, 9, 45, 43, 38, 1543, 1905, 398, 4, 1649, 26, 2, 5, 163, 11, 3215, 2, 4, 1153, 9, 194, 775, 7, 2, 2, 349, 2637, 148, 605, 2, 2, 15, 123, 125, 68, 2, 2, 15, 349, 165, 4362, 98, 5, 4, 228, 9, 43, 2, 1157, 15, 299, 120, 5, 120, 174, 11, 220, 175, 136, 50, 9, 4373, 228, 2, 5, 2, 656, 245, 2350, 5, 4, 2, 131, 152, 491, 18, 2, 32, 2, 1212, 14, 9, 6, 371, 78, 22, 625, 64, 1382, 9, 8, 168, 145, 23, 4, 1690, 15, 16, 4, 1355, 5, 28, 6, 52, 154, 462, 33, 89, 78, 285, 16, 145, 95])\n",
      " list([1, 14, 47, 8, 30, 31, 7, 4, 249, 108, 7, 4, 2, 54, 61, 369, 13, 71, 149, 14, 22, 112, 4, 2401, 311, 12, 16, 3711, 33, 75, 43, 1829, 296, 4, 86, 320, 35, 534, 19, 263, 4821, 1301, 4, 1873, 33, 89, 78, 12, 66, 16, 4, 360, 7, 4, 58, 316, 334, 11, 4, 1716, 43, 645, 662, 8, 257, 85, 1200, 42, 1228, 2578, 83, 68, 3912, 15, 36, 165, 1539, 278, 36, 69, 2, 780, 8, 106, 14, 2, 1338, 18, 6, 22, 12, 215, 28, 610, 40, 6, 87, 326, 23, 2300, 21, 23, 22, 12, 272, 40, 57, 31, 11, 4, 22, 47, 6, 2307, 51, 9, 170, 23, 595, 116, 595, 1352, 13, 191, 79, 638, 89, 2, 14, 9, 8, 106, 607, 624, 35, 534, 6, 227, 7, 129, 113])\n",
      " list([1, 4, 2, 2, 33, 2804, 4, 2040, 432, 111, 153, 103, 4, 1494, 13, 70, 131, 67, 11, 61, 2, 744, 35, 3715, 761, 61, 2, 452, 2, 4, 985, 7, 2, 59, 166, 4, 105, 216, 1239, 41, 1797, 9, 15, 7, 35, 744, 2413, 31, 8, 4, 687, 23, 4, 2, 2, 6, 3693, 42, 38, 39, 121, 59, 456, 10, 10, 7, 265, 12, 575, 111, 153, 159, 59, 16, 1447, 21, 25, 586, 482, 39, 4, 96, 59, 716, 12, 4, 172, 65, 9, 579, 11, 2, 4, 1615, 5, 2, 7, 2, 17, 13, 2, 12, 19, 6, 464, 31, 314, 11, 2, 6, 719, 605, 11, 8, 202, 27, 310, 4, 3772, 3501, 8, 2722, 58, 10, 10, 537, 2116, 180, 40, 14, 413, 173, 7, 263, 112, 37, 152, 377, 4, 537, 263, 846, 579, 178, 54, 75, 71, 476, 36, 413, 263, 2504, 182, 5, 17, 75, 2306, 922, 36, 279, 131, 2895, 17, 2867, 42, 17, 35, 921, 2, 192, 5, 1219, 3890, 19, 2, 217, 4122, 1710, 537, 2, 1236, 5, 736, 10, 10, 61, 403, 9, 2, 40, 61, 4494, 5, 27, 4494, 159, 90, 263, 2311, 4319, 309, 8, 178, 5, 82, 4319, 4, 65, 15, 2, 145, 143, 2, 12, 2, 537, 746, 537, 537, 15, 2, 4, 2, 594, 7, 2, 94, 2, 3987, 2, 11, 2, 4, 538, 7, 1795, 246, 2, 9, 2, 11, 635, 14, 9, 51, 408, 12, 94, 318, 1382, 12, 47, 6, 2683, 936, 5, 2, 2, 19, 49, 7, 4, 1885, 2, 1118, 25, 80, 126, 842, 10, 10, 2, 2, 4726, 27, 4494, 11, 1550, 3633, 159, 27, 341, 29, 2733, 19, 4185, 173, 7, 90, 2, 8, 30, 11, 4, 1784, 86, 1117, 8, 3261, 46, 11, 2, 21, 29, 9, 2841, 23, 4, 1010, 2, 793, 6, 2, 1386, 1830, 10, 10, 246, 50, 9, 6, 2750, 1944, 746, 90, 29, 2, 8, 124, 4, 882, 4, 882, 496, 27, 2, 2213, 537, 121, 127, 1219, 130, 5, 29, 494, 8, 124, 4, 882, 496, 4, 341, 7, 27, 846, 10, 10, 29, 9, 1906, 8, 97, 6, 236, 2, 1311, 8, 4, 2, 7, 31, 7, 2, 91, 2, 3987, 70, 4, 882, 30, 579, 42, 9, 12, 32, 11, 537, 10, 10, 11, 14, 65, 44, 537, 75, 2, 1775, 3353, 2, 1846, 4, 2, 7, 154, 5, 4, 518, 53, 2, 2, 7, 3211, 882, 11, 399, 38, 75, 257, 3807, 19, 2, 17, 29, 456, 4, 65, 7, 27, 205, 113, 10, 10, 2, 4, 2, 2, 9, 242, 4, 91, 1202, 2, 5, 2070, 307, 22, 7, 2, 126, 93, 40, 2, 13, 188, 1076, 3222, 19, 4, 2, 7, 2348, 537, 23, 53, 537, 21, 82, 40, 2, 13, 2, 14, 280, 13, 219, 4, 2, 431, 758, 859, 4, 953, 1052, 2, 7, 2, 5, 94, 40, 25, 238, 60, 2, 4, 2, 804, 2, 7, 4, 2, 132, 8, 67, 6, 22, 15, 9, 283, 8, 2, 14, 31, 9, 242, 955, 48, 25, 279, 2, 23, 12, 1685, 195, 25, 238, 60, 796, 2, 4, 671, 7, 2804, 5, 4, 559, 154, 888, 7, 726, 50, 26, 49, 2, 15, 566, 30, 579, 21, 64, 2574])]\n"
     ]
    }
   ],
   "source": [
    "# Inspect a sample review and its label\n",
    "print(\"--- Review ---\")\n",
    "print(X_train[7])\n",
    "print(\"--- Label ---\")\n",
    "print(y_train[7])\n",
    "\n",
    "print(X_train[:4]) #Checking the data structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
      "1641221/1641221 [==============================] - 0s 0us/step\n",
      "--- Review (with words) ---\n",
      "['the', 'of', 'and', 'local', 'of', 'their', 'br', 'of', 'attention', 'widow', 'and', 'captures', 'parties', 'and', 'and', 'and', 'and', 'excitement', 'and', 'and', 'and', 'and', 'of', 'and', 'english', 'and', 'like', 'it', 'years', 'of', 'and', 'unintentional', 'this', 'hitchcock', 'of', 'and', 'learn', 'everyone', 'is', 'quite', 'and', 'it', 'and', 'such', 'it', 'bonus', 'film', 'of', 'too', 'seems', 'he', 'of', 'enough', 'for', 'be', 'and', 'editing', 'and', 'and', 'please', 'great', 'this', 'of', 'shoots', 'thing', '3', 'and', \"it's\", 'mentioning', 'and', 'and', 'given', 'parties', 'and', 'back', 'out', 'interesting', 'times', 'no', 'all', 'average', 'talking', 'some', 'of', 'nor', 'seems', 'into', 'best', 'at', 'of', 'every', 'cast', 'i', 'i', 'inside', 'keep', 'inside', 'large', 'viewer', 'who', 'obscure', 'and', 'and', 'and', 'movie', 'of', 'and', 'entirely', \"you've\", 'see', 'because', 'you', 'deals', 'successful', 'her', 'anything', 'it', 'of', 'dedicated', 'and', 'hard', 'and', 'further', \"that's\", 'takes', 'as', 'with', 'by', 'br', 'of', 'and', 'in', 'minute', 'and', 'they', 'of', 'westerns', 'watch', 'seemed', 'and', \"it's\", 'lee', 'if', 'oh', 'this', 'japan', 'film', 'around', 'get', 'an', 'of', 'and', 'always', 'life', 'was', 'between', 'of', 'and', 'with', 'group', 'rate', 'code', \"film's\", 'was', 'although', 'of', 'arts', 'had', 'death', 'time', 'and', 'of', 'anyway', 'romantic', 'their', 'won', 'in', 'kevin', 'only', 'flying', \"it's\", 'and', 'only', 'cut', 'show', 'if', 'and', 'is', 'star', 'stay', 'movies', 'both', 'and', 'stay', 'and', 'of', 'music', 'of', 'tell', 'missing', 'they', 'of', 'here', 'really', 'me', 'we', 'value', 'some', 'silent', 'music', 'as', 'had', 'thought', 'and', 'realized', 'she', 'in', 'sorry', 'reasons', 'is', 'and', '10', 'this', 'of', 'and', 'shoots', 'if', 'average', 'remembered', 'in', 'at', 'is', 'over', 'worse', 'film', 'is', 'and', 'it', 'for', 'had', 'absolutely', 'in', 'naive', 'want', 'it', 'for', 'had', 'absolutely', 'in', 'j', 'want', 'it', 'for', 'had', 'back', 'for', 'it', 'absolutely', 'in', 'one', 'want', 'shots', 'has', 'that', 'movie', 'of', 'here', 'write', 'whatsoever', 'it', 'is', 'and', 'set', 'got', 'worse', 'of', 'where', 'and', 'once', 'for', 'of', 'accent', 'after', 'saw', 'she', 'film', 'of', 'rest', 'little', 'and', 'camera', 'if', 'best', 'way', 'elements', 'know', 'of', 'and', 'also', 'an', 'were', 'sense', 'or', 'in', 'realistic', 'actually', 'satan', \"he's\", 'score', 'br', 'any', 'himself', 'in', 'another', 'type', 'english', 'this', 'is', 'and', 'was', 'tom', 'for', 'dating', 'get', \"it's\", 'such', 'from', 'fantastic', 'will', 'pace', 'new', 'years', 'of', 'guy', 'game', 'in', 'murders', 'this', 'us', 'hard', 'lives', 'film', 'and', 'fact', 'that', 'out', 'end', 'is', 'getting', 'together', 'br', 'and', 'of', 'seen', 'in', 'of', 'jail', 'for', 'sees', 'utterly', 'it', 'meet', \"it's\", 'depth', 'is', 'had', 'do', 'you', 'for', 'was', 'rather', 'convince', 'in', 'why', 'last', 'very', 'has', 'i', 'i', 'throughout', 'never', 'keep', 'viewer', 'who', 'of', 'becoming', 'switch', 'and', 'entirely', 'music', 'even', 'interest', 'scene', 'music', 'is', 'far', 'br', 'voice', 'riveting', 'is', 'again', 'something', 'br', 'decent', 'and', 'she', 'this', 'is', 'shoots', 'not', 'director', 'have', 'against', 'people', 'they', 'line', 'cinematography', 'film', 'is', 'couples', 'br', 'and', 'and', 'is', 'and', 'of', 'you', 'it', 'sees', 'hero', \"he's\", 'if', \"can't\", 'is', 'time', 'husband', 'silly', 'and', 'result', 'music', 'image', 'sequences', \"it's\", 'chase', 'music', 'is', 'veteran', 'include', 'and', 'freeman', 'not', 'of', 'and', 'it', 'along', 'are', 'of', 'hearing', 'cutting', 'music', 'his', 'get', 'scene', 'but', 'of', 'fact', 'correct', 'i', 'i', 'means', 'this', 'and', 'this', 'blockbuster', 'as', 'there', 'for', 'disappointed', 'along', 'wrong', 'few', 'has', 'that', 'if', 'his', 'weird', 'way', 'not', 'girl', 'display', 'of', 'love', 'who', 'so', 'friendship', 'in', 'we', 'down', 'it', 'director', 'in', 'situation', 'line', 'has', 'was', 'big', 'why', 'was', 'your', 'supposed', 'last', 'but', 'especially', 'i', 'i', 'of', 'and', 'and', 'of', 'and', 'internet', 'br', 'never', 'give', 'theme', 'rest', 'or', 'really', 'that', 'best', 'and', 'release', 'in', 'for', 'so', 'multi', 'random', 'their', 'even', 'interest', 'is', 'judge', 'once', 'arts', 'like', 'have', 'then', 'own', 'is', 'and', 'has', 'have', 'one', 'is', 'you', 'for', 'off', 'his', 'dutch', 'we', 'they', 'an']\n",
      "--- Label ---\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "# Map word IDs back to words\n",
    "word2id = imdb.get_word_index()\n",
    "id2word = {i: word for word, i in word2id.items()}\n",
    "print(\"--- Review (with words) ---\")\n",
    "print([id2word.get(i, \" \") for i in X_train[7]])\n",
    "print(\"--- Label ---\")\n",
    "print(y_train[7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike the Bag-of-Words approach, where we simply summarized the counts of each word in a document, this representation essentially retains the entire sequence of words (minus punctuation, stopwords, etc.). This is critical for RNNs to function. But it also means that now the features can be of different lengths! The following block of code shows the maximum/minimum review lengths in the data set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Maximum review length: 2697\n",
      "Minimum review length: 70\n"
     ]
    }
   ],
   "source": [
    "print('Maximum review length: {}'.format(\n",
    "len(max((X_train + X_test), key=len))))\n",
    "\n",
    "print('Minimum review length: {}'.format(\n",
    "len(min((X_train + X_test), key=len))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pad sequences\n",
    "\n",
    "In order to feed this data into my RNN, all input documents must have the same length. Let's limit the maximum review length to `max_words` by truncating longer reviews and padding shorter reviews with a null value (0). You can accomplish this easily using the [`pad_sequences()`](https://keras.io/preprocessing/sequence/#pad_sequences) function in Keras. For now, set `max_words` to 500."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras_preprocessing.sequence import pad_sequences\n",
    "\n",
    "# Setting the maximum number of words per document (for both training and testing)\n",
    "max_words = 500\n",
    "\n",
    "# Pad sequences in X_train and X_test\n",
    "X_train = pad_sequences(X_train, maxlen=max_words)\n",
    "X_test = pad_sequences(X_test, maxlen=max_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Design an RNN model for sentiment analysis\n",
    "\n",
    "Building my architecture in the code cell below. I've imported some layers from Keras that I might need. (Any other preferred layers / transformations can be used instead).\n",
    "\n",
    "Remember that your input is a sequence of words (technically, integer word IDs) of maximum length = `max_words`, and your output is a binary sentiment label (0 or 1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes: Embedding\n",
    "\n",
    "- In an embedding, words are represented by dense vectors where a vector represent the projection of the word into a continuous vector space. \n",
    "- The position of a word within the vector space is learned from text (based on the context words that surround the word when it is used).\n",
    "- The position of a word in the learned vector space = its embedding\n",
    "- Examples: Word2Vec(by Google); GloVe(by Stanford)\n",
    "\n",
    ">Keras offers an embedding layer that can be used for NNs on text data. It **requires** that the input data be integer-encoded so that each word is represented by a unique integer as the data set used in this analysis. (Tokenizer API provided with Keras can be used in such data preparation).\n",
    "\n",
    "- The embedding layer is initialized with random weights and will learn an embedding for **all of the words in the training data set**.\n",
    "- This embedding layer is defined as the first hidden layer of a NN.\n",
    "- This embedding layer had three arugments:\n",
    "> **input_dim** (= the size of the vocab in the text data. For example,if the given data is coded to values between 0-10, then the size of vocab would be 11 words.)<br>\n",
    "**output_dim** (= the size of the vector space in which words will be embedded. It defines the size of the output vectors from this layer for each word.)<br>\n",
    "**input_length** (= the length of input sequences; e.g., if all of the documents in the given data set comprise 1000 words, this would be 1000).\n",
    "\n",
    "In the following code block, this part refers to `Embedding(vocabulary_size, embedding_size, input_length = max_words))`.\n",
    "Earlier we set vocabulary size = 5,000; max_words = 500 (all input sequences were padded to have the length of 500 earlier).\n",
    "\n",
    "The output of the embedding layer is a 2D vector with one embedding for each word in the input sequence of words. In the given example, the output will be 500 vectors of 32 dimensions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_1 (Embedding)     (None, 500, 32)           160000    \n",
      "                                                                 \n",
      " bidirectional_1 (Bidirectio  (None, 100)              33200     \n",
      " nal)                                                            \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 1)                 101       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 193,301\n",
      "Trainable params: 193,301\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, LSTM, Dense, Dropout, Bidirectional\n",
    "\n",
    "# Design your model\n",
    "embedding_size = 32 #64 #128 \n",
    "model = Sequential()\n",
    "model.add(Embedding(vocabulary_size, embedding_size, input_length=max_words))\n",
    "model.add(Bidirectional(LSTM(50, dropout=0.4, return_sequences=False), merge_mode='concat'))\n",
    "\n",
    "#model.add(LSTM(64, dropout=0.5, return_sequences=False))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "print(model.summary())\n",
    "\n",
    "#It is not included in the code, but usually the output \n",
    "#from the embedding layer is *flattened* before being passed to the Dense layer.\n",
    "#model.add(Flatten()) before model.add(Dense(...))\n",
    "#The result of the flatten layer will be a 500*32-element squashed vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A summary of the layers above shows:\n",
    "\n",
    "**Embedding layer**<br>\n",
    "There is one embedding layer in this NN model. The embedding of this NN has a vocabulary of 5,000 and an input length of 500. I chose a embedding space of 32 dimensions. As mentioned earlier, the output of this embedding layer has 500 vectors (i.e., one for each word) of the 32 dimensions (i.e., 500 x 32 matrix).\n",
    "\n",
    "**LSTM layer**<br>\n",
    "There is one bi-directional LSTM layer.\n",
    "\n",
    "**Dense layer**<br>\n",
    "There is one Dense layer in this NN.\n",
    "\n",
    "A total of 193,301 parameters were trained in this RNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and evaluate your model\n",
    "\n",
    "Now we are ready to train our model. In Keras world, we first need to _compile_ our model by specifying the **loss function** and **optimizer** we want to use while training, as well as any **evaluation metrics** we would like to measure. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tompile your model, specifying a loss function, optimizer, and metrics\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once compiled, we can kick off the training process. There are two important training parameters that we have to specify - **batch size** and **number of training epochs**, which  determine the total training time (together with your model architecture).\n",
    "\n",
    "Training may take a while, so grab a cup of coffee, or better, go for a hike! If possible, consider using a GPU, as a single training run can take several hours on a CPU.\n",
    "\n",
    "> **Tip**: You can split off a small portion of the training set to be used for validation during training. This will help monitor the training process and identify potential overfitting. You can supply a validation set to `model.fit()` using its `validation_data` parameter, or just specify `validation_split` - a fraction of the training data for Keras to set aside for this purpose (typically 5-10%). Validation metrics are evaluated once at the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "88/88 [==============================] - 64s 680ms/step - loss: 0.6073 - accuracy: 0.6639 - val_loss: 0.5758 - val_accuracy: 0.6972\n",
      "Epoch 2/5\n",
      "88/88 [==============================] - 60s 682ms/step - loss: 0.4317 - accuracy: 0.8091 - val_loss: 0.3339 - val_accuracy: 0.8648\n",
      "Epoch 3/5\n",
      "88/88 [==============================] - 59s 673ms/step - loss: 0.3141 - accuracy: 0.8736 - val_loss: 0.3121 - val_accuracy: 0.8760\n",
      "Epoch 4/5\n",
      "88/88 [==============================] - 61s 688ms/step - loss: 0.2518 - accuracy: 0.9041 - val_loss: 0.3088 - val_accuracy: 0.8756\n",
      "Epoch 5/5\n",
      "88/88 [==============================] - 60s 684ms/step - loss: 0.2307 - accuracy: 0.9119 - val_loss: 0.3091 - val_accuracy: 0.8712\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7ff1a90d7d90>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.callbacks import TensorBoard\n",
    "from time import time\n",
    "tensorboard = TensorBoard(log_dir=\"logs/{}\".format(time()), histogram_freq=1, write_graph=False)\n",
    "\n",
    "# Specify training parameters: batch size and number of epochs\n",
    "batch_size = 256 # 128 # 64 \n",
    "num_epochs = 5\n",
    "\n",
    "# Train your model\n",
    "model.fit(X_train, y_train,\n",
    "                  validation_split=0.1,\n",
    "                  shuffle = True,\n",
    "                  batch_size=batch_size, epochs=num_epochs,\n",
    "                  verbose=1, callbacks=[tensorboard])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "cache_dir=\"./model\"\n",
    "# Save your model, so that we can quickly load it in future (and perhaps resume training)\n",
    "model_file = \"rnn_model.h5\"  # HDF5 file\n",
    "model.save(os.path.join(cache_dir, model_file))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "cache_dir=\"./model\"\n",
    "\n",
    "# Later you can load it using keras.models.load_model()\n",
    "from keras.models import load_model\n",
    "#model = load_model(os.path.join(cache_dir, model_file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have trained your model, it's time to see how well it performs on unseen test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.8743199706077576\n"
     ]
    }
   ],
   "source": [
    "# Evaluate your model on the test set\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)  # returns loss and other metrics specified in model.compile()\n",
    "print(\"Test accuracy:\", scores[1])  # scores[1] should correspond to accuracy because we passed in metrics=['accuracy']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy improved compared to the models with the regular classifiers (Naive Bayes: 0.689; Gradient Boost Decision Tree: 0.81)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Extensions\n",
    "\n",
    "There are several ways in which we can build upon this notebook.\n",
    "\n",
    "[1] Improving the accuracy of your model by experimenting with different architectures, layers and parameters. How good can you get without taking prohibitively long to train? How do you prevent overfitting?\n",
    "\n",
    "[2] Deploying your model as a mobile app or web service. What do you need to do in order to package your model for such deployment? How would you accept a new review, convert it into a form suitable for your model, and perform the actual prediction? (Note that the same environment you used during training may not be available.)\n",
    "\n",
    "[3] Multi-class classificaiton: The dataset actually includes a more fine-grained review rating that is indicated in each review's filename (which is of the form `<[id]_[rating].txt>` where `[id]` is a unique identifier and `[rating]` is on a scale of 1-10; note that neutral reviews > 4 or < 7 have been excluded). \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}