Part2_TVMLAI_CNN_LSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Natural Language Processing in Keras, Part 2\n",
    "\n",
    "Alright, let's look at some code! First, let's load in the libraries we will need.\n",
    "\n",
    "# How does this relate to deep learning?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# LSTM and CNN for sequence classification in the IMDB dataset\n",
    "import numpy as np\n",
    "import keras\n",
    "from keras.datasets import imdb, reuters\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Input, Dense, LSTM, Dropout, RepeatVector\n",
    "\n",
    "from keras.layers.noise import GaussianNoise\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.layers.convolutional import Convolution1D, MaxPooling1D\n",
    "\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras_tqdm import TQDMNotebookCallback\n",
    "\n",
    "import helper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Keras recently rolled out V2.0 with some breaking changes. I'll try to keep this tutorial as v1/v2 compatible as possible in case people are still running v1. \n",
    "If you see errors like `TypeError: Received unknown keyword arguments: {'epochs': 3}`, or size/shape mismatches, you probably have a version mismatch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.2\n"
     ]
    }
   ],
   "source": [
    "print(keras.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Fix random seed for reproducibility\n",
    "np.random.seed(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Preprocess the dataset into suitable shape to feed to NN\n",
    "\n",
    "We will load in the data into memory, and then process it from the current form into a form that our model can accept. \n",
    "\n",
    "### Note on sequence length\n",
    "\n",
    "The variable `sequence_len` here describes the length of the review data that we feed into the network. The longer this is, the better context the network can formulate about the review, and hopefully lead to better accuracy. However, this comes at a cost of longer training times. Initially, I was using len = 500. I cut it down to 50 so that this demo can run in less time, and amazingly the network functions nearly as well! That is the awesome power of deep learning. Feel free to experiment with any of these parameters!\n",
    "\n",
    "### Basic parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\n",
    "nb_top_words           = 5000  # Keep the top n words, zero the rest. These will be our 'vocab' in our abridged corpus\n",
    "sequence_len           = 50    # This is the length in the \"time axis\". Originally was 500, but it turns out 50 works *almost* as well, and way faster!\n",
    "embed_vector_len       = 32    # Size of the feature vector each word will map to\n",
    "nb_lstm                = 100   # Number of LSTM nodes\n",
    "batch_size             = 64    # Number of samples to feed into the model for each forward/backward pass\n",
    "\n",
    "DEFAULT_EPOCHS         = 1     # For testing the whole notebook quickly. \n",
    "RUN_EVERYTHING         = False # switch for letting the whole notebook execute (including training, which can take a while)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Load the data\n",
    "In case you missed the preproccessing part, or for some reason the data isn't working, you can get the [data for this step directly](https://www.dropbox.com/s/cu1kpojxzai5ru4/imdb_rotten_proc.zip?dl=0). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "data_path = '/media/mike/tera/data/nlp/techvalley/' # Point this to the path to where IMDB data is stored\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = helper.unpackage_dataset(data_path + 'imdb.npz')\n",
    "\n",
    "(x_rotten_train, y_rotten_train), (x_rotten_test, y_rotten_test) = helper.unpackage_dataset(data_path + 'rotten.npz')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The data is split into training and test sets so we can perform simple validation. A better approach would be k-fold cross-validation, but that is a topic for another talk. \n",
    "<script>/* unused section, ignore this!\n",
    "\n",
    "# This is for running data from a slightly different data set. You can probably ignore this, but I'm keeping it\n",
    "# in the tutorial in case I need to make last minute changes. \n",
    "\n",
    "# (x_train, y_train), (x_test, y_test) = imdb.load_data(nb_words=nb_top_words, skip_top=0, index_from=3)#nb_words=nb_top_words)\n",
    "# twitter_x = np.load('data/twitter_phrases.npy') # pre-converted indexes from Twitter set using IMDB word list\n",
    "# twitter_y = np.load('data/twitter_labels.npy')\n",
    "# rotten_x = np.load('data/rotten_phrases.npy') # pre-converted indexes from Twitter set using IMDB word list\n",
    "# rotten_y = np.load('data/rotten_labels.npy')\n",
    "\n",
    "\n",
    "## Let's peer into the data. I always like to start by getting a feel for the data\n",
    "\n",
    "We processed the data ourselves, but since I wrote this section before the preprocessing section, it's a little redundant. \n",
    "\n",
    "> This data is in the form of a sequence of integers.  Each `int` maps to a word in the corpus dictionary. It is a giant lookup table, with the index roughly corresponding to the frequency rank of the word. This is necessary because computers do not directly process words, only numbers. \n",
    "\n",
    "word_index = imdb.get_word_index()\n",
    "index_word = {value: key for (key, value) in word_index.items()} # flip key:value pairs to get the integer as the key\n",
    "index_word.update({0: \" \"})\n",
    "list(index_word.items())[:10]\n",
    "\n",
    "sample_num = 3\n",
    "x_train[sample_num][:10]\n",
    "\n",
    "' '.join([index_word[idx] for idx in x_train[sample_num]])\n",
    "\n",
    "*/ </script>\n",
    "\n",
    "<script> /* \n",
    "Unused stuff, ignore this!\n",
    "# wat.\n",
    "Ok, I totally did not expect that. It seems super nonsensical. I dug into it and I think the situation is that the Keras IMDB data has already been processed with bag-of-words. I left this in here because it's important to remember that **Data science is a science**. Unexpected things happen all the time. \n",
    "</script>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Resize the arrays\n",
    "#### Next, we need to pad the tensors out to the proper dimension in the time axis. \n",
    "Even though LSTM can handle variable length data, the backend still prefers rectangular tensors. We will pad/crop our variable-length movie reviews so that they are all exactly the same length. Once we have regularly-sized tensors, we are ready to build the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x_train = sequence.pad_sequences(x_train, maxlen=sequence_len)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=sequence_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 50) (25000, 50)\n"
     ]
    }
   ],
   "source": [
    "# twitter_x = sequence.pad_sequences(twitter_x, maxlen=sequence_len)\n",
    "x_rotten_test = sequence.pad_sequences(x_rotten_test, maxlen=sequence_len)\n",
    "print(x_train.shape, x_test.shape,)# twitter_x.shape, rotten_x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Without further ado, let's build an RNN in 5 lines of code. I'll walk you through each layer in detail. \n",
    "___\n",
    "# (Model 1) Basic LSTM\n",
    "\n",
    "First, we initialize the model.\n",
    "\n",
    "> ```python\n",
    "model = Sequential()```\n",
    "\n",
    "The [Keras Sequential Model](https://keras.io/getting-started/sequential-model-guide/) is based around building up the model layer-by-layer, like a cake. This is the easiest to graps for beginners, and works well, since many, if not most, neural networks can be represented this way. \n",
    "Calling model.add(layer) sticks the layer onto the topmost, and that becomes the new top. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Embedding\n",
    "\n",
    "We start off with an efficient embedding layer which maps our vocabulary to a lower-dimensional space.\n",
    "\n",
    "> ```python\n",
    "model.add(Embedding(nb_top_words,                 \n",
    "                    embed_vector_len, \n",
    "                    input_length=sequence_len\n",
    "                   ))\n",
    "```\n",
    "\n",
    "#### One-hot encoding\n",
    "\n",
    "Currently, we have every word mapped to some integer, which is great because the computer can parse it, but another problem arises: These numbers are sort of arbitrary, and not much like the \"thought vector\" idea from before. Take the numbers 55 (time) and 56 (she): they have absolutely no conection, even though they are numerically close. Since neural networks are essentially giant (non)linear equations, this is not ideal. Larger numbers would get more 'weight' than smaller ones. What we need to do is map it in to some sort of categorical representation, where each word is on 'equal footing'. \n",
    "\n",
    "We achieve this by using a [*one hot* representation](https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science). This is where you have an Nx1 vector, where the dimension associated with the integer has value 1 ('hot') and all other are 0. N is the size of your vocabulary. Each word would look like this:\n",
    "\n",
    "[0,0,0,0,0,0,0,...,0,**1**,0,0,...,0,0,0]\n",
    "<br>        ^ this is the $kth$ dim, where $k$ corresponds to the index in the table\n",
    "\n",
    "\n",
    "Here is a toy example:\n",
    "\n",
    "```\n",
    "here    = [0,0,0,0,1,0]\n",
    "is      = [0,1,0,0,0,0]\n",
    "a       = [1,0,0,0,0,0]\n",
    "toy     = [0,0,0,1,0,0]\n",
    "example = [0,0,0,0,0,1]\n",
    "```\n",
    "\n",
    "Normally, you would have way more dimensions (typically thousands). Fortunately, Keras takes care of this automatically. We give it a sequence of M integers, and it'll automatically convert it to a NxM matrix.\n",
    "\n",
    "Currently, the \"shape\" of each data vector is Nx1, where N = 5000. That's really problematic, due to the [curse of dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality) and also computational constraints. We want to transform our **uncondensed** and **sparse** data into a **compact** and **dense** representation, in our case a 32-vector. This process is known as embedding and is what gives us our **dense** thought vector. **Word2Vec** is a popular word embedding system. Keras Embedding layer works very similarly, so if you want to know more, I suggest checking out how Word2Vec works. \n",
    "\n",
    "Keras will do this automatically for us with the Embedding layer. After embedding, we will have a much more manageable 32x50 matrix. Each \"thought\" has dimensionality 32, and a sequence consists of 50 of these vectors in order. \n",
    "\n",
    "![Socher-Bilingual-tsne](http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/img/Socher-BillingualTSNE.png)\n",
    "<center> t-SNE visualization of the bilingual word embedding. Green is Chinese, Yellow is English. (Socher et al. (2013a))</center>\n",
    "\n",
    "### LSTM\n",
    "\n",
    "> ```python\n",
    "model.add(LSTM(nb_lstm))```\n",
    "\n",
    "\n",
    "The LSTM (long short-term memory) cell is a neuron with memory. This allows the neuron to remember things over a span of time (such as from the start of a review to the end). It accomplishes this by have a memory state, which can be written to and read from. It's like a tiny ~~casette tape~~  [~~floppy disk~~](http://i.imgur.com/Osxo1UF.jpg)  USB flash drive. In short, the inputs from the prior layer (mathematically) control gates. These gates determine whether to erase, write, and/or read from the memory cell. The actual workings of these units are quite involved, so I won't go into much detail. LSTMs have been covered really well in depth in a lot of places. In particular I recommend the articles by [Chris Olah](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) and [Andrej Karpathy](http://karpathy.github.io/2015/05/21/rnn-effectiveness/). \n",
    "\n",
    "![lstm](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png)\n",
    "\n",
    "![LSTM](http://tc.sinaimg.cn/maxwidth.800/tc.service.weibo.com/cdn_images_1_medium_com/58ad765e09eacb5116c9dfc5897c7296.png)\n",
    "\n",
    "### Dense\n",
    "\n",
    "> ```python\n",
    "model.add(Dense(1, activation='sigmoid'))```\n",
    "\n",
    "\n",
    "Finally, we have a densely-connected layer. This is your \"typical\" neural network layer - each node from the prior layer connects to each node of the following. In this case, we are crunching down to a single node since we want a single answer - \"Positive\" or \"Negative\". We'll use a sigmoid activation to squash the output to the range 0-1. \n",
    "\n",
    "That's it for our network! Pretty simple, right? \n",
    "\n",
    "> [\"Keras is so good that it is effectively cheating in machine learning\"](https://news.ycombinator.com/item?id=13872670)\n",
    "\n",
    "### Compiling \n",
    "\n",
    "> ```python\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])```\n",
    "\n",
    "All that's left is compiling the model. This tells the model which loss function, optimizer, and metrics to use. \n",
    "\n",
    "- **Optimizer**: This is the algorithm which describes how we approach gradient descent. Adam is pretty modern and works quite well for a lot of problems, so it is typically the first go-to when picking hyperparameters\n",
    "- **Metrics**: This does not actually affect the direct training of the model. Rather, it gives us humans a way to track the performance of the model over time. This can also be used for automatically early-stopping to avoid overfitting. \n",
    "- **Loss function**: This determines how the \"penalty\" for incorrect predictions is calculated. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Brief tangent: Cross-entropy\n",
    "I'm working on a [simple summary of Cross-Entropy](https://github.com/xkortex/TechValleyMachineLearning/blob/master/CrossEntropy.ipynb). If you would like to know more, then check out that post, otherwise it's a bit of a tangent for this particular project. \n",
    "\n",
    "For now, all we need to know is cross-entropy is a very common loss function, and many Keras models use binary (yes/no problems) or categorical (multiple labels) cross-entropy. You are probably familiar with Mean Squared Error, which is a commonly used loss function if you are performing continuous regression. Cross-entropy is used to predict labels (logistic regression). The legendary [Andrew Ng Coursera Course](https://www.coursera.org/learn/machine-learning) covers this in more detail. \n",
    "___\n",
    "# Let's write the code for the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Really basic LSTM model. \n",
    "model = Sequential()\n",
    "model.add(Embedding(nb_top_words,                 \n",
    "                    embed_vector_len, \n",
    "                    input_length=sequence_len\n",
    "                   ))\n",
    "model.add(LSTM(nb_lstm))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 50, 32)            160000    \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 100)               53200     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 101       \n",
      "=================================================================\n",
      "Total params: 213,301.0\n",
      "Trainable params: 213,301\n",
      "Non-trainable params: 0.0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Have a look at our model\n",
    "print(model.summary())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Note the shape of each layer\n",
    "Each layer will have a shape (N, a, b, ...). \n",
    "\n",
    "- The \"None\" for the first dimension indicates that the model will be fit to whatever our batch size is. \n",
    "- The embedding layer is (None, 50, 30), indicating it has a variable batch size, a time sequence length of 50, and a vector size of 32. \n",
    "- The LSTM is (None, 100). Since it is taking in a sequence, our sequence length parameter disappears, and we are left with the number of nodes (100)\n",
    "- The dense layer smashes the output of 100 LSTMs down to 1 node\n",
    "\n",
    "\n",
    "# Fit the model to the data\n",
    "This is where the magic happens! The Tensor Gnomes will do their ritual dance, and the weights will manifest new values.\n",
    "\n",
    "...What do you mean, that's not how it works?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# This will take about 20-60 seconds per epoch on a GPU. CPU will take anywhere from 30s (i7 w/ 8 threads) to several minutes per epoch. \n",
    "# Larger networks will see a bigger advantage to GPU, though CPU did not do as poorly as I expected!\n",
    "# We'll be making improvements to the model, so you don't have to run this just yet.\n",
    "RUN_MODEL1 = False # I'm just using a switch here so I can run Kernel -> Restart and Run All\n",
    "if RUN_MODEL1 or RUN_EVERYTHING: \n",
    "    model.fit(x_train, y_train, \n",
    "              validation_data=(x_test, y_test), \n",
    "              epochs=DEFAULT_EPOCHS, \n",
    "              batch_size=batch_size, \n",
    "              verbose=0, # Some versions of Jupyter bork on Keras' progress bar. We replace it with Keras-TQDM instead\n",
    "              callbacks=[TQDMNotebookCallback()])\n",
    "    loss, metric = model.evaluate(x_test, y_test, verbose=0)\n",
    "    print('\\nIMDB:\\nLabel ratio: {:.2f}%'.format(np.mean(y_test*100)))\n",
    "    print('Loss: {}\\nAccuracy: {:.2f}%'.format(loss, metric*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This network will get us to about 87% accuracy. However, we had to stop pretty early because of the risk of overfitting. A model that overfits easy is often a strong sign that the model will generalize poorly to new, unseen data, or data from different distributions. Let's see how it performs on a similar dataset, the ~~Twitter~~ Rotten Tomatoes dataset. This will be challenging, as the reviews are much shorter, which gives the RNN less time to 'get up to speed'.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "if RUN_MODEL1 or RUN_EVERYTHING:\n",
    "    loss, metric = model.evaluate(x_rotten_test, y_rotten_test)\n",
    "    print('\\nRotten dataset:\\nLabel ratio: {:.2f}%'.format(np.mean(y_rotten_test*100)))\n",
    "    print('Loss: {}\\nAccuracy: {:.2f}%'.format(loss, metric*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Woohoo!\n",
    "### It's not quite as good as the IMDB, but considering it's from a totally different corpus, this is quite good.\n",
    "Let's see if we can do a bit better with the generalization. Since the dataset we want to extrapolate to (Twitter, Rotten Tomatoes, etc) is different in several ways, we want our IMDB-trained network to be robust to noise, idiosyncracies, and quirks unique to IMDB that do not generalize to other formats.  \n",
    "\n",
    "___\n",
    "# ( Model 2) Adding dropout and noise\n",
    "\n",
    "**Dropout** is a technique for reducing overfitting in neural networks by preventing over-adaptation to the training data. The general idea is, every cycle you randomly drop a certain percentage of nodes or connections. This forces the network to compensate by distributing over multiple nodes and prevents any given node from getting too \"specialized\". \n",
    "\n",
    "Adding noise is another way of reducing overfitting and improving generalization. By adding noise to the vectors, this forces the network to learn to compensate, just as humans process stimuli in a noisy environment.\n",
    "\n",
    "![dropout](http://cdn-ak.f.st-hatena.com/images/fotolife/o/olanleed/20131130/20131130221427.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dropout, Noise, Batch Normalization, and other similar regularization techniques are largely empirical. That's a fancy way of saying, \"We have no mathematical underpinning as to why these work well\". Data scientists just sort of messed around, or used large hyperparameter searches to find configurations that work well. If you hear to someone refer to deep learning as \"voodoo science\" or \"dark arts\", this is what they are talking about.\n",
    "\n",
    "![no idea](https://img.memesuper.com/5da3ce3ad2ddff8c4b752e089ede7d8d_download-pin-have-no-idea-dog-meme-i-have-no-idea-what-im-doing_455-290.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\n",
    "# Simple LSTM with noise and dropout\n",
    "dropout_rate = 0.2 # Rate of input units to drop\n",
    "sigma=0.5         # Amount of noise to add (in terms of standard deviation)\n",
    "\n",
    "model=Sequential()\n",
    "model.add(Embedding(nb_top_words,                 \n",
    "                    embed_vector_len, \n",
    "                    input_length=sequence_len\n",
    "                   ))\n",
    "model.add(GaussianNoise(stddev=sigma))\n",
    "model.add(Dropout(dropout_rate))\n",
    "model.add(LSTM(nb_lstm))\n",
    "model.add(Dropout(dropout_rate))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_5 (Embedding)      (None, 50, 32)            160000    \n",
      "_________________________________________________________________\n",
      "gaussian_noise_3 (GaussianNo (None, 50, 32)            0         \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 50, 32)            0         \n",
      "_________________________________________________________________\n",
      "lstm_4 (LSTM)                (None, 100)               53200     \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 1)                 101       \n",
      "=================================================================\n",
      "Total params: 213,301.0\n",
      "Trainable params: 213,301.0\n",
      "Non-trainable params: 0.0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note the shape of each layer\n",
    "\n",
    "- The gaussian noise and dropout layers are (None, 50, 30), since they are merely applying to the output of embedding (like a filter)\n",
    "- The LSTM is (None, 100), like before\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# This will take about 1-6 minutes per epoch on a GPU. \n",
    "RUN_MODEL2 = False\n",
    "if RUN_MODEL2 or RUN_EVERYTHING:\n",
    "    model.fit(x_train, y_train, \n",
    "              validation_data=(x_test, y_test), \n",
    "              nb_epoch=DEFAULT_EPOCHS, \n",
    "              batch_size=batch_size, \n",
    "              verbose=0, # Some versions of Jupyter bork on Keras' progress bar. We replace it with Keras-TQDM instead\n",
    "              callbacks=[TQDMNotebookCallback()])\n",
    "    loss, metric = model.evaluate(x_test, y_test, verbose=0)\n",
    "    print('IMDB:\\nLabel ratio: {:.2f}%'.format(np.mean(y_test*100)))\n",
    "    print('Loss: {}\\nAccuracy: {:.2f}%'.format(loss, metric*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "if RUN_MODEL2 or RUN_EVERYTHING:\n",
    "    loss, metric = model.evaluate(x_rotten_test, y_rotten_test)\n",
    "    print('\\nRotten Dataset:\\nLabel ratio: {:.2f}%'.format(np.mean(y_rotten_test*100)))\n",
    "    print('Loss: {}\\nAccuracy: {:.2f}%'.format(loss, metric*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Accuracy dropped a little bit to about 64%, but we are running the same number of epochs. \n",
    "Noise layers trade training speed for generalizability. If we run this for a total of 6 epochs, we get up to about 74%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n",
    "# (Model 3) LSTM + CNN = REAL ULTIMATE POWER!\n",
    "\n",
    "![POWAH!](https://memecrunch.com/meme/9IO88/phenomenal-cosmic-power/image.jpg?w=650&c=1)\n",
    "\n",
    "#### LSTMs are awesome. Convolutional neural networks are awesome. I was blown away when I learned you can simply and easily combine both into the same model!\n",
    "\n",
    "### But why does this work?\n",
    "\n",
    "Convolutional layers look at local structures in the data. Think of each node in a CNN as a filter which looks for a specific \"shape\" in the data. In image classifiers, this is visual features. Low-level features are things like edges and gradients. Higher level features correspond to more complex shapes.\n",
    "\n",
    "![cnn](https://upload.wikimedia.org/wikipedia/commons/6/63/Typical_cnn.png)\n",
    "\n",
    "In NLP, this looks at groups of words, or N-grams (sequence of N words). For instance, many English sentences contain N-grams of the form [**subject verb object**]. For example (dropping articles for simplicity):\n",
    "\n",
    "- John rode bike\n",
    "- Suzie hit ball\n",
    "- Bobby made memes\n",
    "\n",
    "Another common sequence is [**subject copula predicate**] (where copula verbs are 'linking' verbs: `is`, `are`, `was`, `will be`): \n",
    "\n",
    "- Roses are red\n",
    "- Movie was bad\n",
    "- Keras is awesome\n",
    "- CUDA is rad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "nb_filter     = 256     # This is the number of convolutional filters to use\n",
    "filter_length = 5  # This is the size of the filter kernel. Since this is 1D, the kernel is Nx1\n",
    "pool_length   = 2    # Size of our max pooling structures\n",
    "dropout_rate  = 0.2 # Rate of input units to drop\n",
    "stddev        = 0.5         # Amount of noise to add (in terms of standard deviation)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(nb_top_words, embed_vector_len, input_length=sequence_len))\n",
    "model.add(GaussianNoise(stddev=sigma))\n",
    "model.add(Convolution1D(filters=nb_filter, kernel_size=filter_length, padding='same', activation='relu'))\n",
    "model.add(MaxPooling1D(pool_size=pool_length))\n",
    "model.add(Dropout(dropout_rate))\n",
    "model.add(LSTM(nb_lstm))\n",
    "model.add(Dropout(dropout_rate))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_6 (Embedding)      (None, 50, 32)            160000    \n",
      "_________________________________________________________________\n",
      "gaussian_noise_4 (GaussianNo (None, 50, 32)            0         \n",
      "_________________________________________________________________\n",
      "conv1d_3 (Conv1D)            (None, 50, 256)           41216     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_2 (MaxPooling1 (None, 25, 256)           0         \n",
      "_________________________________________________________________\n",
      "dropout_9 (Dropout)          (None, 25, 256)           0         \n",
      "_________________________________________________________________\n",
      "lstm_5 (LSTM)                (None, 100)               142800    \n",
      "_________________________________________________________________\n",
      "dropout_10 (Dropout)         (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 1)                 101       \n",
      "=================================================================\n",
      "Total params: 344,117.0\n",
      "Trainable params: 344,117.0\n",
      "Non-trainable params: 0.0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note the shape of each layer\n",
    "Each layer will have a shape (N, a, b, ...). \n",
    "\n",
    "- The convolutional layer is (None, 50, 256). It has sequence length 50 and 256 filters. It basically takes our input thought vector sequence, and creates a sequence of N-gram thought vectors\n",
    "- The max pooling layer is (None, 25, 256). It takes two adjacent points in the sequence, and only retains the maximum. This halves the number of points in the sequence.\n",
    "- Dropout, LSTM and dense layers are like before. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# This will take about 0.5-2 minutes per epoch on a GPU. \n",
    "RUN_MODEL3 = False\n",
    "if RUN_MODEL3 or RUN_EVERYTHING:\n",
    "    model.fit(x_train, y_train, \n",
    "              validation_data=(x_test, y_test), \n",
    "              nb_epoch=DEFAULT_EPOCHS, \n",
    "              batch_size=batch_size, \n",
    "              verbose=0, # Some versions of Jupyter bork on Keras' progress bar. We replace it with Keras-TQDM instead\n",
    "              callbacks=[TQDMNotebookCallback()])\n",
    "    \n",
    "    loss, metric = model.evaluate(x_test, y_test, verbose=0)\n",
    "    print('IMDB:\\nLabel ratio: {:.2f}%'.format(np.mean(y_test*100)))\n",
    "    print('Loss: {}\\nAccuracy: {:.2f}%'.format(loss, metric*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "if RUN_MODEL3 or RUN_EVERYTHING:\n",
    "#     loss, metric = model.evaluate(twitter_x, twitter_y)\n",
    "#     print('Label ratio: {:.2f}%'.format(np.mean(twitter_y*100)))\n",
    "#     print('Loss: {}\\nMetric: {:.2f}%'.format(loss, metric*100))\n",
    "    loss, metric = model.evaluate(x_rotten_test, y_rotten_test)\n",
    "    print('\\nRotten Data:\\nLabel ratio: {:.2f}%'.format(np.mean(y_rotten_test*100)))\n",
    "    print('Loss: {}\\nMetric: {:.2f}%'.format(loss, metric*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Excellent! After 3 epochs, we get about 73% accuracy on the Rotten Tomatoes database, when trained on the IMDB database!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# References\n",
    "\n",
    "## Datasets\n",
    "\n",
    "http://www.cs.cornell.edu/people/pabo/movie-review-data/\n",
    "http://snap.stanford.edu/data/web-Amazon.html\n",
    "\n",
    "## Papers\n",
    "- [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215)\n",
    "- [Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432)\n",
    "- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://arxiv.org/abs/1406.1078)\n",
    "\n",
    "## Code\n",
    "\n",
    "- [Keras Examples: IMDB CNN](https://github.com/fchollet/keras/blob/master/examples/imdb_cnn.py)\n",
    "- [Machine Learning Mastery: Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras](http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/)\n",
    "- [Microway: Building a Movie Review Sentiment Classifier using Keras and Theano Deep Learning Frameworks](https://www.microway.com/hpc-tech-tips/keras-theano-deep-learning-frameworks/)\n",
    "- [Using  Pre-trained Word Embeddings in Keras](https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html)\n",
    "\n",
    "## Tangentally related\n",
    "\n",
    "- [McCormick: Word2vec Tutorial](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___\n",
    "# Bonus feature: Pure CNN Model\n",
    "\n",
    "I was not sure how much ground I would be able to cover, so this section is here in case people are having a great time, and are really thirsty for knowledge! This is a pure-CNN based model, and works by essentially dumping all the N-grams together (global pooling). It does not care about the respective order of N-grams. And yet, it works quite well! It is a bit slower to train than the LSTM with seq_len=50, however."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mike/ve/keras/lib/python3.5/site-packages/keras/datasets/imdb.py:44: UserWarning: The `nb_words` argument in `load_data` has been renamed `num_words`.\n",
      "  warnings.warn('The `nb_words` argument in `load_data` '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000 train sequences\n",
      "25000 test sequences\n",
      "Pad sequences (samples x time)\n",
      "x_train shape: (25000, 400)\n",
      "x_test shape: (25000, 400)\n"
     ]
    }
   ],
   "source": [
    "'''This example demonstrates the use of Convolution1D for text classification.\n",
    "Gets to 0.89 test accuracy after 2 epochs.\n",
    "90s/epoch on Intel i5 2.4Ghz CPU.\n",
    "10s/epoch on Tesla K40 GPU.\n",
    "'''\n",
    "\n",
    "from __future__ import print_function\n",
    "\n",
    "from keras.preprocessing import sequence\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation, BatchNormalization\n",
    "from keras.layers import Embedding\n",
    "from keras.layers import Conv1D, GlobalMaxPooling1D\n",
    "from keras.datasets import imdb\n",
    "\n",
    "# set parameters:\n",
    "max_features = 1000\n",
    "maxlen = 400\n",
    "batch_size = 32\n",
    "embedding_dims = 50\n",
    "filters = 250\n",
    "kernel_size = 3\n",
    "hidden_dims = 250\n",
    "epochs = 2\n",
    "\n",
    "print('Loading data...')\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(nb_words=max_features)\n",
    "print(len(x_train), 'train sequences')\n",
    "print(len(x_test), 'test sequences')\n",
    "\n",
    "print('Pad sequences (samples x time)')\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
    "print('x_train shape:', x_train.shape)\n",
    "print('x_test shape:', x_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "# we start off with an efficient embedding layer which maps\n",
    "# our vocab indices into embedding_dims dimensions\n",
    "model.add(Embedding(max_features,\n",
    "                    embedding_dims,\n",
    "                    input_length=maxlen))\n",
    "model.add(Dropout(0.2))\n",
    "\n",
    "# we add a Convolution1D, which will learn filters\n",
    "# word group filters of size filter_length:\n",
    "for i in range(1):\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Conv1D(filters,\n",
    "                     kernel_size,\n",
    "                     padding='valid',\n",
    "                     activation='relu'))\n",
    "# we use max pooling:\n",
    "model.add(GlobalMaxPooling1D())\n",
    "\n",
    "# for i in range(1):\n",
    "#     model.add(BatchNormalization())\n",
    "#     model.add(Conv1D(filters*2,\n",
    "#                      kernel_size,\n",
    "#                      padding='valid',\n",
    "#                      activation='relu'))\n",
    "\n",
    "# We add a vanilla hidden layer:\n",
    "model.add(Dense(hidden_dims))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "# We project onto a single unit output layer, and squash it with a sigmoid:\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_4 (Embedding)      (None, 400, 50)           50000     \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 400, 50)           0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 400, 50)           200       \n",
      "_________________________________________________________________\n",
      "conv1d_2 (Conv1D)            (None, 398, 250)          37750     \n",
      "_________________________________________________________________\n",
      "global_max_pooling1d_1 (Glob (None, 250)               0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 250)               62750     \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 250)               0         \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 250)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1)                 251       \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 150,951.0\n",
      "Trainable params: 150,851.0\n",
      "Non-trainable params: 100.0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "RUN_FINAL = False\n",
    "if RUN_FINAL:\n",
    "    model.fit(x_train, y_train,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              validation_data=(x_test, y_test), verbose=0, callbacks=[TQDMNotebookCallback()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}