dl-text_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile('./data/labeledTrainData.tsv.zip', 'r') as z:\n",
    "    z.extractall('./data/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train = pd.read_csv('./data/labeledTrainData.tsv', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5814_8</td>\n",
       "      <td>1</td>\n",
       "      <td>With all this stuff going down at the moment w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2381_9</td>\n",
       "      <td>1</td>\n",
       "      <td>\\The Classic War of the Worlds\\\" by Timothy Hi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7759_3</td>\n",
       "      <td>0</td>\n",
       "      <td>The film starts with a manager (Nicholas Bell)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3630_4</td>\n",
       "      <td>0</td>\n",
       "      <td>It must be assumed that those who praised this...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9495_8</td>\n",
       "      <td>1</td>\n",
       "      <td>Superbly trashy and wondrously unpretentious 8...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id  sentiment                                             review\n",
       "0  5814_8          1  With all this stuff going down at the moment w...\n",
       "1  2381_9          1  \\The Classic War of the Worlds\\\" by Timothy Hi...\n",
       "2  7759_3          0  The film starts with a manager (Nicholas Bell)...\n",
       "3  3630_4          0  It must be assumed that those who praised this...\n",
       "4  9495_8          1  Superbly trashy and wondrously unpretentious 8..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(data_train.shape)\n",
    "data_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.<br /><br />Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.<br /><br />The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.<br /><br />Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.<br /><br />Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train['review'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.review.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "\n",
    "def clean_str(string):\n",
    "    \"\"\"\n",
    "    Tokenization/string cleaning for all datasets \n",
    "    Every dataset is lower cased\n",
    "    \"\"\"\n",
    "    string = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`]\", \" \", string)     \n",
    "    string = re.sub(r\"\\'s\", \" \\'s\", string) \n",
    "    string = re.sub(r\"\\'ve\", \" \\'ve\", string) \n",
    "    string = re.sub(r\"n\\'t\", \" n\\'t\", string) \n",
    "    string = re.sub(r\"\\'re\", \" \\'re\", string) \n",
    "    string = re.sub(r\"\\'d\", \" \\'d\", string) \n",
    "    string = re.sub(r\"\\'ll\", \" \\'ll\", string) \n",
    "    string = re.sub(r\",\", \" , \", string) \n",
    "    string = re.sub(r\"!\", \" ! \", string) \n",
    "    string = re.sub(r\"\\(\", \" \\( \", string) \n",
    "    string = re.sub(r\"\\)\", \" \\) \", string) \n",
    "    string = re.sub(r\"\\?\", \" \\? \", string) \n",
    "    string = re.sub(r\"\\s{2,}\", \" \", string)    \n",
    "    return string.strip().lower()\n",
    "\n",
    "texts = []\n",
    "labels = []\n",
    "for idx in range(data_train.review.shape[0]):\n",
    "    text = BeautifulSoup(data_train.review[idx], \"lxml\")\n",
    "    texts.append(clean_str(text.get_text()))\n",
    "    labels.append(data_train.sentiment[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"with all this stuff going down at the moment with mj i 've started listening to his music , watching the odd documentary here and there , watched the wiz and watched moonwalker again maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent moonwalker is part biography , part feature film which i remember going to see at the cinema when it was originally released some of it has subtle messages about mj 's feeling towards the press and also the obvious message of drugs are bad m'kay visually impressive but of course this is all about michael jackson so unless you remotely like mj in anyway then you are going to hate this and find it boring some may call mj an egotist for consenting to the making of this movie but mj and most of his fans would say that he made it for the fans which if true is really nice of him the actual feature film bit when it finally starts is only on for 20 minutes or so excluding the smooth criminal sequence and joe pesci is convincing as a psychopathic all powerful drug lord why he wants mj dead so bad is beyond me because mj overheard his plans \\\\? nah , joe pesci 's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno , maybe he just hates mj 's music lots of cool things in this like mj turning into a car and a robot and the whole speed demon sequence also , the director must have had the patience of a saint when it came to filming the kiddy bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene bottom line , this movie is for people who like mj on one level or another \\\\( which i think is most people \\\\) if not , then stay away it does try and give off a wholesome message and ironically mj 's bestest buddy in this movie is a girl ! michael jackson is truly one of the most talented people ever to grace this planet but is he guilty \\\\? well , with all the attention i 've gave this subject hmmm well i do n't know because people can be different behind closed doors , i know this for a fact he is either an extremely nice but stupid guy or one of the most sickest liars i hope he is not the latter\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.utils.np_utils import to_categorical\n",
    "\n",
    "from keras.layers import Embedding\n",
    "from keras.layers import Dense, Input, Flatten, concatenate, TimeDistributed\n",
    "from keras.layers import Conv1D, MaxPooling1D, Dropout, Concatenate, SpatialDropout1D, GlobalAveragePooling1D, GlobalMaxPooling1D\n",
    "from keras.layers import Bidirectional, LSTM, GRU, CuDNNLSTM, CuDNNGRU\n",
    "from keras.models import Model\n",
    "\n",
    "from keras import backend as K\n",
    "from keras.engine.topology import Layer, InputSpec\n",
    "from keras import initializers\n",
    "\n",
    "MAX_SEQUENCE_LENGTH = 1000\n",
    "MAX_NB_WORDS = 20000\n",
    "EMBEDDING_DIM = 100\n",
    "VALIDATION_SPLIT = 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\keras_preprocessing\\text.py:175: UserWarning: The `nb_words` argument in `Tokenizer` has been renamed `num_words`.\n",
      "  warnings.warn('The `nb_words` argument in `Tokenizer` '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 81988 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer(nb_words=MAX_NB_WORDS)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of data tensor: (25000, 1000)\n",
      "Shape of label tensor: (25000, 2)\n"
     ]
    }
   ],
   "source": [
    "data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "labels = to_categorical(np.asarray(labels))\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of positive and negative reviews in traing and validation set \n",
      "[10073.  9927.]\n",
      "[2427. 2573.]\n"
     ]
    }
   ],
   "source": [
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0])\n",
    "\n",
    "x_train = data[:-nb_validation_samples]\n",
    "y_train = labels[:-nb_validation_samples]\n",
    "x_val = data[-nb_validation_samples:]\n",
    "y_val = labels[-nb_validation_samples:]\n",
    "\n",
    "print('Number of positive and negative reviews in traing and validation set ')\n",
    "print(y_train.sum(axis=0))\n",
    "print(y_val.sum(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 400000 word vectors in Glove 6B 100d.\n"
     ]
    }
   ],
   "source": [
    "GLOVE_DIR = \"../pre_trained word embeddings/glove.6B\"\n",
    "embeddings_index = {}\n",
    "f = open(os.path.join(GLOVE_DIR, 'glove.6B.100d.txt'), encoding='utf-8')\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "print('Total %s word vectors in Glove 6B 100d.' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))\n",
    "for word, i in word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        # words not found in embedding index will be all-zeros.\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_layer = Embedding(len(word_index) + 1,\n",
    "                            EMBEDDING_DIM,\n",
    "                            weights=[embedding_matrix],\n",
    "                            input_length=MAX_SEQUENCE_LENGTH,\n",
    "                            trainable=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TextCNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model.\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         8198900   \n",
      "_________________________________________________________________\n",
      "conv1d_4 (Conv1D)            (None, 996, 256)          128256    \n",
      "_________________________________________________________________\n",
      "max_pooling1d_3 (MaxPooling1 (None, 1, 256)            0         \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 256)               65792     \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 128)               32896     \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 2)                 258       \n",
      "=================================================================\n",
      "Total params: 8,426,102\n",
      "Trainable params: 8,426,102\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:15: UserWarning: Update your `Conv1D` call to the Keras 2 API: `Conv1D(activation=\"relu\", filters=256, kernel_size=5, strides=1, padding=\"valid\")`\n",
      "  from ipykernel import kernelapp as app\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:17: UserWarning: Update your `MaxPooling1D` call to the Keras 2 API: `MaxPooling1D(pool_size=996)`\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:37: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 23s 1ms/step - loss: 0.3280 - acc: 0.8358 - val_loss: 0.3656 - val_acc: 0.8817\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0751 - acc: 0.9739 - val_loss: 0.4081 - val_acc: 0.8796\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0251 - acc: 0.9913 - val_loss: 0.6895 - val_acc: 0.8639\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0097 - acc: 0.9970 - val_loss: 0.7501 - val_acc: 0.8730\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0130 - acc: 0.9955 - val_loss: 0.7973 - val_acc: 0.8491\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0088 - acc: 0.9970 - val_loss: 0.8852 - val_acc: 0.8798\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0039 - acc: 0.9988 - val_loss: 0.8906 - val_acc: 0.8806\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0029 - acc: 0.9990 - val_loss: 1.0366 - val_acc: 0.8774\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0069 - acc: 0.9981 - val_loss: 0.9393 - val_acc: 0.8746\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 22s 1ms/step - loss: 0.0133 - acc: 0.9952 - val_loss: 0.9065 - val_acc: 0.8717\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x14804ae2fd0>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Convolution1D, Dropout, Activation \n",
    "\n",
    "NB_FILTER = 256\n",
    "FILTER_LENGTH = 5\n",
    "\n",
    "print (\"Training model.\")\n",
    "model = Sequential()\n",
    "model.add(embedding_layer)\n",
    "model.add(Convolution1D(nb_filter=NB_FILTER,\n",
    "                        filter_length=FILTER_LENGTH,\n",
    "                        border_mode='valid',\n",
    "                        activation='relu',\n",
    "                        subsample_length=1))\n",
    "# use max pooling:\n",
    "model.add(MaxPooling1D(pool_length=model.output_shape[1]))\n",
    "# We flatten the output of the conv layer,\n",
    "# so that we can add a vanilla dense layer:\n",
    "model.add(Flatten())\n",
    "# We add a vanilla hidden layer:\n",
    "model.add(Dense(256, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Activation('relu'))\n",
    "# We project onto a single unit output layer, and squash it with a sigmoid:\n",
    "model.add(Dense(2, activation='sigmoid'))\n",
    "\n",
    "model.summary() \n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['acc'])\n",
    "\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),nb_epoch=10, batch_size=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:9: UserWarning: Update your `Conv1D` call to the Keras 2 API: `Conv1D(activation=\"relu\", filters=128, kernel_size=3)`\n",
      "  if __name__ == '__main__':\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:10: UserWarning: Update your `MaxPooling1D` call to the Keras 2 API: `MaxPooling1D(pool_size=998)`\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:9: UserWarning: Update your `Conv1D` call to the Keras 2 API: `Conv1D(activation=\"relu\", filters=128, kernel_size=4)`\n",
      "  if __name__ == '__main__':\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:10: UserWarning: Update your `MaxPooling1D` call to the Keras 2 API: `MaxPooling1D(pool_size=997)`\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:9: UserWarning: Update your `Conv1D` call to the Keras 2 API: `Conv1D(activation=\"relu\", filters=128, kernel_size=5)`\n",
      "  if __name__ == '__main__':\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:10: UserWarning: Update your `MaxPooling1D` call to the Keras 2 API: `MaxPooling1D(pool_size=996)`\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:27: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_8 (InputLayer)            (None, 1000)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1000, 100)    8198900     input_8[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_25 (Conv1D)              (None, 998, 128)     38528       embedding_1[11][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_26 (Conv1D)              (None, 997, 128)     51328       embedding_1[11][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_27 (Conv1D)              (None, 996, 128)     64128       embedding_1[11][0]               \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_22 (MaxPooling1D) (None, 1, 128)       0           conv1d_25[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_23 (MaxPooling1D) (None, 1, 128)       0           conv1d_26[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_24 (MaxPooling1D) (None, 1, 128)       0           conv1d_27[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_5 (Concatenate)     (None, 3, 128)       0           max_pooling1d_22[0][0]           \n",
      "                                                                 max_pooling1d_23[0][0]           \n",
      "                                                                 max_pooling1d_24[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "flatten_7 (Flatten)             (None, 384)          0           concatenate_5[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_18 (Dense)                (None, 256)          98560       flatten_7[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_13 (Dropout)            (None, 256)          0           dense_18[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dense_19 (Dense)                (None, 128)          32896       dropout_13[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dropout_14 (Dropout)            (None, 128)          0           dense_19[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dense_20 (Dense)                (None, 2)            258         dropout_14[0][0]                 \n",
      "==================================================================================================\n",
      "Total params: 8,484,598\n",
      "Trainable params: 8,484,598\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 31s 2ms/step - loss: 0.2699 - acc: 0.8694 - val_loss: 0.3798 - val_acc: 0.8708\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 30s 1ms/step - loss: 0.0303 - acc: 0.9901 - val_loss: 0.6883 - val_acc: 0.8714\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 30s 1ms/step - loss: 0.0087 - acc: 0.9976 - val_loss: 0.8204 - val_acc: 0.8774\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 30s 2ms/step - loss: 0.0062 - acc: 0.9978 - val_loss: 0.8247 - val_acc: 0.8684\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 30s 1ms/step - loss: 0.0033 - acc: 0.9991 - val_loss: 1.0677 - val_acc: 0.8710\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 30s 1ms/step - loss: 0.0025 - acc: 0.9992 - val_loss: 1.0593 - val_acc: 0.8738\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 30s 1ms/step - loss: 0.0087 - acc: 0.9970 - val_loss: 0.9717 - val_acc: 0.8752\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 31s 2ms/step - loss: 0.0039 - acc: 0.9988 - val_loss: 1.1179 - val_acc: 0.8696\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 30s 1ms/step - loss: 0.0045 - acc: 0.9982 - val_loss: 1.1851 - val_acc: 0.8728\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 30s 2ms/step - loss: 0.0040 - acc: 0.9990 - val_loss: 1.1477 - val_acc: 0.8720\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1481f3d2b00>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# applying a more complex convolutional approach\n",
    "convs = []\n",
    "filter_sizes = [3,4,5]\n",
    "\n",
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "\n",
    "for fsz in filter_sizes:\n",
    "    l_conv = Conv1D(nb_filter=128,filter_length=fsz,activation='relu')(embedded_sequences)\n",
    "    l_pool = MaxPooling1D(pool_length=MAX_SEQUENCE_LENGTH-fsz+1)(l_conv)\n",
    "    convs.append(l_pool)\n",
    "l_merge = Concatenate(axis=1)(convs) \n",
    "l_flat = Flatten()(l_merge)\n",
    "l_dense1= Dense(256, activation='relu')(l_flat)\n",
    "l_dropout1 = Dropout(0.5)(l_dense1)\n",
    "l_dense = Dense(128, activation='relu')(l_dropout1)\n",
    "l_dropout2 = Dropout(0.3)(l_dense)\n",
    "preds = Dense(2, activation='softmax')(l_dropout2)\n",
    "model = Model(sequence_input, preds)\n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['acc'])\n",
    "\n",
    "model.summary() \n",
    "\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),nb_epoch=10, batch_size=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - simplified convolutional neural network\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         8198900   \n",
      "_________________________________________________________________\n",
      "conv1d_7 (Conv1D)            (None, 996, 128)          64128     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_5 (MaxPooling1 (None, 199, 128)          0         \n",
      "_________________________________________________________________\n",
      "conv1d_8 (Conv1D)            (None, 195, 128)          82048     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_6 (MaxPooling1 (None, 39, 128)           0         \n",
      "_________________________________________________________________\n",
      "conv1d_9 (Conv1D)            (None, 35, 128)           82048     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_7 (MaxPooling1 (None, 1, 128)            0         \n",
      "_________________________________________________________________\n",
      "flatten_4 (Flatten)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 128)               16512     \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 2)                 258       \n",
      "=================================================================\n",
      "Total params: 8,443,894\n",
      "Trainable params: 8,443,894\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "l_cov1= Conv1D(128, 5, activation='relu')(embedded_sequences)\n",
    "l_pool1 = MaxPooling1D(5)(l_cov1)\n",
    "l_cov2 = Conv1D(128, 5, activation='relu')(l_pool1)\n",
    "l_pool2 = MaxPooling1D(5)(l_cov2)\n",
    "l_cov3 = Conv1D(128, 5, activation='relu')(l_pool2)\n",
    "l_pool3 = MaxPooling1D(35)(l_cov3)  # global max pooling\n",
    "l_flat = Flatten()(l_pool3)\n",
    "l_dropout1 = Dropout(0.5)(l_flat)\n",
    "l_dense = Dense(128, activation='relu')(l_dropout1)\n",
    "l_dropout2 = Dropout(0.3)(l_dense)\n",
    "preds = Dense(2, activation='softmax')(l_dropout2)\n",
    "\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['acc'])\n",
    "print(\"model fitting - simplified convolutional neural network\")\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:2: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "20000/20000 [==============================] - 11s 554us/step - loss: 0.4344 - acc: 0.7590 - val_loss: 0.3536 - val_acc: 0.8756\n",
      "Epoch 2/5\n",
      "20000/20000 [==============================] - 10s 483us/step - loss: 0.1241 - acc: 0.9559 - val_loss: 0.4720 - val_acc: 0.8718\n",
      "Epoch 3/5\n",
      "20000/20000 [==============================] - 10s 482us/step - loss: 0.0504 - acc: 0.9832 - val_loss: 0.6448 - val_acc: 0.8688\n",
      "Epoch 4/5\n",
      "20000/20000 [==============================] - 10s 485us/step - loss: 0.0323 - acc: 0.9896 - val_loss: 0.7590 - val_acc: 0.8674\n",
      "Epoch 5/5\n",
      "20000/20000 [==============================] - 10s 485us/step - loss: 0.0161 - acc: 0.9945 - val_loss: 0.8486 - val_acc: 0.8684\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x148101604e0>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=5, batch_size=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:9: UserWarning: Update your `Conv1D` call to the Keras 2 API: `Conv1D(activation=\"relu\", filters=128, kernel_size=3)`\n",
      "  if __name__ == '__main__':\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:9: UserWarning: Update your `Conv1D` call to the Keras 2 API: `Conv1D(activation=\"relu\", filters=128, kernel_size=4)`\n",
      "  if __name__ == '__main__':\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:9: UserWarning: Update your `Conv1D` call to the Keras 2 API: `Conv1D(activation=\"relu\", filters=128, kernel_size=5)`\n",
      "  if __name__ == '__main__':\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:30: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - more complex convolutional neural network\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_3 (InputLayer)            (None, 1000)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1000, 100)    8198900     input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_10 (Conv1D)              (None, 998, 128)     38528       embedding_1[6][0]                \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_11 (Conv1D)              (None, 997, 128)     51328       embedding_1[6][0]                \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_12 (Conv1D)              (None, 996, 128)     64128       embedding_1[6][0]                \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_8 (MaxPooling1D)  (None, 199, 128)     0           conv1d_10[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_9 (MaxPooling1D)  (None, 199, 128)     0           conv1d_11[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_10 (MaxPooling1D) (None, 199, 128)     0           conv1d_12[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 597, 128)     0           max_pooling1d_8[0][0]            \n",
      "                                                                 max_pooling1d_9[0][0]            \n",
      "                                                                 max_pooling1d_10[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_13 (Conv1D)              (None, 593, 128)     82048       concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_11 (MaxPooling1D) (None, 118, 128)     0           conv1d_13[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_14 (Conv1D)              (None, 114, 128)     82048       max_pooling1d_11[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_12 (MaxPooling1D) (None, 3, 128)       0           conv1d_14[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "flatten_5 (Flatten)             (None, 384)          0           max_pooling1d_12[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "dense_10 (Dense)                (None, 128)          49280       flatten_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_11 (Dense)                (None, 2)            258         dense_10[0][0]                   \n",
      "==================================================================================================\n",
      "Total params: 8,566,518\n",
      "Trainable params: 8,566,518\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 25s 1ms/step - loss: 0.2533 - acc: 0.8770 - val_loss: 0.4783 - val_acc: 0.8748\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0549 - acc: 0.9816 - val_loss: 0.6458 - val_acc: 0.8528\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0306 - acc: 0.9916 - val_loss: 0.7718 - val_acc: 0.8688\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0194 - acc: 0.9953 - val_loss: 0.8442 - val_acc: 0.8714\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0167 - acc: 0.9967 - val_loss: 1.3710 - val_acc: 0.8724\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0152 - acc: 0.9976 - val_loss: 1.1906 - val_acc: 0.8640\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0108 - acc: 0.9982 - val_loss: 1.4663 - val_acc: 0.8666\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0095 - acc: 0.9988 - val_loss: 1.6611 - val_acc: 0.8730\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0130 - acc: 0.9980 - val_loss: 1.4141 - val_acc: 0.8688\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 24s 1ms/step - loss: 0.0129 - acc: 0.9981 - val_loss: 1.3900 - val_acc: 0.8678\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x148107d7c18>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# applying a more complex convolutional approach\n",
    "convs = []\n",
    "filter_sizes = [3,4,5]\n",
    "\n",
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "\n",
    "for fsz in filter_sizes:\n",
    "    l_conv = Conv1D(nb_filter=128,filter_length=fsz,activation='relu')(embedded_sequences)\n",
    "    l_pool = MaxPooling1D(5)(l_conv)\n",
    "    convs.append(l_pool)\n",
    "l_merge = Concatenate(axis=1)(convs)    \n",
    "# l_merge = Merge(mode='concat', concat_axis=1)(convs)\n",
    "l_cov1= Conv1D(128, 5, activation='relu')(l_merge)\n",
    "l_pool1 = MaxPooling1D(5)(l_cov1)\n",
    "l_cov2 = Conv1D(128, 5, activation='relu')(l_pool1)\n",
    "l_pool2 = MaxPooling1D(30)(l_cov2)\n",
    "l_flat = Flatten()(l_pool2)\n",
    "l_dense = Dense(128, activation='relu')(l_flat)\n",
    "preds = Dense(2, activation='softmax')(l_dense)\n",
    "\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(\"model fitting - more complex convolutional neural network\")\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bi-LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - Bidirectional LSTM\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_9 (InputLayer)         (None, 1000)              0         \n",
      "_________________________________________________________________\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         8198900   \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 200)               161600    \n",
      "_________________________________________________________________\n",
      "dense_21 (Dense)             (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,360,902\n",
      "Trainable params: 8,360,902\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:13: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  del sys.path[0]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 50s 2ms/step - loss: 0.3962 - acc: 0.8282 - val_loss: 0.3171 - val_acc: 0.8844\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 49s 2ms/step - loss: 0.1533 - acc: 0.9480 - val_loss: 0.3891 - val_acc: 0.8710\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 49s 2ms/step - loss: 0.0722 - acc: 0.9763 - val_loss: 0.3632 - val_acc: 0.8862\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 49s 2ms/step - loss: 0.0394 - acc: 0.9868 - val_loss: 0.5082 - val_acc: 0.8852\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 48s 2ms/step - loss: 0.0278 - acc: 0.9913 - val_loss: 0.5018 - val_acc: 0.8818\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 49s 2ms/step - loss: 0.0162 - acc: 0.9950 - val_loss: 0.5982 - val_acc: 0.8870\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 48s 2ms/step - loss: 0.0110 - acc: 0.9965 - val_loss: 0.6488 - val_acc: 0.8792\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 49s 2ms/step - loss: 0.0068 - acc: 0.9982 - val_loss: 0.8897 - val_acc: 0.8714\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 48s 2ms/step - loss: 0.0047 - acc: 0.9985 - val_loss: 0.9378 - val_acc: 0.8786\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 48s 2ms/step - loss: 0.0033 - acc: 0.9992 - val_loss: 1.0610 - val_acc: 0.8760\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x148238d19e8>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "l_lstm = Bidirectional(CuDNNLSTM(100))(embedded_sequences)\n",
    "preds = Dense(2, activation='softmax')(l_lstm)\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(\"model fitting - Bidirectional LSTM\")\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RCNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model fitting - RCNN\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_4 (InputLayer)            (None, 1000)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1000, 100)    8198900     input_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "spatial_dropout1d_3 (SpatialDro (None, 1000, 100)    0           embedding_1[3][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_3 (Bidirectional) (None, 1000, 200)    121200      spatial_dropout1d_3[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, 999, 64)      25664       bidirectional_3[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_2 (Glo (None, 64)           0           conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 64)           0           conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 128)          0           global_average_pooling1d_2[0][0] \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 2)            258         concatenate_1[0][0]              \n",
      "==================================================================================================\n",
      "Total params: 8,346,022\n",
      "Trainable params: 8,346,022\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 57s 3ms/step - loss: 0.4715 - acc: 0.7631 - val_loss: 0.3183 - val_acc: 0.8660\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 54s 3ms/step - loss: 0.3096 - acc: 0.8705 - val_loss: 0.2766 - val_acc: 0.8812\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 53s 3ms/step - loss: 0.2381 - acc: 0.9021 - val_loss: 0.2530 - val_acc: 0.8938\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 53s 3ms/step - loss: 0.1834 - acc: 0.9275 - val_loss: 0.2509 - val_acc: 0.9004\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.1455 - acc: 0.9439 - val_loss: 0.3003 - val_acc: 0.8926\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 56s 3ms/step - loss: 0.1039 - acc: 0.9620 - val_loss: 0.2585 - val_acc: 0.9106\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0770 - acc: 0.9723 - val_loss: 0.3142 - val_acc: 0.9076\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 56s 3ms/step - loss: 0.0570 - acc: 0.9799 - val_loss: 0.3096 - val_acc: 0.9092\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 56s 3ms/step - loss: 0.0381 - acc: 0.9864 - val_loss: 0.3588 - val_acc: 0.9072\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 55s 3ms/step - loss: 0.0320 - acc: 0.9879 - val_loss: 0.4044 - val_acc: 0.9082\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x16de8a595f8>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "l_dropout = SpatialDropout1D(0.3)(embedded_sequences)\n",
    "l_gru = Bidirectional(CuDNNGRU(100, return_sequences=True))(l_dropout)\n",
    "l_cov = Conv1D(64, kernel_size = 2, padding = \"valid\", kernel_initializer = \"he_uniform\")(l_gru)\n",
    "avg_pool = GlobalAveragePooling1D()(l_cov)\n",
    "max_pool = GlobalMaxPooling1D()(l_cov)\n",
    "l_conc = concatenate([avg_pool, max_pool])\n",
    "preds = Dense(2, activation='softmax')(l_conc)\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(\"model fitting - RCNN\")\n",
    "model.summary()\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HRNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 81988 unique tokens.\n",
      "Shape of data tensor: (25000, 15, 100)\n",
      "Shape of label tensor: (25000, 2)\n",
      "Number of positive and negative reviews in traing and validation set\n",
      "[ 9998. 10002.]\n",
      "[2502. 2498.]\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer, text_to_word_sequence\n",
    "from nltk import tokenize\n",
    "\n",
    "MAX_SENT_LENGTH = 100\n",
    "MAX_SENTS = 15\n",
    "\n",
    "reviews = []\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for idx in range(data_train.review.shape[0]):\n",
    "    text = BeautifulSoup(data_train.review[idx], \"lxml\")\n",
    "    text = clean_str(text.get_text())\n",
    "    texts.append(text)\n",
    "    sentences = tokenize.sent_tokenize(text)\n",
    "    reviews.append(sentences)\n",
    "    labels.append(data_train.sentiment[idx])\n",
    "\n",
    "data = np.zeros((len(texts), MAX_SENTS, MAX_SENT_LENGTH), dtype='int32')\n",
    "\n",
    "for i, sentences in enumerate(reviews):\n",
    "    for j, sent in enumerate(sentences):\n",
    "        if j < MAX_SENTS:\n",
    "            wordTokens = text_to_word_sequence(sent)\n",
    "            k = 0\n",
    "            for _, word in enumerate(wordTokens):\n",
    "                if k < MAX_SENT_LENGTH and tokenizer.word_index[word] < MAX_NB_WORDS:\n",
    "                    data[i, j, k] = tokenizer.word_index[word]\n",
    "                    k = k + 1\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Total %s unique tokens.' % len(word_index))\n",
    "\n",
    "labels = to_categorical(np.asarray(labels))\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)\n",
    "\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0])\n",
    "\n",
    "x_train = data[:-nb_validation_samples]\n",
    "y_train = labels[:-nb_validation_samples]\n",
    "x_val = data[-nb_validation_samples:]\n",
    "y_val = labels[-nb_validation_samples:]\n",
    "\n",
    "print('Number of positive and negative reviews in traing and validation set')\n",
    "print(y_train.sum(axis=0))\n",
    "print(y_val.sum(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_layer = Embedding(len(word_index) + 1,\n",
    "                            EMBEDDING_DIM,\n",
    "                            weights=[embedding_matrix],\n",
    "                            input_length=MAX_SENT_LENGTH,\n",
    "                            trainable=True,\n",
    "                            mask_zero=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_5 (InputLayer)         (None, 15, 100)           0         \n",
      "_________________________________________________________________\n",
      "time_distributed_1 (TimeDist (None, 15, 200)           8319500   \n",
      "_________________________________________________________________\n",
      "bidirectional_4 (Bidirection (None, 200)               180600    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 402       \n",
      "=================================================================\n",
      "Total params: 8,500,502\n",
      "Trainable params: 8,500,502\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "model fitting - Hierachical attention network\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:18: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 125s 6ms/step - loss: 0.5248 - acc: 0.7323 - val_loss: 0.4053 - val_acc: 0.8270\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 121s 6ms/step - loss: 0.3296 - acc: 0.8594 - val_loss: 0.4344 - val_acc: 0.8120\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 122s 6ms/step - loss: 0.2492 - acc: 0.8996 - val_loss: 0.3192 - val_acc: 0.8744\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 122s 6ms/step - loss: 0.1835 - acc: 0.9298 - val_loss: 0.3468 - val_acc: 0.8696\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 122s 6ms/step - loss: 0.1294 - acc: 0.9527 - val_loss: 0.3730 - val_acc: 0.8686\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 124s 6ms/step - loss: 0.0801 - acc: 0.9729 - val_loss: 0.4922 - val_acc: 0.8670\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 124s 6ms/step - loss: 0.0423 - acc: 0.9864 - val_loss: 0.5912 - val_acc: 0.8586\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 123s 6ms/step - loss: 0.0197 - acc: 0.9942 - val_loss: 0.7258 - val_acc: 0.8602\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 125s 6ms/step - loss: 0.0096 - acc: 0.9971 - val_loss: 1.2480 - val_acc: 0.8450\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 123s 6ms/step - loss: 0.0065 - acc: 0.9985 - val_loss: 1.0831 - val_acc: 0.8530\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x16e6d6dc518>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_input = Input(shape=(MAX_SENT_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sentence_input)\n",
    "l_lstm = Bidirectional(GRU(100))(embedded_sequences)\n",
    "sentEncoder = Model(sentence_input, l_lstm)\n",
    "\n",
    "review_input = Input(shape=(MAX_SENTS,MAX_SENT_LENGTH), dtype='int32')\n",
    "review_encoder = TimeDistributed(sentEncoder)(review_input)\n",
    "l_lstm_sent = Bidirectional(GRU(100))(review_encoder)\n",
    "preds = Dense(2, activation='softmax')(l_lstm_sent)\n",
    "model = Model(review_input, preds)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "model.summary()\n",
    "print(\"model fitting - Hierachical attention network\")\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=10, batch_size=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### char-models\n",
    "\n",
    "> https://offbit.github.io/how-to-read/\n",
    "\n",
    "> https://github.com/offbit/char-models\n",
    "\n",
    "> https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total chars: 71\n",
      "Sample doc['\"the premise is amazing and the some of the acting, notably sally kellerman and anthony rapp, is charming...', 'but this film is near unwatchable.', 'the music sounds as if it comes from some sort of the royalty free online site and the lyrics as if they were written with a rhyming dictionary open on the lap.', 'most of the singing is off-key.', 'i think they may have filmed with the singing accapella and put in the music under it...', 'the dialogue is really stupid and trite.', 'the movie works best when it is actually talking about the real estate but unfortunately it strays to often into stupid farcical sub-plots.', 'i found myself checking my watch after ther first twenty minutes and after 40 wondering \\'when is it ever going to end.\\'\"']\n",
      "Sample chars in X:[-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1\n",
      " -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 61 19 69  9 42 50 37 34\n",
      " 42  6 47 42  6 50 19 47 42 45 51 69  6 47 13 34 52 13 53 42 15  6 13 14\n",
      " 45 37 51 42 69 42 37 34 13 58 42  6 50 34 34 13 51 58 42 50 51 50 58 42\n",
      " 45 50 37 34 42 31 13 42 49 69 42 49 52 13 51 45  9 42 50 37 34 42 53  6\n",
      " 69 42 50 34 13 49 42 50  6 13  9  6 47 42 50 50 51 31 42 45 34  9 69 45\n",
      " 47 51 42 50 37 34 42 31 47 42 34 51 47 49 42 50 14 47 49 42 14 47 51 31\n",
      " 42 49 50 14 47 52 42 34 13 42 31 13 42 49 69 42 49 53  6 54 47 49 42 52\n",
      " 13 49 54 14 42 50 37 34]\n",
      "y:0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\keras\\layers\\recurrent.py:2068: UserWarning: `implementation=0` has been deprecated, and now defaults to `implementation=1`.Please update your layer call.\n",
      "  warnings.warn('`implementation=0` has been deprecated, '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "lambda_1 (Lambda)            (None, 512, 71)           0         \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 508, 196)          69776     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 508, 196)          0         \n",
      "_________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1 (None, 254, 196)          0         \n",
      "_________________________________________________________________\n",
      "conv1d_2 (Conv1D)            (None, 252, 196)          115444    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 252, 196)          0         \n",
      "_________________________________________________________________\n",
      "max_pooling1d_2 (MaxPooling1 (None, 126, 196)          0         \n",
      "_________________________________________________________________\n",
      "conv1d_3 (Conv1D)            (None, 124, 256)          150784    \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 124, 256)          0         \n",
      "_________________________________________________________________\n",
      "max_pooling1d_3 (MaxPooling1 (None, 62, 256)           0         \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 256)               394240    \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 256)               0         \n",
      "=================================================================\n",
      "Total params: 730,244\n",
      "Trainable params: 730,244\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 15, 512)           0         \n",
      "_________________________________________________________________\n",
      "time_distributed_1 (TimeDist (None, 15, 256)           730244    \n",
      "_________________________________________________________________\n",
      "bidirectional_2 (Bidirection (None, 256)               394240    \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 128)               32896     \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 129       \n",
      "=================================================================\n",
      "Total params: 1,157,509\n",
      "Trainable params: 1,157,509\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/15\n",
      "20000/20000 [==============================] - 459s 23ms/step - loss: 0.6942 - acc: 0.5020 - val_loss: 0.6932 - val_acc: 0.5078\n",
      "Epoch 2/15\n",
      "20000/20000 [==============================] - 449s 22ms/step - loss: 0.6912 - acc: 0.5136 - val_loss: 0.6685 - val_acc: 0.6014\n",
      "Epoch 3/15\n",
      "20000/20000 [==============================] - 452s 23ms/step - loss: 0.6221 - acc: 0.6649 - val_loss: 0.5909 - val_acc: 0.6904\n",
      "Epoch 4/15\n",
      "20000/20000 [==============================] - 448s 22ms/step - loss: 0.5458 - acc: 0.7250 - val_loss: 0.4628 - val_acc: 0.7730\n",
      "Epoch 5/15\n",
      "20000/20000 [==============================] - 446s 22ms/step - loss: 0.4156 - acc: 0.8124 - val_loss: 0.4049 - val_acc: 0.8186\n",
      "Epoch 6/15\n",
      "20000/20000 [==============================] - 448s 22ms/step - loss: 0.3698 - acc: 0.8387 - val_loss: 0.3797 - val_acc: 0.8342\n",
      "Epoch 7/15\n",
      "20000/20000 [==============================] - 461s 23ms/step - loss: 0.3331 - acc: 0.8566 - val_loss: 0.3581 - val_acc: 0.8454\n",
      "Epoch 8/15\n",
      "20000/20000 [==============================] - 450s 23ms/step - loss: 0.3023 - acc: 0.8721 - val_loss: 0.3706 - val_acc: 0.8488\n",
      "Epoch 9/15\n",
      "20000/20000 [==============================] - 449s 22ms/step - loss: 0.2776 - acc: 0.8845 - val_loss: 0.3635 - val_acc: 0.8514\n",
      "Epoch 10/15\n",
      "20000/20000 [==============================] - 447s 22ms/step - loss: 0.2512 - acc: 0.8989 - val_loss: 0.3506 - val_acc: 0.8480\n",
      "Epoch 11/15\n",
      "20000/20000 [==============================] - 468s 23ms/step - loss: 0.2292 - acc: 0.9104 - val_loss: 0.3838 - val_acc: 0.8458\n",
      "Epoch 12/15\n",
      "20000/20000 [==============================] - 452s 23ms/step - loss: 0.2161 - acc: 0.9150 - val_loss: 0.3706 - val_acc: 0.8534\n",
      "Epoch 13/15\n",
      "20000/20000 [==============================] - 448s 22ms/step - loss: 0.1933 - acc: 0.9247 - val_loss: 0.4107 - val_acc: 0.8574\n",
      "Epoch 14/15\n",
      "20000/20000 [==============================] - 454s 23ms/step - loss: 0.1791 - acc: 0.9318 - val_loss: 0.4217 - val_acc: 0.8502\n",
      "Epoch 15/15\n",
      "20000/20000 [==============================] - 456s 23ms/step - loss: 0.1660 - acc: 0.9374 - val_loss: 0.4397 - val_acc: 0.8550\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x23876118a20>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from keras.models import Model\n",
    "from keras.layers import Dense, Input, Dropout, MaxPooling1D, Conv1D\n",
    "from keras.layers import LSTM, Lambda\n",
    "from keras.layers import TimeDistributed, Bidirectional\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import re\n",
    "import keras.callbacks\n",
    "import sys\n",
    "import os\n",
    "\n",
    "\n",
    "def binarize(x, sz=71):\n",
    "    return tf.to_float(tf.one_hot(x, sz, on_value=1, off_value=0, axis=-1))\n",
    "\n",
    "\n",
    "def binarize_outshape(in_shape):\n",
    "    return in_shape[0], in_shape[1], 71\n",
    "\n",
    "\n",
    "def striphtml(html):\n",
    "    p = re.compile(r'<.*?>')\n",
    "    return p.sub('', html)\n",
    "\n",
    "\n",
    "def clean(s):\n",
    "    return re.sub(r'[^\\x00-\\x7f]', r'', s)\n",
    "\n",
    "\n",
    "data = pd.read_csv(\"./data/labeledTrainData.tsv\", header=0, delimiter=\"\\t\", quoting=3)\n",
    "txt = ''\n",
    "docs = []\n",
    "sentences = []\n",
    "sentiments = []\n",
    "\n",
    "for cont, sentiment in zip(data.review, data.sentiment):\n",
    "    sentences = re.split(r'(?<!\\w\\.\\w.)(?<![A-Z][a-z]\\.)(?<=\\.|\\?)\\s', clean(striphtml(cont)))\n",
    "    sentences = [sent.lower() for sent in sentences]\n",
    "    docs.append(sentences)\n",
    "    sentiments.append(sentiment)\n",
    "\n",
    "num_sent = []\n",
    "for doc in docs:\n",
    "    num_sent.append(len(doc))\n",
    "    for s in doc:\n",
    "        txt += s\n",
    "\n",
    "chars = set(txt)\n",
    "\n",
    "print('total chars:', len(chars))\n",
    "char_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "indices_char = dict((i, c) for i, c in enumerate(chars))\n",
    "\n",
    "print('Sample doc{}'.format(docs[1200]))\n",
    "\n",
    "maxlen = 512\n",
    "max_sentences = 15\n",
    "\n",
    "X = np.ones((len(docs), max_sentences, maxlen), dtype=np.int64) * -1\n",
    "y = np.array(sentiments)\n",
    "\n",
    "for i, doc in enumerate(docs):\n",
    "    for j, sentence in enumerate(doc):\n",
    "        if j < max_sentences:\n",
    "            for t, char in enumerate(sentence[-maxlen:]):\n",
    "                X[i, j, (maxlen - 1 - t)] = char_indices[char]\n",
    "\n",
    "print('Sample chars in X:{}'.format(X[1200, 2]))\n",
    "print('y:{}'.format(y[1200]))\n",
    "\n",
    "ids = np.arange(len(X))\n",
    "np.random.shuffle(ids)\n",
    "\n",
    "# shuffle\n",
    "X = X[ids]\n",
    "y = y[ids]\n",
    "\n",
    "X_train = X[:20000]\n",
    "X_test = X[20000:]\n",
    "\n",
    "y_train = y[:20000]\n",
    "y_test = y[20000:]\n",
    "\n",
    "filter_length = [5, 3, 3]\n",
    "nb_filter = [196, 196, 256]\n",
    "pool_length = 2\n",
    "# document input\n",
    "document = Input(shape=(max_sentences, maxlen), dtype='int64')\n",
    "# sentence input\n",
    "in_sentence = Input(shape=(maxlen,), dtype='int64')\n",
    "# char indices to one hot matrix, 1D sequence to 2D \n",
    "embedded = Lambda(binarize, output_shape=binarize_outshape)(in_sentence)\n",
    "# embedded: encodes sentence\n",
    "for i in range(len(nb_filter)):\n",
    "    embedded = Conv1D(filters=nb_filter[i],\n",
    "                      kernel_size=filter_length[i],\n",
    "                      padding='valid',\n",
    "                      activation='relu',\n",
    "                      kernel_initializer='glorot_normal',\n",
    "                      strides=1)(embedded)\n",
    "\n",
    "    embedded = Dropout(0.1)(embedded)\n",
    "    embedded = MaxPooling1D(pool_size=pool_length)(embedded)\n",
    "\n",
    "bi_lstm_sent = \\\n",
    "    Bidirectional(LSTM(128, return_sequences=False, dropout=0.15, recurrent_dropout=0.15, implementation=0))(embedded)\n",
    "\n",
    "# sent_encode = merge([forward_sent, backward_sent], mode='concat', concat_axis=-1)\n",
    "sent_encode = Dropout(0.3)(bi_lstm_sent)\n",
    "# sentence encoder\n",
    "encoder = Model(inputs=in_sentence, outputs=sent_encode)\n",
    "encoder.summary()\n",
    "\n",
    "encoded = TimeDistributed(encoder)(document)\n",
    "# encoded: sentences to bi-lstm for document encoding \n",
    "b_lstm_doc = \\\n",
    "    Bidirectional(LSTM(128, return_sequences=False, dropout=0.15, recurrent_dropout=0.15, implementation=0))(encoded)\n",
    "\n",
    "output = Dropout(0.3)(b_lstm_doc)\n",
    "output = Dense(128, activation='relu')(output)\n",
    "output = Dropout(0.3)(output)\n",
    "output = Dense(1, activation='sigmoid')(output)\n",
    "\n",
    "model = Model(inputs=document, outputs=output)\n",
    "\n",
    "model.summary()\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, y_train, validation_data=(X_test, y_test), batch_size=16,\n",
    "          epochs=15, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}