sentiment_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Classification\n",
    "\n",
    "This notebook shows how to implement sentiment classification in tensorflow. It's based on tweets on various subjects. Each of these tweets have some embedded emotions. Be it positive or negative, sentiments behing such tweets may significantly impact a company, or a person's, brand.\n",
    "\n",
    "As for the machine learning, the following concepts are covered:\n",
    "\n",
    "- Data prerocessing\n",
    "    - stopword removal\n",
    "    - tokenization\n",
    "    - padding\n",
    "    - labels encoding\n",
    "- Neural Networks\n",
    "    - embedding layers\n",
    "    - pooling layers\n",
    "    - fully-connected layers\n",
    "- Callbacks\n",
    "- Transfer learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "The notebook is based on Emotions in Text dataset from Kaggle. It contains 31k tweets representing various sentiments. After building (and training) the neural network, you will be able to classify sentiment in any short text as belonging to one of those categories:\n",
    "- positive\n",
    "- negative\n",
    "- neutral\n",
    "\n",
    "To start, [download the data](https://www.kaggle.com/c/16295/download-all) and place it in your working directory. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "\n",
    "from utils import (AccuracyCallback, classify_sentence, create_embedding_layer,\n",
    "                   load_glove_embeddings, load_text_data,\n",
    "                   plot_training_progress, unpack_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure GPU\n",
    "gpu_devices = tf.config.experimental.list_physical_devices('GPU')\n",
    "for device in gpu_devices: \n",
    "    tf.config.experimental.set_memory_growth(device, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_FILE = \"tweet-sentiment-extraction.zip\"\n",
    "DATA_DIR = \"data/tweets\"\n",
    "\n",
    "unpack_file(DATA_FILE, DATA_DIR)\n",
    "\n",
    "TRAIN_FILE = DATA_DIR + \"/train.csv\"\n",
    "TEST_FILE = DATA_DIR + \"/test.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data preprocessing\n",
    "SENTIMENT_TO_LABEL = {\n",
    "        \"positive\": 2,\n",
    "        \"neutral\": 1,\n",
    "        \"negative\": 0,\n",
    "    }\n",
    "LABEL_TO_SENTIMENT = {label: sentiment for sentiment, label in SENTIMENT_TO_LABEL.items()}\n",
    "\n",
    "train_texts, train_sentiments = load_text_data(TRAIN_FILE, 2, 3, remove_stopwords=True)\n",
    "train_labels = [SENTIMENT_TO_LABEL[sentiment] for sentiment in train_sentiments]\n",
    "train_labels = tf.keras.utils.to_categorical(train_labels, dtype=int)    \n",
    "\n",
    "test_texts, test_sentiments = load_text_data(TEST_FILE, 1, 2)\n",
    "test_labels = [SENTIMENT_TO_LABEL[sentiment] for sentiment in test_sentiments]\n",
    "test_labels = tf.keras.utils.to_categorical(test_labels, dtype=int)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's have a look at sample tweets. \n",
    "\n",
    "tweets_to_show = 15\n",
    "\n",
    "for i in range(tweets_to_show):\n",
    "    idx = random.randrange(0,len(train_texts))\n",
    "    print(\"Tweet: {}\".format(\" \".join(train_texts[idx])))\n",
    "    print(\"Label: {}\".format(train_sentiments[idx]))\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenization and padding\n",
    "EMBEDDING_DIM = 16    \n",
    "MAX_WORDS = 1e5        # max words in a dictionnary\n",
    "MAX_SEQUENCE_LEN = 50  # max tweet length\n",
    "\n",
    "tokenizer = Tokenizer(num_words=MAX_WORDS, oov_token=\"<oov>\")\n",
    "tokenizer.fit_on_texts(train_texts)\n",
    "\n",
    "sequences_train = tokenizer.texts_to_sequences(train_texts)\n",
    "padded_train = pad_sequences(sequences_train, padding = \"post\", maxlen = MAX_SEQUENCE_LEN)\n",
    "\n",
    "sequences_test = tokenizer.texts_to_sequences(test_texts)\n",
    "padded_test = pad_sequences(sequences_test, padding = \"post\", maxlen = MAX_SEQUENCE_LEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network\n",
    "\n",
    "Let's build a simple neural network for multiclass classification. We will include trainable embedding layer.\n",
    "It will enable the NN to learn multidimensional relationships between different words in our text corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model architecture\n",
    "\n",
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Embedding(len(tokenizer.word_index)+1, EMBEDDING_DIM, input_length = MAX_SEQUENCE_LEN),\n",
    "    tf.keras.layers.GlobalAveragePooling1D(),\n",
    "    tf.keras.layers.Dense(24, activation = \"relu\"),\n",
    "    tf.keras.layers.Dense(3, activation = \"softmax\")\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now it's time to train our model. \n",
    "\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "accuracy_callback = AccuracyCallback(0.9)\n",
    "\n",
    "model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.001), \n",
    "              loss=\"categorical_crossentropy\", \n",
    "              metrics=[\"acc\"])\n",
    "\n",
    "history = model.fit(padded_train, \n",
    "                    train_labels, \n",
    "                    epochs=200, \n",
    "                    verbose=1, \n",
    "                    callbacks=[accuracy_callback], \n",
    "                    validation_data=(padded_test, test_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's evaluate its performance on test data\n",
    "test_metrics = model.evaluate(padded_test, test_labels)\n",
    "\n",
    "for i in range(len(test_metrics)):\n",
    "    print(\"Test {}: {}\".format(model.metrics_names[i], test_metrics[i]))\n",
    "\n",
    "plot_training_progress(history, \"acc\")\n",
    "plot_training_progress(history, \"loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer Learning\n",
    "\n",
    "Can we do any better than that? \n",
    "\n",
    "Instead of training our model for long hours, let us build upon well-trained models. More specifically, let's include pre-trained word embeddings.We will work with Global Vectors for Word Representation. These embeddings were trained by Stanford researchers on on massive amount of english texts. \n",
    "\n",
    "To start, [download the GloVe embeddings](http://nlp.stanford.edu/data/glove.6B.zip) and place them in your working directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "GLOVE_FILE = \"glove.6B.zip\"\n",
    "word2vec = load_glove_embeddings(GLOVE_FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's design a model with pretrained Embeddings layer\n",
    "\n",
    "# Embedding layer\n",
    "sentence_indices = tf.keras.layers.Input(shape=(MAX_SEQUENCE_LEN,), dtype=\"int32\")\n",
    "embedding_layer = create_embedding_layer(tokenizer.word_index, word2vec, MAX_WORDS)\n",
    "embeddings = embedding_layer(sentence_indices)\n",
    "\n",
    "# Dense layers\n",
    "x = tf.keras.layers.GlobalAveragePooling1D()(embeddings)\n",
    "x = tf.keras.layers.Dense(24, activation='relu')(x)\n",
    "x = tf.keras.layers.Dense(3, activation='softmax')(x)\n",
    "\n",
    "model = tf.keras.models.Model(inputs=sentence_indices, outputs=x)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Time to train model based on pretrained word embeddings. \n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.001),\n",
    "              loss=\"categorical_crossentropy\", \n",
    "              metrics=[\"acc\"])\n",
    "\n",
    "history = model.fit(padded_train, \n",
    "                    train_labels,\n",
    "                    validation_data=(padded_test, test_labels),\n",
    "                    epochs=200, \n",
    "                    verbose=1, \n",
    "                    callbacks=[accuracy_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's evaluate its performance on test data. \n",
    "# Does it do any better than when we built the embeddings from scratch?\n",
    "\n",
    "test_metrics = model.evaluate(padded_test, test_labels)\n",
    "\n",
    "for i in range(len(test_metrics)):\n",
    "    print(\"Test {}: {}\".format(model.metrics_names[i], test_metrics[i]))\n",
    "\n",
    "plot_training_progress(history, \"acc\")\n",
    "plot_training_progress(history, \"loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have evaluated models by accuracy and log loss. Now, let's see how our model performs on random texts.\n",
    "\n",
    "Can it correctly classify your tweet?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Out of sample prediction\n",
    "\n",
    "sentence = \"I hate you!!!\"\n",
    "label = classify_sentence(model, tokenizer, sentence, MAX_SEQUENCE_LEN)\n",
    "\n",
    "print(\"Sentiment: {}\". format(LABEL_TO_SENTIMENT[label]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "solid_goggles",
   "language": "python",
   "name": "solid_goggles"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}