text_generator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Generation\n",
    "\n",
    "This notebook shows how to implement text generating model in tensorflow. It's based on sarcastic posts on Reddit. These posts cover wide variety of subjects. Yet, they all share the same feature: sarcastic intent. Can you train a model able to generate similar comments?\n",
    "\n",
    "As for the machine learning, the following concepts are covered:\n",
    "\n",
    "- Data prerocessing\n",
    "    - stopword removal\n",
    "    - tokenization\n",
    "    - creating n-grams\n",
    "- Recurrent Neural Networks\n",
    "    - embedding layers\n",
    "    - LSTM layers\n",
    "    - fully-connected layers\n",
    "    - regularization\n",
    "    - dropout\n",
    "- Text generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "The notebook is based on Sarcasm on Reddit dataset. The dataset was generated by scraping comments from Reddit containing the \"\\s\" tag. This tag is often used by Redditors to indicate that their comment is in jest and not meant to be taken seriously, and is generally a reliable indicator of sarcastic comment content.\n",
    "\n",
    "The dataset contains 1.3 million sarcastic statements, which we will use to build a sarcasm generator. \n",
    "\n",
    "To start, [download the data](https://www.kaggle.com/danofer/sarcasm/download) and place it in your working directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "\n",
    "from utils import (create_n_grams, generate_text, load_text_data,\n",
    "                   plot_training_progress, unpack_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure GPU\n",
    "gpu_devices = tf.config.experimental.list_physical_devices('GPU')\n",
    "for device in gpu_devices: \n",
    "    tf.config.experimental.set_memory_growth(device, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_FILE = \"sarcasm.zip\"\n",
    "DATA_DIR = \"sarcasm\"\n",
    "\n",
    "unpack_file(DATA_FILE, DATA_DIR)\n",
    "\n",
    "TRAIN_FILE = DATA_DIR + \"/train-balanced-sarcasm.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data loading\n",
    "\n",
    "corpus, labels = load_text_data(TRAIN_FILE, 1, 0)\n",
    "corpus = [sentence for i, sentence in enumerate(corpus) if int(labels[i]) == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's have a look at sample posts. \n",
    "\n",
    "posts_to_show = 15\n",
    "\n",
    "for i in range(posts_to_show):\n",
    "    idx = random.randrange(0,len(corpus))\n",
    "    print(\"-\" + \" \".join(corpus[idx]))\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data preprocessing\n",
    "\n",
    "MAX_WORDS = 1000     # max words in a dictionnary\n",
    "SEQUENCE_LEN = 50    # maximum sequence length\n",
    "MAX_DOCS = 5000      # number of posts used for training\n",
    "\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(corpus)\n",
    "\n",
    "if MAX_WORDS:\n",
    "    tokenizer.word_index = {e:i for e,i in tokenizer.word_index.items() if i <= MAX_WORDS}\n",
    "\n",
    "total_words = len(tokenizer.word_index) + 1\n",
    "\n",
    "predictors, predictands = create_n_grams(corpus, tokenizer, SEQUENCE_LEN, MAX_DOCS)\n",
    "predictands = tf.keras.utils.to_categorical(predictands, num_classes=total_words, dtype=int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Neural Network\n",
    "\n",
    "Let's build a recurrent neural network.  We will formulate this probelm as multiclass classification. As the input, we will use a sequence of words. As the output, we will use the last word in this sequence. Nural network will then learn words combination that often appear together. As a result, we will be able to \"predict\" the next word, given a sequence of previous words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model architecture\n",
    "\n",
    "model = tf.keras.models.Sequential()\n",
    "model.add(tf.keras.layers.Embedding(total_words, 100, input_length=SEQUENCE_LEN-1))\n",
    "model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(50, return_sequences = True)))\n",
    "model.add(tf.keras.layers.Dropout(0.2))\n",
    "model.add(tf.keras.layers.LSTM(50)),\n",
    "model.add(tf.keras.layers.Dense(total_words/2, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01)))\n",
    "model.add(tf.keras.layers.Dense(total_words, activation='softmax'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now it's time to train our model. \n",
    "\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', \n",
    "              optimizer='adam', \n",
    "              metrics=['acc'])\n",
    "\n",
    "history = model.fit(predictors, \n",
    "                    predictands, \n",
    "                    epochs=100, \n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's evaluate its training progress\n",
    "\n",
    "plot_training_progress(history, \"acc\", plot_validation=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see how it performs with text generation. Start a sequence below, and let the neural network finish it for you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed_text = \"I have never seen such a good movie!\"\n",
    "next_words = 15\n",
    "\n",
    "generate_text(model, tokenizer, seed_text, next_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How sarcastic does this sentence sound? Or does it sound like a sentence at all?\n",
    "\n",
    "Language models are known for taking really long time to train. At the end of the day, you are trying to represent relelationship between all existing words! So take a step backward. Increase the number of epochs, retrain your model, and try again!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "solid_goggles",
   "language": "python",
   "name": "solid_goggles"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}