exercise_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DATA20001 Deep Learning 2018 - Exercise 4\n",
    "\n",
    "**Due Sunday December 02, before 23:59**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4.1 (3 points)\n",
    "\n",
    "### Summary\n",
    "\n",
    "In this exercise we will implement a Convolutional Neural Network to perform sentiment analysis on the classic IMDB movie review dataset.\n",
    "\n",
    "This is binary classification task, meaning we will be predicting either a positive (1) or negative (0) label. Your output should be a real value between 0 and 1. Use binary cross-entropy as your loss function.\n",
    "\n",
    "Use 1D convolution and pooling layers. Use torch.nn.Embedding as the first layer to get the word embeddings for the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using GPU.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torchtext import datasets, vocab\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    print('Using GPU.')\n",
    "    device = torch.device('cuda')\n",
    "else:\n",
    "    print('Using CPU.')\n",
    "    device = torch.device('cpu')\n",
    "\n",
    "glove = vocab.GloVe(name='6B', dim=50)\n",
    "train_loader, test_loader = datasets.IMDB.iters(batch_size=4, device=device, vectors=glove)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "The input data is divided into two sets: the test set and the training set. They are both identical in structure. They have around 25000 instances each, and two attributes: `text` and `label`. We access them through two iterators `train_loader` and `test_loader` that perform batching and bucketing. \n",
    "\n",
    "Batching groups together several instances into the same tensor, and bucketing groups only instances with similar lengths, in order to minimize padding (all the instances within a certain range of length fall into the same \"bucket\"). For instance, if we have 4 instances, with lengths `[1, 2, 3, 4]` and we want to make batches of size 2, the naïve way of batching them is padding them to the length of the longest sequence (or some hard limit):\n",
    "\n",
    "<code>\n",
    "[[1, 0, 0, 0],\n",
    " [1, 2, 0, 0]]\n",
    "[[1, 2, 3, 0],\n",
    " [1, 2, 3, 4]]\n",
    "</code>\n",
    "\n",
    "With bucketing, we can separate them in two buckets, one of length 2 and one of length 4, and have batches of different length, but with much less padding:\n",
    "\n",
    "<code>\n",
    "[[1, 0],\n",
    " [1, 2]]\n",
    "[[1, 2, 3, 0],\n",
    " [1, 2, 3, 4]]\n",
    "</code>\n",
    "\n",
    "This is why all the input batches are of different length.\n",
    "\n",
    "For the test set, the labels are 1 or 2, so you have to subtract 1 to be able to use cross-entropy. The CNN layers expect the batch to be the first dimension, so you have to transpose the first two dimensions in the iterator.\n",
    "\n",
    "You can traverse the training set using the following loop, for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in train_loader:\n",
    "    data = batch.text.transpose(0, 1).to(device)\n",
    "    target = (batch.label - 1).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embedding layer\n",
    "\n",
    "The embedding layer performs a lookup on the embedding matrix, which is a matrix of size N x d, where N is the size of the vocabulary and d is the length of the embeddings. The lookup operation retrieves the row of the matrix corresponding to each of the words in the input.\n",
    "\n",
    "You can use the method `nn.Embedding.from_pretrained(glove.vectors)` to create the embedding layer, then call it like any other regular layer in the `forward` method.\n",
    "\n",
    "The input will be a batch of sentences containing the word indices, with shape `[batch_size, sentence_length]` (the variable `data` in the above example) and the output will be a batch of sentences containing the word embeddings, of shape `[batch_size, sentence_length, d]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4.2 (3 points)\n",
    "\n",
    "For this exercise, do the same task as above, but replace the CNN with an LSTM. Everything else in your network should remain more or less the same.\n",
    "\n",
    "Evaluate your results using the test data, comparing the two models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}