tensorflow_tutorial_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train and visualize a model in Tensorflow - Part 3: TensorFlow's API\n",
    "\n",
    "We showed in [Part 2](https://github.com/PLN-FaMAF/tensorflowTutorial2018/blob/master/tensorflow_tutorial_2.ipynb) how to train a neural network model with scikit-learn's `MLPClassifier`. We also experienced how painfully slow the process is.\n",
    "\n",
    "TensorFlow's API has a steeper learning curve than Scikit-Learn, but offers a much better performance than the latter. In the first versions of TensorFlow, the only way to code a neural network was doing the math from scratch. This is what we will see here to have a point of comparison with the next tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "from sklearn.metrics import accuracy_score, classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminaries\n",
    "\n",
    "First we need a couple of functions that we will use in the training (and evaluation) process. These are two: `one_hot_encoding` and `next_batch`.\n",
    "\n",
    "The `one_hot_encoding` function takes an array of labels in scalar format (that is numbered from zero to the maximum number of labels there is) and convert each label into a one-hot vector: a vector with size equal to the number of unique labels, that has a one in only one dimension (the label's position when sorted) and zero in all the other dimensions. It is needed as TensorFlow's API only deals with binary labels.\n",
    "\n",
    "The `next_batch` function is needed in order to traverse the dataset. It takes a dataset, a batch size and an offset and returns `batch_size` elements starting from the offset from the dataset. This is needed because the neural network we are about to create can't fit more than a certain number of elements in memory at the same time as it's too expensive.\n",
    "\n",
    "We also load the data in this step so it's ready to train the neural network in the following steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_hot_encoding(labels, num_classes):\n",
    "    \"\"\"Convert class labels from scalars to one-hot vectors.\"\"\"\n",
    "    num_labels = labels.shape[0]\n",
    "    index_offset = np.arange(num_labels) * num_classes\n",
    "    one_hot = np.zeros((num_labels, num_classes))\n",
    "    one_hot.flat[index_offset + labels.ravel()] = 1\n",
    "    return one_hot\n",
    "\n",
    "def next_batch(data, target, offset, \n",
    "               batch_size, train_data=True):\n",
    "    \"\"\"Takes the next batch from data and target\n",
    "    given the offset used. Returns the data and\n",
    "    target batch as well as the offset for the \n",
    "    next iteration. If the data is for training, it \n",
    "    shuffles the content.\"\"\"\n",
    "    start = offset\n",
    "    end = offset + batch_size\n",
    "    num_examples = data.shape[0]\n",
    "    \n",
    "    if start == 0 and train_data:\n",
    "        perm = np.random.permutation(num_examples)\n",
    "        data = data[perm]\n",
    "        target = target[perm]\n",
    "        \n",
    "    if end > num_examples and train_data:\n",
    "        perm = np.random.permutation(num_examples)\n",
    "        data = data[perm]\n",
    "        target = target[perm]\n",
    "        start = 0\n",
    "        end = batch_size\n",
    "    elif end > num_examples:\n",
    "        end = num_examples\n",
    "\n",
    "    return end, data[start:end], target[start:end]\n",
    "\n",
    "# Loading the data\n",
    "\n",
    "newsgroup = np.load('./resources/newsgroup.npz')\n",
    "train_data = newsgroup['train_data']\n",
    "train_target = newsgroup['train_target']\n",
    "test_data = newsgroup['test_data']\n",
    "test_target = newsgroup['test_target']\n",
    "labels = newsgroup['labels']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Architecture\n",
    "\n",
    "Before training the neural network, we need to create it. Unlike Scikit-Learn, which internally defines the algorithm based on the parameters given to the model's class, in this case we need to define the operations of the network layer by layer. This is what makes TensorFlow much more difficult but it also make it much more flexible when we need to design novel and more complex neural networks (something plainly impossible on Scikit-Learn).\n",
    "\n",
    "The operations in TensorFlow are symbolic, we tell it what it needs to do (in this case create the operations for a multilayer perceptron), and then we compile it. You can think of TensorFlow as a DSL to do math.\n",
    "\n",
    "As TensorFlow is way faster than Scikit-Learn, we can improve the architecture and use two hidden layers for this case instead of one like with Scikit-Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of training instances given \n",
    "# to the network on each epoch\n",
    "batch_size = 100\n",
    "\n",
    "# Size of the input layer\n",
    "input_size = train_data.shape[1]\n",
    "\n",
    "# Number of classes (size of the output layer)\n",
    "num_classes = labels.shape[0]\n",
    "\n",
    "# Size of the hidden layers\n",
    "hidden_layer_1 = 5000\n",
    "hidden_layer_2 = 2000\n",
    "\n",
    "# Define the placeholders, this are needed \n",
    "# for the network to be given data, in this case\n",
    "# we have a placeholder for the data (x) and for\n",
    "# the target (y). Remember as the operations\n",
    "# are symbolic, we can't just feed the neural network\n",
    "# the raw dataset.\n",
    "x = tf.placeholder(tf.float32, [None, input_size])\n",
    "y = tf.placeholder(tf.float32, [None, num_classes])\n",
    "\n",
    "# We define a scope (important to keep named structure)\n",
    "# and define the operations in the first hidden layer.\n",
    "# What it basically does is take the input layer and\n",
    "# apply a matrix multiplication with a non-linearity\n",
    "# (the `relu` function).\n",
    "with tf.name_scope('hidden_layer_1'):\n",
    "    W_h1 = tf.Variable(\n",
    "        tf.truncated_normal(\n",
    "            [input_size, hidden_layer_1],\n",
    "            stddev=1.0 / np.sqrt(input_size)\n",
    "        ),\n",
    "        name='W_h1'\n",
    "    )\n",
    "    b_h1 = tf.Variable(\n",
    "        tf.zeros([hidden_layer_1]),\n",
    "        name='b_h1'\n",
    "    )\n",
    "    h1 = tf.nn.relu(tf.matmul(x, W_h1) + b_h1)\n",
    "\n",
    "# Same as before, we define the operations\n",
    "# for the second hidden layer\n",
    "with tf.name_scope('hidden_layer_2'):\n",
    "    W_h2 = tf.Variable(\n",
    "        tf.truncated_normal(\n",
    "            [hidden_layer_1, hidden_layer_2],\n",
    "            stddev=1.0 / np.sqrt(hidden_layer_1)\n",
    "        ),\n",
    "        name='W_h2'\n",
    "    )\n",
    "    b_h2 = tf.Variable(\n",
    "        tf.zeros([hidden_layer_2]),\n",
    "        name='b_h2'\n",
    "    )\n",
    "    h2 = tf.nn.relu(tf.matmul(h1, W_h2) + b_h2)\n",
    "\n",
    "# The last layer (output), is similar to the hidden \n",
    "# layers but in this case we don't apply the \n",
    "# non-linearity as the result is needed to calculate \n",
    "# the cost via the softmax function\n",
    "with tf.name_scope('output_layer'):\n",
    "    W_o = tf.Variable(\n",
    "        tf.truncated_normal(\n",
    "            [hidden_layer_2, num_classes],\n",
    "            stddev=1.0 / np.sqrt(hidden_layer_2)\n",
    "        ),\n",
    "        name='W_o'\n",
    "    )\n",
    "    b_o = tf.Variable(\n",
    "        tf.zeros([num_classes]),\n",
    "        name='b_o'\n",
    "    )\n",
    "    logits = tf.matmul(h2, W_o) + b_o\n",
    "\n",
    "# We define the cost function as the mean of the \n",
    "# softmax cross-entropy given the labels (or target) \n",
    "# y and the result of the output layer in the\n",
    "# previous step\n",
    "cost = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits(\n",
    "        labels=y, logits=logits\n",
    "    )\n",
    ")\n",
    "\n",
    "# Finally, we calculate the prediction of the \n",
    "# neural network using the argmax of the logit\n",
    "y_hat = tf.argmax(logits, 1)\n",
    "\n",
    "# We define the train step that we will keep calling\n",
    "# in each epoch to fit the neural network. This uses\n",
    "# an optimizer algorithm (Adam in this case) to minimize\n",
    "# the cost function described early.\n",
    "train_step = tf.train.AdamOptimizer(0.01)\\\n",
    "    .minimize(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Once the architecture is defined we train the neural network with the dataset. Also, unlike Scikit Learn is not as easy as to call a `fit` method of the model that will do all the work magically in the backend. In this case we need to keep feeding the model with a batch of data so the model fits that batch by adjusting the weights of the network. The process is repeated many times (as many as needed, this is a hyperparameter) until we think is ok to stop it (generally because the model is converging as it's not improving anymore)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss for epoch 100: 0.654\n",
      "Loss for epoch 200: 0.189\n",
      "Loss for epoch 300: 0.110\n",
      "Loss for epoch 400: 0.407\n",
      "Loss for epoch 500: 0.017\n",
      "Loss for epoch 600: 0.002\n",
      "Loss for epoch 700: 0.193\n",
      "Loss for epoch 800: 0.062\n",
      "Loss for epoch 900: 0.009\n",
      "Loss for epoch 1000: 0.209\n",
      "Loss for epoch 1100: 0.211\n",
      "Loss for epoch 1200: 0.004\n",
      "Loss for epoch 1300: 0.373\n",
      "Loss for epoch 1400: 0.078\n",
      "Loss for epoch 1500: 0.003\n",
      "Loss for epoch 1600: 0.045\n",
      "Loss for epoch 1700: 0.000\n",
      "Loss for epoch 1800: 0.000\n",
      "Loss for epoch 1900: 0.065\n",
      "Loss for epoch 2000: 0.001\n"
     ]
    }
   ],
   "source": [
    "# First we need to define a session, which is\n",
    "# TensorFlow's way to excecute a piece of code.\n",
    "# Then we initialize the variables given by the\n",
    "# neural network code we designed previously.\n",
    "# As this is a Jupyter notebook, the session is\n",
    "# interactive. I recommend reading more about this\n",
    "# in TensorFlow's documentation.\n",
    "sess = tf.InteractiveSession()\n",
    "tf.global_variables_initializer().run()\n",
    "\n",
    "# The offset is set to zero for the first time\n",
    "offset = 0\n",
    "\n",
    "# We train the network for 2000 epochs\n",
    "for epoch in range(1, 2001):\n",
    "    # For each epoch we obtain the batch of data\n",
    "    # needed to fit the network\n",
    "    offset, batch_data, batch_target =\\\n",
    "        next_batch(train_data, train_target,\n",
    "                   offset, batch_size)\n",
    "    # We run the train step operation (defined before)\n",
    "    # and return the loss value every 100 epochs\n",
    "    _, loss = sess.run(\n",
    "        [train_step, cost],\n",
    "        feed_dict={\n",
    "            x: batch_data,\n",
    "            y: one_hot_encoding(batch_target, num_classes)\n",
    "        })\n",
    "    if epoch % 100 == 0:\n",
    "        print(\"Loss for epoch %02d: %.3f\" % (epoch, loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "The evaluation step, similaryly to the train step, is done by traversing the whole dataset exactly once (while for training we traverse many times the whole dataset). We store for each batch of data the results and we use that to compare it later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.86\n",
      "\n",
      "Classification Report\n",
      "=====================\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.92      0.97      0.95       160\n",
      "          1       0.74      0.83      0.78       195\n",
      "          2       0.87      0.79      0.83       197\n",
      "          3       0.77      0.75      0.76       196\n",
      "          4       0.69      0.84      0.76       192\n",
      "          5       0.96      0.76      0.85       196\n",
      "          6       0.84      0.71      0.77       194\n",
      "          7       0.86      0.93      0.89       198\n",
      "          8       0.99      0.88      0.94       199\n",
      "          9       0.74      0.99      0.85       199\n",
      "         10       0.95      0.96      0.96       200\n",
      "         11       0.99      0.89      0.94       198\n",
      "         12       0.92      0.76      0.83       196\n",
      "         13       0.83      0.92      0.87       198\n",
      "         14       0.88      0.92      0.90       197\n",
      "         15       0.93      0.92      0.92       200\n",
      "         16       0.98      0.78      0.87       182\n",
      "         17       0.97      0.92      0.94       188\n",
      "         18       0.75      0.93      0.83       155\n",
      "         19       0.86      0.79      0.82       126\n",
      "\n",
      "avg / total       0.87      0.86      0.86      3766\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# First we define the initial offset to zero.\n",
    "# The number of test examples is needed to calculate\n",
    "# the maximum number of epochs needed.\n",
    "# And a list with the predictions of each batch of data\n",
    "offset = 0\n",
    "test_examples = test_data.shape[0]\n",
    "predictions = []\n",
    "\n",
    "# For each batch in the dataset we run the prediction\n",
    "# operation (y_hat) given the data.\n",
    "for _ in range(np.int(test_examples / batch_size) + 1):\n",
    "    offset, batch_data, _ = next_batch(\n",
    "        test_data, test_target, offset, batch_size, False)\n",
    "    predictions.append(sess.run(y_hat, feed_dict={x: batch_data}))\n",
    "\n",
    "# Finally, concatenate the predictions and check the performance\n",
    "predictions = np.concatenate(predictions)\n",
    "accuracy = accuracy_score(test_target, predictions)\n",
    "\n",
    "print(\"Accuracy: %.2f\\n\" % accuracy)\n",
    "\n",
    "print(\"Classification Report\\n=====================\")\n",
    "print(classification_report(test_target, predictions))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow-tutorial]",
   "language": "python",
   "name": "conda-env-tensorflow-tutorial-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}