Entailment with TensorFlow-GRU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll start by doing all necessary imports, and we’ll let our Jupyter Notebook know that it should display graphs and images in the notebook itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import urllib\n",
    "import sys\n",
    "import os\n",
    "import zipfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The files we're about to use may take five minutes or more to download, so if you're following along by running the program in the corresponding notebook, feel free to start running the next few cells. In the meantime, let’s explore textual entailment in further detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_zip_file = \"glove.6B.zip\"\n",
    "glove_vectors_file = \"glove.6B.50d.txt\"\n",
    "\n",
    "snli_zip_file = \"snli_1.0.zip\"\n",
    "snli_dev_file = \"snli_1.0_dev.txt\"\n",
    "snli_full_dataset_file = \"snli_1.0_train.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from six.moves.urllib.request import urlretrieve\n",
    "    \n",
    "#large file - 862 MB\n",
    "if (not os.path.isfile(glove_zip_file) and\n",
    "    not os.path.isfile(glove_vectors_file)):\n",
    "    urlretrieve (\"http://nlp.stanford.edu/data/glove.6B.zip\", \n",
    "                 glove_zip_file)\n",
    "\n",
    "#medium-sized file - 94.6 MB\n",
    "if (not os.path.isfile(snli_zip_file) and\n",
    "    not os.path.isfile(snli_dev_file)):\n",
    "    urlretrieve (\"https://nlp.stanford.edu/projects/snli/snli_1.0.zip\", \n",
    "                 snli_zip_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unzip_single_file(zip_file_name, output_file_name):\n",
    "    \"\"\"\n",
    "        If the outFile is already created, don't recreate\n",
    "        If the outFile does not exist, create it from the zipFile\n",
    "    \"\"\"\n",
    "    if not os.path.isfile(output_file_name):\n",
    "        with open(output_file_name, 'wb') as out_file:\n",
    "            with zipfile.ZipFile(zip_file_name) as zipped:\n",
    "                for info in zipped.infolist():\n",
    "                    if output_file_name in info.filename:\n",
    "                        with zipped.open(info) as requested_file:\n",
    "                            out_file.write(requested_file.read())\n",
    "                            return\n",
    "\n",
    "unzip_single_file(glove_zip_file, glove_vectors_file)\n",
    "unzip_single_file(snli_zip_file, snli_dev_file)\n",
    "# unzip_single_file(snli_zip_file, snli_full_dataset_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Now that we have our GloVe vectors downloaded, we can load them into memory, deserializing the space separated format into a Python dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_wordmap = {}\n",
    "with open(glove_vectors_file, \"r\",encoding = \"utf-8\") as glove:\n",
    "    for line in glove:\n",
    "        name, vector = tuple(line.split(\" \", 1))\n",
    "        glove_wordmap[name] = np.fromstring(vector, sep=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Once we have our words, we need our input to contain entire sentences and process it through a neural network. Let's start with making the sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence2sequence(sentence):\n",
    "    \"\"\"\n",
    "     \n",
    "    - Turns an input sentence into an (n,d) matrix, \n",
    "        where n is the number of tokens in the sentence\n",
    "        and d is the number of dimensions each word vector has.\n",
    "    \n",
    "      Tensorflow doesn't need to be used here, as simply\n",
    "      turning the sentence into a sequence based off our \n",
    "      mapping does not need the computational power that\n",
    "      Tensorflow provides. Normal Python suffices for this task.\n",
    "    \"\"\"\n",
    "    tokens = sentence.lower().split(\" \")\n",
    "    rows = []\n",
    "    words = []\n",
    "    #Greedy search for tokens\n",
    "    for token in tokens:\n",
    "        i = len(token)\n",
    "        while len(token) > 0 and i > 0:\n",
    "            word = token[:i]\n",
    "            if word in glove_wordmap:\n",
    "                rows.append(glove_wordmap[word])\n",
    "                words.append(word)\n",
    "                token = token[i:]\n",
    "                i = len(token)\n",
    "            else:\n",
    "                i = i-1\n",
    "    return rows, words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To better visualize the word vectorization process, and to see what the computer sees when it looks at a sentence, we can represent the vectors as images. Each row represents a single word, and the columns represent individual dimensions of the vectorized word.  The vectorizations are trained in terms of relationships to other words, and so what the representations actually mean is ambiguous.  The computer can understand this vector language, and that’s the most important part to us. Generally speaking, two vectors that contain similar colors in the same positions represent words that are similar in meaning.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3X+YXVV97/H3Z34nM0kmIYFAEgggIqgIEhGqWPD6VPzRWi291Var1ts8XuvTequ2tHpb61O99qmXWnu99aatYh+t1aJYam2tVTEgIA2IQASqKBAkASY/Z/JzfnzvH2ePTrLWISfnnJl9Zufzep7z5Jw1a++91tl71qysvfb6KiIwM7Pq6Sq7AGZmNjvcwJuZVZQbeDOzinIDb2ZWUW7gzcwqyg28mVlFVbKBl3SFpPslfV/SVWWXp1mSPibpcUn3zEhbJukrkr5X/Lu0zDIeK0lrJH1d0r2SNkv6rSJ9vtdrQNJtkr5T1OuPivTTJX2rqNdnJPWVXdZjJalb0rclfbH4PO/rdLyoXAMvqRv4CPAS4FzgNZLOLbdUTbsGuOKItKuAr0bEWcBXi8/zyQTw9og4B7gY+I3i/Mz3eh0EXhgRzwLOB66QdDHwJ8CfFfXaCbypxDI267eAe2d8rkKdjguVa+CBi4DvR8QPIuIQ8PfAK0ouU1MiYiOw44jkVwCfKN5/Avj5OS1UiyJia0TcUbwfpdZwrGL+1ysiYqz42Fu8AnghcG2RPu/qJWk18DLgr4vPYp7X6XhSxQZ+FbBlxudHirSqOCkitkKtsQROLLk8TZO0FrgA+BYVqFcxlHEn8DjwFeABYFdETBRZ5uO1+CHgd4Cp4vMJzP86HTeq2MArk+b1GDqMpCHgc8DbImJP2eVph4iYjIjzgdXU/id5Ti7b3JaqeZJeDjweEbfPTM5knTd1Ot70lF2AWfAIsGbG59XAoyWVZTY8JunkiNgq6WRqvcV5RVIvtcb9UxHx+SJ53tdrWkTsknQDtXsMw5J6ih7vfLsWnwf8nKSXAgPAYmo9+vlcp+NKFXvw/wGcVdzp7wNeDVxfcpna6Xrg9cX71wP/WGJZjlkxhvs3wL0RcfWMH833eq2QNFy8XwC8iNr9ha8DVxbZ5lW9IuL3ImJ1RKyl9nv0tYj4FeZxnY43quJqkkWP40NAN/CxiHhfyUVqiqRPA5cBy4HHgD8EvgB8FjgVeBj4xYg48kZsx5L0fOBG4G5+Mq77+9TG4edzvc6jdsOxm1rH6bMR8V5JZ1C70b8M+Dbw2og4WF5JmyPpMuAdEfHyqtTpeFDJBt7MzKo5RGNmZriBNzOrLDfwZmYV5QbezKyi3MCbmXWQIxd3a0WlG3hJ68suQ7tVsU5QzXpVsU5Q3Xp1kCMXd2tapRt4oIoXYhXrBNWsVxXrBNWtV+mOXNytVVVv4M3M5pMjF3drSeXWoulZMBi9S5YB0Lt4KQtWrgnVeZar61D6g6nedC2lqPNnsGsil5Y/2FR3Zo2mzH67Dua3n1xQ27530VIWnrQmAHQMx5/sT4+f+17qPvaWKb4yl2D3wfx1OdmfqeyMffYNLmVwRVGvyUy5csfPHilfh+4de9PEoQXZ7ScWpGXtGk/3Gl35Epyz+gkATl3Vw7pnDQTAvY+syObtmmxsv9Gd3Tx/vnLfVeY7rR0rkzez05nnum/hMEPLaucqe15y11Wd3yFl6p/7Hay332wzWOfCyF2bo2OPjkRE/uQ04MWXD8b2HXW+3CPcftfBzcCBGUkbImLD9IeZi7sVTw63rHINfO+SZZzxq799WFp3nYeoF29JW8ixlelv0nTjeqT+nekVt2Akf7IPLkmv8Fyju+jhQ9ntd5zbn6QN7Egv2IHt+ePvWZue6tz3kvuFBYjMldKzL01b8oP92e13n5E2plO9+WP170nrNTGQFiz3Bxbyjcniv7s1Pf66C7Lb73jaQJI2+Fj6vY4vzLdat3zwo0naJe94czZv/+7G9ntoUf5YufOVu676d+f/8Ga/18wl1Lu3zh/uvnT7ngPp70W98g/sTA82uirfLOX2m0vLdqaAxQ8eSNK+tvFdD2UzN2hkxyTf+vLqhvL2nvzAgYhY9yRZksXdJH0yIl7bbPk8RGNm1rRgMqYaeh11T/nF3Zpu3KGCPXgzs7kSwFQHL4fvBt7MrAVT7bkfepiIuAG4odX9uIE3M2tSEIw3MPxSllLG4CUNS3pL8f6ydjyxZWY21wKYJBp6laGsm6zDwFtKOraZWdtMEQ29ylDWEM0HgDOLCPTjwF5J1wLPAG6nFiEmJF0IXA0MASPAGyJia0llNjM7TACTHRw0qawe/FXAA0UE+ncCFwBvA84FzgCeVwRm/gvgyoi4EPgYMC9D75lZdU01+CpDp9xkvS0iHgEoevVrgV3UevRfqcVpphvI9t6LxY/WQ+3pVTOzuRAljq83olMa+JnPVE5SK5eAzRFxydE2Lh733QCwYOWazv22zaxSIiCzikXHKGuIZhRYdJQ89wMrJF0CIKlX0tNnvWRmZg0Tkw2+ylBKDz4itkv6pqR7gP3AY5k8hyRdCXxY0hJqZf0QsHluS2tmlhfAVAf34EsboomIX66T/tYZ7+8EXjBnhTIzO0Zl9c4b0Slj8GZm807tQSc38GZmlRPAeL3F7juAG3gzsyYFYrKDV113A29m1oKpelFyOkDlGnhNQe/Y4be1Fz+cD/1zcDiN3jSQidJ0oM75y4XHGx/K/zXfdXa6k4GRNG3bxWnkJoDVX0/DJ+04J42SNDGYP34uyk/faPp8Xb0oSWOr0v2edNP2JG3XM/MPmi27a1eS9sRzhrN5R1el52VwW1rWXJ0Adj01TVt69lOStB1nppGbAHoyQam2XZSWael92c350M61Sdqe0/Ln5ZSb07BaT5yXntfufKAvevam12Duutx7Sv74g4+m3+uCkfQi2P70vuz2k5nLdcVd40laLqIZwOPPTpughdvy01J2p6eQXHy+3rH8dTG6NhOicWM2a8M8Bm9mVlli0mPwZmbVU4vo5AbezKxyIsShSIfvOoUbeDOzFkx5DN7MrHpqN1k9RGNmVkGdfZO1tJJJWifpw0fJMzZX5TEzO1bTN1kbeZWhzMXGNgGbyjq+mVk7THbwg05t/bMi6V2S7pf075I+Lekdkm6QtK74+XJJDxbvL5P0xeL9kKSPS7pb0l2SfuGI/S6XdIukl7WzvGZmrQjEePQ09CpD245aBMh+NbX4qj3AHdQCaDfifwK7I+KZxb5+/DikpJOA64F3R8RX2lVeM7NWHU83WS8FrouIfQCSrj+GbV9E7Y8DABGxs3jbC3wV+I2I+Ea9jQ+LyTrkmKxmNjcCHT9DNJCNPjsx4zj5xT9qC0rU2/Z24MVPetCIDRGxLiLW9SwYbLSsZmYt6+SbrO086kbglZIWSFoE/GyR/iBwYfH+yjrb/hvw40hOM4ZoAvg14GmSrmpjWc3MWhYBk9HV0KsMbTtqRNwBfAa4E/gccGPxow8C/13SzcDyOpv/MbBU0j2SvgNcPmO/k9SGby6X9JZ2ldfMrFW1m6zdDb2ORtKApNskfUfSZkl/1Gr52nprNyLeB7wPQNJ7irT7gPNmZHt3kX4DcEPxfgx4fWZ/Q8W/hzjKMI2ZWRnaeJP1IPDCiBiT1AvcJOlfIuLWZnfoJ1nNzJoUqG0BPyIigOmHO3uLV35x/AbNWgMfEe+ZrX2bmXWKdk6TlNRNbWLJU4CPRMS3Wtlf507gNDPrcAFMRVdDL2C5pE0zXuuT/UVMRsT5wGrgIknPaKV8HqIxM2uajiVk30hErGskY0TsknQDcAVwT5OFq14D37NviuXfOTx+6ZYX5+fGL//OZJJ2aFH6n5q9q/PH2pOZ1f/Uv3wsm7d3bEWSNrEwE+f01r3Z7Q+cmMaTHF2bOc538xdb/650KG9iQZp3fDC/fS72597TlyRpfXvS7xRgYlEavHPhE/m821ekl+XjF6X5zvhcGqcWYPDRzGU9mR6rXl1HT0/jlC7+fnqu+kbz5f/8IxckaSd8Nx/sdnR1+r0svyfNO7o6PwujN/MVdB9Mz1W9mK57zkjrtXRzutO+Nb3Z7UdPS9O6Dqbf39CP0jitAPtXpLFe68XaPfGO9PvuHUuPlfsdBtj+jPYH5ghoaIZMIyStAMaLxn0BtQdA/6SVfVaugTczmysRmh5+aYeTgU8U4/BdwGcj4out7NANvJlZC9r1EFNE3EVtLa+2cQNvZtak2nrwnbsWjRt4M7OmdXZEJzfwZmZNqk2TdA/ezKxyptei6VQt/d9C0lpJTc/RNDOb7zp5ueBZ78FL6i5WhDQzq5TacsGdO0TTjj8rPZI+UcRSvVbSQkkPSvoDSTcBvyjpfEm3Fnmuk7RU0omSbgeQ9CxJIenU4vMDxX6ukfRhSTdL+oGkeuvJm5mVYirU0KsM7WjgzwY2RMR5wB5ges32AxHx/Ij4e+Bvgd8t8twN/GFEPA4MSFpMLdzfJuBSSacBj0+H/qM2+f/5wMuBD+QKIGn99PoO4+P5J0HNzNqttppkw2vRzLl2DNFsiYhvFu8/Cfxm8f4zAJKWAMMzYqp+AviH4v3NwPOAFwDvp7bugvhJsBCAL0TEFPDdIgB3IiI2ABsAFg+taml5TTOzRtWWKqj2NMkjG9Tpz410pW+k1ns/DfhH4HeL7Wc+nntwxvvOHewys+NQW5cqaLt2lOxUSZcU718D3DTzhxGxG9gp6dIi6XXAdG9+I/Ba4HtFL30H8FLgm5iZzQNTqKFXGdrRwN8LvF7SXcAy4C8zeV4P/GmR53zgvQAR8WDx843FvzcBuyJiZxvKZWY2q6Zn0TTyKkNLQzRFA31u5kdrj8h3J3BxnX2cOuP9+6mNxU9/fsMReYeaLqyZ2Szo5CEaP8lqZtakdsZknQ1u4M3MmhTAhHvwZmbV5CGaOTS+qIutlx4eom/wR/mp8XtXpicmFwZv6KH8f8GiO5O+fVc2b9/u4SRtsj8NVzZ6Rj68YNdEWoe+3enxp+qc0X0r07zKLCDRk4+CR99Yevz9y9NFloa/tz+7/dZLFiZp/bvz5yVXhon9afkfuyj/XS18LA3j1r8tDY3Xtyd//MEfZX5hM6e6Z396HIDHRtNy9Twlf2IWPZyehK7xtFwnfPdAdvvJ/sYWutJIvq6LtqRp48NpLMrctVJP3860rAdXpOcf4OSb0+vlwPL09wLy38vkQHpi6oX8O7Q2/x22pMSnVBtRuQbezGyuOOCHmVmFuQdvZlZBDvhhZlZRgZiY8k1WM7NK8hi8mVkVRWcP0ZT6fwtJvynpXkmfKrMcZmbNmB6D79SAH2X34N8CvCQiflhyOczMmuIefIakjwJnANdLerukLxQh/W6VdF6R58OS/qB4/2JJGyV17h0NMzuuBGJyqquhVxlKaywj4s3Ao8Dl1Faf/HYR0u/3qYX4A7gK+CVJlwMfBt5YrBtvZtYROnk9+LKHaKY9H/gFgIj4mqQTJC2JiN2Sfp3aevH/IyIeyG0saT2wHqB38dK5KrOZHefCN1kbkvuGpheeeCawHTil3sYRsSEi1kXEuu6F+fVJzMxmQ4Qaeh2NpDWSvl5MPNks6bdaLVunNPAbgV8BkHQZMBIReySdBrwduAB4iaTnlldEM7MjNTaDpsFe/gTw9og4h1qApN+QlAuo1LBOGaJ5D/DxIqTfPmohAAX8DfCOiHhU0puAayQ9JyJmYVk4M7Nj10jvvLH9xFZga/F+VNK9wCrgu83us9QGPiLWzvj4ikyWF83Iezu14Rozs44QAZNT7R+Dl7SW2sjFt1rZT6f04M3M5qVjmCGzXNKmGZ83RMSGIzNJGgI+B7wtIva0UjY38GZmTQqOaYhmJCLWPVkGSb3UGvdPRcTnWyyeG3gzs+a1bxmCGfcd742Iq9uxz06ZRWNmNi9FNPZqwPOA1wEvlHRn8XppK2VzD97MrAVtnEVzE/lngprmBt7MrEm1WTSdOxDiBt7MrAUNDr+Uwg28mVkL2jVEMxvcwJuZNSlobJ2ZsriBNzNrQQeP0LiBNzNrWkDMwlIF7eKYrGZmLWjXcsGzoewevGOymtm81smzaByT1cysSdNr0XRqD94xWc3MmhVAqLFXCcoeopnmmKxmNi918hBNpzTwLcdkBTYALDh5TQd/3WZWLfIsmgY4JquZzU/R4KsEndKDfw+OyWpm8014qYK6HJPVzOa9Dh4U7pQevJnZPOUevJlZNXXwxG038GZmzZqeB9+h3MCbmbXA8+DnUM/+4ITN44elaTKfd8vrJpK0xTcOJGlja/NncOjBTOLJK7J5Dy3pTdIOLEtnqR5anO8NRHeatvKW/UnaxGD+lPbsT481uibdae/efF2H79+bpO1+ymC6/dZd2e2X3d+XpD1xfp2ypodi0YOZfW7OZAR2P2VhkjY1lB5/76r8d92Tfq3ZG2mjq/LlX74oLdfIgsXZvNt+Ki1D13i638UP5I+1YtOeJG3/yel52bcicwEBu85O01bfkP5eDG1N0wB6Dqb7ffiKJUnawm11WsGutF6D2/K/sPuXp8daMJLmHVudn/192qfT7R/Kl+rYuIE3M6soD9GYmVWT3IM3M6ugEHTwUgVu4M3MWuEevJlZRbmBNzOrqA5u4BtaTVLSzbNdkAbKcJmkL5ZdDjOzH6tCwI+I+KnZLoiZ2XzUybNoGu3Bjx3Zg5b0fyS9oXj/oKT3S7pF0iZJz5b0ZUkPSHpzkeeyIqbqdZK+K+mj0/FVJf1Mse0dkv5B0lCRfoWk+yTdBLyq3ZU3M2tZm9aDl/QxSY9LuqddRWtnwI8tEXEJcCNwDXAlcDHw3hl5LqIWwOOZwJnAqyQtB94NvCging1sAn5b0gDwV8DPApcCK9tYVjOztlA09mrANcAV7SxbO2+yXl/8ezcwFBGjwKikA5KGi5/dFhE/AJD0aWqxWA8A5wLfrMX4oA+4BXga8MOI+F6R/5MUcVePNDMma/+C4VwWM7PZ0abx9YjYKGltW3ZWOJYGfoLDe/xHLtpysPh3asb76c/Txzny71hQW0z5KxHxmpk/kHR+Jn/WzJisi4ZXd/CImJlVSonh+BpxLEM0DwHnSuqXtAT4L00c7yJJpxdj778E3ATcCjxP0lMAJC2U9FTgPuB0SWcW274mu0czszI1Pga/vLhHOf3Kjki0U6M9+IiILZI+C9wFfA/4dhPHuwX4ALUx+I3AdRExVdys/bSk/iLfuyPiP4sv4J8ljVD7Y/CMJo5pZjZr1HjAj5GIWDeLRUkctYGXdAKwAyAifgf4nSPzzIytGhHXULtZcNjPivH1fRHxS5ntvwY8J5P+r9TG4s3MOtN8HaKRdAq1XvcH56Y4ZmbzR6MzaBqZRVNMPLkFOFvSI5Le1Gr5nrQHHxGPAk9t9SDFvm4AbmjHvszMOkb7ZtG0/T6j16IxM2tFBw/RuIE3M2tBJy9VULkGfrJP7FlzeLXq3eUeujWNk6qp9Gwt3Zzf/uDS9L9me8/IP2i157T0qx56NI0n2b+7XpzQtBL14q82aioNU8pUPnQnI88aStIWZuJhTi5L89Wz8NH8b0Yu/uyhJen3cmhZpgLkzyG3ZU7iC5+b3T73HfRMNP5bfO25n0zSfvqWd2bznnBnWq+JNKQsC5/IX8Q7npHGeu0bS/NOpZc6AEvvTdMm+9MyHRjOXxgHlqd5+3dm9rkgf1337U6/1x1n56/rRVvSeo2dkpZrxbcPZbfv334gm96SOKZZNHOucg28mdmccg/ezKyi3MCbmVVTJ4/Bt3M1STMz6yDuwZuZtaKDe/Bu4M3MmuVZNGZmFeYefHuotmKZIqKD/2aa2fGi1iCVXYr6Zv0mq6TflnRP8XqbpD+R9JYZP3+PpLcX798p6T8k3SXpj4q0tZLulfR/gTuANbNdZjOzhrUpJutsmNUGXtKFwBuB51KLz/rrwN9TC/Yx7b8C/yDpZ4CzqMVtPR+4UNILijxnA38bERdExEOZ46yfXkR/Yv/e2auQmdlMbVxNcjbM9hDN86kF9dgLIOnz1AJon1gsRbwC2BkRD0v6TeBn+EkgkSFqDf7DwEMRcWu9g8wM2bfwxDUd/B8mM6ucDh4wnu0Gvt46mtcCVwIrqfXop/P+r4j4f4ftoBaE1t1yM+tIx/MY/Ebg54s4q4PAK4EbqTXqr6bWyF9b5P0y8GuShgAkrZJ04iyXz8ysNR08Bj+rPfiIuEPSNcBtRdJfR8S3ASQtAn4UEVuLvP8m6RzgliK83xjwWiBdstDMrBOU2Hg3YtanSUbE1cDVmfRnZtL+HPjzzG4cbNvMOlInD9HMq3nwZmYdxw28mVk1eakCM7MqOt7H4M3MqkrUnwveCSrXwEcXjA8d/pV3H8znXbxlIkkbW5nGeMzFnQTo39n4n+7+Xen/4w4sTWepLno4H09yx7n9SdrAjnSfA9vzk472L0+PteDxtPxR52rNxUk9uChNHOjPX1K5mJ71bk7l6tW7LxOTNXP8uvudSr+XU765P7v9jqcNJGmDj6fbjy/MzzI+sXswSRv+z/z/4/t3N7bfA8P5Y+XO196T0u+lf3f++BMDmfirS9Pte/fmt+/alm7fcyA9AYcW5cvfvyet/6FF+WtofDA9Vt9Yeqx9J+YD0Pbsm6UJee7Bm5lVk2fRmJlVlRt4M7MKcsAPM7MKcw/ezKyaOnkMftYDfuRIGp4O+iHpMklfLKMcZmYta+NiY5KukHS/pO9LuqrVopXSwAPDwFuOmsvMrMO1K+CHpG7gI8BLgHOB10g6t5WyldXAfwA4U9KdwJ8CQ5KulXSfpE8VsVeRdKGkb0i6XdKXJZ1cUnnNzFJBLeBHI6+juwj4fkT8ICIOUVtW/RWtFK+sBv4q4IGIOB94J3AB8DZqf7XOAJ4nqRf4C+DKiLgQ+BjwvpLKa2aWmA663aaQfauALTM+P1KkNa1TbrLeFhGPABS9+rXALmrLBH+l6NB3A1tzG0taD6wH6F28dA6Ka2ZWaPwm63JJm2Z83lCEG52We468pVu4ndLAz1xMYJJauQRsjohLjrbxzJisC1Y6JquZzR1Fw03OSESse5KfPwKsmfF5NfBos+WC8oZoRoFFR8lzP7BC0iUAknolPX3WS2Zm1qhGZ9A09jfgP4CzJJ0uqY9aWNPrWyleKT34iNgu6ZuS7gH2A49l8hySdCXwYUlLqJX1Q8DmuS2tmVl97ZoHHxETkt5KLT51N/CxiGipvSttiCYifrlO+ltnvL8TeMGcFcrM7Bi1c6mCiPgS8KV27a9TxuDNzOanDr7r5wbezKxZjU+BLIUbeDOzVriBNzOrnukHnTqVG3gzsxZoqnNb+Mo18N2HYPHDh9/W7tudxl4FmOpNHwM46dbdSdromfkp++OD6fb1Yk9Gd/qQWu7u+xPPTmOvApxwz3iStuOcNPbkzrPrBFXNJC/5Xpq2IBMjFGDnWemlMrEwzdc1kcYzBeiaTH8JljyQj4n60EvTHU/mv5asqb70WN0Hn5vm68l/V6Nr07S9p6T1X3p/fvrE6V9Yn6Qt788f68Da9BweWpzmG9yab0Syex1P8+bimQIcWpKm5661qb465V/W2KM0I+vy39XwPWn815U37cjmfeSKZUnaijvTsv7wVfkyPXFRX5q4MZu1ccewUmQZKtfAm5nNJUd0MjOrKvfgzcyqyTdZzcyqKIDGFxubc27gzcxa4DF4QNJYRAzN1fHMzGab58GbmVVVREcP0cz5evCShiR9VdIdku6W9Ioi/c2S7ixeP5T0dUlvkvRnM7b9dUlXz3WZzczqaWPIvrYrI+DHAeCVEfFs4HLgf0tSRHy0iNH6HGqRTa6mFnT254r4rABvBD5eQpnNzPLaF/Cj7coYohHwfkkvoBZrfBVwErCt+PmfA1+LiH8CkPQ14OWS7gV6I+LuZIczYrL2LXRMVjObOx6DP9yvACuACyNiXNKDwACApDcApwFvnZH/r4HfB+6jTu99ZkzWoWWOyWpmcySAzDIcnaKMBn4J8HjRuF9OrUFH0oXAO4BLI+LHE48i4luS1gDPBs4robxmZnW5B3+4TwH/JGkTcCe1njnUeu3LgK9LAtgUEf+t+NlngfMjYudcF9bM7El18CyaOWvgp+fAR8QIcEkmyxufZPPnA3/2JD83MytFJ/fgy5hF0zBJw5L+E9gfEV8tuzxmZodpdAbNcTSLpmERsQt4atnlMDPLESDfZDUzqyZ5DN7MrIIc0WluTfXAgaWHhxfrGk/DggH07E+Xgdvz1DRe2o5z87cqug6maaf+c36iz9iZS5K0H12e2f5f8kvT7T05PVWTmQhk3YfyodW6DqVp+1Zmth/P17Vnb5o2dmp6Za/60kh2+/2npw+g7V+ZD+83sD2tw4Fl6bHq3dzqfSKtw9AXb0+3P+fM7Pajpw7nd3yEhY+l4eIAuhal53B8Uf5Xbdl96YnZ9tzMia1T1wMnpN9Vz74089DWfCjGyf60XD3707x7TszHTJzMhPJb8lhap549mToBO5+VHmvxlnyIzP4dab16RtNzMLBtMLt972g2uUWdvRZN5Rp4M7O51MmzaNzAm5m1wj14M7MKis6eRdPR8+DNzDreHMyDl/SLkjZLmpK0rtHt3MCbmbVAEQ29WnQP8Cpg47Fs5CEaM7NWzMEYfETcC1Cs09WwUht4Se8BxiLig2WWw8ysKUEtqkWHcg/ezKxJ4piGX5YXq+hO21DEsqjtS/p3IPN0Cu+KiH9spnxz3sBLehfwq8AW4AngdknnAx8FFgIPAL8WETslPQf4G2AvcBPwkoh4xlyX2cysrqmGu/AjEVH3BmlEvKg9BfqJOb3JWgT1eDVwAbUbBs8pfvS3wO9GxHnA3cAfFukfB94cEZcA+UfxzMzKMj1E08irBHM9i+ZS4LqI2BcRe4DrgUFgOCK+UeT5BPACScPAooi4uUj/u3o7lbRe0iZJmyb2Z56pNzObJXMxi0bSKyU9Qi2Wxj9L+nIj25UxBt9oTRu+XTwzJuvCEx2T1czm0NzMorkOuO5Yt5vrHvxG4JWSFkhaBPwstfH1nZIuLfK8DvhGEZ5vVNLFRfqr57isZmZHUSw21sirBHPag4+IOyR9hlos1ofiOVp3AAACoklEQVSAG4sfvR74qKSFwA/4Sfi+NwF/JWkvcAOwey7La2b2pALo4KUK5nyIJiLeB7wv86OLM2mbixuvSLoK2JTJY2ZWGgf8aN7LJP0etXI+BLyh3OKYmR3BDXxzIuIzwGfKLoeZWVYAU27gzcwqyBGdzMyqyw383IojJn/uOTUfk3XBSDrV/uCSdOZo3646B8rM1N97ehrTFWqxYo+0+qvphTG+KF/WiYH0YP07M7E3tzX+wO/I09NC7VuRnzk7kImHOXx/mu/g6jT2LMDBJWm99p9QL/5reqzeTOzPgZE6v1iZGGrdy9KYsLvOysf+zO330JL0+PtO7M1u/42fTtfO+/kb3pnNu395eg5W3DmRpI2dUucafiIt68HhtKx7V+a3n8yEWt11VpqoOpdVz4H0+LvPSOOvLr0vv72m0mtg9+n566J3ND3WvlPSuL6DW/LXRcxGaxfAZOeuNlbJBt7MbG4EhBt4M7Nq8hCNmVkFeRaNmVmFuQdvZlZRbuDNzCooAiY7N1SFG3gzs1a4B29mVlFu4M3Mqig8i2a2SVoPrAfoHUqfWDQzmxUB4QedZpdD9plZabxUgZlZBUXAlBt4M7Nq6uCbrHMddLslkr4k6ZSyy2FmNi2mphp6lWFe9eAj4qVll8HM7Ccc8MPMrJq82JiZWTUFEF6qwMysgsIBP8zMKis6eIhG0cE3CJoh6QngoeLjcmCkxOLMhirWCapZryrWCapVr9MiYkWzG0v6V2rfRyNGIuKKZo/VjMo18DNJ2hQR68ouRztVsU5QzXpVsU5Q3XpV0byaB29mZo1zA29mVlFVb+A3lF2AWVDFOkE161XFOkF161U5lR6DNzM7nlW9B29mdtxyA29mVlFu4M3MKsoNvJlZRbmBNzOrqP8PM1VXGDgGhMAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAH11JREFUeJzt3XmUJlWd5vHvU1lZudVOVbEWFKAoJbLW0OxNM3SLtEcYG8eNVno8XaO0W48ybS9npKePx3Z0bI6t3XY5oqjYLriDI9Iow6IgVVCUBaiNUEgBUmRt1JrLm7/5443CrLw3qTczo/J9M+r5nBMnI+57I+LeiMhf3rwR7w1FBGZmVl3Tml0AMzPbvxzozcwqzoHezKziHOjNzCrOgd7MrOIc6M3MKq7ygV7SRZJ+IekRSe9vdnnGS9K1kjZIWjssbb6kWyT9e/FzXjPLOFaSFkv6kaSHJT0o6d1F+pStl6ROST+V9EBRp78t0o+WdE9Rp69ImtHsso6HpDZJ90u6sViuRL2qrtKBXlIb8EnglcBS4A2Slja3VOP2OeCiEWnvB26NiBcDtxbLU8kg8N6IOB44A/iz4vxM5Xr1ARdExEnAycBFks4APgz8Q1GnzcBbm1jGiXg38PCw5arUq9IqHeiB04FHIuLRiOgHvgxc0uQyjUtE3A5sGpF8CXBdMX8dcOmkFmqCIuLpiLivmN9GPYAczhSuV9RtLxbbiymAC4AbivQpVac9JB0B/CHwf4plUYF6HQiqHugPB54Ytry+SKuKgyPiaagHTWBRk8szbpKWAKcA9zDF61V0b6wGNgC3AL8CtkTEYJFlql6H1wD/HRgqlg+iGvWqvKoHemXSPOZDi5E0E/g68J6IeK7Z5ZmoiKhFxMnAEdT/qzw+l21ySzUxkl4FbIiIVcOTM1mnVL0OFNObXYD9bD2weNjyEcBTTSrL/vCMpEMj4mlJh1JvQU4pktqpB/nrI+IbRfKUrxdARGyRdBv1+w9zJU0vWr9T8To8G3i1pIuBTmA29Rb+VK/XAaHqLfp7gRcXTwbMAF4PfKfJZSrTd4C3FPNvAb7dxLKMWdHH+xng4Yj42LCPpmy9JC2UNLeY7wIupH7v4UfAZUW2KVUngIj4y4g4IiKWUP89+mFEvIkpXq8Dhao+emXRArkGaAOujYgPNrlI4yLpX4HzgQXAM8AHgG8BXwWOBH4NvDYiRt6wbVmSzgHuAH7Gb/t9/4p6P/2UrJekE6nflGyj3pD6akT8T0nHUH8YYD5wP3B5RPQ1r6TjJ+l84H0R8aoq1avKKh/ozcwOdFXvujEzO+A50JuZVZwDvZlZxTnQm5lVnAO9mVkLGjmA3EQcEIFe0vJml2F/qGK9qlgncL1sXEYOIDduB0SgB6p6MVaxXlWsE7heNgYjB5CbqAMl0JuZTSUjB5CbkEqPdTO9qyfa58ynffY8ug5ZHAAa5fth0/rTD4ba0zGbYpQ/jdMGc2n5nQ21ZcaCymx3Wl9+/VpXff32WfPoPrio1xj2X+tI9587LqN+lS5TfGUux7a+/DVa68hUttjmjJ559Cxc/PyuVcuUK7f/7J7ydWjbtCNNnNmVXX+wKy3rtIF0qzEtX4Ljj3gWgCMPn86ykzoD4OH1C7N5p9Ua2260ZVfPn6/cscoc0/q+MnkzGx1+rmd0z2Xm/Pr5yp6X3HU1yu+QMvXP/Q6Ott1sSBzlwhh5be7evYX+gR2jXUYNecXv9cTGTaMc3BFWrel7ENg9LGlFRKyAvQeQK76FPGGVDvTtc+ZzzJv/215pbaN8OXv2E2mk3H5I+hu1J8iO1LE5vfK6evMnvW9OeqXngu+sX/dn19+0tCNJ69yUXuWdG/P7f25JetpzxyX3iwsQmatm+s40bc6ju7Lrbz0mDapD7fl9dTyX1muwMy1Y7g8t5IPK7C/dne5/2SnZ9Te9tDNJ63kmPa4D3fno9ZOPfipJO/N9b8vm7dja2Hb7Z+X3lTtfueuqY2v+D3D2uGYuofYdo/wBn5GuP313+nsxWvk7N6c723Z4PkTltptLyzaqgNnrdu+1fO99n8zmG4veTTXuufmIhvK2H/qr3RGxbJSPkwHkJH0xIi4fb9ncdWNmVoqgFkMNTS+4lfwAcuMO8lDxFr2Z2WQJYKhFh+N3oDczK8lQOfdOnxcRtwG3TXQ7DvRmZiUIgoF9dMs0iwO9mVkJAqi568bMrNrcR29mVmEB1Fr0RU4O9GZmJWnNHvomP0cvaa6kK4v588sYpc3MrBmCoNbgNNma/YWpucCVTS6DmdmERcBAg9Nka3bXzd8Dx0paDQwAOyTdAJwArKL+RvmQdBrwMWAm0AtcERFPN6vQZmYpURt11KXmanaL/v3AryLiZOAq4BTgPcBS4BjgbEntwD8Cl0XEacC1wAdH26Ck5ZJWSlpZ25UZvMrMbD8IYCgamyZbs1v0I/00ItYDFK38JcAW6i38WyQBtAGjtuaLEeBWAM+PWGlmNhlatUXfaoF++BiKNerlE/BgRJzZnCKZme1b/QtTrRnom911sw2YtY88vwAWSjoTQFK7pJft95KZmY1BAAMxraFpsjW1RR8RGyXdJWktsAt4JpOnX9JlwMclzaFe5muABye3tGZmowtErelt57ymd91ExBtHSX/HsPnVwHmTVigzs3EYGu1tPU3W9EBvZlYFrdxH70BvZlYKUWtC/3sjHOjNzEpQf8OUA72ZWWVFiP5oa3YxshzozcxKMuQ+ejOz6qrfjHXXjZlZhflmbFO09cOsJ4ZGpOWHv1EtTet+Nn2NQO9J+RM5c33jrxwY6En/vZv15GCStuG0juz6i1b1JWlbj5mRpHVvyNd1sDvdf9/cNF/Hlvz6u+en6894Ls23+biu7PoHrd2eJo7yZp7HLk2/ON25Md3/9B359be+JE2ft+pFSVrbhkyZgE1/nPa5bsrkO+qb+f1fs3lJkjbYkf/3ftMZ6a/jrHVpvqH27Oq09adp0wbTcm0+Pr//eT9P886/L63tjmMyFwsw0JP+bvS3p/vatTC///7Zaf1nbB3lGs5cWm3prwU7D83va8N5e++r7/GJd7n4ZqyZ2QGg5i9MmZlVVyAGojVDamuWysxsivHNWDOzigvkrhszs6rzzVgzswqLwI9XmplVWf1mbDlDIEjqBG4HOqjH6Rsi4gPj3Z4DvZlZSUq8GdsHXBAR2yW1A3dK+r8Rcfd4NuZAb2ZWgkClvXgkIgLY8y2+9mLKf3usAa3ZoWRmNgXVmNbQ1AhJbZJWAxuAWyLinvGWq2UCvaRLJS0dtnyFpMOaWSYzs0YFMBTTGpqABZJWDpuWJ9uLqEXEycARwOmSThhv2Sa160ZSW0RkRpUB4FLgRuChYvkKYC3w1CQUzcxsgjSWVwn2RsSyRjJGxBZJtwEXUY+JY1Zai17SEkk/l3SdpDWSbpDULWmdpP8h6U7gtZKOlfR9Sask3SHppZLOAl4NfETSakl/ASwDri+W/1DSN4ft6/clfaOsspuZTVQAA9HW0LQvkhZKmlvMdwEXAj8fb9nKbtG/BHhrRNwl6VrgyiJ9d0ScAyDpVuBtEfHvkn4H+KeIuEDSd4AbI+KGIt8rgfdFxEpJAv63pIUR8SzwJ8BnSy67mdm4RWhPt0wZDgWuk9RGvUH+1Yi4cbwbKzvQPxERdxXzXwTeVcx/BUDSTOAs4Gv12A3UnxN9QRERkr4AXC7ps8CZwJtzeYu+ruUAM7rnjbMaZmZjV9YXpiJiDXBKKRuj/EA/8vGfPcs7ip/TgC3FDYax+izwXWA38LWISAdwByJiBbACYOb8xeN+HMnMbCzq49G35lg3ZT91c6SkM4v5NwB3Dv8wIp4DHpP0WgDVnVR8vA0Y/paJvZYj4inqN2b/BvhcyeU2M5ug+humGpkmW9l7fBh4i6Q1wHzgnzN53gS8VdIDwIPAJUX6l4GrJN0v6VjqwfxTxc3YPe+TuZ5699BDIzdqZtZM9ccr1dA02cruuhmKiLeNSFsyfCEiHqP+mBAj0u8Clg5L+hXw9RHZzgE+PfFimpmVq8yxbso2ZYZAkLSKel//e5tdFjOznMoPUxwR64Bxf3Orge2ftr+2bWY2UfVhilvzZuyUadGbmbW6ZvS/N8KB3sysBPXRKyvedWNmdiCrD4HgQG9mVmFu0ZuZVV6rfjO20oF+sAs2vnzvA995wtZs3tmfn52k1WakJ61vcX92/S270iF7DrtjZzbv3J3ps7YDs9O0WU8MZdevdaZ5lRnsYd0b8yNAHPnVgSTtmdPbk7Rp+aoy75fpSNMD3Y23ZKb9+pkkbcOrjs3mPeKHaSEevzgt69Hf7cuur1pnkvb07y9K0obSTdYNpiNtdD2R/tp0rd+cXf26R85I0gYOzgeDnifT8zV9d5o2NJhff9fCNH3eI2n5B3vyv/adG9O8m06dn6QNdOf3P2N7WtaOrem1MntdfqTyp9+ZnsM5n+3J5h3sSssw98dPJGmdJx+eXf+Qn+xd180bJz5aip+6MTM7ALjrxsyswsp8Z2zZHOjNzEoQwKBb9GZm1eauGzOzKmvSyJSNcKA3MytBK794xIHezKwkbtGbmVXYnhePtCIHejOzEgRicMg3Y83MKq1V++jH9edH0rskPSzpSUmfKLtQZmZTTlTvnbFXAq8EfhdYVl5x8iRNj4h0IA4zsxbRyn30Y27RS/oUcAzwHWDesPSjJN0qaU3x80hJbZIeVd1cSUOSzivy3yHpRZJ6JF0r6V5J90u6pPj8Cklfk/Rd4AeSDpV0u6TVktZKOrecQ2BmVo5WbdGPOdBHxNuAp4DfA4YP2fcJ4PMRcSJwPfDxiKgBvwSWAucAq4BzJXUAR0TEI8BfAz+MiP9QbPMjkvYMWXcm8JaIuAB4I3BzRJwMnASsHnNtzcz2k0DUhqY1NE22Mm/Gngm8ppj/AvC/ivk7gPOAo4EPAX8K/D/g3uLzPwBeLel9xXIncGQxf0tEbCrm7wWuldQOfCsisoFe0nJgOcD0OfNyWczM9otK3Yxt0J4Bnu8AzgVOB74HzAXOB24vPhfwRxFxcjEdGREPF5/teH5jEbdT/4PxJPAFSW/O7jRiRUQsi4hlbT35sazNzMoWLXwztsxA/2Pg9cX8m4A7i/l7gLOAoYjYTb3L5b9S/wMAcDPwTkkCkHRKbuOSjgI2RMSngc8Ap5ZYdjOzCYtQQ9O+SFos6UfF040PSnr3RMpVZtfNu6h3rVwFPAv8CUBE9El6Ari7yHcH8AbgZ8Xy3wHXAGuKYL8OeFVm++cDV0kaALYD2Ra9mVlzlNpaHwTeGxH3SZoFrJJ0S0Q8NJ6NjSvQR8SSYvZzxURErAMuGCX/ucPmvwR8adjyLuot/JHrPL/tYvk64LrxlNfMbDI00lpvbDvxNPB0Mb9N0sPA4cDkBXozM9tbBNSGyu9/l7QEOIV6N/i4ONCbmZVkDE/dLJC0ctjyiohYMTKTpJnA14H3RMRz4y2XA72ZWQmCMXXd9EbEC44qUDxK/nXg+oj4xkTK5kBvZlaK8m7GFg+mfAZ4OCI+NtHtteaYmmZmU1BEY1MDzgb+GLigGPZltaSLx1sut+jNzEpS4lM3d0J5X7N1oDczK0H9qZvW7CRxoDczK0mD3TKTrtKBfvouWPDA0F5p23vzA53tmpeeoel9adrsBzqy6x/2w01J2oYz8/vq2JJudyhzJrYem28dtG9P09v60nyHfzt/ejee0JakLVhTS9I2H5fmA+jcmqb1zUv/yzzkx9uy6/e9bHFmm0OZnPDEH8xI0qbvSPel/vz6uw5O83b/Jj3+ka9q1pzH0n1tPGVuNm/3F9O8z5yRL2utMz2vB9+dHuzeU+Zk15++O02LaWn9B0YZAqqtLy1X94b0WO2en7+udi5Kyz99d7p+/8z8we74t/R3a8ch2az0z0nr1fPkwiRt+yH5ss7f1L93QkkBuqyum7JVOtCbmU2WoLFxbJrBgd7MrCQt2nPjQG9mVoqA2A9DIJTBgd7MrCTuujEzqzg/dWNmVmFjHOtmUjnQm5mVIQAHejOzanPXjZlZpalln7rZrwMzSFonacH+3IeZWcuIBqdJ5ha9mVkZonVvxpbWopfUI+kmSQ9IWivpdcVH75R0n6SfSXppkXe+pG9JWiPpbkknFulXS7pW0m2SHpX0rmHbv1zST4txmf9F0hhGJzEzmwQt2qIvs+vmIuCpiDgpIk4Avl+k90bEqcA/A+8r0v4WuD8iTgT+Cvj8sO28FHgFcDrwAUntko4HXgecHREnAzXgTSWW3cysBGpwmlxlBvqfARdK+rCkcyNiz7B7e951uApYUsyfA3wBICJ+CBwkac+QfDdFRF9E9AIbgIOB/wicBtwraXWxfEyuEJKWS1opaeVA3/YSq2dmtg9DDU6TrLQ++oj4paTTgIuBD0n6QfHRngF0a8P2l/uTtucfmuED7u5ZR8B1EfGXDZRjBbACYOb8xS36sJOZVU4LP0dfZh/9YcDOiPgi8FHg1BfIfjtF14uk86l37zz3AvlvBS6TtKhYZ76ko0opuJlZSUp8Z2ypynzq5uXARyQNAQPA24EbRsl7NfBZSWuAncBbXmjDEfGQpL8BfiBpWrH9PwMeL6nsZmYT16J9CGV23dwM3Dwiecmwz1cC5xfzm4BLMtu4esTyCcPmvwJ8pazympmVrkW7bvwcvZlZSVT1Fr2Z2QEtBC06BIIDvZlZWdyiNzOrOAd6M7OKc6A3M6uwFv7ClAO9mVlJWvWpm/06Hr2Z2QGlpNEri1F8N0haW0axKt2ir82A7UfsPZqxBvN5Z60fSNKeW9KepLVvz5+lZ86el6TNXpff2c5F6WHPtQQOeqiWXX/X/PTv8/yHdiVpuxd1ZNdftKovSdtwapq389l8XYemp/+etu1O807bnR5TgA2/OyvdZnqoAZj3ULrd/nR1Nr68O7v+QQ+l56D7xvuStG2vWZbf/+p0NOyhzADZQ6P8Jn3vH65J0l7xF3+ezTs0Pa1r36K0XgOz8t0DM57LrD8rvVba+rOr03tieg3kzuus9fnrsr07LVfuWu3alB/Va+bTafrmF+cPbPdvMtfFvBlJmkYZQGxk3shc0+NRYov+c8An2Htk33GrdKA3M5tUJfXRR8TtkpaUsjEc6M3MytGkl4o0woHezKwsjQf6BZJWDlteUQyxvl840JuZlWS0ewIZvRGRvzG0HzjQm5mVpUW7bvx4pZlZCRSNT/vclvSvwE+Al0haL+mtEymbW/RmZmUp76mbN5SyoYIDvZlZWVq068aB3sysJK06BIIDvZlZGWJMT91Mqpa+GSvpx80ug5lZw0oa66ZsLd2ij4izml0GM7OGtWjXTau36LcXP8+XdJukGyT9XNL1klpz4GczO2CV9Xhl2Vo60I9wCvAeYClwDHB2LpOk5ZJWSlpZ27ljMstnZtaSplKg/2lErI+IIWA1sCSXKSJWRMSyiFjW1t0zqQU0swOc++gnbPgg6jWmVtnNrOpa+KkbB0szs7K06M1YB3ozsxIIf2FqXCJiZvHzNuC2YenvaFKRzMxG50BvZlZhTXp0shEO9GZmZfHNWDOzanOL3sys6hzozcwqrElfhmqEA72ZWUncdWNmVnUO9M0x1Lb3cttgPl+tKx32J/f6x8FZ+UEzOzanZzja8nmnDaZ5ax1p3hlb8oXdflhHkrZtSWeS1rmxll1/83EzMnkz5R9lfNC+OekH7Znx42qz0jIBdP8m3ddQe35f7TvTxxg0lO5/2ijndaA7c14H08zdz/QlaQC756V16NiWHteBWn7YqDnTupK00Vp93b1pufrmtCVp7dvzGxh5rUP+uup5Mv9oyGBn5rhmLqHBrvyF0daXlmv67jStb3b+WHVuTnc2fZRxCXPXS+5YjXZdtG/b+wPVyonQHgLBzKzK3EdvZlZtKqZW5EBvZlYWt+jNzKrNT92YmVWdA72ZWYX5xSNmZgcAt+jNzKqtVfvop9LLwc3MWluJLweXdJGkX0h6RNL7J1IsB3ozs5IoGpv2uR2pDfgk8EpgKfAGSUvHW66mBnpJcyVdWcyfL+nGZpbHzGzcgvqLRxqZ9u104JGIeDQi+oEvA5eMt2jNbtHPBa5schnMzCZsz8vBy2jRA4cDTwxbXl+kjUuzb8b+PXCspNXAALBD0g3ACcAq4PKICEmnAR8DZgK9wBUR8XSzCm1mltX4zdgFklYOW14RESuGLedGUxj3rd5mB/r3AydExMmSzge+DbwMeAq4Czhb0j3APwKXRMSzkl4HfBD4L7kNSloOLAdonz1v/9fAzKygaDgW90bEshf4fD2weNjyEdTj4rg0O9CP9NOIWA9QtPKXAFuot/BvkQTQBozami/+Kq4A6DpkcYs+7GRmlVPu6JX3Ai+WdDTwJPB64I3j3VirBfrhg4LXqJdPwIMRcWZzimRm1piynqOPiEFJ7wBupt64vTYiHhzv9pod6LcBs/aR5xfAQklnRsRPJLUDx02k0mZm+0OZQyBExPeA75WxraYG+ojYKOkuSWuBXcAzmTz9ki4DPi5pDvUyXwM40JtZa2nRzuJmt+iJiGy/U0S8Y9j8auC8SSuUmdlYNf7o5KRreqA3M6sMB3ozs+ra84WpVuRAb2ZWEg21ZqR3oDczK0O5z9GXyoHezKwkfsOUmVnVuUVvZlZtvhnbBNEG/XP2PvIDs/Nnom9eeigWPjCYpA1050d27np2IEnbsKwjm7fnybQMuxamg9X1nptdnTn3p2mhdP3HL8kNgAdzHkzTZ69Ly7/+wrbs+gsy++/cVEvSBrtGWf+OJ5O0p1+ZH4F1x6J0G9uOSf8/Pu4zG7PrP/bahUnazj8/K0mL/KFi20vS47Ipk2/Og/nr4kW3XZGk9SzK58392z/31l8laTuXLcmuv+3I9Bpu60uvtd3z8/sf7E7TZj2RFmr74fnzOn1nuq8ZO9L1azOyq7NrfrrdWlc+b99B6Qk7+gvpddV7Xv666juofa/loemjXABjEUDjg5pNqkoHejOzyeQ+ejOzCvNz9GZmVRfhrhszs6pzi97MrOoc6M3Mqs0tejOzKgug1pqR3oHezKwkbtGbmVWdn7oxM6u2Vm3R578LvR9I2l78PEzSDY3mz6RfKmlp2eUzM5uQGMM0ySYt0O8REU9FxGUT2MSlgAO9mbUUAapFQ9Nk22egl9Qj6SZJD0haK+l1ktZJWlB8vkzSbcX81ZKulXSbpEclvSuzvSWS1hbz3ZK+KmmNpK9IukfSsmF5P1js925JB0s6C3g18BFJqyUdW9JxMDObMEU0NE22Rlr0FwFPRcRJEXEC8P195H8p8ArgdOADktpfIO+VwOaIOBH4O+C0YZ/1AHdHxEnA7cCfRsSPge8AV0XEyRGRDu1nZtYMU7zr5mfAhZI+LOnciNi6j/w3RURfRPQCG4CDXyDvOcCXASJiLbBm2Gf9wI3F/CpgSQNlRdJySSslrazt2NHIKmZmJYjfjnezr2mS7fOpm4j4paTTgIuBD0n6ATDIb/9IdI5YpW/YfG0f+3ihQaAHIp4/IvvazvDyrgBWAHQevrhF74GbWRVN2aduJB0G7IyILwIfBU4F1vHbbpY/msD+7wT+c7GfpcDLG1hnGzBrAvs0M9s/pmqLnnrw/YikIWAAeDvQBXxG0l8B90xg//8EXCdpDXA/9a6bfXUNfRn4dHGj9zL305tZSwia8kRNIxrpurkZuDnz0XGZvFePWD5h2PzM4uc6YE/6buDyiNhdPEFzK/D48PzF/A3ADcX8XfjxSjNrRZMQ5yW9FrgaOB44PSJW7mudZn8zthv4UfFkjoC3R0R/k8tkZjYuk/To5FrgNcC/NLpCUwN9RGwDlu0zo5nZVDAJgT4iHgaQGn+hebNb9GZm1RCAXw5uZlZdYkzfel0gaXjf+ori0fD6tqR/Aw7JrPfXEfHtsZbNgd7MrCxDDTfpeyNi1G7riLiwnALVOdCbmZWhhbtuJn30SjOzqpqMQc0k/SdJ64EzgZsk5R5/34tb9GZmZZmcp26+CXxzLOtUOtBPG4Cep/ZOq/Xm/4np6k3/59p2eHp4hmbk97XzkI4kbf7Dg9m8tRnpY1Hzf55eIHMeyw/8uXNBmjbQk6YtHnWc0bRcG05N99Xz6/zakRnQY8fB6bGa/ev8VyI2/85hmW3mHxWbNpjuq/PZ9BxuOCtzUICep9L1D77psSRtyzlHZdfv2JzWq39OWtauZ/P/s994zieStEtvvSqbNzKX5o4zjk7Sth/Wll1/+s40rW9uWta2/nwwqnWkeQd60rQZz+XXj0yxdixKE6fvzq6OhtLttvXlr4vO3jTv1tMOTdJqo4ydu+Pgvcs11N74o4qja87wBo2odKA3M5s0AUzVIRDMzKwxzXipSCMc6M3MyuJAb2ZWYQFk7jO0Agd6M7NS+GasmVn1OdCbmVVYALXW/GqsA72ZWSkCwoHezKza3HVjZlZhfurGzOwA4Ba9mVnFOdCbmVVYBNRqzS5FVuUCvaTlwHKA9pnzmlwaMzugtGiLvnIvHomIFRGxLCKWTe/KjN1rZra/RDQ2TbLKtejNzJojWvapmynZopf0PUnp2yvMzJolIGKooWmyTckWfURc3OwymJklPASCmVmFRcCQA72ZWbW16FM3DvRmZiUJt+jNzKrMLx4xM6s2D2pmZlZtAYSHQDAzq7Dwi0fMzCovWrTrRtGiNw/KIOlZ4HFgAdDb5OLsD1WsVxXrBK5XqzsqIhZOZAOSvk/9eDSiNyIumsj+xqLSgX4PSSsjYlmzy1G2KtarinUC18uaa0qOdWNmZo1zoDczq7gDJdCvaHYB9pMq1quKdQLXy5rogOijNzM7kB0oLXozswOWA72ZWcU50JuZVZwDvZlZxTnQm5lV3P8HUZAkzQS1u/EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def visualize(sentence):\n",
    "    rows, words = sentence2sequence(sentence)\n",
    "    mat = np.vstack(rows)\n",
    "    \n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    shown = ax.matshow(mat, aspect=\"auto\")\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    fig.colorbar(shown)\n",
    "    \n",
    "    ax.set_yticklabels([\"\"]+words)\n",
    "    plt.show()\n",
    "    \n",
    "visualize(\"The quick brown fox fox jumped over the lazy dog.\")\n",
    "visualize(\"The pretty flowers shone in the sunlight.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "Recurrent Neural Networks (also known as RNNs) are a sequence-learning tool for neural networks. This type of neural network has only one layer’s worth of hidden inputs, which is re-used for each input from the sequence, along with a “memory” that’s passed ahead to the next input’s calculations. These are calculated using matrix multiplication where the matrix indices are trained weights, just like they are in a fully-connected layer. \n",
    "\n",
    "The same calculations are repeated for each input in the sequence, meaning that a single “layer” of a recurrent neural network can be unrolled into many layers. In fact, there will be as many layers as there are inputs in the sequence. This allows the network to process a very complex sentence. TensorFlow includes its own implementation of a vanilla RNN cell, BasicRNNCell, which can be added to your TensorFlow graph as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_size = 64\n",
    "rnn = tf.contrib.rnn.GRUCell(rnn_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "# Defining the constants for our network\n",
    "\n",
    "Since we aren’t going to use a vanilla RNN layer in our network, let's clear out the graph and add an LSTM layer, which TensorFlow also includes by default. Since this is going to be the first part of our actual network, let's also define all the constants we'll need for the network, which we'll talk about as they come up:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Constants setup\n",
    "max_hypothesis_length, max_evidence_length = 30, 30\n",
    "batch_size, vector_size, hidden_size = 128, 50, 64\n",
    "\n",
    "lstm_size = hidden_size\n",
    "\n",
    "weight_decay = 0.0001\n",
    "\n",
    "learning_rate = 1\n",
    "\n",
    "input_p, output_p = 0.5, 0.5\n",
    "\n",
    "training_iterations_count = 100000\n",
    "\n",
    "display_step = 10\n",
    "\n",
    "def score_setup(row):\n",
    "    convert_dict = {\n",
    "      'entailment': 0,\n",
    "      'neutral': 1,\n",
    "      'contradiction': 2\n",
    "    }\n",
    "    score = np.zeros((3,))\n",
    "    for x in range(1,6):\n",
    "        tag = row[\"label\"+str(x)]\n",
    "        if tag in convert_dict: score[convert_dict[tag]] += 1\n",
    "    return score / (1.0*np.sum(score))\n",
    "\n",
    "def fit_to_size(matrix, shape):\n",
    "    res = np.zeros(shape)\n",
    "    slices = [slice(0,min(dim,shape[e])) for e, dim in enumerate(matrix.shape)]\n",
    "    res[slices] = matrix[slices]\n",
    "    return res\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_data_into_scores():\n",
    "    import csv\n",
    "    with open(\"snli_1.0_dev.txt\",\"r\") as data:\n",
    "        train = csv.DictReader(data, delimiter='\\t')\n",
    "        evi_sentences = []\n",
    "        hyp_sentences = []\n",
    "        labels = []\n",
    "        scores = []\n",
    "        for row in train:\n",
    "            hyp_sentences.append(np.vstack(\n",
    "                    sentence2sequence(row[\"sentence1\"].lower())[0]))\n",
    "            evi_sentences.append(np.vstack(\n",
    "                    sentence2sequence(row[\"sentence2\"].lower())[0]))\n",
    "            labels.append(row[\"gold_label\"])\n",
    "            scores.append(score_setup(row))\n",
    "        \n",
    "        hyp_sentences = np.stack([fit_to_size(x, (max_hypothesis_length, vector_size))\n",
    "                          for x in hyp_sentences])\n",
    "        evi_sentences = np.stack([fit_to_size(x, (max_evidence_length, vector_size))\n",
    "                          for x in evi_sentences])\n",
    "                                 \n",
    "        return (hyp_sentences, evi_sentences), labels, np.array(scores)\n",
    "    \n",
    "data_feature_list, correct_values, correct_scores = split_data_into_scores()\n",
    "\n",
    "l_h, l_e = max_hypothesis_length, max_evidence_length\n",
    "N, D, H = batch_size, vector_size, hidden_size\n",
    "l_seq = l_h + l_e\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll also reset the graph to not include the RNN cell we added earlier, since we won't be using that for this network:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With both those out of the way, we can define our LSTM using TensorFlow as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm = tf.contrib.rnn.GRUCell(lstm_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss of certain pieces of crucial memory means that complicated relationships required for first order logic have a harder time forming with dropout, and so for our LSTM layer we’ll skip using dropout on internal gates, instead using it on everything else. Thankfully, this is the default implementation of Tensorflow’s DropoutWrapper for recurrent layers:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_drop =  tf.contrib.rnn.DropoutWrapper(lstm, input_p, output_p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "With all the explanations out of the way, we can finish up our model. The first step is tokenizing and using our GloVe dictionary to turn the two input sentences into a single sequence of vectors. Since we can’t effectively use dropout on information that gets passed within an LSTM, we’ll use dropout on features from words, and on final output instead -- effectively using dropout on the first and last layers from the unrolled LSTM network portions. \n",
    "\n",
    "The final output from the LSTMs will be passed into a set of fully connected layers, and then from that we’ll get a single real-valued score that indicates how strong each of the kinds of entailment are, which we use to select our final result and our confidence in that result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# N: The number of elements in each of our batches, \n",
    "#   which we use to train subsets of data for efficiency's sake.\n",
    "# l_h: The maximum length of a hypothesis, or the second sentence.  This is\n",
    "#   used because training an RNN is extraordinarily difficult without \n",
    "#   rolling it out to a fixed length.\n",
    "# l_e: The maximum length of evidence, the first sentence.  This is used\n",
    "#   because training an RNN is extraordinarily difficult without \n",
    "#   rolling it out to a fixed length.\n",
    "# D: The size of our used GloVe or other vectors.\n",
    "hyp = tf.placeholder(tf.float32, [N, l_h, D], 'hypothesis')\n",
    "evi = tf.placeholder(tf.float32, [N, l_e, D], 'evidence')\n",
    "y = tf.placeholder(tf.float32, [N, 3], 'label')\n",
    "# hyp: Where the hypotheses will be stored during training.\n",
    "# evi: Where the evidences will be stored during training.\n",
    "# y: Where correct scores will be stored during training.\n",
    "\n",
    "# lstm_size: the size of the gates in the LSTM, \n",
    "#    as in the first LSTM layer's initialization.\n",
    "lstm_back = tf.contrib.rnn.GRUCell(lstm_size)\n",
    "# lstm_back:  The LSTM used for looking backwards \n",
    "#   through the sentences, similar to lstm.\n",
    "\n",
    "# input_p: the probability that inputs to the LSTM will be retained at each\n",
    "#   iteration of dropout.\n",
    "# output_p: the probability that outputs from the LSTM will be retained at \n",
    "#   each iteration of dropout.\n",
    "lstm_drop_back = tf.contrib.rnn.DropoutWrapper(lstm_back, input_p, output_p)\n",
    "# lstm_drop_back:  A dropout wrapper for lstm_back, like lstm_drop.\n",
    "\n",
    "\n",
    "fc_initializer = tf.random_normal_initializer(stddev=0.1) \n",
    "# fc_initializer: initial values for the fully connected layer's weights.\n",
    "# hidden_size: the size of the outputs from each lstm layer.  \n",
    "#   Multiplied by 2 to account for the two LSTMs.\n",
    "fc_weight = tf.get_variable('fc_weight', [2*hidden_size, 3], \n",
    "                            initializer = fc_initializer)\n",
    "# fc_weight: Storage for the fully connected layer's weights.\n",
    "fc_bias = tf.get_variable('bias', [3])\n",
    "# fc_bias: Storage for the fully connected layer's bias.\n",
    "\n",
    "# tf.GraphKeys.REGULARIZATION_LOSSES:  A key to a collection in the graph\n",
    "#   designated for losses due to regularization.\n",
    "#   In this case, this portion of loss is regularization on the weights\n",
    "#   for the fully connected layer.\n",
    "tf.add_to_collection(tf.GraphKeys.REGULARIZATION_LOSSES, \n",
    "                     tf.nn.l2_loss(fc_weight)) \n",
    "\n",
    "x = tf.concat([hyp, evi], 1) # N, (Lh+Le), d\n",
    "# Permuting batch_size and n_steps\n",
    "x = tf.transpose(x, [1, 0, 2]) # (Le+Lh), N, d\n",
    "# Reshaping to (n_steps*batch_size, n_input)\n",
    "x = tf.reshape(x, [-1, vector_size]) # (Le+Lh)*N, d\n",
    "# Split to get a list of 'n_steps' tensors of shape (batch_size, n_input)\n",
    "x = tf.split(x, l_seq,)\n",
    "\n",
    "# x: the inputs to the bidirectional_rnn\n",
    "\n",
    "\n",
    "# tf.contrib.rnn.static_bidirectional_rnn: Runs the input through\n",
    "#   two recurrent networks, one that runs the inputs forward and one\n",
    "#   that runs the inputs in reversed order, combining the outputs.\n",
    "rnn_outputs, _, _ = tf.contrib.rnn.static_bidirectional_rnn(lstm, lstm_back,\n",
    "                                                            x, dtype=tf.float32)\n",
    "# rnn_outputs: the list of LSTM outputs, as a list. \n",
    "#   What we want is the latest output, rnn_outputs[-1]\n",
    "\n",
    "classification_scores = tf.matmul(rnn_outputs[-1], fc_weight) + fc_bias\n",
    "# The scores are relative certainties for how likely the output matches\n",
    "#   a certain entailment: \n",
    "#     0: Positive entailment\n",
    "#     1: Neutral entailment\n",
    "#     2: Negative entailment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to test the accuracy and begin to add in optimization constraints, we need to show TensorFlow how to calculate the accuracy, or -- the percentage of correctly predicted labels.\n",
    "\n",
    "We also need to determine a loss, to show how poorly the network is doing. Since we have both classification scores and optimal scores, the choice here is using a variation on softmax loss from Tensorflow: tf.nn.softmax_cross_entropy_with_logits. We add in regularization losses to help with overfitting, and then prepare an optimizer to learn how to reduce the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-17-97a0fabce845>:10: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See tf.nn.softmax_cross_entropy_with_logits_v2.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with tf.variable_scope('Accuracy'):\n",
    "    predicts = tf.cast(tf.argmax(classification_scores, 1), 'int32')\n",
    "    y_label = tf.cast(tf.argmax(y, 1), 'int32')\n",
    "    corrects = tf.equal(predicts, y_label)\n",
    "    num_corrects = tf.reduce_sum(tf.cast(corrects, tf.float32))\n",
    "    accuracy = tf.reduce_mean(tf.cast(corrects, tf.float32))\n",
    "\n",
    "with tf.variable_scope(\"loss\"):\n",
    "    cross_entropy = tf.nn.softmax_cross_entropy_with_logits(\n",
    "        logits = classification_scores, labels = y)\n",
    "    loss = tf.reduce_mean(cross_entropy)\n",
    "    total_loss = loss + weight_decay * tf.add_n(\n",
    "        tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES))\n",
    "\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "\n",
    "opt_op = optimizer.minimize(total_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can train the network! If you installed TQDM, you can use it to keep track of progress as the network trains.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|                                                  | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 0.0, Minibatch Loss= 1.093953, Training Accuracy= 0.39844\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|▌                                        | 10/782 [00:04<06:01,  2.14it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 10.0, Minibatch Loss= 1.093795, Training Accuracy= 0.39062\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  2%|▉                                        | 19/782 [00:05<03:40,  3.46it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 20.0, Minibatch Loss= 1.098048, Training Accuracy= 0.33594\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  4%|█▌                                       | 29/782 [00:06<02:45,  4.54it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 30.0, Minibatch Loss= 1.097638, Training Accuracy= 0.35938\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  5%|██                                       | 39/782 [00:07<02:18,  5.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 40.0, Minibatch Loss= 1.080854, Training Accuracy= 0.37500\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  6%|██▌                                      | 49/782 [00:08<02:02,  5.99it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 50.0, Minibatch Loss= 1.078458, Training Accuracy= 0.48438\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  8%|███                                      | 59/782 [00:09<01:51,  6.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 60.0, Minibatch Loss= 1.086889, Training Accuracy= 0.44531\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  9%|███▌                                     | 69/782 [00:10<01:44,  6.84it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 70.0, Minibatch Loss= 1.086531, Training Accuracy= 0.39062\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|████▏                                    | 79/782 [00:11<01:39,  7.09it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 80.0, Minibatch Loss= 1.093783, Training Accuracy= 0.38281\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 12%|████▋                                    | 90/782 [00:12<01:35,  7.28it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 90.0, Minibatch Loss= 1.088816, Training Accuracy= 0.42188\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 13%|█████▏                                   | 99/782 [00:13<01:32,  7.40it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 100.0, Minibatch Loss= 1.081175, Training Accuracy= 0.42969\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 14%|█████▋                                  | 110/782 [00:14<01:28,  7.55it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 110.0, Minibatch Loss= 1.085680, Training Accuracy= 0.40625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 15%|██████                                  | 119/782 [00:15<01:26,  7.69it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 120.0, Minibatch Loss= 1.086339, Training Accuracy= 0.44531\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 16%|██████▌                                 | 129/782 [00:16<01:23,  7.82it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 130.0, Minibatch Loss= 1.095140, Training Accuracy= 0.41406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 18%|███████▏                                | 140/782 [00:17<01:21,  7.91it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 140.0, Minibatch Loss= 1.080731, Training Accuracy= 0.39062\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 19%|███████▋                                | 150/782 [00:18<01:19,  7.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 150.0, Minibatch Loss= 1.083202, Training Accuracy= 0.46094\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 20%|████████▏                               | 160/782 [00:20<01:18,  7.96it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 160.0, Minibatch Loss= 1.085042, Training Accuracy= 0.43750\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 22%|████████▋                               | 170/782 [00:21<01:15,  8.06it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 170.0, Minibatch Loss= 1.076136, Training Accuracy= 0.40625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 23%|█████████▏                              | 180/782 [00:22<01:13,  8.17it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 180.0, Minibatch Loss= 1.088537, Training Accuracy= 0.39844\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 24%|█████████▋                              | 190/782 [00:22<01:11,  8.26it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 190.0, Minibatch Loss= 1.075635, Training Accuracy= 0.42188\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 26%|██████████▏                             | 200/782 [00:23<01:09,  8.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 200.0, Minibatch Loss= 1.042440, Training Accuracy= 0.50000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 27%|██████████▋                             | 210/782 [00:24<01:07,  8.44it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 210.0, Minibatch Loss= 1.051872, Training Accuracy= 0.49219\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 28%|███████████▎                            | 220/782 [00:25<01:05,  8.52it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 220.0, Minibatch Loss= 1.052626, Training Accuracy= 0.46094\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 29%|███████████▊                            | 230/782 [00:26<01:04,  8.57it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 230.0, Minibatch Loss= 1.050965, Training Accuracy= 0.44531\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 31%|████████████▎                           | 240/782 [00:27<01:02,  8.64it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 240.0, Minibatch Loss= 1.075476, Training Accuracy= 0.39844\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 32%|████████████▊                           | 250/782 [00:28<01:01,  8.70it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 250.0, Minibatch Loss= 1.050797, Training Accuracy= 0.50781\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 33%|█████████████▎                          | 260/782 [00:29<00:59,  8.76it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 260.0, Minibatch Loss= 1.090953, Training Accuracy= 0.39844\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 35%|█████████████▊                          | 270/782 [00:30<00:58,  8.80it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 270.0, Minibatch Loss= 1.085470, Training Accuracy= 0.35938\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 36%|██████████████▎                         | 280/782 [00:31<00:56,  8.85it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 280.0, Minibatch Loss= 1.077751, Training Accuracy= 0.39062\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 37%|██████████████▊                         | 290/782 [00:32<00:55,  8.90it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 290.0, Minibatch Loss= 1.085060, Training Accuracy= 0.41406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 38%|███████████████▎                        | 300/782 [00:33<00:53,  8.94it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 300.0, Minibatch Loss= 1.068324, Training Accuracy= 0.41406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 40%|███████████████▊                        | 310/782 [00:34<00:52,  8.95it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 310.0, Minibatch Loss= 1.061959, Training Accuracy= 0.43750\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 41%|████████████████▎                       | 320/782 [00:35<00:51,  8.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 320.0, Minibatch Loss= 1.036167, Training Accuracy= 0.49219\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 42%|████████████████▉                       | 330/782 [00:37<00:50,  8.92it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 330.0, Minibatch Loss= 1.040532, Training Accuracy= 0.47656\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 43%|█████████████████▎                      | 339/782 [00:38<00:49,  8.92it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 340.0, Minibatch Loss= 1.096496, Training Accuracy= 0.33594\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 45%|█████████████████▊                      | 349/782 [00:38<00:48,  8.96it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 350.0, Minibatch Loss= 1.054416, Training Accuracy= 0.52344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 46%|██████████████████▎                     | 359/782 [00:39<00:47,  8.98it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 360.0, Minibatch Loss= 1.073833, Training Accuracy= 0.48438\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 47%|██████████████████▊                     | 369/782 [00:40<00:45,  9.00it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 370.0, Minibatch Loss= 1.063901, Training Accuracy= 0.44531\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 49%|███████████████████▍                    | 380/782 [00:42<00:44,  9.00it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 380.0, Minibatch Loss= 1.045103, Training Accuracy= 0.53125\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 50%|███████████████████▉                    | 390/782 [00:43<00:43,  9.03it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 390.0, Minibatch Loss= 1.054666, Training Accuracy= 0.48438\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 51%|████████████████████▍                   | 400/782 [00:44<00:42,  9.06it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 400.0, Minibatch Loss= 1.003888, Training Accuracy= 0.52344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 52%|████████████████████▉                   | 410/782 [00:45<00:40,  9.09it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 410.0, Minibatch Loss= 1.035823, Training Accuracy= 0.50000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 54%|█████████████████████▍                  | 420/782 [00:46<00:39,  9.12it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 420.0, Minibatch Loss= 1.026305, Training Accuracy= 0.58594\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 55%|█████████████████████▉                  | 430/782 [00:47<00:38,  9.15it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 430.0, Minibatch Loss= 1.045424, Training Accuracy= 0.50000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 56%|██████████████████████▌                 | 440/782 [00:47<00:37,  9.18it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 440.0, Minibatch Loss= 1.023090, Training Accuracy= 0.49219\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 58%|███████████████████████                 | 450/782 [00:48<00:36,  9.21it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 450.0, Minibatch Loss= 0.997493, Training Accuracy= 0.61719\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 59%|███████████████████████▌                | 460/782 [00:49<00:34,  9.23it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 460.0, Minibatch Loss= 1.049472, Training Accuracy= 0.50781\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 60%|████████████████████████                | 470/782 [00:50<00:33,  9.24it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 470.0, Minibatch Loss= 1.014120, Training Accuracy= 0.52344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 61%|████████████████████████▌               | 480/782 [00:51<00:32,  9.25it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 480.0, Minibatch Loss= 1.047596, Training Accuracy= 0.50781\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 63%|█████████████████████████               | 490/782 [00:52<00:31,  9.27it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 490.0, Minibatch Loss= 1.078415, Training Accuracy= 0.41406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 64%|█████████████████████████▌              | 500/782 [00:53<00:30,  9.30it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 500.0, Minibatch Loss= 1.051723, Training Accuracy= 0.50000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 65%|██████████████████████████              | 510/782 [00:54<00:29,  9.32it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 510.0, Minibatch Loss= 1.016728, Training Accuracy= 0.50000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 66%|██████████████████████████▌             | 520/782 [00:55<00:28,  9.33it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 520.0, Minibatch Loss= 1.019636, Training Accuracy= 0.52344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 68%|███████████████████████████             | 530/782 [00:56<00:26,  9.35it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 530.0, Minibatch Loss= 1.000236, Training Accuracy= 0.54688\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 69%|███████████████████████████▌            | 540/782 [00:57<00:25,  9.35it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 540.0, Minibatch Loss= 1.045448, Training Accuracy= 0.48438\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 70%|████████████████████████████            | 549/782 [00:58<00:24,  9.37it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 550.0, Minibatch Loss= 1.062236, Training Accuracy= 0.42188\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 71%|████████████████████████████▌           | 559/782 [00:59<00:23,  9.38it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 560.0, Minibatch Loss= 1.018278, Training Accuracy= 0.56250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 73%|█████████████████████████████           | 569/782 [01:00<00:22,  9.40it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 570.0, Minibatch Loss= 1.065813, Training Accuracy= 0.50000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 74%|█████████████████████████████▌          | 579/782 [01:01<00:21,  9.42it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 580.0, Minibatch Loss= 1.000301, Training Accuracy= 0.58594\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 75%|██████████████████████████████▏         | 589/782 [01:02<00:20,  9.42it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 590.0, Minibatch Loss= 0.982817, Training Accuracy= 0.53125\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 77%|██████████████████████████████▋         | 599/782 [01:03<00:19,  9.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 600.0, Minibatch Loss= 1.011085, Training Accuracy= 0.52344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 78%|███████████████████████████████▏        | 610/782 [01:04<00:18,  9.44it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 610.0, Minibatch Loss= 1.062790, Training Accuracy= 0.48438\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 79%|███████████████████████████████▋        | 620/782 [01:05<00:17,  9.46it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 620.0, Minibatch Loss= 1.014338, Training Accuracy= 0.50781\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 81%|████████████████████████████████▏       | 630/782 [01:06<00:16,  9.47it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 630.0, Minibatch Loss= 0.989471, Training Accuracy= 0.53125\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 82%|████████████████████████████████▋       | 640/782 [01:07<00:14,  9.48it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 640.0, Minibatch Loss= 1.021406, Training Accuracy= 0.50781\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 83%|█████████████████████████████████▏      | 650/782 [01:08<00:13,  9.48it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 650.0, Minibatch Loss= 0.990550, Training Accuracy= 0.53906\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 84%|█████████████████████████████████▊      | 660/782 [01:09<00:12,  9.47it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 660.0, Minibatch Loss= 0.961677, Training Accuracy= 0.61719\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 86%|██████████████████████████████████▎     | 670/782 [01:10<00:11,  9.48it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 670.0, Minibatch Loss= 0.990464, Training Accuracy= 0.54688\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 87%|██████████████████████████████████▊     | 680/782 [01:11<00:10,  9.50it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 680.0, Minibatch Loss= 0.988465, Training Accuracy= 0.60156\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 88%|███████████████████████████████████▎    | 690/782 [01:12<00:09,  9.52it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 690.0, Minibatch Loss= 0.981487, Training Accuracy= 0.54688\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 90%|███████████████████████████████████▊    | 700/782 [01:13<00:08,  9.54it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 700.0, Minibatch Loss= 0.982812, Training Accuracy= 0.53125\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 91%|████████████████████████████████████▎   | 710/782 [01:14<00:07,  9.55it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 710.0, Minibatch Loss= 0.993268, Training Accuracy= 0.52344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 92%|████████████████████████████████████▊   | 720/782 [01:15<00:06,  9.57it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 720.0, Minibatch Loss= 0.961345, Training Accuracy= 0.60938\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 93%|█████████████████████████████████████▎  | 730/782 [01:16<00:05,  9.59it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 730.0, Minibatch Loss= 1.018493, Training Accuracy= 0.52344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 95%|█████████████████████████████████████▊  | 740/782 [01:17<00:04,  9.60it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 740.0, Minibatch Loss= 0.947984, Training Accuracy= 0.61719\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 96%|██████████████████████████████████████▎ | 750/782 [01:17<00:03,  9.62it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 750.0, Minibatch Loss= 0.960487, Training Accuracy= 0.58594\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 97%|██████████████████████████████████████▊ | 760/782 [01:18<00:02,  9.63it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 760.0, Minibatch Loss= 0.930914, Training Accuracy= 0.66406\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|███████████████████████████████████████▍| 770/782 [01:19<00:01,  9.64it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 770.0, Minibatch Loss= 0.999531, Training Accuracy= 0.57031\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████▉| 780/782 [01:20<00:00,  9.65it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 780.0, Minibatch Loss= 0.941837, Training Accuracy= 0.58594\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████| 782/782 [01:21<00:00,  9.65it/s]\n"
     ]
    }
   ],
   "source": [
    "# Initialize variables\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# Use TQDM if installed\n",
    "tqdm_installed = False\n",
    "try:\n",
    "    from tqdm import tqdm\n",
    "    tqdm_installed = True\n",
    "except:\n",
    "    pass\n",
    "\n",
    "# Launch the Tensorflow session\n",
    "sess = tf.Session()\n",
    "sess.run(init)\n",
    "\n",
    "# training_iterations_count: The number of data pieces to train on in total\n",
    "# batch_size: The number of data pieces per batch\n",
    "training_iterations = range(0,training_iterations_count,batch_size)\n",
    "if tqdm_installed:\n",
    "    # Add a progress bar if TQDM is installed\n",
    "    training_iterations = tqdm(training_iterations)\n",
    "\n",
    "for i in training_iterations:\n",
    "\n",
    "    # Select indices for a random data subset\n",
    "    batch = np.random.randint(data_feature_list[0].shape[0], size=batch_size)\n",
    "    \n",
    "    # Use the selected subset indices to initialize the graph's \n",
    "    #   placeholder values\n",
    "    hyps, evis, ys = (data_feature_list[0][batch,:],\n",
    "                      data_feature_list[1][batch,:],\n",
    "                      correct_scores[batch])\n",
    "    \n",
    "    # Run the optimization with these initialized values\n",
    "    sess.run([opt_op], feed_dict={hyp: hyps, evi: evis, y: ys})\n",
    "    # display_step: how often the accuracy and loss should \n",
    "    #   be tested and displayed.\n",
    "    if (i/batch_size) % display_step == 0:\n",
    "        # Calculate batch accuracy\n",
    "        acc = sess.run(accuracy, feed_dict={hyp: hyps, evi: evis, y: ys})\n",
    "        # Calculate batch loss\n",
    "        tmp_loss = sess.run(loss, feed_dict={hyp: hyps, evi: evis, y: ys})\n",
    "        # Display results\n",
    "        print(\"Iter \" + str(i/batch_size) + \", Minibatch Loss= \" + \\\n",
    "              \"{:.6f}\".format(tmp_loss) + \", Training Accuracy= \" + \\\n",
    "              \"{:.5f}\".format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your network is now trained! You should see accuracies around 50-55%, which can be improved by careful modification of hyperparameters and increasing the dataset size to include the entire training set. Usually, this will correspond with an increase in training time.\n",
    "\n",
    "Feel free to modify the following code by inserting your own sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Negative entailment\n"
     ]
    }
   ],
   "source": [
    "evidences = [\"Maurita and Jade both were at the scene of the car crash.\"]\n",
    "\n",
    "hypotheses = [\"Multiple people saw the accident.\"]\n",
    "\n",
    "sentence1 = [fit_to_size(np.vstack(sentence2sequence(evidence)[0]),\n",
    "                         (30, 50)) for evidence in evidences]\n",
    "\n",
    "sentence2 = [fit_to_size(np.vstack(sentence2sequence(hypothesis)[0]),\n",
    "                         (30,50)) for hypothesis in hypotheses]\n",
    "\n",
    "prediction = sess.run(classification_scores, feed_dict={hyp: (sentence1 * N),\n",
    "                                                        evi: (sentence2 * N),\n",
    "                                                        y: [[0,0,0]]*N})\n",
    "print([\"Positive\", \"Neutral\", \"Negative\"][np.argmax(prediction[0])]+\n",
    "      \" entailment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, once we're done playing with our model, we'll close the session to free up system resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}