tcn_mnist.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "tcn_mnist.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sunnnymskang/NIST_BLE_challenge/blob/master/tcn_mnist.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b10XJFD-y-fS",
        "colab_type": "text"
      },
      "source": [
        "# Solving Sequential MNIST with Temporal Convolutional Networks(TCNs)\n",
        "\n",
        "- Sequential MNIST: Based on the work of [Aymeric Damien](https://github.com/aymericdamien/TensorFlow-Examples/) and [Sungjoon](https://github.com/sjchoi86/tensorflow-101/blob/master/notebooks/rnn_mnist_simple.ipynb)\n",
        "- Temporal Convolutional Networks: [Bai, S., Kolter, J. Z., & Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling.](http://arxiv.org/abs/1803.01271)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dqicde8-y-fW",
        "colab_type": "text"
      },
      "source": [
        "### MNIST Dataset Overview\n",
        "\n",
        "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flattened and converted to a 1-D numpy array of 784 features (28*28).\n",
        "\n",
        "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n",
        "\n",
        "To classify images using a recurrent neural network, we consider every image row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then handle 28 sequences of 28 timesteps for every sample.\n",
        "\n",
        "More info: http://yann.lecun.com/exdb/mnist/\n",
        "\n",
        "### Temporal Convolutional Networks Overview\n",
        "\n",
        "![TCNs](https://cdn-images-1.medium.com/max/1000/1*1cK-UEWHGaZLM-4ITCeqdQ.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeeOZueJy-fW",
        "colab_type": "text"
      },
      "source": [
        "## System Information"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PSj1MFEyy-fa",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "outputId": "5ca37e54-e340-4d60-e7b9-5b0548cf680c"
      },
      "source": [
        "!pip install watermark"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting watermark\n",
            "  Downloading https://files.pythonhosted.org/packages/33/ec/4b5b4674bdccebdf2c1de6a597579adac99f2a160ae402671eaebd3521f4/watermark-1.6.0-py3-none-any.whl\n",
            "Requirement already satisfied: ipython in /usr/local/lib/python3.6/dist-packages (from watermark) (5.5.0)\n",
            "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.6/dist-packages (from ipython->watermark) (4.3.2)\n",
            "Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.6/dist-packages (from ipython->watermark) (0.8.1)\n",
            "Requirement already satisfied: pexpect; sys_platform != \"win32\" in /usr/local/lib/python3.6/dist-packages (from ipython->watermark) (4.5.0)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.6/dist-packages (from ipython->watermark) (4.3.0)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.6/dist-packages (from ipython->watermark) (0.7.4)\n",
            "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.6/dist-packages (from ipython->watermark) (39.2.0)\n",
            "Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.6/dist-packages (from ipython->watermark) (1.0.15)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.6/dist-packages (from ipython->watermark) (2.1.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from traitlets>=4.2->ipython->watermark) (1.11.0)\n",
            "Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.6/dist-packages (from traitlets>=4.2->ipython->watermark) (0.2.0)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.6/dist-packages (from pexpect; sys_platform != \"win32\"->ipython->watermark) (0.5.2)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->watermark) (0.1.7)\n",
            "Installing collected packages: watermark\n",
            "Successfully installed watermark-1.6.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PJ589Jbgy-fk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "outputId": "9aba0a8e-df18-4774-b896-9a017f649c19"
      },
      "source": [
        "%load_ext watermark\n",
        "%watermark -v -m -p tensorflow,numpy -g"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPython 3.6.3\n",
            "IPython 5.5.0\n",
            "\n",
            "tensorflow 1.8.0\n",
            "numpy 1.14.3\n",
            "\n",
            "compiler   : GCC 7.2.0\n",
            "system     : Linux\n",
            "release    : 4.4.111+\n",
            "machine    : x86_64\n",
            "processor  : x86_64\n",
            "CPU cores  : 2\n",
            "interpreter: 64bit\n",
            "Git hash   :\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1KgKXmJ1CatC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "4f7f54b8-23bc-404d-93c7-7bd9d28fc895"
      },
      "source": [
        "# From https://stackoverflow.com/questions/48750199/google-colaboratory-misleading-information-about-its-gpu-only-5-ram-available\n",
        "# memory footprint support libraries/code\n",
        "!ln -sf /opt/bin/nvidia-smi /usr/bin/nvidia-smi\n",
        "!pip install gputil\n",
        "!pip install psutil\n",
        "!pip install humanize\n",
        "\n",
        "import psutil\n",
        "import humanize\n",
        "import os\n",
        "import GPUtil as GPU\n",
        "GPUs = GPU.getGPUs()\n",
        "# XXX: only one GPU on Colab and isn't guaranteed\n",
        "gpu = GPUs[0]\n",
        "\n",
        "def printm():\n",
        "  process = psutil.Process(os.getpid())\n",
        "  print(\"Gen RAM Free: \" + humanize.naturalsize( psutil.virtual_memory().available ), \" I Proc size: \" + humanize.naturalsize( process.memory_info().rss))\n",
        "  print('GPU RAM Free: {0:.0f}MB | Used: {1:.0f}MB | Util {2:3.0f}% | Total {3:.0f}MB'.format(gpu.memoryFree, gpu.memoryUsed, gpu.memoryUtil*100, gpu.memoryTotal))\n",
        "\n",
        "printm()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting gputil\n",
            "  Downloading https://files.pythonhosted.org/packages/45/99/837428d26b47ebd6b66d6e1b180e98ec4a557767a93a81a02ea9d6242611/GPUtil-1.3.0.tar.gz\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from gputil) (1.14.3)\n",
            "Building wheels for collected packages: gputil\n",
            "  Running setup.py bdist_wheel for gputil ... \u001b[?25l-\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/17/0f/04/b79c006972335e35472c0b835ed52bfc0815258d409f560108\n",
            "Successfully built gputil\n",
            "Installing collected packages: gputil\n",
            "Successfully installed gputil-1.3.0\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.6/dist-packages (5.4.5)\n",
            "Collecting humanize\n",
            "  Downloading https://files.pythonhosted.org/packages/8c/e0/e512e4ac6d091fc990bbe13f9e0378f34cf6eecd1c6c268c9e598dcf5bb9/humanize-0.5.1.tar.gz\n",
            "Building wheels for collected packages: humanize\n",
            "  Running setup.py bdist_wheel for humanize ... \u001b[?25l-\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/69/86/6c/f8b8593bc273ec4b0c653d3827f7482bb2001a2781a73b7f44\n",
            "Successfully built humanize\n",
            "Installing collected packages: humanize\n",
            "Successfully installed humanize-0.5.1\n",
            "Gen RAM Free: 12.7 GB  I Proc size: 258.9 MB\n",
            "GPU RAM Free: 11439MB | Used: 0MB | Util   0% | Total 11439MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5zj3MnAMy-fq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 351
        },
        "outputId": "27b5e289-1898-4bc9-b176-abd7cbc49021"
      },
      "source": [
        "from pathlib import Path\n",
        "import random \n",
        "from datetime import datetime\n",
        "\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "# Import MNIST data\n",
        "from tensorflow.examples.tutorials.mnist import input_data\n",
        "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-5a5cf057777d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;31m# Import MNIST data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexamples\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtutorials\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmnist\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0minput_data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m \u001b[0mmnist\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minput_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_data_sets\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"/tmp/data/\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mone_hot\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'tensorflow.examples.tutorials'",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A_x00hIey-fw",
        "colab_type": "text"
      },
      "source": [
        "## Building TCNs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TcFQu3F0y-fy",
        "colab_type": "text"
      },
      "source": [
        "###  Causal Convolution"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-u-dyRIPy-fy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class CausalConv1D(tf.layers.Conv1D):\n",
        "    def __init__(self, filters,\n",
        "               kernel_size,\n",
        "               strides=1,\n",
        "               dilation_rate=1,\n",
        "               activation=None,\n",
        "               use_bias=True,\n",
        "               kernel_initializer=None,\n",
        "               bias_initializer=tf.zeros_initializer(),\n",
        "               kernel_regularizer=None,\n",
        "               bias_regularizer=None,\n",
        "               activity_regularizer=None,\n",
        "               kernel_constraint=None,\n",
        "               bias_constraint=None,\n",
        "               trainable=True,\n",
        "               name=None,\n",
        "               **kwargs):\n",
        "        super(CausalConv1D, self).__init__(\n",
        "            filters=filters,\n",
        "            kernel_size=kernel_size,\n",
        "            strides=strides,\n",
        "            padding='valid',\n",
        "            data_format='channels_last',\n",
        "            dilation_rate=dilation_rate,\n",
        "            activation=activation,\n",
        "            use_bias=use_bias,\n",
        "            kernel_initializer=kernel_initializer,\n",
        "            bias_initializer=bias_initializer,\n",
        "            kernel_regularizer=kernel_regularizer,\n",
        "            bias_regularizer=bias_regularizer,\n",
        "            activity_regularizer=activity_regularizer,\n",
        "            kernel_constraint=kernel_constraint,\n",
        "            bias_constraint=bias_constraint,\n",
        "            trainable=trainable,\n",
        "            name=name, **kwargs\n",
        "        )\n",
        "        \n",
        "    def call(self, inputs):\n",
        "        padding = (self.kernel_size[0] - 1) * self.dilation_rate[0]\n",
        "        if self.data_format == 'channels_first':\n",
        "            inputs = tf.pad(inputs, tf.constant([[0, 0], [0, 0], [padding, 0]], dtype=tf.int32))\n",
        "        else:\n",
        "            inputs = tf.pad(inputs, tf.constant([(0, 0,), (padding, 0), (0, 0)]))\n",
        "        return super(CausalConv1D, self).call(inputs), inputs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4FIx5P1ty-f2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "11417725-57d2-47cf-b4a1-948527867f23"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "with tf.Graph().as_default() as g:\n",
        "    x = tf.random_normal((32, 10, 4)) # (batch_size, length, channel)\n",
        "    with tf.variable_scope(\"tcn\"):\n",
        "        conv = CausalConv1D(8, 3, activation=tf.nn.relu)\n",
        "    output = conv(x)\n",
        "    init = tf.global_variables_initializer()\n",
        "    \n",
        "with tf.Session(graph=g) as sess:\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "    res, inputs = sess.run(output)\n",
        "    print(inputs.shape)\n",
        "    print(inputs[0, :, 0])\n",
        "    print(res.shape)    \n",
        "    print(res[0, :, 0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(32, 12, 4)\n",
            "[ 0.          0.          0.88044167 -1.2032915   0.29814827 -1.1900542\n",
            "  1.0388981  -0.07884882 -1.2475842  -0.48100722 -1.7068204  -1.0657506 ]\n",
            "(32, 10, 8)\n",
            "[0.         0.         1.2657795  1.5949045  0.         0.\n",
            " 0.         0.         0.27693093 0.        ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_CYDJDCGy-f-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "0e700a07-5b49-46f2-a088-6cff89548eb5"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "with tf.Graph().as_default() as g:\n",
        "    x = tf.expand_dims(\n",
        "        tf.expand_dims(tf.constant([1, 0, 0, 1, 0, 0, 1], dtype=tf.float32), axis=0),\n",
        "        axis=-1) # (batch_size, length, channel)\n",
        "    with tf.variable_scope(\"tcn\"):\n",
        "        conv = CausalConv1D(8, 2, dilation_rate=2, activation=None)\n",
        "    output = conv(x)\n",
        "    init = tf.global_variables_initializer()\n",
        "    \n",
        "with tf.Session(graph=g) as sess:\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "    res, inputs = sess.run(output)\n",
        "    print(inputs.shape)\n",
        "    print(inputs[0, :, 0])\n",
        "    print(res.shape)    \n",
        "    print(res[0, :, 0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1, 9, 1)\n",
            "[0. 0. 1. 0. 0. 1. 0. 0. 1.]\n",
            "(1, 7, 8)\n",
            "[0.1447475  0.         0.48786867 0.1447475  0.         0.48786867\n",
            " 0.1447475 ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ByuyggHey-gI",
        "colab_type": "text"
      },
      "source": [
        "###  Spatial Dropout\n",
        "\n",
        "Reference: https://stats.stackexchange.com/questions/282282/how-is-spatial-dropout-in-2d-implemented\n",
        "\n",
        "Actually, simply setting noise_shape in tf.layers.Dropout will do the trick."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YRTsgwSGy-gK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "outputId": "aa51c39b-e40f-4769-c4f7-f42c9bd9862d"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "with tf.Graph().as_default() as g:\n",
        "    x = tf.random_normal((32, 4, 10)) # (batch_size, channel, length)\n",
        "    dropout = tf.layers.Dropout(0.5, noise_shape=[x.shape[0], x.shape[1], tf.constant(1)])\n",
        "    output = dropout(x, training=True)\n",
        "    init = tf.global_variables_initializer()\n",
        "    \n",
        "with tf.Session(graph=g) as sess:\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "    res = sess.run(output)\n",
        "    print(res.shape)   \n",
        "    print(res[0, :, :])\n",
        "    print(res[1, :, :])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(32, 4, 10)\n",
            "[[ 0.          0.         -0.          0.          0.         -0.\n",
            "  -0.         -0.          0.         -0.        ]\n",
            " [-1.3023647  -1.628788    0.20181039  1.554159    5.3209696   1.7473009\n",
            "  -0.41964796 -0.6231473  -2.0326247  -0.21218014]\n",
            " [ 1.1885451  -1.3423481  -1.3000014  -1.132894   -0.20258099 -1.6488353\n",
            "   1.3672652   3.4905746  -0.01186325 -0.8049923 ]\n",
            " [ 0.          0.          0.         -0.         -0.          0.\n",
            "   0.          0.          0.         -0.        ]]\n",
            "[[ 0.8047661  -2.3998349   0.68522704  0.7751469  -0.6081628  -4.7214503\n",
            "  -1.3095977   0.8691299  -2.2773757  -0.2609347 ]\n",
            " [-0.          0.          0.         -0.          0.         -0.\n",
            "   0.         -0.         -0.          0.        ]\n",
            " [ 0.         -0.          0.          0.         -0.          0.\n",
            "   0.         -0.          0.         -0.        ]\n",
            " [ 0.         -0.         -0.          0.          0.         -0.\n",
            "   0.          0.         -0.          0.        ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xdTffJqQy-gU",
        "colab_type": "text"
      },
      "source": [
        "### Temporal blocks\n",
        "\n",
        "Note: `tf.contrib.layers.layer_norm` only supports `channels_last`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9bHYwhRxy-gW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Redefining CausalConv1D to simplify its return values\n",
        "class CausalConv1D(tf.layers.Conv1D):\n",
        "    def __init__(self, filters,\n",
        "               kernel_size,\n",
        "               strides=1,\n",
        "               dilation_rate=1,\n",
        "               activation=None,\n",
        "               use_bias=True,\n",
        "               kernel_initializer=None,\n",
        "               bias_initializer=tf.zeros_initializer(),\n",
        "               kernel_regularizer=None,\n",
        "               bias_regularizer=None,\n",
        "               activity_regularizer=None,\n",
        "               kernel_constraint=None,\n",
        "               bias_constraint=None,\n",
        "               trainable=True,\n",
        "               name=None,\n",
        "               **kwargs):\n",
        "        super(CausalConv1D, self).__init__(\n",
        "            filters=filters,\n",
        "            kernel_size=kernel_size,\n",
        "            strides=strides,\n",
        "            padding='valid',\n",
        "            data_format='channels_last',\n",
        "            dilation_rate=dilation_rate,\n",
        "            activation=activation,\n",
        "            use_bias=use_bias,\n",
        "            kernel_initializer=kernel_initializer,\n",
        "            bias_initializer=bias_initializer,\n",
        "            kernel_regularizer=kernel_regularizer,\n",
        "            bias_regularizer=bias_regularizer,\n",
        "            activity_regularizer=activity_regularizer,\n",
        "            kernel_constraint=kernel_constraint,\n",
        "            bias_constraint=bias_constraint,\n",
        "            trainable=trainable,\n",
        "            name=name, **kwargs\n",
        "        )\n",
        "       \n",
        "    def call(self, inputs):\n",
        "        padding = (self.kernel_size[0] - 1) * self.dilation_rate[0]\n",
        "        inputs = tf.pad(inputs, tf.constant([(0, 0,), (1, 0), (0, 0)]) * padding)\n",
        "        return super(CausalConv1D, self).call(inputs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "re2siwPgy-ga",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TemporalBlock(tf.layers.Layer):\n",
        "    def __init__(self, n_outputs, kernel_size, strides, dilation_rate, dropout=0.2, \n",
        "                 trainable=True, name=None, dtype=None, \n",
        "                 activity_regularizer=None, **kwargs):\n",
        "        super(TemporalBlock, self).__init__(\n",
        "            trainable=trainable, dtype=dtype,\n",
        "            activity_regularizer=activity_regularizer,\n",
        "            name=name, **kwargs\n",
        "        )        \n",
        "        self.dropout = dropout\n",
        "        self.n_outputs = n_outputs\n",
        "        self.conv1 = CausalConv1D(\n",
        "            n_outputs, kernel_size, strides=strides, \n",
        "            dilation_rate=dilation_rate, activation=tf.nn.relu, \n",
        "            name=\"conv1\")\n",
        "        self.conv2 = CausalConv1D(\n",
        "            n_outputs, kernel_size, strides=strides, \n",
        "            dilation_rate=dilation_rate, activation=tf.nn.relu, \n",
        "            name=\"conv2\")\n",
        "        self.down_sample = None\n",
        "\n",
        "    \n",
        "    def build(self, input_shape):\n",
        "        channel_dim = 2\n",
        "        self.dropout1 = tf.layers.Dropout(self.dropout, [tf.constant(1), tf.constant(1), tf.constant(self.n_outputs)])\n",
        "        self.dropout2 = tf.layers.Dropout(self.dropout, [tf.constant(1), tf.constant(1), tf.constant(self.n_outputs)])\n",
        "        if input_shape[channel_dim] != self.n_outputs:\n",
        "            # self.down_sample = tf.layers.Conv1D(\n",
        "            #     self.n_outputs, kernel_size=1, \n",
        "            #     activation=None, data_format=\"channels_last\", padding=\"valid\")\n",
        "            self.down_sample = tf.layers.Dense(self.n_outputs, activation=None)\n",
        "        self.built = True\n",
        "    \n",
        "    def call(self, inputs, training=True):\n",
        "        x = self.conv1(inputs)\n",
        "        x = tf.contrib.layers.layer_norm(x)\n",
        "        x = self.dropout1(x, training=training)\n",
        "        x = self.conv2(x)\n",
        "        x = tf.contrib.layers.layer_norm(x)\n",
        "        x = self.dropout2(x, training=training)\n",
        "        if self.down_sample is not None:\n",
        "            inputs = self.down_sample(inputs)\n",
        "        return tf.nn.relu(x + inputs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-CMZKL1jy-ge",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "9b76c2fb-130a-4255-dbb2-65ad7960ba6a"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "with tf.Graph().as_default() as g:\n",
        "    x = tf.random_normal((32, 10, 4)) # (batch_size, length, channel)\n",
        "    tblock = TemporalBlock(8, 2, 1, 1)\n",
        "    output = tblock(x, training=tf.constant(True))\n",
        "    init = tf.global_variables_initializer()\n",
        "    \n",
        "with tf.Session(graph=g) as sess:\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "    res = sess.run(output)\n",
        "    print(res.shape)   \n",
        "    print(res[0, :, 0])\n",
        "    print(res[1, :, 1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(32, 10, 8)\n",
            "[0.         1.534969   0.         1.4863     2.3868496  0.9489206\n",
            " 0.         0.42759717 0.         0.        ]\n",
            "[0.04622638 0.         3.0465143  0.350043   0.         0.42763186\n",
            " 0.         0.         0.         0.        ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WhzHzZtMy-gk",
        "colab_type": "text"
      },
      "source": [
        "### Temporal convolutional networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GeLztD1Ly-gm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TemporalConvNet(tf.layers.Layer):\n",
        "    def __init__(self, num_channels, kernel_size=2, dropout=0.2,\n",
        "                 trainable=True, name=None, dtype=None, \n",
        "                 activity_regularizer=None, **kwargs):\n",
        "        super(TemporalConvNet, self).__init__(\n",
        "            trainable=trainable, dtype=dtype,\n",
        "            activity_regularizer=activity_regularizer,\n",
        "            name=name, **kwargs\n",
        "        )\n",
        "        self.layers = []\n",
        "        num_levels = len(num_channels)\n",
        "        for i in range(num_levels):\n",
        "            dilation_size = 2 ** i\n",
        "            out_channels = num_channels[i]\n",
        "            self.layers.append(\n",
        "                TemporalBlock(out_channels, kernel_size, strides=1, dilation_rate=dilation_size,\n",
        "                              dropout=dropout, name=\"tblock_{}\".format(i))\n",
        "            )\n",
        "    \n",
        "    def call(self, inputs, training=True):\n",
        "        outputs = inputs\n",
        "        for layer in self.layers:\n",
        "            outputs = layer(outputs, training=training)\n",
        "        return outputs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cRTtl0mey-go",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "c8ca7755-35ea-4a41-9ef7-5770643f7b8b"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "with tf.Graph().as_default() as g:\n",
        "    x = tf.random_normal((32, 10, 4)) # (batch_size, length, channel)\n",
        "    tcn = TemporalConvNet([8, 8, 8, 8], 2, 0.25)\n",
        "    output = tcn(x, training=tf.constant(True))\n",
        "    init = tf.global_variables_initializer()\n",
        "    \n",
        "with tf.Session(graph=g) as sess:\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "    res = sess.run(output)\n",
        "    print(res.shape)   \n",
        "    print(res[0, :, 0])\n",
        "    print(res[1, :, 1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(32, 10, 8)\n",
            "[0.6988858  0.         0.7061405  0.15213768 0.93309164 0.\n",
            " 1.3751144  2.6659122  1.8665429  0.        ]\n",
            "[0.         0.         4.748131   0.24300182 0.         0.\n",
            " 0.         0.         1.4196995  0.8674514 ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zCMhWfWjy-g0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "121c2c0f-7066-4f57-fe2c-9d29a8606c6b"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "g = tf.Graph()\n",
        "with g.as_default():\n",
        "    Xinput = tf.placeholder(tf.float32, shape=[None, 10, 4])\n",
        "    tcn = TemporalConvNet([8, 8, 8, 8], 2, 0.25)\n",
        "    output = tcn(Xinput, training=tf.constant(True))\n",
        "    print(tcn.layers[0].down_sample)    \n",
        "    init = tf.global_variables_initializer()\n",
        "    \n",
        "with tf.Session(graph=g) as sess:\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "    res = sess.run(output, {Xinput: np.random.randn(32, 10, 4)})\n",
        "    print(res.shape)   \n",
        "    print(res[0, :, 0])\n",
        "    print(res[1, :, 1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<tensorflow.python.layers.core.Dense object at 0x7fddee04f940>\n",
            "(32, 10, 8)\n",
            "[0.        0.        2.1618829 0.        3.862938  2.824544  4.999326\n",
            " 1.7338551 6.2948    1.4054765]\n",
            "[0.        0.        0.        0.        0.        0.        0.\n",
            " 7.947054  5.4993486 0.       ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5jYugVyby-g-",
        "colab_type": "text"
      },
      "source": [
        "## Sequential MNIST"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "41qAk9lAy-hC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e2f114c1-f9b7-401d-99be-f240426bed49"
      },
      "source": [
        "# Training Parameters\n",
        "learning_rate = 0.001\n",
        "batch_size = 64\n",
        "display_step = 500\n",
        "total_batch = int(mnist.train.num_examples / batch_size)\n",
        "print(\"Number of batches per epoch:\", total_batch)\n",
        "training_steps = 3000\n",
        "\n",
        "# Network Parameters\n",
        "num_input = 1 # MNIST data input (img shape: 28*28)\n",
        "timesteps = 28 * 28 # timesteps\n",
        "num_classes = 10 # MNIST total classes (0-9 digits)\n",
        "dropout = 0.1\n",
        "kernel_size = 8\n",
        "levels = 6\n",
        "nhid = 20 # hidden layer num of features"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of batches per epoch: 859\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bP37UtN5y-hG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "089539a5-b6c1-4cb0-ca1c-deacde5b3cbe"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "graph = tf.Graph()\n",
        "with graph.as_default():\n",
        "    tf.set_random_seed(10)\n",
        "    # tf Graph input\n",
        "    X = tf.placeholder(\"float\", [None, timesteps, num_input])\n",
        "    Y = tf.placeholder(\"float\", [None, num_classes])\n",
        "    is_training = tf.placeholder(\"bool\")\n",
        "    \n",
        "    # Define weights\n",
        "    logits = tf.layers.dense(\n",
        "        TemporalConvNet([nhid] * levels, kernel_size, dropout)(\n",
        "            X, training=is_training)[:, -1, :],\n",
        "        num_classes, activation=None, \n",
        "        kernel_initializer=tf.orthogonal_initializer()\n",
        "    )\n",
        "    prediction = tf.nn.softmax(logits)\n",
        "   \n",
        "    # Define loss and optimizer\n",
        "    loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(\n",
        "        logits=logits, labels=Y))\n",
        "    \n",
        "    with tf.name_scope(\"optimizer\"):\n",
        "        # optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
        "        optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
        "        # gvs = optimizer.compute_gradients(loss_op)\n",
        "        # for grad, var in gvs:\n",
        "        #     if grad is None:\n",
        "        #         print(var)\n",
        "        # capped_gvs = [(tf.clip_by_value(grad, -.5, .5), var) for grad, var in gvs]\n",
        "        # train_op = optimizer.apply_gradients(capped_gvs)    \n",
        "        train_op = optimizer.minimize(loss_op)\n",
        "\n",
        "    # Evaluate model (with test logits, for dropout to be disabled)\n",
        "    correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
        "    accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
        "\n",
        "    # Initialize the variables (i.e. assign their default value)\n",
        "    init = tf.global_variables_initializer()\n",
        "    saver = tf.train.Saver()\n",
        "    print(\"All parameters:\", np.sum([np.product([xi.value for xi in x.get_shape()]) for x in tf.global_variables()]))\n",
        "    print(\"Trainable parameters:\", np.sum([np.product([xi.value for xi in x.get_shape()]) for x in tf.trainable_variables()]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "All parameters: 108992.0\n",
            "Trainable parameters: 36330\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IjwOnIUmy-hM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 278
        },
        "outputId": "79b8a85e-9f56-458d-ecc9-3eea13094548"
      },
      "source": [
        "# Start training\n",
        "log_dir = \"logs/tcn/%s\" % datetime.now().strftime(\"%Y%m%d_%H%M\")\n",
        "Path(log_dir).mkdir(exist_ok=True, parents=True)\n",
        "tb_writer = tf.summary.FileWriter(log_dir, graph)\n",
        "config = tf.ConfigProto()\n",
        "config.gpu_options.allow_growth = False\n",
        "best_val_acc = 0.8\n",
        "with tf.Session(graph=graph, config=config) as sess:\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "    for step in range(1, training_steps+1):\n",
        "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
        "        # print(np.max(batch_x), np.mean(batch_x), np.median(batch_x))\n",
        "        # Reshape data to get 28 * 28 seq of 1 elements\n",
        "        batch_x = batch_x.reshape((batch_size, timesteps, num_input))\n",
        "        # Run optimization op (backprop)\n",
        "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y, is_training: True})\n",
        "        if step % display_step == 0 or step == 1:\n",
        "            # Calculate batch loss and accuracy\n",
        "            loss, acc = sess.run([loss_op, accuracy], feed_dict={\n",
        "                X: batch_x, Y: batch_y, is_training: False})\n",
        "            # Calculate accuracy for 128 mnist test images\n",
        "            test_len = 128\n",
        "            test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))\n",
        "            test_label = mnist.test.labels[:test_len]\n",
        "            val_acc = sess.run(accuracy, feed_dict={X: test_data, Y: test_label, is_training: False})\n",
        "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
        "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
        "                  \"{:.3f}\".format(acc) + \", Test Accuracy= \" + \\\n",
        "                  \"{:.3f}\".format(val_acc))\n",
        "            if val_acc > best_val_acc:\n",
        "                best_val_acc = val_acc\n",
        "                save_path = saver.save(sess, \"/tmp/model.ckpt\")\n",
        "                print(\"Model saved in path: %s\" % save_path)\n",
        "    print(\"Optimization Finished!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Step 1, Minibatch Loss= 3.6770, Training Accuracy= 0.109, Test Accuracy= 0.156\n",
            "Step 500, Minibatch Loss= 0.1022, Training Accuracy= 0.953, Test Accuracy= 0.945\n",
            "Model saved in path: /tmp/model.ckpt\n",
            "Step 1000, Minibatch Loss= 0.2515, Training Accuracy= 0.922, Test Accuracy= 0.992\n",
            "Model saved in path: /tmp/model.ckpt\n",
            "Step 1500, Minibatch Loss= 0.0310, Training Accuracy= 0.984, Test Accuracy= 0.992\n",
            "Step 2000, Minibatch Loss= 0.1406, Training Accuracy= 0.953, Test Accuracy= 0.984\n",
            "Step 2500, Minibatch Loss= 0.0131, Training Accuracy= 1.000, Test Accuracy= 0.984\n",
            "Step 3000, Minibatch Loss= 0.0228, Training Accuracy= 1.000, Test Accuracy= 0.984\n",
            "Step 3500, Minibatch Loss= 0.1202, Training Accuracy= 0.969, Test Accuracy= 1.000\n",
            "Model saved in path: /tmp/model.ckpt\n",
            "Step 4000, Minibatch Loss= 0.0847, Training Accuracy= 0.984, Test Accuracy= 1.000\n",
            "Step 4500, Minibatch Loss= 0.0906, Training Accuracy= 0.953, Test Accuracy= 0.992\n",
            "Step 5000, Minibatch Loss= 0.0346, Training Accuracy= 0.984, Test Accuracy= 1.000\n",
            "Optimization Finished!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91elDQ4Xy-hS",
        "colab_type": "text"
      },
      "source": [
        "## Permuted"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KMtyE5m-97FK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "training_steps = 5000"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ri6zKrm9Ie30",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eWX3IRfR0z5O",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "800eba07-4ffc-492d-d5fc-7906308a9f51"
      },
      "source": [
        "tf.reset_default_graph()\n",
        "graph = tf.Graph()\n",
        "with graph.as_default():\n",
        "    tf.set_random_seed(10)\n",
        "    # tf Graph input\n",
        "    X = tf.placeholder(\"float\", [None, timesteps, num_input])\n",
        "    Y = tf.placeholder(\"float\", [None, num_classes])\n",
        "    is_training = tf.placeholder(\"bool\")\n",
        "    \n",
        "    # Permute the time step\n",
        "    np.random.seed(100)\n",
        "    permute = np.random.permutation(784)\n",
        "    X_shuffled = tf.gather(X, permute, axis=1)\n",
        "    \n",
        "    # Define weights\n",
        "    logits = tf.layers.dense(\n",
        "        TemporalConvNet([nhid] * levels, kernel_size, dropout)(\n",
        "            X_shuffled, training=is_training)[:, -1, :],\n",
        "        num_classes, activation=None, \n",
        "        kernel_initializer=tf.orthogonal_initializer()\n",
        "    )\n",
        "    prediction = tf.nn.softmax(logits)\n",
        "   \n",
        "    # Define loss and optimizer\n",
        "    loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(\n",
        "        logits=logits, labels=Y))\n",
        "    \n",
        "    with tf.name_scope(\"optimizer\"):\n",
        "        # optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
        "        optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
        "        # gvs = optimizer.compute_gradients(loss_op)\n",
        "        # for grad, var in gvs:\n",
        "        #     if grad is None:\n",
        "        #         print(var)\n",
        "        # capped_gvs = [(tf.clip_by_value(grad, -.5, .5), var) for grad, var in gvs]\n",
        "        # train_op = optimizer.apply_gradients(capped_gvs)    \n",
        "        train_op = optimizer.minimize(loss_op)\n",
        "\n",
        "    # Evaluate model (with test logits, for dropout to be disabled)\n",
        "    correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
        "    accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
        "\n",
        "    # Initialize the variables (i.e. assign their default value)\n",
        "    init = tf.global_variables_initializer()\n",
        "    saver = tf.train.Saver()\n",
        "    print(\"All parameters:\", np.sum([np.product([xi.value for xi in x.get_shape()]) for x in tf.global_variables()]))\n",
        "    print(\"Trainable parameters:\", np.sum([np.product([xi.value for xi in x.get_shape()]) for x in tf.trainable_variables()]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "All parameters: 108992.0\n",
            "Trainable parameters: 36330\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oprHW-Qry-ha",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        },
        "outputId": "7f544e34-7cb8-44fb-ff2e-8ad3ee669d6f"
      },
      "source": [
        "# Start training\n",
        "log_dir = \"logs/tcn/%s\" % datetime.now().strftime(\"%Y%m%d_%H%M\")\n",
        "Path(log_dir).mkdir(exist_ok=True, parents=True)\n",
        "tb_writer = tf.summary.FileWriter(log_dir, graph)\n",
        "config = tf.ConfigProto()\n",
        "config.gpu_options.allow_growth = True\n",
        "best_val_acc = 0.8\n",
        "with tf.Session(graph=graph, config=config) as sess:\n",
        "    # Run the initializer\n",
        "    sess.run(init)\n",
        "    for step in range(1, training_steps+1):\n",
        "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
        "        # print(np.max(batch_x), np.mean(batch_x), np.median(batch_x))\n",
        "        # Reshape data to get 28 * 28 seq of 1 elements\n",
        "        batch_x = batch_x.reshape((batch_size, timesteps, num_input))\n",
        "        # Run optimization op (backprop)\n",
        "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y, is_training: True})\n",
        "        if step % display_step == 0 or step == 1:\n",
        "            # Calculate batch loss and accuracy\n",
        "            loss, acc = sess.run([loss_op, accuracy], feed_dict={\n",
        "                X: batch_x, Y: batch_y, is_training: False})\n",
        "            # Calculate accuracy for 128 mnist test images\n",
        "            test_len = 128\n",
        "            test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))\n",
        "            test_label = mnist.test.labels[:test_len]\n",
        "            val_acc = sess.run(accuracy, feed_dict={X: test_data, Y: test_label, is_training: False})\n",
        "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
        "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
        "                  \"{:.3f}\".format(acc) + \", Test Accuracy= \" + \\\n",
        "                  \"{:.3f}\".format(val_acc))\n",
        "            if val_acc > best_val_acc:\n",
        "                best_val_acc = val_acc\n",
        "                save_path = saver.save(sess, \"/tmp/model.ckpt\")\n",
        "                print(\"Model saved in path: %s\" % save_path)\n",
        "    print(\"Optimization Finished!\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Step 1, Minibatch Loss= 3.7196, Training Accuracy= 0.125, Test Accuracy= 0.062\n",
            "Step 500, Minibatch Loss= 0.3547, Training Accuracy= 0.906, Test Accuracy= 0.906\n",
            "Model saved in path: /tmp/model.ckpt\n",
            "Step 1000, Minibatch Loss= 0.2223, Training Accuracy= 0.922, Test Accuracy= 0.945\n",
            "Model saved in path: /tmp/model.ckpt\n",
            "Step 1500, Minibatch Loss= 0.4307, Training Accuracy= 0.906, Test Accuracy= 0.961\n",
            "Model saved in path: /tmp/model.ckpt\n",
            "Step 2000, Minibatch Loss= 0.1025, Training Accuracy= 0.938, Test Accuracy= 0.977\n",
            "Model saved in path: /tmp/model.ckpt\n",
            "Step 2500, Minibatch Loss= 0.2563, Training Accuracy= 0.891, Test Accuracy= 0.961\n",
            "Step 3000, Minibatch Loss= 0.1184, Training Accuracy= 0.969, Test Accuracy= 0.969\n",
            "Step 3500, Minibatch Loss= 0.1279, Training Accuracy= 0.953, Test Accuracy= 0.969\n",
            "Step 4000, Minibatch Loss= 0.0419, Training Accuracy= 0.984, Test Accuracy= 0.984\n",
            "Model saved in path: /tmp/model.ckpt\n",
            "Step 4500, Minibatch Loss= 0.3604, Training Accuracy= 0.938, Test Accuracy= 0.969\n",
            "Step 5000, Minibatch Loss= 0.0682, Training Accuracy= 0.984, Test Accuracy= 0.977\n",
            "Optimization Finished!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "chQz9B84y-he",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}