tensorflow_tutorial_5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train and visualize a model in Tensorflow - Part 5: Using Tensorboard\n",
    "\n",
    "Neural networks have been widely critized because of the lack of interpretation of their internal parameters. In this notebook we will present some techniques visualize the model structure and its behaviour during training.\n",
    "\n",
    "The lack of interpretability leads, among other thigns, to make neural models error prone. While this is true, we still have some tools to try to debug our network and to understand what the model is doing.\n",
    "\n",
    "Tensorflow comes with its own (and very complete) visualization tool: Tensorboard. In the rest of this tutorial, we will explain how to use Tensorboard to inpect the progress of the network while training, check the execution graph and compare different models.\n",
    "\n",
    "As first step, we use the same code as in previous notebooks to load the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the dataset into a numpy keyed structure\n",
    "newsgroups = np.load('./resources/newsgroup.npz')\n",
    "\n",
    "# Define the batch size\n",
    "batch_size = 100\n",
    "\n",
    "def dataset_input_fn(dataset):\n",
    "    \"\"\"\n",
    "    Creates an input function using the `numpy_input_fn` method from\n",
    "    tensorflow, based on the dataset we want to use.\n",
    "    \n",
    "    Args:\n",
    "        dataset: String that represents the dataset\n",
    "        (should be `train` or `test`)\n",
    "    \n",
    "    Returns:\n",
    "        An `numpy_input_fn` function to feed to an estimator\n",
    "    \"\"\"\n",
    "    assert dataset in ('train', 'test'),\\\n",
    "        \"The selected dataset should be `train` or `test`\"\n",
    "    \n",
    "    return tf.estimator.inputs.numpy_input_fn(\n",
    "        # A dictionary of numpy arrays that match each array with the\n",
    "        # corresponding column in the model. For this case we only\n",
    "        # have \"one\" colum which represents the whole array.\n",
    "        x={'input_data': newsgroups['%s_data' % dataset]},\n",
    "        # The target array\n",
    "        y=newsgroups['%s_target' % dataset],\n",
    "        # The batch size to iterate the data in small fractions\n",
    "        batch_size=batch_size,\n",
    "        # If the dataset is `test` only run once\n",
    "        num_epochs=1 if dataset == 'test' else None,\n",
    "        # Only shuffle the dataset for the `train` data\n",
    "        shuffle=dataset == 'train'\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_size = newsgroups['train_data'].shape[1]\n",
    "num_classes = newsgroups['labels'].shape[0]\n",
    "\n",
    "feature_columns = [tf.feature_column.numeric_column(\n",
    "    'input_data', shape=(input_size,))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the execution graph\n",
    "\n",
    "For the first experiment, we define the same model as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_task_type': 'worker', '_session_config': None, '_num_worker_replicas': 1, '_tf_random_seed': None, '_service': None, '_save_checkpoints_steps': None, '_is_chief': True, '_num_ps_replicas': 0, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f939ba74358>, '_keep_checkpoint_max': 5, '_model_dir': 'models/bigDNN', '_save_summary_steps': 100, '_log_step_count_steps': 100, '_master': '', '_keep_checkpoint_every_n_hours': 10000, '_save_checkpoints_secs': 600}\n"
     ]
    }
   ],
   "source": [
    "big_model = tf.estimator.DNNClassifier(\n",
    "    feature_columns=feature_columns,\n",
    "    hidden_units=(5000, 2000,),\n",
    "    n_classes=num_classes,\n",
    "    model_dir=\"models/bigDNN\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating the DNNClassifier object, we can already lunch tensoboard and explore the graph. To start Tensorboard, in your terminal go to the folder where you downloaded the tutorial and type:\n",
    "\n",
    "```$ tensorboard --dirname models```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize learning\n",
    "\n",
    "Now, we would like to inspect the performance of the classifier while it is training. In the contrib.learn module of tensorflow we found an Experiment class that will run the train and evaluation cycle for us. Even if this not hard to implement using a for loop, we recommend to use the functions provided by tensorflow as they support training on multiple servers.\n",
    "\n",
    "In the following cell, we indicate that we want to train our classifier over 2000 epocs, and evaluate its performance in the test dataset every 100 iterations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Experiment.continuous_train_and_eval (from tensorflow.contrib.learn.python.learn.experiment) is experimental and may change or be removed at any time, and without warning.\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1, loss = 299.495\n",
      "INFO:tensorflow:Saving checkpoints for 100 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 72.389.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:09:06\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-100\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:09:09\n",
      "INFO:tensorflow:Saving dict for global step 100: accuracy = 0.754912, average_loss = 0.730839, global_step = 100, loss = 72.43\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-100\n",
      "INFO:tensorflow:Saving checkpoints for 101 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 101, loss = 49.0593\n",
      "INFO:tensorflow:Saving checkpoints for 200 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 43.0473.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:09:38\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-200\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:09:42\n",
      "INFO:tensorflow:Saving dict for global step 200: accuracy = 0.868295, average_loss = 0.446303, global_step = 200, loss = 44.231\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-200\n",
      "INFO:tensorflow:Saving checkpoints for 201 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 201, loss = 19.6646\n",
      "INFO:tensorflow:Saving checkpoints for 300 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 13.404.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:10:12\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-300\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:10:15\n",
      "INFO:tensorflow:Saving dict for global step 300: accuracy = 0.874934, average_loss = 0.442255, global_step = 300, loss = 43.8298\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-300\n",
      "INFO:tensorflow:Saving checkpoints for 301 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 301, loss = 0.90614\n",
      "INFO:tensorflow:Saving checkpoints for 400 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 1.82181.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:10:44\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-400\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:10:47\n",
      "INFO:tensorflow:Saving dict for global step 400: accuracy = 0.886883, average_loss = 0.378348, global_step = 400, loss = 37.4962\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-400\n",
      "INFO:tensorflow:Saving checkpoints for 401 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 401, loss = 2.07265\n",
      "INFO:tensorflow:Saving checkpoints for 500 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 4.2747.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:11:17\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-500\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:11:21\n",
      "INFO:tensorflow:Saving dict for global step 500: accuracy = 0.883962, average_loss = 0.406269, global_step = 500, loss = 40.2634\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-500\n",
      "INFO:tensorflow:Saving checkpoints for 501 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 501, loss = 3.57495\n",
      "INFO:tensorflow:Saving checkpoints for 600 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.265194.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:11:50\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-600\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:11:53\n",
      "INFO:tensorflow:Saving dict for global step 600: accuracy = 0.892193, average_loss = 0.399234, global_step = 600, loss = 39.5662\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-600\n",
      "INFO:tensorflow:Saving checkpoints for 601 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 601, loss = 0.314049\n",
      "INFO:tensorflow:Saving checkpoints for 700 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.193443.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:12:22\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-700\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:12:25\n",
      "INFO:tensorflow:Saving dict for global step 700: accuracy = 0.891131, average_loss = 0.410332, global_step = 700, loss = 40.666\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-700\n",
      "INFO:tensorflow:Saving checkpoints for 701 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 701, loss = 0.133136\n",
      "INFO:tensorflow:Saving checkpoints for 800 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.145754.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:12:54\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-800\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:12:57\n",
      "INFO:tensorflow:Saving dict for global step 800: accuracy = 0.893521, average_loss = 0.4148, global_step = 800, loss = 41.1088\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-800\n",
      "INFO:tensorflow:Saving checkpoints for 801 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 801, loss = 0.519953\n",
      "INFO:tensorflow:Saving checkpoints for 900 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.111203.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:13:26\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-900\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:13:29\n",
      "INFO:tensorflow:Saving dict for global step 900: accuracy = 0.892193, average_loss = 0.417853, global_step = 900, loss = 41.4114\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-900\n",
      "INFO:tensorflow:Saving checkpoints for 901 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 901, loss = 0.0925495\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.119844.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:13:58\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1000\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:14:01\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.890866, average_loss = 0.431849, global_step = 1000, loss = 42.7985\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1000\n",
      "INFO:tensorflow:Saving checkpoints for 1001 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1001, loss = 0.121873\n",
      "INFO:tensorflow:Saving checkpoints for 1100 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.170773.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:14:30\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1100\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:14:34\n",
      "INFO:tensorflow:Saving dict for global step 1100: accuracy = 0.889804, average_loss = 0.441736, global_step = 1100, loss = 43.7784\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1100\n",
      "INFO:tensorflow:Saving checkpoints for 1101 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1101, loss = 0.0519304\n",
      "INFO:tensorflow:Saving checkpoints for 1200 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.100178.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:15:02\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1200\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:15:06\n",
      "INFO:tensorflow:Saving dict for global step 1200: accuracy = 0.891397, average_loss = 0.445988, global_step = 1200, loss = 44.1998\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1200\n",
      "INFO:tensorflow:Saving checkpoints for 1201 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1201, loss = 0.0631971\n",
      "INFO:tensorflow:Saving checkpoints for 1300 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.188712.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:15:35\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1300\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:15:39\n",
      "INFO:tensorflow:Saving dict for global step 1300: accuracy = 0.894318, average_loss = 0.441023, global_step = 1300, loss = 43.7077\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1300\n",
      "INFO:tensorflow:Saving checkpoints for 1301 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1301, loss = 0.0687607\n",
      "INFO:tensorflow:Saving checkpoints for 1400 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0517873.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:16:09\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1400\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:16:13\n",
      "INFO:tensorflow:Saving dict for global step 1400: accuracy = 0.892459, average_loss = 0.43528, global_step = 1400, loss = 43.1385\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1400\n",
      "INFO:tensorflow:Saving checkpoints for 1401 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1401, loss = 0.128617\n",
      "INFO:tensorflow:Saving checkpoints for 1500 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0401762.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:16:41\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1500\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:16:45\n",
      "INFO:tensorflow:Saving dict for global step 1500: accuracy = 0.894583, average_loss = 0.43879, global_step = 1500, loss = 43.4864\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1500\n",
      "INFO:tensorflow:Saving checkpoints for 1501 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1501, loss = 0.0549837\n",
      "INFO:tensorflow:Saving checkpoints for 1600 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 2.65912.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:17:13\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1600\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:17:17\n",
      "INFO:tensorflow:Saving dict for global step 1600: accuracy = 0.893786, average_loss = 0.441514, global_step = 1600, loss = 43.7563\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1600\n",
      "INFO:tensorflow:Saving checkpoints for 1601 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1601, loss = 0.0546657\n",
      "INFO:tensorflow:Saving checkpoints for 1700 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 3.18227.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:17:45\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1700\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:17:49\n",
      "INFO:tensorflow:Saving dict for global step 1700: accuracy = 0.890335, average_loss = 0.457028, global_step = 1700, loss = 45.2939\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1700\n",
      "INFO:tensorflow:Saving checkpoints for 1701 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1701, loss = 0.0424062\n",
      "INFO:tensorflow:Saving checkpoints for 1800 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0429752.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:18:18\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1800\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:18:22\n",
      "INFO:tensorflow:Saving dict for global step 1800: accuracy = 0.891397, average_loss = 0.455336, global_step = 1800, loss = 45.1262\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1800\n",
      "INFO:tensorflow:Saving checkpoints for 1801 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1801, loss = 0.237482\n",
      "INFO:tensorflow:Saving checkpoints for 1900 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0232902.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:18:51\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1900\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:18:55\n",
      "INFO:tensorflow:Saving dict for global step 1900: accuracy = 0.893521, average_loss = 0.450725, global_step = 1900, loss = 44.6692\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-1900\n",
      "INFO:tensorflow:Saving checkpoints for 1901 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1901, loss = 0.050638\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into models/bigDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0470318.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:19:24\n",
      "INFO:tensorflow:Restoring parameters from models/bigDNN/model.ckpt-2000\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:19:27\n",
      "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.893521, average_loss = 0.447487, global_step = 2000, loss = 44.3483\n",
      "INFO:tensorflow:Stop training model as max steps reached\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "({'accuracy': 0.89352095,\n",
       "  'average_loss': 0.44748718,\n",
       "  'global_step': 2000,\n",
       "  'loss': 44.348335},\n",
       " [])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_experiment = tf.contrib.learn.Experiment(\n",
    "    big_model,\n",
    "    train_input_fn=dataset_input_fn(\"train\"),\n",
    "    eval_input_fn=dataset_input_fn(\"test\"),\n",
    "    train_steps=2000,\n",
    "    train_steps_per_iteration=100\n",
    ")\n",
    "big_experiment.continuous_train_and_eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The DNNClassifier will automatically record the accuracy of the model, along with the loss. We can see the progress of these metrics in Tensorboard while the model is training, and eventually stop the training if the model starts to overfit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing experiments\n",
    "\n",
    "Now, what if we want to compare the performance with another model?\n",
    "\n",
    "We can define a smaller model and see the same metrics as before in the same graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_task_type': 'worker', '_session_config': None, '_num_worker_replicas': 1, '_tf_random_seed': None, '_service': None, '_save_checkpoints_steps': None, '_is_chief': True, '_num_ps_replicas': 0, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f939b1a0080>, '_keep_checkpoint_max': 5, '_model_dir': 'models/littleDNN', '_save_summary_steps': 100, '_log_step_count_steps': 100, '_master': '', '_keep_checkpoint_every_n_hours': 10000, '_save_checkpoints_secs': 600}\n"
     ]
    }
   ],
   "source": [
    "little_model = tf.estimator.DNNClassifier(\n",
    "    feature_columns=feature_columns,\n",
    "    hidden_units=(500, 100,),\n",
    "    n_classes=num_classes,\n",
    "    model_dir=\"models/littleDNN\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Experiment.continuous_train_and_eval (from tensorflow.contrib.learn.python.learn.experiment) is experimental and may change or be removed at any time, and without warning.\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1, loss = 299.566\n",
      "INFO:tensorflow:Saving checkpoints for 100 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 56.1963.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:04\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-100\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:05\n",
      "INFO:tensorflow:Saving dict for global step 100: accuracy = 0.845459, average_loss = 0.519421, global_step = 100, loss = 51.4773\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-100\n",
      "INFO:tensorflow:Saving checkpoints for 101 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 101, loss = 54.9002\n",
      "INFO:tensorflow:Saving checkpoints for 200 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 17.1108.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:10\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-200\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:10\n",
      "INFO:tensorflow:Saving dict for global step 200: accuracy = 0.889538, average_loss = 0.36914, global_step = 200, loss = 36.5837\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-200\n",
      "INFO:tensorflow:Saving checkpoints for 201 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 201, loss = 5.11427\n",
      "INFO:tensorflow:Saving checkpoints for 300 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 10.3941.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:15\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-300\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:16\n",
      "INFO:tensorflow:Saving dict for global step 300: accuracy = 0.891397, average_loss = 0.374263, global_step = 300, loss = 37.0915\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-300\n",
      "INFO:tensorflow:Saving checkpoints for 301 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 301, loss = 7.42572\n",
      "INFO:tensorflow:Saving checkpoints for 400 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.673701.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:21\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-400\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:21\n",
      "INFO:tensorflow:Saving dict for global step 400: accuracy = 0.894583, average_loss = 0.364217, global_step = 400, loss = 36.0958\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-400\n",
      "INFO:tensorflow:Saving checkpoints for 401 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 401, loss = 2.25545\n",
      "INFO:tensorflow:Saving checkpoints for 500 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 1.00817.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:26\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-500\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:27\n",
      "INFO:tensorflow:Saving dict for global step 500: accuracy = 0.894849, average_loss = 0.369481, global_step = 500, loss = 36.6175\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-500\n",
      "INFO:tensorflow:Saving checkpoints for 501 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 501, loss = 5.22804\n",
      "INFO:tensorflow:Saving checkpoints for 600 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 1.01435.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:31\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-600\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:32\n",
      "INFO:tensorflow:Saving dict for global step 600: accuracy = 0.899097, average_loss = 0.365666, global_step = 600, loss = 36.2394\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-600\n",
      "INFO:tensorflow:Saving checkpoints for 601 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 601, loss = 0.458158\n",
      "INFO:tensorflow:Saving checkpoints for 700 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.632507.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:37\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-700\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:38\n",
      "INFO:tensorflow:Saving dict for global step 700: accuracy = 0.896176, average_loss = 0.379473, global_step = 700, loss = 37.6077\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-700\n",
      "INFO:tensorflow:Saving checkpoints for 701 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 701, loss = 0.255389\n",
      "INFO:tensorflow:Saving checkpoints for 800 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.429725.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:42\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-800\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:43\n",
      "INFO:tensorflow:Saving dict for global step 800: accuracy = 0.89538, average_loss = 0.374744, global_step = 800, loss = 37.1391\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-800\n",
      "INFO:tensorflow:Saving checkpoints for 801 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 801, loss = 0.283583\n",
      "INFO:tensorflow:Saving checkpoints for 900 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.278232.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:48\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-900\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:48\n",
      "INFO:tensorflow:Saving dict for global step 900: accuracy = 0.896176, average_loss = 0.391692, global_step = 900, loss = 38.8187\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-900\n",
      "INFO:tensorflow:Saving checkpoints for 901 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 901, loss = 0.452167\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 4.05153.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:53\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1000\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:54\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.898035, average_loss = 0.383275, global_step = 1000, loss = 37.9845\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1000\n",
      "INFO:tensorflow:Saving checkpoints for 1001 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1001, loss = 0.269749\n",
      "INFO:tensorflow:Saving checkpoints for 1100 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.219075.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:27:58\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1100\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:27:59\n",
      "INFO:tensorflow:Saving dict for global step 1100: accuracy = 0.898566, average_loss = 0.384257, global_step = 1100, loss = 38.0819\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1100\n",
      "INFO:tensorflow:Saving checkpoints for 1101 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1101, loss = 0.362429\n",
      "INFO:tensorflow:Saving checkpoints for 1200 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.24257.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:04\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1200\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:05\n",
      "INFO:tensorflow:Saving dict for global step 1200: accuracy = 0.898035, average_loss = 0.386132, global_step = 1200, loss = 38.2678\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1200\n",
      "INFO:tensorflow:Saving checkpoints for 1201 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1201, loss = 0.163779\n",
      "INFO:tensorflow:Saving checkpoints for 1300 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.117459.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:09\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1300\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:10\n",
      "INFO:tensorflow:Saving dict for global step 1300: accuracy = 0.898566, average_loss = 0.387477, global_step = 1300, loss = 38.401\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1300\n",
      "INFO:tensorflow:Saving checkpoints for 1301 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1301, loss = 0.414389\n",
      "INFO:tensorflow:Saving checkpoints for 1400 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.938866.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:15\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1400\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:15\n",
      "INFO:tensorflow:Saving dict for global step 1400: accuracy = 0.900956, average_loss = 0.391698, global_step = 1400, loss = 38.8194\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1400\n",
      "INFO:tensorflow:Saving checkpoints for 1401 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1401, loss = 0.102871\n",
      "INFO:tensorflow:Saving checkpoints for 1500 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.266778.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:20\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1500\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:21\n",
      "INFO:tensorflow:Saving dict for global step 1500: accuracy = 0.899097, average_loss = 0.397177, global_step = 1500, loss = 39.3624\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1500\n",
      "INFO:tensorflow:Saving checkpoints for 1501 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1501, loss = 0.109481\n",
      "INFO:tensorflow:Saving checkpoints for 1600 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.118837.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:26\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1600\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:26\n",
      "INFO:tensorflow:Saving dict for global step 1600: accuracy = 0.89538, average_loss = 0.417409, global_step = 1600, loss = 41.3674\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1600\n",
      "INFO:tensorflow:Saving checkpoints for 1601 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1601, loss = 0.196953\n",
      "INFO:tensorflow:Saving checkpoints for 1700 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.116243.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:31\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1700\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:32\n",
      "INFO:tensorflow:Saving dict for global step 1700: accuracy = 0.898301, average_loss = 0.403951, global_step = 1700, loss = 40.0337\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1700\n",
      "INFO:tensorflow:Saving checkpoints for 1701 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1701, loss = 0.0788048\n",
      "INFO:tensorflow:Saving checkpoints for 1800 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.145794.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:37\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1800\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:37\n",
      "INFO:tensorflow:Saving dict for global step 1800: accuracy = 0.899363, average_loss = 0.399621, global_step = 1800, loss = 39.6045\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1800\n",
      "INFO:tensorflow:Saving checkpoints for 1801 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1801, loss = 0.103939\n",
      "INFO:tensorflow:Saving checkpoints for 1900 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.109243.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:42\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1900\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:43\n",
      "INFO:tensorflow:Saving dict for global step 1900: accuracy = 0.898035, average_loss = 0.406908, global_step = 1900, loss = 40.3268\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-1900\n",
      "INFO:tensorflow:Saving checkpoints for 1901 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:step = 1901, loss = 0.124407\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into models/littleDNN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0824199.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-16-21:28:48\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from models/littleDNN/model.ckpt-2000\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-16-21:28:48\n",
      "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.896973, average_loss = 0.4181, global_step = 2000, loss = 41.436\n",
      "INFO:tensorflow:Stop training model as max steps reached\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "({'accuracy': 0.89697289,\n",
       "  'average_loss': 0.41810048,\n",
       "  'global_step': 2000,\n",
       "  'loss': 41.435959},\n",
       " [])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "little_experiment = tf.contrib.learn.Experiment(\n",
    "    little_model,\n",
    "    train_input_fn=dataset_input_fn(\"train\"),\n",
    "    eval_input_fn=dataset_input_fn(\"test\"),\n",
    "    train_steps=2000,\n",
    "    train_steps_per_iteration=100\n",
    ")\n",
    "little_experiment.continuous_train_and_eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally we can compare to a model with a single layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_task_type': 'worker', '_session_config': None, '_num_worker_replicas': 1, '_tf_random_seed': None, '_service': None, '_save_checkpoints_steps': None, '_is_chief': True, '_num_ps_replicas': 0, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f939b63dc88>, '_keep_checkpoint_max': 5, '_model_dir': 'models/single_layer_NN', '_save_summary_steps': 100, '_log_step_count_steps': 100, '_master': '', '_keep_checkpoint_every_n_hours': 10000, '_save_checkpoints_secs': 600}\n"
     ]
    }
   ],
   "source": [
    "single_layer_model = tf.estimator.DNNClassifier(\n",
    "    feature_columns=feature_columns,\n",
    "    hidden_units=(1000,),\n",
    "    n_classes=num_classes,\n",
    "    model_dir=\"models/single_layer_NN\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Experiment.continuous_train_and_eval (from tensorflow.contrib.learn.python.learn.experiment) is experimental and may change or be removed at any time, and without warning.\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1, loss = 299.513\n",
      "INFO:tensorflow:Saving checkpoints for 100 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 44.5894.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:14:07\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-100\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:14:08\n",
      "INFO:tensorflow:Saving dict for global step 100: accuracy = 0.881572, average_loss = 0.44388, global_step = 100, loss = 43.9908\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-100\n",
      "INFO:tensorflow:Saving checkpoints for 101 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 101, loss = 13.4816\n",
      "INFO:tensorflow:Saving checkpoints for 200 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 46.6798.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:14:16\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-200\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:14:17\n",
      "INFO:tensorflow:Saving dict for global step 200: accuracy = 0.896442, average_loss = 0.34751, global_step = 200, loss = 34.4401\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-200\n",
      "INFO:tensorflow:Saving checkpoints for 201 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 201, loss = 18.9586\n",
      "INFO:tensorflow:Saving checkpoints for 300 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 3.81309.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:14:25\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-300\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:14:26\n",
      "INFO:tensorflow:Saving dict for global step 300: accuracy = 0.906267, average_loss = 0.318438, global_step = 300, loss = 31.5588\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-300\n",
      "INFO:tensorflow:Saving checkpoints for 301 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 301, loss = 2.31559\n",
      "INFO:tensorflow:Saving checkpoints for 400 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 5.00304.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:14:35\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-400\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:14:36\n",
      "INFO:tensorflow:Saving dict for global step 400: accuracy = 0.910515, average_loss = 0.303765, global_step = 400, loss = 30.1047\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-400\n",
      "INFO:tensorflow:Saving checkpoints for 401 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 401, loss = 4.69084\n",
      "INFO:tensorflow:Saving checkpoints for 500 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 5.89316.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:14:45\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-500\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:14:46\n",
      "INFO:tensorflow:Saving dict for global step 500: accuracy = 0.908391, average_loss = 0.310978, global_step = 500, loss = 30.8196\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-500\n",
      "INFO:tensorflow:Saving checkpoints for 501 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 501, loss = 0.821377\n",
      "INFO:tensorflow:Saving checkpoints for 600 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.856056.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:14:54\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-600\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:14:55\n",
      "INFO:tensorflow:Saving dict for global step 600: accuracy = 0.912639, average_loss = 0.29796, global_step = 600, loss = 29.5294\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-600\n",
      "INFO:tensorflow:Saving checkpoints for 601 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 601, loss = 3.05731\n",
      "INFO:tensorflow:Saving checkpoints for 700 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.894425.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:15:03\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-700\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:15:04\n",
      "INFO:tensorflow:Saving dict for global step 700: accuracy = 0.909453, average_loss = 0.30824, global_step = 700, loss = 30.5482\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-700\n",
      "INFO:tensorflow:Saving checkpoints for 701 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 701, loss = 1.59988\n",
      "INFO:tensorflow:Saving checkpoints for 800 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.872111.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:15:12\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-800\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:15:13\n",
      "INFO:tensorflow:Saving dict for global step 800: accuracy = 0.914233, average_loss = 0.297888, global_step = 800, loss = 29.5222\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-800\n",
      "INFO:tensorflow:Saving checkpoints for 801 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 801, loss = 0.692904\n",
      "INFO:tensorflow:Saving checkpoints for 900 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.684748.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:15:21\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-900\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:15:22\n",
      "INFO:tensorflow:Saving dict for global step 900: accuracy = 0.912374, average_loss = 0.301172, global_step = 900, loss = 29.8477\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-900\n",
      "INFO:tensorflow:Saving checkpoints for 901 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 901, loss = 0.537427\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.501335.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:15:30\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1000\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:15:31\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.911577, average_loss = 0.304919, global_step = 1000, loss = 30.219\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1000\n",
      "INFO:tensorflow:Saving checkpoints for 1001 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1001, loss = 0.553138\n",
      "INFO:tensorflow:Saving checkpoints for 1100 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.541838.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:15:39\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1100\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:15:40\n",
      "INFO:tensorflow:Saving dict for global step 1100: accuracy = 0.912108, average_loss = 0.307117, global_step = 1100, loss = 30.437\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1100\n",
      "INFO:tensorflow:Saving checkpoints for 1101 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1101, loss = 0.393182\n",
      "INFO:tensorflow:Saving checkpoints for 1200 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.459115.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:15:48\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1200\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:15:50\n",
      "INFO:tensorflow:Saving dict for global step 1200: accuracy = 0.913967, average_loss = 0.306814, global_step = 1200, loss = 30.4069\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1200\n",
      "INFO:tensorflow:Saving checkpoints for 1201 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1201, loss = 1.20336\n",
      "INFO:tensorflow:Saving checkpoints for 1300 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.611669.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:15:58\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1300\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:15:59\n",
      "INFO:tensorflow:Saving dict for global step 1300: accuracy = 0.913436, average_loss = 0.304805, global_step = 1300, loss = 30.2078\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1300\n",
      "INFO:tensorflow:Saving checkpoints for 1301 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1301, loss = 0.401791\n",
      "INFO:tensorflow:Saving checkpoints for 1400 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.282411.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:16:07\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1400\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:16:09\n",
      "INFO:tensorflow:Saving dict for global step 1400: accuracy = 0.913967, average_loss = 0.310675, global_step = 1400, loss = 30.7895\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1400\n",
      "INFO:tensorflow:Saving checkpoints for 1401 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1401, loss = 2.78239\n",
      "INFO:tensorflow:Saving checkpoints for 1500 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.276821.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:16:17\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1500\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:16:18\n",
      "INFO:tensorflow:Saving dict for global step 1500: accuracy = 0.913436, average_loss = 0.309018, global_step = 1500, loss = 30.6253\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1500\n",
      "INFO:tensorflow:Saving checkpoints for 1501 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1501, loss = 0.325394\n",
      "INFO:tensorflow:Saving checkpoints for 1600 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.366973.\n",
      "INFO:tensorflow:Evaluating model now.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:16:26\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1600\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:16:27\n",
      "INFO:tensorflow:Saving dict for global step 1600: accuracy = 0.909984, average_loss = 0.318948, global_step = 1600, loss = 31.6094\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1600\n",
      "INFO:tensorflow:Saving checkpoints for 1601 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1601, loss = 0.339565\n",
      "INFO:tensorflow:Saving checkpoints for 1700 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.179703.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:16:35\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1700\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:16:37\n",
      "INFO:tensorflow:Saving dict for global step 1700: accuracy = 0.910515, average_loss = 0.316406, global_step = 1700, loss = 31.3575\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1700\n",
      "INFO:tensorflow:Saving checkpoints for 1701 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1701, loss = 0.276317\n",
      "INFO:tensorflow:Saving checkpoints for 1800 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.24035.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:16:45\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1800\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:16:46\n",
      "INFO:tensorflow:Saving dict for global step 1800: accuracy = 0.914764, average_loss = 0.311063, global_step = 1800, loss = 30.828\n",
      "INFO:tensorflow:Training model for 100 steps\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1800\n",
      "INFO:tensorflow:Saving checkpoints for 1801 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1801, loss = 0.236172\n",
      "INFO:tensorflow:Saving checkpoints for 1900 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.205219.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:16:54\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1900\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:16:55\n",
      "INFO:tensorflow:Saving dict for global step 1900: accuracy = 0.912108, average_loss = 0.316286, global_step = 1900, loss = 31.3456\n",
      "INFO:tensorflow:Training model for 100 steps\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-1900\n",
      "INFO:tensorflow:Saving checkpoints for 1901 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:step = 1901, loss = 0.196552\n",
      "INFO:tensorflow:Saving checkpoints for 2000 into models/single_layer_NN/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.367257.\n",
      "INFO:tensorflow:Evaluating model now.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-17-22:17:03\n",
      "INFO:tensorflow:Restoring parameters from models/single_layer_NN/model.ckpt-2000\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-17-22:17:04\n",
      "INFO:tensorflow:Saving dict for global step 2000: accuracy = 0.912108, average_loss = 0.314485, global_step = 2000, loss = 31.1671\n",
      "INFO:tensorflow:Stop training model as max steps reached\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "({'accuracy': 0.91210836,\n",
       "  'average_loss': 0.31448451,\n",
       "  'global_step': 2000,\n",
       "  'loss': 31.167068},\n",
       " [])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "single_layer_experiment = tf.contrib.learn.Experiment(\n",
    "    single_layer_model,\n",
    "    train_input_fn=dataset_input_fn(\"train\"),\n",
    "    eval_input_fn=dataset_input_fn(\"test\"),\n",
    "    train_steps=2000,\n",
    "    train_steps_per_iteration=100\n",
    ")\n",
    "single_layer_experiment.continuous_train_and_eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now go to Tensoboard and analyze how the different models behaved, comparing final performance and training time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:env_edm2]",
   "language": "python",
   "name": "conda-env-env_edm2-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}