4.1_MNIST_mlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Digits classification using a multi-layer perceptron (MLP)\n",
    "\n",
    "A multilayer perceptron (MLP) is a feedforward artificial neural network model that maps sets of input data onto a set of appropriate outputs. An MLP consists of multiple layers of nodes in a directed graph, with each layer fully connected to the next one. Except for the input nodes, each node is a neuron (or processing element) with a nonlinear activation function. MLP utilizes a supervised learning technique called backpropagation for training the network. MLP is a modification of the standard linear perceptron and can distinguish data that are not linearly separable.\n",
    "\n",
    "A multilayer perceptron is a logistic regressor where\n",
    "instead of feeding the input to the logistic regression you insert a\n",
    "intermediate layer, called the hidden layer, that has a nonlinear\n",
    "activation function (usually tanh or sigmoid) . One can use many such\n",
    "hidden layers making the architecture deep. The tutorial will also tackle\n",
    "the problem of MNIST digit classification.\n",
    "\n",
    "A good reference for machine learning or MLP is the famous textbook [Pattern Recognition and Machine Learning\" by Christopher M. Bishop, section 5](https://books.google.de/books/about/Pattern_Recognition_and_Machine_Learning.html?id=kTNoQgAACAAJ&source=kp_cover&redir_esc=y)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import timeit\n",
    "\n",
    "import numpy\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import theano\n",
    "import theano.tensor as T\n",
    "from os.path import expanduser\n",
    "PREFIX = expanduser(\"~\")\n",
    "\n",
    "from logistic_sgd import LogisticRegression, load_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Having a glance at the data\n",
    "\n",
    "In this example we use written digit recognition. The MNIST database of handwritten digits has a training set of 50,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. Essentially, the data consists of image patches of size $28 x 28$ and their corresponding label $[0-9]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... loading data\n"
     ]
    }
   ],
   "source": [
    "# load the dataset\n",
    "datasets = load_data(PREFIX+'/ml-bootcamp/mnist/mnist.pkl.gz')\n",
    "\n",
    "train_set_x, train_set_y = datasets[0]\n",
    "valid_set_x, valid_set_y = datasets[1]\n",
    "test_set_x, test_set_y = datasets[2]\n",
    "\n",
    "#f = gzip.open(PREFIX+'/ml-bootcamp/mnist/mnist.pkl.gz', 'rb')\n",
    "#train_set, valid_set, test_set = pickle.load(f,encoding='latin1')\n",
    "#f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 784)\n",
      "The label is: 5\n",
      "Min: 0.0 | Max: 0.99609375\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPwAAAD8CAYAAABTq8lnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfUuMbNtZ3re6u7q6q6pfh5N7bVkXLgNGkSVbkTyByAwQ\nAiHxmBh5EguZiAEhKPLAdgaYhEhJLGFZMEAh2MgmyGDFsmMihWCkEDwJxgiCE2wCkq9kW77nXt97\nzzld/X6sDLq/fb766197V3dXd1fV/j9paa+9ux6rquvb/7/+Z8o5IxAItANL972AQCBwdwjCBwIt\nQhA+EGgRgvCBQIsQhA8EWoQgfCDQIlyb8CmlH0kpfS2l9HcppfdPc1GBQOB2kK7jh08pLQP4WwA/\nBOBbAP4cwLtzzl+Vx4SDPxC4R+Sck712XQn/DgB/n3N+Ked8AuD3APzETRYXCARuH9cl/FsAfEPO\nv3l5LRAIzDCuS/hQ1wOBOcR1Cf8tAC/I+Qu4kPKBQGCGcV3CfxnA96WUXkwprQL4aQCfn96yAoHA\nbWDlOk/KOZ+mlP4ZgP8OYBnAx9RCHwgEZhPXcstN9MLhlgsE7hXTdMsFAoE5RBA+EGgRgvCBQIsQ\nhA8EWoQgfCDQIgThA4EWIQgfCLQIQfhAoEUIwgcCLUIQPhBoEYLwgUCLEIQPBFqEIHwg0CIE4QOB\nFiEIHwi0CEH4QKBFCMIHAi1CED4QaBGC8IFAixCEDwRahCB8INAiBOEDgRYhCB8ItAhB+ECgRQjC\nBwItQhA+EGgRgvCBQItwrWaSgflGSmMtx6b2unxtzvXaJO9t/67nOefa4b1vSglLS0tj69LXtmv0\nwNcvvW/T2m6rh+NVEYRvCUqkmyb5l5eXsbS0VDx672fXUnfTODs7w9nZGU5PT0eOnC8tLWFlZQXL\ny8sjg9eWlpZqR0qpSMycM87Pz3F+fl6959nZ2dh53fVZIH0QfoFRItd1JNwkWFlZcUen08HKysrY\nmrx1kHje8eTkBMfHxzg+Ph6b8/07nQ46nQ5WV1fH5iR+6WihBM054+zsDCcnJzg9PcXJyUntXAdv\nFkH4wK2gjrx1UvSmIOFWV1fHRqfTKarTPJYkL6Xz0dERDg8PR8by8jJSSjg/P0en00G328Xa2trY\ncW1trboBkPx63ul0qs9hic4jbzJHR0fFo86p1Zyfn+P09PTG3+80cCPCp5ReAvAUwBmAk5zzO6ax\nqMD04e1tS/vs62JlZQWrq6sjJFPi1d1oSHhVxfV8aWkJh4eH2Nvbw/7+PlZWVkYIdXJygk6ng7W1\nNayvr6PX66HX643Mu93u2I1Ir6lK7x2Pj49xeHiIg4ODkaO9dnBwUL0WyX5bdpOr4qYSPgP4wZzz\n69NYTODmqNsj89was6Yt4Uk6DhLPu8Houd1z2/ne3h663S46nU61dpKd793tdtHr9TAYDNDv9zEY\nDKqxvr5eqwGQpNYQx/nh4SH29/erwZuPntsbEW0Li0J4AJiNTxIYQ8kgVtoj3xTLy8uV1FxfX0e/\n3x8ZTRqG7qm9sbu7W5EdeCbZj46OsLy8XBGe7725uVmNra2t6sazvr4+dlNaX1/H0tJSrQX+4OAA\nw+GwGru7u9V8dXW1Ijv3+6enpzg+PnbtA/eFaUj4P04pnQH4Dznn/ziFNQVuCVaqKtmnQXhV6SnZ\nB4MBNjY2MBgMxjQKq2Wogc87drvdijyUnsfHxzg4OHAl/ObmJra3t7Gzs4Pt7e3qxkMV385LEp5j\nb28Pu7u7ePr0KZ4+fVrdQFTrsGvjTWBRJPz355y/nVL6BwC+kFL6Ws75i9NY2LzC+8dO+s9usmA3\n/W0SC7i6pzy32U1AcpPgdu4RXklvrfp2nnMes4SrsUxV+I2NjUqyb21tYXt7G4PBoCK4Ep5HJTyt\n6nq+tLRUkVnfWz0CanNQd9+s4EaEzzl/+/L4akrpswDeAaC1hPf2pnZeeh6PSgpLEKuGe4+pe/86\nl9Q01E6rwl9Xpbfr4jnV8H6/X7nB1L/d7/crab61tYXNzc3qhtPv9ytVnhJZb3RKbJKdcw7eWDis\nx4AWel3brPjfiWsTPqXUA7Ccc95NKfUB/DCAfzW1lc0prmoJt2T33FHeufc3zxBmCVWnMt8UnqHu\nKka7ps/JrYJH9pQS+v1+JdG3trZGNAwSnlZ5fm5Vw3nUwBk9WsJ7xGdsANc3K/534iYS/nkAn738\nx60A+N2c8x9NZVVzCu/Hq1KYj9HH67wk3eqknp7X3WiWlpZGfOQakMLjTVGyfnPukdye12k3GuBC\nMgGobgy9Xq+S7BxU8fv9/ogLriThbXScSmqSmST3jirhT09PRzSGWcC1CZ9z/jqAt01xLXMNj+x2\n8HF61OeWrNNe1Jo36tR9Skj1PdvjTeEF3OjQz+p9F032j263OybZSfaVlRX0er1q707prhKelnQO\nEp57d0pzDd3VeZOE5zlvTIsm4QMOPPXUiyW3R8aB2+iv0lBDkaqn3l6faylJXh5virobEzWIOsNi\nUzIKCUTJTq2I38f6+vqIkVB98P1+v7Kkq7FS31vj30l2ldat3sMHxlEiu0oSj/AcpZDUJsmpkWJ2\nK6HD+p/t/KaoS56xRkErzVNKY0Yya0DTGwJfl9+P+t816EaPpW0PgJH3I+FtTLx6BEpjYVX6gI86\n0tdZ0ZeWlkbCPTnseWmsrq662V+6Dhtqqufr6+tT++yTGiwtSplnVlJSstPvzxvX2tramKtNfe8M\nigH80Fkr4ZXoDPBpkvKLbLRrJer24CX1W/fYJYMV99glkmuM+rQIb+f3DZIMeGY116g1j/gq6bmt\n0ZgCldjUIlRj0CODZUpZecPhEHt7e9XY39+vYuetSs89fKj0cwzPesyxvLw8RkjdH2u2WMkqPalK\n7yWBcH9qia5zSkGbLjqJ9L0LKOlsJpodKnn1Om8alL5ra2vY39+vtizWx65zT6rrGA6HVZTd06dP\nsbu7i729vYr0arSzEn5WSB+EnxB1VnhKd09l1kyt0s1C3WbXNdrpDaUUoKM3IWvou0+QDEp4LwvN\nyzW3e2wlu9WCrES3Q410+ro839vbq2LoOSjpuV6uw+7hZwVB+CtAVUh77HQ6WF9fHwnt1DmTM0q+\nZlrpJ3XDedeb/NjWD6+uqfuAlXokHQmr2Wh7e3sj5LMWdGbMMdTVakf0EmggjUd4zzrP84ODg4r0\nVOuHw2Gl2quGEVb6BYDdE2vgC63ETNrQiK+trS0MBoMi0fVGUhd8Uxd62hR4k9KzSDu9WdyXhPdI\noBlmTEXVrDQloecn5/dSiiZUw5xa5C3h9bX1nKq7psRyULrb8luh0s8xdL9upasl/M7ODnZ2dvDg\nwQPs7Oxgc3Oz6CO324SmENrSY3SN3rzuRjELsCr9/v7+SHaaGsK8aDhqSaUbpSV5XU06Wy/v7Oys\n0jxsAQwe9Tn2ZjIrCMJPCM/dpuo1Az8Yz729vY3v+q7vwsOHD/Hw4UNsb28XjWl1++7Svtz7G9ep\nR52XIgDvmvCetLOWckpSGsreeOONak9cinX37Cs6JilAWVeU0vrhNbru6OioUt9LsQOzgCD8FWBV\nbyW7zcPe2dnBw4cP8dxzz+G5557DgwcPigY/dSWViNt0tHNv7Tx68/tEyWinhH/8+PFYIIvdgzd9\nxiY/f0nV51yNg577Tn3upeN9Iwh/BXi+d5WyGvnF8FXeBFgAom7MArwfqP2x1v14625Mda9jJfzR\n0VFlJKNaf3p6OiY5lfxNaCJ8KdLP7vFL1WpnhdR1CMJfAbYgAveNnpHH+wEBz374s/jjKBV+sDHu\netR5k42i9Dx9by0uwf0xrfQe4XWdTVCJXdoWWM1Bz+3efhZV9iYE4a8AWyTB+xFY67H+OIBnmoH+\nWGcF9mbmxbTzcfp4zj13Ja/X3Sh41JBWNZBRtSfhS6MJJcltyV46ehrBrP0PmxCEnxA25po/AEov\nj+RWkgCjW4FZ+6GUftjqT67LZrNVdEh2aysoaQtWwlOttxK+tI4mNElwT6uxN/mS5jYvCMJfAd4P\nwPPflkiv+/2rSqe7gCW8blPqpCsJQn/3+fl51WkGwIi05/vYY87Pik+oSq/GOzZzaNpiNH0+b/9f\npzl4Wzl7k5gXBOGvCE/CAyhKeM9tdH5+PpK5NSvQz+YZpzwJqPOzszN0Op2xPX2dHcBqTiWVnhLe\nPl+Pk3y+puGtSzWQuhvFPCAIfwXYO31KqVLVvYANa7xLKY2QfdZ+KFbS2uwx+2O3P3wNI9UgJSvd\nS3Pdw1tL/XA4HHl9Puc6n0+f612zf7fPL90o5gFB+CvAkl2vT6LWk/BqAJslqMqqLij6mktGL70B\nEHQ1MqS1jvAqQa1K70l4b93X/bxXea3SzWYW/5clBOGvCCW9XlPppBJKs740tpvqLzUArQhTUlWb\nYuUnXX9prj5wLdvEeZOfem1tDUdHR1hbW8Px8THW1tZGss4s7Jq9NFib+hq4GYLwV0CTIYdEt2Gh\nzJSrS3FVSejtjwE0JtFcZf2eoUoz1FSy8ryO7Dnn2r5tTA+uG5pXrlVj5kmCzjqC8BMi51xJJCvl\nuZe3qZ3D4XCkc2pdK+VOp1NLppyzW2aa7z8p4euCSvRG5fVP89xaek1zz71CIF7vOD1n1lmpiETg\n5gjCXxG636QFmoTXvSeJQ3JSApbKRNOdVYoCyzlX0pKDN6FJu8ZYjUSPp6en1Y1KM9SePHlSzUu+\na85Lpbd0XlcbXwkfEv52EIS/AuwPT33r5+fPOplSUrIOOlMzSxLw+Pi42s97aZn0gbN6DknAHPdJ\ngz+sRLf+dk1JffLkCR4/fow33nijOtZZ6fn57I3Mq9Hnqf3dbrdKM9WEFI1SDNwcQfhrwBq7PJV+\nf39/pGbc2dnZyA+cxi0atjqdTrHaCrPESAAAFdlXV1evRIhSYA0rutiU1Ndeew2vvfYaXn/99aI7\njsem2nteGyptHUVbgar0oc5PF0H4K8IjO630arTTajK0gJPg6+vrI8UZlfClem2q2jNyjVl5V5Hw\nda43VemfPHlSEf7VV1/Fd77znSLRlfB1dfSZOagNJlWKl1T6kPDTQxD+Cqjz25LwR0dHI5JdbwaU\naEp0SrLV1dVirjV94Lpnp8SkFJxk7Z5Kr+9jVXol/KNHj8Ys/DbirtSzjseNjQ3s7+9jMBhUqruu\nX8s9h9HudhCEvybsD9BKeS0dZd12lvCU8PbvtriCNl/Q/TBvIl7hBxsgZMNXKU0p3XWopX53d7fo\n0uN5U6XdptxzegQo6e0NIXBzBOGnBJWeJJTtTuqp0dzzK+G92ujn5+eu/57qMrcQpRp4NCzaAhM2\nBZV11kk4qtw2Ks4LK7VbBi3DBaDSfnSroyG89AaQ9JT0QfjpIQg/RaiU11pxdt9s+5TRLVeqh87E\nFU9V1pLTnm8bwIgnga9pw1a5d9c66yXC16nXKrE1Mi7njKOjo5HvRcl+eHg44vfXSrA0VAZujiD8\nFKGkVilWIvvq6ioODg4qwnvWeT4HQJHwtimFWu61Zp4n4Ul0JZvupb19dIn4VsLrdcYteEZO2g64\nHmutD1/89NBI+JTSxwH8GIBXcs5vvbz2AMDvA/geAC8BeFfO+fEtrnPmwR87k2TqyO6F2C4vL48l\n32hCDoAxNd6SnXt7NfAxgQUYLxJpa8ZNIuH1s3JuvwObZ8Brnhp/eHhYfQ5bBpoSPlT66WESCf/b\nAH4dwCfl2gcAfCHn/OGU0vsvzz9wC+ubK6hkU7IvLy9XRC91kWFtvNIAUNtnjp1vVLJTtec1NR4e\nHR1V6jz97torTY1m1jVmya6kt4+z+f9KdttBxzNahko/XTQSPuf8xZTSi+byjwN45+X8EwD+BEH4\n6sduDVdqQPOaQdCQ5YXU8piS3z/exuJbyc6wXrXQexL+yZMnldGuJOGJknqtYcdch0YjWi+Gfida\nDNRqN6HSTw/X3cM/n3N+dDl/BOD5Ka1nbqE/dv7IAb8JpWdJ11x5LzGljvAcfD913ylZrEvOqvTW\nSl/aQzcRnkdb1vv09LToOrSFPa2PPzAd3Nhol3POKaW4/WJ0b+uhqetMXSTb0tLSSJcTrwMKVX5t\nyWSNbbpOdSXam4znhpv0O/DmgdnAdQn/KKX0ppzzyymlNwN4ZZqLWmRoOK7NuJuEaPYxddFv9jW0\nWQaDdhjuqiWsbNTgLHSnCUwH12138nkA77mcvwfA56aznHagRFhPwtrn2efXSWd9DjBKeMbhsx8e\nY9zX19dHDIEauBPEn39M4pb7FC4MdA9TSt8A8EsA/h2AT6eU3otLt9xtLnJR4Km4atQqSW+PwFaq\n16njAMYMeSR8r9cb67Ou0X+2f7wGEwXmD5NY6d9d+NMPTXktrYFHFs/PPamEr1PpLelp0KNKr643\nDcjpdruVy8zrezeLjTQCzYhIuzuG7uH1XP/uHfXvTduBUnaZp9Jrzrnm8qtKX1Llg/TzhyD8HcKS\nve563dwju51PYrSzVnzG1q+trY1U69GQ2MB8Iwh/xyiRpqS+e9esOl9S5T2VXvfwSvalpaUq8o6V\neaxKH6SffwTh7xFXJU/Oo/3XbEsmhtp6BTY0eEaNdxqK2+v1RspKk/S2jDbXEj73+UMQfk5AKa6h\nsXt7e2NGNQ3qYYw69+NaJw54RnyS1VbFpdtOq+SWIgGD8POBIPwcwRbK5B4beGYH4D5dVXf61TX6\nDnhGeD7PIzxJv76+7vbMizj3+UIQfo6gEv7g4KCS7Nq7ziO7ldCq2utzSpKdw/bPY/nsiHWfHwTh\n5wQkFrPNbNmq09NTl+zdbrcir01WUWMcMK7SU7JzbvvEcU0RgTc/CMLPEVSlB0YlvuaXW8nOOvCa\nkqsZe5zXqfRra2vFsl1B+PlBEH6OoHXilOydTqcKhVWyU0KzXBRvCFTj2eCSxj0r1S3plfC2UGVg\nPhCEnxOoSq957ZTSHtl7vR76/X6V386mljT00WhHK75qBUp+3gB0HeoeDMLPD4LwcwStqMMKMtyP\ns1WTlq3q9XrY3d2tCEuXnCbS5JzHkmq479dMOu1vrxV7tJBHXViwlwdQ8ukHbg9B+DmDxt4rSayP\nnh1kSNKUEvr9/khzB0p7Ep3EJ+EHg8FIbXgtMMle7nrNRvt553VlvAK3jyD8HKIUdnt2djZSuorV\ncCl9SXbeNEh29qezbawGg0FF9pTSCMkt6Q8PD4sdZbROvR3UTiJ4524QhJ9DeMQoReFp0wdtM61k\n15sA9/R0w6l/35LdlpT2qu3qft/2zNPKP4G7QRB+zuCp9DxqUA5Dbin52b1GpTjj7m2/+W6366r9\nlux27gXm6Jw1+LwmHYG7QRB+DlFS6VXCW7JTomp6LJtQktxqtbdkp3uP5LZkZ9MI7W9vjwcHB26T\nTa6t7vMFpoMg/ByhZLADRlV6S3bG3qsa3+v1RppNWAlv1X7eSEpk1zr2XjNMr6OurlcRabi3hyD8\nnKEun54EsmRnUQslu/Zotyo9gErtV7XcEtwOtp4uDRJbyV4qoRW4HQTh5xy8AWhcPa+TqGzjNBgM\nqmYTHPTb7+/vj7jQNBqPr0dfvba80j71TYRXCe/l9tsa+qXKPd7nD0yGIPyCwCt7pXHuS0tLlUGP\n3WYYkKP95euGdrTRjDvd+9f1uPcCdzR4Ry37+hk0914/q/38gWYE4RcImr1Gsuv+XItUDofDkVJW\nGlevnW1t/L0WzLA3gSajndbI0847HFqdR331PC+V7wqyT44g/ILBRrWpyq9ReMPhcKQyLYDqBsAj\nbx6U/nycV2ijzh2nW4s6wjOqzxulOn76uQPNCMIvEEpE4LBReCQg9//9fh+9Xm/EB0/prdVxVNrb\n4Bov4IbD2zYo6KdnfXzPX69NN6MpxtURhF8wKOE5Z/86lfBU4/k4BsZoKC1ddABGpPHy8rLb+MLb\nd+tQ6e61rmKCDuMIrL+ec0VE6V0NQfgFgko6zaQj6Snh1RWmLjzukzWJRq8BF+66ksGsKXHG9qmz\nhKfNwAvOYZUdfU9tOx2YDEH4BYJnxFJSUMKTUDYSz0bbsXAlCW9LZNlRSoflke9ro+o4bNMLTbhh\nfIB9XpD9agjCLxhK0pdWcPrDrYRnTTy10Fs/u5bDskdVwT3Jq++vxjzeUEh4ddXp81n8Q633tuRW\n+OmbEYRvEWywi+0My3LWpRBYddOp+05ddyShPXKufe36/f5IlJ91B+pYXl4eybZTC76ulZ9Tj3be\nZgThWwRVk5UolI5aLMM+9ujoaKTGPSW/XlOprzcTPXK7sLa2Vhn31OpfN2jF16Fr5dxK+yD7MwTh\nWwQawFh4grBGNX0sXWT7+/tjBS55TjceJfHKykp1rnt2ax+wufl1En5lZaUquFEy7On2QL0I/IyB\nCQifUvo4gB8D8ErO+a2X134ZwM8CePXyYR/MOf/hbS0ycHOQCCQ84d0EbDEN+u5ZFJNH27qKZD0/\nP69UfK1/r4S3ZGdyj6rwVqW3VnzvMynJVeoHLjCJhP9tAL8O4JNyLQP4SM75I7eyqsCtQNV0awVn\nll2J7Lu7u9jY2MDGxkZV+soWxdSYd17T4pZKeJubf3JyUqvO8wZQctkxClCLaXAdYcl/hkbC55y/\nmFJ60flTfItzBhuQw7ryLHetBNKoPKrxzIHXohnAs5h6dZVpwo0lvJJdI/GsKq8kt3YBlexMs1XQ\nlaheipD0N9vD/0JK6Z8A+DKA9+WcH09pTYFbAkmiYanqR6eB7vDw0DXQlchOyzuhZC/1sut0OmMB\nOqVsOg7PmKixBV5yTUj4UVyX8L8B4F9fzn8FwK8CeO9UVhS4NUySXaaSkymtvAFYd5sdXk96DbP1\ngnd0r69hubpGjbu3YbxekUzN0gMi/FZxLcLnnF/hPKX0WwD+YGorCtwbbDy8jXlnHL41nDEoxjaf\n1Pna2pprkLN7c0237Xa7Y3tygjcdvk6n06laatmhhUHajmsRPqX05pzzty9PfwrAV6a3pMB9wabW\nWtcd9/XWfcfEG9uayp7rNkGHVtfRxhgalKMBQnpjUFfe/v4+Dg4OqpsSb0j0QMQefjK33KcAvBPA\nw5TSNwB8CMAPppTehgtr/dcB/NytrjJwJ1AJzyIael3j8IFn4a68ESjBS+TXuQbdABjZ39sa+iSw\nrb6j5baGwyGGw+FIyi9DhgMXmMRK/27n8sdvYS2BGQD38JzrTYA56pp0o1V0SoTXPnX9fn/E6Of5\n5bVyrva9K5Fdo/54Y1D3ot6k2o6ItAtUUILznMY2qvjWHUayawNKDval5zg6OnK73LAzrar0SvbT\n01N0u93KWm/Jzsg/j+zcggQuEIQPVLAx6ZzrHlrVeJKO7jQlOOd6zfa105h6ACPqPcmuVn410FGN\nt/3r1aZwcHAwlgzUdgThAxWs286muWpUmzcsye1R9+wkK/vX6Y2A2wZbWMOWyGYw0Pr6Og4PDwGM\nttva29sbqdkXCMIHDOqyzGyEnvXHa+FKLVPNBBxV4Xu9XuU35xaiLp9e18D35Y2GNwEm+QyHw8pA\nqHv7urTZtljwg/CBK8Gmn+qeX/f2tjjF+fk5er3eSHgupbsNtNH3skU0tA4+E3W4Hq26q4PXvFp8\n+vw2IAgfmBgkoBJEi2RS5bdk542g1+thY2OjqkzLaDw1ElqprtesZFepnVJyya6DWohG9PEztMVP\nH4QPXAmeCqx+e02ztRl5DIypk/Ae6QklvCX70tKSK9V1MF+AWxJNqW0LgvCBa8Emp5BM+nfeADqd\nDo6Pj7GxsVGFv5LwtNzb1/ZIr4S3ZLeE9wZwUXWX1X70M7QFQfjAlVC319aAHS2MyeCXvb29sRRb\nlfBKvjrSlzrg1En3tbW1KmVW180tSVsQhA9MDN3Dl66pGm+z6ajSq4TXvbS+ngcvpl7TcD2y67nt\nVqMZfG1BED5wJZQkPI/aAAPAyHySPXzpPTUAiEeb+95ktPMKgGit/Dbs5YPwgRFYN5idWzLba/a6\nXtOU2LqWU1dZo17zGlTqKD23TQjCByqUiNJU+MIjmnf+8OFD7OzsYGtrC4PBAOvr62OhryUyExrv\nb/3oDPLRYB+Ow8PDMWOhtR+0AUH4QIUSsTWRxYbT6rW6m0Ed4W05a67Fwqt2oz51G9mnZCfhvYo8\nbSE7EIQPCEi6Ujspr/OMHksx9nyNhw8f4sGDB9jc3KwIzyy4JrITuv+2banrJDwHw36thG8LgvCB\nEWjOuZXiTFrRBBa95lWc1fmDBw+KKj1hyV5S55XoWoNPya6kp4TXeH/NxGsLgvCBCrrf9lo/sdCE\nDrq+ut1urfTvdDrY2trC9vY2tra20O/3a1V6rkfhkV2bS2rvOUt2Et5qByHhA62FLSGlxSZIeC1K\naY+aH+/NB4PBSDMLz2jHdZRgSV+XnecNfX4kzwRajZKEJ2m1qk2v1xsZ3I9bVZ+Dz2WrKn1OScJb\n2CKbqso3qfP0/evrtC01FgjCLxzq1OKSX5qjRFYOS3I7lNyW7FqdhloByX6V0NY60qt6742SRT4I\nH5hL1AWb1FnfedRikF5L6KYilWrQs2q9V2yS7ztJMIxKZeuWswY8Vdnt8F63TQjCLxC8IBk9ltow\nU233jHJqmPPCVbV9dJPbzrPoa417Cy+LziO+R/Y2GuQmQRB+wVAXDeep6V5ByBLJ6yz0VM9LrZ5t\n0wjWmiuF15aI2kR2K+HrSN/Gm0EQfoFgyW4DXyyRryK9tT6ct1dnAI314Vt/vm0v5an0JdW7LtLO\nU+etWt/0Hm1AEH6BoHt3j2jqVivVjm9qFVVyuWmjCHvT8W5CejNqcsPp3JPwTXv40uu1EUH4BYP1\npdta7qwYyy4wHOoqszcBnuve2+7PaW0vJdDYGH07J5oIeZU9fJvdbyUE4RcESiwbOKNGORJ4MBhU\ngTA8Wr+6dbupKu4duQ492mteam0JTdLdI3ydpT4QhJ9J1KWIlshDP3pJ5V5dXa2i3DTaTc+V5F4j\nibrEmEmbPZRIrOeWqDxnAA2LaNijLaHlVdRpO4LwM4SS9LPSu7RXtsY0PXa7XfT7/UqyDwaD6pxH\na6izPnMXvj8mAAAPg0lEQVSvcMVVClh4JNbBIBpPYp+dnVXkVqLr/I033sDrr7+Op0+fYjgc4uDg\nAMfHx61Lga1DEH7GUFeEokmlbrLCc69eOlq3m2oJXs77VavVlIJmOBgiawcj6Uhwj/QHBwd4+vQp\nHj9+jCdPnlSEZ4Zc4AJB+BlCqcoM5+rD9vzaXi/2Jqu8npf887YslZXwk8ILi9U9tya5eAkw7FTr\nkZ4tpnZ3d7G7u1up96yOGxL+ArWETym9AOCTAJ4DkAH8Zs7511JKDwD8PoDvAfASgHflnB/f8lpb\nAau+69zLQ1dfuGdws6Su87dbq7sXEWdvRlzzpPBSWzm3+/KSJLek1+s6VKUPXCDV3flSSm8C8Kac\n81+llAYA/gLATwL4GQDfyTl/OKX0fgA7OecPmOfGLfWKKPmuS4Ezdm7dbXruqex2lEpXcdRtNyYh\nvU1yUXWdnWn29vaqo86VxJb0Wv6aee+edtA2KZ9zHvun1Er4nPPLAF6+nA9TSl8F8BYAPw7gnZcP\n+wSAPwHwAe81ApPBI5AlvPrSbVCMutpKxjmvWo2eT1qgkuvV4yRQld7LY6fKTtXcHpXc3tHeQHQE\nLjDxHj6l9CKAtwP4MwDP55wfXf7pEYDnp76ylqJEdkpdzVxTH3m/3x9zu6nrbTAYjO3/1Q7AnHRd\ngzf31jspSmmtzGPXvfjTp09HxpMnT0akvJ0fHByM1Khra4GLJkxE+Et1/jMAfjHnvGsio3Ko76Mo\nkcALRuFcrfClbDarpttz61tXsg8Gg8aac5PARq15fvTSNTXEWaPc8fHxCLn1qIT39u4kfZC6GY3/\n5ZRSBxdk/52c8+cuLz9KKb0p5/xySunNAF65zUXOC+r86DyWhs1m8wx0TQUo7I1Aq9A0+dInQV1Q\nDP3opYqy1grv7bVLqvxwOMT+/n5tbfnAZGiy0icAHwPwNznnj8qfPg/gPQD+/eXxc87TW4mSMcvL\nUde5GuVKTRGbkl70mhLelpG2e/NJUUpc4fAMcXpUontHz1inBjvVBvi6kfN+NTRZ6X8AwJ8C+Gtc\nuOUA4IMAvgTg0wC+GwW3XBvV/KbkkbosspWVFTeG3XOr2aOWjCrltHe73WL2GudNKEXA2brwJKWd\nMzS2btQF12hVWu8YxB+FZ6WvJfxN0FbClyS5LSllB/fo1squ10oaAI+lWnTcHni+fT1vgq0PZy3h\ndSQuneu8VGnWayLhWeOD8KO4slsuMDk8y3bJ0m6PapQbDAbY3NzExsYGNjc3q/lgMKj1oXvNIOy5\n1Tquu4f33Gqnp6fFABjrQy9Jckrq0ihlxYUVfnIE4aeMOqleR8Zut4ter1cRnQ0beNzc3KyNsqME\n9wpU2kIT1yE7gBHj3NnZ2YgP/eTkpMpYGw6H1VGHZ1nX81LijO0S48XkByZDEH7KqAucsXXedWjw\nDAm/s7Mz0p6pripsp9Nxg2M8YpfmTfAkvPZys3703d3dkWPJh865SmvPG8A1eC7BwGQIwhfgEaGk\nDqulva6emyW5knVtbW1MqtvhJc2otjAJrB9dpaP1o1tiNe2xldw6eK0UEssRJaluH0F4gSf59FiX\nmlp3LBFeid/tdkdUedthtS6RZVJYyWnndT70s7OzRj86febeUD86Lfdt7dF+nwjCG5RCSkvZaqVa\n66VouRLxu93uWAUaNlwk4e1N5Kq+9FJ5KM1HLyW2WD+650tXn7lNfNnb2xtx02ngTBD97hCEd+Cp\n7KwoUyrjbKWwZ5yry2XXSDoNmbWEt77z61jZS4Yxr7+6nqv7zJs3pbZq3LwGzpSMbnEjmD6C8AJv\nX66EV+OajWNfX193SV0iup3bm4kNqLFloK/SponwSkmpVJ8kMKY06lJTS350lfCxd78bBOEv4fnQ\n9aiEp7+cqvfm5iZ6vV5xj14ysnnSvm7LUArNvY5Kr241jpIFnaMpqEb96N7cutnURqBrDNwegvAO\nPNJTpVd/uVrUNzY2al1mTGCp673WZPgreQgmgS0vpamp3J8fHBxU8evWl64VYUuVY20Enp3X1bPj\nGgO3iyC8oORu0wIU6i/f2tqqfOUaGONJaq9xgxcJ520npuVLt4RXY5wSnnXhrC/dI7nu1UtdXS2p\nS6m1gdvHQhG+ya3GY5Mv3Tuura1Voa7qG9/e3sb29rZLeCW+Z9Cz+enXhZKl5Edneqq37+bwSK7H\nprj4UovmcLnNDhaG8HUGN4/MXiOFutHtdrGzs1MRnIPus6Y9vPWhX8ePTpQMXLaDqvWnl+q5c5RC\nYulHVx+6bfQQpJ4PLAzhAb9Vso1lL+2fvWowthEjJbxKegbIkPBNUrxUhGJSeCqwtb571vCTk5PG\n0FavaKSeq4tOX19Jr+uJG8DsYSEJXwptte2N7T67tLfmc23JKB3sveZF2HlEnwbZLcFoebf+dI5S\nRVjOPf+59aN7GWxWwtubUWB2sDCEt6q7layen9sGzpTi1LkX1y6r6otngYpSlppXdnoapLd7dBLe\ni4Q7PDwcsbh7Vnh9HgNtrB+9ZH0PA9x8YGEID4zWdbfRbaurq25HVI6m/udaHtq7aXS7XXdL4ZH7\nOvXkFKUikSrhvTRUWt9tzTjObQUZW1XG68Ve2sOHlJ9NLBThrUqvxFV3mldRptfrFSvFeNsAO7fp\nqSUDItd5E8IT1gpuY95VZd/b2xvLXtPMtt3d3caKNhq4Y+PyS9I9yD5bWBjCeyq9zUZbX1+vyjlr\nNZnNzU30+32386rOS4EzHLoWPdbNvfM6lFxeqtLbpg6U5E+ePKmGVw66VJxSo+GafOhB8NnGwhAe\nwJh0ZxaabdygYbFqaa+T4qurq0VjIPfoQPnHX2dws48twSOjkpLqeWlYkquk393drS0+EUReDCwM\n4W2SC8mufdesoc0OGzCjxjvPym4LP9YFnTSpw01lmqiye6GrlOyeQc7zq+/t7VUGOVravWCdcK0t\nHhaG8ABG1HlKaCW717hBz0vZbirFbfEJVcdVGnrkLhm9rB/bQ865mJzCc8+H7vnTGRdPwnN/HiRf\nfCwM4e3+XQnPvXsd8Zne6gXg2Ag5a3VPKbn7aUt0GwhjjWSAr9rzdUtpp5qrXoqm866phI/AmXZg\nYQgPPJPwVqX3Gi96xG8Kr21yqVmy2wITddKZHU5LVu6cc5HESuCmMlS2v5tV6fle4VNfTCwM4XUP\nP6lKb0epK4wn0UsqvZLe7rmVcPZ4cnIy8jp2fn5+Pqaa26E+c29uB29CpZz0IP3iYWEID2DMJafB\nMk0Gu16vV0y6seT25kpMle62JpwXAcfrhEf4s7OzYmIL/eyl3uhaP84rUmnrygXBFxcLQ3jdw1vf\nuxdh55Gfr6PH0jX7N8CX8FaVL2WrHR0d1UrWs7OzYuoqI+XqCN2UuhqEbwcWhvD2B2tdYUpA3cce\nHh5idXV1Ku9f1yaJ4a5emyUSXj+HPdLPXiL9cDh0/fP6HQQCC0N4AFUHU0aZraysYGlpqfrR699Y\n2YUhtmtrazd67zorPElfqvbKZBWgHKxzfn4+or7z5mGrv5b86YEAsECE1+QRlmtiFRmSkdKVZFd1\nvtvt3vj9mwJjSga7qxrtmAzD11CV3SN9IEDUEj6l9AKATwJ4Dhf94X8z5/xrKaVfBvCzAF69fOgH\nc85/eJsLbYIlPKPgSPajoyPs7++P9VTnvNPpTOX96+q+exloqvbb19N5znmstJQGztTt0QMBoknC\nnwD4Fznnv0opDQD8RUrpC7gg/0dyzh+59RVOCEt4XiPZDw4OxhJjNFZ+GoSvSzyx2Wel+uz2NXWu\n5aWsH90jfCBgUUv4nPPLAF6+nA9TSl8F8JbLP18/r/MWoIQHRiW7rRrrlbm6SRFJwjOUWcNhaTSV\naqZR0LZ/shKejw0fesBDmvQHkVJ6EcD/BPAPAbwPwM8AeALgywDel3N+bB5/p7+0lNJIkotXXqqu\ncOVN8tKJJrdXKXnGi6P3/i9Nbjd9XrjZAjnnsR/1RIS/VOf/BMC/yTl/LqX0HJ7t338FwJtzzu81\nz7nzX1ldNNwk46YokcxztV3Hij6pH91bQ6B9uBbhU0odAP8VwH/LOX/U+fuLAP4g5/xWcz1+bYHA\nPcIj/JL3QCJdiL2PAfgbJXtK6c3ysJ8C8JVpLTIQCNweaiV8SukHAPwpgL/GhWUeAP4lgHcDeNvl\nta8D+Lmc8yPz3JDwgcA94tp7+OsgCB8I3C+urNIHAoHFQhA+EGgRgvCBQIsQhA8EWoQgfCDQIgTh\nA4EWIQgfCLQIQfhAoEUIwgcCLUIQPhBoEYLwgUCLEIQPBFqEIHwg0CIE4QOBFiEIHwi0CEH4QKBF\nCMIHAi3CrVW8CQQCs4eQ8IFAixCEDwRahDshfErpR1JKX0sp/V1K6f138Z5XQUrppZTSX6eU/jKl\n9KUZWM/HU0qPUkpfkWsPUkpfSCn9v5TSH6WUtmdsfb+cUvrm5Xf4lymlH7mntb2QUvofKaX/m1L6\nPymlf355fSa+v5r13cn3d+t7+JTSMoC/BfBDAL4F4M8BvDvn/NVbfeMrIKX0dQD/KOf8+n2vBQBS\nSv8YwBDAJ9ngI6X0YQDfyTl/+PKmuZNz/sAMre9DAHbvu8FoSulNAN6kDVAB/CQuWqPd+/dXs753\n4Q6+v7uQ8O8A8Pc555dyzicAfg/AT9zB+14VM9McM+f8RQBvmMs/DuATl/NP4OJHci8orA+Yge8w\n5/xyzvmvLudDAGyAOhPfX836gDv4/u6C8G8B8A05/yaefcBZQQbwxymlL6eU/ul9L6aA56XZxyMA\nz9/nYgr4hZTS/04pfew+txzEZRu0twP4M8zg9yfr+1+Xl279+7sLws+D3+/7c85vB/CjAH7+UmWd\nWeSLfdisfa+/AeB7cdGR6NsAfvU+F3OpLn8GwC/mnHf1b7Pw/V2u7z/jYn1D3NH3dxeE/xaAF+T8\nBVxI+ZlBzvnbl8dXAXwWF9uQWcOjy/0fe/u9cs/rGUHO+ZV8CQC/hXv8Di8boH4GwO/knD93eXlm\nvj9Z33/i+u7q+7sLwn8ZwPellF5MKa0C+GkAn7+D950IKaVeSmnjct4H8MOYzeaYnwfwnsv5ewB8\nruaxd45ZaTBaaoCKGfn+7rtB651E2qWUfhTARwEsA/hYzvnf3vqbToiU0vfiQqoDwAqA373v9aWU\nPgXgnQAe4mK/+UsA/guATwP4bgAvAXhXzvnxjKzvQwB+EA0NRu9obV4D1A8C+BJm4Pu7SYPWqbx/\nhNYGAu1BRNoFAi1CED4QaBGC8IFAixCEDwRahCB8INAiBOEDgRYhCB8ItAhB+ECgRfj/eMkJglLN\nit4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd1f1626f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Now let's have a look how the data looks like. Essentially it is a matrix with number of columns corresponding to \n",
    "# flattened image size(=28x28) and number of rows equal to number of samples.\n",
    "\n",
    "# So far it is just symbolic, now we have to get the actual data\n",
    "trainX = train_set_x.eval()\n",
    "trainY = train_set_y.eval()\n",
    "print(trainX.shape)\n",
    "\n",
    "# we want to look at index 0\n",
    "idx = 0\n",
    "# Plot the first image in the training set\n",
    "plt.imshow(trainX[idx,:].reshape((28,28)),cmap=plt.get_cmap('gray'))\n",
    "# What's the corresponding label?\n",
    "print(\"The label is: \"+str(trainY[idx]))\n",
    "\n",
    "# It is important to look at the range of the data, often normalization (range [0,1]) is required\n",
    "print(\"Min: \"+str(numpy.min(trainX[:]))+\" | Max: \"+str(numpy.max(trainX[:])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hidden layer for the MLP\n",
    "The hidden layer consists of three elements. A weight matrix $W$, a bias element $b$ and a non-linear activation e.g. $tanh$ function. Given input $x$ is essentially just computed $tanh(Wx+b)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class HiddenLayer(object):\n",
    "    def __init__(self, rng, input, n_in, n_out, W=None, b=None,\n",
    "                 activation=T.tanh):\n",
    "        \"\"\"\n",
    "        Typical hidden layer of a MLP: units are fully-connected and have\n",
    "        sigmoidal activation function. Weight matrix W is of shape (n_in,n_out)\n",
    "        and the bias vector b is of shape (n_out,).\n",
    "\n",
    "        NOTE : The nonlinearity used here is tanh\n",
    "\n",
    "        Hidden unit activation is given by: tanh(dot(input,W) + b)\n",
    "\n",
    "        :type rng: numpy.random.RandomState\n",
    "        :param rng: a random number generator used to initialize weights\n",
    "\n",
    "        :type input: theano.tensor.dmatrix\n",
    "        :param input: a symbolic tensor of shape (n_examples, n_in)\n",
    "\n",
    "        :type n_in: int\n",
    "        :param n_in: dimensionality of input\n",
    "\n",
    "        :type n_out: int\n",
    "        :param n_out: number of hidden units\n",
    "\n",
    "        :type activation: theano.Op or function\n",
    "        :param activation: Non linearity to be applied in the hidden\n",
    "                           layer\n",
    "        \"\"\"\n",
    "        self.input = input\n",
    "        # end-snippet-1\n",
    "\n",
    "        # `W` is initialized with `W_values` which is uniformely sampled\n",
    "        # from sqrt(-6./(n_in+n_hidden)) and sqrt(6./(n_in+n_hidden))\n",
    "        # for tanh activation function\n",
    "        # the output of uniform if converted using asarray to dtype\n",
    "        # theano.config.floatX so that the code is runable on GPU\n",
    "        # Note : optimal initialization of weights is dependent on the\n",
    "        #        activation function used (among other things).\n",
    "        #        For example, results presented in [Xavier10] suggest that you\n",
    "        #        should use 4 times larger initial weights for sigmoid\n",
    "        #        compared to tanh\n",
    "        #        We have no info for other function, so we use the same as\n",
    "        #        tanh.\n",
    "        if W is None:\n",
    "            W_values = numpy.asarray(\n",
    "                rng.uniform(\n",
    "                    low=-numpy.sqrt(6. / (n_in + n_out)),\n",
    "                    high=numpy.sqrt(6. / (n_in + n_out)),\n",
    "                    size=(n_in, n_out)\n",
    "                ),\n",
    "                dtype=theano.config.floatX\n",
    "            )\n",
    "            if activation == theano.tensor.nnet.sigmoid:\n",
    "                W_values *= 4\n",
    "\n",
    "            W = theano.shared(value=W_values, name='W', borrow=True)\n",
    "\n",
    "        if b is None:\n",
    "            b_values = numpy.zeros((n_out,), dtype=theano.config.floatX)\n",
    "            b = theano.shared(value=b_values, name='b', borrow=True)\n",
    "\n",
    "        self.W = W\n",
    "        self.b = b\n",
    "\n",
    "        lin_output = T.dot(input, self.W) + self.b\n",
    "        self.output = (\n",
    "            lin_output if activation is None\n",
    "            else activation(lin_output)\n",
    "        )\n",
    "        # parameters of the model\n",
    "        self.params = [self.W, self.b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLP definition\n",
    "Here we define a MLP. We connect a hidden layer with a logistic regression layer. \n",
    "To make it deep we could stack up multiple hidden layers like a pancake. \n",
    "The end of the network remains the same, a logistic regression layer (for classification)\n",
    "\n",
    "\n",
    "### Regularization\n",
    "\n",
    "In machine learning we train our model from data, trying to prepare it to do well on new examples, not the ones it has already seen. A typical problem is overfitting. That is a learnt statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. A model that has been overfit will generally have poor predictive performance, as it can exaggerate minor fluctuations in the data. A way to combat overfitting is through regularization. There are several techniques for regularization. Here we use L1/L2 regularization. These terms are simply added to the cost. Each term is associated with a weight factor to adjust the relative importance. \n",
    "\n",
    "In principle, adding a regularization term to the loss will encourage smooth network mappings in a neural network (e.g. by penalizing large values of the parameters, which decreases the amount of nonlinearity that the network models). Minimizing the sum of the cost and the regularization terms will in theory correspond to finding the right trade-off between the fit to the training data and the “generality” of the solution that is found. To follow Occam’s razor principle, this minimization should find us the simplest solution (as measured by our simplicity criterion) that fits the training data.\n",
    "\n",
    "L2 punishes large parameters, whereas L1 enforces sparsity.\n",
    "\n",
    "\n",
    "### Early stopping\n",
    "\n",
    "Early-stopping combats overfitting by monitoring the model’s performance on a validation set. A validation set is a set of examples that we never use for gradient descent, but which is also not a part of the test set. The validation examples are considered to be representative of future test examples. We can use them during training because they are not part of the test set. If the model’s performance ceases to improve sufficiently on the validation set, or even degrades with further optimization, then the heuristic implemented here gives up on much further optimization.\n",
    "The choice of when to stop is a judgement call and a few heuristics exist, but these tutorials will make use of a strategy based on a geometrically increasing amount of patience.\n",
    "\n",
    "\n",
    "### Initialization\n",
    "It is a very important and not trivial, because it is necessary that signals reach deep into the network.\n",
    "\n",
    "* If the weights in a network start too small, then the signal shrinks as it passes through each layer until it’s too tiny to be useful.\n",
    "\n",
    "* If the weights in a network start too large, then the signal grows as it passes through each layer until it’s too massive to be useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class MLP(object):\n",
    "    \"\"\"Multi-Layer Perceptron Class\n",
    "\n",
    "    A multilayer perceptron is a feedforward artificial neural network model\n",
    "    that has one layer or more of hidden units and nonlinear activations.\n",
    "    Intermediate layers usually have as activation function tanh or the\n",
    "    sigmoid function (defined here by a ``HiddenLayer`` class)  while the\n",
    "    top layer is a softmax layer (defined here by a ``LogisticRegression``\n",
    "    class).\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, rng, input, n_in, n_hidden, n_out):\n",
    "        \"\"\"Initialize the parameters for the multilayer perceptron\n",
    "\n",
    "        :type rng: numpy.random.RandomState\n",
    "        :param rng: a random number generator used to initialize weights\n",
    "\n",
    "        :type input: theano.tensor.TensorType\n",
    "        :param input: symbolic variable that describes the input of the\n",
    "        architecture (one minibatch)\n",
    "\n",
    "        :type n_in: int\n",
    "        :param n_in: number of input units, the dimension of the space in\n",
    "        which the datapoints lie\n",
    "\n",
    "        :type n_hidden: int\n",
    "        :param n_hidden: number of hidden units\n",
    "\n",
    "        :type n_out: int\n",
    "        :param n_out: number of output units, the dimension of the space in\n",
    "        which the labels lie\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        # Since we are dealing with a one hidden layer MLP, this will translate\n",
    "        # into a HiddenLayer with a tanh activation function connected to the\n",
    "        # LogisticRegression layer; the activation function can be replaced by\n",
    "        # sigmoid or any other nonlinear function\n",
    "        self.hiddenLayer = HiddenLayer(\n",
    "            rng=rng,\n",
    "            input=input,\n",
    "            n_in=n_in,\n",
    "            n_out=n_hidden,\n",
    "            activation=T.tanh\n",
    "        )\n",
    "\n",
    "        # The logistic regression layer gets as input the hidden units\n",
    "        # of the hidden layer.\n",
    "        self.logRegressionLayer = LogisticRegression(\n",
    "            input=self.hiddenLayer.output,\n",
    "            n_in=n_hidden,\n",
    "            n_out=n_out\n",
    "        )\n",
    "        # end-snippet-2 start-snippet-3\n",
    "        # L1 norm ; one regularization option is to enforce L1 norm to\n",
    "        # be small\n",
    "        self.L1 = (\n",
    "            abs(self.hiddenLayer.W).sum()\n",
    "            + abs(self.logRegressionLayer.W).sum()\n",
    "        )\n",
    "\n",
    "        # square of L2 norm ; one regularization option is to enforce\n",
    "        # square of L2 norm to be small\n",
    "        self.L2_sqr = (\n",
    "            (self.hiddenLayer.W ** 2).sum()\n",
    "            + (self.logRegressionLayer.W ** 2).sum()\n",
    "        )\n",
    "\n",
    "        # negative log likelihood of the MLP is given by the negative\n",
    "        # log likelihood of the output of the model, computed in the\n",
    "        # logistic regression layer\n",
    "        self.negative_log_likelihood = (\n",
    "            self.logRegressionLayer.negative_log_likelihood\n",
    "        )\n",
    "        # same holds for the function computing the number of errors\n",
    "        self.errors = self.logRegressionLayer.errors\n",
    "\n",
    "        # the parameters of the model are the parameters of the two layer it is\n",
    "        # made out of\n",
    "        self.params = self.hiddenLayer.params + self.logRegressionLayer.params\n",
    "        # end-snippet-3\n",
    "\n",
    "        # keep track of model input\n",
    "        self.input = input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def test_mlp(param):\n",
    "    \"\"\"\n",
    "    Demonstrate stochastic gradient descent optimization for a multilayer\n",
    "    perceptron\n",
    "\n",
    "    This is demonstrated on MNIST.\n",
    "\n",
    "    :type learning_rate: float\n",
    "    :param learning_rate: learning rate used (factor for the stochastic\n",
    "    gradient\n",
    "\n",
    "    :type L1_reg: float\n",
    "    :param L1_reg: L1-norm's weight when added to the cost (see\n",
    "    regularization)\n",
    "\n",
    "    :type L2_reg: float\n",
    "    :param L2_reg: L2-norm's weight when added to the cost (see\n",
    "    regularization)\n",
    "\n",
    "    :type n_epochs: int\n",
    "    :param n_epochs: maximal number of epochs to run the optimizer\n",
    "\n",
    "    :type dataset: string\n",
    "    :param dataset: the path of the MNIST dataset file from\n",
    "                 http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz\n",
    "\n",
    "\n",
    "   \"\"\"\n",
    "    datasets = load_data(param['data'])\n",
    "    \n",
    "    batch_size = param['batch_size']\n",
    "    n_hidden = param['nhidden']\n",
    "    learning_rate = param['lr']\n",
    "    L1_reg = param['L1_reg']\n",
    "    L2_reg = param['L2_reg']\n",
    "    n_epochs = param['nepochs']\n",
    "\n",
    "    train_set_x, train_set_y = datasets[0]\n",
    "    valid_set_x, valid_set_y = datasets[1]\n",
    "    test_set_x, test_set_y = datasets[2]\n",
    "\n",
    "    # compute number of minibatches for training, validation and testing\n",
    "    n_train_batches = int(train_set_x.get_value(borrow=True).shape[0] / batch_size)\n",
    "    n_valid_batches = int(valid_set_x.get_value(borrow=True).shape[0] / batch_size)\n",
    "    n_test_batches = int(test_set_x.get_value(borrow=True).shape[0] / batch_size)\n",
    "\n",
    "    ######################\n",
    "    # BUILD ACTUAL MODEL #\n",
    "    ######################\n",
    "    print('... building the model')\n",
    "\n",
    "    # allocate symbolic variables for the data\n",
    "    index = T.lscalar()  # index to a [mini]batch\n",
    "    x = T.matrix('x')  # the data is presented as rasterized images\n",
    "    y = T.ivector('y')  # the labels are presented as 1D vector of\n",
    "                        # [int] labels\n",
    "\n",
    "    rng = numpy.random.RandomState(1234)\n",
    "\n",
    "    # construct the MLP class\n",
    "    classifier = MLP(\n",
    "        rng=rng,\n",
    "        input=x,\n",
    "        n_in=28 * 28,\n",
    "        n_hidden=n_hidden,\n",
    "        n_out=10\n",
    "    )\n",
    "\n",
    "    # start-snippet-4\n",
    "    # the cost we minimize during training is the negative log likelihood of\n",
    "    # the model plus the regularization terms (L1 and L2); cost is expressed\n",
    "    # here symbolically\n",
    "    cost = (\n",
    "        classifier.negative_log_likelihood(y)\n",
    "        + L1_reg * classifier.L1\n",
    "        + L2_reg * classifier.L2_sqr\n",
    "    )\n",
    "    # end-snippet-4\n",
    "\n",
    "    # compiling a Theano function that computes the mistakes that are made\n",
    "    # by the model on a minibatch\n",
    "    test_model = theano.function(\n",
    "        inputs=[index],\n",
    "        outputs=classifier.errors(y),\n",
    "        givens={\n",
    "            x: test_set_x[index * batch_size:(index + 1) * batch_size],\n",
    "            y: test_set_y[index * batch_size:(index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    validate_model = theano.function(\n",
    "        inputs=[index],\n",
    "        outputs=classifier.errors(y),\n",
    "        givens={\n",
    "            x: valid_set_x[index * batch_size:(index + 1) * batch_size],\n",
    "            y: valid_set_y[index * batch_size:(index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "\n",
    "    # start-snippet-5\n",
    "    # compute the gradient of cost with respect to theta (sotred in params)\n",
    "    # the resulting gradients will be stored in a list gparams\n",
    "    gparams = [T.grad(cost, param) for param in classifier.params]\n",
    "\n",
    "    # specify how to update the parameters of the model as a list of\n",
    "    # (variable, update expression) pairs\n",
    "\n",
    "    # given two lists of the same length, A = [a1, a2, a3, a4] and\n",
    "    # B = [b1, b2, b3, b4], zip generates a list C of same size, where each\n",
    "    # element is a pair formed from the two lists :\n",
    "    #    C = [(a1, b1), (a2, b2), (a3, b3), (a4, b4)]\n",
    "    updates = [\n",
    "        (param, param - learning_rate * gparam)\n",
    "        for param, gparam in zip(classifier.params, gparams)\n",
    "    ]\n",
    "\n",
    "    # compiling a Theano function `train_model` that returns the cost, but\n",
    "    # in the same time updates the parameter of the model based on the rules\n",
    "    # defined in `updates`\n",
    "    train_model = theano.function(\n",
    "        inputs=[index],\n",
    "        outputs=cost,\n",
    "        updates=updates,\n",
    "        givens={\n",
    "            x: train_set_x[index * batch_size: (index + 1) * batch_size],\n",
    "            y: train_set_y[index * batch_size: (index + 1) * batch_size]\n",
    "        }\n",
    "    )\n",
    "    # end-snippet-5\n",
    "\n",
    "    ###############\n",
    "    # TRAIN MODEL #\n",
    "    ###############\n",
    "    print('... training')\n",
    "\n",
    "    # early-stopping parameters\n",
    "    patience = 10000  # look as this many examples regardless\n",
    "    patience_increase = 2  # wait this much longer when a new best is\n",
    "                           # found\n",
    "    improvement_threshold = 0.995  # a relative improvement of this much is\n",
    "                                   # considered significant\n",
    "    validation_frequency = min(n_train_batches, patience / 2)\n",
    "                                  # go through this many\n",
    "                                  # minibatche before checking the network\n",
    "                                  # on the validation set; in this case we\n",
    "                                  # check every epoch\n",
    "\n",
    "    best_validation_loss = numpy.inf\n",
    "    best_iter = 0\n",
    "    test_score = 0.\n",
    "    start_time = timeit.default_timer()\n",
    "\n",
    "    epoch = 0\n",
    "    done_looping = False\n",
    "\n",
    "    while (epoch < n_epochs) and (not done_looping):\n",
    "        epoch = epoch + 1\n",
    "        for minibatch_index in range(n_train_batches):\n",
    "\n",
    "            minibatch_avg_cost = train_model(minibatch_index)\n",
    "            # iteration number\n",
    "            iter = (epoch - 1) * n_train_batches + minibatch_index\n",
    "\n",
    "            if (iter + 1) % validation_frequency == 0:\n",
    "                # compute zero-one loss on validation set\n",
    "                validation_losses = [validate_model(i) for i\n",
    "                                     in range(n_valid_batches)]\n",
    "                this_validation_loss = numpy.mean(validation_losses)\n",
    "\n",
    "                print(\n",
    "                    'epoch %i, minibatch %i/%i, validation error %f %%' %\n",
    "                    (\n",
    "                        epoch,\n",
    "                        minibatch_index + 1,\n",
    "                        n_train_batches,\n",
    "                        this_validation_loss * 100.\n",
    "                    )\n",
    "                )\n",
    "                \n",
    "                # Early stopping\n",
    "                # Here we do some model checking. We use the validation set to evaluate the model.\n",
    "                # If we notice a substantial drop in the loss, we test the performance on the test set.\n",
    "                # if we got the best validation score until now\n",
    "                if this_validation_loss < best_validation_loss:\n",
    "                    #improve patience if loss improvement is good enough\n",
    "                    if (\n",
    "                        this_validation_loss < best_validation_loss *\n",
    "                        improvement_threshold\n",
    "                    ):\n",
    "                        patience = max(patience, iter * patience_increase)\n",
    "\n",
    "                    best_validation_loss = this_validation_loss\n",
    "                    best_iter = iter\n",
    "\n",
    "                    # test it on the test set\n",
    "                    test_losses = [test_model(i) for i\n",
    "                                   in range(n_test_batches)]\n",
    "                    test_score = numpy.mean(test_losses)\n",
    "\n",
    "                    print(('     epoch %i, minibatch %i/%i, test error of '\n",
    "                           'best model %f %%') %\n",
    "                          (epoch, minibatch_index + 1, n_train_batches,\n",
    "                           test_score * 100.),flush=True)\n",
    "\n",
    "            if patience <= iter:\n",
    "                done_looping = True\n",
    "                break\n",
    "\n",
    "    end_time = timeit.default_timer()\n",
    "    print(('Optimization complete. Best validation score of %f %% '\n",
    "           'obtained at iteration %i, with test performance %f %%') %\n",
    "          (best_validation_loss * 100., best_iter + 1, test_score * 100.))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... loading data\n",
      "... building the model\n",
      "... training\n",
      "epoch 1, minibatch 2500/2500, validation error 10.770000 %\n",
      "     epoch 1, minibatch 2500/2500, test error of best model 11.520000 %\n",
      "epoch 2, minibatch 2500/2500, validation error 8.870000 %\n",
      "     epoch 2, minibatch 2500/2500, test error of best model 9.200000 %\n",
      "epoch 3, minibatch 2500/2500, validation error 7.930000 %\n",
      "     epoch 3, minibatch 2500/2500, test error of best model 8.120000 %\n",
      "epoch 4, minibatch 2500/2500, validation error 7.340000 %\n",
      "     epoch 4, minibatch 2500/2500, test error of best model 7.490000 %\n",
      "epoch 5, minibatch 2500/2500, validation error 6.740000 %\n",
      "     epoch 5, minibatch 2500/2500, test error of best model 6.960000 %\n",
      "epoch 6, minibatch 2500/2500, validation error 6.220000 %\n",
      "     epoch 6, minibatch 2500/2500, test error of best model 6.500000 %\n",
      "epoch 7, minibatch 2500/2500, validation error 5.770000 %\n",
      "     epoch 7, minibatch 2500/2500, test error of best model 6.190000 %\n",
      "epoch 8, minibatch 2500/2500, validation error 5.440000 %\n",
      "     epoch 8, minibatch 2500/2500, test error of best model 5.910000 %\n",
      "epoch 9, minibatch 2500/2500, validation error 5.130000 %\n",
      "     epoch 9, minibatch 2500/2500, test error of best model 5.680000 %\n",
      "epoch 10, minibatch 2500/2500, validation error 4.920000 %\n",
      "     epoch 10, minibatch 2500/2500, test error of best model 5.370000 %\n",
      "epoch 11, minibatch 2500/2500, validation error 4.720000 %\n",
      "     epoch 11, minibatch 2500/2500, test error of best model 5.250000 %\n",
      "epoch 12, minibatch 2500/2500, validation error 4.630000 %\n",
      "     epoch 12, minibatch 2500/2500, test error of best model 5.110000 %\n",
      "epoch 13, minibatch 2500/2500, validation error 4.560000 %\n",
      "     epoch 13, minibatch 2500/2500, test error of best model 4.970000 %\n",
      "epoch 14, minibatch 2500/2500, validation error 4.390000 %\n",
      "     epoch 14, minibatch 2500/2500, test error of best model 4.800000 %\n",
      "epoch 15, minibatch 2500/2500, validation error 4.310000 %\n",
      "     epoch 15, minibatch 2500/2500, test error of best model 4.610000 %\n",
      "epoch 16, minibatch 2500/2500, validation error 4.190000 %\n",
      "     epoch 16, minibatch 2500/2500, test error of best model 4.450000 %\n",
      "epoch 17, minibatch 2500/2500, validation error 4.060000 %\n",
      "     epoch 17, minibatch 2500/2500, test error of best model 4.370000 %\n",
      "epoch 18, minibatch 2500/2500, validation error 3.970000 %\n",
      "     epoch 18, minibatch 2500/2500, test error of best model 4.310000 %\n",
      "epoch 19, minibatch 2500/2500, validation error 3.940000 %\n",
      "     epoch 19, minibatch 2500/2500, test error of best model 4.220000 %\n",
      "epoch 20, minibatch 2500/2500, validation error 3.820000 %\n",
      "     epoch 20, minibatch 2500/2500, test error of best model 4.130000 %\n",
      "epoch 21, minibatch 2500/2500, validation error 3.800000 %\n",
      "     epoch 21, minibatch 2500/2500, test error of best model 4.090000 %\n",
      "epoch 22, minibatch 2500/2500, validation error 3.720000 %\n",
      "     epoch 22, minibatch 2500/2500, test error of best model 4.040000 %\n",
      "epoch 23, minibatch 2500/2500, validation error 3.690000 %\n",
      "     epoch 23, minibatch 2500/2500, test error of best model 3.980000 %\n",
      "epoch 24, minibatch 2500/2500, validation error 3.650000 %\n",
      "     epoch 24, minibatch 2500/2500, test error of best model 3.910000 %\n",
      "epoch 25, minibatch 2500/2500, validation error 3.550000 %\n",
      "     epoch 25, minibatch 2500/2500, test error of best model 3.910000 %\n",
      "epoch 26, minibatch 2500/2500, validation error 3.500000 %\n",
      "     epoch 26, minibatch 2500/2500, test error of best model 3.860000 %\n",
      "epoch 27, minibatch 2500/2500, validation error 3.450000 %\n",
      "     epoch 27, minibatch 2500/2500, test error of best model 3.780000 %\n",
      "epoch 28, minibatch 2500/2500, validation error 3.430000 %\n",
      "     epoch 28, minibatch 2500/2500, test error of best model 3.750000 %\n",
      "epoch 29, minibatch 2500/2500, validation error 3.450000 %\n",
      "epoch 30, minibatch 2500/2500, validation error 3.420000 %\n",
      "     epoch 30, minibatch 2500/2500, test error of best model 3.640000 %\n",
      "epoch 31, minibatch 2500/2500, validation error 3.400000 %\n",
      "     epoch 31, minibatch 2500/2500, test error of best model 3.630000 %\n",
      "epoch 32, minibatch 2500/2500, validation error 3.320000 %\n",
      "     epoch 32, minibatch 2500/2500, test error of best model 3.610000 %\n",
      "epoch 33, minibatch 2500/2500, validation error 3.300000 %\n",
      "     epoch 33, minibatch 2500/2500, test error of best model 3.600000 %\n",
      "epoch 34, minibatch 2500/2500, validation error 3.280000 %\n",
      "     epoch 34, minibatch 2500/2500, test error of best model 3.550000 %\n",
      "epoch 35, minibatch 2500/2500, validation error 3.300000 %\n",
      "epoch 36, minibatch 2500/2500, validation error 3.280000 %\n",
      "epoch 37, minibatch 2500/2500, validation error 3.270000 %\n",
      "     epoch 37, minibatch 2500/2500, test error of best model 3.480000 %\n",
      "epoch 38, minibatch 2500/2500, validation error 3.250000 %\n",
      "     epoch 38, minibatch 2500/2500, test error of best model 3.470000 %\n",
      "epoch 39, minibatch 2500/2500, validation error 3.220000 %\n",
      "     epoch 39, minibatch 2500/2500, test error of best model 3.450000 %\n",
      "epoch 40, minibatch 2500/2500, validation error 3.190000 %\n",
      "     epoch 40, minibatch 2500/2500, test error of best model 3.400000 %\n",
      "epoch 41, minibatch 2500/2500, validation error 3.160000 %\n",
      "     epoch 41, minibatch 2500/2500, test error of best model 3.370000 %\n",
      "epoch 42, minibatch 2500/2500, validation error 3.130000 %\n",
      "     epoch 42, minibatch 2500/2500, test error of best model 3.340000 %\n",
      "epoch 43, minibatch 2500/2500, validation error 3.100000 %\n",
      "     epoch 43, minibatch 2500/2500, test error of best model 3.330000 %\n",
      "epoch 44, minibatch 2500/2500, validation error 3.090000 %\n",
      "     epoch 44, minibatch 2500/2500, test error of best model 3.280000 %\n",
      "epoch 45, minibatch 2500/2500, validation error 3.090000 %\n",
      "epoch 46, minibatch 2500/2500, validation error 3.060000 %\n",
      "     epoch 46, minibatch 2500/2500, test error of best model 3.250000 %\n",
      "epoch 47, minibatch 2500/2500, validation error 3.050000 %\n",
      "     epoch 47, minibatch 2500/2500, test error of best model 3.240000 %\n",
      "epoch 48, minibatch 2500/2500, validation error 3.040000 %\n",
      "     epoch 48, minibatch 2500/2500, test error of best model 3.210000 %\n",
      "epoch 49, minibatch 2500/2500, validation error 3.030000 %\n",
      "     epoch 49, minibatch 2500/2500, test error of best model 3.180000 %\n",
      "epoch 50, minibatch 2500/2500, validation error 3.040000 %\n",
      "epoch 51, minibatch 2500/2500, validation error 3.030000 %\n",
      "     epoch 51, minibatch 2500/2500, test error of best model 3.130000 %\n",
      "epoch 52, minibatch 2500/2500, validation error 3.000000 %\n",
      "     epoch 52, minibatch 2500/2500, test error of best model 3.120000 %\n",
      "epoch 53, minibatch 2500/2500, validation error 3.000000 %\n",
      "epoch 54, minibatch 2500/2500, validation error 3.000000 %\n",
      "epoch 55, minibatch 2500/2500, validation error 3.000000 %\n",
      "epoch 56, minibatch 2500/2500, validation error 3.000000 %\n",
      "epoch 57, minibatch 2500/2500, validation error 2.990000 %\n",
      "     epoch 57, minibatch 2500/2500, test error of best model 3.060000 %\n",
      "epoch 58, minibatch 2500/2500, validation error 2.980000 %\n",
      "     epoch 58, minibatch 2500/2500, test error of best model 3.040000 %\n",
      "epoch 59, minibatch 2500/2500, validation error 2.940000 %\n",
      "     epoch 59, minibatch 2500/2500, test error of best model 3.030000 %\n",
      "epoch 60, minibatch 2500/2500, validation error 2.950000 %\n",
      "epoch 61, minibatch 2500/2500, validation error 2.930000 %\n",
      "     epoch 61, minibatch 2500/2500, test error of best model 3.020000 %\n",
      "epoch 62, minibatch 2500/2500, validation error 2.950000 %\n",
      "epoch 63, minibatch 2500/2500, validation error 2.960000 %\n",
      "epoch 64, minibatch 2500/2500, validation error 2.960000 %\n",
      "epoch 65, minibatch 2500/2500, validation error 2.950000 %\n",
      "epoch 66, minibatch 2500/2500, validation error 2.920000 %\n",
      "     epoch 66, minibatch 2500/2500, test error of best model 3.030000 %\n",
      "epoch 67, minibatch 2500/2500, validation error 2.910000 %\n",
      "     epoch 67, minibatch 2500/2500, test error of best model 3.020000 %\n",
      "epoch 68, minibatch 2500/2500, validation error 2.900000 %\n",
      "     epoch 68, minibatch 2500/2500, test error of best model 2.990000 %\n",
      "epoch 69, minibatch 2500/2500, validation error 2.900000 %\n",
      "epoch 70, minibatch 2500/2500, validation error 2.860000 %\n",
      "     epoch 70, minibatch 2500/2500, test error of best model 2.970000 %\n",
      "epoch 71, minibatch 2500/2500, validation error 2.860000 %\n",
      "epoch 72, minibatch 2500/2500, validation error 2.870000 %\n",
      "epoch 73, minibatch 2500/2500, validation error 2.860000 %\n",
      "epoch 74, minibatch 2500/2500, validation error 2.850000 %\n",
      "     epoch 74, minibatch 2500/2500, test error of best model 2.960000 %\n",
      "epoch 75, minibatch 2500/2500, validation error 2.810000 %\n",
      "     epoch 75, minibatch 2500/2500, test error of best model 2.940000 %\n",
      "epoch 76, minibatch 2500/2500, validation error 2.800000 %\n",
      "     epoch 76, minibatch 2500/2500, test error of best model 2.950000 %\n",
      "epoch 77, minibatch 2500/2500, validation error 2.810000 %\n",
      "epoch 78, minibatch 2500/2500, validation error 2.810000 %\n",
      "epoch 79, minibatch 2500/2500, validation error 2.800000 %\n",
      "epoch 80, minibatch 2500/2500, validation error 2.810000 %\n",
      "epoch 81, minibatch 2500/2500, validation error 2.810000 %\n",
      "epoch 82, minibatch 2500/2500, validation error 2.810000 %\n",
      "epoch 83, minibatch 2500/2500, validation error 2.820000 %\n",
      "epoch 84, minibatch 2500/2500, validation error 2.810000 %\n",
      "epoch 85, minibatch 2500/2500, validation error 2.820000 %\n",
      "epoch 86, minibatch 2500/2500, validation error 2.820000 %\n",
      "epoch 87, minibatch 2500/2500, validation error 2.810000 %\n",
      "epoch 88, minibatch 2500/2500, validation error 2.820000 %\n",
      "epoch 89, minibatch 2500/2500, validation error 2.820000 %\n",
      "epoch 90, minibatch 2500/2500, validation error 2.820000 %\n",
      "epoch 91, minibatch 2500/2500, validation error 2.800000 %\n",
      "epoch 92, minibatch 2500/2500, validation error 2.800000 %\n",
      "epoch 93, minibatch 2500/2500, validation error 2.810000 %\n",
      "epoch 94, minibatch 2500/2500, validation error 2.800000 %\n",
      "epoch 95, minibatch 2500/2500, validation error 2.800000 %\n",
      "epoch 96, minibatch 2500/2500, validation error 2.800000 %\n",
      "epoch 97, minibatch 2500/2500, validation error 2.800000 %\n",
      "epoch 98, minibatch 2500/2500, validation error 2.780000 %\n",
      "     epoch 98, minibatch 2500/2500, test error of best model 2.840000 %\n",
      "epoch 99, minibatch 2500/2500, validation error 2.790000 %\n",
      "epoch 100, minibatch 2500/2500, validation error 2.780000 %\n",
      "     epoch 100, minibatch 2500/2500, test error of best model 2.850000 %\n",
      "epoch 101, minibatch 2500/2500, validation error 2.770000 %\n",
      "     epoch 101, minibatch 2500/2500, test error of best model 2.850000 %\n",
      "epoch 102, minibatch 2500/2500, validation error 2.760000 %\n",
      "     epoch 102, minibatch 2500/2500, test error of best model 2.850000 %\n",
      "epoch 103, minibatch 2500/2500, validation error 2.740000 %\n",
      "     epoch 103, minibatch 2500/2500, test error of best model 2.850000 %\n",
      "epoch 104, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 105, minibatch 2500/2500, validation error 2.770000 %\n",
      "epoch 106, minibatch 2500/2500, validation error 2.780000 %\n",
      "epoch 107, minibatch 2500/2500, validation error 2.760000 %\n",
      "epoch 108, minibatch 2500/2500, validation error 2.770000 %\n",
      "epoch 109, minibatch 2500/2500, validation error 2.770000 %\n",
      "epoch 110, minibatch 2500/2500, validation error 2.780000 %\n",
      "epoch 111, minibatch 2500/2500, validation error 2.770000 %\n",
      "epoch 112, minibatch 2500/2500, validation error 2.760000 %\n",
      "epoch 113, minibatch 2500/2500, validation error 2.760000 %\n",
      "epoch 114, minibatch 2500/2500, validation error 2.760000 %\n",
      "epoch 115, minibatch 2500/2500, validation error 2.760000 %\n",
      "epoch 116, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 117, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 118, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 119, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 120, minibatch 2500/2500, validation error 2.700000 %\n",
      "     epoch 120, minibatch 2500/2500, test error of best model 2.800000 %\n",
      "epoch 121, minibatch 2500/2500, validation error 2.680000 %\n",
      "     epoch 121, minibatch 2500/2500, test error of best model 2.800000 %\n",
      "epoch 122, minibatch 2500/2500, validation error 2.680000 %\n",
      "epoch 123, minibatch 2500/2500, validation error 2.680000 %\n",
      "epoch 124, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 125, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 126, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 127, minibatch 2500/2500, validation error 2.690000 %\n",
      "epoch 128, minibatch 2500/2500, validation error 2.690000 %\n",
      "epoch 129, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 130, minibatch 2500/2500, validation error 2.710000 %\n",
      "epoch 131, minibatch 2500/2500, validation error 2.710000 %\n",
      "epoch 132, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 133, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 134, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 135, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 136, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 137, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 138, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 139, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 140, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 141, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 142, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 143, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 144, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 145, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 146, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 147, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 148, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 149, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 150, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 151, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 152, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 153, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 154, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 155, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 156, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 157, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 158, minibatch 2500/2500, validation error 2.710000 %\n",
      "epoch 159, minibatch 2500/2500, validation error 2.710000 %\n",
      "epoch 160, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 161, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 162, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 163, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 164, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 165, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 166, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 167, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 168, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 169, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 170, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 171, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 172, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 173, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 174, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 175, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 176, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 177, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 178, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 179, minibatch 2500/2500, validation error 2.740000 %\n",
      "epoch 180, minibatch 2500/2500, validation error 2.750000 %\n",
      "epoch 181, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 182, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 183, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 184, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 185, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 186, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 187, minibatch 2500/2500, validation error 2.730000 %\n",
      "epoch 188, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 189, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 190, minibatch 2500/2500, validation error 2.720000 %\n",
      "epoch 191, minibatch 2500/2500, validation error 2.710000 %\n",
      "epoch 192, minibatch 2500/2500, validation error 2.710000 %\n",
      "epoch 193, minibatch 2500/2500, validation error 2.710000 %\n",
      "epoch 194, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 195, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 196, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 197, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 198, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 199, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 200, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 201, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 202, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 203, minibatch 2500/2500, validation error 2.700000 %\n",
      "epoch 204, minibatch 2500/2500, validation error 2.690000 %\n",
      "epoch 205, minibatch 2500/2500, validation error 2.680000 %\n",
      "     epoch 205, minibatch 2500/2500, test error of best model 2.550000 %\n",
      "epoch 206, minibatch 2500/2500, validation error 2.680000 %\n",
      "epoch 207, minibatch 2500/2500, validation error 2.680000 %\n",
      "epoch 208, minibatch 2500/2500, validation error 2.690000 %\n",
      "epoch 209, minibatch 2500/2500, validation error 2.690000 %\n",
      "epoch 210, minibatch 2500/2500, validation error 2.690000 %\n",
      "epoch 211, minibatch 2500/2500, validation error 2.690000 %\n",
      "epoch 212, minibatch 2500/2500, validation error 2.680000 %\n",
      "epoch 213, minibatch 2500/2500, validation error 2.670000 %\n",
      "     epoch 213, minibatch 2500/2500, test error of best model 2.550000 %\n",
      "epoch 214, minibatch 2500/2500, validation error 2.670000 %\n",
      "epoch 215, minibatch 2500/2500, validation error 2.670000 %\n",
      "epoch 216, minibatch 2500/2500, validation error 2.670000 %\n",
      "epoch 217, minibatch 2500/2500, validation error 2.680000 %\n",
      "epoch 218, minibatch 2500/2500, validation error 2.680000 %\n",
      "epoch 219, minibatch 2500/2500, validation error 2.670000 %\n",
      "epoch 220, minibatch 2500/2500, validation error 2.670000 %\n",
      "epoch 221, minibatch 2500/2500, validation error 2.660000 %\n",
      "     epoch 221, minibatch 2500/2500, test error of best model 2.540000 %\n",
      "epoch 222, minibatch 2500/2500, validation error 2.660000 %\n",
      "epoch 223, minibatch 2500/2500, validation error 2.660000 %\n",
      "epoch 224, minibatch 2500/2500, validation error 2.650000 %\n",
      "     epoch 224, minibatch 2500/2500, test error of best model 2.520000 %\n",
      "epoch 225, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 226, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 227, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 228, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 229, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 230, minibatch 2500/2500, validation error 2.660000 %\n",
      "epoch 231, minibatch 2500/2500, validation error 2.660000 %\n",
      "epoch 232, minibatch 2500/2500, validation error 2.660000 %\n",
      "epoch 233, minibatch 2500/2500, validation error 2.660000 %\n",
      "epoch 234, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 235, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 236, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 237, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 238, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 239, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 240, minibatch 2500/2500, validation error 2.650000 %\n",
      "epoch 241, minibatch 2500/2500, validation error 2.650000 %\n",
      "Optimization complete. Best validation score of 2.650000 % obtained at iteration 560000, with test performance 2.520000 %\n"
     ]
    }
   ],
   "source": [
    "param = {\n",
    "    'data': PREFIX+'/ml-bootcamp/mnist/mnist.pkl.gz',\n",
    "    'lr': 0.01,\n",
    "    'L1_reg': 0.00,\n",
    "    'L2_reg': 0.0001,\n",
    "    'batch_size': 20,\n",
    "    # number of words in the context window\n",
    "    'nhidden': 50,\n",
    "    'nepochs': 1000}\n",
    "\n",
    "test_mlp(param)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}