from_scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Networks from Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "~~Plagiarized from~~ Inspired by [Andrej Karparthy's cs231n](http://cs231n.github.io/neural-networks-case-study/) and [Colah's Blog](http://colah.github.io/posts/2015-08-Backprop/).\n",
    "\n",
    "This notebook recreates neural network algorithms using only the linear algebra library `numpy` (uses other libraries for performance benchmark and visualization). The sample data is a 2-dimensional data with three classes in a form of spirals. \n",
    "\n",
    "The content consists of the following aspects of neural networks:\n",
    "\n",
    "* Weight initialization\n",
    "* Loss functions\n",
    "    * Cross-entropy loss\n",
    "    * Hinge loss\n",
    "    * Regularization loss\n",
    "* Layers\n",
    "    * Linear\n",
    "    * Sigmoid\n",
    "    * Tanh\n",
    "    * ReLu\n",
    "    * Swish\n",
    "    * Softmax\n",
    "    * Hierarchical Softmax (TBD)\n",
    "    * SVM\n",
    "    * Dropout\n",
    "    * Batch Normalization\n",
    "    * Convolution (TBD)\n",
    "    * Pooling (TBD)\n",
    "    * RNN (TBD)\n",
    "    * LSTM (TBD)\n",
    "    * Embeddings (TBD)\n",
    "    * Capsule (TBD)\n",
    "* Training\n",
    "    * Linear > Softmax\n",
    "    * Linear > SVM\n",
    "    * [Linear > Activation]++ > [Linear > Output]\n",
    "* Optimization\n",
    "    * Gradient Descent (GD/SGD/Mini-batch SGD) (TBD)\n",
    "    * RMS Prop (TBD)\n",
    "    * Momentum (TBD)\n",
    "    * Adam (TBD)\n",
    "* Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "#usual data science stuff\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix, f1_score, classification_report\n",
    "from sklearn.metrics import roc_curve, roc_auc_score, accuracy_score\n",
    "\n",
    "#utils\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Generate Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The `spiral_gen` function generates a 2-dimensional set of data in a nonlinear shape with 3 classes and 3000 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsnXecFOX9+N/PzJbba9zB0Y7jKEdH\n6SCKvQU1sSTGYNTYEmISE5PoL9EUjRqNmsToV03sPXZFQTAIKEWkHb1IL8cBBxx3x7W9LTPP749n\n97iye3XbHfN+vfYFOzM78+zezPN5Pl1IKbGwsLCwsAiixXsAFhYWFhaJhSUYLCwsLCzqYQkGCwsL\nC4t6WILBwsLCwqIelmCwsLCwsKiHJRgsLCwsLOphCQYLCwsLi3pYgsHCwsLCoh6WYLCwsLCwqIct\n3gNoC1lZWbJ///7xHoaFhYVFh2L16tXFUsruzR3XIQVD//79yc/Pj/cwLCwsLDoUQoh9LTnOMiVZ\nWFhYWNTDEgwWFhYWFvWIiGAQQrwshDgihNgUZr8QQvyfEGKnEGKDEGJcnX03CiF2BF43RmI8FhYW\nFhZtJ1Iaw6vA1Cb2XwIMDrymA/8BEEJ0Be4DTgMmAfcJITIjNCYLCwsLizYQEcEgpVwMlDRxyBXA\n61KxHMgQQvQGvgXMk1KWSClLgXk0LWAsLCwsLKJMrHwMfYD9dd4XBraF225hYWFhESc6TLiqEGI6\nygxFbm5unEdjEW82rz/Eh/9dx8H9x3Gl2Jl8Vn+umjYah7PD3NIWFglLrDSGA0DfOu9zAtvCbW+E\nlPJ5KeUEKeWE7t2bzc+w6IB4vQZfL9rNWy/ns+CzbVRVekMel79sH088/CW7thfjdvsoKa5mzowt\n3PbDd1ifH/L2sbCwaAWxEgwzgR8FopMmA8ellIeAucDFQojMgNP54sA2i5OMkmPV/O5nH/Pqf1Yw\nd+Y3vPPqau6c/hF7dh6rd5yUkjdfzMfrMRqdwzAkTz26kLKS6lgN28KiUxKpcNW3gWXAUCFEoRDi\nViHEbUKI2wKHzAF2AzuBF4CfA0gpS4AHgVWB1wOBbRYnGa88s4zjpW48NX4AvB4Dd7WPpx5dhJSy\n9riKcg8V5TVhz2OakqULd0d9vBYWnZmIGGSllNc2s18Cvwiz72Xg5UiMw6Jj4vX42bz+EKYpG+2r\nrPCwf18Zuf1VFLMzyQaND6vFMCSlxyyNwcKiPViZzxZxxzBl2LleCIHX469973TaGHtaX7Qwd25S\nko2hI3tGfpAWFicRVgiHRZvZsOYA8z7dSvnxGkaN78NFlw0jvUtSq8/jctnp07cL+/eWNdonBPQf\n2LXetpt/NpmiA8cbHa/rgq7dUxg7qS8WFhZtR9S133YUJkyYIK3qqrFFSsmKr/by2cdbOHa0CtOU\nuKu9mKbab7druJId3P/4ZXTtltzq8+/cdpTH7p2Pz2cok5IAh0Pn5p9P5oxzBoYcz8ql+/j0w00c\nLDyOzaZxxrkD+f71Y0lOcbT361pYdEqEEKullBOaPc4SDBYt4b3X1zB/9jY8dcw6DdE0wZnnDeTW\nX57RpmsUHShn9keb2L3zGD17p3HpVSMZNNQKTbawiBQtFQyWKcmiWcpK3cyd+Q1+v9nkcaYpWbOy\nkFvbeJ1efdLbLFQsLCwihyUYLHBXe9mz8xiuZAf987oihKi3f9G8Hc0KhSDh3cgWFhYdBUswnOTM\nnrGZj99ej27TME2J3aHTp28XumS6GH9aX5JTHMyfvbXF56uq8PKnO2ZxyVUjOf3sAQgBSxbs4rOP\nt1B+vIaBQ7px9XVj6dfAoWxhYZE4WD6GkxS/z2DhvB28++oavN7GWcRBNI1aB3NrcDptTJySS3KK\ng4Wf7ziRqRxwKt/94MXkDclq4+gtLCzaguVjsAjLgs+28f4ba6lx+2huXdAWoQDg8fhZsWQfpjQx\n/HUuIlVW89uv5POnv1kV1i0sEhFLMJxkLFu8h7dfWY2vCS0hUhiGiaYJQqUq795eHPXrW1hYtA1L\nMJwkSCn59INNfPjWuma1hEih6SLsPmeSPTaDsLCwaDVWSYyThM8/3crMDzbGTCiAKmeR5GosAOx2\nnXMvHhy7gVhYWLQKS2PoJBiGydKFu1k0byeG32Ty2QM471uDcTptSCmZ9f7GkKWq24vTaeOam8ay\nZvl+dm4txuv1Y7PpIOAnvzqDXtnpPHrvPAzDxDAkQsCgod256trRER+LhYVFZLAEQyfANCVPPPQl\n2zYfxhOY/A8UlLFkwU7ue+wSgLBNb9qDpgvOPH8gF0wdygVTh7Jj61G2bCgiOdnOpDP7k5HpAuDJ\nV65mff4Bykrd5A3JYsCgbhEfi4WFReSwBEMnYPP6Q2zbcqRWKIDqhnakqIIlX+wkvYsLwpv724QQ\nggf+eRl9A+WwAYYM78GQ4T0aHWu360w43WrHamHRUbAEQycgf1lBbYObuqiw0DWYholpRM65YLNp\nPPjEt8nO6RKxc1pYWCQOlmDoBOzfWxp2X7iwVCEgNd2Ju8qL39+00Ehy6Zw/dRglxVWMGp/NkOE9\nSM9wtWvMFhYWiUtEBIMQYirwJKADL0opH2mw/1/AeYG3yUAPKWVGYJ8BbAzsK5BSXh6JMXVmykqq\nee+Ntaxevh8hoMbta/U5nEk2Hn/+u7z/5loWz99JTY0fTYhGXdTsdo2zLxzM928YyyfvbeC1Z1ci\npURKOPuCPH54ywRsdj1SX83CwiIBaLdgEELowDPARUAhsEoIMVNKuSV4jJTyN3WO/yUwts4p3FLK\nMe0dx8lCdZWX++6aQ/nxmnaZh5xOGw6njetunch1t05ESsmubcX8/f4FSCnxeQ1VNyk3g6uvG8Os\n9zcyZ8bmepFNSxbswus1+LFVEbWDYAIlQDWQAnQl4s4ni05BJDSGScBOKeVuACHEO8AVwJYwx18L\n3BeB63Z6pJSNKp0unr+TynJPu4SC3a5x1oWD6m0TQjBoWHeeePl75H9dwPEyFUE07JSeGIZkzsdb\nGoW7er0GyxbvYdqN40lNd7Z5PBaxoAZYA/hRAkIDHMC4wL8WFieIhGDoA+yv874QOC3UgUKIfsAA\n4Is6m5OEEPmoO/YRKeXHERhTh+bLudt5/421VFV6SXLZ+c7VI7n0qlOocfv4+N0NLS6BHQpnko2c\n3Ayu+P6pIfe7XHbOuiCv3rbKihoMI/Q17XadI4crLMHQCIlanR8ADCALyEZZW+PBJsBT570BuIFv\nACunxKI+sXY+TwM+kFLWXXr2k1IeEEIMBL4QQmyUUu5q+EEhxHRgOkBubucLffT5DJYs2Mkn722k\nrMRdu73G7eP9N9dx9HAlm9cX4a5uvT+hLjdMn8iUc/MCNYxaRmqqA02EPt7nM8jqntKuMXVOdgIH\nUatzgHKUkJhA7B+7GqAqzL5SwAe0pERJUEu1zE+dnUiUxDgA1O2+nhPYFoppwNt1N0gpDwT+3Q0s\npL7/oe5xz0spJ0gpJ3Tv3rnaPfp9Bn/74+e8/fLqekKhFgkLP99JSXG4h7vlrFt1oFVCAcBm17nw\nsqE4nPVXu3aHzvjT+loRSo2oor5QIPB/D/WV6/biB4qBYygNoKnjwv3NRTOfBfACm4FFqEd0NVDR\nmoFadDAisXRZBQwWQgxACYRpwA8bHiSEGAZkAsvqbMsEqqWUHiFEFjAFeCwCY+pQrPhqH4X7yprs\niwBgRCAXYc2K/fh8BvZWRhJdfd0YvF6DhZ/vQNc1DL/BhMm53PKLye0eU+ejmFAVZZVwOIyypoIS\nFH7ARevXaAdQWkndCX8kEMwqlyht4EiYsQSxAU2ZAU2Ub6KmznnKgXyUFXlQG8Zukei0WzBIKf1C\niNuBuSgD6stSys1CiAeAfCnlzMCh04B3ZP3OQMOB54QQQW/YI3WjmU4WFs7bgcfTOEEtGpimZPuW\nI4wc3btVn9N0jet/PJHvXTeG4sOVZHZLJjXtZPcrBG32DX8HSfjJWENNspuBStTELoA8lA+iJRxH\nCYWGfp9NKPeeM3D+EprXBnJo2jR0DKUxhPo+B1DfZVTzQ7boUETE2CmlnAPMabDt3gbv/xLic18D\nob2gJwmfz/qGHd8ciek1j7XDJOVy2euVwTg5qUQF3VWjJtUkYBjQBaUBHAzzOQ3oBaxFCZW6k+0O\n1ITekjpShTQWCgS2bQxc41iYYxqyF6WxBEuZlAHbUY5pOyrtqCnhUoLSkKxufJ0JK/M5jhw7WsV7\nr69tUylsTWucjNYiBAzIs4rYtQ0DZQrawYlJV6IExHpU5PYh1Ao7FK7Ay0fjFbgZOG8GjSOXgsce\nBPYEPh+OSpQ20VJMYBtqYj+EEgpBPIFX6GZLJ8a2EWUlPrXO2GuAXSgBpQE9USY0a8rpCFh/pTiy\nenkBTdt/wzNmYg4Xf3soj//1y1aV0+7RM5WcfhltuubJTQWwDiUcwvkPCgnvXwDllA5lAgriBr7i\nhK/gAGpF375ItOaRqO+3vYn9zVGK0qJyUMJpDyc0DQP1XUpRUVmWTyLRsQRDHPH7zTY5lG12jaEj\nejD81N707J0eslaS0KBrt2SOl9YgAU1AekYSv7v/wkZJcxahqAGKUJNyBmpV3ZQfSKImxOYmvZpm\n9psoX0EayskbCwyUUGsvxajJP9RCRaK++zGgc0UVdkYswRAj1q0q5KO313OkqIKevdO4/JpTmTtr\na5vMQQLBpCn9APjJHWdw729mNzpGmlBZ4eVPj0ylcF8Z3bonM+yUXq0OVT05OQxs5YQTuWHoaThS\nUCv9XbRVE1RIYicUgkTKz9WU9mqgfBiWYEh0LMEQAxbP38kbz6+sDUfdu6uEpx5djGyLjwDok9uF\nkuJqumalYPhNklw2atyNV7OmKcnsltyoMc7Rw5WUH3eTndMFV7JVDqE+PpRQaJiD0BKyUXkKMeyf\n2qEIluGwSHQswRBlDMPknVdXN8pRaKtQACVYHr1vHmeeN5Crrh2NEaZEhq5r9UJKj5e5eerRRezd\nVYLNpuH3m0y9fDjfu26MZV6q5Vg7PluE0jYsQmOi/CxlqAiupu65GpSm5g4c2wtruood1i8dZY4e\nrmxXbaNweD0GixfsYuzEvpx2Zn9WLN1Xr/eCw6lzyRXDsdmUzVtKyT/+soDC/WWYhqw9du6sb8js\n6uKCS4dFfIwdEw8t1xAaUhDJgXRSDgdeAhXBJFETfx6QGjimBBXpFDTlFaOc8ONQ5igbKrrLIlpY\n4QFRJjnFEbYAXXvx+0z+9dCXnHZmf848dyB2h44zyYbTaWPq5cO5/JoTiUd7d5Vw+FBFo6qsXo/B\nu6+t4fNZ31BV6Wl4iZOAatQK1h94WZN7bJCo39tACYJ8lDZhopLzTE6Y5EyUiW8FKgdkJargQojy\nMRYRQci2BNHHmQkTJsj8/Px4D6PFPHrvPLZsKIra+Z1JNp546XvouqD8eA1dMpNxOOrHwq/6eh8v\nPrUsbFMfh0NHt2nc89eL6Tewa9TGmjjUoFalwSQ1iYo+KqPtGoNF+7ChSmzsoPmMbVD+itOx1rct\nRwixWko5obnjrF80ShwsPM4TD3/JT6e9zd7dJVG9lpSS/OUFOJPsdO+Z1kgoAPTJzWhSc/F6DdzV\nPp75+2I6zmJBokw/Lc/jOPG5dajwUjPw+WATG0soxA8/yvHf0r9nUNuwiDSWjyEKHD5Uzv3/bw6e\nGn+bsppbi89nUFnRtBkoO6cLQ0f2YNumI/h84R+80pJqDh+qoFd2eqSH2Q58qEncjgoJFShH705O\nJJx1B4ZS/5b2BV5J1F8DlRE+O9mi42AS3pwkUfeGjlUmvPVYgiGC+P0mO745wifvbQgZPhotNE1j\n2MiezR73q7vP5b8vrGLpot34faFXxkII/E0IjtgiURm0dUtVJ6Gqeu6i/ur+KEp7GIfK4t2EMhcJ\nlFAYwInq8NW0XsuwSEwa9gKRqPtlH+pvrKH+7v2xBETLsQRDhNi8/hBPP7YY05Rh7fjRIm9IFgMH\nN1/EzOm0ccvtp3P99In88VezOFJUGfKY7L6JUjKjCOUMrqt2VaNs0A0JlnUobLA/uHLcjbrdywLn\ntej4aJxwVoOqx1SJEgrBRYOBuocMlP/CoiVYPoYIUFpSzRMPf0l1lTfmQqH/wK78/oGLWvUZh8PG\nT+6YgtNpq82E1jSBw6lz6y9PT6Ds6D20LllMoDSJUJioWkCWUOg8OFBC4UjgtRF1zzTUhk1UrabY\nafEdHUtjiABLFuxqW6XTduJw6Hz3utG1uQqtYcjwHjzwr8v47OMt7N19jJy+GUy9YkSCldRubfis\npGnnseVY7lzUbR7UHCJwfGqdbUbgZccyM9XHEgwRoPhIZVibfTTRdIHN1vbm8r2y07n554ncga2p\ncs+hsBO+qYxF+7ABZ6JqOK2J81iCtObvHOwFVojKl6hAmZ1AaR6DONGTwsISDK3ANCWffbyZ/33y\nDZUVHnrndGHaTeMYMrwHy5fsxVMTW1XVNCVDRtS/mctK3WxaexBd1xg1vg8pqR25Nk0GqlpnSxmD\n6kfcGU0GKagIq3hFUwWje7oEXsfjNI62IlGJcaGEiQf4BjUdngw5PM0TER+DEGKqEGKbEGKnEOLu\nEPtvEkIcFUKsC7x+XGffjUKIHYHXjZEYT7R4/dkVfPzuBsqP12CakgMFZTz1yCKSXDZS05xoemzV\n0WGn9KzXu/nTDzdx1/SPeP35lbz67HLuuOUDvl60O6ZjiixDaJmKL1CRSkm0rANaPBFAP2A0MLYV\nn5MojSgeaEDdVrDNlQ5PVJrSMExUgIIFREAwCCF04BngEmAEcK0QYkSIQ9+VUo4JvF4MfLYrcB+q\nUe0k4D4hRCIZuWspK3Wz5MtdjZrieL0Grz23kt8/cBETT89Fj5Fw0DToN+DE6mbrpsN88t4GfD4T\nT42fGrcfn9fg5WeWc/hQrEs4l6ASyJajVmLVbTxPMs1HkmgoYVAGLCLxi9idAgxErUwzaHmfZzdt\n/x1FnVdr0VC9IXLrbOushoa2/r6dj0hoDJOAnVLK3VJKL/AOcEULP/stYJ6UskRKWQrMA6ZGYEwR\nZ9/uknqr87qUl9Xwh1/N5OwLB5HVI5VYFCrVbTpnnpdX+37e7K2NKrgCmIbJ4vnhInWiwX5UdEgp\najIrQtXBqWjj+brQuNVlXbJRRdba3sc6dgygcW/kfjT9/YIEC8o1hUAVl6v7WGuoXtKn0frHPQPV\nrnNsgzHmhj68RcRL62kJdlSo80aUL6IzmiRbRiQEQx/qZyAVBrY15HtCiA1CiA+EEMFMo5Z+Nu5k\nZLqajDwKFrQrLamOerazzaZx7c0T6NXnRHby8VJ3yHnDMCTlZbEqNuZHqeMNHfEGoXMPWkJ1iPMF\nEUSm81i0EajJtH+IfUmoiTeN9kXGpKCcw6ehTHBpgW39gIkoB2trAiQEKpO8a4hx9SR0sx0nSvg1\n9T1iG87dOjycaM+6C1W072QsLBk7nXAW8LaU0iOE+CnwGnB+a04ghJgOTAfIzW3PiqVt5A7IpFv3\nFA7uD+90k2b042E0TfD7By9iyPD6TudR4/uwd3dJvdLboArsjRjdm9hwnPCTwnHUr9Oaye8IJzqp\nhSLRoo8EyvSVidKWilEr7Z5AUyVG0lC9kP0ozWcdjSfxoMnsWIh9Oaiy1cF1Xm/q+wRA/VYaLcv4\n1lACITnMfoEyiVWiJlIDFdGTxYkKtR0xs7zu/WSiHP07UT24Ty4ioTEc4EStAVB36YG6B0gpj0kp\ng6L3RWB8Sz9b5xzPSyknSCkndO8e+9aAlRUezjo/r8ljpJRRK7EdxO7QGwkFgPOnDiE52V7PAW6z\naXTrnsLE02MlSJsyiTQnEEyUf2ATShiU07jsRaIjUBNjMmqSHAYMpmmhUBcbynTWH/VoBn8zDdWo\nZgRK8DgD+1JQpp7BNP8oC9Sj1vA4ERhvUp1rZdOyyTAV9R1HojQIgTLHjK1zvqbG0xEojvcA4kIk\nNIZVwGAhxADUpD4N+GHdA4QQvaWUhwJvL0d5JAHmAg/XcThfDNwTgTFFlK++2MWrz65oNiNY1wXZ\nOV3Yt6c1IZatI1wyW2qakwcev4wP31rP6uUFaLpgyjkDueIHo7CF8Y1Eni6EXpUK1Ioy3O9noGLj\n65qNikg8jaA5TFQ5Bo322eH7oSbao6jfIIsTiVl9aLu1tT/KlHOIEzkiXVATux01/rY6qeuSBkwG\nFjZxTCZKiMY6MKK1dLR7MDK0WzBIKf1CiNtRk7wOvCyl3CyEeADIl1LOBH4lhLgcdSeUADcFPlsi\nhHgQJVwAHpBSJlQd3SNFFbz67IpGJpqGaBo4XXYK9kZPKABUV3lZ+uVuppw3sNG+jK7J3Hr76dx6\n++lRHUN4BGoFu54TWcg6aoU7uInP7aexL6GjPpBB4dCX9k2wySgBEUkEyv8wAPV7O6m/so9khZxg\n8cJwGl9QM1pJYgcOSOArlKDuR/OaUOfAatTTDB+9tY5PP9wc0kQkAour5GQHXbNcHNxfjmFE//e0\nOzTu/PMFDD+1V9Sv1Tb8KLOQB7V67EbTk84KOleooACmkNgROLFgLSqMuCEaSkvJQvlMgm08OwI9\nUePuRsuiyRILq1FPhKgor2nSb5CUZMcwJPv3Ho+JUADweU1mvr8xJtdqGzaUuWMgaqXV3G3WUSaF\n1tCRfCPRYgCh//YOTiQidkP5KTqKED0MbAGW0nxWvg9lXd+N8lV0nPu8s2aqRIxTxmSzdOGekOUu\npAR3dXzC7w4famteQCLSExXJ0lkmU4lK7hvOyV1/JwOlGWxHTZLB9qnDqW9m64W6B2pQU5IdZV46\nSMvCkQXKZxErK3SwlPs61OInlFO/DNhAfZNqEqpfSOJPu4k/wjgzZmIOvfukc6CgDF+gUJ4QRD1X\noXvPFIqPVIW9Tt9+idIzoT0EI5GOxXsgbUSgVr+hCveZqBiLTDrOajgaBM0uHtTkGO63CCbnBUlB\naRwtEQwZnCiIF2t2o3xkySi/SV/UtLqR+kEYBspcuhOlISU2limpGXRd4w8PXcyl3x1Jt6xkumQk\nkdHV1fwH20lmtxTsIXo3A9jtGpdfMyrqY4guErWi2o7Kio6HtqCj4v/zaHlNph4oYRAs13E6yokb\njpMz3LE+ArVabq2AtKG0iab2D0QFPMSzVasPladTgDIxfUNos5FE5eYkPpbG0AzrVhXywX/XcfhQ\nOZndkrli2ihWLS2g9Fh0s4mrq7zc89eLee3ZFezdpVRkISCzWzI3/WwyeUOa79iW2BxDPUzxNB/Z\nUKkzGsoW3FRxOA0V0RMqWTDcd5CczGUVIsMQ1MRbyokQ21RgFPUFjZPEyVI+Qvg1d8cwl1qCoQm+\nXrSbV/69vLZw3uGDFbz89PKoX1fTBMNO6cnAwVnc/8/LkFLirvbh9fjpkulCxKIYU9Q5SvwfklNR\ndukdNF8xdBChhQIoU8mhENsFVhnn9qKjhIAb5Xdw0bjPMyiz03bq31Ot7ecRScLd2x3DBGwJhjCY\npuTtV1Y3qqYaq2tfeuWJzFMhBMkpDpJTOnJvhYbE24qpoZKrdtK8gGrKNg4qHr8YpR0EJyINZXYK\nNYlZtB4X9X0QDemNsuPXbe2ZhTJTxqpWWF2CiYJ17y2dpvN5EgdLMIShvMyNuyp+Bb8yu4WrU9MZ\nKCMxbK2FtExrCUbThCMJVahuH8pEZkeZqHq2d4AWrSIHVc7Di/obSFTjpnjgQmmZ+1Emri6oBLno\n+ycjgSUYwuBKthOv5D8hlDnJ4/Ez4611LF6g+kAMO6Un024aR06/hGxZ0UI8qMzoeJuR+qKchc2h\noR7o5rQ1J8oebhFfNE5kJ28mPtoCqEVBNxK/cVRo4q3PJyzOJDvjTuuLzR77nygnNwMpJY/dO4/5\nc7ZTVenF5zPYuO4gD979P4oOJnp9maY4GO8BoEJIB9L0ZO9EaQmnELpctkViY3Ci1lSsCZoRj6Jq\nfhWTOI7xlmEJhia4+eeT6Z/XFYdTx5kUG+VK0wQ/v+tstm46zP59Zfh8dXwcErweg5nvJXLWc3O4\niZ+2kISqUDo68D6X0I+ACxWGOpaOuuKziKdGmoGqAbUJFbq6EVhG0yXkEwvLlNQEySkO/vzIJezZ\neYz9+0p55d/LMaNc9mL6r6eQ3bcLy5fswetpHOpompKtmxO9fWVTpBG/iKRJ1K9v0wcVjXSAExEs\nLlQUTGeI/DqZsaE0wnis1ENlYEtUMmcSqgz7UdSipLleHfHBEgxhOLj/OAcPHKdX7zQGDOrGgEHd\n2LDmIGuWF0S1JtJHb63DU+Pj0w83hc167pLRkSs89kY5aWMtGEbQuOhZsLFOP1T0ioMT5a0tOjYC\nFQG0hfj7s4KYwF7qRysdRC1QmuttHlsswdAAd7WXJx9eyK7txeg2DcNv0i+vK7/54/n8aPokCnaX\nUFbqxlPjj0ppjNJjbt58YVVY4eN02rjkyo7cUSqYrbotxtdtyp9gx8o36Ix0R+WqbKP5PJVY0bB3\nt4nSWHuQSJqD5WNowEtPL2fH1qN4vYZKKvMa7NlxjOefXEp6lyR+d/+F2O0aui6iUi/JME18/tAr\nHE0TXHjZUCaeEfvWpu3Dh6plEzSNxWMFtysO17SIP11RfbDTqD/daagghL7Ev5aViXJSJw6WxlCH\n6iova1ftx99gYvb7TTatO0hFeQ2vPbuC6iofphkdc5ImBGYYiTNqXDbX/GhcVK4bHQzUai1YIkCi\nTEnxKOeRyM1gLKKLhqpqWoSy8wtUvkOwHWklzZfQjjaJYu5SWBpDHSorPOha6J/EpmuUHK1i8/qi\nqAgFoYHDqXPGuQNwhCie53TaGDMxJ+LXjS5bOREyaKBu/kOEaesdZeK9KrSIL8Fe1mOBMdRvNRur\nMhXJhA5q0Em08uwREQxCiKlCiG1CiJ1CiLtD7P+tEGKLEGKDEGKBEKJfnX2GEGJd4DUzEuNpK12z\nUtD18NEomVkpyCiFmwlg2o3jmHbTeJKS7fX6S+u6IDXNyRnnDIjKtaODFxW/3XAlZBKfMtsdTaha\nxI4+xMZ4Uk3jcNWgSSuxklbbLRiEEDrwDHAJKvTjWiHEiAaHrQUmSClHAR8Aj9XZ55ZSjgm8Lm/v\neNqDzaZx1bWjcTjrr9gdTp0mLxzbAAAgAElEQVTvXHMqaelOevcJ7SCy2TVEO35N04QP31pPksvO\n/f+4lElT+tXmT5x+zkD+8o9LcCZ1pFWvh8QK+YxnWWaLxMZGfEpVpKB6M5xCYj0rkRGTk4CdUsrd\nAEKId4ArUHFiAEgpv6xz/HLg+ghcNypc/J3huJIdzHhnPSXFVWR2Teb0cweQ//U+PnhjLbpNQ9ME\nUspa57PNpnH19WNwV/uYM2Mzhl+2ydxkGCb7dpcwcHAWP7vzrAh/s1iTRHi7aTyqXh6k5X0XLE4u\nyohPox8biVpPKxKCoQ+qUlSQQlQYQDhuBT6r8z5JCJGPCll5REr5cQTG1C7OuiCPsy7IA6BgbykP\n/v6z2iqrfp+JbtNIctpxu70IBJom+PDN9Uw6sx9PvfZ9io9U8fmsrXy9cDeGKZEtFBJSqsZAnQM7\n6qY/Qn0BoaEiQQ4T2xBCAyWMOqZgqDpwFH9lDWmDstH0jteEPrEpIz4ZyfEr0tkcMY1KEkJcD0wA\nzqmzuZ+U8oAQYiDwhRBio5SyUWyhEGI6MB0gNzd24Zoz3l6Pz1u/9LbhN6n2K9OEROIN7F/19T76\n5GYgTcmyRbvRNIFhtDzaQNcFffsnlq2xfQxFOdbq9irIRtUe8hHbuklJdMRYi4q9RSyc9iCl63ch\nbDq6y8HpT9/BgGvOjdg1TL/B/lnLOLRwHa7eXRl0w0Wk9OkesfMnPnbUvRHryKDELbci2ltBVAhx\nOvAXKeW3Au/vAZBS/q3BcRcCTwHnSClD1lwWQrwKfCql/KCpa06YMEHm5+e3a9wt5Y6bP6CstOUV\nGl3JdgzDbFMfB92m8fRr3+9kfRdAZRVvRanrArUeieVqSUPZchNTbQ+H4fXxwcDrqC4qgbpapy7Q\nbDaQkuyLJzDpnz/DcHvwVdVgT3Wh2XTSBvXBcHuwpyU32djJW17FnLPuoGJPEf5KN5rTjtA1znnz\nD/S78swYfMtEwIuqZRRLwaCjDCtNtYWNPEKI1VLKCc0dFwmNYRUwWAgxABWHOA34YYPBjAWeA6bW\nFQpCiEygWkrpEUJkAVOo75iOOxldXa0SDO7qtk94drvGpnWHmDSlX/MHdxj8qDLbwd9FEn2hEJwI\nJUpTyCPRwgHDcXzbfqoKj5J56gB2vDK3sVAAMCSmoX7DwtnLKZyzAt1uw/D6lT1SUyn5Qtdw9chk\nwqPTybvuQvZ+sIjVf3qZyj1FJOdkMfpP11O6YQ/HtxdietT5gv8uvv5vTCv6AHtqx+gf0HaCIdQ6\nsRUMPVFRSl6UJb4SVY4ll0Qoy9JuwSCl9Ashbgfmon7dl6WUm4UQDwD5UsqZwN9R3/b9wOqlIBCB\nNBx4TghhopZ1j0gpt4S8UJy47Lun8ML/LY1ZJ7d49YCIHodR9v1YkY6K9tBQ8elZJKIJyfB42fvR\nEgo/XU7l3iKc3btwfOt+qvYfBU1gVLXQ/yIBKTE8dYRtQJBIv0n1wWMs/enjFC3ZyK4352FUq6Jy\nlXuKWPHLp5HIWmFQF6FrHJi7iv7fO7u9XzWBkahFSzmxNyMdpLEptQqV93Mq8S7R0m5TUjyIpSkJ\n4J7bP+FgYfR7INjtOk++8j1cLjtrVhayfPEehC4489w8Ro3P7qC9nrcR+x4MwagnHaUxjCOW7jRp\nmnhKKrCnJ6M7GocYV+47zKdn3E7NkeNII/atY1uEJrCnuuh93ljGPnATXU8dGO8RRYESVGnsUH8D\nB/ELcXaiyr5H/nmPpSmpU1NZ7uFwUfRD2ewOnetunYDLZedfD33Jti1H8NSo2kLr8w8wdmIOt/32\nzA4oHIKr91iuyIKLHQOlru8hVr12d7z6P/LveRFvWSVCEwz60cVM+tcvsCU5MH1+9s5YwvLbn8JT\nfDwm42mWcJHDpsRXXk3BzK85OG81lyx8nKwJQ2M9uihTSnht1oa6d+MRseRDRezFz4xnCYZmKCmp\nVtFFUb6ONCV+v8mqZQVs23wET51eDJ4aP2tXFvLl3O1UVXix2XUmnpFLVo/42yKbpyf1G7THGomq\nkRN9wbDr7S9Y9vMnMWpOrDR3vvY51YdKOPv1u5kx8haqDxRHfRytork5T0r81TWsvPM/XLroiZgM\nKXbYCC8Z7ai+HIdRrlMP6h5u3CMl8kjibf60TEnN4Hb7+OWP3sPni/7EZnfo9BuYyc6toScPVcZJ\n5U0IIfjBTeO46LJhUR9X+6lEqezBbOhYm090ILq2ctMw+G/G5fhb6hvoYAibzk3ez+M9jAhTA6yg\n8aJFQ7k/GwYsGKhYmxqiq0WkAhOjcuaWmpISzyuXYLhcds6/ZGi92kUtog0WH5/XCCsUQJXNMAOa\nhc9n8O5rayg60BH6P6eiQvPiZYqIfrz4xn++12mFAoA9tSM3hwpHEuqe1Bq8eqEqrzZER6Vh9Q18\nNonIRhAFQ7nj32/FEgwtYNpN47n4261cmcdAETMNk6ULd0f/Qu2mBuWA3kpstYXggxZ9x+mGv74Z\n9WvEky4jOlMIdV16oRy9g1H3yQSUsAi3srOhwp9PD7wiWedIokxVW1C+sfhhCYYWoGmCiVP6keSK\nj0smnL/ZMCTu6kQuDmeibvIVwA5i78TLQZXyiq4Tz1fpxl/ZebUFgJJ1u9j3ydJ4DyNKOFAZ+X1R\nDufWEI37ugLIRzWX2oR6fpYBO4lVpJQlGFpIVo/URg18oondoTN0ZA8mn9WfM8/Lw2Zv/KdyJtkY\nPSGRy0nvQ8Vlm8SnFk0OscgsrSk+jgjTx6OzYFR72Pj3d+M9jARDokJeo4EBFKCen2qU1l2I8nFE\nXzh07rs5gmRkuhgzIQe7PTYFzHxeg4xMF7f99kyu+/EEMjJd9YSDw6mTNySLkaN7x2Q8baOQ+Ham\nio2G5y2tQMrE6sAVDUo37sbwqmQ47/FKStbvwlPSEXxc0cCHMo3GcsEjOZEpHV2sqKRW4PH4eemp\nZaxZUYBu0/F5/RhG9H4/u0Pj+p9M4tyLBlNV6WXuzC2s+GofdofGuRcN5txvDcFmS2TZ/mXzh0SV\nc4lWNdWKvUWUbNjF9pc+o3DWsqhcIxERdp2MEf0o37YfzWHH8Pjof/XZTHnhLmxJna3GVzhM1Mrd\nTXw0YSdwRps+2dKoJEswtIHKcg+lJdVk9UxlzYoCXv33itoKq5EmO6cLf3s6rv2L2sEK4utEG0Ok\nO2MZHi+Lrn+Ygk++RvoTNGs5xuguB/2uOotz3vxDvIcSI44C3xD7sOsgNqBt/VqscNUokprupG//\nTFwuO72yu+DzRe8GqSjvyE7NQTS+xWJ5y+2J+BlX/b/nKJi5zBIKdTDcXvZ9tISaYwmSzR11yomf\nUIBY9C+3BEM7efOFlURT6RoyvGNUBQ1NN1Q4XzDSwx7YFqvb7jiRbAZk+g22vzQH6YtF9mvHQnPY\nqCo4gpQSw+vrhMUg69JUQEMXVG5DMCciGvSK0nlPYJXEaAf5ywvYvSN6je2dSTa+e92YqJ0/NnSj\nfoJZEdGL5AjFSmAskNbuM/kq3ZheSyiEwl/j4dCi9cy77B7cR8pwZKQy6u5rOeXOazpgfa/maGrC\n70P9ktpbUBn/kcIeuEZ0sTSGNlJRXsNzj38VnZMLGDm6F3/627fIyc2IzjXiRldi67AzUDHhm9p9\nXUeXFOxpyZEYVFg0hw1bSsfLMpZ+k/z/9xzuolIwJd6SCtb95XVW//GleA8tChxqYl+whHkyquz7\n+AhfewixMCVZGkMDSkuqWb54LxXlNQw7pSenjMkOWQ5j1dcFUWsfPOWcgdwwfSKu5M4Y5eFAZZju\nJrahrEeBNagS3G37wwkh6HXeGApmRG5BoDlt5Fw6mfS8Pgz60UWk5WWz76MlVO47TOboPJKzu2F6\n/RxZvplVv/lPxK4bcUyJbCB4/dU1bHnyQ0bf88OoC9TY0lSjqYb7nKiaSyGbVraSYI+R6GMJhjrk\nLy/g2ce/Ainx+Uzmz9lGTm4Gdz94EQ5n/Z/KXe3FiFLC2/Kv9rJ5/SFGjOpFcoqDnNwujBqfQ7fu\nrc3KTFT6ohrqHECp3BUxum45ypTVttwP0+fnwP9WRmQkaXnZjLjjewz76bfR7PXvrbzrLmx0fI/T\nhuMrq2Ld/a9H5PqxQmgai274G96ySnqeeSrDb7+S5F7xbULTfrqh7t2GGmjDibsSleQZCaEASsDE\nZrFohasGqK7ycsfNHzQKO7U7dC69agTfvba+rX/PzmM8/Me5MensJgTouuCcQO5Cdk6XBM9faA6J\nelgKUHbYFJSAiKQttilGoB5aDWUPdqE0ihqUqSt0YbR9nyxl4Q8eqOdnMFJcuFPTsB8toaJbd6pT\nM0iqqSLrUCFl3XvhSUomrayY1IoyhK6hu5x8Z+W/yRiW26aRm36DtQ++zqbH3sX0+uITRt9KhKYh\nTRPNacfmcnLZ10+1+fsnBh6U76quv0lDLXbGoDTSEmAjkdWKz6G91v+YNuoRQkwFnkSVH3xRSvlI\ng/1O4HWUwe0Y8AMp5d7AvnuAW1HG4F9JKedGYkytZV1+YUiTkc9rsHj+rkaCYcCgbpw6NpuNaw9G\nXThICX6/ZMFn21m8YCd2u40bpk/kjHM6aletPajszeBD4yVqdrmQ1O0eu7fBvl2oVVlf1OPRDTU2\njap9h8kc3ZNBN55KysWncO9dDSvhyvATtZRomiCtSxJZWyu4eLCJrrf+IddsOuPvv5kB3z2bLU/P\nYMerc8FoPPkIuw2hiZBtO2ONNNX4TI8Pr9fH8tv/j6nz/xHnUbUHJ6os9h7UdKajtNBcTvR32Erk\nTaWxe0barTEIIXRgO3ARJ4p5XFu3d7MQ4ufAKCnlbUKIacBVUsofCCFGAG+jKp1lA/OBIVLKJmfa\naGgMC+ft4L8v5uP1NI46SU1z8swb1zTabhomX36+g/mzt8ak9WddHE6dO++9gGEje8b0uu3HC3xN\nh1jqNkA9KhJPjclPr23KAdk8qelOnnjpe+0usfKK7cLaHs91saW56HrqQIpXb084zUJoGj+q+R+a\nLTblZWJPuD4P7WUC7Y2ui6XGMAnYKaXcHbjwO8AV1F+WXQH8JfD/D4CnhYphuwJ4R0rpAfYIIXYG\nzhfzGgOnjO5du7Kpi6YJxkwMHR6m6RoXXDKUCy4ZyprlBfzfo4uimtNQF6/HYNb7GzugYChHqcMd\nL0FMRV0KPnq7/YlcleUefvz9t+jRO5WqCi9CE4yf3JcrrxlF16yW+5IyTxlA6YbGpdel18+57/4Z\no8ZHcf42Dsxbzc6XP2v3uCOBRCJa29+kQ6ERHUl8kFj1NImEYOhD/apOhaiuLCGPkVL6hRDHUTp6\nH2B5g89GP0g3BFk9UrngsmF8+dn22raaNptGksveyIwUikHDe2CzaTHp9Bak6OAJLeXY0SpmfbiR\nTWsPkZLq4KJvD2PKuQMTMIa848c7bF4XOV/IkUMn+okv+nwnX3+5m7yhWaSnJzFuci5jJ+aQ5Aof\nnjjpH7cx/4o/Y7hPjElPdjJw2vmk9FHNZtLzshn4g/OY8PCtvD/wOozqWPlyQpPUq2snr0brQPnN\nIt0rvmMJhpgghJgOTAfIzY2O42rajeMYMqw7n3+6lYryGk4dm80lV44kI7Ppev5lJdUs/HwH6Rku\nSkuqMaNYWK8WAbn9VXRH8ZFK7v3tbGrcPgxDcvQwvP7sSrZtPsKtt58e/bG0ii4om2zH0xiC9Mm1\nUVgQnfH7fCZbN6kolpVfF6DpgkuvHMnV148JKeSzLxzPhbMeIv/3z1G6YQ/ObumM/PX3GHnn9xsd\n6+qRybVFH/JW96vi6nvonN3gGjIC5aCONBVEIlmzOSIhGA6gPHVBcgLbQh1TKISwoWaHYy38LABS\nyueB50H5GCIw7kYIIRg/OZfxk1sueHbvKObRP8/DMEx8PrPWgS0EUTUrORw6l19zKgAz3lmPu9pL\nXUuYx+Nn2eI9XHbVSHr1SY/eQFqNQDVZX4dStzuegLj+J11Z8VX7fAwtxTQkc2d9Q2q6k0uuGBHy\nmOzzx3L5qmdbdD57qovLlv4fC674M57jlRhxaDBUseMgJRt203VURw2eaAkpKLfpIUKHtbbVsuAm\nFoIhEvrcKmCwEGKAEMIBTANmNjhmJnBj4P9XA19I5fWeCUwTQjiFEANQ/fWiIWajgpSSZx//ipoa\nf60JyQw4Anv1SeeBxy8N232tPdjtGrf/7mwGDFKlJjasPkgI9wgAm9bHZgJrHWmossHDUG0SOxbp\nXXRuui12wtbnNZj94aaInS9r3BCu2fc2lyz4Z3xadkrJlqc+iv11Y84gVF6DhtKSdVSf6GG0PcJo\nN7FYTLVbMEgp/cDtwFxULdr3pJSbhRAPCCGC9aJfAroFnMu/Be4OfHYz8B7KUf0/4BfNRSQlEsVH\nKik9FrqsdPHhSpYu3BMyBLa9mKZk/96y2vfOpNCKn6aJuLUjbR4dlbCTi4o3aMmtKIiTC6oR501N\n5+UPe9Onb2x8OBXlHswQYaltxXu8iqLFG3D1yETY9fD9Y6PE0RXfxPR68UFH5TWMR5WyOBWYDJTR\ndue0G5U0F10i4gGSUs6RUg6RUuZJKR8KbLtXSjkz8P8aKeX3pZSDpJSTghFMgX0PBT43VEqZGGET\nLaRpU5GgotwdlUY+hiGZN3tr7fvzpg7B4Wgc+ielZNykvo22Jx4pqOSdkaj+CSmoejB1JysR2NYf\nZX2Mv/NS13UefiqHZ9++hum/PgOHM3pjciXb0dqQ9xCK8l0H+XDIj1h736sULVyn5ighcGSkkjk6\nj/GPTY/IdZqibNNevpr+T0yjw6wD20EqqiJqJuo+bm9o+z6i3d4z/k9XB6Z7z1S6hHFMd+uewrhJ\nuWFX8+3leKmbJQt2IaXkW98ZzpARPXAm2dA0lePgcOj84q6zSU7pSPWWeqBWWJOAKahVVioqMzkH\nlVQUrLWUTWyT4kKRBJyJy+Vkyrl5vPDudfTKjo6JKVTP77aydPo/8ZRW1EYnSb8Bpomw6Vye/x/6\nTp0UsWs1xY4X5/D5JXd34BLdNcA2VM7CWiBEwiPFgX1LUAGYhUQmw/9gBM4RnkS1M3QIhBBM//UU\n/nH/Agy/id9vYrNp6DaN6b8+g34Du9GjVyqHDpTjj3AYq5Tw+vMr2LOzmB/99DTuuu8Cdmw9yrZN\nh0lOdXDalP6kpjsxTRkVc1b0EajJPzvEPg3ljhoIlAZeBlBF+1djLaUHMJyGa6vf3X8hD/z+M8pK\n3BG9WlWll+oqb7sFveHxcnjxhpBJcabHy7H87RRGqB5USyhatJ7DSzbS6+xRMbtmZKhGVe01UQKg\nGnXv5QIDULE1W6hfNsOPyqyP1PWjh1UrKQIUH6lk/uytFOwro9+Arlx46dDagnc1bh+fvLeBOTO2\nNHOWtmG3azzyzBVk9ThR38fvN5nx9joWfLYdd7WPXtlpXHvLBMZMyInKGBKLAlSpgmBpgkgJ5FRU\n2QMdVU8pfLOWHVuP8Ne7I1vZxWbT+NeL3yU9o+nQ6ebw13h5I/XSkIIBXcOenIS/xhvTZkRZpw0n\nPa83PaacyqAbLsKe2r7vGBs2oCb/hmioqLsNRK96sIZaFLXeTBzTWkknO1k9Upl2c+jfOsllJ61L\n9OK2fT6TN19cxc/vOrvWz/Dcv75i7apCfIGCgEUHK3jmscXc/vtzGD0+MZy30SMXpWUcR03i7WmU\nkoRa/aWj6uu3jN59urTxeuFJTXNwrLiKtC5J7Upa3PbCpwghGpXIBsAw8VXEvkf3sfxtFK/4hoJP\nvmb9X9/gOyv+TUpO95iPo3WUhtkuiH5JeY1od3GzfAwxYP7sbVE9/4Y1B/n7X+YjpeTo4QpWL99f\nKxSCeL0G7762JqrjSByCxe8ygNE03dikFypR/zzgTJRzOwfl65gc2N+6XgKuZDsiwk9WRbmXh//w\nOX/69aeUlbRt8t711gJW3/MiMoLRTZEgOB5/VQ01R8pY/sun4jyiltDUHzhameUCtUgZT7Sb9Vga\nQwyoqoxuCQLDb7Jj61Fmf7iJsjI3RpgH/1DhydKsvS4pqJyJo6hko3LUaq4rKoeibl0iO0pDaB9r\nVxXisOt4Ilh11zBMDAMK95XxzwcW8OAT32n1Odbd/1rcy2E0hzRM9s9ejjTNBC+b0RPlAG6oeUki\nFzGko+7PnkAWarqOzZRtCYYY0LN3Gvt2h1M9I4M0Je+/ua7JY5xJeq2D/OQi2HchNgUHt20+HFGh\n0JCCvWXccvWb2HSd1HQn51w4iKlXjsDpbPpxriyIVMOYKNMh/J4DUeZKNyrwIfhM9UQ1g4oEBmoh\nowfOG7vp+mSbIeJEYkQFeTwGv7rpfbZuPhzvoXRqMjJd2CMYXhoKwy/xePyqeOIHG3n4D3PxN9NR\nMLVfB6jEKwS9zxub4NoCqEl6AqomUi5K+zwdZUaKtGArRRWYiJ3ATPRfv8NT4/axf190tYWWYhqS\nqkovjz/4BZUViW1S6MicEeOqtj6fyaED5axZsb/J48b+5Ub05AbRVE0M05bqwtm9CyJCiXXNomvY\n05OZ/PQvY3O9diNQJp48lF/KQVPRau1bIHpQXeFigyUYIkz+sgLu/sUn3HL1f/ntTz5i0bydyFCh\ngXFEmpLlS/bGexidjuIjlezYekQlF/7ubJxJtpiZ7Tw1ftavLmzymIHTzmfS32/DkZGKLSUJNBGy\nFIawaXx75b+5ofxTfnj4I1y9u0Vr2PWvKwS2JAf29I7c2zyH0NOqhkrY1Ou8WisoLMHQIfnqi108\n96+vOHSgHMNvcuxoFR/8dy2uJurpA1E3OzTE6zUoKa6K6TU7M5UVHh758zzuvn0m/3zgC+64+UM2\nrj3Ek69czcDBsZlUNU2QmtbUalUx7GeXc+3hD7li3Qs4M9NCd39LTqK68OiJz/z02xEdazik36Dm\nWDlr/vhSTK4XHVJRAqBu4TwNVTgvG5XRPyLwvncrzx27/A5LMEQI05S8+9oavA3DRD0GXl94R6Su\nC0aMim5MckOSXDbyhmTF9JqdmScfXsj2b47g8xq4q334fAaL5+/g81lbGR6jv63NpnHWBYNadKxm\nt5Gel40RpieDNCX+KlWO2zQMDi/dFLMie9JvsG/GVzG5VvTozQkBMAIVBh307+go81MPWq8xtFaQ\ntB1LMESIivIa3NWhHzSbTSe3f0aj7bouGDG6Nz17x66Es82mkdE1mdyBXWtLhIMSbLM/2sTtP3qP\nG696g9/97GPylxfEbFwdlaID5ezddQyjgePX6zH47OPNUW/apOkCu13jBzeOIye38T3WFNkXjlPm\npAZIv0Hv88cCsH/WMg5/tTGykUJ2vck5MfEdzy3BhhIAWShhEIrWJPGd0sR5Io8Vrhohklz20Nmk\nqDyD0eNz2L/veG3BMCFUhuytt5/O7372cVTHpusCw5DY7RrpGUkcLargnttn4nLZufaW8ZxxzkD+\n+9IqFs/fiTcQZnn4UAXPPf4Vxq/O4LQz+0d1fB2Zo0cqMcNMmu5qH7M+iFwfhSBCKNPRqWOzGTWu\nD+Mm9yWza+uS8AAmPDqdQ1+uw19VowrpAbaUJIb/6ip8FdVsfHM+u/47H3+km/k0oUEDDJh2XmSv\nl7Bk0Hw3wwyU2Sm2ZUIswRAhnE4bk87ox8qv99UrmKfrgu69Upn76Tf1qkgKITClREoZ9bDtoFCw\nO3SOl9ZgGBLDMPB5DZ7711I++O86yo5VNyoR7vUavPPqaiZN6ZeAvaMTgx69UiNeILE5pFR/080b\nipj+6zNJSW1bYb0ug3O4ct0LbHzsbQ4uWIurZyYjf3M1R1ds5ZOx05F+E9Mfu5pJAAjBuAduiu01\nY46JCkH1obLqK5o4tgxVnXUCKuopNliCIYLceNtplB6rZtf2YjRdIE3omZ2Gz2fUrsSDmKbk2NEq\nig5UoOsavih3ZfL5zNoucw05diS8I7qsxI3X48eZFN0U/I6K12Og6SI2fb4boOsaWzYcYuIZbe/C\nltqvJ6c/8+va94cWruObp2dguKNb7z8cvc8fi7NrIrWijTQVwHpOVGVtyaLCg6rKOjyK46qPJRgi\nSJLLzt1/vZjCfaUc2H+cHr3S6J/XlduufSf0ByQcPVzB1CuG88m7GxIy4VPXBXZ77GybHY20dCea\nFh/BABIR4ZLq257/tNbxHA8GXHNOi44zvD5qjpaRlNUF3dlReo6YqF7nbdHCjtJhBIMQoivwLqry\n2F7gGillaYNjxgD/QVV/MoCHpJTvBva9imrdFSzic5OUsum6Du3EX11D9cFjJGd3w5YcnaqnOf0y\nyemXWfu+R+80CvaESHITqsva57O+SUihANArO519e0pV/R+HzqQp/ejRK/rNyBMNn8/g7VdWs2LJ\nXkzDZMykHL577Ri690xl8LAebN9yOCrd+prCNCWnjI5spIq3rDKi52sty27/P/p//1ycGakh90vT\nZM19r7LlyQ+RhhKMw39xBeMeuhVNT/QFTAltz16O7b3Vrn4MQojHgBIp5SNCiLuBTCnl7xscMwSQ\nUsodQohsYDUwXEpZFhAMn0opP2jNddvSj8H0G6y661m2vTAboWtIw2ToTy5j4j9uQ7NF94Zau3I/\n//7HkkahrClpDgyfQU1Nx2hvqGkCXde4+oYxTL18RLyHEzOqq73c+ZOPqK6qH3WW5LJz76NTOV5W\nwxsvrOTY0SqEUP0wouF3cDh0vF4DXVd/hx9HITBg2/OfsvLO/8RVazjtqV8y4hdXhtyXf88LbHlq\nRr1igHqyk6HTv81pj/88VkNsIweBHbStJHd3VGRS+2hpP4b2xoVdAbwW+P9rQKO/ppRyu5RyR+D/\nB4EjtC5OKyKsvOtZtr04G8PtwV/pxnB72PbibFbe9WzUrz12Ul9umD4RTa+v9rurfB1GKIBaofp8\nBu+/sZZ9e2KXhRlvnn38q0ZCAVS5kz/9+lOeePhLig6Wk9nVxbnfGszZF+ZFZRy9+qSTNzSL3n3S\nmXBGblQ0t7wbLiK1XzcZ9hIAACAASURBVE/0pPiZZ8o27w253e/2NBIKAEa1h23PzcJXGdmueZGn\nrX067EDLclQiRXsFQ08p5aHA/4topnylEGISyrVet7/dQ0KIDUKIfwkhwqZuCiGmCyHyhRD5R48e\nDXdYSHyVbra/8GnIG2r7C7PxVUX/hsrsltLIVm8mWKmMluL3mfzlzjnMfG9DB+7X2zL8fpONa8L3\n1zVNiafGj2lIig5W8L+Pv+GLz3ZEZSwFe0rZta2YwoLjLFu8l4f/MJcP/7s2otewuZx8e9nTjLrn\nh9i7xKE0hRCk5YVq5wrVB4+F9aloNp2q/YlePTYFVe69NaSj+oVEr9lXKJoVDEKI+UKITSFeV9Q9\nTqoZIuwsIYToDbwB3CylDOpS96CCdCeifrHfh/k4UsrnpZQTpJQTundvncJRfbAYEcb+KHSN6oOh\nWvRFlk3rDuKpab3TyZnUdDJQvDBNySfvb+Szj+u3LPX7VLmNhmazjorfZySk8JOmxOs1mPPxFtas\n3I+vmdyA1mBPS2bMn2/gzBfuxBbrNptSsvbeV9j/6bJGu1y9MpFhKsiaPj/J2bEpP9J2qmldEx8N\nGEm0m/KEolnns5TywnD7hBCHhRC9pZSHAhN/SJEthEgHZgN/lFIur3PuoLbhEUK8AtzVqtG3kOTs\nrNoEnoZIv0FyDIqEpaQ4sNm0kKWRhaifWGqza9zy88lMPKMfDqeNQweOc+9vZzcKeY03fp/Ju6+t\noehgOdf9eCJzZmzms4+31BYNPG/qEH5w4zj0WFXnjALOJBvduqdQ3ERIbzzx+0yefmwxdpvGldNG\nccmVI9t0ngOf57Pq/z1L6ea9ODJSGX77VYz6/TTSBvSidOOeCI+6aQy3l4XTHuQHB97D0eWEE9qe\n4mLQjd9i5+ufY7jr+BhcDgZOO7/esYmHH+Vebc3icAix1hSCtPeJnQncGPj/jcAnDQ8QQjiAGcDr\nDZ3MAWGCUNlTVwKRTxMF7KkuBt8yFd1V31Klu5wMvvWSmDQfP/2cgSHVYIfTxrjT+pKa5sRu1xkx\nqhf3PnoJU87Lw+G04fcZrF1Z2KhVZyKxeMFOHvjdZ8yZsRlPjR+v18DrNZg/eyt/vft/FOxNjLLj\nbUEIwbQbx5PIVRoMv0lNjZ+P3l7P4gU7W/35A/PyWfDde5UAMCXekgo2/f1dlv7kn0xd8E/smbGP\nQjMNk63PzmLfJ0spmPU1/mrlDD/tyV8w8Ifnoyc5sKcnoyc56P/9czj933fEfIzNI4EDwNfAElon\nFBxEu69zU7Q3Kqkb8B6qU8U+VLhqiRBiAnCblPLHQojrgVeAzXU+epOUcp0Q4guUI1qgAnxvk1I2\nGy/Xpqgkn58Vv36GHa/8D82uY/oMBt88ldOe+AWaPTbpHF8t2Mmrz65EaMoUI4Tg7AvyuGH6pJCZ\nxX6fwcN/+pz9e0qbNM0EP5qAFg9AVY8945yB3PyLyTHJoHZXeykrddM1K6XZrmZNsWndQV79zwqO\nHo5PCGeXDCcVFd5W5Uh0657C4y98t1XX+XjMTyjdsLvRds1pZ+RvrmbT39+NW59ozWlHd9qRhsmZ\nr/yOAVerPAdPaQWVe4tI7dczQRPiJLARaIuZWkf1HI/892ppVFK7BEO8aItgCOKrqKbqQDEpfbKw\np7W+vkx7KS9zs3rFfrweg1Hjs+ndJ3ykwqJ5O3jzxVXNmpBsdi3mZRlai92u89PfTGlXlm5zeL0G\nrz+3guWL96DbNExDcuG3h/H968ciBBwqLMcwTPr07YIWMG+Vl7k5XuamqspH9x6pdOuuHK47tx3l\nkT/Pi6umZrfrmKb6u7Y0R0LTBK98dH2rrvOq4+KQplZh00jKysBdlBgRaLrLyZXrXyB9UJ94D6UF\nlKNMR60lFSUUouNXaKlgOOkyn+1pyWQMy43b9dMzXJz3rSEtOnbpwj3NCoX+eV2RJgkfPurzGbz1\n0ipGju7N7BmbWb5oDwjV7eyiS4eyesV+8pcVkOSyc963BjNydO+w2kV5mZvlS/ZSVupm6IienDq2\nN5qu8fLTy8hfXlCv/Mf82VupLK9hy4Yiyo/XIITA6bRx4aVDmTd7KxXldWzVuipM97M7z+Kjt9bH\n3Xzn8xkIoQREWrodXdfI6Z9JeambvXtKQzaACgq21mBPc+EtbawVSb+Jt7K6TWOPBqbfYNuLs5n4\nyPR4D6UFFLfhMxowmHg4mxty0gmGjkTDvIe65A3J4ld3n0tGVxd33NKq/MC4UXLMzW9/8hFerx/D\nrya1OTM2M+ejzQgNfF41mW9cc4BzLh7MdbdObHSO9asP8PRjizBNid9nMnfmN2T1SOHO+y4gf9m+\nRvWgvB6DxfN31dvmCdjjG2IYknX5B/jXQ19yYP/xRvvjgZRKC7jmxvFMOXdg7faH/jCX3TuK62mK\nDqfOVdNGtfoaXUfnUbSw8e8BYES6smpdNIHusGMaJtLXvP1d+vxUFSR6SGqQljqlBCfCDnNR1VTj\nTwK71CzOviAvpI1ctwlyB3Zl985iDMMku29bE2dij7vaVysUQEXV+P1mrVAA8HgMFs7dQWFBWb3P\nejx+nvn7Yrweo3ZC9PtNig5W8Odff4oeoTaaWzcdrjXhJAI1NX6OHKpfgfM3fzyPMRNysNk1nEk2\nklPsXHPDOKac1/rkum7jB0dqqK1C6BpXbnyJMX+6jp7njkZzNr1StqUk0fvcMTEaXXtpMqUL1XQn\n+H2dqIY+A6I6otZgaQwJzOQz+7N88V62bj5cPwdCwqLPt7Ns0W7SuyRx7c0T2Ln1aMKFs7YHn99g\nxlv/v70zj4+qOhvwc+6dLStkAxIgJGGTYFhkkU1WRRBkdUGp2lalitXWaqvW1ra2frXWilq17guu\nuGFRUYqCC7KJsiNg2IQQIIQAWSbJLOf7407ihMxkJiSzJJzn94PMnHvuve+cuXPfe867baBjZlsy\ns5PpN6gTK5btwuXy/Rkr7c5mjfcoL41MdlFf2GJM9ZR/bJyFm+8YRXlZNeVlVSSnxp12fekuU0ew\n7ZGFfl26Q4XJZqHq2En6/fFqzv7tLBad8wtK9xTi9lFZTph0rCmJ5MweF1YZT58YoCt1Y3m9KfR6\nbcfwzelHtMwYzjjjc0vD7ZZsXn+QtV/t5ds1++ulZqhZFz/v/G48/9gqysui54bWVGriPhpjXDeZ\nBE5ny7umGyIpJZYHn5yGKURZbqWUvN39Ksp2Fwbu3IyY4mxMWvlvkvOMJbLqE2VsvO9Vdr/2KS6H\nE0ubeCoOFCF0jS4zzmPQA3OIad/YyOFIU46RHykYl+04YHBIpVFeSa2MhoLcTCaNJ169HLNZZ+M3\nBXy8aBu7dxZTXeWsEzx3aiDdaSEId6LHRqHr4GdS0SIRAv46bzKds5ICd24Ch77cxMfjbg/rrCEm\nI4XL9y9o5UWgJLCK4COeRxPKVAfhSqKnCBOVdidaA1FWjioXmiboP6gTd/11PM8suIKbfjuS2Dhj\nHVNo0CbJ1vSa7lGsFKB1KQUAq83M4UMNVfhqHtqPyKPbzyaE/DzedLvmwlauFAAqMSq1BYO3ITqy\nKMXQQujUpa3fpHspaXHEJdTNhllcVMaTD31JeZlxUUo3HD9WGdSMISZOmZ7CSaCbYzgKJR3ftpcD\nH64O3LEZ0a1m3K1Nk9dDI/inqehZJlOKoYWw6ZsCXD6iT00mjWtuOLfezeWFJ1afduGYyoow1/k9\ngzGbNfoN6kiHjEQ0HylTBJDbJ7SpEb79y0u8128O9jAkk/Rmy4Nv8vHY23BVtR67WH2sGLaDQAgM\nz6ToQCmGFoDL5ea5x1bh8pGALykllt4+qnjt2Orf31vTRIOunS3Q7NRi0U06N9w6gnsemED79ARs\nNmO2ZjbrWKw6N/1uZMhmDNLtZtmlf2HjX+ZDBFJeOMvsHF23k+1Pvh/2c4eXXAIvEY0gmpxEo0cS\nhV8O7DvuUykAlBRXUHayivjEugkCfc0uasjrn07vfhn8981N2CscEapXfGaj6YK4OAu3/mEMthjD\nDvS3Ry5m/dr97PzuCEnJsQwfk0ObtqFL8PjtH55n3ztfhOz4weCyV/H9cx/R+1czIypH0ygH9mO4\nnSYCnTBmCjXEAUMxkun5ohvRdiuOLmkUPjGbddx+HuMlRsDbqXTq0pZ9u327yI2/uBdn98tgzIU9\nWLEsn1Vf7GXf7mOnVS9C0TgsVp2ERBtXXT+IPgM61klJbjJpDBrWJaT5pGqQUrJlXnREzLuqgzXO\nRiNFwDaMX6LEKF9fAJyDkfeoBivQH9gE1NhVBEYgXKdwCRs0aimpBZDeKdHnk6PQBN16phETW78M\n45XXDsJkrv/1ZnVL4ex+RoUsi0Vn7ISe3P1/F5LXPyNaHCJaLUITzL39PB58ajr9B3eOaJ0K6XL7\nDCQLN5rVTM4VYyMtxmniBrZ7/tY8uEmMG/8OH/3bYiwZ5QG9MCqz9SIaf3hKMQTBvoUr+O+AX/Ba\n2nQWj7mVQ19uCuv5hRDcfMcoYmLNWKzGerPVZiIhwcr1twz1uc9Zvdvz69+PIb1TYm0itnEX9eDu\n+8b77D9xWi4WS+i9X1oDQuBT6TaErmsMGNKZ/oM6+zQyhxvNpCNMkf2+dZuFuE5p9P51S11GKsW/\nx1EpP84MvNGAVIxaC2GujtcIVIBbALbMe4v1f3yxtlAIGE85aeeehTU5kezLx5A1cyRaGH5k5WVV\nfPXZbg4dLKVLdhJDzsvCagucidHpdKPrIqBb5MrPdzP/qbW4XJLqqjN7WUnXhV+vLrNF5yfXDeTl\nZ75uMCLbZNZwOd1YrSbaJsfyh/svJCExMhW5anCU2cl/aQkF/1tHyba9lO0Kb7SzNx3G9Gfcwr9g\nSYxAbelm4SRGGRlfCkAA52HUVogeVORzM+CsqOT19jNxlvvPMCnMOm3PyuTitU+gW+sv6bQ0nA6j\njvAH72w5420OA4d2ZuO6gjoZW61WExOm5TLjir4sWbSN1573nXPfFmNi7ISe2GJMZGYl0XdAx9oa\nEKHi5K6DnNixn8RuGbTp0bne9sqi4ywadCNVR0/grKhC6FrECvCAMWMYOf9OsjzFd1oeEvgK3wFs\nSRi5j6ILVY+hGSjZvAcR4McsHS5KNu/hvb7XM23jMy1eOZjMOv0HdeL9tzdHWpSIYrOZGDAkkwsv\nzuWtV9azf+8x4uNtDBmZRV7/dDavP8iw0Tn8sKeElZ/vqRd8qOsaM2f3O+3Edo3BUW5n+WX3cmj5\nBjSrGXe1g7QhuYx79y916iB/c/dzVBQWIx3GE24klQKAq7KaLQ+91YIVgwB6YxiUa4zPGsYsoWcE\n5Wo6TbpqhRDJQoilQojvPX99JnQRQriEEBs8/xZ5tWcLIdYIIfKFEAs89aGjBktSPO4gc8eczC9g\ny0NvhVii8NA5K4ku2cn1bmoms4beQI2IcGO26NhiTCGx3cXEmhk4tAs9cttx1ZzBxMVbOXHCzgfv\nbOFvdy7h4f9bzq3XvovVZqJdh/gf4w8sOlariZvvGBUWpQCw8oZ5HFq+AVdlNY4T5bjs1RxZuZUv\nrvkHUkqKN+RTsHQde976vFYphAPNU5e5IeyHWm49cIMkYAjQBWgH5HjeR6/9IBiaWvP5AeCYlPJ+\nIcSdQJKU8g4f/cqklPE+2t8E3pVSviGEeBLYKKX8T6DzhtPG8F6/6ynZYhRJD0R8dgcu3fVqGKQK\nPeVl1Tz76Eo2rS9A1zU0IZh2RR8OHzzJiuW7jepmwki1EQmEMFKB9MrrwMEDJ9i1o3EVs3ST5jc2\nJKdHCjf/bhTJqXFUVTq49bp3/WattVh1Lr9mAG2TY9i1vYik1FiGjswOmy3BUVrB6+1n4Kr0kara\nYiI+sz32wmKEScdxojwsMtWQPu4crMkJ7HtvhW+FJATdrhnPec//LqxyncmExcYghNgBjJZSFgoh\n0oHPpJT15lC+FIMwLKFFQAcppVMIMRT4s5TywkDnDadiKN1TyOJRtxpPYlXVuKv9r7vHdEhm1sHW\nMWuooay0irLSKlLT4mrTPu/JL2bdqh84XFjK+q/3h6XetMmsMfqC7mz8poCS4gqcnpu6yaQRE2vG\nXuGobWsIIUDT/SuFOb8exvDRPxa7WbFsF/OfXtugvSW1XTz/enp6Iz9R81C69xDv5V3boB0skjRk\nxzDFxzB1/dMkds0Is1RnLuHKrtpeSlnj1nAI/2WLbEKIdUKI1UKIaZ62FOC4lLLmF3cA8FvlWwgx\nx3OMdUVFRU0UO3gSstO5dPerjHrtbgb960Y6jPFjUNIEiT07s/PZDykv8C2flJKWZuyPT7DSISOx\nTi2A7G4pTL0sj03fFpy2UmhMUs0OGQncce8FjDy/GydK7HUUgNPpprysusFIb7NZQ9MEOd1TmHv7\neQ265Q4dmVPn/dGi8oBG+JMn7EF+kuYnrmNqxN1OG8KXUhC6Rocx/bh4zeNKKUQpAY3PQohPMJxu\nT+Vu7zdSSimE8HfX6yKlLBBC5ADLhBCbMUIEg0ZK+TTwNBgzhsbs21Q0k07nSUMA6DV3Kstm/Ikf\nPlgJ3u6MbsnRtd9x9OvtcMtj9L17Nn3v/glgzDpW/+oxCj7+GqEJOl88lHMfvom4jmnh/BjNyrbN\nh04rZbIQRvUxh8NVr7aEbtI4b2xXBg3LZNfOo7RNimHgsC7ExRmmpw/f3eJTAfjLOgvGctP1twyj\nXYcEUtLi2Lf7mN8UILFx5noxBplZSdhiTEaFOD906hLaWgkNoZlN9LvnKtbd+WxQdZMjiiaYVfBm\nCyy2c+YRUDFIKc/3t00IcVgIke61lOQzc5uUssDzd7cQ4jOM2PB3gLZCCJNn1tAJI5Y8qhFCMObt\nP/Hh8FsoXrcT6VUb2GX/cR16499fo/2IPNr2zuL9wXOpLilDut1I4If3vuLIyq3M3P4S5oSGjXPR\nir+lGDBu/rpJw+2WSLcEAZoQdMpMYuzEHgwdlc32LYd57IHPkW6Jw+HGajORnBLLZVefQ1y8pTY6\n2xuL1WQsAwWZqtls1jlvXFd65f34XJOZnURquzgKC07WUSgWi84Fk86qd4y+AzvSpm0Mjuoyn3EN\nFovOZVf1D0qeUJF7ywzW/+lFnNGuGNyS8oPFSjG0AJrqrroIuAa43/P3v6d28HgqVUgpq4QQqcBw\n4AHPDGM5cAnwhr/9o5EfFq7g+Na9dZTCqbgqqtj2+Huk9OuGs7yyTl/pcuM4UUH+y0vpNXdqOERu\ndnrldfD59C40wTmDO3HjbedRaXcQE2uhoqwKW6ylzhJO3wEd+ed/pvHVZ3soKS6nZ+/29B/cuUFP\nnsHDu7DgpW/rtVusOrl9OrBt0yEc1S6kNCLD09rHM3Fq3VTGQghuu2ccD977KcVF5WiawOlwMXBo\nJlMv71Pv2Lqu8Yf7J/DSk2tYv/YArprvURqpSmZfO6iO4okEjpMVDdq+ogZh1GBQRD9NNT6nAG8C\nmcA+4DIp5TEhxEDgBinldUKIYcBTGAlFNOBhKeVznv1zMJRCMrAe+ImUMmANvEiX9lw6+fccWLwm\nYL+4zPYkdu9I4af1b2YAWZeOYsyCe5pbvLCxfMlOXnt+Xe3N2GzWsNpM/PnBSaS1r+eE1iys/Hw3\nzz++GqTE6XRjsZjI7dOBm+8cxb7dx/h86feUlVbRf3Bnzh2R5TdltZSSvbuOcfxYBZnZyaSkBY6+\ndTpcuFxuLFYTLpcMmztqINxOF68mTYkOA7QmEAifD02axczsY+9hio1s9PeZTFgC3KSUxcA4H+3r\ngOs8r1diZI3ytf9uQl39OgTIIJcy7IeKkW7ffTWzifjsyD5pNpUxF/YgMzuZpR9sp/hoObl9OjBu\nYk8S24Tuhz9sVA698jqwdsU+7HYHZ/dNp2vPVIQQ5HRPJad7alDHEUKQ3S0FwwciOExmvdYIb/KR\n0TZUOCurKVq9DWHSScrLxpIYV8e+o5l0zrrhYr57YhEue7C1hUOEWyJsZgQSt5cLrWY1k3vLDKUU\nWggqJcZpsOuVpay88eEmPaHpsVambXxWeWUoGmT3guV8df2/cDucuKsdIA03z76/v5LcX89k96uf\nkv/yUoQwntIPr9wakaI73mg2C3m/vYyt895BulwIXSf31zPo/6dr0PTo9aA6E1C5kkKI2+Hk4wt+\nS/E3O09LOWhmE52nDCN9dF9yrhiLNTmx7vGdLo5tyEezmEjKy0G63BxYvIaSzXtIyEknc/oITLao\nChJXhIDiDfl8OPwWn7MAYTUTk9aW6pLS2mvQFGcjPjud49/vhwgmQew4YSDjF/8Dt8NJZfFJrMkJ\n6BZlW4gGlGIIMW6Hkz1vfsauVz+hZPMeKouO1zUACvxn5NUEuCV6rBWhaVzwwf/RYaRh+Nz77pd8\ndf2/kE4XUrqNzJNCUH2yHGd5FaZYK3qMhUlfPEKbnvUTpSlaD1/+7B/kz1/a4mqtDn/ut/T42YRI\ni6HwQbgC3M5YNLOJrrPPZ/zi+7lk1yucdcMU46ZtNWNJTkBryPvC4ybpqqjCWWbn0xn34HY4Kd6Q\nzxdX/53qklIcpRU4yyqpOFhMRcFRnKV2cLtxltmpOnqST2e0XKO1oi6Ocjt73/mCXa9+gv3wsdr2\nk7sOtjiloFnNxGUEb7dRRCcqu2ozoFvMnPvwTQx8YA6OE+VYkhN4M+tK7AXB5e+RTheHvthE/ktL\n6hjs/O8gKdt3mBM79qtZQwvnh/dX8vmV9yE0zYiMdzrpc9ds+v3xKtqPyKNo7XZklLiiCrMJzWJC\nOl219o56fXSNdiPODr9wimZFzRiaEd1ixpbWFk3XyZo5Muj9nGWVFH29nRM79jcYG+GNpus4SitO\nV1RFFFBx8CifzfobzvJKzwzRjqvSweYH3qBgydfk/nIaphhr4AOFCymxJMQwbfNzjF14b72U9HqM\nhSGP3ow5rmVnFlUoxRAy0scGX6RDut1s/OvLxHZMhWB94wUk5WXXvi3bd5gtD73Fpn+8TsnWvY2U\nVhEJdr3yic8HAWd5JVsfeYfYjFQmr/w31tQ2EZCuPtLpwlFq5/DnG+kyZTiX7n2N3r+5hNRBPcme\nNYaJy+fR4+cTIy2mohlQS0khwmS1oMdY6qTJaAiXvZqDS78JKr23Hmtl8ENza4sCbZn3Ft/e/byx\nFOF2s+Hel+n+84kMefSXp5XPSBEeKg4dw13le+nQXmjYGtr26sKkrx5lYd61UbGk5CyvZP/iNfS4\n9iLiOqYx+MEbIy2SIgSoGUOISBvSC3cji6I4yysbVAzCpGOKs5E5ZTidJxtJ/Uq27uXbP7yAq7Ia\nd5UD6XDhsleR/+LHHFi8BmdFJY6yyGX/bGkc+mITn13xVz4a+xs2P7iA6hNlOO1VdWp+NxcdRvXF\nFF9/2UWzmMm4YABgzCbX/uaJqEmQJ3SNmHaRSxqoCA/KXTWEfDLlD+z/YFWzH1ezmLG0iWPKt0+y\n7ZF32TrvbZ/pja0piVR7irMk981h2FO/IfWcHs0uT3Mi3W5O7DyAbjGRkOM/+M/tcCKlbDb/+Krj\nZXw++28ULFlXq5w1qwWk2xhbIUgd1JPhT99GUu+sZjmn2+liYZ/rOLlzf70HAs1qpsf1F3Hw0/Wc\n/O6HZjlfoxGinleUHmPloi8fjvrrSOEbFccQBZTuKeTdXj8NSYIzYdLpds14hBDsfHZxUPuY4mOY\nvvk54rv4K5sRWQ58vJYVP/8njtIKpFsS36UdoxfcQ3LejzUSSvcUsvKGeRQu3wBS0v68PIb959Ym\neWeVHyhi0YBfUFkUIBO8EJgTYpjx3YvEpjfeJbOy5CQrr/sXBz5ai6uymsSzMinbU+jXwyfS2Non\n4ThRXmtkdjtdDHxgDr1vnhFhyRSni4pjiAISstPpf+/P0GOsRlBbMyKdLvYvWkXnyUN9Lkf4wllu\nZ9Xch6OyWFDJlj0su+TP2A8dw1leictexYnt+/lo1K1UHS8DjKf698+9icJP1xsBgC43hz7fxAfD\nfon9yOnXDl5723+oPHoycEcpcVU5+O6JRYH7nsLehSt4o91M9i1cgavSsDud3P6DYWOIvq8DNEGH\nUX24ZNcrnDtvLufOm8ule15TSuEMQSmGENPnd7OY9NWj9Jo7lYTufgvUnRa6zUKnSeeS3LeroXxq\n8GdwlnBgydesu/OZoI5vP1LC5gcXsOrmR9n16ie4qoIzpJ8OWx5806ch1l3tYNernwDw/Qsf46yo\nm8IcKXHZq9n+5Punfe79H6wKOpDMXeWgaNXWRh3/+La9fH7l3/yWuIxGdJuFPndcQWx6Cj2um0SP\n6yYR20HVUThTUF5JYSClXzdSHr2ZAx+vZfll9+JsBmOw0DUypw9n1/ylOMrs6DEWdJsZ6XbjKLX7\nfwp1S7Y9+i5n3TiFhKz62V1LtuyheH0+lcUn+PYPz4Nb4qqsJv+l//HtH19g8urHiGmXhJSS6pJS\nNKu5WfzWS7bu9XnjdFZUcdzjfntk5VZcFfXzBrkqqzmyckujziel5NBnG9i9YDmuRiz1CZNOm56d\ngurrKK3g0Ocb2f7U+369j6INzWzCmprI8GduJ6V/90iLo4gQSjGEkY7jB9JuWG8Or9iCq4leLtLl\nZvvj/0XoWq0No6HC6964qxy83XU2CdnpDPznDWRNH8HJ3YUsnXQXJ/ML0C2mem62zjI75ZXVrPnV\nY/S4bhIrb3yY8n2HAcgYP4ARz97epMpcKf27cWxDfj35TXE2kvt0BSCxWwaaxVTfZqNraBYzm+5/\nnbjMdnSZPqLBwDCXw8kH597EsY35jV7GkS43R1ZvZ9drn5IzawxC8z3p3v70B6y99Qk0sx4ddRIC\nYIqzMezp35A26CwSctL9fi7FmYEyPocZt8NJ/ktL2Pj31yjbezjiuXD0WCu9bp7Oln8uCCqGQph0\nhFnH7a04dA1zQiwdzz+H9iPy6HbNeCxt/BfqKd17iC0PLuDQl5tJzEnn7Nsuw5bWlv/2n1O7/m6c\nTGBNSeDSXa9iucrkcAAAEgBJREFUToildO8h3jv72vquo0JgirHgqnKgx1jRLSYmLH+ojtHam8Wj\nfs3hLzcH/KwNIcw61uREMqcMI/fm6SSd/WOw4ZFVW/n4gt/6nN1EA5rlx+dB6ZZoZhM950xi8ENz\nVdxLK0d5JUU50u3mi6v/zr73vjLiD1rQ+nNDmGKt6HE2Ll79OAnZ6fW2H9u8m/cHz62ztCLMOgld\nO3o8dJy1yjKpTw5j3ryHNj1+9DgqWPI1n8++z4gREeC0V4E0jPF15Iiz0ef3V9L9ZxPrrI2f2Lmf\nhb1/7ne8hVlHNiL+RHhmKyPn31mbBmX5rHvZ+9YXEVf6/tCsZiYu+xeHV2xBOl1kTh1G29ysSIul\nCANhUQxCiGRgAZAF7MUo7VlySp8xwDyvprOAWVLK94QQLwKjgBo/wZ9KKTcEOm9rUAw1HNu4ix/e\nX8mGv70SFZGtgdCtZlxBrJdrVgttenSk5y8m0/P6yWhm4yn1rW6zKdt9KPCJhCChazozd8zHZa9C\ns5pri7y4nS4Or9yK42Q5y2b+ye+NXLOY0Cxmxn90P+2HG4nd8uf/jy9//kBQs6PGYIqPodvVF7Dv\n3RXYi45HvFhOQ2RfPprRr/8x0mIoIkC4FMMDwDEp5f1CiDuBJCnlHQ30TwbygU5SygqPYvhASvl2\nY87bmhRDDT+8v5LPZv0NIQROe2V0ujACSX27UrJxV9D99VgrHc7rwwWL/4672sH8mEbk0jHp6Ga9\n1t7RpmdnRr1+N9+/sISdzy1GOly4g4gINifFk3vzdHY8+T5VJaWNmhG0NmI7pTFz53xV6OkMJVyK\nYQcwWkpZKIRIBz6TUvZsoP8cYJSUcrbn/YsoxVBLZfEJ9r3zJce3/8B3T7yHrG7kDUwTxHRIwX4w\nuHTfp0NMegr2wuJG7WOKt3H+ovuIy2zHO92uapoAwoj8bilePuFAs5iNILkgmLLuP6SoqOUzlnAF\nuLWXUhZ6Xh8CAoXUzgJeP6XtPiHEJiHEPCGEX1cSIcQcIcQ6IcS6oqKiJogcvdhS2tBzzmTOfWgu\nk796DGFqZH1ct2z0TbuxnM7xnWWVHPhoLbbUNk0P9JMopXAKQtfqpcCu30nQ/ecTlVJQBEVAxSCE\n+EQIscXHv6ne/aQx9fA7/fDMKPKAJV7Nd2HYHAYByYDfZSgp5dNSyoFSyoFpaWmBxG7xpA7owUVf\nPIylbTzmhFijVGgwRKnBUwiwJMaROXW4/w7NcZIzEJe9irZ52eixXhH2NX91QVyX9ox69fcMf+a2\nyAmpaFEEjGOQUp7vb5sQ4rAQIt1rKelIA4e6DFgopax93POabVQJIV4Abg9S7jOCdkNymVX4Fgc+\nWkvR1zvY9sg7uKud9TxwWgLfPfk+BUu/IXVAT5LP6c6xb7//caMQoGvQxM8Vk5GMvSC0M6ZoRJhN\nZIztz4inb+O7x9+jfH8RGRcMoOecyViTEiItnqIF0lQbwz+BYi/jc7KU8nd++q4G7pJSLvdqq1Eq\nAsNzqVJKeWeg87ZWG0Mg7IePsf2pD9j+xCKqjpe2CC+mUxFmHel0h2Rmo5lNuJ3OqDXcB0Pa0Fyy\nZo5kw19fxuHJjBsIPdbKtA3PkNiteVOuKFof4TI+pwBvApnAPgx31WNCiIHADVLK6zz9soCvgM5S\nSrfX/suANIyFkg2efcoCnfdMVQw1SLebg598y/7Fa9g1fwnVx4O7gSiinza9Muk0cTB6jJXC5Rso\nWuk7L5NmMaF7PItGzr+LzCnDwimmooWiAtzOEA6v2Mz/Jt6J2+EMSXpvRXgxZj0uI3DOpCPd7rrf\nqxCYE2MZ8cxtWJISaD/i7NpKfgpFIFTa7TOE9iPymLb5OXrdPB3LGbyerFlbR9ovt8OI/JZOl5Ee\nRAja5nYxgvXMJlIH9mDSikfJumQUGePOUUpBERJax6/pDCchqwOD/3kD3a66gA9H/KpZsre2NKyp\nbbEXhC5+I1JIl5suM0fS+1czjNxRZ7DyV4QPNWNoRST36cqUdf/Bmtom0qKEneriIArttECky5g5\nWJMTlVJQhA2lGFoZbXp0ZvQbf6xbuAcQJs2IGm6lqRCCyd8UFQjDvVSYg5usm2JtyrCsCDtKMbRC\nMsb2Z+TLdxHbMdVIPmcxkTllODO2vcCgB35BzhVj0WPqKwihaz7bWwQtILZNs5g4a+5UJix9AM1X\npLKgTrS7Kc5GxwmDaTesdxilVCiUV1KrRkpJZdFxTHG2OlXWpJSsmvsw+fOXIp0uYzYhYfBDc1k1\n9+GGDyqIyjiB5P7dObnjB5xhqIEgTDqmxBicpfbACfkEWJIS6DJ9BP3/fA1xHY2o/c0PLmD9n17C\nXW2kXDfF20jITidr1hj2v/cVeoyVnnMmN1gMSKFoLMpdVRGQkq17ObB4DXqMlawZI4jNSOW1djOo\nOnrC9w7CyM5Zsd93ripzm1ic5VW1leSk29209NZmDRzBpa/WrGYyxg+kcOk3SE3gDpGCMMXaGLvw\nL2SMO4eT+Qf54Ny5VPsLRBMw4oXf0f3qC31uLl7/PTuf+4iq4hN0vngYWZeMRLeYQyK3QgHKXVUR\nBEm9s8j77eXk/nIasRmpAHS8cJD/HWTD221pbbls3+tMXvUYSWdnNbnmQdcrLwicHM6Du8pB4Sff\nMnnN45hjrH7zJgWVcK5BJO2Hn43QNNr06ETO7HF+ewpd96sUAFL6d2foY7cw+vU/0vXKcUopKKIG\npRgUdehx7cTaiNpTSRuSS/rofpjibPU3CkHb3Cxi01NI6dfNKFvaRH5YuKJRq1aaWWfns4sN/38/\nM2HpcgdfLU8TPxqJhcAUa2XwI7/EFPvj58+5Ypzf8bKmJDZCeoUielCKQVGHDqP6kpSXjWat+/Sq\nx1oZ9tStdJlxHub4mHrps/UYC33vurL2fXPcFB2lFWjmRqQel1CyeTfO8srAfRtA6BrmhBgu/Pgf\nnHXDxST370aX6SMY/79/0vPai+r0bTesN7EZKfXHI9ZK71/PbJIcCkWkUIpBUQchBBM+/Rfdfzah\nNo1z2pBeTFz2EMl5OZhsFiZ99Sgp53RHt1kwxdmwtWvLqFd+T9q5vWqP0+uX05oujJSGcTfIdNqa\nxURy366Nr2PhQWiC2M5pdJ19PlO+eYqM8wcw5JFfMvWbpxj79p9p78M7SAjBhf97gIScdEzxMZgT\nY9GtZnIuH8PZt192WnIoFJFGGZ8VDSKlRPi5MZcXFOEsrySxW8d6njPOymoW9Z/DifyC2vrHwqSj\nmU247MEbhk1xNmI7plJRcNQwarvd9LvnanY+8yEVh47hLLWjx1gRusb4xX/HltaW/57zi4bPITz/\neV37eqyV3F/NYOB91wUtmzdSSo6u3U7FoWOkDuhBXKfWXzNE0fJQXkmKiOMos7Pt0XfJn/8/EIJu\nV19AxvkD+Wj0rUErh7ShuUxa8SjHt+3DcbKc5H7dMMVYcTuc/LBoJUVrviMusz05V4zBlmJEfO9+\nYxkrrnsQTddx2qvq1K/QzCbMbWJJ6JpByaY9aBYT7spqcn5yPsP+cyvaac42FIqWgFIMiqilorCY\nDffOZ/+Ha9DMOjmzz+fg0m84tiG/TtlOPcbKBR/cR/qY/o0+h6O0goOffIvb6aLySAk7n/sIx8ly\nOk8ZRt7vZhHbIZmTuw5Svv8IbXO7ENMuqTk/okIRlSjFoGhROEorWH3Lv9n9xnKky01c53YMeeQm\nOk8eGmnRFIpWg1IMihaJ2+HEaa/CnBDr17ahUChOj7AEuAkhLhVCbBVCuD1V2/z1myCE2CGEyPeU\nAK1pzxZCrPG0LxBCtNBEPYrmQjObsCTGKaWgUESQprqrbgFmAF/46yCE0IHHgYlALnCFECLXs/kf\nwDwpZTegBLi2ifIoFAqFook0STFIKb+TUu4I0G0wkC+l3C2lrAbeAKYK45FwLPC2p99LQDM4vysU\nCoWiKYQjwK0jsN/r/QFPWwpwXErpPKVdoVAoFBEkYLUQIcQnQAcfm+6WUv63+UXyK8ccYA5AZmZm\nuE6rUCgUZxwBFYOU8vwmnqMA6Oz1vpOnrRhoK4QweWYNNe3+5HgaeBoMr6QmyqRQKBQKP4RjKelr\noLvHA8kCzAIWScNPdjlwiaffNUDYZiAKhUKh8E2T4hiEENOBfwNpwHFgg5TyQiFEBvCslPIiT7+L\ngIcBHXheSnmfpz0HwxidDKwHfiKlDJgrQQhRBOwLQsRU4GijP1joiVa5QMl2OkSrXBC9skWrXBC9\nsjWHXF2klAETebXIALdgEUKsCyaYI9xEq1ygZDsdolUuiF7ZolUuiF7ZwimXSrutUCgUijooxaBQ\nKBSKOrR2xfB0pAXwQ7TKBUq20yFa5YLolS1a5YLolS1scrVqG4NCoVAoGk9rnzEoFAqFopG0eMUQ\nrRlehRDJQoilQojvPX/rVYIRQowRQmzw+lcphJjm2faiEGKP17Z+zSFXsLJ5+rm8zr/Iqz2SY9ZP\nCLHK851vEkJc7rWt2cfM33Xjtd3qGYN8z5hkeW27y9O+QwhxYVNlaaRcvxFCbPOM0adCiC5e23x+\nr2GU7adCiCIvGa7z2naN5/v/XghxTZjlmucl004hxHGvbSEbMyHE80KII0KILX62CyHEox65Nwkh\nzvHaFprxklK26H9AL6An8Bkw0E8fHdgF5AAWYCOQ69n2JjDL8/pJ4MZmkusB4E7P6zuBfwTonwwc\nA2I9718ELgnRmAUlG1Dmpz1iYwb0ALp7XmcAhUDbUIxZQ9eNV5+5wJOe17OABZ7XuZ7+ViDbcxw9\njHKN8bqWbqyRq6HvNYyy/RR4zMe+ycBuz98kz+ukcMl1Sv+bMWKuwjFmI4FzgC1+tl8EfIRRrXwI\nsCbU49XiZwwyejO8TvUcL9jjXgJ8JKWsaKbzN0RjZasl0mMmpdwppfze8/ogcAQjwDIU+LxuGpD5\nbWCcZ4ymAm9IKauklHuAfM/xwiKXlHK517W0GiPlTDgIZsz8cSGwVEp5TEpZAiwFJkRIriuA15vp\n3A0ipfwC46HQH1OB+dJgNUYqoXRCOF4tXjEESSQyvLaXUhZ6Xh8C2gfoP4v6F+J9nqnjPCGEtZnk\naoxsNiHEOiHE6polLqJozIQQgzGe/nZ5NTfnmPm7bnz28YzJCYwxCmbfUMrlzbUYT5w1+Ppem4tg\nZZvp+Z7eFkLU5FKLijHzLLtlA8u8mkM5ZoHwJ3vIxitgEr1oQERJhtdTaUgu7zdSSimE8Ov+5dH+\necASr+a7MG6OFgw3tTuAe8MsWxcpZYEwUpcsE0JsxrjxnTbNPGYvA9dIKd2e5iaNWWtECPETYCAw\nyqu53vcqpdzl+wgh4X3gdSlllRDiFxgzrrFhPH8gZgFvSyldXm2RHrOw0iIUg4ySDK+NkUsIcVgI\nkS6lLPTcxI40cKjLgIVSSofXsWuenKuEEC8AtwcrV3PJJqUs8PzdLYT4DOgPvEOEx0wIkQh8iPFg\nsNrr2E0aMx/4u2589TkghDABbTCuq2D2DaVcCCHOx1C4o6RXDjI/32tz3eQCyialLPZ6+yyGbalm\n39Gn7PtZuOTyYhZwk3dDiMcsEP5kD9l4nSlLSZHI8LrIc7xgjltvPdNzY6xZ05+GUUa1uQgomxAi\nqWYpRgiRCgwHtkV6zDzf30KMNde3T9nW3GPm87ppQOZLgGWeMVoEzBKG11I20B1Y20R5gpZLCNEf\neAqYIqU84tXu83ttJrmClS3d6+0U4DvP6yXAeI+MScB46s6iQyqXR7azMAy5q7zaQj1mgVgEXO3x\nThoCnPA8BIVuvJrLsh6pf8B0jLW1KuAwsMTTngEs9up3EbATQ8vf7dWeg/GDzQfeAqzNJFcK8Cnw\nPfAJkOxpH4iRebamXxaG5tdO2X8ZsBnj5vYKEN+MYxZQNmCY5/wbPX+vjYYxA34COIANXv/6hWrM\nfF03GMtTUzyvbZ4xyPeMSY7Xvnd79tsBTGzm6z6QXJ94fg81Y7Qo0PcaRtn+Dmz1yLAcOMtr3597\nxjIf+Fk45fK8/zNw/yn7hXTMMB4KCz3X9QEMm9ANwA2e7QJ43CP3Zry8L0M1XiryWaFQKBR1OFOW\nkhQKhUIRJEoxKBQKhaIOSjEoFAqFog5KMSgUCoWiDkoxKBQKhaIOSjEoFAqFog5KMSgUCoWiDkox\nKBQKhaIO/w/7hkcUAFdc5AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10305b080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of X: (3000, 2)\n",
      "Shape of y: (3000,)\n"
     ]
    }
   ],
   "source": [
    "D = 2 #2 features\n",
    "K = 3 #3 classes\n",
    "N = 3000 #number of samples\n",
    "\n",
    "X,y = spiral_gen(int(N/K),D,K)\n",
    "print('Shape of X:',X.shape)\n",
    "print('Shape of y:',y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Weight Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We usually initialize the bias terms $b_i$ as zero vectors, and the weight terms $W_i$ as randomized matrices with different scaling terms.\n",
    "\n",
    "* Constant scaling\n",
    "* Xavier scaling: $\\sqrt{\\frac{1}{nodes_{previous}}}$\n",
    "* He scaling: $\\sqrt{\\frac{2}{nodes_{previous}}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def init_weights(D,K,h_list,scaling,const=0.1):\n",
    "    nb_layers = len(h_list) + 1\n",
    "    W_list = [None] * nb_layers\n",
    "    b_list = [None] * nb_layers\n",
    "    \n",
    "    #randomize weights\n",
    "    for i in range(nb_layers):\n",
    "        #scaling term\n",
    "        if scaling=='constant':\n",
    "            scaler = const\n",
    "        elif scaling=='xavier':\n",
    "            scaler = np.sqrt(1.0 / h_list[i - 1])\n",
    "        elif scaling=='he':\n",
    "            scaler = np.sqrt(2.0 / h_list[i - 1])\n",
    "        else:\n",
    "            raise ValueError('Can only use \"constant\",\"xavier\" or \"he\" scaling')\n",
    "            \n",
    "        #input layer\n",
    "        if (i == 0):\n",
    "            W_list[i] = np.random.randn(D,h_list[i])\n",
    "            b_list[i] = np.zeros((1,h_list[i]))\n",
    "        #output layer\n",
    "        elif (i==nb_layers-1):\n",
    "            W_list[i] = np.random.randn(h_list[i-1],K) * scaler\n",
    "            b_list[i] = np.zeros((1,K))\n",
    "        #hidden layers\n",
    "        else:\n",
    "            W_list[i] = np.random.randn(h_list[i-1],h_list[i]) * scaler\n",
    "            b_list[i] = np.zeros((1,h_list[i]))\n",
    "    return W_list, b_list\n",
    "\n",
    "W_list, b_list = init_weights(2,3,[100,50],'he')\n",
    "W, b = W_list[0], b_list[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Loss Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Loss functions are objective functions to be minimized during the training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Cross-Entropy Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "For each example, the loss is the negative log of predicted probability of the correct class\n",
    "$$L_i = -\\sum_{k=1}^{K} y_klog(p_k)$$\n",
    "where \n",
    "* $y_k$ is the true probability of the class; in 1 or 0\n",
    "* $p_k$ is probability of a class\n",
    "* $i$ is an example\n",
    "\n",
    "The higher probability of the correct class, the lower the loss (for incorrect classes $y_k = 0$ so loss is 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'loss')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8XNV99/HPT7sljSRrt2XJ8iIv\nIBA2XsEEBwihhCVpEgIEkjQ0tDSkPE2atuny5OmetE2eNiUhcSANZDEUkoBDIJCExWw2yPu+yfKm\nXbK1etFy+seMVUG8jLHu3NHc7/v1mpdGozu+v2PZ33vn3HPPMeccIiKS+JL8LkBERGJDgS8iEhAK\nfBGRgFDgi4gEhAJfRCQgFPgiIgGhwBcRCQgFvohIQCjwRUQCIsXvAkYqLCx0lZWVfpchIjJmrFmz\nps05VxTNtnEV+JWVldTW1vpdhojImGFm+6LdVl06IiIBocAXEQkIBb6ISEAo8EVEAkKBLyISEAp8\nEZGAUOCLiATEmA/8oSHH/S/sYuXOVr9LERGJa2M+8JOSjO+srOM325r9LkVEJK6N+cAHKM3JoLnr\nuN9liIjEtYQI/JKcDJq6jvldhohIXEuYwG9W4IuInFFCBH5pbjot3ccZGnJ+lyIiErcSIvBLcjIY\nHHK09aofX0TkdBIm8AGaOxX4IiKnkxCBX3oy8NWPLyJyWgkR+CfP8DVSR0Tk9BIi8Auz00gyaFHg\ni4icVkIEfkpyEkWhdJ3hi4icQUIEPpy8+UoXbUVETiehAl9dOiIip5cwgV+q6RVERM4oYQK/JCed\nI339HOsf9LsUEZG4lECBHx6a2aJ+fBGRU0qYwC/N1Vh8EZEzSZjA181XIiJnlnCBr5E6IiKnljCB\nn5ORwrjUZJo6FfgiIqeSMIFvZpTk6G5bEZHTSZjAh5M3X2mUjojIqSRU4Jfm6uYrEZHT8TzwzSzZ\nzNaZ2dNe7+vkYubOaalDEZF3isUZ/n3Athjsh5KcDE4MDNF5tD8WuxMRGVM8DXwzmwR8AHjQy/2c\nVKqx+CIip+X1Gf6/A38GDJ1uAzO728xqzay2tbX1vHZWkpMOoKGZIiKn4Fngm9kNQItzbs2ZtnPO\nLXPOzXPOzSsqKjqvfWo+HRGR0/PyDP9y4CYzqwceBa4ysx96uD+KT57hq0tHROS3eBb4zrkvOecm\nOecqgVuBF5xzd3i1P4D0lGTys9IU+CIip5BQ4/ABysePY197r99liIjEnZgEvnPuJefcDbHY16zS\nHLY1dmssvojIOyTcGf6sCSE6ek/Q2qMLtyIiIyVc4M8sDQGwo6nb50pEROJLwgX+rNIcALY3KvBF\nREZKuMDPz0qjOJTOdp3hi4i8TcIFPsCsCTlsb+ryuwwRkbiSmIFfGmJXSw8Dg6ed0UFEJHASNvBP\nDAxRr/H4IiLDEjLwT47U2aYLtyIiwxIy8KcXZ5OcZBqaKSIyQkIGfnpKMtOKsnThVkRkhIQMfICZ\nkSkWREQkLGEDf1ZpiENHjtJ1TMsdiohAggc+wE7144uIAIkc+BMiUywo8EVEgAQO/Im5GYQyUnTh\nVkQkImED38yYXZrD5kMKfBERSODAB5hXOZ7NhzrpPT7gdykiIr5L6MBfPK2AgSFH7b7DfpciIuK7\nhA78SyePJyXJWFXX7ncpIiK+S+jAz0xLoaY8T4EvIkKCBz7A4qkFbDzYSY/68UUk4BI+8BdNLWBw\nyFFb3+F3KSIivkr4wJ87OY/UZGNVnQJfRIIt4QM/My2Fmkl5vKF+fBEJuIQPfAgPz9x8qJNuTaQm\nIgEWiMAf7sfXeHwRCbBABP7civGRfnx164hIcAUi8MelJTOnfDxv7FHgi0hwBSLwAa6cWcTGg500\ndR7zuxQREV8EJvDff2EpAM9tafK5EhERfwQm8KcXZ1NVnM2zmxv9LkVExBeBCXyA36ku5c29HbT3\nHPe7FBGRmAtU4F9XPYEhB89vbfa7FBGRmAtU4M+eEGJyQSa/3Kx+fBEJHs8C38wyzOxNM9tgZlvM\n7G+92tc51MR1F5by+p42Oo/qrlsRCRYvz/CPA1c552qAS4DrzGyRh/uLynXVpfQPOn6zTd06IhIs\nngW+C+uJfJsaeTiv9hetmkl5TMjN4Fl164hIwHjah29myWa2HmgBfuWcW32Kbe42s1ozq21tbfWy\nHACSkozrqkt5eWcrnX3q1hGR4PA08J1zg865S4BJwAIzqz7FNsucc/Occ/OKioq8LGfYRy6dxImB\nIX627mBM9iciEg9iMkrHOXcEeBG4Lhb7O5sLJ+Zy8aRcHn3rAM753sskIhITXo7SKTKzvMjzccD7\ngO1e7e9c3baggu1N3aw7cMTvUkREYsLLM/wJwItmthF4i3Af/tMe7u+c3Fgzkcy0ZB59c7/fpYiI\nxISXo3Q2OufmOOcuds5VO+f+zqt9vRvZ6SncVDORn29o1EpYIhIIgbrT9p1uW1DB0f5Bnlrf4Hcp\nIiKeC3TgXzwpl9kTcliubh0RCYBAB76ZcfvCCrY0dPHm3g6/yxER8VSgAx/gI3MnkZ+Vxrdf3uN3\nKSIingp84I9LS+ZTl1XywvYWtjd1+V2OiIhnAh/4AJ9YPJnMtGS+83Kd36WIiHhGgQ/kZaZx+4IK\nVmxo4EBHn9/liIh4QoEfcdcVU0gyePAVneWLSGJS4EdMyB3Hh+aU8ehbB2jt1pq3IpJ4FPgj3LN0\nOgNDjvtf2OV3KSIio06BP8KUwixunV/Oj1bvp76t1+9yRERGlQL/He67uorU5CT+7fkdfpciIjKq\nFPjvUJyTwWeumMLTGxvZeFBTJ4tI4lDgn8Jn3jOV/Kw0vvLsdi2QIiIJQ4F/CqGMVD531XRe39PO\nC9tb/C5HRGRUKPBP4+MLJzO9OJsvr9jC0RODfpcjInLeFPinkZaSxN/fXM3Bw0e5/0UN0xSRsS+q\nwDez+8wsx8IeMrO1Znat18X5bfG0An53ThnLVtaxu6Xb73JERM5LtGf4n3bOdQHXAuOBO4GveFZV\nHPnLD8xmXGoyf/3kZl3AFZExLdrAt8jX64EfOOe2jHgtoRVmp/Nn181iVV0HT6w56Hc5IiLvWrSB\nv8bMnicc+M+ZWQgY8q6s+HL7ggrmV47n736+lUNHjvpdjojIuxJt4N8F/AUw3znXB6QCv+dZVXEm\nKcn42kcvYdA5/uyJDQwNqWtHRMaeaAN/MbDDOXfEzO4A/hro9K6s+FNRkMlff+ACXtvdzg9W7fO7\nHBGRcxZt4D8A9JlZDfAFYA/wiGdVxanbFpSzdGYR//zsNva09vhdjojIOYk28AdceIjKzcD9zrlv\nAiHvyopPZsa/fPhiMlKT+dyP13GsXzdkicjYEW3gd5vZlwgPx/yFmSUR7scPnOKcDL5+Sw1bG7v4\n259v9bscEZGoRRv4HwOOEx6P3wRMAv7Vs6ri3FWzSvjDK6ex/M39PLnukN/liIhEJarAj4T8j4Bc\nM7sBOOacC1wf/kh/eu0MFlTm85c/26S7cEVkTIh2aoVbgDeBjwK3AKvN7CNeFhbvUpKT+MZtcxiX\nmszdj6yhs6/f75JERM4o2i6dvyI8Bv+TzrlPAAuAv/GurLGhNDeDB+64lAOH+7h3+VoGBgNzL5qI\njEHRBn6Sc27kxPDt5/DehLZgSj7/+MGLeGVXG//wi21+lyMiclopUW73SzN7Dlge+f5jwDPelDT2\n3DK/nJ3N3Tz46l6mF2dzx6LJfpckIvJbogp859wXzezDwOWRl5Y5537mXVljz5eun01dWy//96nN\nFIfSufbCUr9LEhF5m6i7ZZxzP3HOfT7yUNi/Q3KScf/tc7hoUh6fW76O2voOv0sSEXmbMwa+mXWb\nWdcpHt1m1nWW95ab2YtmttXMtpjZfaNbevzJTEvhvz41n7K8cdz1cC07mzVcU0TixxkD3zkXcs7l\nnOIRcs7lnOXPHgC+4Jy7AFgEfNbMLhitwuNVflYaD396AekpSdz50Gr2tff6XZKICODhSBvnXKNz\nbm3keTewDSjzan/xpDw/kx/ctZATA0Pc/t3VHOjo87skEZHYDK00s0pgDrD6FD+728xqzay2tbU1\nFuXExMzSED+4ayHdx/q5/cFVNGjhFBHxmeeBb2bZwE+A/xNZF/dtnHPLnHPznHPzioqKvC4npqrL\ncvnBXQs50tvP7d9dpdWyRMRXnga+maUSDvsfOed+6uW+4lVNeR4P37WA9t4T3PLtN6hvU5++iPjD\ns8A3MwMeArY5577u1X7GgrkV41n+mUUc7R/klu+8wS6N3hERH3h5hn854fnzrzKz9ZHH9R7uL65V\nl+Xy2N2LALjlO2+wbv9hnysSkaDxcpTOq845c85d7Jy7JPII9HQMVSUhHv/DxeSMS+X2767mhe3N\nfpckIgGiCdBibHJBFk/84WVML87mM4+s4bG39vtdkogEhALfB0WhdB69exGXTy/kz3+yiX/55XaG\nhpzfZYlIglPg+yQrPYWHPjmP2xZU8K2X9nDPj9bQd2LA77JEJIEp8H2UmpzEP32omr+54QJ+tbWZ\nj377DY3VFxHPKPB9ZmbctWQKD31yPvvb+7jxP1/l9T1tfpclIglIgR8n3jurmCfvvZz8rDTufOhN\nHnylDufUry8io0eBH0emFWXz5Gcv59oLSviHX2zjnh+upfOoFkcXkdGhwI8z2ekpfOvjc/nL62fx\n623N3PCfr7Dx4BG/yxKRBKDAj0Nmxt3vmcZjf7CYwUHHhx94nQdfqdPQTRE5Lwr8OHbp5PE8c98V\nLJ1ZzD/8Yhuf+N6bNHcd87ssERmjFPhxLi8zjWV3Xso/fegiavd18P5/X8mzmxr9LktExiAF/hhg\nZty+sIKnP3cF5eMzuedHa/nj5es40nfC79JEZAxR4I8h04uz+ekfXcafXDODZzY18r7/v5Jfb9UE\nbCISHQX+GJOanMR911Tx5GcvpyArjd9/pJZ7f7yWtp7jfpcmInFOgT9GVZflsuLeJXz+fTN4fksz\n13z9ZR6vPaCbtUTktBT4Y1haShJ/fHUVz9y3hGlF2XzxiY3cumwVu1u0opaI/DYFfgKYXhzi8T9Y\nzD//7kVsb+rmd/7jFb76y+2afVNE3kaBnyCSkozbFlTwwheu5KaaMh54aQ9X/dvLrNjQoG4eEQEU\n+AmnIDudr91Sw0/uWUxBdhp/vHwdH1u2is2HOv0uTUR8psBPUJdOzmfFvUv4xw9Vs7ulhxvvf5U/\nfXyD7tQVCTAFfgJLTjI+vnAyL31xKXdfMZUV6xtY+q8v8fXnd9BzXP37IkGjwA+AnIxUvnT9bH79\n+Su5enYx33hhN1f+y4s8/Ho9JwaG/C5PRGJEgR8gFQWZ3H/7XJ767OVUlWTz5RVbuOprL/HEmoMM\naiZOkYSnwA+gmvI8ln9mEd//vfnkZabyp49v4P3/vpKnNzZoCmaRBKbADygzY+nMYn5+7xIe+Phc\nAO798Tp+5z9e4ZlNjQp+kQRk8TRGe968ea62ttbvMgJpcMjx9MYGvvGbXexp7WVGSTaffe90brh4\nIslJ5nd5InIaZrbGOTcvqm0V+DLSyeC//4Xd7GrpYUphFvdcOY0PzikjLUUfCEXijQJfztvQkOP5\nrU385wu72dLQRWlOBr9/xRRuW1BBVnqK3+WJSIQCX0aNc46Vu9p44KXdrKrrICcjhTsWTeZTl1VS\nnJPhd3kigafAF0+s23+YZSvr+OWWJlKSjJsvKePTl0/hgok5fpcmElgKfPHUvvZeHnxlL0+sOcjR\n/kEWTy3g00umcNWsYl3gFYkxBb7ERGdfP8vf2s/Dr9fT2HmM8vxx3LloMh+bV0FuZqrf5YkEggJf\nYmpgcIjntzbz/dfqebO+g4zUJG6uKePOxZOpLsv1uzyRhKbAF99saejkh6v28eS6Bo72D1JTnsfH\nF1RwQ80EMtM0ukdktMVF4JvZ94AbgBbnXHU071HgJ47Oo/38dO1BfrhqH3taewmlp/DBOWXcuqCc\nCyfqrF9ktMRL4L8H6AEeUeAHl3OOt+oP8+PV+3hmcxMnBoa4qCyXj80v58aaieSOU1+/yPmIi8CP\nFFIJPK3AF4AjfSd4ct0hHn3rANubuklPSeK66lI+emk5l00rIEkjfETO2ZgKfDO7G7gboKKi4tJ9\n+/Z5Vo/EB+ccmw518njtQZ5af4iuYwNMzM3gg3PK+PClk5hWlO13iSJjxpgK/JF0hh88x/oHeX5r\nMz9de5CVO1sZclAzKZcPzinjxpqJFGan+12iSFxT4MuY1NJ1jKfWN/CzdYfY2thFcpJxRVUhN9VM\n5NoLS8nWHD4iv0WBL2PejqZunlx/iBXrGzh05CjpKUlcM7uEG2smsHRmMRmpyX6XKBIX4iLwzWw5\nsBQoBJqBLzvnHjrTexT48k5DQ461+w/z1PoGnt3cSFvPCbLTU7hmdjEfuHgiV1QVKvwl0OIi8N8N\nBb6cycDgEKvqOvj5hgae29rEkb7+4fC/rnoCS2cWKfwlcBT4kvD6B4d4fU87z2xsHA7/zLRk3juz\nmPdXl/LemUWEMjTGXxKfAl8CpX9wiNV1HTyzuZHntzTR1nOCtOQkLp9ewLUXlnL17GKKQ5q7XxKT\nAl8CazDS5//c5iae29rEgY6jmMGc8jyuuaCEay8oYVpRNma6yUsSgwJfhPANXjuau3l+SzPPb21i\n86EuACYXZHL1rBKunl3M/Mp8rdUrY5oCX+QUGjuP8pttLfxqazNv1LVzYmCI7PQUrqgq5L2zilk6\ns0hdPzLmKPBFzqL3+ACv7W7jhe0tvLijheau4wBcVJbL0plFLJ1ZxCXl47WCl8Q9Bb7IOXDOsbWx\nixe3t/DSjlbW7j/MkIPccaksmV7IlTOKeM+MIkpzdfYv8UeBL3IeOvv6eWV3Ky/vaGXlrtbhs/8Z\nJdlcUVXEkqpCFk7J14IuEhcU+CKjxDnH9qZuXtnVyiu72li9t4MTA0OkJScxd3IeS6YXcvn0Qi4q\nyyUlWRd/JfYU+CIeOdY/yFv1Hby6q41XdrWxtTE88ieUkcKiqQVcPq2Ay6YXUlWsoZ8SGwp8kRhp\n7znO63vaeX1PG6/ubuNAx1EACrPTWDS1gMXTClg8tYAphVk6AIgnFPgiPjnQ0ccbkQPAG3Xtw/3/\nJTnpLJpawKKpBSyckq8DgIyacwl8XXUSGUXl+ZmU52dyy/xynHPsbevl9T3trN7bwet72nlqfQMA\nRaF0FkzJZ+GUfBZMyWdGcUhLPIrnFPgiHjEzphZlM7UomzsWTR4+AKze28HquvBB4BcbG4HwEND5\nleOZV5nP/Mp8LirL1R3AMuoU+CIxMvIAcNuCCpxzHDx8lDf3dvDm3g7equ/g19taAEhPSaKmPI95\nk8czvzKfuRXjyc3U7J9yftSHLxJHWruPU1vfQe2+w9TWd7CloYuBofD/0aribC6dPJ65k8czt2I8\n04p0HUB00VYkYfSdGGDDgU7W7AsfBNbtP0Ln0X4g3A00pyKPuRXhA0BNea7WAAggXbQVSRCZaSnh\noZ3TCoDwko91bT2s3XeENfsOs3b/YV7a0QqAWfhTwJzy8VxSkUfNpDxmlGTrhjAZpjN8kTGu82g/\n6w8cYf3+I6w/cJh1B45wpC/8KSAzLZnqslwuKQ8fAGrKcynLG6euoASiM3yRAMkdl8qVM4q4ckYR\nEJ4OYl97X/ggEHl8/7V6TgwOAVCQlcbFk3K5OHIAuHhSHoXZ6X42QWJEgS+SYMyMysIsKguz+OCc\nMgBODAyxrbGLjQePsOFgJxsOHOGlna2c/IA/MTeDiyIHgeqyXC4qyyU/K83HVogXFPgiAZAWGeZZ\nU57HnZHXeo8PsKUhfBDYdKiTTQc7eW5L8/B7yvLGUV2WQ/XEXKrLcrmwLEcLxIxxCnyRgMpKT2FB\n5E7fkzqP9rOloZPNhzrZdKiLzYfefhAoDqWHw39iTuSRy6TxuiYwVijwRWRY7rhULptWyGXTCodf\n6z7Wz9aGLjYd6mRrQxdbGrp4eWcrg5H7A3IyUpg9IRz+F0zMYfaEEFXFId0pHIcU+CJyRqGMVBZO\nLWDh1ILh1471D7K9qZstDZ1saehiW2MXy9/cz9H+QQBSk41pRdlcMCGH2cOPEAW6OOwrBb6InLOM\n1GQuKc/jkvK84dcGh8JzBW1r7GJrYxdbG7p4dXcbP113aHibolB6OPxLQ8yaEGJmSQ7Ti7P1aSBG\nFPgiMiqSk4zpxdlML87mxpqJw6+39xxnW2M325u62NbYzbbGLv5rT/vwMNGUJGNqURYzS3OYVRpi\nZkmImaUhyvLGaQbRUabAFxFPFWSns6QqnSVV/3tdoH9wiPq2XrY1dbOjqYsdTd2s23+Yn29oGN4m\nKy2ZqpLwAWBGaYgZJdnMKAlRHErXReJ3SYEvIjGXmpxEVUmIqpIQjPg00H2sn53NPeyIHAh2Nvfw\nq23NPFZ7YHib3HGpzCjJpqokxIzi8EGgqiREYXaaDgRnocAXkbgRykjl0snjuXTy+Le93tZznJ3N\n3exs6mZHcw+7mrt5ekMDXccGhrfJy0ylqjib6cUhqoqzqSoJdy+V5mToQBChwBeRuFeYnU5hdvrb\nhos652jpPs6u5h52tXSzs7mH3S3dPLu5keWRuYQAstNTmFaUxbTI9YXpReGvFfmZgZtYToEvImOS\nmVGSk0FJTsbbrg8452jrOcHulh52t/awu7mb3a09vLa7jZ+u/d8RQ6nJRmVBFtOKsplWHP4aXqAm\ni5wEnWZagS8iCcXMKAqlUxRKH55W+qSuY/3UtfaGDwYtPexp7WFnSze/2tY8fCMZhIePTivKCh8A\nCk8eDLIoyxs3pj8VKPBFJDByMlJ/6/4BCE8ut7+jjz2tPdS19ka+9vDMpsbhqaYh/KlgckEWUwqz\nmFoY/jqlMIspRVkUZcf/6CFPA9/MrgP+A0gGHnTOfcXL/YmIvBtpKUnD9xC8U0fvCepOHgjaeqhv\n66WutZeXd7QO30sA4WsFlYWZVBaEDwaVIw4IeZnxMfOoZ4FvZsnAN4H3AQeBt8xshXNuq1f7FBEZ\nbflZaeRn5TOvMv9trw8OORqOHKWurZe9rT3Ut/dR19bLhoNHeGZTIyN6iMgdlxo+ABRkMrkga/jA\nUFmQRV5masw+GXh5hr8A2O2cqwMws0eBmwEFvoiMeclJRnl+JuX5mcOLz5x0fGCQAx1HqW/rpb69\nl72Rr2/VH+apDQ2MXGgwJyOFmaUh/vsPFnse/F4GfhlwYMT3B4GF79zIzO4G7gaoqKjwsBwRkdhI\nT0k+bRdR+GDQx772Purb+9jX3suJgaGYnOX7ftHWObcMWAbhNW19LkdExFPhg0GI6cWhmO/by/FF\nh4DyEd9PirwmIiI+8DLw3wKqzGyKmaUBtwIrPNyfiIicgWddOs65ATO7F3iO8LDM7znntni1PxER\nOTNP+/Cdc88Az3i5DxERic7YvUdYRETOiQJfRCQgFPgiIgGhwBcRCQhzLn7udTKzVmDfObylEGjz\nqJx4pnYHi9odLOfa7snOuaKzbxZngX+uzKzWOTfP7zpiTe0OFrU7WLxst7p0REQCQoEvIhIQYz3w\nl/ldgE/U7mBRu4PFs3aP6T58ERGJ3lg/wxcRkSjFfeCb2XVmtsPMdpvZX5zi5+lm9ljk56vNrDL2\nVY6+KNr9eTPbamYbzew3ZjbZjzq9cLa2j9juw2bmzCwhRnJE024zuyXye99iZj+OdY1eiOLfeoWZ\nvWhm6yL/3q/3o87RZGbfM7MWM9t8mp+bmX0j8ney0czmjsqOnXNx+yA8y+YeYCqQBmwALnjHNn8E\nfDvy/FbgMb/rjlG73wtkRp7fkwjtjrbtke1CwEpgFTDP77pj9DuvAtYB4yPfF/tdd4zavQy4J/L8\nAqDe77pHod3vAeYCm0/z8+uBZwEDFgGrR2O/8X6GP7wurnPuBHByXdyRbgYejjx/ArjaYrUisHfO\n2m7n3IvOub7It6sILzCTCKL5nQP8PfBV4Fgsi/NQNO3+DPBN59xhAOdcS4xr9EI07XZATuR5LtAQ\nw/o84ZxbCXScYZObgUdc2Cogz8wmnO9+4z3wT7UubtnptnHODQCdQEFMqvNONO0e6S7CZwOJ4Kxt\nj3y8LXfO/SKWhXksmt/5DGCGmb1mZqvM7LqYVeedaNr9/4A7zOwg4enWPxeb0nx1rhkQFd/XtJXz\nY2Z3APOAK/2uJRbMLAn4OvApn0vxQwrhbp2lhD/RrTSzi5xzR3ytynu3Ad93zn3NzBYDPzCzaufc\nkN+FjTXxfoYfzbq4w9uYWQrhj3ztManOO1GtB2xm1wB/BdzknDseo9q8dra2h4Bq4CUzqyfcv7ki\nAS7cRvM7PwiscM71O+f2AjsJHwDGsmjafRfw3wDOuTeADMLzzSQyT9YEj/fAj2Zd3BXAJyPPPwK8\n4CJXPcaws7bbzOYA3yEc9onQl3vSGdvunOt0zhU65yqdc5WEr1/c5Jyr9afcURPNv/UnCZ/dY2aF\nhLt46mJZpAeiafd+4GoAM5tNOPBbY1pl7K0APhEZrbMI6HTONZ7vHxrXXTruNOvimtnfAbXOuRXA\nQ4Q/4u0mfBHkVv8qHh1RtvtfgWzg8cg16v3OuZt8K3qURNn2hBNlu58DrjWzrcAg8EXn3Jj+NBtl\nu78AfNfM/oTwBdxPjfWTOjNbTvjgXRi5NvFlIBXAOfdtwtcqrgd2A33A743Kfsf435uIiEQp3rt0\nRERklCjwRUQCQoEvIhIQCnwRkYBQ4IuIBIQCX+Q8mNlLCXDTlwSEAl/kLMws2e8aREaDAl8Czcwq\nzWy7mf3IzLaZ2RNmlmlm9Wb2VTNbC3zUzC6JTFi20cx+ZmbjR/wxd5rZejPbbGYL/GqLyNko8EVg\nJvAt59xsoIvwGgsA7c65uc65R4FHgD93zl0MbCJ8Z+RJmc65SyLv+14M6xY5Jwp8ETjgnHst8vyH\nwJLI88cAzCwXyHPOvRx5/WHCC1ictByG5zjPMbM870sWOXcKfJHw/Cyn+r73PN8vElcU+CJQEZln\nHeB24NWRP3TOdQKHzeyKyEt3Ai+P2ORjAGa2hPCshp0e1yvyrsT1bJkiMbID+KyZfQ/YCjzAb6+q\n9Eng22aWSXhK4pGzFx4zs3XEWB18AAAATUlEQVSEZzv8dAzqFXlXNFumBJqZVQJPO+eqfS5FxHPq\n0hERCQid4YuIBITO8EVEAkKBLyISEAp8EZGAUOCLiASEAl9EJCAU+CIiAfE/zYrdCaBgJJMAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1080f7f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p = np.arange(0.01, 1, 0.01)\n",
    "plt.plot(p,-np.log(p))\n",
    "plt.xlabel('prob')\n",
    "plt.ylabel('loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cross entropy loss: 0.9982815421178499\n"
     ]
    }
   ],
   "source": [
    "def cross_entropy(probs,y):\n",
    "    #probs: N x K\n",
    "    N = probs.shape[0]\n",
    "    #print('All normalized probs:',probs.shape)\n",
    "    \n",
    "    #normalized prob of correct label\n",
    "    correct_logprobs = -np.log(probs[range(N),y])\n",
    "    #print('-log(normalized probs) when label is correct:',correct_logprobs.shape)\n",
    "    \n",
    "    #data loss\n",
    "    data_loss = np.sum(correct_logprobs) / N\n",
    "    \n",
    "    return(data_loss)\n",
    "\n",
    "#test\n",
    "probs = np.random.random((3000,3))\n",
    "print('Cross entropy loss: {}'.format(cross_entropy(probs,y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Hinge Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "For an example, the loss per class per example is positive only when the score of wrong class is more than score of correct class by margin.\n",
    "\n",
    "$$L_{i} = \\sum_{other \\ne correct}^{K} max(0,scores_{other} - scores_{correct} + \\Delta)$$\n",
    "\n",
    "Loss is positive only when the score of correct class is lower than an incorrect class, by magnitude more than margin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQcAAAELCAYAAADUaUPcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAECdJREFUeJzt3XvsX3V9x/HnyxakhSIoNdMCk0SR\nGZkDq4KokeuYGNyM8xIx6hZJjAqo87qpWzK3OJHIjJJVkGEgmImYEXUC3nVbkAJuFUoZQeTqrFHk\noo7be398T9kP9unvd1p+p+f7bZ+P5Jt+z/l9z/m82/T3+p7r+6SqkKRHeszYBUiaToaDpCbDQVKT\n4SCpyXCQ1GQ4SGoyHCQ1GQ6SmgwHSU1Lxy5griRlWknDeRCoqvT57FSFw2OAXcYuQtqO/WYLPusX\ntaQmw0FSk+EgqclwkNRkOEhqMhwkNQ0aDkmOTbIhyfVJ3jvkWJIWV4ZqE5dkCXAdcDRwC3A58Jqq\numZzyyxJyuscpOH8Bnig50VQQ245PBe4vqpuqKp7gc8BLxtwPEmLaMhwWAXcPGf6lm7ewyQ5Mcna\nJGttdStNj9Evn66qNcAamOxWjFyOpM6QWw63AvvMmd67mydpBgwZDpcDT0uyX5KdgVcDFw04nqRF\nNNhuRVXdn+StwMXAEuAzVXX1UONJWlyDncrcGp7KlIY1LacyJc0ww0FSk+EgqclwkNRkOEhqGv0K\nyblWAe8cYL3vH2Cd0vbOLQdJTYaDpCbDQVKT4SCpyXCQ1GQ4SGoyHCQ1GQ6SmgwHSU2Gg6Qmw0FS\nk+EgqclwkNRkOEhqMhwkNRkOkpoMB0lNhoOkJsNBUpPhIKnJcJDUNFXdp1cARwyw3r8ZYJ1gV2tt\n39xykNRkOEhqMhwkNRkOkpoMB0lNhoOkpsHCIck+Sb6Z5JokVyc5eaixJC2+Ia9zuB94Z1VdmWQF\ncEWSS6vqmgHHlLRIBttyqKrbq+rK7v1dwHpg1VDjSVpc2+SYQ5KnAAcBl22L8SQ9eoOHQ5LdgC8A\np1TVnY2fn5hkbZK1vxi6GEm9DRoOSXZiEgznVdWFrc9U1ZqqWl1Vq/ccshhJW2TIsxUBzgLWV9Vp\nQ40jaRhDbjkcBrwOOCLJD7rXSwYcT9IiGuxUZlV9D8hQ65c0LK+QlNRkOEhqMhwkNRkOkpoMB0lN\nU9VgdtlSOPAJA6z4vwdYJ8M0rrVpraaFWw6SmgwHSU2Gg6Qmw0FSk+EgqclwkNRkOEhqMhwkNRkO\nkpoWDIckh/WZJ2n70mfL4RM950najmz23ookhwLPB1YmececH+0OLBm6MEnjmu/Gq52B3brPrJgz\n/07gFUMWJWl8mw2Hqvo28O0k/1hVP96GNUmaAn2OOZyZZI9NE0n2THLxgDVJmgJ9wmGvqrpj00RV\n/QJ44nAlSZoGfcLhwST7bppI8ttADVeSpGnQpxPUnwPfS/JtJs+heCFw4qBVSRrdguFQVV9NcjBw\nSDfrlKr62bBlSRpbnyskAxwLHFxVXwKWJ3nu4JVJGlWfYw6fAg4FXtNN3wV8crCKJE2FPsccnldV\nBye5CiZnK5LsPEg1uwNHLf5qD/za4q8TGKSr9RAdrcGu1tpyfbYc7kuyhO4MRZKVwIODViVpdH3C\n4e+BLwJPTPJh4HsM9wUnaUr0OVtxXpIrgCOZnMr8w6paP3hlkkY1bzh0uxNXV9UBwLXbpiRJ02De\n3YqqegDYMPcKSUk7hj5nK/YErk7yfeCeTTOr6vjBqpI0uj7h8IHBq5A0dfocc/jLqjp8awfo1rEW\nuLWqXrq165G0bfU55vBgksc9ijFOBjy7Ic2YPrsVdwPrklzKw485nLTQgkn2Bo4DPgy8Y4GPS5oi\nfcLhwu61NT4OvJuH96B8mCQn0t0Cvu/yrRxF0qLrcxHUOd29FPt3szZU1X0LLZfkpcBPq+qKJC+e\nZ/1rgDUAq58Qm8hIU2LBcOh+sc8BbmRyheQ+SV5fVd9ZYNHDgOOTvATYBdg9yblVdcKjK1nSttBn\nt+JjwDFVtQEgyf7A+cCz51uoqt4HvK9b5sXAnxkM0uzoc+PVTpuCAaCqrgN2Gq4kSdOgz5bD2iRn\nAud2069lct1Cb1X1LeBbW1SZpFH1CYc3A28BNp26/C6T7lCStmN9wmEpcHpVnQYPXfH42EGrkjS6\nPsccvg4smzO9DBiq8ZqkKdEnHHapqrs3TXTvvVxJ2s71CYd7uudWAJDk2cCvhytJ0jToc8zhFODz\nSW5jchHUbwGvGqSaFcARg6x5EIN0tR6gozXY1Vpbrs/l05cnOQB4ejfrYZdPJzm6qi4dqkBJ4+iz\nW0FV3VdVP+xej7yv4iMD1CVpZL3CYQFZhHVImjKLEQ7eSSlthxYjHCRthxYjHG5chHVImjILhkOS\nP06yonv/F0kunHvdQ1W9fMgCJY2jz5bDB6rqriQvYPIM7LOAM4YtS9LY+oTDA92fxwFrqurLwM7D\nlSRpGvQJh1uT/AOTqyK/kuSxPZeTNMP6/JK/ErgY+P2qugN4PPCuQauSNLoFw6GqfgX8FHhBN+t+\n4L+GLErS+PqcrfgQ8B66ZrFM+keeu/klJG0P+uxW/BFwPN3TrqrqNuZ5SI2k7UOfcLi3qoruMukk\nuw5bkqRp0Ccc/qk7W7FHkjcxaRH36WHLkjS2Pv0cTk1yNHAnk54OH7R/g7T9mzccuk7TX6uqwwED\nQdqBzLtbUVUPAA8medw2qkfSlOjTQ/JuYF2SS+nOWABU1UmbX0TSrOsTDhd2L0k7kD4HJM9JsjOw\nfzdrQ6OP5OLYncl9nzuwQTpag12ttcUWDIckLwbOYdLUJcA+SV5fVd8ZtjRJY+qzW/Ex4Jiq2gCQ\nZH/gfODZQxYmaVx9LoLaaVMwAFTVdUzur5C0Heuz5bA2yZn8381WrwXWDleSpGnQJxzeDLwF2HTq\n8rvApwarSNJU6BMOS4HTq+o0eOiqyccOWpWk0fU55vB1YNmc6WVMbr5aUJI9klyQ5Nok65McujVF\nStr2+mw57FJVd2+aqKq7kyzvuf7Tga9W1Su6ayX6LidpZH22HO6Z+5yKJKuBXy+0UHc/xouYtLKn\nqu7telBKmgF9thxOBj6f5LZu+klMOlEvZD9gI3B2kmcBVwAnV9U98y8maRr02XLYDziIyVmLS4EN\n9Ht47lLgYOCMqjqIyU1b733kh5KcmGRtkrUbf967bkkD6/vEqzuBPYDDmZzG7PPEq1uAW6rqsm76\nAiZh8TBVtaaqVlfV6pWP71m1pMFt6ROvPt33iVdV9RPg5iRP72YdCVyzVVVK2ub6HHPY9MSro4GP\nbOETr94GnNedqbgBeOPWlSlpW+sTDq8EjgVOrao7kjyJnk+8qqofAKsfRX2SRtKnn8OvmNPspapu\nB24fsihJ4/OBuJKaDAdJTYaDpCbDQVKT4SCpqc+pzG1n5+Ww7zMXf71HfX/x1zlj7Go9YVfr/txy\nkNRkOEhqMhwkNRkOkpoMB0lNhoOkJsNBUpPhIKnJcJDUZDhIajIcJDUZDpKaDAdJTYaDpCbDQVKT\n4SCpyXCQ1GQ4SGoyHCQ1GQ6SmqarwSwrgRMXf7X7Lv4qARvXYuNa2H6b1rrlIKnJcJDUZDhIajIc\nJDUZDpKaDAdJTYaDpKZBwyHJ25NcneSHSc5PssuQ40laPIOFQ5JVwEnA6qp6JrAEePVQ40laXEPv\nViwFliVZCiwHbht4PEmLZLBwqKpbgVOBm4DbgV9W1SWP/FySE5OsTbJ248a7hipH0hYacrdiT+Bl\nwH7Ak4Fdk5zwyM9V1ZqqWl1Vq1euXDFUOZK20JC7FUcBP6qqjVV1H3Ah8PwBx5O0iIYMh5uAQ5Is\nTxLgSGD9gONJWkRDHnO4DLgAuBJY1421ZqjxJC2uQfs5VNWHgA8NOYakYXiFpKQmw0FSk+Egqclw\nkNRkOEhqmrLu03sBfzp2Ef0N0dXajtbAbHW1HqKjNYzf1dotB0lNhoOkJsNBUpPhIKnJcJDUZDhI\najIcJDUZDpKaDAdJTYaDpCbDQVKT4SCpyXCQ1GQ4SGoyHCQ1GQ6SmgwHSU2Gg6Qmw0FSk+Egqclw\nkNSUqhq7hock2Qj8uOfH9wJ+NmA5i2mWaoXZqneWaoXx6/3tqlrZ54NTFQ5bIsnaqlo9dh19zFKt\nMFv1zlKtMFv1ulshqclwkNQ0y+GwZuwCtsAs1QqzVe8s1QozVO/MHnOQNKxZ3nKQNKCZC4ckxybZ\nkOT6JO8du575JNknyTeTXJPk6iQnj13TQpIsSXJVki+NXctCkuyR5IIk1yZZn+TQsWvanCRv7/4P\n/DDJ+Ul2GbumhcxUOCRZAnwS+APgGcBrkjxj3KrmdT/wzqp6BnAI8JYprxfgZGD92EX0dDrw1ao6\nAHgWU1p3klXAScDqqnomsAR49bhVLWymwgF4LnB9Vd1QVfcCnwNeNnJNm1VVt1fVld37u5j85101\nblWbl2Rv4DjgzLFrWUiSxwEvAs4CqKp7q+qOcaua11JgWZKlwHLgtpHrWdCshcMq4OY507cwxb9s\ncyV5CnAQcNm4lczr48C7gQfHLqSH/YCNwNndbtCZSXYdu6iWqroVOBW4Cbgd+GVVXTJuVQubtXCY\nSUl2A74AnFJVd45dT0uSlwI/raorxq6lp6XAwcAZVXUQcA8wlcegkuzJZAt3P+DJwK5JThi3qoXN\nWjjcCuwzZ3rvbt7USrITk2A4r6ouHLueeRwGHJ/kRia7a0ckOXfckuZ1C3BLVW3aEruASVhMo6OA\nH1XVxqq6D7gQeP7INS1o1sLhcuBpSfZLsjOTgzoXjVzTZiUJk33i9VV12tj1zKeq3ldVe1fVU5j8\nu36jqqb2262qfgLcnOTp3awjgWtGLGk+NwGHJFne/Z84kik9eDrX0rEL2BJVdX+StwIXMzni+5mq\nunrksuZzGPA6YF2SH3Tz3l9VXxmxpu3J24Dzui+KG4A3jlxPU1VdluQC4EomZ7CuYgaulPQKSUlN\ns7ZbIWkbMRwkNRkOkpoMB0lNhoOkJsNBUpPhoFEluTHJXmPXof/PcNBW6e4u1HbMcNiBJNk1yZeT\n/EfXdORVSZ6T5N+6ed9PsiLJLknOTrKuu+Px8G75NyS5KMk3gK93896V5PIk/5nkrzY3To/aliX5\nlyRvGvQfQb2Z/juWY4Hbquo4eKgnwlXAq6rq8iS7A79m0vClqurAJAcAlyTZv1vHwcDvVtXPkxwD\nPI1Jn40AFyV5EbCyMc58dmNys9dnq+qzi/kX1tZzy2HHsg44OslHkrwQ2Be4vaouB6iqO6vqfuAF\nwLndvGuZPIVsUzhcWlU/794f072uYnLfwAFMwuJh41TVLxeo65+Bsw2G6WI47ECq6jom3/zrgL8G\nXr4Vq7lnzvsAf1tVv9e9nlpVZz1ynCQfXGCd/woc292xqClhOOxAkjwZ+FVVnQt8FHge8KQkz+l+\nvqI70Phd4LXdvP2ZbGFsaKzyYuBPumY2JFmV5ImNcRbqs/BB4BdM+oNqSnjMYcdyIPDRJA8C9wFv\nZvLt/4kky5gcbzgK+BRwRpJ1TG4xfkNV/c8jv9ir6pIkvwP8e/ezu4ETgKc2xlnIycBnkvxdVb37\n0f9V9Wh5y7akJncrJDW5W6FtIskXmTRYnes9VXXxGPVoYe5WSGpyt0JSk+EgqclwkNRkOEhqMhwk\nNf0vYqsgqwDMhloAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1081be358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p = np.random.random((10, 10))\n",
    "delta=1\n",
    "for i in range(0,p.shape[0]):\n",
    "    fk=i+1\n",
    "    for j in range(0,p.shape[1]):\n",
    "        fcorrect=j+1\n",
    "        if i==j:\n",
    "            p[i,j]=0\n",
    "        else:\n",
    "            p[i,j] = max(0,fk-fcorrect+delta)\n",
    "\n",
    "plt.imshow(p, cmap='hot', interpolation='nearest')\n",
    "plt.xlabel('scores_k')\n",
    "plt.ylabel('scores_correct')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hinge loss: 2.0147138212706346\n"
     ]
    }
   ],
   "source": [
    "def hinge(scores,y):\n",
    "    #scores: N x K\n",
    "    N = probs.shape[0]\n",
    "    \n",
    "    correct_scores = scores[range(N),y]\n",
    "   \n",
    "    # need to broadcast by row not columns\n",
    "    margin = scores.T - correct_scores + delta \n",
    "    margin = margin.T\n",
    "    \n",
    "    #zero loss at the correct label\n",
    "    margin[range(N),y] = 0 \n",
    "    \n",
    "    #element-wise max\n",
    "    margin = np.maximum(np.zeros(margin.shape),margin) \n",
    "    data_loss = np.sum(margin) / N\n",
    "\n",
    "    \n",
    "    return(data_loss)\n",
    "\n",
    "#test\n",
    "scores = np.random.random((3000,3))\n",
    "print('Hinge loss: {}'.format(hinge(scores,y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Regularization loss is usually added to the loss function in order to prevent overfitting. It penalizes more complex weight terms.\n",
    "\n",
    "Data loss:\n",
    "\n",
    "$$L_{data} = \\frac{1}{N}\\sum_{n=1}^{N}L_i$$\n",
    "\n",
    "Regularization loss:\n",
    "\n",
    "$$l1 = \\frac{\\lambda}{2}|W|$$\n",
    "$$l2 = \\frac{\\lambda}{2}W^2$$\n",
    "\n",
    "where $W$ is the weights\n",
    "\n",
    "Overall loss:\n",
    "$$L_{overall} = L_{data} + L_{reg}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L2 Loss: 0.010537394039478272\n",
      "L1 Loss: 0.00942190784134485\n"
     ]
    }
   ],
   "source": [
    "def reg_loss_calc(W_list, l2=True, reg=1e-3):\n",
    "    loss = 0\n",
    "    if(l2):\n",
    "        for i in range(len(W_list)):\n",
    "            loss += 0.5 * reg * np.sum(W_list[i]* W_list[i])\n",
    "    else:\n",
    "        for i in range(len(W_list)):\n",
    "            loss += 0.5 * reg * np.sum(abs(W_list[i]))\n",
    "    return(loss)\n",
    "\n",
    "#test\n",
    "W_list = []\n",
    "for i in range(4):\n",
    "    W_list.append(np.random.randn(D,K))\n",
    "\n",
    "print('L2 Loss: {}'.format(reg_loss_calc(W_list,l2=True)))\n",
    "print('L1 Loss: {}'.format(reg_loss_calc(W_list,l2=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Layers are roughly classified into linear, activation and output layers. The output of their function is what passed to the next layer in the forward pass of training and the derivative is what passed to the previous layer in the backward pass."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Forward Pass\n",
    "\n",
    "$$scores = X \\cdot W + b$$\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "$$\\frac{\\partial previous}{\\partial W} = \\frac{\\partial previous}{\\partial scores} \\cdot \\frac{\\partial scores}{\\partial W} = \\frac{\\partial previous}{\\partial scores} \\cdot X$$\n",
    "\n",
    "$$\\frac{\\partial previous}{\\partial b} = \\frac{\\partial previous}{\\partial scores} \\cdot \\frac{\\partial scores}{\\partial b} = \\frac{\\partial previous}{\\partial scores} \\cdot 1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'out')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8lfX9/vHXm71n2BDClBVQDOAW\nN05EbKvWPbB2ftt+BSwOKlpFq63fuooVR4daGYKC4oLiQgGVLFYIK6wwwwxkvH9/JM0vpYwguXOf\ncT0fDx6c+5xPcq6bhFy5z33O+5i7IyIiAlAt7AAiIhI5VAoiIlJGpSAiImVUCiIiUkalICIiZVQK\nIiJSRqUgIiJlVAoiIlJGpSAiImVqhB3gWCUkJHhSUlLYMUREosrChQu3uHuLo62LulJISkpiwYIF\nYccQEYkqZra6Iuv08JGIiJRRKYiISBmVgoiIlAmsFMxsopnlmln6EdYMNrNvzSzDzP4VVBYREamY\nII8UXgaGHO5GM2sCPAtc4e69ge8FmEVERCogsFJw97nAtiMsuQ6Y4u5rStfnBpVFREQqJsxzCt2B\npmY2x8wWmtmNh1toZiPMbIGZLdi8eXMVRhQRiS9hlkIN4GTgUuAi4D4z636ohe4+wd1T3D2lRYuj\nvvZCRCSmFBQV8+ycLBat3RH4fYX54rUcYKu77wH2mNlcoB+wLMRMIiIRJX1dHqMmp5Kxfic/OruQ\nfh2aBHp/YZbCNOBpM6sB1AIGAX8IMY+ISMTILyji/z5azp/nZtO0Xi2e+2F/Lk5uE/j9BlYKZvYa\nMBhIMLMc4AGgJoC7P+/ui83sPSAVKAb+4u6HffqqiEi8mL9qG6Mmp5K9eQ9Xn9ye+y7tReN6Navk\nvgMrBXe/tgJrHgceDyqDiEg02b2/kMffW8Kr81bTtnFdXr11IGd1r9rzqFE3EE9EJBbNWZrLmKnp\nrM/bx02nJnH3RSdQv3bV/4hWKYiIhGjH3gM8+E4mU75eR5cW9XnzzlNJSWoWWh6VgohISGambeD+\naels31vAT87pws/O7UadmtVDzaRSEBGpYrk787l/WgbvZWykT7tGvHLrQHq3bRx2LEClICJSZdyd\nNxfm8NA7meQXFjNqSA/uOLMTNapHzsBqlYKISBVYu20v90xJ49OsLQxMasajw5Pp3KJB2LH+i0pB\nRCRARcXOq1+s4rH3llLNYNyVffjhwESqVbOwox2SSkFEJCBZubsYOSmVr9fsYPAJLXh4WDLtmtQN\nO9YRqRRERCpZQVExf/7XCv7voyzq1a7OH37QjytPbIdZZB4dlKdSEBGpROnr8rh7UiqLN+zk0uQ2\njL2iNy0a1g47VoWpFEREKkF+QRF//HA5L3ySTfP6tfjzDSdzUe/WYcc6ZioFEZHj9NXKbYyenEr2\nlj38IKUDv7m0J43rVs0Au8qmUhAR+Y525Rcw/r0l/G3eGjo0q8vfbhvEGd0Swo51XFQKIiLfwewl\nuYyZmsaGnfncdkYnfn1hd+rViv4fqdG/ByIiVWjbngOMeyeTqd+so1vLBky+6zT6JzYNO1alUSmI\niFSAu/NO6gbGTs8gb18BPz+vGz85pwu1a4Q7wK6yqRRERI5i0858xkxN58PFm+jbvjF/v2MQPVo3\nCjtWIIJ8O86JwGVArrv3OcK6AcAXwDXuPimoPCIix8rdeWP+Wh6euZgDhcWMuaQnt5yeFFED7Cpb\nkEcKLwNPA68eboGZVQfGA+8HmENE5Jit2bqX0VNS+XzFVgZ1asb44X1JSqgfdqzABfkezXPNLOko\ny34GTAYGBJVDRORYFBU7L322kt+/v5Qa1arxu2HJXDOgQ8QOsKtsoZ1TMLN2wDDgHFQKIhIBlm0q\nGWD37dodnNujJQ8P60ObxpE9wK6yhXmi+Y/AKHcvPtqQKDMbAYwASExMrIJoIhJPDhQW89ycFTw9\nezkNatfgqWtO5Ip+baNigF1lC7MUUoDXS//RE4BLzKzQ3d86eKG7TwAmAKSkpHiVphSRmLZo7Q5G\nTkpl6aZdDD2xLfdf1ovmDaJngF1lC60U3L3Tvy+b2cvAO4cqBBGRIOw7UMSTHyzlxU9X0rJhHf5y\nYwrn92oVdqzQBfmU1NeAwUCCmeUADwA1Adz9+aDuV0TkaD5fsYV7pqSxeuterhuUyOiLe9CoTnQO\nsKtsQT776NpjWHtzUDlERP5tZ34Bj8xcwmtfraFj83r8445BnNYlugfYVTa9ollE4sJHizcxZmo6\nubvyuePMTvzqghOoWyu2RlRUBpWCiMS0rbv389u3M5m+aD09WjfkzzecTL8OTcKOFbFUCiISk9yd\n6YvWM3Z6Brv3F/LL87tz1+Au1KoRuyMqKoNKQURizoa8fdw7NZ2PluRyYocmPHZ1X7q3ahh2rKig\nUhCRmFFc7Lw+fy2PzFxMQXEx917ak1tO70T1OBlRURlUCiISE1Zt2cPoKanMy97GaV2a88hVyXRs\nHvsD7CqbSkFEolphUTETP1vJE+8vo1aNaowfnsz3UzrE5YiKyqBSEJGotXjDTkZNTiU1J4/ze7bi\noSv70LpxnbBjRTWVgohEnf2FRTwzewXPzs6icd2aPH3dSVya3EZHB5VApSAiUeXrNdsZNSmV5bm7\nGXZSO+67rBfN6tcKO1bMUCmISFTYe6CQJ95fxsTPVtK6UR1eunkA5/RoGXasmKNSEJGI93nWFkZP\nSWPNtr1cf0oio4b0oKEG2AVCpSAiEStvXwGPzFzM6/PX0imhPm+MOIVBnZuHHSumqRREJCK9n7GR\ne99KZ8vu/dx5dmd+eX536tTUALugqRREJKJs3rWfsW9nMCN1Az1aN+QvN6XQt70G2FUVlYKIRAR3\n561v1/HbtzPZu7+IX13QnR+drQF2VU2lICKhW79jH2OmpjF76WZOSmzCY8P70k0D7EKhUhCR0BQX\nO3//ag3j311CUbHzwOW9uPHUJA2wC1GQ79E8EbgMyHX3Poe4/YfAKMCAXcBd7r4oqDwiElmyN+9m\n9OQ0vlq1jTO6JvDIVcl0aFYv7FhxL8gjhZeBp4FXD3P7SuBsd99uZhcDE4BBAeYRkQhQWFTMXz5d\nyR8+WEbtGtV47Oq+fO/k9hpRESECKwV3n2tmSUe4/fNym/OA9kFlEZHIkLm+ZIBd2ro8LurdinFD\n+9CykQbYRZJIOadwG/Bu2CFEJBj7C4t4+uMsnpuzgib1avLMdf25JLm1jg4iUOilYGbnUFIKZxxh\nzQhgBEBiYmIVJRORyrBw9TZGTU4jK3c3w/u3595Le9JUA+wiVqilYGZ9gb8AF7v71sOtc/cJlJxz\nICUlxasonogchz37C3l81lJe+WIVbRvX5eVbBjD4BA2wi3ShlYKZJQJTgBvcfVlYOUSk8n2yfDP3\nTElj3Y593HhKR+4e0oMGtUN/YEIqIMinpL4GDAYSzCwHeACoCeDuzwP3A82BZ0sfVyx095Sg8ohI\n8PL2FvDQjEzeXJhD5xb1+eedpzIgqVnYseQYBPnso2uPcvvtwO1B3b+IVK330jdy37R0tu05wI8H\nd+Hn53XTALsopOM5ETkuubvyGTs9g5lpG+nVphEv3TyAPu0ahx1LviOVgoh8J+7O5K/XMe6dTPYV\nFHH3RScw4qzO1KyuAXbRTKUgIsds7ba9/GZqGp8s30JKx6Y8OrwvXVs2CDuWVAKVgohUWHGx89d5\nqxn/3hIMeHBob64f1JFqGmAXM1QKIlIhWbm7GT05lQWrt3NW9xb8blgf2jfVALtYo1IQkSMqKCpm\nwtxsnvpoOXVrVueJ7/Xjqv7tNKIiRqkUROSw0tflMXJSKpkbdnJxn9b8dmhvWjbUALtYplIQkf+S\nX1DEUx8tZ8LcbJrVr8Xz1/dnSJ82YceSKqBSEJH/MH/VNkZNTiV78x6+d3J77r20F43r1Qw7llQR\nlYKIALB7fyGPv7eEV+etpm3juvz1toGc2a1F2LGkiqkURIQ5S3MZMzWd9Xn7uOnUJO6+6ATqa4Bd\nXNJXXSSObd9zgHEzMpny9Tq6tKjPpB+dyskdNcAunqkUROKQu/Nu+kbun5bOjr0F/OzcrvzknK4a\nYCcqBZF4k7szn/umpTMrYxPJ7Rrz6q2D6NW2UdixJEKoFETihLvz5sIcHnonk/2FxYy+uAe3n9GJ\nGhpgJ+WoFETiQPkBdgOTmvHo8GQ6t9AAO/lvKgWRGFZU7Lz6xSoee28p1QzGDe3NDzXATo5ApSAS\no5Zv2sWoyal8vWYHg09owcPDkmnXpG7YsSTCBfkezROBy4Bcd+9ziNsNeAq4BNgL3OzuXweVRyRe\nFBQV8/ycFfzp4yzq1a7OH37QjytP1AA7qZggjxReBp4GXj3M7RcD3Ur/DAKeK/1bRL6jtJw87p60\niCUbd3F5v7Y8cHkvEhrUDjuWRJHASsHd55pZ0hGWDAVedXcH5plZEzNr4+4bgsokEqvyC4r4w4fL\neGFuNi0a1mbCDSdzYe/WYceSKBTmOYV2wNpy2zml16kURI7Bl9lbGT0ljZVb9nDNgA7cc0lPGtfV\nADv5bqLiRLOZjQBGACQmJoacRiQy7MovYPx7S/jbvDV0aFaXv98+iNO7JoQdS6JcmKWwDuhQbrt9\n6XX/xd0nABMAUlJSPPhoIpFt9pJcxkxNY+POfG4/oxO/urA79WpFxe94EuHC/C6aDvzUzF6n5ARz\nns4niBzZtj0HGPdOJlO/WUe3lg2YfNdpnJTYNOxYEkOCfErqa8BgIMHMcoAHgJoA7v48MJOSp6Nm\nUfKU1FuCyiIS7dydGWkbeGBaBnn7Cvj5ed34yTldqF1DA+ykcgX57KNrj3K7Az8J6v5FYsWmnfnc\n+1Y6H2Ruom/7xvzt9kH0bKMBdhIMPQgpEqHcnTfmr+XhmYs5UFjMby7pwa2na4CdBEulIBKBVm/d\nwz1T0vh8xVYGdWrG+OF9SUqoH3YsiQMqBZEIUlTsvPTZSn7//lJqVqvG74Ylc82ADhpgJ1VGpSAS\nIZZu3MXIyaksWruD83q05KFhfWjTWAPspGqpFERCdqCwmOfmrODp2ctpWKcmT11zIlf0a6sBdhIK\nlYJIiL5du4NRk1JZumkXV5QOsGuuAXYSIpWCSAj2HSjiyQ+W8uKnK2nZsA4v3pTCeT1bhR1LRKUg\nUtU+X7GF0ZPTWLNtL9cNSmT0xT1oVEcD7CQyqBREqsjO/AIembmE175aQ1Lzerx2xymc2qV52LFE\n/kOFSsHMarv7/qNdJyKH9mHmJsa8lcbmXfsZcVZnfnl+d+rW0ogKiTwVPVL4AuhfgetEpJytu/fz\n27czmb5oPT1aN2TCDSn069Ak7Fgih3XEUjCz1pS88U1dMzsJ+Pdz5BoB9QLOJhK13J3pi9YzdnoG\nu/cX8svzu3PX4C7UqqERFRLZjnakcBFwMyXvdfBkuet3Ab8JKJNIVNuQt497p6bz0ZJcTuzQhMeu\n7kv3Vg3DjiVSIUcsBXd/BXjFzIa7++QqyiQSlYqLndfmr+GRmUsoKnbuvbQnt5zeieoaUSFRpKLn\nFPqYWe+Dr3T3Bys5j0hUWrllD6Mnp/Llym2c3rU5jwzrS2JzPcIq0aeipbC73OU6wGXA4sqPIxJd\nCouKmfjZSp54fxm1alTj0auS+cGADhpRIVGrQqXg7k+U3zaz3wOzAkkkEiWWbNzJqEmpLMrJ44Je\nrXjoyj60alQn7Fgix+W7vnitHiUnn0Xizv7CIp6ZvYJnZ2fRuG5Nnr7uJC5NbqOjA4kJFX3xWhrg\npZvVgJbAuAp83BDgKaA68Bd3f/Sg2xOBV4AmpWtGu/vMCqcXqWLfrNnOyEmpLM/dzVUnteO+y3rR\ntH6tsGOJVJqKHilcBjQFzqTkB/hMd194pA8ws+rAM8AFQA4w38ymu3tmuWX3Av909+fMrBcwE0g6\ntl0QCd7eA4U88f4yJn62kjaN6vDSzQM4p0fLsGOJVLqKlsJQ4A5gCiUvYHvJzF5w9z8d4WMGAlnu\nng1gZq+Xfp7ypeCUvBAOoDGw/hiyi1SJz7O2MHpKyQC7G07pyMghJ9BQA+wkRlW0FG4HTnH3PQBm\nNp6SMRdHKoV2wNpy2znAoIPWjAXeN7OfAfWB8yuYRyRwefsKeGTmYl6fv5ZOCfV5Y8QpDOqsAXYS\n2ypaCgYUldsu4v+PvDge1wIvu/sTZnYq8Fcz6+Puxf9x52YjgBEAiYmJlXC3Ikf2fsZG7n0rna17\nDvCjs7vwP+d3o05NDbCT2FfRUngJ+NLMppZuXwm8eJSPWQd0KLfdvvS68m4DhgC4+xdmVgdIAHLL\nL3L3CcAEgJSUFEckIJt37Wfs2xnMSN1Aj9YNefGmASS3bxx2LJEqU9HXKTxpZnOAM0qvusXdvznK\nh80HuplZJ0rK4BrguoPWrAHOA142s56UvDBucwWzi1Qad+etb9fx27cz2bu/iP+9sDt3nt2FmtU1\nwE7iS4Vfp+DuXwNfH8P6QjP7KSUvcqsOTHT3DDN7EFjg7tOBXwMvmNkvKTnpfLO760hAqtS6HfsY\nMzWNOUs30z+xZIBd15YaYCfxKdB3Xit9zcHMg667v9zlTOD0IDOIHE5xsfP3r9bw6MzFODD28l7c\ncGqSBthJXNPbcUpcyt68m9GT0/hq1TbO6JrAI1cl06GZBtiJqBQkrhQWFfPCJyv5w4fLqFOjGo9d\n3ZfvndxeIypESqkUJG5krt/JyMmLSF+3k4t6t2Lc0D601AA7kf+gUpCYl19QxNMfZ/H8v1bQpF4t\nnvthfy5ObhN2LJGIpFKQmLZw9TZGTkplxeY9XNW/Hfdf1osm9TTATuRwVAoSk/bsL+TxWUt55YtV\ntG1cl1duHcjZ3VuEHUsk4qkUJObMXbaZe6aksT5vHzee0pG7h/SgQW19q4tUhP6nSMzYsfcAD81Y\nzKSFOXRuUZ9/3nkqA5KahR1LJKqoFCQmvJu2gfumZbB97wF+ck4XfnauBtiJfBcqBYlqubvyeWBa\nBu+mb6R320a8cusAerfVADuR70qlIFHJ3Zm0MIeHZixmX0ERI4ecwB1ndtYAO5HjpFKQqJOzfS/3\nTEnjk+VbGJDUlEeH96VLiwZhxxKJCSoFiRrFxc5f561m/HtLMODBob25flBHqmmAnUilUSlIVMjK\n3c3oyaksWL2ds7u34OFhfWjfVAPsRCqbSkEiWkFRMRPmZvPUh8upV7s6T36/H8NOaqcBdiIBUSlI\nxEpfl8fISalkbtjJpcltGHtFb1o0rB12LJGYplKQiJNfUMRTHy1nwtxsmtWvxfPXn8yQPq3DjiUS\nF1QKElHmr9rGqEmpZG/Zw/dT2jPmkl40rlcz7FgicSPQJ3Wb2RAzW2pmWWY2+jBrvm9mmWaWYWb/\nCDKPRK7d+wu5f1o633v+Cw4UFfO32wbx2NX9VAgiVSywIwUzqw48A1wA5ADzzWx66fsy/3tNN+Ae\n4HR3325mLYPKI5FrztJcxkxNZ33ePm45PYn/vfAE6muAnUgogvyfNxDIcvdsADN7HRgKZJZbcwfw\njLtvB3D33ADzSITZvucA42ZkMuXrdXRt2YBJPzqNkzs2DTuWSFwLshTaAWvLbecAgw5a0x3AzD4D\nqgNj3f29ADNJBHB33k3fyP3T0tmxt4CfnduVn57bldo1NMBOJGxhH6PXALoBg4H2wFwzS3b3HeUX\nmdkIYARAYmJiVWeUSpS7M5/7pqUzK2MTye0a8+qtg+jVtlHYsUSkVJClsA7oUG67fel15eUAX7p7\nAbDSzJZRUhLzyy9y9wnABICUlBQPLLEExt15c0EO42ZkcqCwmHsu7sFtZ3SihgbYiUSUIEthPtDN\nzDpRUgbXANcdtOYt4FrgJTNLoOThpOwAM0kI1m4rGWD3adYWBnZqxvjhfemUUD/sWCJyCIGVgrsX\nmtlPgVmUnC+Y6O4ZZvYgsMDdp5fedqGZZQJFwN3uvjWoTFK1ioqdVz5fxeOzllK9mvHQlX24bmCi\nBtiJRDBzj65HY1JSUnzBggVhx5CjWL5pF6Mmp/L1mh2cc0ILHh6WTNsmdcOOJRK3zGyhu6ccbV3Y\nJ5olxhQUFfP8nBX86eMs6teuzlPXnMgV/dpqgJ1IlFApSKVJzdnByEmpLNm4i8v7teWBy3uR0EAD\n7ESiiUpBjlt+QRF/+GAZL3ySTYuGtZlww8lc2FsD7ESikUpBjsu87K2MnpzKqq17uXZgB0Zf3JPG\ndTWvSCRaqRTkO9mVX8Cj7y7h71+uIbFZPf5x+yBO65oQdiwROU4qBTlmHy/ZxJip6Wzamc/tZ3Ti\nVxd2p14tfSuJxAL9T5YK27bnAA++ncFb366nW8sGPHvXaZyUqAF2IrFEpSBH5e68nbqBsdMz2Lmv\ngF+c140fn9NFA+xEYpBKQY5oY14+976VzoeLN9GvfWPG3zGIHq01wE4kVqkU5JDcndfnr+V3MxZT\nUFzMvZf25JbTO1FdIypEYppKQf7L6q17GD05jS+yt3Jq5+Y8OjyZjs01wE4kHqgUpExRsTPx05U8\n8cFSalarxiNXJXPNgA4aUSESR1QKAsDSjbsYOWkRi3LyOL9nSx66MpnWjeuEHUtEqphKIc4dKCzm\nmdlZPDsni0Z1avKna0/isr5tdHQgEqdUCnHs27U7GDlpEcs27ebKE9ty/+W9aVa/VtixRCREKoU4\ntO9AEU9+sJQXP11Jq0Z1mHhzCuf2aBV2LBGJACqFOPP5ii2MnpzGmm17+eGgREZf3IOGdTTATkRK\nqBTixM78Ah6ZuYTXvlpDUvN6vD7iFE7p3DzsWCISYaoF+cnNbIiZLTWzLDMbfYR1w83MzeyobxUn\nx+7DzE1c8OS/eGP+Gu48qzPv/uIsFYKIHFJgRwpmVh14BrgAyAHmm9l0d888aF1D4BfAl0FliVdb\nd+9n7NuZvL1oPT1aN+SFG1Po275J2LFEJIIF+fDRQCDL3bMBzOx1YCiQedC6ccB44O4As8QVd2f6\novWMnZ7B7v2F/PL87tw1uAu1agR6YCgiMSDIUmgHrC23nQMMKr/AzPoDHdx9hpmpFCrBhrx93Ds1\nnY+W5HJihyY8dnVfurdqGHYsEYkSoZ1oNrNqwJPAzRVYOwIYAZCYmBhssChVXOy8Nn8Nj8xcQlGx\nc99lvbj5tCQNsBORYxJkKawDOpTbbl963b81BPoAc0pfPdsamG5mV7j7gvKfyN0nABMAUlJSPMDM\nUWnllj2MnpzKlyu3cVqX5jx6VV8Sm9cLO5aIRKEgS2E+0M3MOlFSBtcA1/37RnfPA8re1NfM5gD/\ne3AhyOEVFhXz4qcrefKDZdSqUY1Hr0rmBxpgJyLHIbBScPdCM/spMAuoDkx09wwzexBY4O7Tg7rv\neLB4w05GTU4lNSePC3q14qEr+9CqkQbYicjxCfScgrvPBGYedN39h1k7OMgssWJ/YRHPfJzFs3NW\n0KReTZ65rj+XJLfW0YGIVAq9ojmKfL1mO6MmpbI8dzdXndSO+y7rRVMNsBORSqRSiAJ7DxTy+1nL\neOnzlbRpVIeXbh7AOT1ahh1LRGKQSiHCfbp8C/dMTWXttn3ccEpHRg45QQPsRCQwKoUIlbevgIdn\nZPLPBTl0SqjPGyNOYZDmFYlIwFQKEWhWxkbueyudrXsOcNfgLvzivG7UqVk97FgiEgdUChFk8679\njH07gxmpG+jZphEv3jSA5PaNw44lInFEpRAB3J2p36zjwXcy2bu/iP+9sDt3nt2FmtU1wE5EqpZK\nIWTrduzjN1PS+NeyzfRPLBlg17WlBtiJSDhUCiEpLnb+/uVqHn13CQ48cHkvbjxVA+xEJFwqhRCs\n2Lyb0ZNTmb9qO2d2S+B3w5Lp0EwD7EQkfCqFKlRYVMyET7L544fLqVOjGo9f3ZerT26vERUiEjFU\nClUkY30eoyankr5uJxf1bsW4oX1oqQF2IhJhVAoByy8o4k8fL+fP/8qmSb1aPPfD/lyc3CbsWCIi\nh6RSCNCCVdsYOTmV7M17GN6/Pfdd1pMm9TTATkQil0ohAHv2F/L4rKW88sUq2jauy6u3DuSs7i3C\njiUiclQqhUo2d9lm7pmSxvq8fdx0ahJ3X3QC9Wvrn1lEooN+WlWSHXsP8NCMxUxamEPnFvV5885T\nSUlqFnYsEZFjolKoBO+mbeC+aRls33uAHw/uws81wE5EolSgpWBmQ4CnKHmP5r+4+6MH3f4r4Hag\nENgM3Oruq4PMVJlyd+XzwLQM3k3fSO+2jXjl1gH0bqsBdiISvQIrBTOrDjwDXADkAPPNbLq7Z5Zb\n9g2Q4u57zewu4DHgB0FlqizuzqSFOTw0YzH7CooYOeQERpzZmRoaYCciUS7II4WBQJa7ZwOY2evA\nUKCsFNx9drn184DrA8xTKdZu28tvpqbxyfItDEhqyqPD+9KlRYOwY4mIVIogS6EdsLbcdg4w6Ajr\nbwPePdQNZjYCGAGQmJhYWfmOSXGx8+oXq3hs1lIMeHBob64f1JFqGmAnIjEkIk40m9n1QApw9qFu\nd/cJwASAlJQUr8JoAGTllgywW7B6O2d3b8HDw/rQvqkG2IlI7AmyFNYBHcptty+97j+Y2fnAGOBs\nd98fYJ5jVlBUzIS52Tz14XLq1a7OE9/rx1X922mAnYjErCBLYT7Qzcw6UVIG1wDXlV9gZicBfwaG\nuHtugFmOWfq6PO6elMriDTu5tG8bxl7emxYNa4cdS0QkUIGVgrsXmtlPgVmUPCV1ortnmNmDwAJ3\nnw48DjQA3iz97XuNu18RVKaKyC8o4o8fLueFT7JpVr8Wz19/MkP6tA4zkohIlQn0nIK7zwRmHnTd\n/eUunx/k/R+rr1ZuY/TkVLK37OH7Ke0Zc0kvGterGXYsEZEqExEnmsO2e38h499dwl/nraZ907r8\n9baBnNlNA+xEJP7EfSnMXprLvVPTWZ+3j1tP78SvL+yuAXYiErfi9qff9j0HGPdOJlO+WUfXlg2Y\n9KPTOLlj07BjiYiEKu5Kwd2ZkbaBB6ZlkLevgJ+f25WfnNuV2jU0wE5EJK5KYdPOfO57K533MzeR\n3K4xf7t9ED3bNAo7lohIxIibUpi9JJefv/4NBwqLuefiHtx2RicNsBMROUjclEKnhPr0T2zK2Ct6\n0ymhfthxREQiUtyUQlJCfV4xUtMSAAAEvElEQVS5dWDYMUREIpoePxERkTIqBRERKaNSEBGRMioF\nEREpo1IQEZEyKgURESmjUhARkTIqBRERKWPuHnaGY2Jmm4HVx/AhCcCWgOJEsnjdb4jffdd+x5dj\n3e+O7n7UN4qJulI4Vma2wN1Tws5R1eJ1vyF+9137HV+C2m89fCQiImVUCiIiUiYeSmFC2AFCEq/7\nDfG779rv+BLIfsf8OQUREam4eDhSEBGRCoqZUjCzIWa21MyyzGz0IW6vbWZvlN7+pZklVX3KyleB\n/f6VmWWaWaqZfWRmHcPIWdmOtt/l1g03MzezmHh2SkX228y+X/o1zzCzf1R1xqBU4Hs90cxmm9k3\npd/vl4SRszKZ2UQzyzWz9MPcbmb2f6X/Jqlm1v+479Tdo/4PUB1YAXQGagGLgF4Hrfkx8Hzp5WuA\nN8LOXUX7fQ5Qr/TyXfGy36XrGgJzgXlASti5q+jr3Q34Bmhaut0y7NxVuO8TgLtKL/cCVoWduxL2\n+yygP5B+mNsvAd4FDDgF+PJ47zNWjhQGAlnunu3uB4DXgaEHrRkKvFJ6eRJwnplZFWYMwlH3291n\nu/ve0s15QPsqzhiEiny9AcYB44H8qgwXoIrs9x3AM+6+HcDdc6s4Y1Aqsu8ONCq93BhYX4X5AuHu\nc4FtR1gyFHjVS8wDmphZm+O5z1gphXbA2nLbOaXXHXKNuxcCeUDzKkkXnIrsd3m3UfJbRbQ76n6X\nHkZ3cPcZVRksYBX5encHupvZZ2Y2z8yGVFm6YFVk38cC15tZDjAT+FnVRAvVsf4MOKq4eY/meGdm\n1wMpwNlhZwmamVUDngRuDjlKGGpQ8hDSYEqOCueaWbK77wg1VdW4FnjZ3Z8ws1OBv5pZH3cvDjtY\nNImVI4V1QIdy2+1LrzvkGjOrQcnh5dYqSReciuw3ZnY+MAa4wt33V1G2IB1tvxsCfYA5ZraKksda\np8fAyeaKfL1zgOnuXuDuK4FllJREtKvIvt8G/BPA3b8A6lAyHyiWVehnwLGIlVKYD3Qzs05mVouS\nE8nTD1ozHbip9PLVwMdeeqYmih11v83sJODPlBRCrDy+fMT9dvc8d09w9yR3T6LkXMoV7r4gnLiV\npiLf529RcpSAmSVQ8nBSdlWGDEhF9n0NcB6AmfWkpBQ2V2nKqjcduLH0WUinAHnuvuF4PmFMPHzk\n7oVm9lNgFiXPUpjo7hlm9iCwwN2nAy9ScjiZRcmJm2vCS1w5KrjfjwMNgDdLz6uvcfcrQgtdCSq4\n3zGngvs9C7jQzDKBIuBud4/2I+KK7vuvgRfM7JeUnHS+Odp/8TOz1ygp+YTScyUPADUB3P15Ss6d\nXAJkAXuBW477PqP830xERCpRrDx8JCIilUClICIiZVQKIiJSRqUgIiJlVAoiIlJGpSByHMzs87Az\niFQmPSVVRETK6EhB5DiY2e7Svweb2Rwzm2RmS8zs7zEwhVfikEpBpPKcBPwPJbP8OwOnhxtH5Nip\nFEQqz1funlM6lfNbICnkPCLHTKUgUnnKT6AtIkZmi0l8USmIiEgZlYKIiJTRU1JFRKSMjhRERKSM\nSkFERMqoFEREpIxKQUREyqgURESkjEpBRETKqBRERKSMSkFERMr8P9aL12rhCbEjAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1081be278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(0.01, 1, 0.01)\n",
    "w = 1.5\n",
    "b = 0.1\n",
    "plt.plot(x,w*x + b)\n",
    "plt.xlabel('in')\n",
    "plt.ylabel('out')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forward pass: (3000, 100)\n",
      "Backward pass: (2, 100), (1, 100)\n"
     ]
    }
   ],
   "source": [
    "class Linear:\n",
    "    def forward(self,X,W,b):\n",
    "        #X: N x D\n",
    "        #W: D x K\n",
    "        scores = X.dot(W) + b #N x K\n",
    "        return(scores)\n",
    "    def backward(self,X,W,previous,reg=1e-3):\n",
    "        #previous: N x K\n",
    "        dW = X.T.dot(previous)\n",
    "        dW += reg * W\n",
    "        db = np.sum(previous,axis=0,keepdims=True)\n",
    "        return dW,db\n",
    "\n",
    "#test\n",
    "lin = Linear()\n",
    "scores = lin.forward(X,W,b)\n",
    "previous = np.ones_like(scores)\n",
    "dW, db = lin.backward(X,W,previous)\n",
    "print('Forward pass: {}'.format(scores.shape))\n",
    "print('Backward pass: {}, {}'.format(dW.shape,db.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Sigmoid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Forward Pass\n",
    "\n",
    "$$out = \\frac{1}{1 + e^{-scores}}$$\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "$$\\frac{\\partial previous}{\\partial scores} = \\frac{\\partial previous}{\\partial out} \\cdot \\frac{\\partial out}{\\partial scores}\n",
    "\\\\\n",
    "\\frac{\\partial out}{\\partial scores} = (-1) \\cdot (\\frac{1}{1+e^{-x}})^2 \\cdot (-1) \\cdot (\\frac{1}{e^x})^2 \\cdot e^x \n",
    "\\\\\n",
    "= \\frac{e^{-x}}{(1+e^{-x})^2} = \\frac{1}{1+e^{-x}} \\cdot \\frac{e^{-x}}{1+e^{-x}} = out \\cdot \\frac{e^{-x}}{1+e^{-x}} \\\\\n",
    "= out \\cdot \\frac{(1+e^{-x}) - 1}{1+e^{-x}} = out \\cdot (1 - out)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'out')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl0HOWZ7/Hvo92LvMqrLG9gjBew\nsYUDhC1sNgTsTBKIuROywITJQk5yMpkZcjOHcEjOvTeTSe4kd5gQZhISloEQGIgnMRFmG5KAwQsG\nLC9YeJWwZHmVbFlSL8/9o8uiEZItbFWXuvX7nNPurqq3W4+rS/1TvVVdr7k7IiIiAHlRFyAiIn2H\nQkFERDooFEREpINCQUREOigURESkg0JBREQ6KBRERKSDQkFERDooFEREpENB1AV8UGVlZT558uSo\nyxARySpr1qzZ6+6jTtQu60Jh8uTJrF69OuoyRESyipnt6Ek7dR+JiEgHhYKIiHRQKIiISAeFgoiI\ndFAoiIhIh9BCwcx+YWZ7zGx9N8vNzH5iZjVm9oaZzQurFhER6Zkw9xR+CSw6zvKrgWnB7VbgpyHW\nIiIiPRDa9xTc/UUzm3ycJkuA+z01HuhKMxtmZuPcfXdYNYlI7nB32uJJ2mJJWuMJ2uNJ4kknkUwS\nSziJpBNPOvHEsflOLJEM7o8tT5J0xx2SnnpNd3DS5uEkHXB/tw3vb5+ahmQwxPGxZQD+nrrTHqct\nee/8rp9w+YwxzKkY1lursEtRfnmtHNiVNl0bzHtfKJjZraT2Jpg4cWJGihOR8Lg7Ta1xGptb2dPc\nxv4j7TQdjdPUGqPpaCy4j9PcGqOpNc6Rtjht8SStsURwSwVBfxli3ix1P3pISU6HQo+5+73AvQCV\nlZX9ZDMQyV7uTt3Bo+zc18LO/S3s2J+6f+fgURqb22hsbqMtnuzyuQV5RmlJAUMGFDKkpJDSkgKG\nDxzIgKJ8SgryKCnMp6Tw2H1+x3Rhfh6F+UZBXh4FeUZBfuo+P88oCObn5xmF+cG8YDo/z8gzMAwz\ngtu78/IMMMgzw3jvMssDI1gWtIHgNY49N2D27kTabKybNlGJMhTqgIq06QnBPBHJIrFEkup3mlhf\nd4hN9U1s2t3MpvpmDrfFO9oU5BkThg+gfPgAzp08glGlxYwuLWZUcCsbXMyQkkKGDChgQGF+n/hw\n7K+iDIVlwG1m9gjwIeCQjieI9H3t8SRrdx7g1W37eXXbftbuPEBLewKA0pICZowdwsfnlTN9bClT\nRg5i4siBjBs6gPw8fdBng9BCwcweBi4FysysFvgOUAjg7vcAy4FrgBqgBfh8WLWIyKlpbo3x7MY9\nrNjYwIubG2lui2MG08eUcv38CZw7ZQRzK4ZRPmyA/srPcmGefXTjCZY78JWwfr6InJpE0vlTzV4e\nX1NLVXU9bfEko0qL+ejZ47jszNF8aMpIhg4sjLpM6WVZcaBZRDLncFucR1ft4r6XtrFr/1GGDijk\nhsoKPnZOOedUDCNP3UA5TaEgIgA0tcb4txe38ss/b6e5LU7lpOHcvmgGV8wcTXFBftTlSYYoFET6\nudZYggdX7uDu52s40BLjmrPG8oWLpnLOxOFRlyYRUCiI9GMvv72P//nEm2zbe4SLppXxdwvP5KwJ\nQ6MuSyKkUBDphw4djfG/l2/kkVW7mDhiIPffvICLzzjh8L3SDygURPqZdbsO8pWH1lLf1MpfXzKV\nr19+BgOKdMxAUhQKIv2Eu3P/yzv43u83MLq0hMe/dAFzQ76OjmQfhYJIP9AeT/L3j7/BE6/VcfmZ\no/nhDXMYNrAo6rKkD1IoiOS4w21xvvTgGv64ZS/fuPIMbvvI6fqugXRLoSCSw/YebuPz961iw+4m\nfvDJs7m+suLET5J+TaEgkqP2HW7jUz97mbqDR7n3pvlcPmNM1CVJFlAoiOSg5tYYn73vVWoPHOVX\nNy/gvKkjoy5JskSYYzSLSARaYwlu+dVqNu1u5p5Pz1cgyAeiPQWRHJJMOl99+DVWbd/PP39qLh85\nc3TUJUmW0Z6CSA7552e3sGJDA3dcO5Mlc8ujLkeykEJBJEc8XV3PT57dwvXzJ/C5CyZHXY5kKYWC\nSA6o2XOYbzz6OmdPGMp3PzZbo5/JSVMoiGS51liCLz64huKCPO759HxKCnUdIzl5OtAskuX+8Q+b\nqdlzmAduWcD4YQOiLkeynPYURLLYy2/v4xd/3sZnzp/ERdN06Ws5dQoFkSzV3Brjm795nSllg7j9\n6jOjLkdyhLqPRLLU9363kd2HjvLYly5gYJF+laV3aE9BJAu9snUfv169i1svPo15GktZepFCQSTL\nxBNJvrOsmvJhA/ja5dOiLkdyjEJBJMs89MpONtU38w8fnaFhNKXXKRREssi+w2388OnNXHh6GYtm\nj426HMlBCgWRLPKDqs20tCe4c/FMfWtZQqFQEMkSm+qb+PXqXXzugsmcPro06nIkRykURLLED59+\ni8FFBdx22elRlyI5TKEgkgXW7TrIig0NfOHiqQwbWBR1OZLDQg0FM1tkZpvNrMbMbu9i+UQze97M\nXjOzN8zsmjDrEclWP3x6M8MHFnLzhVOiLkVyXGihYGb5wN3A1cBM4EYzm9mp2T8Aj7r7OcBS4F/D\nqkckW72ydR9/3LKXL116GoOL9c1lCVeYewoLgBp33+ru7cAjwJJObRwYEjweCrwTYj0iWcfd+aen\nNzO6tJjPnD856nKkHwgzFMqBXWnTtcG8dHcCnzazWmA58NUQ6xHJOiu37mfV9gPcdtnpGidBMiLq\nA803Ar909wnANcADZva+mszsVjNbbWarGxsbM16kSFR+9uLbjBxUxA2VFVGXIv1EmKFQB6RvyROC\neeluAR4FcPeXgRKgrPMLufu97l7p7pWjRuma8dI/bKpv4oXNjXzugsnaS5CMCTMUVgHTzGyKmRWR\nOpC8rFObncDlAGY2g1QoaFdABLj3xa0MKMznpvMnRV2K9COhhYK7x4HbgCpgI6mzjKrN7C4zWxw0\n+xvgC2b2OvAw8Dl397BqEskW7xw8yrJ177B0QYW+lyAZFer5be6+nNQB5PR5d6Q93gB8OMwaRLLR\nfX/ehgO36HsJkmFRH2gWkU6aWmP8xys7ufbscUwYPjDqcqSfUSiI9DGPr6nlSHuCv7pwatSlSD+k\nUBDpQ9ydB1fuYG7FMM6aMDTqcqQfUiiI9CEvb93H241HuOk8nXEk0VAoiPQhD67cwbCBhXz07HFR\nlyL9lEJBpI9oaGqlqrqBGyor9GU1iYxCQaSPePjVnSSSzl9+aGLUpUg/plAQ6QNiiSQPv7qTS84Y\nxaSRg6IuR/oxhYJIH/D8pj00NLXxaR1glogpFET6gMfW1FI2uJiPTNcFHyVaCgWRiO093MZzm/bw\n8XnlFOTrV1KipS1QJGK/XfcO8aTzyfkToi5FRKEgEiV35zerdzFnwlDOGFMadTkiCgWRKFW/08Sm\n+mbtJUifoVAQidBja2opys9j8ZzOw5eLREOhIBKR9niS366r48pZYxg6sDDqckQAhYJIZJ7fvIcD\nLTF1HUmfolAQiciyde8wclARF51eFnUpIh0UCiIRaG6N8czGBj569jh9N0H6FG2NIhFYsaGBtniS\nxXPGR12KyHsoFEQisOz1dygfNoB5E4dHXYrIeygURDJs3+E2/rhlL9fNGU9enkVdjsh7KBREMmz5\n+noSSVfXkfRJCgWRDFu2ro5powczY5wuayF9j0JBJIPqDh5l1fYDLJ4zHjN1HUnfo1AQyaCn3twN\nwHXqOpI+SqEgkkFV1fWcObaUyWUaclP6JoWCSIY0NrexescBrpo1NupSRLqlUBDJkGc2NuAOC2eN\niboUkW4pFEQypKq6ngnDBzBz3JCoSxHplkJBJAOaW2O8VLOPhbPG6qwj6dNCDQUzW2Rmm82sxsxu\n76bNDWa2wcyqzew/wqxHJCrPb26kPZFkoY4nSB9XENYLm1k+cDdwJVALrDKzZe6+Ia3NNOBbwIfd\n/YCZjQ6rHpEoVVXXM3JQEfMn6VpH0reFuaewAKhx963u3g48Aizp1OYLwN3ufgDA3feEWI9IJNri\nCV7YtIcrZ44hX9c6kj4uzFAoB3alTdcG89KdAZxhZn82s5VmtqirFzKzW81stZmtbmxsDKlckXC8\nVLOPI+0JdR1JVoj6QHMBMA24FLgR+DczG9a5kbvf6+6V7l45atSoDJcocmqqqusZXFzABaePjLoU\nkRMKMxTqgIq06QnBvHS1wDJ3j7n7NuAtUiEhkhMSSWfFhgYunT6K4oL8qMsROaEwQ2EVMM3MpphZ\nEbAUWNapzZOk9hIwszJS3UlbQ6xJJKPW7DjAviPt6jqSrBFaKLh7HLgNqAI2Ao+6e7WZ3WVmi4Nm\nVcA+M9sAPA/8rbvvC6smkUyrqq6nKD+PS6er21OyQ2inpAK4+3Jgead5d6Q9duAbwU0kp7g7VdX1\nfPj0kZSWFEZdjkiPRH2gWSRnbdjdRO2Bo+o6kqyiUBAJSVV1A3kGV8zUBfAkeygURELydHU9lZNG\nUDa4OOpSRHpMoSASgh37jrCpvpmrdJlsyTIKBZEQVFXXA+h4gmQdhYJICKqqG5g5bggVIwZGXYrI\nB6JQEOlle5pbWbvzgPYSJCspFER62YoNwbCbs3U8QbJPj0LBzN53+kRX80Qk1XU0aeRApo8pjboU\nkQ+sp3sKL/dwnki/1tQa4+W392rYTclax73MhZmNJTUGwgAzOwc4tpUPAXQETaST5zftIZZwFupU\nVMlSJ7r20ULgc6Que/2jtPnNwP8MqSaRrFVVXc+o0mLOqdCwm5KdjhsK7v4r4Fdm9gl3fzxDNYlk\npdZYghc2N/Kxc8rJ07CbkqV6epXU2WY2q/NMd7+rl+sRyVp/2rKXFg27KVmup6FwOO1xCXAtqTES\nRCRQVV1PaUkB50/VsJuSvXoUCu7+w/RpM/snUgPkiAgQTyR5ZmMDl505mqICff1HstfJbr0DSR18\nFhFg1fYDHGiJqetIsl6P9hTM7E3Ag8k8YDTw3bCKEsk2VdX1FBXkcckZGnZTsltPjylcCwwHLgKG\nAcvdfU1oVYlkEXdnxYYGLp5WxqDiUEe4FQldT7uPlgAPAGVAIXCfmX01tKpEssj6uibqDh7lKnUd\nSQ7o6Z81fwWc5+5HAMzs+6Quc/H/wipMJFtUVdenht2coW8xS/br6Z6CAYm06QTvXvJCpF+rqq7n\n3MkjGDGoKOpSRE5ZT/cU7gNeMbMngumPAT8PpySR7LG18TBb9hzmjmtnRl2KSK/o6fcUfmRmLwAX\nBrM+7+6vhVaVSJaoqm4A0FjMkjN6fKqEu68F1oZYi0jWqaquZ3b5ECYM10WDJTfoq5ciJ6n+UCvr\ndh1k4UyddSS5Q6EgcpJWbKgHYOFshYLkDoWCyEmqqm5gStkgpo0eHHUpIr1GoSByEg61xFi5dR9X\nzRqjYTclpygURE7Cio0NxJPOIn2LWXJMqKFgZovMbLOZ1ZjZ7cdp9wkzczOrDLMekd7yh/W7GT+0\nhLkVw6IuRaRXhRYKZpYP3A1cDcwEbjSz933Dx8xKga8Br4RVi0hvam6N8eJbe1k0e5y6jiTnhLmn\nsACocfet7t4OPELqwnqdfRf4PtAaYi0ivea5TXtoTyS5+ix1HUnuCTMUyoFdadO1wbwOZjYPqHD3\n34dYh0iveurNekaXFjN/4vCoSxHpdZEdaDazPOBHwN/0oO2tZrbazFY3NjaGX5xIN1ra47zw1h4W\nzhpLXp66jiT3hBkKdUBF2vSEYN4xpcBs4AUz2w6cByzr6mCzu9/r7pXuXjlqlEa2kuj89+ZGWmPq\nOpLcFWYorAKmmdkUMysClgLLji1090PuXubuk919MrASWOzuq0OsSeSULF9fz4hBRSyYPCLqUkRC\nEVoouHscuA2oAjYCj7p7tZndZWaLw/q5ImFpjSV4bmMDC2eNoSBfX/GR3BTqgLLuvhxY3mneHd20\nvTTMWkRO1R+37OVIe4JFs8dFXYpIaPTnjkgPPbV+N0MHFHLBaSOjLkUkNAoFkR5ojydZsaGBK2aM\noVBdR5LDtHWL9MBLb++luTXONTrrSHKcQkGkB556s57BxQVcOK0s6lJEQqVQEDmBtniCP1TXc8WM\n0RQX5EddjkioFAoiJ/DiW3s5dDTGkrnlJ24skuUUCiInsOz1dxg+sFBdR9IvKBREjqOlPc4zGxq4\n5qxxOutI+gVt5SLHsWJDA0djCRbPGR91KSIZoVAQOY5l695h3NASztW1jqSfUCiIdONgSzsvbmnk\nujnjdZls6TcUCiLdeGp9PbGEq+tI+hWFgkg3nnytjqllg5g1fkjUpYhkjEJBpAs797Xwyrb9fHxe\nOWbqOpL+Q6Eg0oXH19ZiBh+fNyHqUkQySqEg0kky6Ty2ppYLTy9j/LABUZcjklEKBZFOVm7bR93B\no3xyvvYSpP9RKIh08tjqWkqLC1g4S5fJlv5HoSCSprk1xvL1u7l2znhKCnVFVOl/FAoiaZa/uZvW\nWFJdR9JvKRRE0jy6upapowYxb+KwqEsRiYRCQSSwcXcTa3YcYOm5FfpugvRbCgWRwIMrd1BUkMf1\n8yuiLkUkMgoFEVIHmJ98rY7rzh7P8EFFUZcjEhmFggip6xwdaU9w0/mToi5FJFIKBen33J0HVu7g\nrPKhzJkwNOpyRCKlUJB+79Vt+3mr4TA3nTdJB5il31MoSL/34Cs7GVJSwHUaN0FEoSD9W93Boyx/\nczfXV1YwoEjfYBZRKEi/9os/bQPg5gunRFyJSN+gUJB+61BLjIdf3cniOeMp1yWyRYCQQ8HMFpnZ\nZjOrMbPbu1j+DTPbYGZvmNmzZqbzASVjHnxlBy3tCW69eGrUpYj0GaGFgpnlA3cDVwMzgRvNbGan\nZq8Ble5+NvAY8I9h1SOSrjWW4L4/b+eSM0YxY5zGYBY5Jsw9hQVAjbtvdfd24BFgSXoDd3/e3VuC\nyZWALk0pGfHEa3XsPdzGX2svQeQ9wgyFcmBX2nRtMK87twBPdbXAzG41s9VmtrqxsbEXS5T+KJ5I\ncu+LWzmrfCjnnzYy6nJE+pQ+caDZzD4NVAI/6Gq5u9/r7pXuXjlq1KjMFic554nX6ti29whf+chp\n+rKaSCcFIb52HZB+uckJwbz3MLMrgG8Dl7h7W4j1iNAeT/LjZ7dwVvlQDbcp0oUw9xRWAdPMbIqZ\nFQFLgWXpDczsHOBnwGJ33xNiLSIA/Hr1LmoPHOVvrjpDewkiXQgtFNw9DtwGVAEbgUfdvdrM7jKz\nxUGzHwCDgd+Y2TozW9bNy4mcstZYgn95bgvnTh7OJWeoG1KkK2F2H+Huy4Hlnebdkfb4ijB/vki6\nB17eQUNTGz9eeo72EkS60ScONIuE7VBLjJ/+99tcNK2M86bqjCOR7igUpF/4v8+8xcGWdv5+0ZlR\nlyLSpykUJOdt3N3E/S9v5398aCKzyzWIjsjxKBQkp7k731lWzdABhXzzqulRlyPS5ykUJKf91xu7\neXXbfv524ZkMG1gUdTkifZ5CQXJWU2uM//X7jcwuH8Knzq048RNEJNxTUkWidNd/baDxcBv33DSf\n/DydgirSE9pTkJy0YkMDj62p5cuXnsbcimFRlyOSNRQKknP2HW7jW//5BrPGD+Grl02LuhyRrKLu\nI8kp7s63n1hP09E4D/3VXIoK9HePyAeh3xjJKfe/vIM/VNfzjavOYPrY0qjLEck6CgXJGa9u2893\nf7eBK2aM5taLNKKayMlQKEhO2H3oKF9+aA0TRwzkR5+aS57ONhI5KTqmIFmvNZbgSw+u5Wh7goe/\ncB5DSgqjLkkkaykUJKvFEkm+8tBaXq89yE//cj7Txug4gsipUPeRZK1k0vnmb17n2U17uGvJbBbN\n1vCaIqdKoSBZyd2587+q+e26d/jbhdO56bxJUZckkhPUfSRZJ5F0/uHJ9Tz86k7++uKpfPnS06Iu\nSSRnKBQkq7TGEnztkdeoqm7gKx85jW9eNV1Da4r0IoWCZI2DLe3c+sAaXt22n+9cN5PPf3hK1CWJ\n5ByFgmSFdbsO8pWH1rKnuZUfL53LkrnlUZckkpMUCtKnuTv3v7yD7/1+A6NLS3jsixcwR1c9FQmN\nQkH6rF37W/j2k+t58a1GLjtzND+6YY5GTxMJmUJB+pxE0vnlS9v5p6rNmMGd183kM+dP1qUrRDJA\noSB9hrvz9IYGflC1mZo9h/nI9FF87y/OonzYgKhLE+k3FAoSuWTS+e+3GvnJc1t4bedBpo4axD2f\nnsfCWWN1uqlIhikUJDIt7XGefO0dfv6nrbzdeIRxQ0v4/ifO4hPzJlCQry/bi0RBoSAZlUw6K7ft\n4z/X1vHUm7s50p5gdvkQfrx0LtecNY5ChYFIpBQKErojbXFeensfz25s4JmNe9h7uI3BxQVce/Z4\nPlk5gcpJw9VNJNJHKBSk1x1saWfV9gOs2r6fV7btZ33dIRJJp7S4gEumj+KqWWO5csYYBhTlR12q\niHQSaiiY2SLgx0A+8O/u/n86LS8G7gfmA/uAT7n79jBrkt7T0h5n5/4WavYcZtPuZjbVN7FxdzN1\nB48CUJSfx9yKYXzxkqmcP7WMBVNGUFSg7iGRviy0UDCzfOBu4EqgFlhlZsvcfUNas1uAA+5+upkt\nBb4PfCqsmqRn3J3DbXEam9vY09xGY3Db09xGQ1MrO/e3sGNfC3sPt3U8Jz/POG3UIOZPGs5fnjeR\n+ROHM6diGCWF2hsQySZh7iksAGrcfSuAmT0CLAHSQ2EJcGfw+DHgX8zM3N1DrCtruDvxpJMIbvGO\n+2TqPhEsc++Ybk8kaY0laI0laIunHrfFkrTGg/tYgtZ4gtZYkubWGM2tcZpaYzQdjdPcGqOpNU7T\n0Rjx5PvfgsJ8Y3RpCRUjBnDZmaOYNHIQFSMGMrVsENPGDKa4QAEgku3CDIVyYFfadC3woe7auHvc\nzA4BI4G9vV3Mo6t28bMX3wbAg3+Ofey5Ow4ciyLHcX93+rhtOpYHczuWv/ucY8vTp4/9/Pe1wUkm\nIZ5M0sXncq/IzzNKCvIoLSlkyIACSksKKRtcxNRRgygtKWBISSFDBxQyekgxowaXBPfFDB1QqG8V\ni+S4rDjQbGa3ArcCTJw48aReY/igIs4cOwSCzzRLve6xSczenXdsOQbHWry7vNM862j9njapudYx\nj/TX7mJ5xzwz8vOMgrzUfb4Z+fnHpvM65hfkGXlp7Qry8sjPg6KCPEoK8ikuzKekMI/igtR9SWE+\nJYX5FBfk6bRPEelWmKFQB1SkTU8I5nXVptbMCoChpA44v4e73wvcC1BZWXlSfz9fOXMMV84cczJP\nFRHpN8L8k3EVMM3MpphZEbAUWNapzTLgs8HjTwLP6XiCiEh0QttTCI4R3AZUkTol9RfuXm1mdwGr\n3X0Z8HPgATOrAfaTCg4REYlIqMcU3H05sLzTvDvSHrcC14dZg4iI9JyOOIqISAeFgoiIdFAoiIhI\nB4WCiIh0UCiIiEgHy7avBZhZI7DjJJ9eRgiX0OglfbU21fXBqK4Prq/Wlmt1TXL3USdqlHWhcCrM\nbLW7V0ZdR1f6am2q64NRXR9cX62tv9al7iMREemgUBARkQ79LRTujbqA4+irtamuD0Z1fXB9tbZ+\nWVe/OqYgIiLH19/2FERE5DhyLhTM7HozqzazpJlVdlr2LTOrMbPNZrawm+dPMbNXgna/Di773ds1\n/trM1gW37Wa2rpt2283szaDd6t6uo5ufeaeZ1aXVd0037RYF67HGzG7PQF0/MLNNZvaGmT1hZsO6\naZeRdXai/7+ZFQfvc02wPU0Oq5a0n1lhZs+b2Ybgd+BrXbS51MwOpb2/d3T1WiHVd9z3xlJ+Eqyz\nN8xsXgZqmp62LtaZWZOZfb1Tm4ysMzP7hZntMbP1afNGmNkKM9sS3A/v5rmfDdpsMbPPdtWmx9w9\np27ADGA68AJQmTZ/JvA6UAxMAd4G8rt4/qPA0uDxPcCXQq73h8Ad3SzbDpRleP3dCXzzBG3yg/U3\nFSgK1uvMkOu6CigIHn8f+H5U66wn/3/gy8A9weOlwK8z8N6NA+YFj0uBt7qo61Lgd5ncpnr63gDX\nAE+RGozwPOCVDNeXD9STOp8/4+sMuBiYB6xPm/ePwO3B49u72u6BEcDW4H548Hj4ydaRc3sK7r7R\n3Td3sWgJ8Ii7t7n7NqAGWJDewFLjZV4GPBbM+hXwsbBqDX7eDcDDYf2MkCwAatx9q7u3A4+QWr+h\ncfen3T0eTK4kNZJfVHry/19CavuB1PZ0uR0bjzUk7r7b3dcGj5uBjaTGQc8WS4D7PWUlMMzMxmXw\n518OvO3uJ/vl2FPi7i+SGlcmXfp21N3n0UJghbvvd/cDwApg0cnWkXOhcBzlwK606Vre/wszEjiY\n9uHTVZvedBHQ4O5bulnuwNNmtiYYpzpTbgt233/Rze5qT9ZlmG4m9RdlVzKxznry/+9oE2xPh0ht\nXxkRdFedA7zSxeLzzex1M3vKzGZlqiZO/N5EvV0tpfs/0KJaZ2PcfXfwuB7oakzhXl1voQ6yExYz\newYY28Wib7v7bzNdT1d6WOONHH8v4UJ3rzOz0cAKM9sU/DURWm3AT4HvkvoF/i6p7q2bT/Vnnmpd\nx9aZmX0biAMPdfMyoayzbGJmg4HHga+7e1OnxWtJdY8cDo4XPQlMy1Bpffa9CY4dLga+1cXiKNdZ\nB3d3Mwv9dNGsDAV3v+IknlYHVKRNTwjmpdtHape1IPjrrqs2vVKjmRUAHwfmH+c16oL7PWb2BKlu\ni1P+Jerp+jOzfwN+18WinqzLXq/LzD4HXAtc7kFnahevEco666Qn//9jbWqD93ooqe0rVGZWSCoQ\nHnL3/+y8PD0k3H25mf2rmZW5e+jX+OnBexPKdtVDVwNr3b2h84Io1xnQYGbj3H130JW2p4s2daSO\nexwzgdQx1ZPSn7qPlgFLg7NCppBK+lfTGwQfNM8DnwxmfRYIa8/jCmCTu9d2tdDMBplZ6bHHpA60\nru+qbW/q1If7F938zFXANEudqVVEard7Wch1LQL+Dljs7i3dtMnUOuvJ/38Zqe0HUtvTc90FWW8J\njln8HNjo7j/qps3YY8c2zGxzRGHzAAAB0klEQVQBqc+ATIRVT96bZcBngrOQzgMOpXWdhK3bvfao\n1lkgfTvq7vOoCrjKzIYH3b1XBfNOTthH1DN9I/VBVgu0AQ1AVdqyb5M6a2QzcHXa/OXA+ODxVFJh\nUQP8BigOqc5fAl/sNG88sDytjteDWzWpLpRMrL8HgDeBN4INclzn2oLpa0id3fJ2JmoL3o9dwLrg\ndk/nujK5zrr6/wN3kQotgJJg+6kJtqepGVhHF5Lq9nsjbT1dA3zx2LYG3Basm9dJHbC/IEPbVZfv\nTafaDLg7WKdvknb2YMi1DSL1IT80bV7G1xmpUNoNxILPsFtIHYd6FtgCPAOMCNpWAv+e9tybg22t\nBvj8qdShbzSLiEiH/tR9JCIiJ6BQEBGRDgoFERHpoFAQEZEOCgUREemgUBA5BWb2UtQ1iPQmnZIq\nIiIdtKcgcgrM7HBwf6mZvWBmj1lq3IeHwr4qqkgYFAoivecc4Oukxu6YCnw42nJEPjiFgkjvedXd\na909SeoSE5MjrkfkA1MoiPSetrTHCbL0KsTSvykURESkg0JBREQ66JRUERHpoD0FERHpoFAQEZEO\nCgUREemgUBARkQ4KBRER6aBQEBGRDgoFERHpoFAQEZEO/x9pMQj1bQXNhAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1084706a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores_p = np.arange(-10, 10, 0.1)\n",
    "out_p = 1. / (1. + np.exp(-scores_p))\n",
    "plt.plot(scores_p, out_p)\n",
    "plt.xlabel('in')\n",
    "plt.ylabel('out')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forward pass: (3000, 100)\n",
      "Backward pass: (3000, 100)\n"
     ]
    }
   ],
   "source": [
    "class Sigmoid:\n",
    "\n",
    "    def forward(self,scores):\n",
    "        #scores: N x K\n",
    "        out = 1.0 / (1.0 + np.exp(-scores))\n",
    "        return out\n",
    "    \n",
    "    def backward(self,scores,previous):\n",
    "        out = self.forward(scores)\n",
    "        dscores = (1.0 - out) * out * previous\n",
    "        return(dscores)\n",
    "#test\n",
    "sig = Sigmoid()\n",
    "out = sig.forward(scores)\n",
    "dscores = sig.backward(scores,previous)\n",
    "print('Forward pass: {}'.format(out.shape))\n",
    "print('Backward pass: {}'.format(dscores.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Tanh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Forward Pass\n",
    "\n",
    "$$out = \\tanh(scores)$$\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "$$\\frac{\\partial previous}{\\partial scores} = \\frac{\\partial previous}{\\partial out} \\cdot \\frac{\\partial out}{\\partial scores}\n",
    "\\\\\n",
    "\\frac{\\partial out}{\\partial scores} = 1 - \\tanh^2(scores) = 1 - out^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'out')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEKCAYAAAA1qaOTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XuUnXV97/H3Z25JSMg9hFzJhQAB\nuY8BxVqFgGgrwVYtdKnR4slpT/HU04vCYS3qQu2R9rT29NSjpgii9QiKcowaVwSEdlVuGTAXQhIy\nkxAyYZKZTG4zmSRz+54/9hPdDDOTSWbv/ey95/Naa695Lr9n7+88s2c+8zy/Zz8/RQRmZmbDVZF2\nAWZmVh4cKGZmlhMOFDMzywkHipmZ5YQDxczMcsKBYmZmOeFAMTOznHCgmJlZTjhQzMwsJ6rSLqCQ\npk6dGvPmzUu7DDOzkvLCCy/si4hpJ2s3ogJl3rx51NXVpV2GmVlJkbRzKO18ysvMzHLCgWJmZjnh\nQDEzs5xwoJiZWU44UMzMLCdSDRRJ90tqlvTSAOsl6Z8k1UvaIOmKrHXLJW1LHssLV7WZmfUn7SOU\nbwI3DrL+vcCi5LEC+CqApMnAXwNXAUuAv5Y0Ka+VmpnZoFL9HEpE/LukeYM0WQZ8KzLjFD8raaKk\nGcC7gMciYj+ApMfIBNN381uxmfV1vLuH9mPdtB/vpu1YNx2dPXT19NLZ00t3T9Dd00tXb9DV3Ut3\nby9dybLegABODEMeAUEkX984T9Kuv3VFOYh5EQ6tvvzt85gyblReX6PYP9g4C9iVNd+YLBto+ZtI\nWkHm6Ia5c+fmp0qzMtV+vJtX9raxbW8buw8cZc/hY+w5fJw9h46yr72T9mPddPb0pl1mUZLSruCN\nbrps1ogPlGGLiJXASoDa2tri+7fBrIg0Hz7G0w2tPN2wj+d27Gdna8ev11UIpo4bxYwJo5k3ZSxv\nnTeZ8WOqGTeq6jeP0VWMramiulJUV1VQXVFBVaUy85UVVFVWUF0hqiorqBAIgTJ/fAVISr5m1p34\no5w9/6Z2xfaXewQr9kDZDczJmp+dLNtN5rRX9vKnClaVWRk52tnDmk17+P4Lu3i6oZUImDCmmqvm\nT+bDtXM4b/qZnDd9HLMmjqGqMu1uVytmxR4oq4DbJT1EpgP+UEQ0SVoD/E1WR/wNwJ1pFWlWig4d\n7eL+/9jBA7/cweFj3cyZPIY/u24RSxdP58IZ46mo8H/+dmpSDRRJ3yVzpDFVUiOZK7eqASLia8Bq\n4H1APdABfCJZt1/S54G1yVPdc6KD3swGFxH88MXd/M3qzbQe6eSGC6fz8WvmcfX8KQ4RG5a0r/K6\n9STrA/jTAdbdD9yfj7rMytWhji4+84P1rNm0l8vnTuSbn1jCxbMnpF2WlYliP+VlZjny6r4jLH/g\neV4/eJS73reY294x30ckllMOFLMRYHPTYT5y33P0RvDQiqu58pzJaZdkZciBYlbmduw7wke/8Rw1\nVRV855NXsWDauLRLsjLlawDNytihji6W3/88vQHfvs1hYvnlQDErU729wacf/hVNh47yLx+r5dyz\nHCaWXw4UszJ1/y938OTWFu5+/0VceY7vnWr550AxK0M79h3h79ZsZeni6XzkKt/DzgrDgWJWZiKC\nO3+4gZqqCr74gbf4XldWMA4UszLz85f38uz2/XzmxguYPn502uXYCOJAMSsjXT293PuzLZx71jhu\nfeuck29glkMOFLMy8sgLjWzfd4Q7brzAdwa2gvM7zqxM9PQGX/u3Bi6dPYHrFp+Vdjk2AjlQzMrE\nz15qYmdrB3/82wvdEW+pcKCYlYGI4Ov/tp0FU8dyw0Vnp12OjVAOFLMysL7xEBt3H+IT75hPpe8g\nbClJNVAk3Shpq6R6SXf0s/7LktYlj1ckHcxa15O1blVhKzcrLg89/xpjqiu5+bKZaZdiI1hqdxuW\nVAl8BbgeaATWSloVES+faBMR/y2r/aeAy7Oe4mhEXFaoes2KVduxLlatf52bLp3JmaOr0y7HRrA0\nj1CWAPURsT0iOoGHgGWDtL8V+G5BKjMrIT9e30RHZw+3+hYrlrI0A2UWsCtrvjFZ9iaSzgHmA7/I\nWjxaUp2kZyXdnL8yzYrbqvW7WThtLJd6KF9LWal0yt8CPBIRPVnLzomIWuAPgX+UtLC/DSWtSIKn\nrqWlpRC1mhXM3sPHeG7Hft5/6UxfKmypSzNQdgPZ94aYnSzrzy30Od0VEbuTr9uBp3hj/0p2u5UR\nURsRtdOmTRtuzWZFZfXGJiLgdy9xZ7ylL81AWQsskjRfUg2Z0HjT1VqSLgAmAc9kLZskaVQyPRW4\nBni577Zm5e7H619n8YzxHjzLikJqgRIR3cDtwBpgM/C9iNgk6R5JN2U1vQV4KCIia9lioE7SeuBJ\n4EvZV4eZjQR7Dx/jxdcO8jsX+4OMVhxSu2wYICJWA6v7LLu7z/zn+tnuaeDivBZnVuSe3NIMwNIL\np6dciVlGqXTKm1kfT2xpZtbEMZw//cy0SzEDHChmJelYVw+/rN/HtRec5au7rGg4UMxK0HM79tPR\n2cO1F/g29VY8HChmJeiprc2MqqrgbQunpF2K2a85UMxK0NP1rSyZP5nR1ZVpl2L2aw4UsxLT0nac\nrXvbfHRiRceBYlZintneCsA1C6emXInZGzlQzErMMw37OHN0FRfNHJ92KWZv4EAxKzFPN7Ry1fwp\nVFX619eKi9+RZiWk6dBRdrZ28Hb3n1gRcqCYlZC6Vw8A8NZ5k1OuxOzNHChmJeSFnQcYU13JBTN8\nuxUrPg4UsxJSt3M/l82ZSLX7T6wI+V1pViKOHO9mc1MbtfMmpV2KWb8cKGYlYt2ug/T0Blee40Cx\n4uRAMSsRda8eQIIrHChWpFINFEk3StoqqV7SHf2s/7ikFknrkscns9Ytl7QteSwvbOVmhbdu1wEW\nnTWO8aOr0y7FrF+pjdgoqRL4CnA90AislbSqn6F8H46I2/tsOxn4a6AWCOCFZNsDBSjdrOAigg2N\nh3i3b1dvRSzNI5QlQH1EbI+ITuAhYNkQt30P8FhE7E9C5DHgxjzVaZa61w8do/VIJ5fOnpB2KWYD\nSjNQZgG7suYbk2V9/b6kDZIekTTnFLc1Kwsbdh0E4OLZE1OuxGxgxd4p/2NgXkRcQuYo5MFTfQJJ\nKyTVSapraWnJeYFmhbC+8RDVlWKxP9BoRSzNQNkNzMman50s+7WIaI2I48nsfcCVQ9026zlWRkRt\nRNROmzYtJ4WbFdqGxoNccPZ4RlV5QC0rXmkGylpgkaT5kmqAW4BV2Q0kzciavQnYnEyvAW6QNEnS\nJOCGZJlZ2entDTY2HuIS959YkUvtKq+I6JZ0O5kgqATuj4hNku4B6iJiFfBfJd0EdAP7gY8n2+6X\n9HkyoQRwT0TsL/g3YVYAO/d30Ha824FiRS+1QAGIiNXA6j7L7s6avhO4c4Bt7wfuz2uBZkVgS9Nh\nAC6c4UCx4lbsnfJmI97mpsNUCBZNH5d2KWaDcqCYFbnNe9qYP3Uso6vdIW/FzYFiVuS27DnM4hke\nP96KnwPFrIi1Heti1/6jDhQrCQ4UsyK2dU8bgD/QaCXBgWJWxDYngXLB2T5CseLnQDErYpubDjN+\ndBUzJoxOuxSzk3KgmBWxLU2ZDnlJaZdidlIOFLMi1dsbbNnT5g55KxkOFLMitetABx2dPe6Qt5Lh\nQDErUpub3CFvpcWBYlakTtxy5bzpPkKx0uBAMStSW/YcZt7UsYyp8S1XrDQ4UMyK1OamNhb7dJeV\nEAeKWRFqP97Na/s73CFvJcWBYlaEtu3NdMif7yMUKyGpBoqkGyVtlVQv6Y5+1v+5pJclbZD0hKRz\nstb1SFqXPFb13daslNU3twNw7lkeA8VKR2ojNkqqBL4CXA80AmslrYqIl7Oa/QqojYgOSX8C/C3w\nB8m6oxFxWUGLNiuQhpYj1FRWMGfSmLRLMRuyNI9QlgD1EbE9IjqBh4Bl2Q0i4smI6EhmnwVmF7hG\ns1TUN7czb+oZVFX6rLSVjjTfrbOAXVnzjcmygdwG/CxrfrSkOknPSrp5oI0krUja1bW0tAyvYrMC\naWhp9+kuKzkl8e+PpI8AtcDfZS0+JyJqgT8E/lHSwv62jYiVEVEbEbXTpk0rQLVmw3O8u4fX9new\ncJoDxUpLmoGyG5iTNT87WfYGkpYCdwE3RcTxE8sjYnfydTvwFHB5Pos1K5SdrR309IaPUKzkpBko\na4FFkuZLqgFuAd5wtZaky4GvkwmT5qzlkySNSqanAtcA2Z35ZiWrIbnCy0coVmpSu8orIrol3Q6s\nASqB+yNik6R7gLqIWEXmFNc44PvJeBCvRcRNwGLg65J6yYTil/pcHWZWsk5cMrxg2tiUKzE7NakF\nCkBErAZW91l2d9b00gG2exq4OL/VmaWjoaWdWRPHcEZNqr+eZqesJDrlzUaS+pZ2Frr/xEqQA8Ws\niPT2Bg3NR1jo011WghwoZkWk6fAxjnb1+AovK0kOFLMi4iu8rJQ5UMyKiG8KaaXMgWJWRBpa2pl4\nRjVTxtakXYrZKXOgmBWR+uZ2Fk4bR/K5K7OS4kAxKyINLe2c6/4TK1EOFLMicbCjk33tnSw8y5cM\nW2lyoJgViYYWd8hbaXOgmBWJhuYjgC8ZttLlQDErEvUt7dRUVTB70hlpl2J2WhwoZkWiobmdBVPH\nUlnhK7ysNA0pUE6MPXKyZWZ2+nxTSCt1Qz1CeWaIy8zsNBzr6mGXh/21EjdooEg6W9KVwBhJl0u6\nInm8Cxj2iV5JN0raKqle0h39rB8l6eFk/XOS5mWtuzNZvlXSe4Zbi1maXm09Qm/4Ci8rbScbwec9\nwMfJjPf+D1nL24D/PpwXllQJfAW4HmgE1kpa1WfkxduAAxFxrqRbgHuBP5B0IZkhgy8CZgKPSzov\nInqGU5NZWn5zhZc/g2Kla9BAiYgHgQcl/X5E/CDHr70EqI+I7QCSHgKW8cax4ZcBn0umHwH+WZl7\nUiwDHoqI48AOSfXJ8/k0nJWk+uZ2JFgw1UcoVrqGOsboWyRd1HdhRNwzjNeeBezKmm8ErhqoTTIG\n/SFgSrL82T7bzhpGLWapOjHs75iayrRLMTttQ+2UbweOJI8e4L3AvDzVlFOSVkiqk1TX0tKSdjlm\n/apvbnf/iZW8IR2hRMTfZ89L+p/AmmG+9m5gTtb87GRZf20aJVUBE4DWIW4LQESsBFYC1NbWxjBr\nNsu53t5g+7523rZwStqlmA3L6X6w8Qwyf8SHYy2wSNJ8STVkOtlX9WmzClieTH8Q+EVERLL8luQq\nsPnAIuD5YdZjlordB49yrKvXRyhW8oZ0hCJpI3Div/sK4Czg88N54aRP5HYyRzqVwP0RsUnSPUBd\nRKwCvgF8O+l0308mdEjafY9MB3438Ke+wstKVX2Lh/218jDUTvnfBSYBvwVMBFZHxAvDffGIWA2s\n7rPs7qzpY8CHBtj2i8AXh1uDWdoaPOyvlYmhnvJaBnwbmApUAw9I+lTeqjIbQRpa2pl0RjWTPeyv\nlbihHqF8Erg6Io4ASLqXzGc+/ne+CjMbKRqaj/joxMrCUI9QROZy4RN6kmVmNkz1Le3uP7GyMNQj\nlAeA5yQ9mszfTKbD3MyGYf+RTvYf6fQRipWFoX4O5R8kPQW8I1n0iYj4Vd6qMhshGnyFl5WRoR6h\nEBEvAi/msRazEcdXeFk58YiNZimqb25nVFUFsyaOSbsUs2FzoJilqKGlnQXTxlHhYX+tDDhQzFJU\n3+KbQlr5cKCYpeRYVw+NB456UC0rGw4Us5RsbzlCeNhfKyMOFLOU+KaQVm4cKGYpaUiG/Z0/1ae8\nrDw4UMxSUt/SzpxJZzC62sP+WnlwoJilpKG53R3yVlYcKGYp6O7pZXvLEc6bfmbapZjlTCqBImmy\npMckbUu+TuqnzWWSnpG0SdIGSX+Qte6bknZIWpc8Livsd2A2PK+2dtDZ0+tAsbKS1hHKHcATEbEI\neCKZ76sD+FhEXATcCPyjpIlZ6/8qIi5LHuvyX7JZ7ryytw3AgWJlJa1AWQY8mEw/SOZ2+G8QEa9E\nxLZk+nWgGZhWsArN8uiVvW1I/gyKlZe0AmV6RDQl03uA6YM1lrQEqAEashZ/MTkV9mVJo/JUp1le\nbNvbztzJZzCmxld4WfkY8u3rT5Wkx4Gz+1l1V/ZMRISkGOR5ZpAZz355RPQmi+8kE0Q1wErgs8A9\nA2y/AlgBMHfu3FP8LszyY+veNp/usrKTt0CJiKUDrZO0V9KMiGhKAqN5gHbjgZ8Cd0XEs1nPfeLo\n5rikB4C/HKSOlWRCh9ra2gGDy6xQjnf38Oq+I7znokEPzM1KTlqnvFYBy5Pp5cCP+jaQVAM8Cnwr\nIh7ps25G8lVk+l9eymu1Zjm0Y98RunvDRyhWdtIKlC8B10vaBixN5pFUK+m+pM2HgXcCH+/n8uDv\nSNoIbASmAl8obPlmp++VvZl7eDlQrNzk7ZTXYCKiFbiun+V1wCeT6X8F/nWA7a/Na4FmebRtbxuV\nFWKBPyVvZcaflDcrsK172pg35QxGVfkKLysvDhSzAtvW3O7TXVaWHChmBXSsq4dXW30PLytPDhSz\nAqpvbifCHfJWnhwoZgX0m3t4+ZYrVn4cKGYF9MredqorxTyP0mhlyIFiVkDb9raxYOo4qiv9q2fl\nx+9qswLaureNRT7dZWXKgWJWIIeOdtF44CgXzhyfdilmeeFAMSuQzU2HAbhwhgPFypMDxaxAXn49\nCRQfoViZcqCYFcjLTYeZOm4UZ505Ou1SzPLCgWJWIJteP8xFPjqxMuZAMSuAzu5e6pvbfLrLypoD\nxawAtjW30dUT7pC3suZAMSuATUmH/GIHipWxVAJF0mRJj0nalnydNEC7nqzRGldlLZ8v6TlJ9ZIe\nToYLNitaGxsPMW5UFQt8yxUrY2kdodwBPBERi4Ankvn+HI2Iy5LHTVnL7wW+HBHnAgeA2/Jbrtnw\nbGg8yFtmjaeiQmmXYpY3aQXKMuDBZPpB4OahbihJwLXAI6ezvVmhdXb3srmpjUtmT0y7FLO8SitQ\npkdEUzK9B5g+QLvRkuokPSvpRGhMAQ5GRHcy3wjMGuiFJK1InqOupaUlJ8WbnYqte9ro7OnlktkT\n0i7FLK+q8vXEkh4Hzu5n1V3ZMxERkmKApzknInZLWgD8QtJG4NCp1BERK4GVALW1tQO9jlnerG88\nCMClPkKxMpe3QImIpQOtk7RX0oyIaJI0A2ge4Dl2J1+3S3oKuBz4ATBRUlVylDIb2J3zb8AsRzY0\nHmTSGdXMnjQm7VLM8iqtU16rgOXJ9HLgR30bSJokaVQyPRW4Bng5IgJ4EvjgYNubFYsNjYe4ZPZE\nMt1/ZuUrrUD5EnC9pG3A0mQeSbWS7kvaLAbqJK0nEyBfioiXk3WfBf5cUj2ZPpVvFLR6syFqP97N\nK3vbuHSOT3dZ+cvbKa/BREQrcF0/y+uATybTTwMXD7D9dmBJPms0y4V1rx2kN6D2nH4/amVWVvxJ\nebM8qtu5Hwkum+sjFCt/DhSzPHph5wHOn34m40dXp12KWd45UMzypKc3+NVrB6md59NdNjI4UMzy\nZMuew7Qf76b2nMlpl2JWEA4Uszx5YecBAK50h7yNEA4Uszx5pqGVWRPH+AONNmI4UMzyoLc3eGZ7\nK29bOMUfaLQRw4FilgcvNx3mYEcX15w7Je1SzArGgWKWB880tALw9oVTU67ErHAcKGZ58MuGfSyc\nNpbp40enXYpZwThQzHKss7uX53fs99GJjTgOFLMcW/vqfjo6e/jt86alXYpZQTlQzHLsic3NjKqq\n4JpzfYRiI4sDxSyHIoIntuzl7QunMKamMu1yzArKgWKWQ9v3HWFnawfXLp6edilmBedAMcuhX2zO\njGZ97QVnpVyJWeGlEiiSJkt6TNK25OubbnYk6d2S1mU9jkm6OVn3TUk7stZdVvjvwuzNfrqxiYtm\njmfWRN9uxUaetI5Q7gCeiIhFwBPJ/BtExJMRcVlEXAZcC3QAP89q8lcn1kfEuoJUbTaIXfs7WLfr\nIO+/dGbapZilIq1AWQY8mEw/CNx8kvYfBH4WER15rcpsGH6yoQmA37l4RsqVmKUjrUCZHhFNyfQe\n4GQ9mLcA3+2z7IuSNkj6sqRRA20oaYWkOkl1LS0twyjZbHA/Xv86l8+dyJzJZ6Rdilkq8hYokh6X\n9FI/j2XZ7SIigBjkeWYAFwNrshbfCVwAvBWYDHx2oO0jYmVE1EZE7bRp/qCZ5cfWPW283HSY91/i\n0102clXl64kjYulA6yTtlTQjIpqSwGge5Kk+DDwaEV1Zz33i6Oa4pAeAv8xJ0Wan6bvPv0ZNZQUf\nuHxW2qWYpSatU16rgOXJ9HLgR4O0vZU+p7uSEEKZgSZuBl7KQ41mQ3K0s4cfvtjIjW85m0lja9Iu\nxyw1aQXKl4DrJW0DlibzSKqVdN+JRpLmAXOAf+uz/XckbQQ2AlOBLxSgZrN+rd7YxOFj3dy6ZG7a\npZilKm+nvAYTEa3Adf0srwM+mTX/KvCmcwgRcW0+6zMbqojg/l/uYMG0sVy9YHLa5Zilyp+UNxuG\n/6jfx6bXD/Of37nAQ/3aiOdAMRuGrz7VwPTxo7jZnfFmDhSz07X21f083dDKbe+Yz6gq31nYzIFi\ndhoigi/8dDNnjx/NR6+el3Y5ZkXBgWJ2Gn6yoYn1uw7yFzec53FPzBIOFLNTdOhoF1/46ctcOGM8\nv3fF7LTLMSsaqVw2bFbK/uanm9nX3sl9H3srlRW+ssvsBB+hmJ2CNZv28HDdLla8cwEXz56Qdjlm\nRcWBYjZEDS3t/MX31nPp7Al8eumitMsxKzoOFLMhaGk7zm3fXEtNVQVf/ciVvkzYrB8OFLOT2H+k\nk49+4zn2Hj7Ov3yslpke3tesX+6UNxvEa60dLH/geV4/eJT7ltdy5TmT0i7JrGg5UMwGsGbTHj7z\nyAYk+L//6SquPMc3fzQbjAPFrI/mtmP8j9VbePRXu3nLrPH8861XMG/q2LTLMit6DhSzxO6DR/nW\nM6/yrad30tMbfOrac7n92nPdAW82RKkEiqQPAZ8DFgNLknFQ+mt3I/C/gErgvog4MRDXfOAhYArw\nAvDRiOgsQOlWZg4f6+LJLc18v66RXzbsA+CmS2fy6aXnMd9HJWanJK0jlJeA3wO+PlADSZXAV4Dr\ngUZgraRVEfEycC/w5Yh4SNLXgNuAr+a/bCtl3T297NzfwSt72ljfeIhntreysfEgvQGzJo7hU9cu\n4kNXzmbO5DPSLtWsJKU1YuNm4GQDEi0B6iNie9L2IWCZpM3AtcAfJu0eJHO040ApcxFBZ08vXT1B\nV3cvnT29dHb30tXTS0dnD4eOdnHoaBeHk6+HjnbR2t7J64eO8vrBo+w6cJTO7l4AqirE5XMncvu7\nz+Wac6fy1nmTqfBtVMyGpZj7UGYBu7LmG4GryJzmOhgR3VnL8zq60V2PbuS5HfuBzB+1bNG3cQy8\n7mTbxhu2jQHX9Tc/1Nd50/MM8jqD1de3xZufd+CaTva8fdv29AZdSZCciqoKMWlsDTMnjuH8s8/k\nusXTOW/6mVxw9pmce9Y4Rle7b8Qsl/IWKJIeB87uZ9VdEfGjfL1uP3WsAFYAzJ0797SeY+bEMZw/\n/cysJ+3zGm9+zUHWnf62b37drLYnfd7+t+t32zfMn6TtoOsG3naw77uvqgpRU1VBdWUFNVUV1FRW\nUF0paqoqk68VjKmuZMKYaiacUc340dVMGFPNGTWVHpbXrIDyFigRsXSYT7EbmJM1PztZ1gpMlFSV\nHKWcWD5QHSuBlQC1tbWn9i9u4k/ffe7pbGZmNqIU861X1gKLJM2XVAPcAqyKzPmQJ4EPJu2WAwU7\n4jEzs/6lEiiSPiCpEXgb8FNJa5LlMyWtBkiOPm4H1gCbge9FxKbkKT4L/LmkejJ9Kt8o9PdgZmZv\npL4duOWstrY26ur6/ciLmZkNQNILEVF7snbFfMrLzMxKiAPFzMxywoFiZmY54UAxM7OccKCYmVlO\njKirvCS1ADtPc/OpwL4clpMrxVoXFG9truvUuK5TV6y1nW5d50TEtJM1GlGBMhyS6oZy2VyhFWtd\nULy1ua5T47pOXbHWlu+6fMrLzMxywoFiZmY54UAZupVpFzCAYq0Lirc213VqXNepK9ba8lqX+1DM\nzCwnfIRiZmY54UDJIulDkjZJ6pVU22fdnZLqJW2V9J4Btp8v6bmk3cPJbfdzXePDktYlj1clrRug\n3auSNibtCnJHTEmfk7Q7q773DdDuxmQ/1ku6owB1/Z2kLZI2SHpU0sQB2hVkn53s+5c0Kvk51yfv\np3n5qiXrNedIelLSy8nvwJ/10+Zdkg5l/XzvznddyesO+nNRxj8l+2uDpCsKVNf5WftinaTDkj7d\np01B9pmk+yU1S3opa9lkSY9J2pZ8nTTAtsuTNtskLR9WIRHhR/IAFgPnA08BtVnLLwTWA6OA+UAD\nUNnP9t8Dbkmmvwb8SZ7r/Xvg7gHWvQpMLfD++xzwlydpU5nsvwVATbJfL8xzXTcAVcn0vcC9ae2z\noXz/wH8BvpZM3wI8XICf3QzgimT6TOCVfup6F/CTQr6nhvJzAd4H/IzMQKBXA8+lUGMlsIfM5zUK\nvs+AdwJXAC9lLftb4I5k+o7+3vfAZGB78nVSMj3pdOvwEUqWiNgcEVv7WbUMeCgijkfEDqAeWJLd\nQJmxZq8FHkkWPQjcnK9ak9f7MPDdfL1GniwB6iNie0R0Ag+R2b95ExE/j8z4OgDPkhnlMy1D+f6X\nkXn/QOb9dJ3yPJZxRDRFxIvJdBuZMYhm5fM1c2gZ8K3IeJbMiK4zClzDdUBDRJzuB6eHJSL+Hdjf\nZ3H2+2igv0fvAR6LiP0RcQB4DLjxdOtwoAzNLGBX1nwjb/5lmwIczPrD1V+bXPotYG9EbBtgfQA/\nl/SCpBV5rKOv25PTDvcPcIg9lH2ZT39E5r/Z/hRinw3l+/91m+T9dIjM+6sgklNslwPP9bP6bZLW\nS/qZpIsKVNLJfi5pv6cgcyQ50D93aewzgOkR0ZRM7wGm99Mmp/sub2PKFytJjwNn97PqrogoiqGE\nh1jjrQx+dPKOiNgt6SzgMUn+DA4iAAADWElEQVRbkv9i8lYb8FXg82T+AHyezCm5Pxruaw63rhP7\nTNJdQDfwnQGeJi/7rJRIGgf8APh0RBzus/pFMqd02pP+sf8HLCpAWUX9c0n6Sm8C7uxndVr77A0i\nIiTl/ZLeERcoEbH0NDbbDczJmp+dLMvWSuZQuyr5r7K/NjmpUVIV8HvAlYM8x+7ka7OkR8mcahn2\nL+FQ95+kfwF+0s+qoezLnNcl6ePA7wLXRXLyuJ/nyMs+62Mo3/+JNo3Jz3oCmfdXXkmqJhMm34mI\nH/Zdnx0wEbFa0v+RNDUi8nrPqiH8XPLynjoF7wVejIi9fVektc8SeyXNiIim5BRgcz9tdpPp5zlh\nNpk+5NPiU15Dswq4Jbn6Zj6Z/zCez26Q/JF6Evhgsmg5kK8jnqXAloho7G+lpLGSzjwxTaZT+qX+\n2uZSn/PWHxjgNdcCi5S5Iq6GzKmCVXmu60bgM8BNEdExQJtC7bOhfP+ryLx/IPN++sVAIZgrSR/N\nN4DNEfEPA7Q5+0RfjqQlZP5+5DXohvhzWQV8LLna62rgUNapnkIY8GxBGvssS/b7aKC/R2uAGyRN\nSk5R35AsOz35vvqglB5k/gg2AseBvcCarHV3kbk6Zyvw3qzlq4GZyfQCMkFTD3wfGJWnOr8J/HGf\nZTOB1Vl1rE8em8ic9inE/vs2sBHYkLyZZ/StLZl/H5mriBoKUVvy89gFrEseX+tbVyH3WX/fP3AP\nmcADGJ28f+qT99OCAuyjd5A5Vbkhaz+9D/jjE+814PZk36wnc3HD2wtQV78/lz51CfhKsj83knWF\nZgHqG0smICZkLSv4PiMTaE1AV/I37DYy/W5PANuAx4HJSdta4L6sbf8oea/VA58YTh3+pLyZmeWE\nT3mZmVlOOFDMzCwnHChmZpYTDhQzM8sJB4qZmeWEA8UsJZKeTrsGs1zyZcNmZpYTPkIxS4mk9uTr\nuyQ9JekRZcZt+U6+7y5slg8OFLPicDnwaTJj7ywArkm3HLNT50AxKw7PR0RjRPSSue3JvJTrMTtl\nDhSz4nA8a7qHEXgncCt9DhQzM8sJB4qZmeWELxs2M7Oc8BGKmZnlhAPFzMxywoFiZmY54UAxM7Oc\ncKCYmVlOOFDMzCwnHChmZpYTDhQzM8uJ/w89d9dUl1ugwgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x108464f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores_p = np.arange(-10, 10, 0.1)\n",
    "out_p = np.tanh(scores_p)\n",
    "plt.plot(scores_p, out_p)\n",
    "plt.xlabel('in')\n",
    "plt.ylabel('out')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forward pass: (3000, 100)\n",
      "Backward pass: (3000, 100)\n"
     ]
    }
   ],
   "source": [
    "class Tanh:\n",
    "    def forward(self,scores):\n",
    "        #scores: N x K\n",
    "        out = np.tanh(scores)\n",
    "        return out\n",
    "    \n",
    "    def backward(self,scores,previous):\n",
    "        out = self.forward(scores)\n",
    "        dscores = (1.0 - np.square(out)) * previous\n",
    "        return(dscores)\n",
    "#test\n",
    "tanh = Tanh()\n",
    "out = tanh.forward(scores)\n",
    "dscores = tanh.backward(scores,previous)\n",
    "print('Forward pass: {}'.format(out.shape))\n",
    "print('Backward pass: {}'.format(dscores.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## ReLu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Forward Pass\n",
    "\n",
    "$$out = max(0,scores)$$\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "$$\\frac{\\partial previous}{\\partial scores} = \\frac{\\partial previous}{\\partial out} \\cdot \\frac{\\partial out}{\\partial scores}\n",
    "\\\\\n",
    "\\frac{\\partial out}{\\partial scores} = \n",
    "\\begin{cases}\n",
    "1 & scores > 0 \\\\\n",
    "0 & otherwise\n",
    "\\end{cases}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'out')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHxZJREFUeJzt3Xl4VdW9//H31xAIEEYTZhAEAQHB\nhIiC1NahKk44VCrCT636WEG4olWL8lS90nodrvyqFrFaq7WGSZGfs1Yc2nK1aCZimKOgAYEQmQwQ\nMq3fH+fYG2MCIck56wyf1/PkyTn77LA/WWfnfFl77b2XOecQEZH4dZTvACIi4pcKgYhInFMhEBGJ\ncyoEIiJxToVARCTOqRCIiMQ5FQIRkTinQiAiEudUCERE4lwL3wEaIiUlxfXt29d3DBGRqJKdnV3i\nnEs93HpRUQj69u1LVlaW7xgiIlHFzL5syHo6NCQiEudUCERE4pwKgYhInAtZITCzP5tZsZkV1FjW\n2czeNbMNwe+dQrV9ERFpmFD2CJ4Dzq21bCbwnnPuOOC94HMREfEoZIXAOfcPYGetxeOBvwQf/wW4\nOFTbFxGRhgn3GEFX59zW4ONtQNf6VjSzG8wsy8yyduzYEZ50IiJxyNtgsQvMkVnvPJnOuaeccxnO\nuYzU1MNeDyEiElNKSg9y32urOVBeFfJthbsQbDez7gDB78Vh3r6ISMSrqnbMWJjHCyu+ZNM3+0K+\nvXAXgleBq4OPrwZeCfP2RUQi3qPvbWB5YQmzxw/l+O7tQ769UJ4+ugD4GBhkZpvN7DrgAeCnZrYB\nOCv4XEREgj5cV8zj72/gZyN7MSGjd1i2GbJ7DTnnJtbz0pmh2qaISDTbsvsAtyzKY1DXdswePwwz\nC8t2dWWxiEgEKK+s5qbMHCqqHE9MSqd1y4SwbTsq7j4qIhLr7n9zDXlFu3liUjrHpiaHddvqEYiI\nePbayq957qNNXHtqP847oXvYt69CICLiUWFxKTOX5JPepyMzxw32kkGFQETEk/3llUzNzKZVYgJz\nJ6XTsoWfj2SNEYiIeOCcY9bSAjYUl/L8taPo3qG1tyzqEYiIeLDgkyKW5m5hxpkD+dFxfm+jo0Ig\nIhJmn23ew72vruK0galMP2OA7zgqBCIi4bRnfwVT52dzdHJLfv/zEznqqPBcNHYoGiMQEQmT6mrH\nr17MY+vuMhbfOJrObVv6jgSoRyAiEjZP/fMLlq0pZtb5x5PeJ3Jm6lUhEBEJg3998Q0Pv7OO80/o\nzjVj+vqO8z0qBCIiIVa8t4zpC3I5pnMbHrjshLDdTK6hNEYgIhJClVXVTF+Qy7dlFfz1ulG0S0r0\nHekHVAhERELokXfXs2LjTuZMGMHgbqGfZKYxdGhIRCRElq3ezrwPP2fiqD5cmt7Ld5x6qRCIiIRA\n0c793Lo4j6E92nPPhUN8xzkkFQIRkWZWVlHFlMxsHDBv0kiSEsM3yUxjaIxARKSZzX59NQVb9vL0\nVRn0ObqN7ziHpR6BiEgzWpq7mcwVX/HLHx/LT4d09R2nQVQIRESayfrt33LXywWM6teZ288e5DtO\ng6kQiIg0g9KDldz4QjZtW7XgDxPTaJEQPR+v0ZNURCRCOee48+XP2FSyj8cmnkiX9km+Ix0RFQIR\nkSZ6/uMveW3l1/zq7EGM6Z/iO84RUyEQEWmC3K928ds3VnPm4C5M+XF/33EaRYVARKSRdu0r56bM\nHLq2T+KRCSMiYpKZxtB1BCIijVBd7ZixKI+S0nJemjKajm0iY5KZxlCPQESkEeZ+UMjf1+/g7guH\nMLxXR99xmkSFQETkCC3fUMKcZeu5+MQeTDq5j+84TaZCICJyBLbtKePmhbkMSE3md5dE3iQzjaFC\nICLSQBVV1Uybn8OBiirmTU6nbavYGGb1UgjM7BYzW2VmBWa2wMyi6+oLEYlLD729lqwvd/HAZcMZ\n0KWd7zjNJuyFwMx6Av8BZDjnhgEJwBXhziEiciTeLtjK0//cyFWjj+GiET18x2lWvg4NtQBam1kL\noA3wtaccIiKHtalkH7e/mM+IXh2Ydf7xvuM0u7AXAufcFuC/ga+ArcAe59zfaq9nZjeYWZaZZe3Y\nsSPcMUVEgO8mmckhIcGYOymdVi0ie5KZxvBxaKgTMB7oB/QA2prZ5NrrOeeecs5lOOcyUlNTwx1T\nRASAu18pYM3Wvfzfn59Ir06RP8lMY/g4NHQWsNE5t8M5VwG8DIzxkENE5JAWf1rE4qzNTD9jAKcP\n6uI7Tsj4KARfAaeYWRsLnIB7JrDGQw4RkXqt/novv3mlgDH9j2bGWQN9xwkpH2MEK4CXgBzgs2CG\np8KdQ0SkPnvLKpiamU3HNok8NjGNhCi9mVxDebkawjl3D3CPj22LiByKc447XsynaNcBFt5wCinJ\nrXxHCjldWSwiUsMzyzfy9qptzDx3MCf17ew7TlioEIiIBGVt2skDb63l7CFduf5H/XzHCRsVAhER\noKT0INPm59KzU2sevnxETNxMrqFi445JIiJNUFXtmLEwj537y1k6dQwdWif6jhRW6hGISNx79L0N\nLC8sYfb4oQzt0cF3nLBTIRCRuPbhumIef38DPxvZiwkZvX3H8UKFQETi1pbdB5ixKI9BXdsxe/yw\nuBoXqEmFQETiUnllNVMzc6iscsybPJLWLWPvZnINpcFiEYlL97+5hpVFu5k3KZ1+KW19x/FKPQIR\niTuvrfya5z7axHVj+zHuhO6+43inQiAicaWwuJSZS/IZeUwnZo4b7DtORFAhEJG4sb+8kqmZ2bRK\nTOAPV6aRmKCPQNAYgYjECeccs5YWsKG4lOevHUX3Dq19R4oYKociEhcWfFLE0twtzDhzID86TrMe\n1qRCICIx77PNe7j31VWcNjCV6WcM8B0n4qgQiEhM27O/gqnzs0lJbsnvf34iR8X4JDONoTECEYlZ\n1dWOX72Yx7Y9ZSz65Wg6t23pO1JEUo9ARGLWH//xBcvWFDPrvONJ79PJd5yIpUIgIjHp48+/4eF3\n1nL+8O5cPaav7zgRTYVARGJO8d4ypi/IpW9KWx68bHjc3kyuoTRGICIxpbKqmukLcik9WEHm9SeT\n3Eofc4ejFhKRmPLIu+tZsXEncyaMYFC3dr7jRAUdGhKRmLFs9Xbmffg5E0f14dL0Xr7jRA0VAhGJ\nCUU793Pr4jyG9mjPPRcO8R0nqqgQiEjUK6uoYkpmNg6YN2kkSYnxO8lMY2iMQESi3uzXV1OwZS9P\nX5VBn6Pb+I4TddQjEJGotjR3M5krvuLGH/fnp0O6+o4TlVQIRCRqrd/+LXe9XMCofp257eyBvuNE\nLRUCEYlKpQcrufGFbNq2asEfJqbRQpPMNJpaTkSijnOOXy/JZ1PJPh6fmEaX9km+I0U1L4XAzDqa\n2UtmttbM1pjZaB85RCQ6Pf/xl7yRv5XbzhnE6P5H+44T9XydNfQo8LZz7mdm1hLQML+INEjuV7v4\n7RurOXNwF248rb/vODEh7IXAzDoApwHXADjnyoHycOcQkeiza185N2Xm0LV9Eo9MGKFJZpqJj0ND\n/YAdwLNmlmtmfzKzth5yiEgUqa52zFiUR0lpOU9MSqdjG00y01x8FIIWQDowzzmXBuwDZtZeycxu\nMLMsM8vasWNHuDOKSISZ+0Ehf1+/g7svHMLwXh19x4kpPgrBZmCzc25F8PlLBArD9zjnnnLOZTjn\nMlJTU8MaUEQiy/INJcxZtp6LT+zBpJP7+I4Tc8JeCJxz24AiMxsUXHQmsDrcOUQkOmzbU8bNC3MZ\nkJrM/ZeeoElmQsDXWUPTgczgGUNfAL/wlENEIlhFVTXT5udwoKKKeZNH0qalbo8WCl5a1TmXB2T4\n2LaIRI+H3l5L1pe7eGxiGgO6JPuOE7N0ZbGIRKS3C7by9D83cvXoY7hoRA/fcWKaCoGIRJyNJfu4\n/cV8RvTuyF3nH+87TsxTIRCRiFJWUcWUF7JJSDDmXplGqxaaZCbUNPIiIhHl7lcKWLvtW579xUn0\n6qS7z4SDegQiEjEWf1rE4qzNTD9jAKcP6uI7TtxQIRCRiLD667385pUCxvQ/mhlnaZKZcFIhEBHv\n9pZVMDUzm45tEnlsYhoJuplcWGmMQES8cs5xx4v5FO06wMIbTiEluZXvSHFHPQIR8eqZ5Rt5e9U2\n7hw3mJP6dvYdJy6pEIiIN1mbdvLAW2s5Z2hXrhvbz3ecuNWgQmBmP+ir1bVMRKShSkoPMm1+Lj07\ntebhy0foZnIeNbRH8HEDl4mIHFZVtWPGwjx27Q9MMtM+KdF3pLh2yMFiM+sG9ARam1ka8F3Jbo/m\nGRaRRnp02XqWF5bw0GXDGdqjg+84ce9wZw2dQ2Bu4V7AnBrLvwXuClEmEYlhH6wr5rH3C7l8ZC8m\nnNTbdxzhMIXAOfcX4C9mdplzbkmYMolIjNqy+wC3LMpjcLd23Dd+mO84EtTQ6wiGmdnQ2gudc/c1\ncx4RiVHlldVMzcyhssoxb/JIWrfUzeQiRUMLQWmNx0nABcCa5o8jIrHq/jfXsLJoN/MmpdMvpa3v\nOFJDgwqBc+6Rms/N7L+Bd0KSSERizmsrv+a5jzZx3dh+jDuhu+84UktjLyhrQ2AAWUTkkAqLS5m5\nJJ+Rx3Ri5rjBvuNIHRrUIzCzzwAXfHoU0AWYHapQIhIb9pdXMjUzm6TEBOZemU5igm5mEIkaOkZw\nAdAJ+BHQEXjTOZcdslQiEvWcc8xaWsCG4lL+eu3JdOuQ5DuS1KOh5Xk88FcgBUgEnjWz6SFLJSJR\nb8EnRSzN3cItZw1k7HEpvuPIITS0R3A9cIpzbh+AmT1I4BYTj4cqmIhEr8827+HeV1dx2sBUpp0+\nwHccOYyG9ggMqKrxvIr/vd2EiMi/7dlfwZTMbFKSW/L7n5/IUZpkJuI1tEfwLLDCzJYGn18MPBOa\nSCISraqrHbcuzmP73jIW/XI0ndu29B1JGqCh1xHMMbMPgbHBRb9wzuWGLJWIRKU//uML3ltbzL0X\nDiG9TyffcaSBGjxVpXMuB8gJYRYRiWIff/4ND7+zlvOHd+fqMX19x5EjoJN6RaTJiveWMX1BLn1T\n2vLgZcM1yUyU0eT1ItIklVXVTF+Qy76DlWRefzLJrfSxEm30jolIkzzy7npWbNzJnAkjGNStne84\n0gg6NCQijbZs9Xbmffg5V57ch0vTdfuxaOWtEJhZgpnlmtnrvjKISOMV7dzPrYvzGNazPXdfMMR3\nHGkCnz2Cm9GcBiJRqayiiimZgduNzZs0kqRETTITzbwUAjPrBZwP/MnH9kWkaWa/vpqCLXt5ZMKJ\n9O7cxnccaSJfPYLfA3cA1Z62LyKNtDR3M5krvuLGH/fnp0O6+o4jzSDshcDMLgCKD3cbazO7wcyy\nzCxrx44dYUonIoeybtu33PVyAaP6dea2swf6jiPNxEeP4FTgIjPbBCwEzjCzF2qv5Jx7yjmX4ZzL\nSE1NDXdGEaml9GAlUzKzaduqBX+YmEYLTTITM8L+Tjrn7nTO9XLO9QWuAN53zk0Odw4RaTjnHL9e\nks+mkn08PjGNLu01yUwsUUkXkcN6/uMveSN/K7edM4jR/Y/2HUeamdcri51zHwIf+swgIoeW+9Uu\nfvvGas4c3IUbT+vvO46EgHoEIlKvXfvKuSkzh67tk5gzQZPMxCrda0hE6lRd7ZixKI+S0nKWTBlD\nhzaJviNJiKhHICJ1mvtBIX9fv4N7LhrCCb06+I4jIaRCICI/sHxDCXOWreeStJ5cOaqP7zgSYioE\nIvI92/aUcfPCXI7rkszvLhmmSWbigAqBiPxbRVU10+bncKCiiicmjaRNSw0jxgO9yyLybw++tZas\nL3fx2MQ0BnRJ9h1HwkQ9AhEB4O2Crfxp+UauHn0MF43o4TuOhJEKgYiwsWQft7+Yz4jeHbnr/ON9\nx5EwUyEQiXNlFVVMeSGbhARj7pVptGqhSWbijcYIROLc3a8UsG77tzx7zUn06qRJZuKRegQicWzx\np0UsztrM9NMH8JNBXXzHEU9UCETi1Oqv9/KbVwo4dcDR3HyWJpmJZyoEInFob1kFUzOz6dgmkUev\nSCNBN5OLaxojEIkzzjnueDGfol0HWHTDKaQkt/IdSTxTj0AkzjyzfCNvr9rGneMGk9G3s+84EgFU\nCETiSNamnTzw1lrOGdqV68b28x1HIoQKgUicKCk9yE3zc+jZqTUPXz5CN5OTf9MYgUgcqKp23Lww\nl937K3h56km0T9IkM/K/VAhE4sCjy9bzP4Xf8NBlwxnaQ5PMyPfp0JBIjPtgXTGPvV/I5SN7MeGk\n3r7jSARSIRCJYVt2H+CWRXkM7taO2RcP8x1HIpQKgUiMKq+sZmpmDlVVjnmTR5KUqJvJSd00RiAS\no+5/cw0ri3bz5OR0+qW09R1HIph6BCIx6LWVX/PcR5u4fmw/zh3W3XcciXAqBCIxprC4lJlL8hl5\nTCd+PW6w7zgSBVQIRGLI/vJKpmZmk5SYwNwr00lM0J+4HJ7GCERihHOOWUsL2FBcyl+vPZluHZJ8\nR5Ioof8uiMSI+Z98xdLcLdxy1kDGHpfiO45EERUCkRjw2eY9/OerqzltYCrTTh/gO45EGRUCkSi3\nZ38FUzKzSUluye9/fiJHaZIZOUJhLwRm1tvMPjCz1Wa2ysxuDncGkVhRXe24dXEe2/eWMXdSOp3b\ntvQdSaKQj8HiSuBXzrkcM2sHZJvZu8651R6yiES1P/7jC95bW8x/XjSUtD6dfMeRKBX2HoFzbqtz\nLif4+FtgDdAz3DlEot3Hn3/Dw++s5YLh3blq9DG+40gU8zpGYGZ9gTRghc8cItGmeG8Z0xfk0jel\nLQ9cNlyTzEiTeCsEZpYMLAFmOOf21vH6DWaWZWZZO3bsCH9AkQhVWVXN9AW57DtYyZOTR5LcSpcD\nSdN4KQRmlkigCGQ6516uax3n3FPOuQznXEZqamp4A4pEsEfeXc+KjTv53SXDGNi1ne84EgN8nDVk\nwDPAGufcnHBvXySaLVu9nXkffs6VJ/fh0vRevuNIjPDRIzgV+D/AGWaWF/w6z0MOkahStHM/ty7O\nY1jP9tx9wRDfcSSGhP3gonNuOaCRLZEjUFZRxZTMbADmTdIkM9K8NMokEgXue301BVv28vRVGfTu\n3MZ3HIkxusWESIR7OWcz81d8xY0/7s9Ph3T1HUdikAqBSARbt+1bZi0t4OR+nbnt7IG+40iMUiEQ\niVClByuZkplNclILHr8yjRaaZEZCRHuWSARyzvHrJflsKtnH4xPT6NJOk8xI6KgQiESg5z/+kjfy\nt3L7OYM55dijfceRGKdCIBJhcr/axW/fWM1Zx3fhl6cd6zuOxAEVApEIsmtfOTdl5tC1fRKPXK5J\nZiQ8dB2BSISornbMWJRHSWk5S6aMoUObRN+RJE6oRyASIeZ+UMjf1+/gnouGcEKvDr7jSBxRIRCJ\nAMs3lDBn2XouSevJlaP6+I4jcUaFQMSzrXsO8B8LczmuSzK/u2SYJpmRsFMhEPGooqqaafNzOVhR\nxbzJI2nTUsN2En7a60Q8evCttWR/uYvHJ6bRPzXZdxyJU+oRiHjydsFW/rR8I9eM6cuFI3r4jiNx\nTIVAxIONJfu4/cV8TuzdkbvOO953HIlzKgQiYVZWUcWUF7JJSDDmTkqnZQv9GYpfGiMQCbO7Xylg\n3fZvefaak+jZsbXvOCLqEYiE0+JPi1ictZnppw/gJ4O6+I4jAqgQiITN6q/38ptXCjh1wNHcfJYm\nmZHIoUIgEgZ7yyqYmplNxzaJPHpFGgm6mZxEEI0RiISYc447XsynaNcBFt1wCinJrXxHEvke9QhE\nQuyZ5Rt5e9U27hw3mIy+nX3HEfkBFQKREPp0007+6621nDu0G9eN7ec7jkidVAhEQqSk9CDT5ufQ\nu1NrHrp8uG4mJxFLhUAkBKqqHTcvzGX3/gqemDSS9kmaZEYilwaLRULg0WXr+Z/Cb3joZ8MZ0qO9\n7zgih6QegUgz+2BdMY+9X8iEjF5MyOjtO47IYakQiDSjLbsPcMuiPAZ3a8d944f5jiPSICoEIs2k\nvLKaqZk5VFU55k0eSVJigu9IIg2iMQKRZnL/m2tYWbSbJyen0y+lre84Ig3mpUdgZuea2TozKzSz\nmT4yiDSn11Z+zXMfbeL6sf04d1h333FEjkjYC4GZJQBzgXHAEGCimQ0Jdw6R5lJYXMrMJflkHNOJ\nX48b7DuOyBHz0SMYBRQ6575wzpUDC4HxHnKINNn+8kqmZmaTlJjAH65MJzFBw24SfXyMEfQEimo8\n3wycHIoNzVr6GZ9s3BmKf1oEgNKDlWzbW8YL151Mtw5JvuOINErEDhab2Q3ADQB9+vRp1L/Ro2Nr\njuua3JyxRH7g7CHdOHVAiu8YIo3moxBsAWpeZdMruOx7nHNPAU8BZGRkuMZs6KbTBzTmx0RE4oqP\nA5qfAseZWT8zawlcAbzqIYeIiOChR+CcqzSzacA7QALwZ+fcqnDnEBGRAC9jBM65N4E3fWxbRES+\nT+e6iYjEORUCEZE4p0IgIhLnVAhEROKcCoGISJwz5xp1rVZYmdkO4MtG/ngKUNKMcZpLpOaCyM2m\nXEdGuY5cpGZrbK5jnHOph1spKgpBU5hZlnMuw3eO2iI1F0RuNuU6Msp15CI1W6hz6dCQiEicUyEQ\nEYlz8VAInvIdoB6RmgsiN5tyHRnlOnKRmi2kuWJ+jEBERA4tHnoEIiJyCDFRCMzscjNbZWbVZpZR\n67U7zazQzNaZ2Tn1/Hw/M1sRXG9R8PbYzZ1xkZnlBb82mVlePettMrPPgutlNXeOerZ5r5ltqZHv\nvHrWOzfYjoVmNjMMuR42s7Vmlm9mS82sYz3rhaXNDvf7m1mr4PtcGNyf+oYqS41t9jazD8xsdfBv\n4OY61vmJme2p8f7eHepcwe0e8n2xgMeC7ZVvZulhyjWoRlvkmdleM5tRa52wtJmZ/dnMis2soMay\nzmb2rpltCH7vVM/PXh1cZ4OZXd2kIM65qP8CjgcGAR8CGTWWDwFWAq2AfsDnQEIdP78YuCL4+Elg\nSojzPgLcXc9rm4CUMLffvcBth1knIdh+xwItg+06JMS5zgZaBB8/CDzoq80a8vsDU4Eng4+vABaF\n4b3rDqQHH7cD1teR6yfA6+HcpxryvgDnAW8BBpwCrPCQMQHYRuB8+7C3GXAakA4U1Fj2EDAz+Hhm\nXfs90Bn4Ivi9U/Bxp8bmiIkegXNujXNuXR0vjQcWOucOOuc2AoXAqJormJkBZwAvBRf9Bbg4VFmD\n25sALAjVNkJkFFDonPvCOVcOLCTQviHjnPubc64y+PRfBGaz86Uhv/94AvsPBPanM4Pvd8g457Y6\n53KCj78F1hCYFzwajAeedwH/AjqaWfcwZzgT+Nw519gLVpvEOfcPoPbE6jX3o/o+j84B3nXO7XTO\n7QLeBc5tbI6YKASH0BMoqvF8Mz/8Izka2F3jA6eudZrTj4DtzrkN9bzugL+ZWXZw3uZwmRbsnv+5\nnq5oQ9oylK4l8L/HuoSjzRry+/97neD+tIfA/hUWwUNRacCKOl4ebWYrzewtMxsapkiHe19871MQ\n6LnV958yH20G0NU5tzX4eBvQtY51mrXtInby+trMbBnQrY6XZjnnXgl3nro0MONEDt0bGOuc22Jm\nXYB3zWxt8H8NIcsGzANmE/jDnU3g0NW1Td1mU3N912ZmNguoBDLr+WdC0mbRxMySgSXADOfc3lov\n5xA49FEaHP/5f8BxYYgV0e9LcCzwIuDOOl721Wbf45xzZhbyUzujphA4585qxI9tAXrXeN4ruKym\nbwh0SVsE/xdX1zrNktHMWgCXAiMP8W9sCX4vNrOlBA5JNPmPp6HtZ2ZPA6/X8VJD2rLZc5nZNcAF\nwJkueHC0jn8jJG1WS0N+/+/W2Rx8rzsQ2L9CyswSCRSBTOfcy7Vfr1kYnHNvmtkTZpbinAvpPXUa\n8L6EZJ86AuOAHOfc9tov+GqzoO1m1t05tzV4qKy4jnW2EBjH+E4vAmOkjRLrh4ZeBa4Ins3Rj0BF\n/6TmCsEPlw+AnwUXXQ2EqodxFrDWObe5rhfNrK2ZtfvuMYHB0oK61m1OtY7LXlLPNj8FjrPAGVYt\nCXSpXw1xrnOBO4CLnHP761knXG3WkN//VQL7DwT2p/frK17NJTgG8Qywxjk3p551un03VmFmowj8\n3Ye0QDXwfXkVuCp49tApwJ4ah0TCod7euY82q6HmflTf59E7wNlm1il4KPfs4LLGCfWoeDi+CHx4\nbQYOAtuBd2q8NovA2R7rgHE1lr8J9Ag+PpZAgSgEXgRahSjnc8CNtZb1AN6skWNl8GsVgcMj4Wi/\nvwKfAfnBnbB77WzB5+cROCvl83BkC74fRUBe8OvJ2rnC2WZ1/f7AfQQKFUBScP8pDO5Px4ahjcYS\nOKSXX6OdzgNu/G5fA6YF22YlgUH3MWHIVef7UiuXAXOD7fkZNc74C0O+tgQ+2DvUWBb2NiNQiLYC\nFcHPsOsIjCu9B2wAlgGdg+tmAH+q8bPXBve1QuAXTcmhK4tFROJcrB8aEhGRw1AhEBGJcyoEIiJx\nToVARCTOqRCIiMQ5FQKRI2RmH/nOINKcdPqoiEicU49A5AiZWWnw+0/M7EMze8kC8yZkhvpuoyKh\noEIg0jRpwAwCc18cC5zqN47IkVMhEGmaT5xzm51z1QRu79DXcx6RI6ZCINI0B2s8riKK7ugr8h0V\nAhGROKdCICIS53T6qIhInFOPQEQkzqkQiIjEORUCEZE4p0IgIhLnVAhEROKcCoGISJxTIRARiXMq\nBCIice7/A6j1BZEFNGf9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1087cd0b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores_p = np.arange(-10, 10, 0.1)\n",
    "out_p = np.maximum(0,scores_p)\n",
    "plt.plot(scores_p, out_p)\n",
    "plt.xlabel('in')\n",
    "plt.ylabel('out')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forward pass: (3000, 100)\n",
      "Backward pass: (3000, 100)\n"
     ]
    }
   ],
   "source": [
    "class ReLu:\n",
    "    def forward(self,scores):\n",
    "        #scores: N x K\n",
    "        out = scores * (scores > 0)\n",
    "        return out\n",
    "    def backward(self, scores, previous):\n",
    "        dscores = 1.0 * (scores > 0) * previous\n",
    "        return(dscores)\n",
    "\n",
    "#test\n",
    "relu = ReLu()\n",
    "out = relu.forward(scores)\n",
    "dscores = relu.backward(scores,previous)\n",
    "print('Forward pass: {}'.format(out.shape))\n",
    "print('Backward pass: {}'.format(dscores.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Swish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Forward Pass\n",
    "\n",
    "$$out = scores \\cdot \\frac{1}{1 + e^{-scores}}$$\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "$$\\frac{\\partial previous}{\\partial scores} = \\frac{\\partial previous}{\\partial out} \\cdot \\frac{\\partial out}{\\partial scores}\n",
    "\\\\\n",
    "\\frac{\\partial out}{\\partial scores} =scores \\cdot \\frac{\\partial \\sigma(scores)}{\\partial scores} + \\sigma(scores) \\cdot 1\\\\\n",
    "= scores  \\cdot \\sigma(scores) \\cdot (1-\\sigma(scores)) + \\sigma(scores) \\\\\n",
    "= scores \\cdot \\sigma(scores) - scores \\cdot (\\sigma(scores))^2 + \\sigma(scores) \\\\\n",
    "= out + \\sigma(scores) \\cdot (1-out)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'out')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VOXd//H3NwlZWBK2AAGURQER\nZI2IaJUqLhX3rVht3SoFpC5VW9Q+PlYff3W32se11bqwuYFSlweXurUWBcK+71tYggkkQEK2+/fH\nDDbEBBLIzD3L53VduXLmzJk5n5yZzHfuc59zH3POISIi8SvBdwAREfFLhUBEJM6pEIiIxDkVAhGR\nOKdCICIS51QIRETinAqBiEicUyEQEYlzKgQiInEuyXeAumjdurXr3Lmz7xgiIlFl9uzZ251zmQdb\nLioKQefOnZk1a5bvGCIiUcXM1tVlOe0aEhGJcyoEIiJxToVARCTOhawQmNlLZrbNzBZWmdfSzD42\nsxXB3y1CtX4REambULYIXgbOrjZvHPCpc64b8GnwtoiIeBSyQuCc+xLIrzb7AuCV4PQrwIWhWr+I\niNRNuPsI2jrnNgentwBta1vQzEaa2Swzm5WXlxeedCIicchbZ7ELXCOz1utkOudecM5lO+eyMzMP\nej6EiEhM+W7XXu77+2KKSytCvq5wF4KtZpYFEPy9LczrFxGJeOUVlfx60hwmfLOOtd/tDvn6wl0I\npgFXB6evBt4N8/pFRCLeI9OX8fWq73jgouPomZUe8vWF8vDRScC/gR5mttHMrgceBM4wsxXAsOBt\nEREJ+mDBZp7/cjVXDT6SSwd2DMs6QzbWkHPuilruOj1U6xQRiWYrthZxx5vz6H9kc+45t1fY1qsz\ni0VEIkBRSRm/em02acmJPHvlQJKTwvfxHBWjj4qIxDLnHLe/OY91+XuY8MsTaJeRGtb1q0UgIuLZ\ns1+sYvqirdz5k2MY3LVV2NevQiAi4tFXK/J4dPoyzuvbnutP7uIlgwqBiIgnGwv2cNOkOXRr04yH\nLjkOM/OSQ4VARMSDkrIKRo2fTXmF47mfD6Rxsr8uW3UWi4iEmXOO/3pnIQs3FfLXX2TTpXUTr3nU\nIhARCbOJ367nzdkbuem0oxl2bK1jb4aNCoGISBjlrC/g3mmLOLV7JjcP6+47DqBCICISNnlFexkz\nPod2Gak8OaIfiQl+OoerUx+BiEgYBEYUzaFgTylTxgyheeNk35G+p0IgIhIGD/3fUmaszufxy/vS\nq32G7zj70a4hEZEQ+/u8XP7y1RquPrETFw8Iz4ii9aFCICISQsu3FvG7t+czsFML7h5+rO84NVIh\nEBEJkcLgiKJNUpJ45soBYR1RtD4iM5WISJSrrHT85vV5bMjfw9M/G0Db9PCOKFofKgQiIiHwzOcr\n+WTJVu4e3pNBXVr6jnNAKgQiIg3si+V5PPbxci7o155rhnT2HeegVAhERBrQhvzAiKI92jbjjxf7\nG1G0PlQIREQaSElZBb96bTbOOZ73PKJofURHShGRCOec466pC1i8uZCXrsmmUyu/I4rWh1oEIiIN\nYPyMdUzJ2cQtw7px2jH+RxStDxUCEZHDNHtdAfe9t5jTjmnDTad18x2n3lQIREQOw7aiEsZMmE1W\nRhpPXN6PhAgZUbQ+1EcgInKIyioqGTthDjuLy5g6ZhAZjRv5jnRIVAhERA7RHz9Yyrdr83lyRD96\nZqX7jnPItGtIROQQvDt3Ey/9aw3XDOnMBf06+I5zWFQIRETqaemWQsa9vYBBnVty9/CevuMcNhUC\nEZF62FkcGFG0WWoS/3tlfxolRv/HqJe/wMxuNbNFZrbQzCaZWeQOyyciEhQYUXQumwqKefaqAbRp\nFhsfXWEvBGbWAbgJyHbO9QYSgRHhziEiUl9//sdKPl26jXvOO5aBnSJ7RNH68NWmSQLSzCwJaAzk\nesohIlInny3bxp8+Xc7F/Tvw88GdfMdpUGEvBM65TcCjwHpgM7DTOfdRuHOIiNTVuu92c/OkORzT\nLp0HLoqOEUXrw8euoRbABUAXoD3QxMyuqmG5kWY2y8xm5eXlhTumiAgAxaUVjBqfg5nx/FUDSUtO\n9B2pwfnYNTQMWOOcy3POlQFTgCHVF3LOveCcy3bOZWdmZoY9pIjIvhFFl24p5MkR/TiyVWPfkULC\nRyFYDww2s8YWaF+dDizxkENE5IBe/fc6ps7ZxK3DujO0RxvfcULGRx/BN8BbQA6wIJjhhXDnEBE5\nkJlr87n/vcUM69mGsT8+2neckPIy1pBz7r+B//axbhGRg9lWWMKYCTl0bJHGY1E6omh9aNA5EZEq\nyioquXFiDrtKyhl//QlkpEXniKL1oUIgIlLFA+8vYebaAp66oj892jXzHScson+QDBGRBvLu3E28\n/PVarjupC+f3be87TtioEIiIsP+Ioneec4zvOGGlQiAica+wpIxRMTaiaH2oj0BE4lpgRNF5bCwo\nZtLIwTEzomh9xFfZExGp5tkvVvHJkq3cPbwnx3eOnRFF60OFQETi1pfL83j0o2Vc0K891wzp7DuO\nNyoEIhKXNhbs4ebJc+jephl/vDj2RhStDxUCEYk7JWUVjB6fQ3mF47mfD6Rxcnx3l8b3Xy8icene\naYtYsGknL/x8IF1aN/Edxzu1CEQkrkz+dj2TZ27gxh8fxZm92vmOExFUCEQkbszfuIN7pi3iR91a\n85szeviOEzFUCEQkLuTvLmX0+Bwym6bw5Ij+JMb4iKL1oT4CEYl5FZWOmyfPIa9oL2+NPpGWTZJ9\nR4ooKgQiEvMe/3gZX63YzoMXH0efjs19x4k42jUkIjHto0VbePqzVYw4/ghGDDrSd5yIpEIgIjFr\nzfbd3PbGPI7rkMG95/fyHSdiqRCISEzaU1rOqNdmk5hoPHvVAFIbJfqOFLHURyAiMcc5x7i3F7B8\nWxGvXDuIji0a+44U0dQiEJGY8/LXa5k2L5fbz+zBKd0zfceJeCoEIhJTZq7N54H3lzCsZ1tGn3qU\n7zhRQYVARGLGtsISxkzIoWOLNB67vC8JOmmsTtRHICIxoayikhsn5rCrpJzXrh9ERloj35GihgqB\niMSEP36wlJlrC3hyRD+OaZfuO05U0a4hEYl60+bl8tK/1nDNkM5c0K+D7zhRR4VARKLasi1F/O6t\n+WR3asFd5/T0HScqqRCISNQqLClj1PjZNE1N4ukrB5CcpI+0Q6E+AhGJSpWVjtvemMf6/D1MumEw\nbdNTfUeKWiqfIhKVnvtyFR8v3spd5/RkUJeWvuNENS+FwMyam9lbZrbUzJaY2Yk+cohIdPrniu08\nOn0Z5/bJ4rqTOvuOE/V87Rp6Evg/59ylZpYMaCAQEamTTTuK+fWkHI5u05SHLumDmU4aO1xhLwRm\nlgGcAlwD4JwrBUrDnUNEok9JWQWjx8+mrMLx3FUDaZKibs6G4GPXUBcgD/ibmc0xs7+aWRMPOUQk\nyvzh74uYv3Enj17Wl66ZTX3HiRk+CkESMAB41jnXH9gNjKu+kJmNNLNZZjYrLy8v3BlFJMK8MWsD\nk77dwKhTj+Ls3u18x4kpPgrBRmCjc+6b4O23CBSG/TjnXnDOZTvnsjMzNYysSDxblLuT/3pnISd2\nbcXtZ3b3HSfmhL0QOOe2ABvMrEdw1unA4nDnEJHosLO4jNHjc2jeuBFPXdGfpEQd9d7QfPW0/BqY\nEDxiaDVwraccIhLB9p00lrujmMkjB5PZLMV3pJjkpRA45+YC2T7WLSLR4/kvV/PJkq3cc+6xZHfW\nSWOhojaWiESkr1dt55HpSxneJ4trddJYSKkQiEjE2VpYwk2T5tCldROdNBYGOhtDRCJKWUUlN07I\nYU9pBZNuGExTnTQWctrCIhJRHvxwKbPWBa401q1tM99x4oJ2DYlIxPhgwWZe/KeuNBZuKgQiEhFW\n5e3ijjfn0f/I5rrSWJipEIiId7v3ljPqtdmkNErkGV1pLOzURyAiXjnnuGvqAlbm7eK1604gKyPN\nd6S4o7IrIl6Nn7GOd+fmctsZ3Tm5W2vfceKSCoGIeDNnfQH3vbeY045pw5ihR/uOE7dUCETEi/zd\npdw4IYe26ak8cXk/EhJ00pgv6iMQkbCrqHTcPHkO23eXMmX0EDIaN/IdKa7VqUVgZj8Y8q+meSIi\ndfHkpyv4asV27ju/F707ZPiOE/fqumvo33WcJyJyQJ8t28ZTn67gsoEd+enxR/iOIxxk15CZtQM6\nAGlm1h/YtxMvHWgc4mwiEmM25O/h1tfn0jMrnfsv7K3B5CLEwfoIzgKuAToCj1eZXwTcFaJMIhKD\nSsoqGDMhh4pKx3NXDSC1UaLvSBJ0wELgnHsFeMXMLnHOvR2mTCISg+57bzELNu3kL7/IplOrJr7j\nSBV1PWqot5n1qj7TOXdfA+cRkRj0zpxNTPxmPaOHHsUZx7b1HUeqqWsh2FVlOhU4F1jS8HFEJNas\n3FbEXVMXMKhLS247o7vvOFKDOhUC59xjVW+b2aPA9JAkEpGYsae0nNHjc2icnMifr+hPUqLOYY1E\nh3pCWWMCHcgiIjVyzvH7qQtZmbeL8defQNv0VN+RpBZ1KgRmtgBwwZsJQBvg/lCFEpHo98asDUyZ\ns4lbh3XnpKM1mFwkq2uL4FygBfAjoDnwgXNudshSiUhUW5xbyD3vLuJH3Voz9jQNJhfp6rrD7gLg\nNaA10Aj4m5n9OmSpRCRqFZWUcePEHJo3bsQTP+1HogaTi3h1bRH8EhjsnNsNYGYPERhi4s+hCiYi\n0cc5x7i3F7A+fw+TbhhM66Yakiwa1LVFYEBFldsV/Ge4CRERAF799zreX7CZO87qwaAuLX3HkTqq\na4vgb8A3ZjY1ePtC4MXQRBKRaDRvww7+5/3FDOvZhpE/6uo7jtRDXc8jeNzMPgdODs661jk3J2Sp\nRCSq7NhTypgJObRplsqjl/XVRWaiTJ3PI3DO5QA5IcwiIlGostJx2xvz2FZUwpujhtC8cbLvSFJP\nOs1PRA7LX75azadLt3H3OT3pd0Rz33HkEHgrBGaWaGZzzOw9XxlE5PDMXJvPw9OXcc5x7bh6SGff\nceQQ+WwR3IwGrhOJWt/t2svYiTkc0SKNBy/po4vMRDEvhcDMOgLDgb/6WL+IHJ7KSsctr8+lYE8Z\nT185gPRUXXw+mvlqEfwJ+C1Q6Wn9InIYnv1iFV+t2M695/WiV3tdfD7ahb0QmNm5wLaDjVVkZiPN\nbJaZzcrLywtTOhE5mG/X5PPYR8s4v297rhiki8/HAh8tgpOA881sLTAZOM3MxldfyDn3gnMu2zmX\nnZmZGe6MIlKD/N2l3DRpDke2bMwDF+ni87Ei7IXAOXenc66jc64zMAL4h3PuqnDnEJH6cc5x+5vz\nyN9dyv/+bADN1C8QM3QegYjUyYv/XMM/lm7j7uE96d1B/QKx5FCvUNYgnHOfA5/7zCAiBzd3ww4e\n/HApZ/Vqyy9O7OQ7jjQwtQhE5IB2FpcxdmIObdNTefiSvuoXiEFeWwQiEtkC1xeYz5adJbwx6kQy\nGqtfIBapRSAitRo/Yx0fLtzCHWf1YMCRLXzHkRBRIRCRGi3K3cn97y1haI9MbtD1BWKaCoGI/MCu\nveWMnTiHFk0a8ZiuLxDz1EcgIvtxzvH7qQtY991uJt4wmFa67nDMU4tARPbz5uyNvDM3l1uGdWdw\n11a+40gYqBCIyPdWbC3inncXMuSoVtz446N9x5EwUSEQEQBKyioYO3EOTVOS+NNP+5GofoG4oT4C\nEQHggfeXsGxrES9fezxt0lN9x5EwUotARPho0RZem7GOX57chaE92viOI2GmQiAS5zbvLOa3b8+n\nd4d07ji7h+844oEKgUgcq6h03Pr6XErLK3lqRH9SkhJ9RxIP1EcgEsee+2IVM1bn8/Clfeia2dR3\nHPFELQKROJWzvoDHP17OuX2yuGxgR99xxCMVApE4VFhSxs2T59AuPZUHLjpOQ0vHOe0aEokzgSEk\nFpK7o4Q3fjWYjDQNLR3v1CIQiTNv52xi2rxcbj69GwM7tfQdRyKACoFIHFmzfTf3vLuQQV1aaggJ\n+Z4KgUicKC2v5KZJc2iUmKAhJGQ/6iMQiROPfbSMBZt28txVA2nfPM13HIkgahGIxIEvl+fx/Jer\n+dkJR3J273a+40iEUSEQiXHbd+3lN2/Mo1ubpvzX8GN9x5EIpF1DIjHMOccdb86jsKSM164fRFqy\nhpCQH1KLQCSG/e1fa/lsWR53n9OTnlnpvuNIhFIhEIlRi3J38uCHSxnWsw2/OLGT7zgSwVQIRGLQ\nntJybpo0h+aNG/HwpX01hIQckPoIRGLQ/e8tZvX23Yy//gRaNkn2HUcinFoEIjHmwwWbmfTtBkae\n0pWTjm7tO45EARUCkRiSu6OYcVMW0KdjBredoauNSd2EvRCY2RFm9pmZLTazRWZ2c7gziMSiikrH\nLa/PpawicLWx5CR9z5O68dFHUA7c5pzLMbNmwGwz+9g5t9hDFpGY8eznK/l2TT6PXtaXzq2b+I4j\nUSTsXxmcc5udcznB6SJgCdAh3DlEYsnsdQU88ckKzuvbnksG6N9J6sdr29HMOgP9gW985hCJZvuu\nNpaVkcoDF/XWoaJSb94KgZk1Bd4GbnHOFdZw/0gzm2Vms/Ly8sIfUCRK3PPOQnJ3FPPkiH6kp+pq\nY1J/XgqBmTUiUAQmOOem1LSMc+4F51y2cy47MzMzvAFFosTUORt5Z24uN5/eXVcbk0Pm46ghA14E\nljjnHg/3+kVixbrvdvP7qQsZ1LklY0/T1cbk0PloEZwE/Bw4zczmBn/O8ZBDJGqVVVRy0+S5JCQY\nT4zQ1cbk8IT98FHn3D8BvWtFDsOfPlnOvA07ePpnA+igq43JYdIZJyJR5utV23nm81X8NPsIhvfJ\n8h1HYoAKgUgUKdhdym9en0eXVk245zxdbUwahgqBSJRwzjFuyny+272Xp67oT5MUDR4sDUOFQCRK\njJ+xjumLtnLHWT3o3SHDdxyJISoEIlFg4aad3P/eEob2yOSXJ3f1HUdijAqBSIQrKilj7MQcWjZJ\n5vHL+5GgQ0WlgWkno0gEc85x55QFbCgoZtINg3W1MQkJtQhEItjEb9fz3vzN/OaM7gzqoiEkJDRU\nCEQi1OLcQv7w98Wc0j2T0ace5TuOxDAVApEItGtvOWMn5tA8rRGPX95X/QISUuojEIkwzjnunrqA\ntd/tZuINg2ndNMV3JIlxahGIRJiJ367n3bm53DqsO4O7tvIdR+KACoFIBJm9roB7py3i1O6ZjPmx\nhpaW8FAhEIkQ24pKGDNhNlkZaTw1or+GlpawUR+BSAQoLa/kxgk5FBaXM2XMIDIa65KTEj4qBCIR\n4H/eX8zMtQU8dUV/emal+44jcUa7hkQ8e2v2Rl799zp+eXIXzu/b3ncciUMqBCIezV5XwF1TF3Bi\n11aM+8kxvuNInFIhEPFkQ/4eRr46i6yMVJ6+cgBJifp3FD/0zhPxoLCkjOtenklZRSUvXXO8BpMT\nr9RZLBJmZRWBI4TWbN/Nq9cP4qjMpr4jSZxTIRAJo8pKxx1vzuOrFdt5+NI+DDmqte9IIto1JBIu\nzjnuf38x78zN5Y6zenB59hG+I4kAKgQiYfPM56v427/Wct1JXRgzVMNKS+RQIRAJg798uZpHpi/j\nwn7t+f3wnphp+AiJHOojEAmxF75cxf/7YCnD+2TxyGW6toBEHhUCkRB67otVPPjhUs7r254nLu+r\ncwUkIqkQiIRAZaXjoelLef6L1Zzftz2PqwhIBFMhEGlge8sruOPN+Uybl8tVg4/kD+f31pDSEtFU\nCEQaUP7uUsZMmM2M1fn87uxjGHVqV3UMS8Tz0lY1s7PNbJmZrTSzcT4yiDS02esKGP7UV+Ss28Gf\nftqP0UOPUhGQqBD2FoGZJQJPA2cAG4GZZjbNObc43FlEGoJzjhf/uYYHP1xK++ZpTBkzhN4dMnzH\nEqkzH7uGBgErnXOrAcxsMnAB0OCFYPnWInYWl7HvO1ngy5l9P/2f+YZ9fz8YRtUvcjXNDzy+pucK\nLvn9Y/j+W2Gd1mH2/XMlmJGQAIlmJCUkBKYTLPBjgd/6xunXqrxd3DVlAd+syefMY9vyyGV9yUjT\n1cUkuvgoBB2ADVVubwROOOAjli2DoUPrvaLKLUVU7Cmt9+OizffFw/5TeGq8TbC4WGBeggVu7z+9\n/zL7Ck9ClenEYHHad7vq88eLSufI3VHC9h3F3G7QqVUTMmekYC/5TiZSfxHbWWxmI4GRAH1SUg7p\nOY5okUa7jNQf3uHAVZ9R0639Z9eydJX5B3igq+VGbc/1n+dzuOC0C86s8XbV5V1ttwPTlc5R6aCi\n0lHmHJXV5rvgvDqzQAulUaKRlGAkJSaQlBhoxew3L8FITkogOTEhao+iqXSObUV72bSjmLLySlo2\nTaFzq8Yk69BQiWI+CsEmoOpoWx2D8/bjnHsBeAEgOzvb8fnn9V5Rk0PLJ0FlFZXs2VvBrtJydu8t\nZ9fe8sDtvYHbu0v/M6+opIyCPWUU7CkN/OwOTO8prajxuZulJNEmPYW26am0S0+lTXoqbdNTaJee\nSruMVDo0T6N105SIOQt3W2EJb+VsZMKM9WzaUczxnVtw6xnd6a7RQyWS1XHXsY9CMBPoZmZdCBSA\nEcDPPOSQg2iUmEBG4wQyGh/6Pu+Ssgp2BAvEd7tK2VZUwtbCvWwtLPn+55s1+WwrKqGsYv9WSKNE\no11GKlkZabTPSKV98zSymqfRofm+eWmkpyWFrJ9ka2EJXy7PY/qiLXy2LI+KSscJXVry4CXHcfLR\nrdU/IzEj7IXAOVduZmOB6UAi8JJzblG4c0h4pDZKpF1GYs276KqorHQU7CllS2EJm3eUsHlnMbk7\nS8jdUczmHSXMXFvA1sLNlFfuXyyaJCeS1TyN9s0DxSIrI432zQNFo33zNNqlp5LaKOGAH9rFpRVs\n37WXDQV7WLaliGVbipi7YQdLtxQB0KZZCiNP6crl2UfQpbXamRJ7vPQROOc+AD7wsW6JTAkJRqum\nKbRqmkKv9jUfellR6di+K7B/fl+x2Dedu7OYxbmFbN+19wePS0ww0holkpacSFqjRJISjNKKSsoq\nKtkd3NVVVcsmyfRqn86F/TtwSrdMjmnXLGJ2UYmEQsR2FotUl5hgtE1PpW16KhxZ8zJ7yyvYsrOE\n3B2B1sSWwhL2lJZTXFpJcVk5xaUVlFU6UhITaJSYQFpyIpnNUshslkJWRio92jUjs2mKdvtIXFEh\nkJiSkpRIp1ZN6NRKu3BE6krHvImIxDkVAhGROKdCICIS51QIRETinAqBiEicUyEQEYlzKgQiInFO\nhUBEJM6Zq89ww56YWR6w7hAf3hrY3oBxGkqk5oLIzaZc9aNc9Rep2Q41VyfnXObBFoqKQnA4zGyW\ncy7bd47qIjUXRG425aof5aq/SM0W6lzaNSQiEudUCERE4lw8FIIXfAeoRaTmgsjNplz1o1z1F6nZ\nQpor5vsIRETkwOKhRSAiIgcQE4XAzC4zs0VmVmlm2dXuu9PMVprZMjM7q5bHdzGzb4LLvW5mySHI\n+LqZzQ3+rDWzubUst9bMFgSXm9XQOWpZ571mtqlKvnNqWe7s4HZcaWbjwpDrETNbambzzWyqmTWv\nZbmwbLOD/f1mlhJ8nVcG30+dQ5WlyjqPMLPPzGxx8H/g5hqWGWpmO6u8vveEOldwvQd8XSzgqeD2\nmm9mA8KUq0eVbTHXzArN7JZqy4Rlm5nZS2a2zcwWVpnX0sw+NrMVwd8tanns1cFlVpjZ1YcVxDkX\n9T9AT6AH8DmQXWX+scA8IAXoAqwCEmt4/BvAiOD0c8DoEOd9DLinlvvWAq3DvP3uBW4/yDKJwe3X\nFUgObtdjQ5zrTCApOP0Q8JCvbVaXvx8YAzwXnB4BvB6G1y4LGBCcbgYsryHXUOC9cL6n6vK6AOcA\nHwIGDAa+8ZAxEdhC4Hj7sG8z4BRgALCwyryHgXHB6XE1ve+BlsDq4O8WwekWh5ojJloEzrklzrll\nNdx1ATDZObfXObcGWAkMqrqABa5JeBrwVnDWK8CFocoaXN/lwKRQrSNEBgErnXOrnXOlwGQC2zdk\nnHMfOef2XVB4BtAxlOs7iLr8/RcQeP9A4P10uoX4mpfOuc3OuZzgdBGwBOgQynU2oAuAV13ADKC5\nmWWFOcPpwCrn3KGesHpYnHNfAvnVZld9H9X2eXQW8LFzLt85VwB8DJx9qDliohAcQAdgQ5XbG/nh\nP0krYEeVD5yalmlIPwK2OudW1HK/Az4ys9lmNjKEOaobG2yev1RLU7Qu2zKUriPw7bEm4dhmdfn7\nv18m+H7aSeD9FRbBXVH9gW9quPtEM5tnZh+aWa8wRTrY6+L7PQWBllttX8p8bDOAts65zcHpLUDb\nGpZp0G0XNdcsNrNPgHY13HW3c+7dcOepSR0zXsGBWwMnO+c2mVkb4GMzWxr81hCybMCzwP0E/nHv\nJ7Dr6rrDXefh5tq3zczsbqAcmFDL04Rkm0UTM2sKvA3c4pwrrHZ3DoFdH7uC/T/vAN3CECuiX5dg\nX+D5wJ013O1rm+3HOefMLOSHdkZNIXDODTuEh20Cjqhyu2NwXlXfEWiSJgW/xdW0TINkNLMk4GJg\n4AGeY1Pw9zYzm0pgl8Rh//PUdfuZ2V+A92q4qy7bssFzmdk1wLnA6S64c7SG5wjJNqumLn//vmU2\nBl/rDALvr5Ays0YEisAE59yU6vdXLQzOuQ/M7Bkza+2cC+mYOnV4XULynqqHnwA5zrmt1e/wtc2C\ntppZlnNuc3BX2bYaltlEoB9jn44E+kgPSazvGpoGjAgezdGFQEX/tuoCwQ+Xz4BLg7OuBkLVwhgG\nLHXObazpTjNrYmbN9k0T6CxdWNOyDanaftmLalnnTKCbBY6wSibQpJ4W4lxnA78FznfO7allmXBt\ns7r8/dMIvH8g8H76R23Fq6EE+yBeBJY45x6vZZl2+/oqzGwQgf/7kBaoOr4u04BfBI8eGgzsrLJL\nJBxqbZ372GZVVH0f1fZ5NB0408xaBHflnhmcd2hC3Ssejh8CH14bgb3AVmB6lfvuJnC0xzLgJ1Xm\nfwC0D053JVAgVgJvAikhyvlCx7/UAAABdklEQVQyMKravPbAB1VyzAv+LCKweyQc2+81YAEwP/gm\nzKqeLXj7HAJHpawKR7bg67EBmBv8ea56rnBus5r+fuA+AoUKIDX4/lkZfD91DcM2OpnALr35VbbT\nOcCofe81YGxw28wj0Ok+JAy5anxdquUy4Ong9lxAlSP+wpCvCYEP9owq88K+zQgUos1AWfAz7HoC\n/UqfAiuAT4CWwWWzgb9Weex1wffaSuDaw8mhM4tFROJcrO8aEhGRg1AhEBGJcyoEIiJxToVARCTO\nqRCIiMQ5FQKRejKzr31nEGlIOnxURCTOqUUgUk9mtiv4e6iZfW5mb1ngugkTQj3aqEgoqBCIHJ7+\nwC0Ern3RFTjJbxyR+lMhEDk83zrnNjrnKgkM79DZcx6RelMhEDk8e6tMVxBFI/qK7KNCICIS51QI\nRETinA4fFRGJc2oRiIjEORUCEZE4p0IgIhLnVAhEROKcCoGISJxTIRARiXMqBCIicU6FQEQkzv1/\nstZg089/ahYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a142198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores_p = np.arange(-10, 10, 0.1)\n",
    "out_p = scores_p / (1. + np.exp(-scores_p))\n",
    "plt.plot(scores_p, out_p)\n",
    "plt.axhline(y=0, color='r', linestyle='-')\n",
    "plt.xlabel('in')\n",
    "plt.ylabel('out')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forward pass: (3000, 100)\n",
      "Backward pass: (3000, 100)\n"
     ]
    }
   ],
   "source": [
    "class Swish:\n",
    "    def sigmoid(self,scores):\n",
    "        out = 1.0 / (1.0 + np.exp(-scores))\n",
    "        return out\n",
    "\n",
    "    def forward(self,scores):\n",
    "        #scores: N x K\n",
    "        out = scores / (1.0 + np.exp(-scores))\n",
    "        return out\n",
    "    \n",
    "    def backward(self,scores,previous):\n",
    "        out = self.forward(scores)\n",
    "        dscores = (out + self.sigmoid(scores) * (1-out)) * previous\n",
    "        return(dscores)\n",
    "#test\n",
    "sw = Swish()\n",
    "out = sw.forward(scores)\n",
    "dscores = sw.backward(scores,previous)\n",
    "print('Forward pass: {}'.format(out.shape))\n",
    "print('Backward pass: {}'.format(dscores.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Randomly chose nodes of previous layer to be *turned off*. The scaler $\\frac{1}{1-p}$ is added to keep the expectation of activation output the same as when there is no Dropout.\n",
    "\n",
    "$$mask = \n",
    "\\begin{cases}\n",
    "\\frac{1}{1-p} & u >= p\\\\\n",
    "0 & u < p\n",
    "\\end{cases}$$\n",
    "\n",
    "where $u$ is a sample from uniform distribution of $[0,1]$ and $p$ is the threshold for percentage of nodes to be dropped.\n",
    "\n",
    "### Forward Pass\n",
    "\n",
    "\n",
    "$$out = mask \\cdot scores$$\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "$$\\frac{\\partial previous}{\\partial scores} = \\frac{\\partial previous}{\\partial out} \\cdot \\frac{\\partial out}{\\partial scores}\n",
    "\\\\\n",
    "\\frac{\\partial out}{\\partial scores} = mask$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forward pass: (3000, 100)\n",
      "Backward pass: (3000, 100)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 0.        ,  0.        ,  3.33333333, ...,  3.33333333,\n",
       "         3.33333333,  3.33333333],\n",
       "       [ 3.33333333,  3.33333333,  3.33333333, ...,  3.33333333,\n",
       "         3.33333333,  3.33333333],\n",
       "       [ 3.33333333,  3.33333333,  0.        , ...,  3.33333333,\n",
       "         0.        ,  0.        ],\n",
       "       ..., \n",
       "       [ 3.33333333,  0.        ,  3.33333333, ...,  0.        ,\n",
       "         3.33333333,  3.33333333],\n",
       "       [ 3.33333333,  3.33333333,  0.        , ...,  0.        ,\n",
       "         0.        ,  3.33333333],\n",
       "       [ 3.33333333,  3.33333333,  3.33333333, ...,  3.33333333,\n",
       "         3.33333333,  3.33333333]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Dropout:\n",
    "    def __init__(self,p,seed):\n",
    "        self.seed = seed\n",
    "        self.p = p\n",
    "    def mask(self,scores,scaling=True):\n",
    "        #if no dropout\n",
    "        if self.p == 1: return np.ones_like(scores)\n",
    "        \n",
    "        np.random.seed(seed=self.seed)\n",
    "        if (scaling): \n",
    "            scaler = 1.0/(1-self.p)\n",
    "        else:\n",
    "            scaler = 1.0\n",
    "        mask = scaler * np.random.binomial(n=1,p=self.p,size=scores.shape)\n",
    "        return(mask)\n",
    "    \n",
    "#test\n",
    "drop = Dropout(p=0.7,seed = 1412)\n",
    "mask = drop.mask(scores)\n",
    "print('Forward pass: {}'.format(mask.shape))\n",
    "print('Backward pass: {}'.format(mask.shape))\n",
    "mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch Normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch Normalization layer converts the activation to a gaussian distribution, then feed it to another linear layer (Norm > Linear). The process is redone every iteration, thus *batch* normalization. The benefits of using Linear > BatchNorm (Norm > Linear) > Activation > Dropout are:\n",
    "\n",
    "* Faster gradient flow\n",
    "* Allows higher learning rates\n",
    "* Reduce dependence on initialization\n",
    "* Regularization\n",
    "\n",
    "### Forward\n",
    "$$normalized_{i,j} = \\frac{scores_{i,j} - \\mu_j}{\\sqrt{{\\sigma_j}^2 + \\epsilon}}$$\n",
    "\n",
    "where $i$ is row of examples, $j$ is column of features and $\\epsilon$ is a small sclaer for numerical stability and regularization.\n",
    "\n",
    "$$out = c  + normalized \\cdot D$$\n",
    "\n",
    "where $c$ and $D$ are learned parameters in order to adjust *how normalized* the output needs to be.\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "#### Scores\n",
    "\n",
    "$$\\frac{\\partial previous}{\\partial scores} = \\frac{\\partial previous}{\\partial out} \\cdot \\frac{\\partial out}{\\partial normalized} \\cdot \\frac{\\partial normalized}{\\partial scores}\n",
    "\\\\\n",
    "\\frac{\\partial out}{\\partial normalized} = D\\\\\n",
    "\\frac{\\partial normalized}{\\partial scores} = \\frac{\\sqrt{{\\sigma_j}^2 + \\epsilon}(1-\\frac{\\partial \\mu_j}{scores_{i,j}}) - (scores_{i,j} - \\mu_j)(\\frac{1}{2})(\\frac{1}{\\sqrt{{\\sigma_j}^2 + \\epsilon}})(2){\\sigma_j}^2(\\frac{\\partial {\\sigma_j}^2}{\\partial scores_{i,j}})}{{\\sigma_j}^2 + \\epsilon}\\\\\n",
    "\\frac{\\partial \\mu_j}{\\partial scores_{i,j}} = \\frac{1}{N}\\frac{\\partial \\sum_{k=1}^{K} scores_{k,j}}{\\partial scores_{i,j}} = \\frac{1}{N}\\\\\n",
    "\\frac{\\partial {\\sigma_j}^2}{\\partial scores_{i,j}} = \\frac{1}{N}\\frac{\\partial \\sum_{k=1}^{K} (scores_{k,j} - \\mu_j)^2}{\\partial scores_{i,j}} = \\frac{2}{N}(scores_{i,j} - \\mu_j)(1-\\frac{1}{N}) \\\\\n",
    "\\frac{\\partial normalized}{\\partial scores} = (1-\\frac{1}{N})(\\sqrt{{\\sigma_j}^2 + \\epsilon} - \\frac{2\\sigma_j(scores_{i,j}-\\mu_j)^2}{N\\sqrt{{\\sigma_j}^2 + \\epsilon}})\n",
    "$$\n",
    "\n",
    "where N is the number of samples in the batch\n",
    "\n",
    "#### Linear parameters\n",
    "\n",
    "$$\n",
    "\\frac{\\partial previous}{\\partial param} = \\frac{\\partial previous}{\\partial out} \\cdot \\frac{\\partial out}{\\partial param}\\\\\n",
    "\\frac{\\partial out}{\\partial c} = 1\\\\\n",
    "\\frac{\\partial out}{\\partial D} = normalized\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Norm:\n",
    "    def forward(self,scores,epsilon = 1e-3):\n",
    "        self.mu = np.mean(scores,axis=0)\n",
    "        self.sd = np.sqrt(np.var(scores+epsilon,axis=0))\n",
    "        self.N = scores.shape[0]\n",
    "        normalized = (scores - self.mu) / self.sd\n",
    "        return(normalized)\n",
    "    def backward(self,scores,previous,epsilon = 1e-3):\n",
    "        out = self.forward(scores,epsilon)\n",
    "        dnormalized = (1-(1.0/N)) * (self.sd - (2 * np.sqrt(np.var(scores,axis=0)) * (scores - self.mu)**2 )/(self.N * self.sd))\n",
    "        dnormalized *= previous\n",
    "        return(dnormalized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variance of output [ 1.  1.]\n",
      "Shape of dnormalized:  (3000, 100)\n"
     ]
    }
   ],
   "source": [
    "norm = Norm()\n",
    "print('Variance of output',np.var(norm.forward(X),axis=0))\n",
    "print('Shape of dnormalized: ',norm.backward(scores,previous).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Assume cross-entropy loss.\n",
    "\n",
    "### Loss\n",
    "\n",
    "For each example, the loss is the negative log of predicted probability of the correct class\n",
    "$$L_i = -\\sum_{k=1}^{K} y_klog(p_k)$$\n",
    "\n",
    "### Forward Pass\n",
    "\n",
    "$$p_{class} = \\frac{e^{scores_{class}}}{\\sum_{k=1}^{K} e^{scores_{k}}}$$\n",
    "\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial scores_{class}} = \\frac{\\partial L}{\\partial p_{class}} \\cdot \\frac{\\partial p_{class}}{\\partial scores_{class}} = - \\frac{1}{p_{class}} \\cdot \\frac{\\partial p_{class}}{\\partial scores_{class}}\n",
    "\\\\\n",
    "\\frac{\\partial p_{class}}{\\partial scores_{class}} = \\frac{\\sum_{k=1}^{K} e^{scores_{k}}e^{scores_{class}} - e^{2scores_{class}}}{(\\sum_{k=1}^{K} e^{scores_{k}})^2} = p_{class} - p_{class}^2 = \n",
    "p_{class}(1-p_{class})\\\\\n",
    "\\frac{\\partial L}{\\partial scores_{class}} = p_{class} - 1 * y_{class}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forward pass: (3000, 100)\n",
      "Backward pass: (3000, 100)\n",
      "Loss: 5.005441414882651\n"
     ]
    }
   ],
   "source": [
    "class Softmax:\n",
    "    def forward(self,scores):\n",
    "        exp_scores = np.exp(scores)\n",
    "        probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)\n",
    "        return(probs)\n",
    "    \n",
    "    def backward(self,scores,y):\n",
    "        N = scores.shape[0]\n",
    "        dscores = self.forward(scores)\n",
    "        dscores[range(N), y] -= 1\n",
    "        dscores /= N\n",
    "        return(dscores)   \n",
    "    \n",
    "    def cross_entropy(self,probs,y):\n",
    "        #probs: N x K\n",
    "        N = probs.shape[0]\n",
    "        #print('All normalized probs:',probs.shape)\n",
    "\n",
    "        #normalized prob of correct label\n",
    "        correct_logprobs = -np.log(probs[range(N),y])\n",
    "        #print('-log(normalized probs) when label is correct:',correct_logprobs.shape)\n",
    "\n",
    "        #data loss\n",
    "        data_loss = np.sum(correct_logprobs) / N\n",
    "\n",
    "        return(data_loss)\n",
    "    \n",
    "#test\n",
    "soft = Softmax()\n",
    "probs = soft.forward(scores)\n",
    "dscores = soft.backward(scores,y)\n",
    "loss = soft.cross_entropy(probs,y)\n",
    "print('Forward pass: {}'.format(probs.shape))\n",
    "print('Backward pass: {}'.format(dscores.shape))\n",
    "print('Loss: {}'.format(loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hierarchical Softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Assume hinge loss.\n",
    "\n",
    "### Loss\n",
    "\n",
    "$$L_{i} = \\sum_{other \\ne correct}^{K} max(0,scores_{other} - scores_{correct} + \\Delta)$$\n",
    "\n",
    "### Forward Pass\n",
    "\n",
    "$$scores_k = (X \\cdot W + b)_k$$\n",
    "\n",
    "### Backward Pass\n",
    "\n",
    "Derivative of loss function by weight\n",
    "$$\\frac{dL}{dscores} = \n",
    "\\begin{cases}\n",
    "\\frac{dL}{dscores_{correct}} \\\\\n",
    "\\frac{dL}{dscores_{other}}\n",
    "\\end{cases}$$\n",
    "\n",
    "$$\\frac{dL}{dscores_{correct}} = \n",
    "\\begin{cases}\n",
    "\\sum_{other \\ne correct}^{K} (-1) & scores_{other} - scores_{correct} + \\Delta > 0\\\\\n",
    "0 & scores_{other} - scores_{correct} + \\Delta <= 0\n",
    "\\end{cases}$$\n",
    "\n",
    "$$\\frac{dL}{dscores_{other}} = \n",
    "\\begin{cases}\n",
    "1  & scores_{other} - scores_{correct} + \\Delta > 0 \\\\\n",
    "0 & scores_{other} - scores_{correct} + \\Delta <= 0\\\\\n",
    "\\end{cases}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Forward pass: (3000,)\n",
      "Backward pass: (3000, 100)\n",
      "Loss: 124.29518160523449\n"
     ]
    }
   ],
   "source": [
    "class SVM:\n",
    "    \n",
    "    def forward(self,scores):\n",
    "        return(scores)\n",
    "    \n",
    "    def forward_correct(self,scores,y):\n",
    "        N = scores.shape[0]\n",
    "        correct_scores = scores[range(N),y]\n",
    "        return(correct_scores)\n",
    "        \n",
    "    def backward(self,scores,y):\n",
    "        N = scores.shape[0]\n",
    "        correct_scores = self.forward_correct(scores,y)\n",
    "        # can broadcast by row not columns\n",
    "        margin = scores.T - correct_scores + delta \n",
    "        margin = margin.T\n",
    "        \n",
    "        #zero loss at the correct label\n",
    "        margin[range(N),y] = 0 \n",
    "        \n",
    "        #element-wise max\n",
    "        margin = np.maximum(np.zeros(margin.shape),margin) \n",
    "        \n",
    "        #binary to vectorize\n",
    "        binary = margin\n",
    "        #all derivatives whose scores are less than zero are zero\n",
    "        binary[margin>0] = 1\n",
    "        \n",
    "        #sum to deduct from those with correct class only; the rest are to be 1\n",
    "        col_sum = np.sum(binary,axis=1)\n",
    "        binary[range(N),y] -= col_sum[range(N)]\n",
    "        binary /= N\n",
    "        return(binary)\n",
    "        \n",
    "    def hinge(self,scores,y):\n",
    "        #scores: N x K\n",
    "        N = probs.shape[0]\n",
    "\n",
    "        correct_scores = scores[range(N),y]\n",
    "\n",
    "        # need to broadcast by row not columns\n",
    "        margin = scores.T - correct_scores + delta \n",
    "        margin = margin.T\n",
    "\n",
    "        #zero loss at the correct label\n",
    "        margin[range(N),y] = 0 \n",
    "\n",
    "        #element-wise max\n",
    "        margin = np.maximum(np.zeros(margin.shape),margin) \n",
    "        data_loss = np.sum(margin) / N\n",
    "        \n",
    "        return(data_loss)\n",
    "    \n",
    "#test\n",
    "svm = SVM()\n",
    "correct_scores = svm.forward_correct(scores,y)\n",
    "dscores = svm.backward(scores,y)\n",
    "loss = svm.hinge(scores,y)\n",
    "print('Forward pass: {}'.format(correct_scores.shape))\n",
    "print('Backward pass: {}'.format(dscores.shape))\n",
    "print('Loss: {}'.format(loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Linear > Softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0: loss 1.0400675167105593\n",
      "Iteration 100: loss 0.7426315244989891\n",
      "Iteration 200: loss 0.7383929886913772\n",
      "Iteration 300: loss 0.7378299545312949\n",
      "Iteration 400: loss 0.7377135218911937\n",
      "Iteration 500: loss 0.7376860547312004\n",
      "Iteration 600: loss 0.73767933138491\n",
      "Iteration 700: loss 0.7376776656269808\n",
      "Iteration 800: loss 0.7376772486055508\n",
      "Iteration 900: loss 0.7376771415078005\n"
     ]
    }
   ],
   "source": [
    "iter = 1000\n",
    "reg = 1e-3\n",
    "step_size = 1e-0\n",
    "\n",
    "#initialize weights\n",
    "W = np.random.randn(D,K)\n",
    "b = np.zeros((1,K))\n",
    "N = X.shape[0]\n",
    "\n",
    "#Architecture: Linear > Softmax\n",
    "lin = Linear()\n",
    "soft = Softmax()\n",
    "\n",
    "for i in np.arange(iter):\n",
    "    #forward pass\n",
    "    scores = lin.forward(X,W,b)\n",
    "    probs = soft.forward(scores)\n",
    "    loss = soft.cross_entropy(probs,y) + reg_loss_calc(W_list=[W])/N\n",
    "    \n",
    "    #report loss\n",
    "    if i % 100 == 0:\n",
    "        print(\"Iteration {}: loss {}\".format(i, loss))\n",
    "    \n",
    "    #backward pass\n",
    "    dscores = soft.backward(scores,y)\n",
    "    dW, db = lin.backward(X,W,dscores,reg=1e-3)\n",
    "    \n",
    "    #update weights\n",
    "    W += -step_size * dW\n",
    "    b += -step_size * db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.5336666666666666\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsvXm4ZGV17/9Ze6i5zjz03M3QTAIK\ntGBABaOSRo1EDFFvYhyIZJRfpuden2u81+T+1E7yi9eQmRjjkBgVJ1BQQBAEVOaZZmgaejynzzzU\nXHvv9/fHrjpj1ZlqPuf9PE893VW1a++3TlV93/Wutd61RCmFRqPRaDYWRqMHoNFoNJr6o8Vfo9Fo\nNiBa/DUajWYDosVfo9FoNiBa/DUajWYDosVfo9FoNiBa/DUajWYDosVfo9FoNiBa/DUajWYDYjV6\nAOUIhNtUqK1/9n67x66I18ARaTQaTfPzyBOvjCilepc7rmnFP9TWz0W//tlFj9+4L0H7dLYBI9Jo\nNJrmx+j50KEVHVfrgVSbqz8W44OPRBo9DI1Go2lpWk78AY7/MMzln+pmMh5s9FA0Go2mJWlJ8S+i\nVwEajUazNlpa/EGvAjQajWYttLz4F9GrAI1Go1k560b8Qa8CNBqNZqWsK/EvolcBGo1GszTrUvxB\nrwI0Go1mKdat+BfRqwCNRqNZzLoXf9CrAI1Go1nIhhD/InoVoNFoND4bSvxBrwI0Go0GNqD4F9Gr\nAI1Gs5HZsOIPehWg0Wg2Lhta/IvoVYBGo9loaPEvoFcBGo1mI6HFfwF6FaDRaDYCWvxLoFcBGo1m\nvaPFfwn0KkCj0axXtPgvg14FaDSa9YgW/xWiVwEajWY9ocV/FehVgEajWS9o8V8DehWg0WhaHS3+\na0SvAjQaTSujxb9C9CpAo9G0Ilr8q4BeBWg0mlZDi38V0asAjUbTKmjxrzJ6FaDRaFoBLf41Qq8C\nNBpNM1MV8ReRL4jIkIg8XeZ5EZHrReSAiDwpIudX47rNjl4FaDSaZqValv8Xgb1LPH8FsLtwuxb4\npypdtyXQqwCNRtNsVEX8lVI/AcaWOORK4MvK5+dAh4hsrsa1WwW9Cqg9ngdKNXoUGk1rYNXpOluB\nI3PuHy08NlCn6zcNV38sxpa9Jl+8INXoobQcngci/m0ugyfgS182eO55//6mfvjgb3qcfnr9x6jR\ntAr1Ev8VISLX4ruFCMV7Gzya2nH8h2Eu/2GYG/claJ/ONno4Tc+hw/DlrxgcfBlsG15/ieK9v6YI\nBCCRgP/zKYNkEsCfFQYGFfv+2uB971Vc/ha9FNBoSlGvbJ9jwPY597cVHpuHUuoGpdQepdQeO9xe\np6E1Dh0LWJ7RUfjMPoOXDgpKCbmccO99wt//oy/0990vZDNQFH4f/9ivf0OYnm7EqDWa5qde4n8z\n8JuFrJ/XAZNKqQ3n8imFjgUszR13Co47/7F8Xtj/nDB4wl8VOK6UfK0IPPNs6ec0mo1OVdw+IvJf\nwGVAj4gcBf43YAMopf4ZuBV4G3AASAEfqsZ11xMbIRZw8GV4+mkhFIILL1R0rGBxd+iw4DiLBdyy\nYHAQdu2EBx5UeN7iYwwDAgHt9tFoSlEV8VdKvW+Z5xXw+9W41npmvcUC0mm45QfCfff77he3YMHb\nFnzz28Lv/rbHea9Z+hwn7VK8+CKLJgDHgS1bYPduxU03C6m0Yr7rxxf/s8+u4hvSaNYReodvE7Ie\nYgGOA5/6jMGtPxAmJgTXFXxxFvKO77v/5xsMssvMcW99s8K2AGYteNtWnHO2oq8XohH48//tsWVz\n8RiFaSqCQcUf/T8eAbtW71CjaW2aKttHM0srrQIGByGZgu3bmRHbn/4Mjh0Hpcr73JWC/c/Ba15d\n/tydnfBnH/f4z68aPP+CIhiASy9VXPUrs5NBby98+v/1mJyEZ/YLwYDinLMhEKjWO9Ro1h+imnRX\nTFv/bnXRr3+20cNoCrbsTTdlLGBsDD53vcHgCT+4ms9DJAK9PTAwCJnMcsFWRU8P/NaHPV56Sbj9\nR0I6BaeeCu97r8f2bXV5GxrNusLo+dAjSqk9yx2nxb+FaJZVgOfBAw8KX/yyFNw2C0W++J1aWaaN\nYSgMY65fXxEKwV980qNv/W730GhqwkrFX/v8W4hmiAUoBX/zf4V/+3chmy368RdS7vHSeN7CjB4h\nl4Mf/FCnaWo0tUKLf4vRyH0B2Sx8ep/wzLOl0y+riecJBw9q8ddoaoUW/xal1quAqSm4627htjv8\nzVQAX/6KcOCl1Vn1a8UwFFu3NqdLUqNZD+hsnxamVhlBDz0MN3zeQMT373/zW8Llb1E88JAsmb2z\ncubn5BuGL/JzN2pZFrz9Ci3+Gk2t0Jb/OqCaq4Bkyhf+fN7PxXccIZ8Xfni74DiVnt3PwX/tHkUg\noLBtRTisePdVissu9e+LKLZsUfzJH3ps3VqNd6TRaEqhLf91QrVWAY8+Vlrk/d25lVj9fkbPJz7u\nsWsnOI4ikYR4DEzTP+I3/pvCcdEbszSaOqDFf52xlhpBSsE9PxG+f6swOlquIUop4V9cUmHx87Ns\n6p892rJYVNvHMCCg16IaTV3QP7V1yGozgr5/i/DVrwkjI0Wf/sotfNNUWJZfVqE8/jmPD8Bn/spg\nYnLFp9cUsYIQ74OOrf6/lt6+rKkMbfmvYxauApSCO38s3HabkM/Da1+r2Hu54js3ScmqmMsRCPh1\ndQ4cEO7/Gbz0kh8gtiz/X/DLL88iuK7innuEK9+5ToK5dhjsECgPsgnw3OVfs+prhCDei4hvqynD\nBLsfpobByVT/epoNgRb/dc7xH4a5/Ach/mTvFN/6N49cDoqW/R0/gh/dKWvqe2vbir2/pNi8CTZv\nUrzh9XDokOKZ/UI0Cq4DX79x8YSSzwsDg+tE+Nv6wQogYqCUgnAbTI9APl3d60Q6Z4QfQMRfSalo\nJ0yWaYthBcEwwcnWZkLStDxa/DcA8bEMX/0nhSxy54gvWqsO5CrOPkvxrivni/jOnbBzp//YkaOl\ni7oFAopTTl7l5ZqRYAzMwIwozwhyrAfGjyz92oUEIhBq88U6n4b05KxgW0Ewy0TASz1umP6kJGbh\nYxXIZ2B6mKVdc5qNhhb/dY44HrHJ7BLyvpYMHuHFl0CkvJhs3wZnnA77n1Mzrh/DUITDfg/elsO0\nAQE3598PRhGjTMgs3ue7agCySUiN+26hUoTaINI+x6UT8yeDqWGI94AsEZZbdE7xr21YhcmocJgd\ngs6t/ipBrwI0BbT4r2eUon24NtVAE4nlj/no73t87xbhJz+BXB5e82rF1e/2J4CWwbQLgmoUDGfl\nW9FmmYCrCNihGfFVwSjYQZg4vvhYw4JIxzyhFhEUBrT3z9wvxUxBRjEg1u3HHuacY/6QCueM98LU\n0OykYZj+ysJzffeQZkOhxX8dE8i6hNJOTYoxlNGkedg2XPUr82vvtwxiQKTTt/CLb7bwj2rrW/ql\nC8XcsCDWC4kRCMV9y960ylr1Ir47rpzwzxyD+Nk/IkseO3O8GfCPnx6GUNQfR/GjUS5MndArgw2E\nFv91iOH4ll0wkVul8K+0FLNi544WFPTlMEzfl29YvtvGMMuIanmxLfW4iKACYejavuRxy51n8TEG\niqUniUXjAChMXiIyO6Ep8SeoqcEVnUvT+mjxX0eYOZfOoSR2rox/eRnEEJRXTtSLgWGFbcO1H1ln\n4m8FC6Ioy1reK7HMq/GalZ63GseLCMqy/QlQW/8bAi3+6wTxFD3HExiemrHbV5PHo/Bz8w1ToMRv\nXwS2bVOccrIinYZP7zOwTHjjGxXveLtq/ZIMsZ4S6ZSlWWsDpGoLf/WRgiuq1BfA8N1EIpBLg1dx\noSdNg9Hiv04IJfOIUvPEfjVSU9zXq9zSrzNN+JM/9Pg/nzaYGAe3sCnsBz+EAweE//6na1ttNAWG\n5Vu8q6D5hXyNtG+e/X8uBclxP2Ad6561JiIdkE35QWInN5sBpWkpdHmHdYLpeCyRebkqFp7GshQX\nnK948ikhkZgVfvA3bT27H/7XJw1eOVSd69cFKwSxHj8nPtK5qpeuV+GXQuC4eCMQgc4tEOtGxEAM\nw/9XDAhGIdrpZyXFda/NVkSL/zogkHYIpapjfc11GXni/3vmeQYf/qDiwEsUWjcuftXhI8Jn/tJg\noJHxQsNcmQUfikNbLwQiiB2CQHjdCnolFDeulVoL+hNEYSKwQv7fVNNSaPFvccJTWboGE9hZr2op\nncXzjGyJcWJHG7eNtvPbT0fYtAkCdvnlRT4Pt9xSJxENRHz3Q2GnLe2boWOLf2vfPLv71Q77ZRcC\nhX4HYhRy640ZwdfCX56ZVcBSxxiG/znMxQ77qwNDe5abFf3JtChW1qFtLEOwRnn8AE5w9utx/Idh\nrneD7LKmIF86lOx5wsuHoKZlBER8cS/sYp0bfJ3ZWGXa0LZptrxBEeVCJlly7JoKKQaEPbfgBpqz\nNyKT8Hc5a5oKLf4tiJ1x6B5IIKo2MqaAdGRx+o5nGhzuitOXnoaxxROAiGJbrfvuRjrnlS8olUI5\nk8/OfKteYUK4TVv6VUYp5e+AjvXMPDbv7x6M+fWFql3wTlMR2u3TgrSNpjFqKPxKYLKvdFtIJ2hy\nvKODVNTGWjA/2Da84201Fv+5O24LrHTD1UpcGJrVM+v/l5J/YzEMPyZQjA+E23yXkF6BNRRt+bcA\n4npEp7IEMi6ObRDI1nYTzlRnEGUI4insrItrCm5gfiB1oi+KN5YhlswirmLLZnj/b3hs317mpFVj\n5btZNzJFd1jT/B3sEHRum72vlL+KmzoBbr5x49rAVEX8RWQv8LeACXxeKbVvwfMfBP4aOFZ46O+V\nUp+vxrXXO4bj0Xt0GlEKQ4Gqw8o5Fw4QmczQNjbbKMSxTcY2RfGswmJRhKnuMFNdfvXKz/1lcu29\ng0XK9Y5cjOeUL3FcoOmErwS12O274AqgFAqjKf4Oi8ZQjNfEesr3JNDUlIrdPiJiAv8AXAGcBbxP\nRM4qcejXlVKvKdy08K+QtrE0hucLP8zZjFWj6ynAdD3axjIYipmbnXPpGkwufoEIiHD1x2J8b3qV\ntkSkw69307ndLzgWWEG5z+TYmnfY1gOl1MxtKWouyJ4LntsUwl8OEfEL3NnhlVUK1FSVavj8LwQO\nKKUOKqVywNeAK6twXg0QTJXP5qm2BCogFzKJTmZnJpsiAlh5FzNf3uX0d9e3r7x3cLQLQvFZX7Fp\n+Vagtcxr8xm/TVip8RdEt1G+faWUv+t17DBkpqs6Sa30XEoplOf5FUSdbFNPlD7if+6d2yDc0ejB\nbCiqIf5bgbmti44WHlvIu0XkSRH5pojU3DO8XlBGmWBmLS4mMN0VxnDLC8ZSzxVZdhUgUgjcGgse\nNiDcvvTJzQCYZklRa2RAVynlW9tTJ/wH0hPgOahyTVyWOI9S3px//dtK3GJKKT+jZuKYX3YhPTn7\n+LzzN8+EICIzO4cJxwuBYE09qFe2z/eAXUqpc4E7gC+VOkhErhWRh0Xk4Xzhi7uREdfDNctb+NWU\nOQWM9kfJhywyEYtykuUEVlYDZ8lVgCxxjlL+/Ein7x7q2gHt/fM2aDWCGdeO582KaTYJk8fnHgQT\nA5CaQOVLW+ALH1NK+UXTxo7450pNQHIMxo+V/LAXijpKQWJ0tlmL5/pNZNz87DjdPBQml2ZDxPA7\nm2nqQjXE/xgw15LfxmxgFwCl1KhSqhgN/DxwQakTKaVuUErtUUrtsZezANc7StF7LEGgijt3lyMX\n9q31ZHsQzxSKJXyKpR6musJlVyLlKLkKKFMRcsZtMpdYD4Ric1IJG5+dPLvCUL6LZ+wwJEdLWOcK\nMtN+jfxcctaS9zyU5/quIc/1hdl1fFdNYth/qev4r80mfTFPT5VcRcwT9anBxa0dPccPqBbHOTng\nr07KrAAaPimYtj8B6BhAzalGts9DwG4ROQlf9N8L/Le5B4jIZqVUMaT/TmB/Fa67rgkn8hhu/YQf\noOtEkrFNMZRpMLwtTnQqSyjl4JpCsj00Mzmslr+7vp2/A27cl5jNCEpNoub2rlUKUDOuCsBfIQQi\nTRy0FD9GsZIWiIlRMKf8lEfP9S181Mp3vqYnfYs93Ob/XZyc/1o3549jNe4lNw+ZqZIuNn/TnNew\nSVZEUJEOPxkgn/Hft24xWRMqFn+llCMifwDchp/q+QWl1DMi8hfAw0qpm4HrROSdgAOMAR+s9Lrr\nHTvrLAq6wtyWKtV1+wh+cNnMu7i2iTINEp1hEqsreLkkV38sxkevc/nluOOLj3JR4faCmGULYjYn\n59u0fFFbyk3USIqVL1cqTm6+spz2zLR/W8QarPVcqqT4K8+DzBQq3NbQCQAKjeftkD/WxMjcAyDS\nNRsfyGd895juMbAqpOHLvDK09e9WF/36Zxs9jIYRncgQH8+UnABqhQeM90fJRmvfmWXeKqAcdhji\nvU1s+eO7YpLjkF1BR/tmI9ZTqGhaXH15vrtpcsCf1KKdheYu80tnlKNWexdm4hlu1u8jEIqBac9f\nNSrPj2+sMsC+HjF6PvSIUmrPssfVYzCa1ZOKB4Da5PMvdU7Hrs9XYtmMoFAc4j01Ff7VGj6lM4z8\nKqEtSWIEkuMoJ4dy85CegslCTe5cyg80jx/zA9Djx5Y8VS03rRUzgrBCfoqwGSjRdU10ptAq0eLf\njChFbCI705yl4A2v7iVK3M+FrUVlHGpJ2YygOWWXa81SE8DcDVvKcyn7KSywjluKbMK39CeOF+It\nC96j8vzHlFty78JMwLkQuK4lS6XyimH4acCaFaPFvwkJph2iU9mZ3byl22lURi5gzEwqHpCM24z1\nN8ZyWrQKsAI1rQoNK7X6lS+K40f9W5nNZTMCud5JjUMmMX//QXpyNtupTn+DUhOA8jzdTnKV6MJu\nTUhkKldzX/94f5RwygHPIxsNrDh/v1bMywhS8bqk+i3bpH1ho/LUOCreO29FopTn5+NvFFJj/iRg\nmPP/NvmMPzGEYsxNSaiVK2ium2kmU6wV4y4NRFv+TYjUIQjff3Sa+FiatvEsvcf8/zcD/3JjB64d\naoIgbwkxyWdgegTl5Gdz85NjG1B0VOnMmtS4HzNITZRfJVVzFMXNavm077pq0uSVZkVb/k1IOhYg\nkCmd6lkpM01O5vYDUBCdzJKJ2uSDjf1KvO0XbCyjtjbJjKW4wJk26woqbM7KZxa+tCA0zTFRNiXF\ndNZIR02tfp/C+e2Qv98il6rJ9dYrWvybkHTUIjJlYufcuqV6ioLQdL7h4m/X0Ps0IxrpKQiEUaa1\nIKhcEP1MQueMNzGLJxVBxXqYmdDdnL8ic3QMYCm026fJiExm2HR42m/YosAxqh9GW8LTXeUrrZ4f\nPeLgejUcx8SAX3RteghcZ7bcQjFfPzWhhb8a5FJ1LxUxUwLECkJbv24evwz6r9NEhKezM3X0oSDS\nao6rpgrXKFseWiATa0yqnGHAxa8yOW27yfCE5yfP1MQsUaAKwu65vp/YsPzgpZOjGSa/dUNyDKwA\nak6/5VpSajVAuM0fh6YkWvybiNjE4jr6hqq9JCkg2RYkH6r/1yEShM9dF6a/0yASEtJZhVkr14/C\n3yg0t5G452hLvxYUd9xGu1DB2JzMHI+iCVLLSUFEUDrvf0m0+DcRplN+a3o1fibl6gG5Bkx3r6CL\nVg34jcsDbO0xCNj+yMJBqbG7QFv3dSU5Brk0KhT3V1i5lL/KivdW7RKldherYjkITVm0+DcR+YBJ\nsMbN2Rd6VDyBRGdjhB/gTedbM8JfpGYWoUjpDB5Nbcmn56+2wK9QWqX+wiWFHwXpUkXwNEV0wLeJ\nmO4Oz9TQL6IW/Dv38bXYsPmQiSfM3BIdIVJtjVsee/Wsw6WUnxbYBKSH06RObNCU0ULf55oVgfNc\nf7+Bductibb8m4hcyGJ0c4y2sTR21p2fi7+A4h7K1QaDs2GbsU0xDNfDNQ1YZXOWanP7Q3muflOA\n4Bzr3/NUQR9qMLYGZ4BMvTTFPb99L+P7/V3B8V0xzvno2XSc0UH3uV3IKj6P43cf5/mvHMBNOZx0\n1S5OetcuDKsF7DkzUP2a5AVEBOXmKiudvUHQ4t9k5EMWo1vi9BwrpHsuwVrq+ruWgTIE12iOGvn/\ndWee83ZbnLbdwCoMqWZxQKFhjUGUUrhZj+/v/QG5idn888kXprjvoz/FDJsYAYNIf4R8IocZsgj3\nheg4vYNT33sK3ed2ceQHRxl7dpz2U9oYe3aM5//9BZyU/x0Z/OkJDnztJd76jTdjmE0+AXhOzT5k\nf8dvxi9JXdz9qymJFv9mZYU+ndX+hAIZh3S8ebIgcnn4wq1ZPvORMGLVsnyz59fqqYNFmJvO8eJX\nX2Lw3kE8x2Pq5WmmDy7tf3bTLm7aZXJytpPZ9MFphn4+zIGvv4QVtPAcDyfpYIZN3PR8w8BJOQw/\nMsLRO46xY+/2hadvLsJ+Cexql4GeSRSIdDKbI638PR16w9citPg3Kam4jT3qVn1lbLgeXQMJrLxL\nPmAy3RnCaXRJh9fZWFUewqKMoXx2fjeoCkmdSHPsruMYtrD98m0ECnGTgfsG+NH7foybqV7g3st4\n5DKz4rVQ+Is4SYe7r/kJZ3/0VbzmT89tThdQIALB2rTmnH9OmbGMVLzPr8qqmYcW/yYlG7FhtPqZ\nKcGUM1Mi2nQcgukEEz1hXNskHzQb0jg7GhKMKsceFmWA2EG/VHQVLMBnb9jPw3/+KGIIYgo/9X7O\n66+/mOe/9AKD952o+PyV4OU8nvnHZ0keS/KGv7ukoWMpSTBWsk9DLZvBgPiBfp3pNY8mNA00AFa+\numkwRTt4btsRwd9E1jmcpnsgwaZDUwST9Q+U3fO4Qzpb2/x7hUCg8n4FQw8N8eAnHsbLebgZFyfp\n4KZd7vnIvQ0X/iJu2uXl77xCeqgJ/d3LlNGeaQ5T9etqqVuI/os0KdVup7iUTVWcBAxP0TmUxMzX\ndq/BQn78mMMLR1wyOf9HX5Mfv4KbHgkt7hq2Su76zbv9zRJNjhkwmVomztAQssnCLt8FKA+mhiCb\nBDdf3e+A3t9REi3+TYprm2TCVlX3o65kUS0KwtP1DY5t7hZSWQhYtVv+5/Jw1yPO8r2Dl2D8+Qky\nI62xazQ/nceKNaFXN5uAfNbvvEWxJr/nx2OcjN8RbHIAUhNzOoatfTUw87pQmx9ojnRCsNhwZmOj\nxb+JGd8UrXsxAgFMt36mbW+H8Hd/GOGiM00Mo/obf5RSuK7iez/Ns/+Q/77K9g5ehiO3Hqnq2GrN\n8//+QqOHUJrpIUgMo9KTkJr0awAttMwzU37j+OQEsHaDYKbvb7gNwm1IuM1vAt+5xS83sYFpQtNA\nM4MIubBFMO1U3U4ptznMk0KwuU5c9UaboE3VA75FRISnDjrc8L3Fq5mrPxbjo9e5/HJ8ZTtBR54a\nrfbwasoL//Ei7bvbGH5khFB3kNPefxpdr+ps9LB88pnlXTGmDZH2kgHi1TJ38hARFCZEuiAxXPG5\nWxUt/k3OdGeIQCaB1GAJkLcEy1UzlUQ9ASdgkqmj+J+1y8SuaX6/4uBA+RjGvN7B04tdOsOPjnDf\ndfcz+fxUzcZYMzx46BOP+P834cX/fInX/dWF7H7fqY0d10pp669ZoFZEUIHG1bRqBrTbp8nJhyzG\n+6PkbWOmnEM15gEBLA8meiNkQhbZoMlUV4iRLbG6pnseGnRx3No5t0SEy85bfjIrFQuYeGGSH/zy\nba0p/Atxwc24/Px/PEi+ARldqyYQppYN4DVa/FuCbMRmeHsbAye145jV/TFkYgHGtsQY3Ron1R6q\ne57/N+/O4ywwzKud7dMZEy44ffmvejEWUOTJzz2Fl2uB1J5VIIbw0z/+Off/8c84dOthvDrGd1aF\n1GPPycYu763Fv4UIT+ewXFW12v6pWP3cO+U4PKT4nzekOXzCXwG4Xm2yfT5zbZg/+rXAivTk8k91\n871pi+GH158/2Ek6vPzdV3jxKwe49/fu57Z33YGbq29qbyXM3QtQiZGglPJ7NW9gtPg3EYbj0TaS\novfoFJ2DSezMnECkUrSNZSoW/qLbyLGE6c7mKG/81EGPa/4yzbs+nuSv/ytb9Q1fxYyPKy6y+dwf\nrOw9/9317byYa6/qOJqGgrHvJB2GHx/hmX98lkwNdpNXhBUsbwQkRqtTqiM1Ufk5Whgt/k2CmXfp\nOzpNdCqHnfMIpfJ0DyQIJfwsFfEURhUamxd/Tpaj2HRoiu5jU9hpx6+A2GAyOX+377Fhj1y++uMR\nEc7cZXLN220uPNOkv1MwDThpk9ARWyw0Vi5X0jGQt4NMt3XhmBaZcJREvIOy6zGB+MkxOs/pxGqz\nCHQEsDsav+Iq4qU9Hv3M43zj3G9xx3vuJDveLPsYvNKWfbFRSyheuLv674lSyt9MtsHdPlXJ9hGR\nvcDfAibweaXUvgXPB4EvAxcAo8B7lFKvVOPa64X4WAbx1LzSC6KgfSRNJmqjDEEJVcn6mStTgaxH\nz0ACJX5mUbI92JD6PkUcFx4/4LJzkzHvh10tV5CI8J5fDPCeX1z8XDKjuPn+PK8MKDJpj9dc0snm\nS19N/uKz+dynFliJZYQpkElx8rMPc/bmad7w96+n/dS2kuM4fs8Ad7z3TpTTBALk+TWBjt8zwJ2/\n8WPedsveRo8IMsnym7FC8aVXBishvbGtfqiC+IuICfwD8FbgKPCQiNyslHp2zmHXAONKqVNF5L3A\nXwLvqfTa64lgpnQuvyiF4So8yyDZHiQ6ubjJeyXMTDYK4uP+0j/Z0Th30KlbDd5xsV3T9M9yohEN\nwfvePKfc9UeuZmQoz5/+9lCpk5R8LBeJ8dyeSznepugdU7g/g1NPhb4FLWu3XLqZDwz8Bs/esJ+H\n/vwRVK7xk4ByFSOPjzJ5YJL2Uxvs8nJz/i7fSOf8iTY5CrHuyo0Br3XiHLWiGpb/hcABpdRBABH5\nGnAlMFf8rwQ+Wfj/N4G/FxFRte3U3VJ4pmCWSHkUBaqwAaroo49N+Evzqpd7VhCfyJJsCxDIuIRS\neZQI6XgAJ1Cf3ZCvP9fCbtDuk1KC8refXotvWZiagn/6l9nzbd2i2HMBnHO24uSTwSg4XM+69kw2\nv2ET33/rrbjZxmfeeDmP1EBs0aNwAAAgAElEQVSq8eIPkJn23TN22K/9k8/4lVmr0QXMjkA+taJD\n1QM/r/BizUk1fmZbgbn73o8CF5U7RinliMgk0A1Ur8B6i5NoD9E+kppn1Xsw4/IBQITprjDpqE3P\nsURNqpOIp+gYShFKOzMupthUlsmuMKn2yoqirQTXVc0Qfphh4NhaBXn+p3PsuH+76XtCIAC/9zse\nr3m1/1znmZ3Ed8WZeHFyXtE4K2Jxzh++isxolv03PFc3F3XHWYt3AasHfs70zS/VZwBLEQzQ9jd/\nVHG00ksZJD75n8se98TdzdP4qNo01Q5fEbkWuBYgFO9d5uj1RTpmY+WDxCazM4ZNLmQx2RuZd1ww\nlafzRLJmZak8QwilnPm/LQXtY2kyMRuvxi0C73nc4T2/GJhp6dhouroNhk5UwyKf/cRyObj+c/DO\nQ3fQnfV9z205j2mjoP3FhlS2YvLG5xARTj41yKGDWdw6eCvuftN3icVLfQDNIYS7L32WTe86u7J6\nPz0d61rYV0I1fsnHgLl947YVHit5jIhYQDt+4HceSqkblFJ7lFJ77HATLDvrScGqH9zZzujmGCe2\ntzG2OTZr9QMoRfuwvzqY+7Wv1q5fBeRtKfmlUEAwvbIaOJVweEjx77fmyOYVjluj2u6r4A//rKcm\n5/XE4Omu02fu2wGDnScF2bTFprffZvuuAL399ozAiQg7TgoSrEM4ZnysuXcAH/hfd+BMZFBzst+U\nt7rvipgG/e8+uxbDaxmqIf4PAbtF5CQRCQDvBW5ecMzNwAcK//9V4C7t7y+NMoR8yMIr0YLPcBVm\niXTPaq4CXMssO5GoOiUBfesneT70mRSf/34O110+na+WX6Wt2wP82m/Gqn9iEcbt+IKHhEjEJN5m\nYpfo5yACVh1aM+aykEo0T0DUiNiET+7CKNScUo7Ho7/6FaaeOI6Xd/ByDl7eXdVKQETY/cm3YoSb\nJ+223lT8TVJKOcAfALcB+4FvKKWeEZG/EJF3Fg77N6BbRA4Afwx8rNLrbkSUITX3+wYzTtlLZOv4\nQxmeUHzrnjzv/l9J9h9y8crs6szmFbkaL0jeflUHX/zOVt56RcFNUI3JZg3nSCY8Usn6BIUHjufJ\nNjoALXDSf7+UX/jp73HeN36dX7j/9zjpT9+IEbXZ/GuvJnZmHyIGylUYa0hIEMug7+1n1GDgrYE0\nqwHe1r9bXfTrn230MJqOzsEkoVR+nrXvFfL/q1X2wbEEy1Ezlr6oQsXPoMl0Z5hcuHGhokvONnn/\nLwXo7zJ4ZdDj376fxbaEP/tAiPicIo3VLBFR/I2k3WkePvsGHuw+h6d6zqp4P0RbdoqrX751xccP\nHM2RStVPkE0T2jtN4m0WVg1Tb8ux/dqL2PE7F2FGZn3zbiqPm8phd1enCfzxrz3BgU/eUfF5molL\nn7/1EaXUnuWOa6qAr2Z5JvoidA0msLMuFEQ/FQ/gWEL7WLbiCUDwd/9OdIex8y6RqZzf4EWBmXGx\nBxOM90frWvN/Lvc/7XL/04t70777z5KcucPg1aeanLzF4I2vthADDJE1dQcrCr5SMDal+Ow3suz9\n7A0AvHb0KV7o2EXWilQ0AeRMm4QVIeYsn3LoeYpMnS1x14XxUZfxUZct2wKEwvUtCLDtw3vmCT+A\nGbExwlZVhN/N5Ekd2LgJh1r8WwxlCKNb4pg5F9PxcIImnmlg5j0Yq87WfAE6RtO4LPYLGgraRtMM\nN0j8l2L/YY/9h32B/MKtOX71MptzTjbZ2W9glvEKlFr5ZvNwz2MON96dY3hS0XFZmms++6WZ5wU4\nefIw+7srcxlkzBA3nvw23nb4x/Rnlm4UM3wi35B9ScU/z4nBHDt2VbirdpVYbaWj29UYg/I8vIzD\n0M3PLn/wOkWLf4viBkzcOX5OqwZN18t5Ua2856tCE9daHxxT/P23/bpIey+0uO5XgzPpoyKCpxTp\njOL6b+UYnfS4+k0BTAO+/9M8P33GneeS/+4FKe5dcP5UIFr5IEXwxOL7O9+C5Tn0ZMbYM/Ik/en5\nE4HnKRKJxvrfXQccB+w6zvnJF0eInV6dlO95k7yCqUeP88InbsOZapZaRvVHi/86wa1y/v1yst41\nkGRsc7SpJ4AiP3zQ4e7HHX7pQos9p1t4Cu59wn+s2EvgiZcWV7XcsjfNNdd9iXtLuOWnAm3Ve+8i\nOKbNYKSPH2x/E28+dj/bkwMzTxdrmTWaen/UL33qLs7+l6swgiZiGH7Td1lbg5fia5RSTDx4mKc+\neGO1h9tyaPFvYQzXIzKZrUv+/VwECGQdYhMZEp2t0Qovk4Ob7nO46b6V/62+WMLiB0iZITqyk4wH\n26rbZlAEVyx+1n8+2w/eMvOwYYBl+ZZ3o7Bt6h70nXzwCE+8/2vs/P2Lie7uIT+ZJnJS16I4wGoQ\nETou2kGgL0puKFnF0bYeWvxbFMPx6D067Zd6pv6GoaEgMp1vGfFfLbd/fJR7L5lvHWaNAHdtuZgT\nkV6kuA+7Bu6vhB3FFQNT+a4eEaG332bweL5hpS/sQGOqvyeePsEzv/sdfwydYS788W8vOkYpBcrv\nUrZSomf2kRt6uWrjbEV0Pf8WJT6exigIPxRKQJc5tmZ6MUeJzLxLbDxNfDTtN6Fp0hTilfDR6yYX\nCT/AHdtez0CkF9cwcQyrZn4Q03Mx1HwffyRqsnVHgEi0/DVNEyLR2vykU0lv3o7aRpAfT/Py39yD\nm86jCu0nnWSOiQeOMH7/KzhTmZnHlyP14tIB9o2AtvxblFCqdAnoahQ8XAkKf8dxbDyDa0D7WGam\nEFx0Kks6ZjPZU1kqZCP4Zup6nti72K0wZccYDnWhjNoWHTI9h7PGXyj5GQaDBv2bA7zyUrbs3Fqr\nTWBKgafKJwHUi+NfeYzJh4+x6apXYUYCjNz+ImM/OQgK7O4IF93928gSg1RKkTkySfb4VP0G3aRo\n8W9RPKN0CeiF1MJWK04wwmwPgLliJQrCiTzpmEOuhbbPb9mb5onrSvuTk1YIr5SqVGtyUwpRHqdN\nHOSCkafLHmYYQixukJj2Fk0AtSz6ZhizZagbTXL/EC99an6PBasjTP+7XoWX9xDLKBkUVkqhHI8n\nfvPr9RpqU9MkH6dmtSTag3gLvt/1svplwf9LN6GBcKEFZatwzXVfKvtcwKt1tFVxytQhLh56FGOZ\nKbu3zyYaN+q6qGrvMOua478awid18drbrmHn71+MtcT+ExFBTGHbBy+o4+iaF235NwKl6BgZwc5m\nGevvx11D8nQ6HsDOeUSns357xzKrfaFxWYJWxiE+liYbtsmFzKZ2AX3i1n+edz8vJj/vO48jsa1Y\nyuGUyZep6fQqBnljZd8DMYT+TQG8XkUq5TI06NQ8xBKNz7cTlVJMTrhMjDm4LgSDQk+fXfddwAC7\n/+KtWPEAUliaLDVJieHX8zm47+46ja550eJfZ6KTk1z23e8SmZ5GGQaG5/HwZZdx8OxVlpcVYaon\nTKIziJV1EQWdQ8lFPX7rtRooRSCvCExkiU5kyURtJvqaMwZw474ET87J5U8bAb5+6jtxZXbCerzn\nHER5KGqzuc3y8uxKHF3VawxTMOtQ5RNgbMRl89ZZt9fYqMPk+OxmuGxWcfxojq3bAwRDdZwADKH9\n/K0zwr8SGh24bha026eeKMUvfvvbxCcmsB2HQC6H5TjsuftuugcGln99CTzTIBexyUZt8sHF5Zgb\nJbUy52YA4WSe9uFU02UBfTN1PU+e+x/zHrt7yy/ME34ARFDFnH6lZm+VUHy9UiiEjuzkqk8RCkld\n5tN0enZp6XlqnvAXUcqfFOqKp1ac4QOgXI8T332mhgNqHbT415GuEycIpVIYC341huOw+4knKj7/\nZHfz5twLEEnk6T42PZM6EkzlCabyDZsQbtyXKNnNaTDaX7ZB+6JbJRRfL4IrJt/f+WYGwj2rdtN1\ndtc+B0d54OT9kTlO+RHmGlAGeurx4yvq6aCUIvHCCIf/aX325F0t2u1TR4LpNKqEYBhAOLWyZtJL\nYbp+GeaFrp+FzH26nisDAQI5j75Dkxje/IuP90bIxurXVu/2j49y77mlt/g3ZCoSwcXih9vfRHd2\nnL1H7iHgLd1Ra3rKYXjIoeiJqvUcOnA8y7YdwSV3+tqB+n2j7P4oJ/3RG2k7b+uywWilFOlXxnns\nXV+u0+iaHy3+dWRk82aMEvl4jmVx7OSTKz6/Y5vL5v4r/Nr8eQuCDejWJ4DpFcYzR6y6hlLkJjOM\n98dKdjFb07VcD8NTuJYxz0rfsjddchMXwFCoC9vNkzMD9Y9PiOCJyWiwk/v79/CmgZ+VPTSb8Rg+\nUftA71xyWTj4YhbThGBIyGbUvOuLQFd3fVJ7T/vUL9F/1dmF667gc1Lw1Ee+VeNRtRZa/OtIPhTi\nqde9jnMeeACrUKjFsSyS8TgvvepVFZ/ftQ3SEZtQKo9RdCcDCDiG76ZIxgN4xU1ZFV9xbZRMDQUC\nWY/uY9MMb4/PNJNxCxOBeMrvZLaCH7q4Hp1DKYJpByV+GeyJnjCiwHQ8/ufRhzhk2ByM72Ak3EUs\nn+Kk6SMk7Qh3bH0Dnhi1E/4VlIPwDJNX4tvwBqRs2ufEeH2Ffy6uC25aEYkK6bRCeWDbQk+fVZds\nn54rTqd/FQ3cleNx9MuPkD26+pjKekaLf53Z/9rXMt7Xx2mPP04wk+HIqady4Jxz1pTuWYqJvgix\n8QzRqRyiFLmQxWR3eF75594jUzOTQykalSEkgOUq+l+ZmtFHJYIohShQBky3h0h2BGcE1HA82kbT\nhJL+MiYVtwlkXeyc5wecFeAquk6kUPilEz5z09mo3efM85U81v0qLJXHq/EO3plrLuemEMETWRQf\nKpLPN7jFIn6Gz0mnBIHqdk5bjh2/87oV1/FRSnHsq4/x8v/3kxqPqvXQ4t8ABnfuZHDnztqcXIRE\nV5hEV/ngr1Em1a34aCOTMYVCCYGZRBg1Mx7xID6RAYFkRwg8Rc+xaUx39pjodH7mPAsxYHF5hjlB\nV0fVKeZQmABEuUAhi2iueCpFd2YcS5UXeNs2yGYaX+Mf6iv8AOYKY0Nu1mH6yUEOfvrHNR5Ra6Kz\nfTYgmbC1ZFCzmZIxF8qKoaBtLEP38Wm6j88X/lLHL/f4/IPqGKz08rxp4AGuPngLITeLWdhBbHgu\ntufw+sGHlnx9ONL4n65l1V/4AYZvea5kds/cx5RSiCG8+Gc/rOfQWorGf4M0dWe6K4xnSMk9AXMD\nw82KAMGMS6Dg2in1fLPjiklPepS4k+JXD97KeSNPs3P6COeO7efdL99Cd3ZiyddHY7V1T4lAW3v5\nEhJ+cLcxjoOjn3+Q/Fhq0QQwdyISEcQy2Py+19R7eC2DFv8NiGcZDG+Pk47aJUW+WBKiSRpIlWUp\nkW/mcaMUm5MniBcatwe9HK8ee463HLufC0aeJuos7iq2ENOUJcs7V0ogCD19Nj19FrZddI35/1g2\n9PZbxNsbI/7OVJaHLv88bmrpdDURoefy3XUaVeuhff4bFM80SLUFCSXzJUU0F7aY7gwRHU0Rzpa2\nsJuZZh/vpvRwxeeIRk1SydrsqA0G/MqYbe0WbQ0S+aVQrsIMLj8ut8WKC9YTbflvYMx86Z4AHpCO\nBciHLOwVlI2eS1Nb3E2C7Tm05RNrfr1SihMDWX+DV42YnvaYnmpg38hl8LIObmppYVdKcfhfHqjT\niFoPLf4bmPh4trSFLJCO+amnlqNWZUU3u8VdV5SHKM+vjVBAPA/by7MzcWzNpx0ZypOYru00qxSM\njjgrKptQa6yOEDuvu4Tzv/1+zr7hKjov2QkKDv/rg2WLtCmlSOwfYviW5+o82tah+dZzmvqgFFYZ\nq14Uy+ehF4+t7qjWDeK59GbGeMPgg/y0fw+DkV4ANqeHeMPAgzP9eVeL5ymmJuuT4uk6NWlRvCqs\n9hAXfPcDWJ3hGTdP+55tvPy5+zj6rw/ScdF2Oi/ZNS/Yq5TCTeZ47KqvNGrYLYEW/yYmNjHBGY8+\nSvvoKCObNvHCeeeRjsWqc3IRXLN0NzB3TnmFbMgkmHHnibwCXEOY7A0TH0tj51e3OljXKEXAzfG6\noUc5ZeowBoq3HfkxjpiAWjJ3fyXUM7e/GrXrKmXL+8/H6gjP8++bkQAn/dEbGPzmUzz9kW9x1t9f\nSfdlp4Ah4BWE/z3/2cBRtwZa/JuU7oEBfvHb38Z0HAyl6D1+nDMffZRkPM4rZ5zBcxdcQD4YrOga\nU50h2kfT83b7egJTnbPnneyN0ntsCrw5WUACo5ujuEGLbDRA52CibE/hdcEKzV/Dczl94gC/MPTY\nor+FpSrvsZhJe3Utmdze2fjuXd2XnoQZWixTynGJndHH1KPHePb3byJ8Uift528lN5pi/L5XUE7j\nd0A3O1r8m5QLf/Qj7PxsKltxm398aoqzH3yQ0x9/nO+///1k4vE1XyPdFgTx+/CajsK1hKnOEJn4\nrPi7tsGJHe1EprLYOZd8wCQVD6DM2dXBZE8E+/g0RiE+sC4ngXITQOFxy3Noy01z4fCTa37/nqeY\nGHdITHmIAW3tJm3tvgAnEy4nBvJ1q+cjAp1djW7XDtkTCWKvUovKOYhlkh+brYSbfnmc9Mvj9R5e\nS1OR+ItIF/B1YBfwCvBrSqlFn4CIuMBThbuHlVLvrOS66x3DcWgfGyv7vAB2LscVX/0q3/3IR1AV\ndNZOx4Ok40uvIJQhfjmFMniWwdD2NtqHU0QSDSgVWk2K6jpTXEjNf27BBCAoNicG2TV9lN1Tr6zZ\nwldKcexwjlxu9nqjww6ppEtbh8XQYP2E3x8PZDMQjtTvmqU49qVH6Lx4F+ac3rxe3iX54gjpV7TY\nV0Kl2T4fA+5USu0G7izcL0VaKfWawk0L/zIow1hW0P1drhk2HzpUn0EthwjTXeHGmv1VUEdRLu25\nqdkMnYWNW+Zcw/Iczh57niuO3sOZky9V5NpJTHvk8/PHrxSkkorBY3m8BngxvCZodzj50FFe2vdj\n3FQOZzqLm86TeOYEz/zudxo9tJanUrfPlcBlhf9/Cbgb+B8VnnPDowyDV844g53PPYdVov7/7IGK\nviNHOH7SSfUb3BJ4lsF4b4SO4dnluKjZKqHFmEHV54eiRb7CipllzwHsGX6KR3vPASkx+RbOb7s5\nQm6Wc0af44zJlyocvE867TVbh8uGNGMvxeA3nmTopmeJnt5DfjxN5oguzVwNKhX/fqVUsfnsINBf\n5riQiDwMOMA+pdR3Sx0kItcC1wKE4r0VDq21efiyywgnEvQdPYpZmAAWSpoApz/xBJsPH+aeK68k\nMj3Na+69l67hYdLRKE9fdBEvn3VWXcediQU4EbEJpn33TzZsY+VdOgeTi4qwVYwqtANb0Gt36deU\nnxyuPnAzk6EODOVRbsoNeHnecvReNldhh+5crCaLvnX3mJhm80RvvEKFTk31WPYrJyI/AjaVeOrj\nc+8opZRI2QaCO5VSx0TkZOAuEXlKKbXIZFJK3QDcANDWv7vJ7KD64to2d7/rXcQmJug9coQL77wT\ng/kTgACm69I+MsJbvvENQqnUzEohPjnJa++6i0A6zfMXXFDXsStDyERny+7mg9aqAsHl9hDM/UJs\nbk/C8WkGo6W+mqVO6tGWS5Axg+SsxTGOoJujzU1jZP1uWuVwxKQjN7Wya66CtnaL8dHKM4IqJRQS\nunoswpHGB3s1tWXZdZ1S6i1KqbNL3G4CTojIZoDCv0NlznGs8O9BfNfQeVV7B+ucREcHL59zDiNb\nt5Y9xgCi09MzK4QiluNwzgMPIEu5juqEW8aKVCVuuaCJYwle4SWe+LeRzVEGd7Xz7//q8vYHvsfW\n5ImZUsjzTzrfbrA8hzcOPMjVL/vVM60Fr7E8h3NH9wMQc1LsnD5a8ryG53Da5EHCbnZ1b34FWJZg\nNoH1v3VHUAv/BqHSr9vNwAeAfYV/b1p4gIh0AimlVFZEeoBLgL+q8Lobjp6BgWUt55INTDyPcCpF\nqoKU0GqQ6AzRMZRatKcgE7FIthUscRHyQXPGtx5O5gmkHRzLIB0P4FmG33i90H/39MmDPNV9Jq7y\nZnz0hudPdKbnFfYkCKdMvsKpU68A8KqJF8mbAZ7sPoPiX+xVY89zzvhsGYBLB37OE7mzeKbzNHKm\nDQpCboZzxp7j7PEXKv5bZLMeqYSL4yrCIZNIzMAwBNOcbZDSCEyt+RuKSsV/H/ANEbkGOAT8GoCI\n7AF+Ryn1W8CZwL+IiIdvpO5TSj1b4XU3HK5lYebWUKFQKTLh8l29woUVQ6K9vabbOTPRAFNdHm1j\nfrliATJRm4neSOnripCOBUjP6dr0zdT13HvJ7P2wm+WXD93BT/svYDDSj6FcTp08xGuHHmck3EXa\nCtGXHqEtn5w9LXDe6DOcO7aflBki7GYW7bo1UJw3+gznjT5T1b+BUoqRYYepidmV2BT+tQMBYS0f\nbzXp6mmCpYembkgzFG4qRVv/bnXRr3+20cNoGs6/5x5OffLJpbN/FlC0fI/s3s3PL7+czqEhLrj7\nbjqHh8kHgzimSSidRhkG2VCIn+7dy/C2bbV7EwBKYToenmn4DdlXyI37Ejx57n+UPy3Nv7ksnXI5\nfrS59kEU593OLpOOLqvhO3o1lXPp87c+opTas9xxeqpvEZ645BLaxsZmsn9W8hP1G5grtr70Ehff\neiubDx/Gcny/QjCTIVA4Bs/DSiS47Kab+P4HPlC9+kElByW49ur9C0sJPzSn8GfSHlNTLspTxOIm\nienmKpG88+QAnguWLRirmIg16wMt/i2Ca1nc/a530TY6ys7nn+eMRx8Fw8BwHAxv6WYrluuy7eDB\nkm0b52J4Hqc8/TRPv+51VR792tmyN801132p0cNYNWOjeSbG3JnYczLhsbB3fCOxbLAsQyvABkZ/\n9C3GVHc3T118Mc++9rVsOnyYnuPHOeuRR1b02uVSu0zXJTo1hZXLseOFF4hOTzO6aRMDO3dWVEKi\nEv7duYv7GnLltePk1TzhBz8ByWt80tUMnV36p7/R0d+AFsW1bY6dcgontm1bkfivZHdt3raZ7uzk\nyn/7NwzPw8rncWybqc5OfnT11bi2vcSrq0vR4r/v1vpcTylFLqcwBOxAZRNdKlVa5ZXyLW6nzm5/\nw2BeeYi2doN4WxMtQzQNQYt/i+PaNkoEWUHgftmG50qx89lnCWRnO3zZ+TwdIyO8/ctfJhWP8/KZ\nZ3LwrLNQNc4L/OIFKe5d8JjnKZSi6jtPJ8byjI3OWuqG4efd20Ghs9MiGFrdZGAY0jTtLEVgy7YA\nIuA4ikDQwLK0f1+jxb/lUYbBWF8f3SdOVBT0FHyff/v4+KLzmJ5HbHqa2PQ0XSdOsOP55/nxu99d\ns9TQubn84LtRTgzmyKR9SQ0Ghb5NNoGgQTbjMTyUJ5tRGIUyyF09K8taUUpx/OjseYt4HuRy/kog\nlcjRv9kmGlt6slNKMT7mMDnuLlmErd5WP+KnkYohBCpr/6BZZzRH5SZNRTzwlrdU5TzmMoFj8IPH\nm44e5awHH1zyuOjkJLGJiVVX2rxxX2Ke8CulOHokO0+gs1nFsSM5MhmXY0dyZDP+c54HkxMuQ4Mr\nU9ipCXeR8C9EKRg+kV+yl63nKQ6/kmV8dGnhrzci0NNrLaqFr9GAtvzXBZO9vTx90UWc9fDDGCtM\nA12K5WIDApz7s58xuGMHY5s3F16k6BwaovvECc549FEiiQQA6WiU+9/2Nsb6+ghmMjiWtWTsQK77\nAjC7kSuV9EoGSpWCkSFn0dyilJ9Z4zhqSfdGPu8xPray1EvPA8eBUsN2HMXIUL7+Fn0Z7IDvdrIs\noaPTaprKnJrmQ2/yWkd0Dw7y6vvuo+/YsZnOX6tluXTQhccdPflkXjntNC666y6sfH4m9jD3dY5l\nkY5EZiaEYyefzINvfjO5OTuPP3rdJB17/2vmvucpXBcS0w5jI6UDqAsDmXMJhYT2TotozFjkAho8\nniWZWN3fp6vHpL3DxChkPSmlGBlymJpyF//RGoRhwLYdgYoD1prWZqWbvLT4rzPMfJ53fOlLhJPJ\nNU8Aq8GbE2xeaqJY2AAeIBmP89gb3sCR007jm6nreeLuAEophk/kmZ5a2n8iAsGQLOu2CQRh89aA\nn9MOjI3kGR9be85lvM2gp89matJlbGTxyqMRiAFtbQYdXbYO5mq0+G9kgqkU599zD7uef74pd77O\nxQOCAejqtonGDIYG8ySml3ecmyb0bbYYPLYyAe7oMunsMnn5QOUFdJZacdST4gS4eWtA79DVzKDL\nO2xgspEIP7viCp4//3wu/sEPiE9MAM1ZAsEA8jk4MZAnGpMVu2NcFwaOOitOOJoYc5mowOKfS6OF\nPxY3CIUMgiGDYEh0PR7NmtDOwXXMWH8/3//AB7j7yiv9QGuDdumudG2ZSq5+FdqkC9eaYRjQt8mm\nvRDM1cKvWSta/Nc7IgycdBLf+9CHePqii5iOx+sen1xxB68NJuRrYeuOgBZ8TVXQ4r9ByESjPHPR\nRXzvwx/myCmn4GkBaTn6NlkEdCaPpkpon/9GQ4T73vEOth84wJkPP0zX0BCiqtxYXVMRpuV39DJN\nCASFaMwk3mbqoK6mqmjx34gUGrwc2b2bYCrFWQ8+yO6nn0ZcF0NPBA1DBKIxg/7NgeUP1mgqRIv/\nBicbifDYZZfx2KWX0nf4MBfffjvBVApTO+DrimlBR4dJe6f+SWrqg3YganxEGNq5k1ve/35ePPdc\nPGmeypTrFSkUXduxK8Cuk0N0dNk6mKupG1r8NfPIh0I8+qY38cP3vQ/XMPQEUCMiEWH7zgDbdwV1\nOQZNQ9DfOk1JJvr6uP2978VbsDdgqclATxQrQwTi7ZYWfU1D0d8+TVkm+vq4+1d+hUQ8jmOauIbB\naH8/A9u2LUoVXa4S6EbBtJZucyAC4YhBNKZ/eprGoqNLmiU5sWMHN3/4w4STSRzbJh/0O4JsOnSI\ni+64g2A6jeC3gAxmMnpTdL8AAAi4SURBVBt2AohEhM4em1DImGneXuydaQeEUNCPoURjJpGo3pmr\naTxa/DXLI0I6Fpv30ODOndx0zTVEEgnyts22gwd53e23r+q0zVIgbUkEOrv8HPvR4dL1/4NBYfO2\n2TZZXd02HR0WuZzCtMC2tZWvaT70t1KzdkRIxePkQyEOnX46zioavItAe2dzNxEvZuN0dFq0tZsl\n/Vq+/37x+zBMIRQ2tPBrmhb9zdRUBc80uefKK7Gi1qzPW6Ctw6Szy5jnB5fC4x1NmNMejRl095p0\ndJn0b7bZtsMvl2wY4rdEXPA+7IAQb2vuSUyjKUXz/fo0Lcs/dn2bxzbZJJOC5yrCUWOmFk007pGY\n9rteReMmoZD/eFePWbZT10oQWVwQrijK+ZxaUbG44s7arh5rSUu9vcMiGDSYmnRwHL+0ciyuyy5o\nWhMt/pqqUOzEZZiUtISDQYNgcLGwdnbZBIMGE2O+oIbCQmLaW1a0wxGD9g7/OpGIQTarSEy7iAGx\nuEnAFoZOLG4MUyrO6jd6sVYUhA2FDUJhXX5B0/po8ddUzG1vuYX7/njtghiJmkSisxNGb78imfBI\npTxsS5FMKnLZ2dnAsoS+TfNbFobCsqhZef/mAD19CsfxcPKQzSos0195GAZ4LhgmOvNGsyGpSPxF\n5Grgk8CZwIVKqYfLHLcX+FvABD6vlNpXyXU1zcOWvWnuu+5IVc8pIsTiJrG4PyF0dCmyGUU262Hb\nQjiy8lRJ0xRM0yQYhGhswXPa9NFsYCoN+D4NXAX8pNwBImIC/wBcAZwFvE9EzqrwupomYMveNNdc\n96WaX0fEt+rbOywiUVNb6hpNFajI9lFK7Ydll80XAgeUUgcLx34NuBJ4tpJraxpPPYRfo9HUhnos\nfLcCc/0CR4GLSh0oItcC1wKE4r21H5lmTXz0ukk69v5Xo4eh0WgqYFm3j4j8SESeLnG7stqDUUrd\noJTao5TaY4fbq316TZW4dJ+2+DWaVmdZy18p9ZYKr3EM2D7n/rbCY5oW5MZ9CZ44V6c6ajStTj12\n+D4E7BaRk0QkALwXuLkO19VUmds/PsqT5/5Ho4eh0WiqQEXiLyLvEpGjwC8At4jIbYXHt4jIrQBK\nKQf4A+A2YD/wDaXUM5UNW9MI7r3kxkYPQaPRVIlKs32+A3ynxOPHgbfNuX8rcGsl19I0jts/PqqF\nX6NZZ+jCbppl0cKv0aw/tPhrluQTt/5zo4eg0WhqgBZ/TVm+mbq+0UPQaDQ1Qlc30SyiuInrCXRK\np0azXtGWv2YR73j2vkYPQaPR1Bht+WvmceO+BPedW90qnRqNpvnQlr9mBr2JS6PZOGjx18ww9fH/\nbPQQNBpNndBuHw037kvw5Ln/oQO8Gs0GQlv+GuS6LzR6CBqNps5oy3+D84lb/1lb/BrNBkRb/huY\n2z8+2ughaDSaBqEt/w1IsffuvbrUnkazYdGW/wbkby9zGz0EjUbTYLT4bzCKmT0ajWZjo90+G4iP\nXjfJk+fqxusajUZb/hsKXbNHo9EU0Zb/BqAY4L1PB3g1Gk0BbflvAK7/9r82eggajabJ0OK/zvlm\n6nqeuFtv4tJoNPPRbp91zI37EjxxrhZ+jUazGG35r2N0SqdGoymHtvzXIcU2jBqNRlMObfmvM7Tw\nazSalaDFfx2xZW9aC79Go1kRWvzXEV+8INXoIWg0mhZBlFKNHkNJRGQYONTocVSRHmCk0YOoMfo9\nrg/W+3tc7+9vp1Kqd7mDmlb81xsi8rBSak+jx1FL9HtcH6z397je399K0W4fjUaj2YBo8ddoNJoN\niBb/+nFDowdQB/R7XB+s9/e43t/fitA+f41Go9mAaMtfo9FoNiBa/GuEiFwtIs+IiCciZTMLRGSv\niDwvIgdE5GP1HGOliEiXiNwhIi8W/u0sc5wrIo8XbjfXe5xrYbnPRUSCIvL1wvMPiMiu+o9y7azg\n/X1QRIbnfG6/1YhxVoKIfEFEhkTk6TLPi4hcX/gbPCki59d7jI1Ei3/teBq4CvhJuQNExAT+AbgC\nOAt4n4icVZ/hVYWPAXcqpXYDdxbulyKtlHpN4fbO+g1vbazwc7kGGFdKnQr8X+Av6zvKtbOK793X\n53xun6/rIKvDF4G9Szx/BbC7cLsW+P/buXvWKKIojOP/B0TT+RIhxhfEgGArBBHtjJVFomhhZYoU\n5juksxH8AjZpYhPBNEYQhBDFKqiNBA1qksaEGEEhZVQ4FnuVwZhkzO7O7DrPD4a9M3tZzuHOnN25\n3Nm7BcTUMlz8myQi5iLi3TbdzgDzEbEYEd+A+8BA86NrmAFgLLXHgMslxtJIecYlm/sE0CdJBcZY\nj3Y/73KJiOfA1y26DAD3omYG2Cepu5joyufiX64jwMfM/lI61i66ImIltT8BXZv065D0StKMpHb4\ngsgzLr/7RMQPYA3oLCS6+uU9766m6ZAJSceKCa1Q7X791cV/6VwHSVPAob+8NRIRD4uOpxm2yjG7\nExEhabOlY8cjYllSDzAtaTYiFhodqzXUI2A8ItYl3aR2l3Oh5JisgVz86xARF+v8iGUg+4vqaDrW\nMrbKUdKqpO6IWEm3y583+Yzl9Loo6RlwGmjl4p9nXH71WZK0C9gLfCkmvLptm19EZHMZBe4UEFfR\nWv76ayZP+5TrJXBS0glJu4HrQFushkkmgcHUHgQ23O1I2i9pT2ofBM4DbwuLcGfyjEs292vAdLTP\nQzPb5vfH3Hc/MFdgfEWZBG6kVT9ngbXMNOb/LyK8NWEDrlCbQ1wHVoEn6fhh4HGm3yXgPbVfwiNl\nx/2POXZSW+XzAZgCDqTjvcBoap8DZoHX6XWo7Lhz5rZhXIBbQH9qdwAPgHngBdBTdswNzu828CaN\n21PgVNkx7yDHcWAF+J6uxSFgGBhO74vaqqeFdG72lh1zkZuf8DUzqyBP+5iZVZCLv5lZBbn4m5lV\nkIu/mVkFufibmVWQi7+ZWQW5+JuZVZCLv5lZBf0EX44eGCk0HvUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a12f438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#prediction\n",
    "def predict(X):\n",
    "    pred = np.argmax(soft.forward(lin.forward(X,W,b)),axis=1)\n",
    "    return(pred)\n",
    "\n",
    "plot_decision_boundary(predict,X,y)\n",
    "print('Accuracy: {}'.format(np.mean(predict(X)==y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Linear > SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0: loss 1.9887632988593225\n",
      "Iteration 100: loss 1.0564541314444627\n",
      "Iteration 200: loss 1.0561157496976274\n",
      "Iteration 300: loss 1.056073754196578\n",
      "Iteration 400: loss 1.0560668301389107\n",
      "Iteration 500: loss 1.05606046965717\n",
      "Iteration 600: loss 1.0560572428193047\n",
      "Iteration 700: loss 1.0560557015821912\n",
      "Iteration 800: loss 1.0560556670293322\n",
      "Iteration 900: loss 1.056055158287627\n"
     ]
    }
   ],
   "source": [
    "iter = 1000\n",
    "reg = 1e-3\n",
    "step_size = 1e-0\n",
    "\n",
    "#initialize weights\n",
    "W = np.random.randn(D,K)\n",
    "b = np.zeros((1,K))\n",
    "N = X.shape[0]\n",
    "\n",
    "#Architecture: Linear > SVM\n",
    "lin = Linear()\n",
    "svm = SVM()\n",
    "\n",
    "for i in np.arange(iter):\n",
    "    #forward pass\n",
    "    scores = lin.forward(X,W,b)\n",
    "    probs = svm.forward_correct(scores,y)\n",
    "    loss = svm.hinge(scores,y) + reg_loss_calc(W_list=[W])/N\n",
    "    \n",
    "    #report loss\n",
    "    if i % 100 == 0:\n",
    "        print(\"Iteration {}: loss {}\".format(i, loss))\n",
    "    \n",
    "    #backward pass\n",
    "    dscores = svm.backward(scores,y)\n",
    "    dW, db = lin.backward(X,W,dscores)\n",
    "    \n",
    "    #update weights\n",
    "    W += -step_size * dW\n",
    "    b += -step_size * db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.525\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsvXeYZNV95/05N1SuzmF6Us8MDDmD\nAIEkwEJoFCwMMhLYlpXldQDbev2uta8k22svFrbXfmWQVzaWbGVLgJCEyAg0iCDCgAgzDDPMDBM7\nTeeqrnTD2T9uVceqTpW7z+d5+unuW7fuPd1V9T3n/KKQUqJQKBSK1YVW7QEoFAqFovIo8VcoFIpV\niBJ/hUKhWIUo8VcoFIpViBJ/hUKhWIUo8VcoFIpViBJ/hUKhWIUo8VcoFIpViBJ/hUKhWIUY1R5A\nIdpao3LThrZqD0OhUCjqihdePjgopWxf6LyaFf9NG9p4/tG/rPYwFAqFoq7Q2j5+aFHnlXsgCoVC\noag9lPgrFArFKkSJv0KhUKxClPgrFArFKkSJv0KhUKxClPgrFArFKkSJv0KhUKxClPgrFArFKkSJ\nv0KhUKxClPgrFArFKkSJv0KhUKxClPgrFArFKkSJv0KhUKxClPgrFArFKkSJv0KhUKxClPgrFArF\nKkSJv0KhUKxClPgrFArFKkSJv0KhUKxClPgrFArFKkSJv0KhUKxCalb8HV1UewgKhUKxYqlZ8d9/\nVOeqm1v52AshxqL+ag9HoVAoVhQ1K/45eh4Mct3nInzshVC1h6JQKBQrhpoX/xw9Dwa56uZWtQtQ\nKBSKElA34p8jtwv4acyo9lAUCoWibqk78QdvF3DbrY1cdXOrmgQUCoViGdSl+E/ntlsblVNYoVAo\nlsiKWDb3PBjkugcBItx40xi/HrWrPSSFQqGoaUqy8hdC/IcQYkAIsbPA40IIcasQYp8Q4hUhxHml\nuG8+cuYghUKhUBSmVGafbwDb5nn8PcDW7NdngK+W6L4Fuerm1sk8AYVCoVDMpCTiL6X8BTA8zylX\nA9+SHs8ATUKIrlLceyFUiOjqwXVBymqPQqGoDypl818HHJn2+9Hssd4K3Z/rPhdh7Tadb5yfqNQt\nFSXGdUEI72s6ff3wzW9pvL7H+31NJ3zsd11OPrnyY1Qo6oWacvgKIT6DZxYiEG0v+fV7Hgxy1YNB\nAO68JU5jLF3yeyhKz6HD8K1vaxx4E0wT3nap5PoPSXw+iMfhb27WmJgA8GaF3j7JLf+gccP1kquu\nVFsBhSIflQr1PAZsmPb7+uyxGUgpb5dSXiClvMAMNpZ1QKpkRH0wNARfukVj/wGBlIJMRvDEk4Kv\n/B9P6J98SpBOQU74Pbxzf3CHIBarxqgVitqnUuJ/D/C72aifi4ExKWXFTD6FyPkDVMZw7fLIowLb\nmXnMsgS7Xxf09Xu7AtvJXwFWCNj1mqoOq1DkoySKJ4T4L+ByoE0IcRT4S8AEkFL+K3A/8F5gH5AA\nPl6K+5aKngeD3PZgkNuAhz8/VO3hrFgOvAk7dwoCAbjwQknTIjZ3hw4LbHuugBsG9PXBpm549jmJ\n6849R9PA51NmH4UiHyURfynlDQs8LoE/LMW9ys1VN7dy401jACpZrEiSSbjvAcGTT3nmFye7gjcN\nuOtuwe//nsu558x/jc2bJG+8wZwJwLZh7VrYulXyk3sEiaRkpunHE/8zzijhH6RQrCDqvrxDObjt\n1kaVLFYktg03f0nj/gcEo6MCxxF44iywbM92/6+3a6QX8Lm/650S0wCYWsGbpuTMMyQd7RAOwf/8\nS5e1XblzJLou8fslf/rHLj6zXH+hQlHfKEP3AuR2AmoXUJi+PphIwIYNTIrt07+EYz0gZWGbu5Sw\n+3U45+zC125uhi983uW739PYs1fi98Fll0mu/Y2pyaC9Hf72f7mMjcGu3QK/T3LmGeDzleovVChW\nHkLWaFZMQ+dWedFv/1O1hzGDG28a4x1rdRUimmV4GL58q0Zfv+dctSwIhaC9DXr7IJVayNkqaWuD\nT33CZf9+wcM/EyQTcOKJcMP1LhvWV+TPUChWFFrbx1+QUl6w0HlK/JfB2m3JVZ0s5rrw7HOCb3xL\nZM02s0U+955aXKSNpkk0bbpdXxIIwF//lUtH6dM9FIoVzWLFX9n8l8FqLhkhJfzj/y/4+n8K0umc\nHX82hY7nx3VnR/QIMhl44EEVpqlQlAtl8y+C6z4XASKrZieQTsP//ifBG/uWJu7LwXUFBw7AdEev\nQqEoHWrlXwJW4k5gfBwe2y546BEvmQrgW98W7NtffuEHzxS0bp0SfoWiXKiVfwlZKcXjnt8Bt39N\nQwjPvn/XDwVXXSl59nkxb/TO4pkZk69pnshPT9QyDHjfe5T4KxTlQol/iZlePK4eQ0QnEp7wW9ZM\nkX/w4akkreUj0XU471zJy694/gPDgPe/TzI0BE886eUHdHXBR3/HZd26Yu+nUCgKocS/jNx2a2Pd\nlYx48VcCO8985Ql/Mat+L6Lni5932dQNti2JT0A0ArrunfE7vyWxHVRilkJRAZT4V4BcpnCt7gSk\nhMd/Ibj3fsHQUKGGKPmEf25JhbmPT7Gmc+psw2BObR9NA5/yQikUFUF91CpIrmRErTmG771P8L3v\nCwYHczb9xa/wdV1iGF5ZhcJ41+zphS/9vcboWJEDXo0Yfoh2QNM677uh0pcVxaFW/lXgus9FuPEm\nz4BeyZ2AlPDozwUPPSSwLHjLWyTbrpL86Ccib1XMhfD5vLo6+/YJnvol7N/vOYgNw/sOzPIdCBxH\n8vjjgqs/sEKcuWYQzABIF9JxcIt2jOS5RwCi7QjhrdWkpoPZCePHwU6V/n6KVYES/ypx262ezeM2\nyt9VTEqv6cm3v+slT+VW9o/8DH72qFhW31vTlGx7t6RrDXStkbz9bXDokGTXbkE4DI4NP7hz7oRi\nWYLevhUi/A2dYPgQQkNKCcEGiA2ClSztfULNk8IPIIS3k5LhZhgr0BbD8IOmg50uz4SkqHuU+NcA\n5Q4RvfOHgocfyVcXX3iitWRHruSM0yTXXD1TxLu7obvbO3bkaP6ibj6f5IQtS7xdLeKPgO6bFOVJ\nQY60wciR+Z87G18IAg2eWFtJSI5NCbbhB72ABzzfcU33JiWhZ19WAVYKYsdRCXOK6Sibf42QSxT7\nacwoqU8gHqeA8OdYTgSP4I39Yk4j9elsWA+nnOztEHJomiQY9Hrw1h26Cfo0O7s/jNAKfHyiHdCy\n0fsKt4KY52MWaIBIK8L0I3TDm1Qau0D3T9n3CyHdWQeEd75mIDQNITRvUjID0LzOmxgUiixq5V9j\n5MJDS7ETkBK+/p/5QzeLJR5f+Jwb/9Dlp/cJfvELyFhwztmS6z7oTQB1g25mBVXLLpylt4rWCzhc\ns2IrsjOj9IfB9MNoz9xzNQNCTZPnek8XSDRo7Jz8PR+TBRmFBpFWz/cw7Rozh5S9ZrQdxgemJg1N\n93YWruOZhxSrCiX+NUouWezOWzyVXY5PYN8+2LmrPOUY5lv15zBNuPY3ZtberxuEBqFmb4Wf+2Oz\n32TDPKtxmCvmmgGRdogPQiDqmXl0o+COQAjPHFdI+CfPQXi7AyHmPXfyfN3nnR87DoGwN47cSyMd\nGO9X/oFVhBL/GscrHgcQWXSy2NiY95l+/gUvqmfxLLYUs6R7Yx0K+kJoumd20QzPVKLpBUS1sNjm\nOy6EQPqC0LJh3vMWus7cczQk808Sc8YBkJ28hBBTE5oU3gQ13reoaynqHyX+dcRCXcV6++Cr/6bR\nk7UwuLNNwgugafM9J+cYlpgmfObTK0z8DX9WFMWCK+/FrMxL8ZzFXrcU5wshkIbpTYBq9b8qUOJf\nZ+R8AjAzRDSd9nrmTkxMj7JZSiSPV35B18m7WxAC1q+XnLBFkkzC396iYejwjndI3v8+Wf8lGSJt\necIp87PcBkilFv7SI7KmqDziLzTPTCQEZJLg1l6mumJpqGifOua6z0X4acybv194UWBbs8MrlyI2\nXkRQITORrsP/8ycur+4U7NghiMcFo2OCBx4UfPmf6/xtpBlLjoSpfSFfJo1dU5FKkTYvZNQX8qKF\nQs0QaoKmLi+KKRvuqqhP6vxTq8iVjOiZMEhnSnXVmStbw5Ccf57klVcF8Tg407KBLUvw2m74i7/S\nOHioVPevAEbAE7eGTk/UlsBKFX6RdRznvjzRX+uFogptWvioBv4whJu9qKSo6rVZjyjxXwH4kjZ3\n3w1uSWrte9cQQhIIeHV7zjpT8omPSfbtJ9u6ce5zDh8RfOnvNHqr6S/U9MWt4ANRaGgHXwhhBsAX\nXLGCXgy5xLV8O0hvgshOBEbA+58q6gpl869zguNpGoeSCFnKgE6BEJL/8ecuTY3QmK2+uWYN+ExJ\nxsp/J8uC++4TfOqTFXAG+0JecTPHBjvjxbrr2bezY3thlY7lxb8bpncsk8iGcDYt2r6/2llU1JGm\nIf0RSMWmDppBL4LASiv/QI2ixL9OMdI2DcMp/Em7LE0VNQ26N8489vZLJT+9V4CV35HsuoI3D0FZ\nywgI4dmlNWMygmbqoWxilW5Cw5qp8gY5pAOpibxjVxRJziHsOlkz0LTciFQcEiPVHJ0iD8rsU4eY\nKZu2nnjZhB8k5507V8CjUfgff+6yfp13zmyEkKwvd9/dUPOk8Hv3nPl98uesyWKGDVvoEGxQK/0S\nI6X0Vgs5H0rOP6Dl/AORGRnIitpAiX8d0jCURCupmWc6Ep8PPv7R/CK+cQP8r792uejCXB3/KUwT\n3v/eMov/9IzbLItNuJqcBBQlZcr+L/L+j4WmeT6BnH8g2OA5jNUOrKoos08dIByX8HgaX8rBNjV8\n6fIm4XzoOq/+TioFhw5DQwN0rZl5zqc/KWlugp9v9+r2rO2Cj/yOy4YNeS9ZQhafzbqayZnDaub/\nYAagef3U71J6u7jxfs83o6g4JRF/IcQ24J8BHfialPKWWY9/DPgH4Fj20FeklF8rxb1XOprt0n40\nhpASTYIscan42QgBJ2+VPPyI4K67Bbru9e9d2wV/8sfuZOtFw4DrPyz58IckrjvVh3fZN11s4pRr\nFy5xnKXmhC8P5cj2nXUHkBKJVhP/hzljyPlrIm2FexIoykrRZh8hhA78C/Ae4DTgBiHEaXlO/YGU\n8pzslxL+RdIwnERzPeGHqcC7chlXdB3GxuGuuwWZjCCZ9L4fPgL/fOvct4sQRQh/qMmrd9O8wSs4\n5luEXXhieNkZtpVASjn5NR9lF2TXAdepCeEvhBDCi9Ayg4urFKgoKaWw+V8I7JNSHpBSZoDvA1eX\n4LoKwJ8o7NQttQRKYMupgoce9gR/Oq4rONYD/f0lulm4BQLRKVuxbnirQGOBXgZWygvbzDf+rOhW\ny7YvpfRKIw8fhlSspJPUYq8lpUS6rhfqaqdreqL0EN7r3rwegk3VHsyqohTivw6Y3rroaPbYbD4o\nhHhFCHGXEKLsluGVgtQKODPLcTMBTw1FODKa/22h6xBbRB3/he8jso5bbdZhDYKN8z9X94Gu5xW1\najp0pZTeans8OzsmR8G1kXMarix8HSndad+9r8WYxaSUXiew0WNe7kNybOr4jOvXzoQghJiKCgpG\ns45gRSWoVLTPT4FNUsqzgEeAb+Y7SQjxGSHEDiHEDiv7xl3NCMfF0Quv8EspcxIY6gxjBQwOx/3k\nkyzX9Tp0FY2Yx06Uz54favbMQy0bobFzqkNVlZg07bjulJimJ2CsZ/pJMNoLiVGklX8FPvuYlNIr\nmjZ8xLtWYhQmhmHkWN4Xe7aoIyXEh6aatbiO10TGsabG6ViQnVxqDSE0r7OZoiKUwuF7DJi+kl/P\nlGMXACnl9EL0XwP+Pt+FpJS3A7cDNHRurb13ZyWRkvZjcXTbrVhAXCbovR0mGv2EYhmY5mvw+Tzn\nrr8UHSYLZHxOmk2mE2nLll/IrVOqbxueTCaTrifUBZFe1msqBpFWpC80eRi8CUP6w14IpOt4iVCZ\nbPc2xwZnWsZschwZbJizW5oUccfyTD2zdxquPdehmk2Ck8z1PZTfEb0AuulNAOnY4oMAFMuiFOL/\nPLBVCLEZT/SvB35r+glCiC4pZe4d+AFgdwnuu6IJxi00p3LCD9DSP8HwmghS1zi+Pkp4PE0gYePo\ngsHGAOf/egqW0VEsL4kxZKhxUsw8EZOTpgpgsqJk7TotheejWEwLxPgQ6ONeyKPreCt85OIzX5Nj\n3oo92OD9X+yM91wn441jKeYlx4LUeF4Tm5c17c6ZZCqFEAIZavKCAayU93erFpNloWjxl1LaQog/\nAh7CC/X8DynlLiHEXwM7pJT3ADcJIT4A2MAw8LFi77vSMdP25Kp7OlMtVUq7BhZ4zmXdcnBMHalr\nxJuDxKcVvPS6ikVYuy1ZdH9hUuMgHWSwMStm6ayYTYv51g1P1OYzE1WTXOXLxYqTYxUX057bRcxh\nGSvkTCKv+EvXhVT+XUalmNxZmQFvsswkvF3N1AkQapnyD1gpzzymaggtCVGLtj/wzD4X/fY/VXsY\nVSM8miI6kso7AZQLFxjpDJMOL64zy3xdxUqCGYRoew2v/LOmn4kRSJfCE15hZpnUpHQ9c9NYrzep\nhZuzzV1mlc4oQLlMRpP+DCcN6QQEIqCbM3eN0vX8G0t0sK9EtLaPvyClvGCh81SGb42SiPqIjqRK\nvsKH+XcNtrn41d70rmKL7S+8aALRbPXN8gn/UsUq3/lCaJ6Zoh7FPz4I/ggyEPVW0+kJSI57j2US\nU1VQpfS+N+cL4vMop68gV6tJioBXHoKZk5BnqvIiyPLvjBT5ULV9ahEpiYymEdlVf9YaXtpb5Pk9\nEzRwfMszsVx1c+tkV7GiyVN2uVzMt/OdnrAlXYeCr8Ks1XFdkY57K/3Rnqy/ZdbfKF3vmHTy5i5M\nRhG5TtkjiOYL5RWaprqKLREl/jWIP2kTHk9PZvPmb6dRHBmfNjmpuMBE1GS4s7gY61xXsZ/GDMai\nRYQFGb6yVoWGxSZNSU8UR456XwWSyyYFcqWTGIFUfGb+QXLMS2qbGKJS/4N8E4B03azzW7FYlNmn\nBgmNZ8pu6x/pDBNM2OC6pMM+7GWu+PORMwfdeJOzPJ+AP1qRdP8Fm7TPblSeGEFG22fsSKR0vXj8\n1UJi2JsENH3m/8ZKeRNDIML0kIRymYKmm5kmI8Xq0fRWRdTKvwYRFXDCdx6NER1O0jCSpv2Y93Op\nye0ElrQL8IXBDNSAkzePmFgpiA0i7WzSlGN7USarTnRk/siaxAiM9XmTYaFdUilHkUtWs5Ke6apG\ng1dqFSX+NUgy4sMtk/blPh5CMtkTQEgIj6Ux0+X5wF73uQg/jRmL8wkEIp79toxMrhTzHM+VVSAV\n88R+NlbSy74dPuyVUUhPlHWsdUcuh0A3y7rq98he3wwsXBNKMQdl9qlBkmGD0LiOmXEqFuopJARi\nFpa/PG+J2271YspvA+68JU5jwWSx8kb3AF5Eiy+I1I1ZTuVcRm5cxYzXMHMnFYGMtDEZx+ZkvB2Z\nrXwA86FW/jVGaCzFmsMxr2GLBFsrvRttHkt3ie+Un+s+F+FjL4Tym4MyE+WNGhnt9YquxQbA8Qqv\neTV6svH6iVEl/KUgk6h4/aDJCrGG32snqam17Xyo/04NEYylaRhOzajdj5xmqinBPQqWhxaQilQu\nVK7nwSDXPQgQ4aabxnh/qwm6Pyu85chuwLuuzAq763h2Ys3wnJd2hlURsVMpJobB8CGn9VsuJ/l2\nAwQbvHEo8qLEv4aIjKbnmHk0WX5JksBEgx8rUPm3Q8gPZ/jXkAoKgoaWLYFcRoeHEfDs9jlcW630\ny0Eu4zbcgvRHZhbDy76+5ZwUhBBIFfc/L0r8awjdLpyaXoqPSaH1tKNBrHURXbTKwO9c5WNdm4bP\nyAmCVmZzgVrdV5SJYcgkvSxiTfeyhu0MRNtLdot82cUyVw5CURAl/jWE5dPxl7k5u8tMR48rIN5c\nHeEHuOI8A585u2RCmVaEQuSP4FGUFys5c7cFXoXSEvUXziv8SEiqUg/zocS/hoi1BjF74zNMP3La\n9+lv8eX6AayAjjltgok3BUg0VG977FayDpeUXlhgDUwAyeNJpAuhzupNvFUjW6unbEXgXMdz6K8S\nc5589pllPU+Jfw2RCRgMdUVoGE5iph2ELCzuuRzKpU4C6aDJ8JoImuPi6BoUaBNZKR5+3uK6K3z4\np63+XVdm9aEMY6tyBMj4/nEe/70nGNntZQVHN0U488YzaDqlidazWhBLeD16tvew59v7cBI2m6/d\nxOZrNqEZdRDAp/vK5tMXQiCdTHGls2uQ8c9/N+/xl7cvf+GmxL/GsAIGQ2ujtB3LhnvOw3Lq+juG\nhtQEjlYbNfL/61GLc7canLRBw8gOqWx+QEHVGoNIKXHSLvdue4DM6FT8+djecZ688Wn0oI7m0wh1\nhrDiGfSAQbAjQNPJTZx4/Qm0ntXCkQeOMvzaCI0nNDD82jB7/nMvdsJ7j/Q93c++7+/nXXe8E02v\n8QnAtcv2InsZvymvJHUu+7eOeOLSOws8UvrduRL/WmWRfsmlfoR8KZtktHaiIDIW/Mf9ab706SDC\nKGeCl+vV6qnAijATy/DG9/bT90Qfru0y/maM2IH57c9O0sFJOoyNTXUyix2IMfDMcfb9YD+G38C1\nXewJGz2o4yRnLgzshM3xFwY5+sgxNm7bMPvytUWwCSh9GejJQIFQM1Mx0tIzAVUh4auQOebJz87X\n+rNyKPGvURJRE3PIKfnOWHNcWnrjGJaD5dOJNQewy5TVu1jee7GJUeIhzIkYstIzu0EVSaI/ybHH\netBMwYar1uPL+k16n+zlZzf8HCdVOse9m3LJpKbEa7bw57AnbLZ/8heccePpnPNnZ9WmCcgXAn95\nWnPOvKaYXBnJaIdXlbVM5DPJFGOOqRRK/GuUdMiEodI7Jv0Je7JEtG7b+JNxRtuCOKaO5dcrUk1z\nNuGAQCux72FOBIjp90pFl2AF+Nrtu9nxP19EaAKhC552n+Ftt17Cnm/upe/J/qKvXwxuxmXX/3mN\niWMTvP22S6s6lrz4I3n7NJS3cbwo2tFf2BwD5TDJVAIl/jWKYZU2DCa3Dp7+scsVdWs+nkQKQAhG\n2kOLbuNYKh5/yebsE3WC/nJOPMKrGFqk+A88P8BzX9zhxcxO4/FPP1HUdUuJk3R480cHueCL5xHs\nqLFoooXKaE+eVuL3wiIaA8lnn6kZk0wlUOJfoyylneJiyDmHCz0mJCAlzQMTHF8fxTEr5xD++a9s\ntl1kcPJGnYBPlHkVWByP/e72OcJfi+g+nfEDsdoT//QE0vDNXf1LF2KDXitGw4csZVVQxyX2V99B\nTnjO33owyVQCJf41imPqpIIGgaRdMrv/Yq4jJARjGeItlRONrlZBIg0+o5zbfwmZ4sovj+wZJTVY\nH1mjVszCiNTgxzsdB18IafgRWq6cB54/xk55XwCBBmSoMfuk5ZeDkFKCrjF+zmXIlIUWNFnTOsDA\n/XtwkysrHHSp1OC7Q5FjZE2YNW+OVbQ7rAB0p3JL2/YmwW1/EiLkp+R2f/A+/K4Lb8YnOKFIk8+R\n++vLJLDnP/dyyT9eXO1hzCU2AGYAaQa8LL/MhJeYNZ3UuPflj0J4+f2cc83f1/3OuZO/u2mb7j9+\nG7+67jtk+ldbI54plPjXMkKQCRr4S7j6z1EoOcwVWWdzhbj2HSZ+szzCD96H/dUDNv/vVwXQysOf\nH1r2tQZfXf5zq8He77xB49YGjr8wSKDVz0kfOYmW05urPSwAxv/71+ccm22OCZ3YytnfuwFzmcI/\nnem7Bs1v4GsPs/Uvr2TXH/y46GvXK0r8a5xYcwBfKu7Z5EuMZQgMR06Wk3AF2D6dVAXF/7RNOmZZ\n4/slB3qnVpVX3dwKwNptSb5xfmLB5x9/cZAnb3qKsT3jZRtj2XDh+S++4P2swxvf3c/Ff38hW284\nsSK3LyZCRpg6Z33rwxhlKjMuhKDlsi1luXa9oMS/xrECBiOdYRqGkjMigIqVSwEYLoy2hwiNZxBS\nkoyYJBr8FQ33PNTnZLN7y7fyv/xck6/+eKZ9t+fBIFc9GJy3q9jo3jEe+PWHcDN14OFdCAccx+GZ\nP3+OTR/oxixRRNf8Ar98Wi7bjObTy9vSs8qlTaqNEv86IB0yOR4yQUraD49jOKXbBqQivoo2cZnN\nXdstrjjPnCztAKV3+jZHBOefrPHCnrkift3nIqzdpufdBbzy5VdXhvBPQ2iCpz/7DEbYYP2V69jw\n7vUzykHUSlaqry2MFizvDtQtU8/qekGJfx0RjGUwHFmy2v6JSGXj+fNxeEDy/92e5E+u87O2TUMI\n0MuwIvvSZ4I88KzFl+/MMDv5N7cLgJn9hY/vOF7ycVQbe8LmzbsPArDvO/vwBwRr1/tqL7RWF3mL\n3EkpkbbrJfDqyy8JLaWk74c7ix1lXVOD+d+rF812aRhM0H50nOa+CczUtJWJlDQMp4oW/lwlUNsQ\nxJoDRV6tNLx6wOWTf5fkms9P8A//lSaZLq2DQ2TLB7/nIpMv/9H8f3OuvzBAqCtU0nHUGlJCKikZ\nHbZxSribLAUNZ3blFXbpSt74m0fZ89/vL+r60nI5+E+/KOoa9Y4S/xpBtxw6jsYIj2cwMy6BhEVr\nb5xA3AtPFK5Ec4v/gOY+ToYtWXNonNZj45hJmznL4SqQynjZvseOu2Ss0o9HCMGpm3Q++T6TC0/V\n6WwW6BpsXiNoikwJTc+DQa66uZW9/flr6Fimn1hDC7ZukAqGiUebKLgfExDdEqH5zGaMBgNfkw+z\nqfo7rukMDzkcPJCm52i6ZiYBZyKDm6eznZuwkEmLruvPASmX1fVNupKBe3fjTKg4/6IRQmwD/hnQ\nga9JKW+Z9bgf+BZwPjAEfFhKebAU914pRIdTCHdKQnJZt42DSVJhE6kJpKAkUT/TZcqXdmnrjSOF\nF1k00VhZh+9sbAde2ufQvUYrS7q/EIIP/5qPD//a3McmUpJ7nrI42CtJJV1OP9hK1x+di3XJGXz5\n5tGZJ+cTHSnxpRJseW0HZ3TFePtX3kbjiQ15x9HzeC+PXP8o0q4NsUVCMiHpO5Zh3UZ/tUdD3907\n6bz2DMhTnG7tb59L5PTO5Ttp/trrAAAgAElEQVSDBRz6l6eLHGH9U7T4CyF04F+AdwFHgeeFEPdI\nKV+bdtongREp5YlCiOuBvwM+XOy9VxL+VP5YfiElmiNxDY2JRj/hsblN3othcrKREB3xsisnmqpn\nDjpxncb7LzHLGv5ZaCIJB+CGd05zfn/6OgYHLP7s9wbyXSTvsUwowusXXEZPg6R9WOL8Ek48ETqm\ntazNRchs2eJnZNhieLC8rTuXQiolyWRcfL7qGgXiu/o5+OUn2fzZt+NazmRiyt6/fJiTb96GVkz5\nEcmqTu7KUYqV/4XAPinlAQAhxPeBq4Hp4n818FfZn+8CviKEELK8nbrrClcX6Hm23EKCzDq+cjb6\nyKjnkCx5uWcJ0dE0Ew0+fCmHQMJCCkEy6sP2VabWz9vOMjCrFIaQb1L4579dThlowfg4fPVfp67X\nlB5l0/gR1if66GDqtWtuMQmFNI4dsWrB8gaAbUt8NVD+5tg3X2Dg3t00v7UbJ2kx8uRBIqd3Ios1\nfwpofddWBh/YU5qB1iml+JitA6bHgR0FLip0jpTSFkKMAa1A6Qqs1znxxgCNg4kZq3oXJk0+AAhB\nrCVIMmzSdixelrIPwpU0DSS8mkLZsUTG04y1BEk0lt8c4DiyZkQQoPfYckM9xYzZedTfxEvtTbzE\nmRiuwxU9T7FxohcAf0DHMG2szMw/XAhoatGxLYdYBXPMfL7aifyxhhIM3Lt78veJPQPFrfqzdN90\n6aoX/5py+AohPiOE2CGE2GElxxZ+wgoiGTGZaPQjhZdpKwVkggZj7TMjTvwJi7ae8gg/gKsJAgkb\nLds/eNL3MJxEq0DNn8dfsnFqxwpCS2uJPiLZGjMIga0bPLrubQz5myYf7lprohte5eHcqdEGneYW\ng441AbZs9aNXqNBqKllDs+8snAmL/ntfW5ajN4cQguCGxoVPXOGU4p19DJjeN2599ljec4QQBtCI\n5/idgZTydinlBVLKC8zgKntxsqv6vu5Ghroi9G9oYLgrMrXqB5CSxuOJSWGePMyiuz7OiwQsU+R9\nU0jAnyx/UszhAcl/3p8hbUlsZ3nRHKXkT77QVpbrukJjZ8vJk7+bPo3uzX7WrDVp7zTZsMlHe+dU\nWWMhBBs3+/FXwB0zMlzbUTD7/uIR7NHUDPOPdJf2XhG6RucHzyjH8OqGUoj/88BWIcRmIYQPuB64\nZ9Y59wAfzf78m8Bjyt6fH6kJrICBmyfKQXMkeh57Zyl3AY6hF5xIZIWsAT/8hcXHv5Tga/dmcJw8\nLRlnj6uMb6V1G3x86Hcjpb+wEIyY0VmHBKGQTrRBx8zTz0EIMCrQmjGThkS8drZfWsgkuKUFLVtz\nStouL/7mtxl/uQfXsnEzNq7lLCkiTAjB1r96V9mziGuZot9JUkob+CPgIWA3cIeUcpcQ4q+FEB/I\nnvZ1oFUIsQ/4LPC5Yu+7GpGaKM0Sfx78KbvgLdIV/KAcH5X88HGLD/7FBLsPObjZmO7ZQp+2JJky\nb0jed20T3/jROt71nqwXtBSTzTKuMRF3SUxUptxEb49FOl3l0hYCNv/3y3jr03/AuXf8Nm996g/Y\n/GfvQAubdH3obCKndiCEhnQk2jICEoSh0fG+U8ow8PqgJHEVUsr7gftnHfuLaT+ngOtKca/VjNQE\nqZBJIGHNWO27JYz/1x2JbQiELSdX+kJ6c05LX5xYc5BMsHLhOIkU/PGtU71XLz1D5yPv9tHZonGw\nz+Xr96YxDcEXPhogOq3/TCnLFeQmnEvTe9n4+jM813omr7adVlw+hBBY+tJCamJjTkWd4b1HMzQ2\n60QbDIwyht4WYsOnL2Lt9WejB6YWHWt/61w6f+N0zNapJvD6MndDQggiZ6yBu14tyXjrDVXbp84Y\n7QjR0hfHTDteQImERNSHbQgah9OlqfZpS0Zbg5iW41X8BHQJesrB7Isz0hmuaM3/6Ty10+Gpnck5\nxz/4hQlO3ahx9ok6W9ZqvONsA6GBJpbXFjIn+FLCWMbl6I0/ZuSJNwF4y9Cr7G3aRNoIFTUBZHST\nuBEiYi9cWtp1JakKr8QdB0aGHEaGHNau9xEIVjY+ZP0nLkAPzZwg9ZCJFjRKMrk7KYvEvtUbcKjE\nv86QmmBobRQ946DbLrZfx9U1dMuF4dK0GBRA01ASh7l2QU1Cw1DSqzJaY+w+7LL7sCeQ/3F/ht+8\n3OTMLTrdnVrBSJl8/oK0BY//yubO7RmOj0k++ZkxmrLCD97/Z8vYYXa3FmcySOkB7tzyXt57+Od0\npuZvFHO835rT7KoS5P49/X0ZNm7yV7QAnNGQ37tdijFI18VN2Qzc89rCJ69QlPjXKY5Px5lm5zSs\n0itDISuqYbmeKtRaJchp9A1LvnK3Vxdp24UGN/2mf7JstBACV0qSKcmtP8wwNOZy3RU+dA3ufdri\n6V1T5pUbbxqjadt/zbl+whcufpBC4AqDe7uvxHBt2lLDXDD4Cp3JmROB60ri8era3x0bbBvMCs75\nE28MEjm5feETF8GMSV7C+Is97P3iQ9jj9dGTuRwo8V8hOHppt+QLyXpL7wTDXeGangByPPiczfaX\nbN59ocEFJxu4Ep542TtmZ+fMl/en5r/ILMZ9DaX724XA1k36Qh08sOEK3nnsKTZkE8Agu/qugdi4\nSr/U+29+jDP+7Vo0v9fURbouZCu0LpXcc6SUjD53mFc/Vp4mNPWEEv86RnNcQmPpisTfT0cAvrRN\nZDRFvDm44Pm1QCoDP3nS5idPLv5/dectcV45K8+qXw/QlB5jxN/gZWWVCiFwhMEvO89jw4H7Jg9r\nGhiGt/KuFqZJxZ2+Y88d4eWPfJ/uP7yE8NY2rLEkoc0tc/wAS0EIQdNFG/F1hMkMTJRwtPWHEv86\nRbNd2o/GvFLPVH5hqEkIxay6Ef9SkNZ8PLb2EvpD7QgkIMpi/oqbYRyhoUvP1COEoL3TpK+nevV/\nzCoVeovv7GfX7//IG0NzkAt//ntzzpHSC0fL1/ylEOFTO8gMvLnwiSuYmirvoFg80ZEkWlb4YaoU\nQz7KphfTlEi3HCIjSaJDSa8JTZ3n8K3dluSVs74z49gj699Gb6gdR9OxNaNsdhDdddDkTBt/KKyz\nbqOPULjwPXUdQuHyfKQTE27xBdWKxBpJ8uY/Po6TtJDZUiP2RIbRZ48w8tRB7PHU5PGFSLwxv4N9\nNaBW/nVKIJG/BHR2PVp2JF7GcWQkhaNB43BqMtcgPJ4mGTEZaysuFLKafOP8BE9M+33cjHA80ILU\nyltgR3dtThvZm/c19Ps1Ort8HNyfLji3lisJTEpwZeEggErR8+1fMbbjGGuuPR095GPw4TcY/sUB\nkGC2hrho++8h5hmklJLUkTHSPRWslFejKPGvU1wtfwno2ZRjrZabYARTPQCmi5WQEIxbJCM2mTpN\nnx///HeBKdvyhBHAzacqpZrcpERIl5NGD3D+YOHespomiEQ14jF3zgRQzoJ4muZ91QITuwfYf/PM\nHgtGU5DOa07HtVyEkb+3b67/78u/+4NKDbWmqZGXU7FU4o1+3Fnv70qt+sWsn/M3oYFgtgVlvfHQ\nlffx8vaZTkWfW25vq+SE8UNcMvAi2gJTdnuHSTiqVXRT1dik116T9yzBzS285aFP0v2Hl2DMk38i\nhEDogvUfO7+Co6td1Mq/GkhJ0+AgZjrNcGcnzjKCp5NRH2bGJRxLe+0dC+z2BdWLEjRSNtHhJOmg\nSSag15UJyBI6z3Scy5HIOgxpc8LYm5R1ehUalra494HQBJ1rfLjtkkTCYaDPLruLJRyduU6UUjI2\n6mSbv4PfL2jrMCueBQyw9a/fhRH1TbZ1nG+SEppXz+fALdsrNLraRYl/hQmPjXH5j39MKBZDahqa\n67Lj8ss5cMYSy8sKwXhbkHizHyPtICQ0D0zMqfFTqd1APnyWxDeaJjyaJhU2Ge2ofR/AXYlbeeTP\nIvxg6zU4YmrCeqntTIR0kZQnuc1wLTbFjy7pOZoull3XZqkMDzp0rZsyew0P2YyNTCXDpdOSnqMZ\n1m3w4Q9UcALQBI3nrVtSP99qO65rBWX2qSRS8mt33010dBTTtvFlMhi2zQXbt9Pa27vw8/Pg6hqZ\nkEk6bGL555ZjrpbUimlfGhCcsGg8nqjpKKCHPz/Ey9t9bF/71hnCD4AQyFxMv5RTX8WQe76USARN\n6aU3MAoEREXm02RyamvpunKG8OeQ0psUKoorFx3hAyAdl/4f7yrjgOoHJf4VpKW/n0AigTbrU6PZ\nNltffrno64+11m7MvQBCcYvWY7HJ0BF/wsKfsGpmQsg1Vu8LdxZs0D7nqxhyzxcCR+jc2/1OeoNt\nSzbTNbeWPwZHumBb3shsu/AIM1UoAz3+Us+iejpIKYnvHeTwV5+pwKhqH2X2qSD+ZBKZRzA0IJhY\nuLLjQuiOV4Z5ofLO0x+u5M5AAL6MS8ehMTR35s1H2kOkI9XrGn7jTWOTRcmrMhUJgYPBgxuuoDU9\nwrYjj+Nz5++oFRu3OT5gk7NElXsO7e1Js36jf95MX7OC/X/NzjCb//QdNJy7bkFntJSS5MERfnXN\ntyo0utpHiX8FGezqQssTj2cbBse2bCn6+rapLxj7L/Hq/1sG+KvQrU8AupsdzzSxahlIkBlLMdIZ\nydvFbFn3clw0V+IY2ryr9LXbkpPF2wYCLZiORUb3Vd4/IQSu0BnyN/NU5wVc0fvLgqemUy7H+8vv\n6J1OJg0H3kij6+APCNIpOeP+QkBLa2VCe0+6+d10XntG9r6LeJ0kvPrpH5Z5VPWFEv8KYgUCvHrx\nxZz57LMY2UIttmEwEY2y//TTi76+Y2oks81etJw5GUCArXlmiomoDzeXlFX0HZdH3tBQwJd2aT0W\n4/iGKEa2mYyTnQiEK71OZov4oAvHpXkggT9pI4VXBnu0LYiQoNsumYCBbQiCcQsz48DzDiO+BibM\nEI+sezuumH+yKIpFlINwNZ2D0fW4vaJg2OfoSGWFfzqOA05SEgoLkkmJdME0BW0dRkWifdreczKd\n15yx6NBTabsc/dYLpI8u3aeyklHiX2F2v+UtjHR0cNJLL+FPpThy4onsO/PMZYV75mO0I0RkJEV4\nPIOQkkzAYKw1OKP8c/uR8cnJIR/VihASgOFIOg+OT+qjFAIhpddNTINYY4CJJv+kgGq2S8NQksCE\nt41JRE18aQcz43oOZwk4kpb+xJy/a/Ln5yR3b9qGIS3cMmfwTtpnFjJTCIErxBz/UA7LqnKLRbwI\nn80n+IHSdk5biI3/7eJF1/GRUnLse7/izf/9izKPqv5Q4l8F+rq76evuLs/FhSDeEiTeUtj5qxUI\ndcsdrWYwpiBbQmAyEEZOjke4EB1NgYCJpgC4krZjMXRn6pxwzJq8zmwKrkmzzltbVsjnkJ0AhHSA\nbBTRdPGUktbUCIYsLPCmqZFOVb/GP1RW+AH0RfqGnLRN7JU+Dvztz8s8ovpERfusQlJBY16nZm3E\n3njMlhVNQsNwitaeGK09M4U/3/kLHZ95UgWdla7FFb3Pct2B+wg4afRsBrHmOpiuzdv6np/3+cFQ\n9T+6hlF54Qc4ft/reaN7ph+TUiI0wRtfeLCSQ6srqv8OUlScWEsQVxN5cwKmO4ZrFQH4Uw6+rGkn\n3+O1jiN02pJDRO0Ev3ngfs4d3El37AhnDe/mg2/eR2t6dN7nhyPlNU8JAQ2NhUtIeM7d6hgOjn7t\nOazhxJwJYPpEJIRAGBpdN5xT6eHVDUr8VyGuoXF8Q5Rk2Mwr8rmSEDXSQKog84l8LY8bKema6Cea\nbdzudzOcPfw6Vx57ivMHdxK2F+4qputi3vLOxeLzQ1uHSVuHgWnm8hG8b4YJ7Z0G0cbqiL89nub5\nq76Gk5g/XE0IQdtVWys0qvpD2fxXKa6ukWjwE5iw8opoJmgQaw4QHkoQTOdfYdcytT7eNcnjRV8j\nHNZJTJQno9bv8ypjNjQaNFRJ5OdDOhLdv/C4nDotLlgJ1Mp/FaNb+XsCuEAy4sMKGJiLKBs9nZpe\ncdcIpmvTYMWX/XwpJf29aS/Bq0zEYi6x8Sr2jVwAN23jJOYXdiklh//t2QqNqP5Q4r+KiY6k86+Q\nBSQjXuipYcslraJrfcVdUaSLkK5XGyGLcF1M16I7fmzZlx0csIjHyjvNSglDg/aiyiaUG6MpQPdN\nl3Le3R/hjNuvpfnSbpBw+N+fK1ikTUpJfPcAx+97vcKjrR9qbz+nqAxSYhRY1QvJwnHouXNLO6oV\ng3Ad2lPDvL3vOZ7uvIC+UDsAXckB3t773GR/3qXiupLxscqEeDp2WVoULwmjMcD5P/4oRnNw0szT\neMF63vzykxz99+doumgDzZdumuHslVLiTGT41bXfrtaw6wIl/jVMZHSUU158kcahIQbXrGHvueeS\njERKc3EhcPT83cCcaeUV0gEdf8qZIfIScDTBWHuQ6HAS01ra7mBFIyU+J8PFAy9ywvhhNCTvPfJz\nbKEDct7Y/cVQydj+UtSuK5a1HzkPoyk4w76vh3xs/tO303fXq+z89A857StX03r5CaAJcLPC/+Hv\nVnHU9YES/xqltbeXX7v7bnTbRpOS9p4eTn3xRSaiUQ6ecgqvn38+lt9f1D3GmwM0DiVnZPu6Asab\np6471h6m/dg4uNOigAQMdYVx/AbpsI/mvnjBnsIrgkUufzXX4eTRfbx14Fdz/heGLL7HYirpVrRk\ncmNz9bt3tV62GT0wV6ak7RA5pYPxF4/x2h/+hODmZhrPW0dmKMHIkweRdvUzoGsdJf41yoU/+xmm\nNRXKlkvzj46Pc8Zzz3HySy9x70c+QioaXfY9kg1+EF4fXt2WOIZgvDlAKjol/o6p0b+xkdB4GjPj\nYPl0ElEfUp/aHYy1hTB7YmhZ/8CKnAQKTQDZ44Zr05CJceHxV5b997uuZHTEJj7uIjRoaNRpaPQE\neCLu0N9rVayejxDQ3FLtdu2Q7o8TOV3OKecgDB1reKoSbvLNEZJvjlR6eHVNUeIvhGgBfgBsAg4C\nH5JSznkFhBAO8Gr218NSyg8Uc9+VjmbbNA4PF3xcAGYmw3u+9z1+/OlPI4vorJ2M+klG599BSE14\n5RQK4BoaAxsaaDyeIBSvQqnQUpJT18niQnLmY7MmAIGkK97HpthRto4fXPYKX0rJscMZMpmp+w0d\nt0lMODQ0GQz0VU74vfFAOgXBUOXumY9j33yB5ks2oU/rzetaDhNvDJI8qMS+GIqN9vkc8KiUcivw\naPb3fCSllOdkv5TwL4DUtAUF3ctyTdF16FBlBrUQQhBrCVZ12V8KbRTSoTEzPhWhM7txyzQFNlyb\nM4b38J6jj3Pq2P6iTDvxmItlzfwLpITEhKTvmIVbBSuGWwPtDseeP8r+W36Ok8hgx9I4SYv4rn52\n/f6Pqj20uqdYs8/VwOXZn78JbAf+vMhrrnqkpnHwlFPofv11jDz1/6dOlHQcOULP5s2VG9w8uIbG\nSHuIpuNT23Ehp6qE5nwGpZ4fSnL9rKhfcPxVXmw/E0SeyTdbkM10MgScNGcOvc4pY/uXPe7pJJNu\nrTQ0m6Qazdjz0XfHKwz85DXCJ7dhjSRJHVGlmUtBseLfKaXMNZ/tAzoLnBcQQuwAbOAWKeWP850k\nhPgM8BmAQLS9yKHVNzsuv5xgPE7H0aPo2QlgtqgJ4OSXX6br8GEev/pqQrEY5zzxBC3Hj5MMh9l5\n0UW8edppFR13KuKjP2TiT3rmn3TQxLAcmvsm5hRhK4ZC3cgWvP48ztvr9t3DWKAJTboUmnJ9rsWV\nR5+gqwQZutMxasz71tqmo+u1471xsxU6FaVjwbecEOJnwJo8D31++i9SSilEwQaC3VLKY0KILcBj\nQohXpZRzlkxSytuB2wEaOrfW2DqosjimyfZrriEyOkr7kSNc+OijaMwVOt1xaBwc5Mo77iCQSEzu\nFKJjY7zlscfwJZPsOf/8io5daoJUeKrsruU3luQILpRDMP0NYRkC19AIpBZpapEuDZk4Kd1Pxpjr\n4/A7GRqcJFra66ZVCFvoNGXGF3fPJdDQaDAyVHxEULEEAoKWNoNgqPrOXkV5WVD8pZRXFnpMCNEv\nhOiSUvYKIbqAgQLXOJb9fkAIsR04FyjNfnmFE29qIt7UxJbXX6fjWP6sUA0Ix2Jzjhu2zZnPPsve\nc85B6tX9MBfKKcg3w2f8OprjojsSTXrhpwBDa8LYfgMpACGIjKTwpZ05jWlmm34M1+aSvh1sHT/I\nzqatvNBxNrZmzHj8rKHdAETsBN2xoxyKrsPRZn48NNfmpLE3CTrppf8DFsAwBLoxVSO/WqzbWFz4\nsKJ+KHazeQ/wUeCW7PefzD5BCNEMJKSUaSFEG3Ap8PdF3nfV0dbbu+DKOW8DE9clmEiQKCIktBTE\nmwM0DSTm5BSkQgYTDVnBEQLLr0/a1oMTFr6kjW1oJKO+Ob19Ew0+ImPpGQ1fJquRCgj4wUnYnDB2\nkBPHDwJw+ugbWLqPV1pPIfcfO314D2eOTJUBuKz3GV7OnMau5pPI6CZICDgpzhx+nTNG9hb9v0in\nXRJxB9uRBAM6oYiGpgl0vbriX+X1gaLCFCv+twB3CCE+CRwCPgQghLgA+G9Syk8BpwL/JoRw8Rap\nt0gpXyvyvqsOxzDQM8uoUCglqWDhrl7BWAzdcYg3NpY1nTMV9jHe4tIw7JUrFkAqbDLaHsp/XyFI\nRnwk5+na5Ooax9dGaBxM4k/ZICARMRlvCfI3H4mz6w+eoiM5SIM1MXVZ4NyhXZw1vJuEHiDopOZk\n3WpIzh3axblDu0rxp08ipWTwuM346JR5Zxzv3j6fYDkvbylpaasxx4OirBT1aksph4B35jm+A/hU\n9uengTOLuY8CDpx+Oie+8krB6J98si3xVv5vffhhnrnqKpoHBjh/+3aajx/H8vuxdZ1AMonUNNKB\nAE9v28bx9evL9jckGgMkGvzotoura15D9iJxfDrDayNzHLknn6WRHi8cBqtLd7KefqVIJd0Zwj+d\n6fH9lST3L2tu0Yk2qKX/akJN9XXCy5deSsPw8GT0z2Jk02tgLlm3fz+X3H8/XYcPY9ieXcGfSuHL\nnoPrYsTjXP6Tn3DvRz9auvpBeQclcMwyiMw04b/zljivnPWd0t9jiaSSLuPjDtKVRKI68VhtlUju\n3uLDdcAwBVoJJmJFfaHEv05wDIPt11xDw9AQ3Xv2cMqLL4Kmodk2mjt/sxXDcVh/4EDeto3T0VyX\nE3buZOfFF5d49JWlMVZ6h+xSGR6yGB12JmP3J+IuWg0trA0TDENTCrCKUS99nTHe2sqrl1zCa295\nC2sOH6atp4fTXnhhUc9dKGVHdxzC4+MYmQwb9+4lHIsxtGYNvd3dRZWQqCQ33jTGE5feWdUx2Jac\nIfzgWaXc6kdyTtLcoj76qx31DqhTHNPk2Akn0L9+/aLEfzHZr5ZpEmtu5uqvfx3NdTEsC9s0GW9u\n5mfXXYdjmvM8uzZ4/2tP8uQynielJJORaAJMX3ETXSKRX+Wl9FbcdoXLH2kaM8pDNDRqyr6vUOJf\n7zimiRQCsYjaAAs2PJeS7tdew5ee6vBlWhZNg4O871vfIhGN8uapp3LgtNOqkjdgZDIIKectZf3k\nZ48s+bqjwxbDQ1MrdU3z4u5Nv6C52cAfWNpkoGmiZtpZCgFr1/sQAmxb4vNrGIay7yuU+Nc9UtMY\n7uigtb+/qNIJAs/m3zgyMuc6uusSicWIxGK09Pezcc8efv7BD1as00coFuPihx6ivacHAYy0tfHL\nd7+b8dZWmvv7uWD7dlr7+vCbkqGwTkubsag69FJKeo5mSCVnSrXretE3mYwkEc/Q2WUSjsw/2Ukp\nGRm2GRtx5i3CVulVP8ILIxWawKfytxTTqA9DrmJenr2yYBL2ktAXcByD5zxec/Qopz333LznhcfG\niIyOUmy1MuG6vOuOO+g4dgzdddFcl+aBAd51xx009/dz5V130d7biyYlVgbGRh0G+hansOOjzhzh\nn42UcLzfmreXretKDh9MMzI0v/BXGiGgrd2YUwtfoQC18l8RjLW3s/Oiizhtxw60RYaBzsdCvgEB\nnPXLX9K3cSPDXV3ZJ0maBwZo7e/nlBdfJBSPA5AMh3nqve9luKMDfyqFbRhL8h2sffNNfKnUZDMb\n8FYsmuNw/uOPo9szwyel9CJrbFvOa96wLJeR4cWFXrou2DbkG7ZtSwYHrMqv6Atg+jyzk2EImpqN\nmqnMqag9lPivEF5961vp2byZs598ko5jx2aI5XJYqEG7AN79gx9wdMsWDp50Ehc99hiGZU36HnLP\ni46NceWdd5IMhSYnhGNbtvDcO99JZp7MY9228ScSREZG0PIktpm2TXR0NO/fKSX092RobDYIR7Q5\nJqC+njQT8cX/f6SEeMymsUlHy0Y9SSkZHLAZH3dK00igBGgadK31Fe2wVqwOxHzb2WrS0LlVXvTb\n/1TtYdQdumXx/m9+k+DERNETwGJwpzmbC00Us3cSuVFNRKP86u1v58hJJ00+JlyX8x97jBN27fIi\nlIRAuO4c+6Rlmgx2dtK5wETn80PXOp8X0w4MD1qMDC8/5jLaoNHWYTI+5jA8aNdEDX6hQUODRlOL\nqZy5Ci7bc/8LUsoLFjpPLRFWGI5p8uBv/RaHTjqpIgtSTS7ctzdfHwIBRGIxLr3/ft793e+y4Y03\nQEouevhhtu7ciS4lmpR5/RASSAUCvHLxxWjB+R2xmTQcOpBhaNDCdd2ihB8gNu5y6ECaoePVF34h\nIBAUbNrip63Dp4RfsSTUyn8F09LfzyUPPEB0dBSo7cbqlmFwdMsWNu3du+A4p5uksgVAVxWRqEYg\noOEPaPgDYlGRTYrVg1r5Kxju7OTej36U7Vdf7Tlaq5SluxhtNm2bDfsX1+Jh+k5jtQm/pkHHGpPG\nrDNXCb9iuSjxX+kIQe/mzfz04x9n50UXEYtGK+6fXKw86fP1K1YAsG6jTwm+oiQo8V8lpMJhdl10\nET/9xCc4csIJuDUsIMee6ZMAAAoHSURBVLMnp1W2uC9IxxoDn4rkUZQIFeq52hCCJ9//fjbs28ep\nO3bQMjCAkKVrrF4M84WVriZy7Rx1HXx+QTji1dpXZZcVpUSJ/2pECI5s3cqRrVvxJxKc9txzbN25\nE+E4k9E7isojBIQjGp1dhbuXKRSlQon/KicdCvGryy/nV5ddRsfhw1zy8MP4Ewn01eZJrTK6AU1N\nOo3N6iOpqAzKgKjwEIKB7m7u+8hHeOOss3BF7VSmXKmIbNG1jZt8bNoSoKnFVM5cRcVQ4q+YgRUI\n8OIVV/DgDTfgaJqaAMpEKCTY0O1jwya/KsegqArqXafIy2hHBw9ffz3urNyA+SYDNVEsDiEg2mgo\n0VdUFfXuUxRktKOD7b/xG8SjUWxdx9E0hjo76V2/fk6o6EKVQFcLujF/mwMhIBjSCEfUR09RXZR3\nSTEv/Rs3cs8nPkFwYgLbNCe7aK05dIiLHnkEfzKJwCu05k+l8k4A+SaG3LGVMmmEQoLmNpNAQJts\n3p77A02fIOD3fCjhiE4orDJzFdVHib9iYYQgGYnMONTX3c1PPvlJQvE4lmmy/sABLn744SVdNhEM\nEkwmSznS0iOgucWLsR86nr/+v98v6Fo/1SarpdWkqckgk5HoBpimWuUrag/1rlQsHyFIRKNYgQCH\nTj4ZO0+3Ewl5s4kdw+D1886rwCCXTy4ap6nZoKFRz7tF8ez3cyuLarogENSU8CtqFvXOVJQEV9d5\n/OqrsUwTK1tEztF19pxzDq9efDG2YeAKgQvYhsHes8/mjXPPRdaY+SMc0Wht12lq0ensMlm/0Yem\nCTRNeC0Rpw1XCM+kE22ofDN7haJYlNlHUTIG1q/nR5/6FOsPHMCXTtPb3U2suRmAni1b6N67F+G6\nHD7pJIY7OwF46ZJLOOfppycbsizUQWw2+Uo650TZyshFVf3MZda2tBnzrtQbmwz8fo3xMRvb9kor\nR6Kq7IKiPlHirygptt/PwVNPnXN8tL2d0fb2Ocdff8tbGO3o4NQdOwhNTNC/fj0b9+4t6DzOEQxp\nNDZ5K+5QSCOdlsRjDkKDSFTHZwoG+i3isZkd1fNtNJpadJpbjEU5YQNBjUBQlV9Q1D9K/BVVp6+7\nm77u7snfd1xxBWdFX+Ottz2GaUgmJiSZ9NQS3jAEHWtmtiwMBMWcZuWdXT7aOiS27WJbkE5LDB3C\nUR1NA9cBTUdF3ihWJUXZ/IUQ1wkhdgkhXCFEwc4xQohtQog9Qoh9QojPFXNPxSpACL74hW46Ok2a\nW32s2+Bj7Xofre0Ga9aabNi0+JaFui7w+3XCEZ2WVoOGJgNd97pf6YbqgqVYvRTr8N0JXAv8otAJ\nQggd+BfgPcBpwA1CiNOKvK9ihSNu+o+pn4W3qm9sMgiFdSXYCkUJKEr8pZS7pZR7FjjtQmCflPKA\nlDIDfB+4upj7KlY2D39+iJe3K7u6QlFOKhHquQ44Mu33o9ljcxBCfEYIsUMIscNKjlVgaIpaRD77\nTLWHoFCseBZ0+AohfgasyfPQ56WUPynlYKSUtwO3AzR0blV1wlYhD115H09+9sjCJyoUiqJYUPyl\nlFcWeY9jwIZpv6/PHlMoFApFlaiE2ed5YKsQYrMQwgdcD9xTgfsq6gy16lcoKkexoZ7XCCGOAm8F\n7hNCPJQ9vlYIcT+AlNIG/gh4CNgN3CGl3FXcsBUrjRtvGlPCr1BUkKKSvKSUPwJ+lOd4D/Deab/f\nD9xfzL0UK5vLbvkmL6MifBSKSqEKuylqAhXaqVBUFiX+iqrzxfv/tdpDUChWHUr8FQqFYhWixF9R\nNdZuS6pVv0JRJZT4KxQKxSpEib+iatx6979XewgKxapFib+iKtx405iK8FEoqogSf0VV+PWoXe0h\nKBSrGtXJS1Fx7rwlzhNn3VntYSgUqxq18ldUlDtvifPKWd+p9jAUilWPEv//2969hMZVhmEc/z8o\ntrtepbYq1WJAuhAvQcTu1EUqkngF3Vghot2060Jx48bLRiiIIkFaN7E0GyMUxLZKdVG1C+2V2rQg\nNsS2XghKS7y9LuZTBptJppk535mZ8/zgMGfmfITnzTfz5szHmYllteTXmbIjmBlu/pbRlq3TfLbB\nyz1mncDN37J55MTnZUcws8TN37LxVzabdQ43f8ti7NKOsiOYWR03fzOzCvJ1/la4l/a+7X/UYtZh\nfOZvZlZBbv5WKK/1m3UmN38rzJqBy/7yNrMO5eZvhdl5z6WyI5hZA27+Vgh/mtess7n5W9vtefU3\nlg6Mlh3DzObg5m9mVkG+zt/aasvWaY7c4bN+s07nM39rK/+HLrPuoIgoO8OsJF0Evis7RxutBH4s\nO0TBXGNv6PUae72+tRFx/XyDOrb59xpJhyOiv+wcRXKNvaHXa+z1+prlZR8zswpy8zczqyA3/3ze\nKTtABq6xN/R6jb1eX1O85m9mVkE+8zczqyA3/4JIekrScUl/S2p4ZYGkAUmnJE1I2pYzY6skLZf0\nsaTT6XZZg3F/Sfo6beO5cy7EfPMiaZGk3en4F5JuyZ9y4Zqo7zlJF+vm7fkycrZC0ruSLkg61uC4\nJO1Iv4Mjku7OnbFMbv7FOQY8DhxsNEDSNcCbwEZgPfCMpPV54rXFNmB/RPQB+9P92VyOiDvTNpgv\n3sI0OS/DwC8RcRvwBvBa3pQLdxXPu9118zaSNWR77AQG5ji+EehL2wvAWxkydQw3/4JExMmIODXP\nsHuBiYg4GxG/A+8DQ8Wna5shYFfa3wU8WmKWdmpmXuprHwMelKSMGVvR7c+7pkTEQeDnOYYMAe9F\nzSFgqaTVedKVz82/XDcC39fdP5ce6xarImIq7f8ArGowbrGkw5IOSeqGPxDNzMt/YyLiT2AaWJEl\nXeuafd49kZZDxiTdnCdaVt3++muJv9itBZL2ATfMcmh7RHyQO08R5qqx/k5EhKRGl46tjYhJSeuA\nA5KORsSZdme1tvoQGI2IGUkvUnuX80DJmayN3PxbEBEPtfgjJoH6M6qb0mMdY64aJZ2XtDoiptLb\n5QsNfsZkuj0r6VPgLqCTm38z8/LvmHOSrgWWAD/lideyeeuLiPpaRoDXM+TKreNff0Xysk+5vgL6\nJN0q6TrgaaArroZJxoFNaX8TcMW7HUnLJC1K+yuBDcCJbAkXppl5qa/9SeBAdM+HZuat739r34PA\nyYz5chkHnk1X/dwHTNctY/a+iPBWwAY8Rm0NcQY4D3yUHl8D7K0b9zDwLbUz4e1l577KGldQu8rn\nNLAPWJ4e7wdG0v79wFHgm3Q7XHbuJmu7Yl6Al4HBtL8Y2ANMAF8C68rO3Ob6XgGOp3n7BLi97MwL\nqHEUmAL+SK/FYWAzsDkdF7Wrns6k52Z/2Zlzbv6Er5lZBXnZx8ysgtz8zcwqyM3fzKyC3PzNzCrI\nzd/MrILc/M3MKsjN38ysgtz8zcwq6B+GBbBgia/YEgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1036662e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#prediction\n",
    "def predict(X):\n",
    "    pred = np.argmax(svm.forward(lin.forward(X,W,b)),axis=1)\n",
    "    return(pred)\n",
    "\n",
    "plot_decision_boundary(predict,X,y)\n",
    "print('Accuracy: {}'.format(np.mean(predict(X)==y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Linear > Activation]++ > [Linear > Output]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the `[Linear > Output]` layer,\n",
    "\n",
    "$$\\frac{dL}{dW_{last}} = \\frac{dL}{dscores_{last}} * \\frac{dscores_{last}}{dW_{last}} $$\n",
    "\n",
    "For the `[Linear > Activation]` layers,\n",
    "\n",
    "$$\\frac{dL}{dW_{nonlast}} = \\frac{dL}{dscores_{previous}} * \\frac{dscores_{previous}}{dscores_{nonlast}} * \\frac{dscores_{nonlast}}{dW_{nonlast}} $$\n",
    "\n",
    "$$\\frac{dscores_{previous}}{dscores_{nonlast}} = \n",
    "\\frac{d(scores_{nonlast}W_{previous} + b_{previous})}{dscores_{nonlast}} = W_{previous}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "class DeepModel:\n",
    "    def __init__(self,X,y,D,K,h_list,p_list,\n",
    "                 activation_list,output_activation,scaling,const=0.1):\n",
    "        #parameters\n",
    "        self.X=X\n",
    "        self.y=y\n",
    "        self.nb_layers = len(h_list) + 1\n",
    "        nb_layers = self.nb_layers\n",
    "        self.h_list = h_list\n",
    "        self.p_list = p_list\n",
    "        self.W_list = [None] * nb_layers\n",
    "        self.b_list = [None] * nb_layers\n",
    "        self.X_list = [None] * nb_layers\n",
    "        self.X_list[0] = self.X\n",
    "\n",
    "        self.act_list = [None] * nb_layers\n",
    "        self.lin_list = [None] * nb_layers\n",
    "        self.drop_list = [None] * nb_layers\n",
    "        self.linout_list = [None] * nb_layers\n",
    "        self.actout_list = [None] * nb_layers\n",
    "        self.mask_list = [None] * nb_layers\n",
    "\n",
    "        self.dscores_list = [None] * nb_layers\n",
    "        self.dW_list = [None] * nb_layers\n",
    "        self.db_list = [None] * nb_layers\n",
    "    \n",
    "        #randomize weights\n",
    "        for i in range(nb_layers):\n",
    "            #scaling term\n",
    "            if scaling=='constant':\n",
    "                scaler = const\n",
    "            elif scaling=='xavier':\n",
    "                scaler = np.sqrt(1.0 / h_list[i - 1])\n",
    "            elif scaling=='he':\n",
    "                scaler = np.sqrt(2.0 / h_list[i - 1])\n",
    "            else:\n",
    "                raise ValueError('Can only use \"constant\",\"xavier\" or \"he\" scaling')\n",
    "            #linear layers\n",
    "            self.lin_list[i] = Linear()\n",
    "            if (i == 0):\n",
    "                self.W_list[i] = np.random.randn(D,h_list[i])\n",
    "                self.b_list[i] = np.zeros((1,h_list[i]))\n",
    "            elif (i==nb_layers-1):\n",
    "                self.W_list[i] = np.random.randn(h_list[i-1],K) * scaler\n",
    "                self.b_list[i] = np.zeros((1,K))\n",
    "            else:\n",
    "                self.W_list[i] = np.random.randn(h_list[i-1],h_list[i]) * scaler\n",
    "                self.b_list[i] = np.zeros((1,h_list[i]))\n",
    "\n",
    "            #activation layers\n",
    "            seed = random.randint(1, 9999)\n",
    "            if (i<nb_layers-1):\n",
    "                self.act_list[i] = activation_list[i]\n",
    "                self.drop_list[i] = Dropout(seed=seed,p=p_list[i])\n",
    "            else:\n",
    "                self.act_list[i] = output_activation\n",
    "                self.drop_list[i] = Dropout(seed=seed,p=1)\n",
    "    \n",
    "    def summary(self):\n",
    "        nb_layers = self.nb_layers\n",
    "        h_list = self.h_list\n",
    "        N = self.X_list[0].shape[0]\n",
    "        \n",
    "        for i in range(nb_layers):\n",
    "                \n",
    "            print('[LIN] {}: Input ({},{}), Weight {}, Bias {}, Output ({},{})'.format(i,\n",
    "                N,self.W_list[i].shape[0],                                                              \n",
    "                self.W_list[i].shape,\n",
    "                self.b_list[i].shape,\n",
    "                N,self.W_list[i].shape[1]))\n",
    "            \n",
    "            if (i < nb_layers - 1):\n",
    "                print('[ACT] {}: Input-Output ({},{}), Dropout {}, Activation {}'.format(i,N,self.W_list[i].shape[1], \\\n",
    "                                                                                         self.p_list[i],self.act_list[i]))\n",
    "            else:\n",
    "                print('[OUT] {}: Input-Output ({},{})'.format(i,N,self.W_list[i].shape[1]))\n",
    "            \n",
    "    def train(self,iter,reg,step_size,use_l2):\n",
    "        nb_layers = self.nb_layers\n",
    "        X_list = self.X_list\n",
    "        act_list = self.act_list\n",
    "        lin_list = self.lin_list\n",
    "        actout_list = self.actout_list\n",
    "        linout_list = self.linout_list\n",
    "        dscores_list = self.dscores_list\n",
    "        dW_list = self.dW_list\n",
    "        db_list = self.db_list\n",
    "        y = self.y\n",
    "        \n",
    "        for j in np.arange(iter):\n",
    "            #forward\n",
    "            for i in range(nb_layers):\n",
    "                linout_list[i] = lin_list[i].forward(X_list[i],self.W_list[i],self.b_list[i])\n",
    "                actout_list[i] = act_list[i].forward(linout_list[i])\n",
    "                #dropout\n",
    "                self.mask_list[i] = self.drop_list[i].mask(actout_list[i])\n",
    "                actout_list[i] *= self.mask_list[i]\n",
    "                if (i < nb_layers-1): X_list[i+1] = actout_list[i]\n",
    "\n",
    "            if type(act_list[nb_layers-1]) is Softmax:\n",
    "                data_loss = act_list[nb_layers-1].cross_entropy(actout_list[nb_layers-1],y)\n",
    "            else:\n",
    "                data_loss = act_list[nb_layers-1].hinge(actout_list[nb_layers-1],y)\n",
    "            reg_loss = reg_loss_calc(self.W_list,l2=use_l2,reg=reg)\n",
    "            loss = data_loss + reg_loss\n",
    "\n",
    "            if j % (iter/10) == 0:\n",
    "                print('Loss at iteration {}: {}'.format(j,loss))\n",
    "\n",
    "            #backward\n",
    "            for i in reversed(range(nb_layers)):\n",
    "                #backward activation\n",
    "                if i < nb_layers-1:\n",
    "                    dscores_list[i] = act_list[i].backward(X_list[i+1],dscores_list[i+1].dot(self.W_list[i+1].T))\n",
    "                    #dropout\n",
    "                    dscores_list[i] *= self.mask_list[i]\n",
    "                else:\n",
    "                    dscores_list[i] = act_list[i].backward(linout_list[i],y)   \n",
    "\n",
    "                #backward linear\n",
    "                dW_list[i],db_list[i] = lin_list[i].backward(X_list[i],self.W_list[i],dscores_list[i],reg=reg)\n",
    "\n",
    "            #update\n",
    "            for i in range(nb_layers):\n",
    "                self.W_list[i] -= step_size * dW_list[i]\n",
    "                self.b_list[i] -= step_size * db_list[i]  \n",
    "\n",
    "    def pred(self,X):\n",
    "        nb_layers = self.nb_layers\n",
    "        X_list = self.X_list\n",
    "        X_list[0] = X\n",
    "        act_list = self.act_list\n",
    "        lin_list = self.lin_list\n",
    "        actout_list = self.actout_list\n",
    "        linout_list = self.linout_list\n",
    "        W_list = self.W_list\n",
    "        b_list= self.b_list\n",
    "        \n",
    "        for i in range(nb_layers):\n",
    "            linout_list[i] = lin_list[i].forward(X_list[i],W_list[i],b_list[i])\n",
    "            actout_list[i] = act_list[i].forward(linout_list[i])\n",
    "            if (i < nb_layers-1): X_list[i+1] = actout_list[i]\n",
    "        pred = np.argmax(actout_list[nb_layers-1], axis=1)\n",
    "        return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LIN] 0: Input (3000,2), Weight (2, 100), Bias (1, 100), Output (3000,100)\n",
      "[ACT] 0: Input-Output (3000,100), Dropout 0.5, Activation <__main__.ReLu object at 0x1025c9588>\n",
      "[LIN] 1: Input (3000,100), Weight (100, 50), Bias (1, 50), Output (3000,50)\n",
      "[ACT] 1: Input-Output (3000,50), Dropout 0.5, Activation <__main__.ReLu object at 0x1025c9630>\n",
      "[LIN] 2: Input (3000,50), Weight (50, 3), Bias (1, 3), Output (3000,3)\n",
      "[OUT] 2: Input-Output (3000,3)\n"
     ]
    }
   ],
   "source": [
    "mod = DeepModel(X,y,D,K,h_list=[100,50],p_list=[0.5,0.5],\n",
    "                activation_list=[ReLu(),ReLu()],output_activation=Softmax(),scaling='he')\n",
    "mod.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss at iteration 0: 1.7984069799173645\n",
      "Loss at iteration 100: 0.3419684761902695\n",
      "Loss at iteration 200: 0.260874974699511\n",
      "Loss at iteration 300: 0.23423617010268266\n",
      "Loss at iteration 400: 0.22285874066340983\n",
      "Loss at iteration 500: 0.20771365374813994\n",
      "Loss at iteration 600: 0.20331201427840848\n",
      "Loss at iteration 700: 0.19988726089061204\n",
      "Loss at iteration 800: 0.18720044373873562\n",
      "Loss at iteration 900: 0.18824603956412556\n"
     ]
    }
   ],
   "source": [
    "mod.train(iter=1000,reg=1e-3,step_size=1e-0,use_l2=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9693333333333334\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztvXmULFl93/n5RUSulZm1vv29XoAW\nCINA6hZgNQxtCaHGGoEtqWVkLaBBxpqRwBKeM9Oe9pmxpcFuy+cg00jjMWCNkIW2RlhqBAIJpJa6\nNQZBM3SzNYhuut/+Xr3acl8i4jd/3MysrKrMqqzKyKUq7+ecPJkZGRlxI5fvvfd3f4uoKhaLxWKZ\nLpxxN8BisVgso8eKv8VisUwhVvwtFotlCrHib7FYLFOIFX+LxWKZQqz4WywWyxRixd9isVimECv+\nFovFMoVY8bdYLJYpxBt3A3oRT+U0mTsx7mZY+uS5ZwPcwEaLWyzj5rHHn7mhqsf22m9ixT+ZO8HL\nf+xd426GpU8evL/IbKE27mZYLFOPs/RTz/a137AbYrFYLJbJw4q/xWKxTCFW/C2RcM+9GT5SmFgr\nosVi2YYVf4vFYplCrPhbLBbLFGLF32KxWKYQK/6WyPiDr8fH3QSLxdInVvwtkXH54ym76GuxHBKs\n+FssFssUYsXfYrFYphAr/pZIec8Ds2xkE+NuhsVi2QMr/pbI+WcPu+NugsVi2QMr/haLxTKFWPG3\nWCyWKcSKv8VisUwhVvwtkXP54yne/Fh63M2wWCy7YMXfYrFYphAr/haLxTKFWPG3WCyWKcSKv8Vi\nsUwhVvwtQ8Eu+losk40Vf8vQuPzx1LibYLFYemDF32KxWKYQK/4Wi8UyhVjxt1gslinEir9lqNjK\nXhbLZGLF3zJUbH5/i2UyseJvsVgsU4gVf4vFYplCIhF/Efl1EbkuIl/q8bqIyAMi8g0ReUJEviOK\n81osFovlYEQ18v8N4O5dXn8dcFvz9lbgP0Z0XovFYrEcgEjEX1X/CljdZZc3AL+phk8DcyJyKopz\nWyafe+7NjGTRNwxBdeinsViOBKPywzsDXOh4frG57cqIzm85AoQhiJhbJ1evwQd+0+HJr5nnJ0/A\nm38y5PnPH30bLZbDwkQ5YYvIWzFmIZLZY2NujWVSePY8/OZ/cXj6mxCLwSvvVN74I0o8DsUi/NI7\nHUolANMrXLmq3P/vHX70jcprX2OnAhZLN0bl7XMJONfx/Gxz2xZU9b2qeoeq3hFLzY6oaZZJZmUF\n/u39Dk89LagK9brwyKPCr/5fRugf/WuhVoWW8BvMvr/3+0KhMI5WWyyTz6jE/yHgJ5teP68ANlTV\nmnwse/JnnxL8YOu2RkP46pPC1WtmVuAH0vW9IvDlr3R/zWKZdqJy9fwd4L8BzxeRiyLyFhH5GRH5\nmeYuHwOeBr4BvA/4n6I4r+XwcM+9GZ64Fuehjwh/+mfC+kZ/73v2vOD7OwXc8+DqVbjlZnCc7qYd\nx4F43Jp9LJZuRGLzV9Uf3eN1BX42inNZDg8SKpm1KuliHSdQ3vW/AQgxDz70YeF//Kch3/7S3Y9x\n6y3K3/4tOzoA34fTp+G225Q/ekgoV5Stph8j/i96UaSXZLEcGWyEr2U4qLJ0qUBmo4YbaIcsCw3f\n2O7/7/c61Gq7H+Z7v0eJeQCbI/hYTHnxi5Tjx2AmDf/6/wg5faq1j+K6SiKh/MI/C4nHhnBtFssR\nYKK8fSyHE7ce4IRKI+6CY2Q+VajjNUJ2s7irwlefhJe+pPc+8/PwL+8L+eBvO3zt60oiDq9+tfKD\n/2CzMzh2DP7N/xmysQFf/qqQiCsvfhHE4xFdoMVyBLHibzkwjh+yeLWI2wgBEIVQwI+5ePVgV+EH\nqNfht37bIZkMeeop4U8/KVTK8LznwY++MeTcWbPfmdPwv/zP4Z7tmZ2F73qFtfFbLP0gOqEhkbkT\nt+nLf+xd426GpRuqJIt15m5UEN1uad800PTrZ+M4iuN02vWVZBJ+8V+FHLfhHhbLvnCWfuoxVb1j\nz/1G0RjLEUKVhSsl5pYrOF2EH8y2/ThYhuF2jx6hXoc/+bh107RYhoU1+1j6RkJl4UqReG1vk86g\nhKHw9NPQudBrsViiw4q/pStOEJIsNRBVqukYQcxl9kZ5JMIPxhR05owVfotlWFjxt+wgWawzv1w2\nY26F3GqVYi5BqtiISPi3+uS3grTCcHOb58H3v86Kv8UyLKzN37IFCULml8uIgqPmByIKmY09HPL7\nQAFx4TvvUOJxJRZTUinlh35QuevV5rmIcvq08s9/PuTMmYFPabFYemBH/pYtJEuNnmb2QUb9rUP+\nvZ+Hn3yh4vtKsQTZDLiuee3H/7HiB9jALItlBNiRvwVUSedrHD+fZ+5Gpesu3YR/L6OMdtwAGi78\n7u9m+EjBw/NgbnZT+KGZi8cKv8UyEqz4W5hZr5FbqeD54b7dNBUT2LVbR9A6ZiyApctF/uALVuH3\njZeA7HGYO2PuPRu+bBkMa/aZJlRJ5+tkNqqIQnkmRnE2QW6teiCTjgosn8kSr/qki3ViVeMJpGLE\nXnXr6KK1jS8H8KooLmgCiKUglgQNoVaEMNj7Pfs+RxKyxxAxn6Y6LsROQH4Z/Gr057NMBVb8pwFV\nUoU6sytbI3Iz+TqZfP1AhwyB0myCIO5SibtUcgm8mk+i4qOONL2EKjumBA7A+hHx4smdAC+OiIOq\nQioHhRvQ6G46OzDp+bbwA4iYuZTOzMNGj7IYXgIcF/zacDoky6HHiv8UkF2tktmo7RjdC9udLvtD\ngVrKpTCf3LLdT3j4CfOT8urdBScUcE4cgcjdRAbceFuU24KcWYK1C7u/dzvxNCRzRqwbFahsbAq2\nlwC3h5ms23bHNZ2SuM0vVqBRhcIyNmDO0okV/yOO+GFX4W+/fpBjAvFqsLOSegd+3KWe9IhXfVq1\nVhTMrOAFbs/3TSxuDBAImjOlxAzi9Fgyyx43phqAWgnKa8Ys1I1kDtKzHSadjOkM8suQXQLZZVlu\nxzHFnNvxmp1Rc7dYEubPmFmCnQVYmljxP8qoMrtcHsqhexTP2sLqyRkya1VmCvV2pHB+IcXJ+OAx\nAyPDjTUF1aEd9VZYBrfHgqsIxJJt8dXEDMQSsH55576OB+m5LUItIigOzJ5oP+9GOyGjOJBZNGsP\nHcfY2qTmMbPHIH99s9NwXDOzCANjHrJMFVb8jzDxWkCy4o8kHUNXRCgupCgupPbed9IQB9LzZoTf\nEtPmneaO7/7W7WLueJA5BsUbkMyakb3r9RzViwiq2lP42/sgxvtHZNd92/u7cbN/YRmSM6YdrU5c\nA8hfszODKcKK/xHE8c3ILlGs79ttE/Y2BSnQiB9BL2HHNbZ8xzNmG8ftIaq9xbbbdhFB4ylYOLfr\nfnsdZ+c+DsruncSOdgA0Oy8R2ezQVEwHlb/a17Eshx8r/kcItx4wf71ErL534ZP90loYbgVtrR+f\nifwcY8VLNEVR9hx59zMyj+I9/R43iv1FBPVipgO0o/+pwIr/EUFCZelyESfcrJe7H0+ezkjcXu+p\nxxwaCRdRZfFyEQRK2TjFuWS7fOOhJbPUxZ2yOwctgBS18EePNE1RXcRfHGMmEoF6BUJ/5K2zRIsV\n/yNCK/3y1pIo/dOKwt1N1lZPZVi6XMD1N8+TWa8Rrwasns70fa7LH0/B7cNZiD4QjmdGvPtg8oX8\ngMye2nxcL0NpzSxYZxY3RxPpOaiVzSKxX9/0gLIcKo6g4XY6cf0QiciNe/thQoHKTIxEuYETbO1g\nHCBR9Vm6mCdW6380+ObH0lE09eB4ScgsGZ/49Py+3npUhV+aC8etG/E0zJ+GzCIiDuI45l4cSMzA\nzLzxSsraWpuHESv+R4B4xSdZjmb01WkyauXsqaVibBxLE6/5XV08BYjXQxYvF3F7BHeNBMftbwSf\nzELuGMTTSCwJ8dSRFfRBaAWudZtDmg6i2RF4SfOZWg4V1uxzyEnlazvSNgxKy/xz43SG0HUIPTNG\n8GMuoTR6+viLQna9OprF4HjaJDcLfGN6yCwa90kw24o3IGgY/3cvZrbVy00Xzrm+7fvTTl9eR46D\nJjJQLWxujKVMbESjZtcHJhQr/ocUr+aTW62SGKIffytVQ4tKNk52rdpzIVmAWG3II38RY5duRrF2\nLr62A6vcGORObqY3aKEBVEtE101a2rQWhMOgaQbqiI2oFk2Us2WisOJ/CIlVfRavFCMd7XeiQCW9\nM29M6DrcOJ1l/noJrxHuOLfx/x9y6ob0/Jb0Bd1cKNv+7GwduSoupHJ2pB8xqmpG+Zml9rYtn3si\nY/ILRZ3wzjIQ1uZ/CMmtVHCGKPwqsHG8+4Ksn3BZPpejMuOxPZpABeP2OUw6I26b9Btw1V7ItETK\npv1fun7G4jhmTaC1PpDKmQVjOwMbK3bkfwiQIGQmb1wq/ZhDfMimlfx8AnUECZVYLSBwhWDbiH79\n+Ay51SrpfA1R8GMOG0tp/MSwk7b1H806zbTMYRPzOcSSMH9287mqmcXlr5m1GcvIiUT8ReRu4N2A\nC7xfVe/f9vqbgX8PXGpu+lVVfX8U5z7qOH7IsYsFRBVHQUcwc66n4qQ3quRWNwuF+DGX1ZMz7cVf\nRMgvpsgvJNvPD4xIs8pLH4R+7xTHTSZO+LowjGjfbWcAVRRnIj6HHW1orddklnrXJLAMlYHFX0Rc\n4NeA7wUuAp8VkYdU9Svbdv09Vf25Qc83beRWK1uidg8SvbsfFHCDkNxqdYtXT6wesHC1xI2z21z6\nDigsbitYKJkFxCwUlldN9OhulFbR7PGJELRudFuA7sbQ2x8GoIp4k5s+2ySb84xnkF/tfwBgiYQo\nbP4vA76hqk+rah34XeANERzXAiTKvb15ov6rKFBPusxs1Ha4cwrgNQLcRjQmpx9/QQaS2U1bseuZ\nUaCX2P2Njapx2+zWftX2iHocnYOqmqjX1fNQLRw4DUTPY/e5n4ahcXX1a5G2YTiI+d7nz0JqbtyN\nmSqiEP8zQGfpoovNbdv5IRF5QkQ+JCLnurxu6YL2yJkzFGkTKCykcILegrHba/2STsArTye3+NqD\nyVJJanb3N7txcN2uojbOBV1VNaPt/DWzobIOoY/2KuKyy3FUw457c+tnVKyqxqNm/ZKJfahsbG7f\ncvzJ6RBEpB05TCrbXAi2jIJReft8BLhFVb8N+DPgA912EpG3isjnRORzjeYPd5qRICRwe4/wo5Q5\nBVZOzNBIelTTOz15WvgRuHI+//vrxHpFinWz56fnTTrkhZtg9kR7tjAuWgKqYbgpprUSbFzu3AnW\nr0B5HW10H4Fv36aqxuy1esEcq7wOpVVYu9T1y94u6qhCcWWzWEsYmCIyQWOznUEDmp3LpCHimMpm\nlpEQxYLvJaBzJH+WzYVdAFR1pePp+4Ff7nYgVX0v8F6A3InbJu/XOUpUOXapaHL2jOiU9ZT5OZRm\nE6QLdQibi8wYN878QqrnTGQ/rFa7m47aZpNOMkvN9Autccr4bf3tYDINjVD3RE3Ua7UAmUU0nm5v\nBtNhaGLGuECGgQmEqjcT3gU+BB0Rs5U8msrtmC21RTxoGFPP9plG6O9cUG0GwSk71x6GvxC9B27M\ndAC1gl0DGDJRiP9ngdtE5FaM6L8R+MedO4jIKVVt/QJfD3w1gvMeaVLFBk4wOuEHWLhWYvVkBnUd\nls9mmcnXSJZ9AlcozSbbncOgNEKgvIF21q7VZlLpzhmfuCb/zoQu7oKYNYp+SiAWV8DNG5fHMGgu\nbGv/ka+VDTNiT+XM5+LXzXuDumnHfsxLQQOq+a4mNhM0F+7oZEaFiKDpOeMM0Kia67YlJofCwP9m\nVfVF5OeAT2BcPX9dVb8sIr8IfE5VHwLeLiKvB3xgFXjzoOc96sR6JFHrLKoSpSQKZnHZbQQEMRd1\nHYrzKYr7S3jZP9U8aICmZptiVmuKWYfPt+sZUZMJ9VhpZb7sV5yCxmA+7a1ZxA4OMEKul7uKv4Yh\nVLvPMkZFe2YVS5rOsl42s5rNHSC9sLk+0Kga85jNIbQvIhnKqerHgI9t2/a/dzz+F8C/iOJc00Lg\nOYSys1D6dpfPKFHAq4cEsRGJba1kbr0Qh151bicBEUGTGSPoteK4m7M/ggbUy2iHSU01NAJa2TDr\nBDPzzc9/a+qMXkRtMmp3AvE0zJ+DoGbqCCQz4Ma2dhKzJ836xj4X2KcZG+E7oZT3SKI2CLsd049N\niNgms83sm8Mz+exXrLrtL+IYM8VhE38wo+lEBk1mzWi6VoJK3rxWL29mQVU19/PdnPgMw1wrMLWG\nBZWkSQ/B1k7ImKrEzAS6zows3bDiP4moklmvtYuz9FtYfV+n2HY8xSz4bk/jMBa6pF0eFruJ1haP\nGA2bAW1d9m2Pjg/hAmWtuHvH1RpJa2BiF5LZrUnbWp9RGKA9C95Hw+ZCexc3X8dB3fjQzn0UseI/\ngSQqPjP52lAXe+txh3iz0LsC5WyM/OKYq2u18OLDC2Fu0p+rY9NdU0Nzmz1l2rZjt5BDKfz7pbwG\nCJrs8MWv5I2ZKJbcmsp5iHTrYDQMbTnJfWLFfwJJ5+s9C6ZExdqJGVJlH8KQ2kw8Ev/9yEhkB8sV\n1Cd7FmnfXqi8vIZmj22ZkaiGxh9/Wiivmk7Acbd+No0qVItmDaTDJWFYM4HOGVvbU+wwmt7GiBX/\nCURG4N984mLB/D0VWK9RnE1QWEgN/bx7Ep+BWHIC3Du7iEmjCoUbaHreeCKFgYnk3W3R+kii3T1r\nymvmM4uljP19jwR8A7dCQ0BMVHNp1cYF7BMr/hNIJRMnXu3u6jko7fWDznoACjMbNaozMRqJ4f8k\nfuhbdpmeJzMm//sQaY8Ut5koNk1BzeCsRnX7W43QbNiiJD1pubMOcbF+83tqHj+WNPEWrQA5S19Y\n8Z9AKjMe6bxLrB4M3fzTQhSShcbQxf9tb9/gB7K7+WMP17sHMHbqeAp1vW2Lyq2I3KL1GZ9gdnYq\ngmaWaHfoQd3MBHy7BrAbE+LXZ2mR3qhy8nzBFGxR8J3olxJ3sXRHfKYDUC8NN+/M+hVjqilch8Ak\nXjM5ekIorRn7vRX+wamXR54/qJ0h1ktA7gQ4dmy7G/bTmSBShdqWPPoCpiYHHc8HpGd6aIFqZoyu\ncvEUuImm8A6xWoE2hT0MTM4bxzOLl36dUXZ+j9z54K6vv/Jd55CXv2JErRkCpVXw4mhHveVh0m02\nQCpn2mHpihX/CSKzvjOPvqPDlyQFSrkEjeQYfg4ikDtpirI7zuYi3jBQTJBQZyHx0B/aSF8/82ke\nfcduid96Y963+d5Xvmszd+Kh6BQ0NBG3MwtoIrM1GV7z+x12TID1+98dK/4ThOv3Dk2P4m/Sazwd\nOFBYHJOnT2oWOmzvIs6QzQXDH93n7/sgjz8crfBs7UQu8JK7jD07+/rnTnZnUDLV2TSZNTOsetnM\nsrLHIjtFt0A9k77aJoTbDSv+E0Qj7pIYcnH2kK0LPaFAcX6MLp7xmS5FXYY0IhTp7sETEZumnOGP\nONudy8ObM4RX/fU9Qz/vgWhUts62wGQojai+cFfhR6FiUz3shhX/CaKwmCJ2pbjF9NPhfLgjHQPs\nf0bQSLrEOjqY4lyScm400+O9PX2GjKpxC4y4AziIecf3zTfoedF1dK3O5yV31Sd7RtDM1TOMTr5d\nUa1w3S7c74EV/wminvRYOZUht1ohVgu2+uJvoxVDud9OoJaKsXoygxOEBK4DERRnGYhacUf64JbZ\nZygzgCF4gOxH+Ov1kOtXGtTqLfGH+QWPeMIhkdifIJZLAfmNgDCEbM4lkzUj6ccfjrdnBBPZEbjD\nS98hImhQHyx19pRgxX/CaCQ9Vk5nWbrUdPfchYPk9Q88B3WEwJmQdA4VU+BE9yrcHgVCpIVB9mPb\nN2UU4dL5OmHH0o7fgOVrZoQq0ixh0Oz0XQ8SCYdsziORFErFkFotJB53qFUD8hthO6i1Wgkp5IVT\nZ+JbOpDOjgAmxDQU+kNL32HqGFdNnYVWTWNLV6z4Typ9rkvu9y8Ur/pUspPkBaHGtz53fKhZPFVD\nk6snohGhMbF0/xzDQMlv+JQrISg0GuA3+inAbjqDFr4PtWpIfqOO08ysbMR+56BAFaoVpVwKmcn0\n7tgfufPBtufQ2GYDqTkg+jTQbUeB9DybPtJqTEA24GsHVvwnlHI2RmwliHxm7AQhC1eKeI2ARtyl\nMJ/EH0FKh11JtJKBRccOj6FGbWs1qAPSGu37vlIuBYgI6RkH1zXtL5cDrlyM3uQQ9lGjRBWuXm4w\ntxCysNjbv37TTHVh9DOBeBoSwynNufWY0v5JafY4rF2M/HyHHSv+E0otHYOV6D1TEmWfVlZ61/dJ\nVIqsL6UIYi6NhDu06fiD9xeZLfRYgHOi8froZIcHSCxh0jEPMAJsjfbX13xWbvhbuqtjJzw21n1q\nw3Mm6puNtQDfV06c3HuG98idD/KSu+rk3vljI2gZkMh0neENt3C8DGWh/7Bj0ztMKF4j2nJ0rXFw\nZ1E+wQSRzS9XWLxS5OSzeRKlMSyU1crN4J9hIiZj6AFpedJUKgEry76xKnTcrl+dDOEH055SIWx7\nFO3F4w/H94w4jow90mi3btGf10rdduwnMqFEXU5xtzFVqxNwQmX+egm3MdxYgx3US+DX2x3AqHPC\n7EWnMF69dEi8SAQafawzdPLInQ+Sv++DQ2pQk1qpe0evIeSvm/TYQSPa38CQ4zsOK1b8J5Qg5lJN\neZHGo/YzqRaFVGHEi2OO16yGJUOc/qvpZPb7rs98uv24Vgv7sr1PAu2qk/tk6LOAWhEaNVN5C7MQ\nrxqa9Ri/CqUVk3OpvN5+bZDZQPt9yZxZaE7PD2WN6TBixX+CWTs5M/I8mwK4wQgVznFNecRYymRk\njFj428JRLezb3r89eKtcHPGMaEDy6wcPchrqLKBwHYrLaGUDyhsmB9D2kXk1D6sXoLQOHHxA0P5N\npXKQyiGpHMwswPxp89ubYqz4TzIi1CMe/bfoDBDrJJTmYvOoSGaHFu0JzYXfRjWSUovV6uES/0I+\nZH21wdXLdZavNajV9tepP/5wfMvMJ1Ja30k1byJyu+HGID0biQtw58BCREBcSC8MfNzDjBX/Cacw\nn0SHNENteELYcexQwI+7VEcp/l5iqNkdTYKvg5uxqpWQZ56u8tTXq5QPYbXGlRsBpWJIfiPg0vk6\n+Y39zQYefceF4XUAe5E7MbSFWhExacSnGCv+E04j6bF2YoZGzGmP1qOYCQjghbB+LE016VFLuOQX\nktw4nRlJ8fQ2US/ubcP8yQ/m5bN+oc6li3WCI5IiRhVuXPcJw/193o++48LovIFaxFMMswC8xYr/\noaCWjrF8LseVW2fx3Wj/DNVMnNXTGVbOZCnPJkcr/GCm/du6s8g7A8c1hWL2yRd/f2UiiptFiQLX\nr9a5frVOqRjs67N+5M4HRzcLkOHFnGxyxL7cfWLF/xCRKtTxAo0st385M0LzTi8CH/LXUb/RXpwd\nymhv9sS+bbzLTx5B90CFUlEp5EOuXWlw+WJ9Xx3AQYvTREVnLMAggwTjBFCMsGWHDyv+E4Tjh+Ru\nlDl2Mc/81RKxaoe9QZXcanVg4W+ZjXxPKMwnBzxaRPg12LhsvDuKNyIP+Gov9iUzxo7cJ+lbj/aC\nYCsf0PqqTxBM2Axgt7Wg4kokqTqicAI4zFjxnxDcRsDxiwVm8nVi9ZBkucHilSLJolmslFBx9mmr\n7Ubr7+T5ysln8yxeyhOr+K2MYWNGTaWnoDGUiF8RAS8ByVmIpTZd/Vyv68Jio0e0cyOWoJBbwHc9\nqqkZitk5dpuPuR7E4602DHwZkbO6EvDM0zUuX6z13QkMfwYQdh/Ztwq1JLPNp/v/3aqqCSabcrNP\nJLl9RORu4N2AC7xfVe/f9noC+E3gdmAF+Eeq+kwU5z4qZFerSKhbUi+IwuyNCtWZGOoIKmbboHTq\nT7wWsnSliIrxLCrNJsavUPUqpGJb/thRmYJEBE3Pdn9RQ2MKCOqEQUDmucd47ptvp/FdL+I/vHPb\nKLGHMMWrZZ7zlc9x08qzHD8ZI5HoPr4qlXyuXpqglWSFSlm5eqnOmZv6Wx955M4Hh5cYrlrqHYyV\nzA7uJVaZ7lE/RDDyFxEX+DXgdcALgR8VkRdu2+0twJqqPg/4FeDfDXreo0ai6ncdO4oqTqAgQmk2\nscU1MwpaSd4chexalZmN4dQ9/avLffrIu3FIZRFx2uaaYSR963ZDHBMMlFnCmT3Bdz94Dyfe+B07\nhd8cZOfNcainMzx5+6v5y793DxcXb+IbuZvJx3Z6G83MeDz3W5IsLE1WoFG1qtTr/c+6hmb+CerN\nKF9Fw7B9o7QCXnzw30Sv2IIpIoqR/8uAb6jq0wAi8rvAG4CvdOzzBuBfNR9/CPhVERGdtCQuYyR0\nBbfLlFsUtFltq2Wjz6wbgY483bNCdr1GKRcnXg1IlhuoCJVsHD8+mEi954FZ6KeMY9PFbxx0E5R3\n/5sD2JZFqMVSPHzmzvamudo6t+QvcLZ8lePV1fYVzi/ESKcdLl1oTIblDVNiMt5nyYdH33GBV/31\nkOoCVAvGPBNLmVlZo2oys0ZRBSyWhkY5ilYeWqKw+Z+hVSbIcLG5res+quoDG8BiBOc+MhRnkztG\n9SFQaZp8ABChsJBi+UxmaNZKCZW562UWrpWYydfJbNQ4dqlAekgzgknnyqUDrj1smxWsJ+b4wrEX\n88c3fy+/edsPc37mVHvXRNLFi+1UMxGYX3TJ5g7a+oMRj+9PWYe6+KuhycnUqABqUnREMTboZfqb\nIiZqwVdE3ioinxORzzUqG+NuzkipZGKUZhOomEhbFainPDaOpbfslyg3WLpcHNrYOHSEZNnH0U2T\nkCjMrlZwBsz5854HZvlIYY/JZn2yFuIWFiP6i3R0BL7r8akzr2QlMdd++dTpWHvdubVrNucyv+Bx\n/GSS59yWwB2Rhaha2d/nX3joqSG1pBvazAx68N+IiJjUEVNOFL/sS8C5judnm9u67iMiHjCLWfjd\ngqq+V1XvUNU7Yqkp65mbo/qrN8+ycirDtXM5Vk9lNkf9AKrMLpfbwtzeTDRyqUAjJl1/FAokKoMv\nUL7ngVk2srssKAZ+R0bHIeWgr3nVAAAgAElEQVR23wc//y+XhnLcUBy+tPD89vNY3OHmWxOcPB3j\n2IkY526Jc+xEbEs+mptuTZAYgXfu2ur+0lY//nB8+KmgOymtgm71BjrQbyVx8PoOR4EoxP+zwG0i\ncquIxIE3Ag9t2+ch4E3Nxz8M/Lm193dHHaGR9Ai9nV+NEyhuF3fPKGcBgef27EiiyjH0zx7eYwhb\nLZhMj6U1YO8/9TB/SmfOxfmRn8xEf2AR1mLZbZuEdNolm3OJdannIAJel99F1NRr+89gOtQkcDQr\n23f+0jeugF/bFP19BgeKCMwsEu2/53Ax8C+pacP/OeATwFeB31fVL4vIL4rI65u7/WdgUUS+AbwD\nuHfQ804j6sjQLSKJqt/zFLVUNFPlyx/vI6FWGECtYGqvdvzJtwu9iQcY7ofy/T84x2/81zN87+vM\nKmhUs6z0i/Y3qygVQ8ql0aTbvnJ5/1lAh+L7n56DhbOQO2Xu03OAGDdQr7UqrQd3T57i0b9M6gA8\nd+I2ffmPvWvczZg45q+WSJYbW8YrYdP/P6q0D74neL62R/qizYyfCZfCfIp6anAnsdN3V/iN2w/g\nbRFLGQFwvLY7ICKQWdoSqBWle2jrP/KBP6nxwU/6ZG6UyOYbA3/eDc9h+aYcHyo/wOMP7+1ec+Vi\nnXJ5dLUWXBdm512yOQ/P6/9qI/P9T+Z2pHRWDZuVatxIvmOtFowZ6QjhLP3UY6p6x577jaIxluhY\nP56mnnQJBULHmGLK2TgbC4nosn36yvpiinLWuNUJ4CokqgELV4skyoOXMrz88RRvfiy9947baVTM\nlH/tAuSvmdQQjaqZIWxchfJae0GwJdoHjQJt3wi5731VPvhJs+ZRXEwTMPgMwAlDHD/kh9Nv59ue\n+PFd9w1DpbrPkfigBAGsrQSc/2aNaqX/c0dm/0/lduTyF3GiE34NITgkZTmHgBX/Q4Y6wsrpLMtn\nsqwen+HaTTnyS2lqM/vPWtkLAeZWKiTz9R0/EEcht1KJ5DyXP57affF3vwR1s15QWjFrBrUCukf1\nru2JwlpBRdSKsH6ZPzl/he/75xU++2SHDVyESmbw2Y8TwvHzeWJVn3vuzfCfH3hTz32XrzXGEpfU\nKlB/7Wr/CeD6mcX0RY9c/tEIf/PCaoewSENEWPE/pARxl3o6Ruiar9AbQtH1XsuyXiOMLBfQPfdm\nDjYD2IvQNwvGG1egtLJjzaAt8sUbZgbRqJgZRGHZzCpKq7z2l2b5lXd3d7L3IhiEC+YPuHS5yMlv\nrlP7dZ+TH9hpMglDpVgcb/HgwAd/H85ekeT/j3BUviMbqF8zM8Uh5JA6LFjxPyIEbrRfZcvHvxcL\nV0qRdQCRzwC2UyuZjKHlVahX0FrZZIZcu2ASyfk1I/qF69Co8JGCx2vfuXsMoueHkfmJtNJrxKsB\nv3S/yy/f9ZYtr7dymY2b/Q64B+4ASqttl184oDtnky1pQhpV0+GHE5RbaQxY8T/EOEFIZrXC4qUC\nszdGF6ouQLzmk1mPLt/9Pfdm9g4AG4hm/vbisrnVt073N7IJXvvORV77zkWTimIXHD+ksYtL7EFp\ndQKzNyq88l2boTOOA94wP5o+iMXY16Jvi4HSP/s12DCzMg0azXKcg33qIgKxpCkWM+VY8T+kOH7I\nsQsFMus1ErWAeG20BmFHIV2IdrHsPQ/M7jnijpo3P5bmzY+luefevX35JQhZuFLkxPk8yYq59mEM\nyF0/5Pv+7O+3OwARaQZ8DeFkfRKLH1wqBooADupmVrZ+GfLXu+5yoBmBZyN8rfgfUrJrFZxQ21/g\nbmaaoVkMOv5wbiMgs1Yhu1IxRWgGMAm99p2LQ54FbI70L3881V/cAbBwtUSi4rdH6MPS4paL7fd9\n8vt5yV1mwTo943LmpjjpmV3qBriQnhnOX7pcCtED1pOILAJYQyitbTMFheBXoVFFw32UpfSn18un\nxZgnk5aDkix3TwEdRcLDflBMxHFmrUrgwOxqtV1rYCZfo5KJsbGUPnDwzXsemOU9zccP3m/K7c0W\nBksut5FN9DXC74bbCIjVgqF/tqFAKbdZU0Ef+B/g234LgETC4cSpOM88VevZtw4rCEwVQu3tBLAX\njz8c5yX3fZDcO39ssIbUisYclMig4pg1m0bT+0wcmD+769tV1axeq03pbMX/kBI63VNAb2cYo/5W\nByOYGgCwtcMRhVSxQSXjU48gKrgl2Kfv3pSefgLEPlLw+IOvG7fDfkf3vXB7LPBG1Rm0vqdyNk5h\nYTOBzz33ZnjwiR/niWYH4DhCJutQLIQ7OoBgiHrmOOY2CI8/HIcoCsAEDRPP0Yk4JupXQxSnqzto\ne1aQvzrY+Y8IVvwPKcXZBLMrFZwOARjVqF96PN6yj0KqWI9E/Ft0CvhrBxTz/RI6w/9ky5kY+aW9\n3V6PHY+hNCh16QCGxexcNIFVMIQKYI4HsydBBBGnp+lHRMxrqdmdnccUYm3+40CVueVljl28iNs4\nmO2xko1TzjZTQDu9R/jjTFvlVX2yqxXiE1MjeB8EIbnlEiee2eDY+Xy7lvKwEEzQV1/7OsKJk3Fu\neU6CE6e8kSwEz2S3SoWqsr7m88xTVZ76epWLz+4vCjiSOIAWmUUQpx0NvFsnJSKQGEJcySHEjvxH\nzMzGBnf94R+SLhRQx8EJQz531108/aIX7e9AIuSXUhTnE3i1AFGYv17aUeN3VLOBbsQbSny9xsx6\njepMjPXjB18DGCXih5y4kG/nS3JDxduotzvYYVxBKFCd2d8syXEFdwRZPgFWbwScOrNpdltd8dlY\nC9p9eq2mXL5Y58y5OIlkf21qdQAvuas+2FrAfuv5HrJxyLCwI/9Rosp3f/jDZNfXifk+8Xodz/e5\n4+GHWbxy5UCHDF2HejpGbSZGI7HT93xcUisdNwdIlRrMLpcPxQxg/np5R6K8Tm8q7bgNgm67bxzA\nnTKZlJH0p5WOUX0Y6hbhb6FqOoX98vjD8WhnArugU57SoRMr/iNk4do1kuUyzrZ/jeP73Pb44wMf\nf2NxtHbw/SBAuthg8VKh7TqSKDdMkrgJ6xAS1e6eVNLlNgjScS9q0jzEDmAim18cfsCShuA3TLt8\nv3f76gMkn3vkzgd55M4H9+8W2qj25eJpPH3qMGVVAnthzT4jJFGpoF2GaQ6QKg8eoesGJg3zdtPP\ndjpfHuXMQIB4PeT4sxvGvt1x8rVjaWqZiBKCHUIEQGHpSpFGwmVlexW3LhTyPsvX/XY6+2H3oVcu\n1zh7U2LXSN/YPuv/dqPlFfTKd51DXr5LcXhxTHrvWHJPs48R/obJ52MB7Mh/pNw4dQqniz+e73lc\nes5zBj6+H3N7+v53Pg4FamMKcDQ2dPPDc3TztnC9zOKlPI4fnZ+6BCFuI9iXKsaq/q4L6MOkNZuI\n1QJjItuFWjVk+ZqPhpuZN4dNvQZP/22N89+skehibhKBhcXofliPvuMC+fs+2D09RHrB+PQnMv3b\n+3tECE8rduQ/QhrJJF98xSt48Wc+g9dMkeh7HqVslqf+zt8Z+PhBzKGSjpEsN9ouoAog4DumKngp\nGydsBWUNfMaD0cukEq+FLF4qsHwu2y4mEzQXNCVUMxLu448uQcj89TKJio+KSYO9vpRC1Pjr15Me\nviekig1i9QDfc6hl4rh+yMLVUs82RkE/C/CCWSNZ194VqtbX/LFZy4IAgoqSnhEqFUVDiMWEpeMe\nyVS048nHH47DwxeAC5szgXgakv2LvqpCNW8Du7ZhxX/EfPU7v5O148f5li98gUS1yoXnPY9vvPjF\nBLFoRkzrx9Nk1qrM5OuIKvWkx8ZiiiC+aRc+diG/JT5gO+PyEBLAC5QTz+TbmqciiCqioA4UZpOU\n5jYjYB0/JLdSIVkyLrPlbIx4LSBWD9u2dAJl4Vp5x3UJHYuu67X2tmFfY9+fb3PHbpHNjcb4UxHX\nasqtzzXZWKOsnNYLUybyAq/60tv2L/zl9eE27hBixX8MXL35Zq7efPNwDi5CcSFFcaH34q/TI0fL\nMF0Z+0VophBozVxUNxdGQ8iuV0GgNJeEUFm6VMANNveZaSab63YN3caknYuuoxpIt87V2Ql0tleB\nRtwFR/jT+1a6esLEYg616vhz/MNohL+T6rUqqTN7pwA3eX/qVvh7YG3+U0g15e0qdJPke7NdVhyF\n3GqVxcsFFi9vFf5u+++1fb/7RIWKmaVdP5cjdISwNdNpvXYsxYP3F3u6QKbS4//ret7ohR9g+aNP\ndvXu6dxmHoup22Dpyvh/QZaRU1hIETrSNSag05d9UhFMPeF4fbj5doaJKDQSHkHM4fq5LIX5JJW0\nR2EuwfVzOfyEh7z913u+fyYzXPdOEcjNOj2XWMzi7ngMBxff/zc0Vss7OoDOjqj9OJEdZdMOFVb8\np5DQc1g+l6UyE+sq8p1miUnvBHoxye1WoJYywg+grkNpLsnayQzFhRSh5/Ch8gO71sJ1Xdk1vfOg\nxBOwdDzG0nGPWKx5nuadF4NjJzyys+MRfz9f47OvfT9BeffUKDaVw+5Ym/+UEroO5VyCZKnRVUTr\nKY/CfJKZlTKpWnQlC0fFpLe3nuw9cv/Eaz7Ko+/YO+ZhZsalXBpOKcJE3GTGzM165MYk8ruhgeIm\n9m5X+ak1/HdFkEr6CDJ536plZLiN7pGsIVDJxGkkPWJ9pI3uZJy5hA4LKiYmoxfGq2WX96ty/Wqd\nYmF485tCISSZ9snmJlMiwppPUK7j5ZI991FVzv+nz7DcDBqDCPIIHSEm85u1jITsWq27UAtUMsb1\n1PN1X2JuhX+TbpHUiok76JbE7fTdFd7y9g/sedwb1xtDFX4wQWMrN3wy2ehSOR8Uby7JmZ+8ncW7\nnkP9RolLH3iMtb9+lvPv+xtu/YVXIV0ioVWV4levs/zRJ7dsf7yjI2iXydwtivgIY8V/WlHF6zGq\nF2XPYKpJcAudZBRoJFzWjqWZu1EmXjUBRrWUx8ax7tlNH/jw+3ic3c09YajkN0bj4hk00wyNU/u9\n2SS3/+Gb8OZTbTPP7B1n+eZ/eJSL7/sb5l5+jvk7b9nSQakqQanO//eD/2XXY2/OsC5M5YzAiv8E\nk1lf5wWf/zyzKyvcOHmSr3/7t1PJHKwM4Q5ECNzu1cCCjjTBtaRLohrs8EMPHGHjWIrsaoVYY3+z\ng6OMYuorbCymqGbiIMLK6aypgQjQZZT64P1Fnvi239pT+IGR+vZLfwHVQ+X0T3wH3lxqi33fTce5\n9RdexdUPfZEv/ZM/4IW/+gYW73qu+WzDpvD/o/0lh2vNCKapE7DiP6EsXrnCd3/4w7i+j6PKscuX\n+dbPf55SNsszL3gBT95+O43E3oEuu5GfT+6oBhYK5Oc3j7txbIZjl/IQdngBCaycmiFIeNRm4sxf\nLfasKXwU6HcdQ4FStlmNa7tq9kjS9qHyAzz+bf0ltKtWwgOlTD4os/PjN/ksvvpW3OROmVI/IPOC\n4+Q/f4mv/Owfkbp1ntnvOEN9pczao8+gB8wRNU1mISv+E8rLPvlJYh1VvlppoLP5PC/6m7/h+V/4\nAn/8Ez9BNXtwP+ZKLgFi6vC6vhJ4Qn4+STW7Kf5BzOHaTbOk8zVi9YBG3KWcjaPu5uxgYylN7HIB\np7k+cBQ7gV4dQGt7KOazKiz2V7Cm22g/DE11rGI+RBzIzbrkZo0Al4oB1640RpbPRwTmF4afKnov\nateKZP6O7rDri+fSWN1Mflf55hqVb0ZbmvGom4UGEn8RWQB+D7gFeAb4EVXd8Q2ISAB8sfn0vKq+\nfpDzHnUc32d2dbXn6wLE6nVe99u/zR/+k3+CDlBZu5JNUMnuPoNQR0w6hR6EnsP1czlml8ukiwcr\nSzkpbF/L2J4RtZusV5Mu1Zk45Wy85wi/k26jfVXl0vk69frmGVeWfcqlgNycx/WroxN+0x6oVSE1\nZjf5Sx94jPnvugU3vblAHjYCSn97g8ozo6vD+/gR9BgadOR/L/ApVb1fRO5tPv9fu+xXUdWXDniu\nqUEdxwh6l/TPLUyUa5VTzz7L5VtvHV3jeiFCYSFFutQYW4RVFG6mCgQxwWvsnMVsz8cTCpRyCQp9\nFNHpzNHTzbZfLIQ0Gls/OFUol5RyaTwdatgjB9Qo2fjsRZ66/y947r13oYEinkPpa8t8+Wf/cGxt\n2m4aOqxmoUHF/w3AXc3HHwAeprv4W/aBOg7PvOAF3Pzkk3i7dACocvzChckQf8wMYO1YmrmOXPSi\nm4K5r4yW+yCK47dkrjCfILve3QW2dfxQTJBccTZBOdfbXn/67gq/cXvZVKj62O7nr1TCSStoFnl6\n5oNy9fef4PoffYWZ5y/RWKtQvTA5lbhamUbh8M0IpJ/yZz3fLLKuqnPNxwKstZ5v288HvgD4wP2q\n2rXbFpG3Am8FSGaP3f7Kn/7PB27bYcdtNHjVH/8xxy9exG12ANsFSYHQdcnPz/OXb3gD6UKBlz7y\nCAvLy1RmZvjSy1/ON1/4wpG3XUIlUTGj1VoqhtcImL9a2pGEbRAOWo1sN9v9tbMZYr4yf63UM+V1\nKLB6MkM91XvctFdqhm6srjRYW5mcfPOLSy5zC2Oq+HMEGGdH4Cz91GOqesde++0p/iLySeBkl5fu\nAz7QKfYisqaq812OcUZVL4nIc4A/B75HVZ/a7by5E7fpy3/sXXu1/8iTWV/n2IULvOxTn8Khu3CF\nQDmbJVkub5kp+J7H43/37/K1228fVXN7cuLZja5upd3oFUPQ+e6GJ4SeQ7Lan2Aq4McEJ1DcLo4g\ngSNcu2UW1w85fiHfsxSmAtduyhF2uMOevrvCu+8KeOLbfquvtnTD95Vnn96Zt3/UJJPCwpJHKj3+\nxd6jwqhNQ/2K/55mH1V9Ta/XROSaiJxS1SsicgroWidNVS81758WkYeBbwd2FX+LoTg3R3Fujuc8\n+STHL13quo8DzBQKO7Z7vs+LP/MZvv7Sl6LueP/MvWIKumlsPeHiBCFuoDhKO93xyskZ/ISHCiBC\nZq1KvBbsGKVvH92HYjySKtk46Y0qudXqDvfW4pxZ9A68ndXQ2vsB5Wycn31HgR/IGpfLR+58ED4G\nT+znw+iC5wmut5kjf1ycuWkw92HLTibVNDSozf8h4E3A/c37P9q+g4jMA2VVrYnIEnAn8MsDnnfq\nWLpypa/yf9txwpBUuUx5AJfQKCjOJ5m7Xt4hutW0RynXFBwRGgm3XY08VWoQr/j4nkMlG98y2gYo\n5+JkNmpbCr60s5F2JKKsZOLtdBXlXAInUDIbm6Ps0myC0uym6K0fT5NZrzKzUccJ1aQ3psR3Z77C\nXemv8sW74zwywGdRq4WUiwF+oKSSLumMg+MIrjte8R/z+GAqmCSvoUHF/37g90XkLcCzwI8AiMgd\nwM+o6k8D3wr8JxEJMYPU+1X1KwOed+oIPA+3Xt//G1Wppnp7o6QKBdwgoDg7O9RwzupMnPxCSG61\nChhRrs7EWO+R6gCRpmj3tp2HrsPy6QyzNyokqj4IlDMx8gspYrUAN1DqSZegM4laq9LZfNLENrjS\nds88fXelvdsDH34fj39l57m/2EcUbi9UlRvLPvn1TVNVHmODiseFg3y9UbKwZMN+Rsm4vYYGWvAd\nJtbmv5Xv+Mu/5HlPPLG79882zAhYuHDbbXz6ta9l/vp1bn/4YeaXl2kkEviuS7JSQR2HWjLJ/3v3\n3SyfPTu8iwBQxfVDQtcxBdkjPO5+O68H7y+2H+c++Zd7ZtMclEo54PLFyYqDaH1k8wsucwve2CN6\nLYN3BJEt+I4LK/5bcX2fV33kI23vn/38RX3X5crNN3Pq/Hk8f9OusN023ojF+OM3vSm6/EETwOm7\nKzzw4fft2L5fb5yDUK2E5PMBGiqZrEux4A89G+d+uPk5ccIAvJjgRNkRWyLjIKYhK/5HlNzKCjd/\n7Wu84POfB8fB8X2ccO9iKy1b+G6e24Hr8uXv/E6+9Iqto45+Uw1bNlldabC+GrR990XAGbNNvxMv\nBjff2jtq2zJZvOqv7+l7Xyv+Rxy30eDk+fMsXb7MCx97rC/x72dsl805LB2PUSwE+A0lkXRIzzjW\nHLAP/IZy/pnajqCt5jr2RHDsxGRW6LLszV4J5yJz9bRMJkEsxqXnPpdrZ8/ywsce23P/fqRbBGIx\n4dmnjXAZM3pALC6cORc/0qYBVaVeVxyBWHywyNZyufu6jKoZcfsjNvs7DoQdsQ25WYdszrr2HFY6\nE87Bwb2GrPgfcoJYDBVBIhpS5jeCLUKhCvWaGcnGPCGT28w0OWrCUFE1xcujZH21werKponGcYzf\nfSwhzM97JJL76wwcRyamgLwInD4bR8QEksUTDp53dDvxaaRbGup+sOJ/iGl5q/zFf1ti/bPLAx9P\nFfweNunAh8BXqlWfYsHn9NnEyDoAv6Fcu1qnWjGSmkgIx0/GiCccatWQ5esNalXFaaZBXljqz2tF\nVbl8cfO4LcIQ6nUzEygX65w4FWMms/tIWVVZW/XZWNvaee68lr2vN1LEuJGKI8Rt/NaRZz8ea1b8\nDyGfeM1HKTz0VDstcKYasD7C81crsLbqs7DYO/dLoxGCGk+SQToJVeXihdqWhdJaTbl0oc6pszEu\nX9hMdRyGsLEe4PvKiVN7e/Pk14Mdwr/z/LB8rbHrukcYKheerY1e2PdABJaOeV1r3FosVvwPAdsT\nhT36MaAj2CiRdJlbCFlfC0aWTnltJSCddkimzIhYVanXlGo1ZH3Nb4u168GJU3ESCSEMm14v+xCj\ncikk7GJCV4Ub1/0dC6iqUCqG+L7uat5oNELWVvtzvQlDMyOKdenrfF+5cb0xMcIfi5vP1/OEuXlv\nYjJzWiYPK/4TxtvebtLV/nen3XaisH5quy4uxZiZcVm50dhzNBsVly40SM8EZDIOy8s+2sXc4Tfg\n8oX6FjfH9Ixw7EQMz+stTGGoBAHU691THatCo979OlXh2uU6s/MeM5mdI/arl2uUiv1/RqpQLPjM\nzrk4zcI5qsqN6z75/Og63L1wHDh1Oj7wgrVlOrDiPwG87e0b/PdfedTY65p53w+SKCyZcjh1Js75\nZ2oj8ycvl0LKpd3rpapu9W8vl5Rnn67jeSalQDbndeyrLF9rUMjvfkwRiCekZ0dXrSrVKw3iCTh1\nJt7uaFZvNPYl/C1WbwSs3gjarrD5jYDChAi/OJDLOcwtxOxirqVvrPiPgR1BUx+DRyM6tuMIZ29K\nsLLcoFg4WBHrUeH7cP2q314/mMk4XL/aX7sdB+YXXa5e2mn66aReg2efrjO34DK/4LK2OljO/EI+\npFSs7bqoOypEIJEUTp052m64luFgxX9EdBvdDwvPE06cijM7H3L9SmNHecBJo1GHa1cazGSk71F5\nEMCVi37f6XzWVwPWBxT+FuMW/kzWIZl0SCQdEsnBFtQt04sV/yHxidd8tP24JfhRje77JZl0OHdL\nnHIp5OrlCVmR3IVyaf+d1KREzI4Kx4HjJ2NW8C0DY8U/Qt729g3m7v4doOWRM35EhJmMy83Pcchv\n+OQ3gonJL7OdaRPyg3DmprgVfkskWPE/IKfvrvBD31Jviz0wdHPOIHiesLAYY37B4+rl+oFG2Zbx\ncvykR9x68lgiwor/PvnT+1bQz3yaR98+3Nzvw0JEOHk6TqkYsr7qU6vZTmDSaJVzdF3j0TSTccnm\nXLuoa4kUK/578Kf3rZg6rU0emeDRfb+ICJmsSybr4vvK2mqD/EY4EW6L04wIzGScvqKTLZZBseK/\nje3mnKMg9rvhecKx43GWjimVcsj1qw32USzMEhGuB3NzLrPz9i9pGQ1T/0s7fXeF37i9DED+vg/y\n+Nunc9QlIqRnXM7d4rB6ozkTsAyVVgrtk6djNirXMnKmVvw/8ZqPtl0wH2lvnU7h78R1hWMn4uRm\nQy6eH3NF8SNMOi0sHbeibxkfUyH+p++uAPDuu4J2vpxJccWcVBJJhzM3xbh0fvLjAw4bIpCd9azw\nW8bKkRX/t719gx/I+maxdoB8OdNMMuly6iwsN9cBVE0ufXGUamXcrZtMXA/CoHfMggik0g4zGSv8\nlvFypMT/Q+UH2o8fvzveYc6xHJR02uWmWx2CwAhXq4pWuRRw/VrDCB3gyPjTHoyTdFqYX4qRTDrt\n4u0IoBCLC8mEqe41k3FtTWTLRHCoxf/03RX+H//P29Vr+kl9bNk/IoK37ZeSnnG5+VaHwDdZJUvF\ngOVr+wsd3l5bdiIRmF8wPvYry92vL5EQTp3dLJO1sBhjbs6jXldcD2IxO8q3TB6HSvxbZQsBY7sf\nQ74cyyYigtcscJLNuV2Lq/R+L8zOu6ytTK5fqYgZtc813S9Xbvg7YiGM/X5niUfHFZIpO7q3TC4T\nL/5ve/sGr77/Azz+cJwn7CLtxCJiUgtfuWQ8hFQBMTV1XUdZX9ssyiICuTmXuXlv4sR/JuOQTAlB\nYBLjdZpolo55rCz7W64jFheyud3r+1osk4johGbTekFyVt97yyvH3QzLPgkDpVQKCQMlNeO0c9HU\naiHFglkgmMm6JJNm+9pqg9UbB+8ARHYurrZEuVHXvmYircjahSVvTxNNtRKS3/DxfZNaOZO1aRcs\nk8Wrv/axx1T1jr32m/iRv+Vw4bjdR8KJhEMisVNY5xdiJBIO66tGUJMpoVjoXrqxk1TaYXbOnCed\ndqjVlGIhQBzIZF3iMeH6tZ2FYbqts5pCL15fi7DJlEMyZdeWLIcfK/6WsZOecUnPbHYYx04opWJI\nuRwS85RSyRSHb+F5wvGTW0sWJlOyo1j5iVNxlo4rvh/iN6BWUzzXzDwcx7hkOi7W88YylQwk/iJy\nD/CvgG8FXqaqn+ux393AuwEXeL+q3j/IeS1Hm87EcwBzC0qtqtRqIbGYkEr37yrpuoLruiQSMJPZ\n9pod+limmEF90L4E/CDwV712EBEX+DXgdcALgR8VkRcOeF7LFCFiRvWzcx7pGdeO1C2WCBho7KOq\nX4U9p80vA76hqk83907Q0QAAAAU6SURBVP1d4A3AVwY5t8VisVgOziiiT84AnZVPLja37UBE3ioi\nnxORz60HNqmYxWKxDIs9R/4i8kngZJeX7lPVP4qyMar6XuC9YFw9ozy2xWKxWDbZU/xV9TUDnuMS\ncK7j+dnmNovFYrGMiVGYfT4L3CYit4pIHHgj8NAIzmuxWCyWHgwk/iLyD0XkIvB3gY+KyCea20+L\nyMcAVNUHfg74BPBV4PdV9cuDNdtisVgsgzCot89/Bf5rl+2Xgb/f8fxjtLPqWywWi2Xc2FyzFovF\nMoVY8bdYLJYpxIq/xWKxTCFW/C0Wi2UKseJvsVgsU4gVf4vFYplCrPhbLBbLFGLF32KxWKYQK/4W\ni8UyhVjxt1gslinEir/FYrFMIVb8LRaLZQqx4m+xWCxTiBV/i8VimUKs+FssFssUYsXfYrFYphAr\n/haLxTKFWPG3WCyWKcSKv8VisUwhVvwtFotlCrHib7FYLFOIFX+LxWKZQqz4WywWyxRixd9isVim\nECv+FovFMoVY8bdYLJYpRFR13G3oiogsA8+Oux0RsgTcGHcjhoy9xqPBUb/Go359N6vqsb12mljx\nP2qIyOdU9Y5xt2OY2Gs8Ghz1azzq19cv1uxjsVgsU4gVf4vFYplCrPiPjveOuwEjwF7j0eCoX+NR\nv76+sDZ/i8VimULsyN9isVimECv+Q0JE7hGRL4tIKCI9PQtE5G4R+ZqIfENE7h1lGwdFRBZE5M9E\n5G+b9/M99gtE5AvN20OjbudB2Ot7EZGEiPxe8/XPiMgto2/lwenj+t4sIssd39tPj6OdgyAivy4i\n10XkSz1eFxF5oPkZPCEi3zHqNo4TK/7D40vADwJ/1WsHEXGBXwNeB7wQ+FEReeFomhcJ9wKfUtXb\ngE81n3ejoqovbd5eP7rmHYw+v5e3AGuq+jzgV4B/N9pWHpx9/O5+r+N7e/9IGxkNvwHcvcvrrwNu\na97eCvzHEbRpYrDiPyRU9auq+rU9dnsZ8A1VfVpV68DvAm8Yfusi4w3AB5qPPwD8gzG2JUr6+V46\nr/1DwPeIiIywjYNw2H93faGqfwWs7rLLG4DfVMOngTkROTWa1o0fK/7j5QxwoeP5xea2w8IJVb3S\nfHwVONFjv6SIfE5EPi0ih6GD6Od7ae+jqj6wASyOpHWD0+/v7oea5pAPici50TRtpBz2/99AeONu\nwGFGRD4JnOzy0n2q+kejbs8w2O0aO5+oqopIL9exm1X1kog8B/hzEfmiqj4VdVstkfIR4HdUtSYi\n/xQzy/nuMbfJEiFW/AdAVV8z4CEuAZ0jqrPNbRPDbtcoItdE5JSqXmlOl6/3OMal5v3TIvIw8O3A\nJIt/P99La5+LIuIBs8DKaJo3MHten6p2Xsv7gV8eQbtGzcT//4aJNfuMl88Ct4nIrSISB94IHApv\nmCYPAW9qPn4TsGO2IyLzIpJoPl4C7gS+MrIWHox+vpfOa/9h4M/18ATN7Hl922zfrwe+OsL2jYqH\ngJ9sev28AtjoMGMefVTV3oZwA/4hxoZYA64Bn2huPw18rGO/vw98HTMSvm/c7d7nNS5ivHz+Fvgk\nsNDcfgfw/ubj7wK+CDzevH/LuNvd57Xt+F6AXwRe33ycBB4EvgH8DfCccbc54uv7t8CXm9/bXwAv\nGHebD3CNvwNcARrN/+JbgJ8Bfqb5umC8np5q/jbvGHebR3mzEb4Wi8UyhVizj8VisUwhVvwtFotl\nCrHib7FYLFOIFX+LxWKZQqz4WywWyxRixd9isVimECv+FovFMoVY8bdYLJYp5P8HoV6Bs0NuFeMA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10817ae80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('Accuracy: {}'.format(np.mean(y==mod.pred(X))))\n",
    "plot_decision_boundary(mod.pred,X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train/Validation Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_valid_split(X,y,p=0.6):\n",
    "    ind = np.arange(X.shape[0])\n",
    "    random.shuffle(ind)\n",
    "    split = round(X.shape[0] * p)\n",
    "    train_ind = ind[:split]\n",
    "    valid_ind = ind[split:]\n",
    "    \n",
    "    X_train = X[train_ind]\n",
    "    y_train = y[train_ind]\n",
    "    X_valid =  X[valid_ind]\n",
    "    y_valid = y[valid_ind]\n",
    "    \n",
    "    return X_train,y_train,X_valid,y_valid\n",
    "\n",
    "#test\n",
    "X_train,y_train,X_valid,y_valid = train_valid_split(X,y,p=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss at iteration 0: 2.2253256373884\n",
      "Loss at iteration 100: 0.41315561061177897\n",
      "Loss at iteration 200: 0.30590611834556747\n",
      "Loss at iteration 300: 0.2737440045810935\n",
      "Loss at iteration 400: 0.25833855643789794\n",
      "Loss at iteration 500: 0.2504375899015064\n",
      "Loss at iteration 600: 0.24543711246687375\n",
      "Loss at iteration 700: 0.24249248361649792\n",
      "Loss at iteration 800: 0.24057510086684486\n",
      "Loss at iteration 900: 0.243548911008853\n"
     ]
    }
   ],
   "source": [
    "mod = DeepModel(X_train,y_train,D,K,h_list=[100,100],p_list=[1,1],\n",
    "                activation_list=[ReLu(),Norm()],output_activation=Softmax(),scaling='he')\n",
    "mod.train(iter=1000,reg=1e-3,step_size=1e-1,use_l2=True)\n",
    "y_pred = mod.pred(X_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUsAAAEmCAYAAADr3bIaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xe8FOXZxvHfdWgWwAaigB2QqIk0\nRVTUWCJYMTExFhBBscfEEnuivho1vonRaKIYRTTGFrtYE7tBFBCJXdQYQJQiKqIg4P3+MXNw4YWz\ncw5nmd1zrm8+82F3Znbm3o1cPDPzzDOKCMzMrGZVeRdgZlYJHJZmZhk4LM3MMnBYmpll4LA0M8vA\nYWlmloHDshGRtKqkByR9JunOFdjOoZIeq8/a8iKpr6S38q7Dyp/cz7L8SDoEOBnoCswBJgAXRcRz\nK7jdgcCJwPYRsXCFCy1zkgLoHBGT8q7FKp9blmVG0snAH4DfAO2ADYE/AfvXw+Y3At5uDEGZhaSm\neddgFSQiPJXJBKwBfAH8uIZ1WpCE6Yfp9AegRbpsF2AKcAowHZgGHJEuOx/4GliQ7mMocB7w14Jt\nbwwE0DR9Pxh4j6R1+z5waMH85wo+tz3wEvBZ+uf2BcueAv4HeD7dzmNAm+V8t+r6f1lQ/wBgL+Bt\n4BPgrIL1twVGA5+m614FNE+XPZN+l7np9z2oYPunAx8BN1fPSz+zWbqPHun79sAMYJe8/9vwlP/k\nlmV56QOsAtxTwzpnA9sB3YCtSQLjnILl65GEbgeSQLxa0loR8WuS1urtEdEyIq6vqRBJqwNXAv0j\nohVJIE5YxnprA6PSddcBfg+MkrROwWqHAEcA6wLNgVNr2PV6JL9BB+BXwHXAYUBPoC9wrqRN0nUX\nAb8A2pD8drsBxwFExE7pOlun3/f2gu2vTdLKHla444h4lyRI/yppNWAEMDIinqqhXmskHJblZR1g\nZtR8mHwocEFETI+IGSQtxoEFyxekyxdExEMkrarN61jPN8BWklaNiGkR8doy1tkbeCcibo6IhRFx\nK/AmsG/BOiMi4u2I+Aq4gyTol2cByfnZBcBtJEF4RUTMSff/Osk/EkTEuIh4Id3vf4BrgZ0zfKdf\nR8T8tJ4lRMR1wCRgDLA+yT9OZg7LMjMLaFPkXFp74IOC9x+k8xZvY6mw/RJoWdtCImIuyaHrMcA0\nSaMkdc1QT3VNHQref1SLemZFxKL0dXWYfVyw/Kvqz0vqIulBSR9J+pyk5dymhm0DzIiIeUXWuQ7Y\nCvhjRMwvsq41Eg7L8jIamE9ynm55PiQ5hKy2YTqvLuYCqxW8X69wYUQ8GhF7kLSw3iQJkWL1VNc0\ntY411cafSerqHBGtgbMAFflMjd0/JLUkOQ98PXBeeprBzGFZTiLiM5LzdFdLGiBpNUnNJPWX9Nt0\ntVuBcyS1ldQmXf+vddzlBGAnSRtKWgM4s3qBpHaS9k/PXc4nOZz/ZhnbeAjoIukQSU0lHQRsATxY\nx5pqoxXwOfBF2uo9dqnlHwOb1nKbVwBjI+JIknOx16xwldYgOCzLTET8jqSP5TkkV2InAycA96ar\nXAiMBSYC/wbGp/Pqsq/HgdvTbY1jyYCrSuv4kOQK8c78/zAiImYB+5BcgZ9FciV7n4iYWZeaaulU\nkotHc0havbcvtfw8YKSkTyX9pNjGJO0P9OPb73ky0EPSofVWsVUsd0o3M8vALUszswwclmZmGTgs\nzcwycFiamWVQVgMJqNlqoVXWzLuMitK9S/viK5mtoA8++A8zZ84s1oe1Vpq03ihi4f+7iWq54qsZ\nj0ZEv/qsoTbKKyxXWZMW3Y/Ku4yK8vwT5+ddgjUCO/TuVe/bjIVf0WLzoj26Fps34epid2eVVFmF\npZk1JgJVzplAh6WZ5UOA6vXIvqQclmaWH7cszcyKEVQ1ybuIzByWZpYfH4abmRUhfBhuZlac3LI0\nM8vELUszswzcsjQzK8ad0s3MinOndDOzjNyyNDMrRtDEndLNzGrmfpZmZhn5nKWZWTG+Gm5mlo1b\nlmZmGbhlaWZWhHxvuJlZNm5Zmpll4JalmVkxvhpuZlac8GMlzMyKc8vSzCwbn7M0M8vALUszswzc\nsjQzK0I+Z2lmlo1blmZmxamCwrJy2sAlVlUlRv/lGO665JDF8847cjcm3nIiL998Asf9qDcArVdv\nwd8vPoQxNxzLuJHHM7B/t7xKLktHHzmEDduvS89uW+VdSkV57NFH+N6Wm7Nl105c9ttL8i5npUge\nwaPMU94clqkTDtyOtz6Ysfj9wP7d6Lhua7Y+7Cq6D7yKO//5KgBHH7Atb34wg95D/syePxvBJcfv\nSbOmldOxttQGHj6Y+x58JO8yKsqiRYv4+c+O574HHublia9z52238sbrr+ddVulJqCr7lDeHJdCh\nbWv69enCiFHjF88bNmAbfjPyaSICgBmfzgUgAlqu2hyA1VdrzuzPv2Lhom9WftFlase+O7H22mvn\nXUZFeenFF9lss05ssummNG/enB8f9FMefOC+vMtaKdyyrDCXndiPs//8GN98E4vnbdJ+bQ7cdSue\nGz6Me397GJt1TALgmrvH0HWjtrx3z6mMHXEcp1758OJANauLDz+cSseOGyx+36FDR6ZOnZpjRSuP\nwzIlqZ+ktyRNknRGKfdVV/37dGH67Lm8/Pa0Jea3aNaE+V8vZMdhwxnx4DiuPX0AAHts24mJkz5i\n0wP+l95Dr+HyX+xNq9Va5FG6WcWrpLAs2dVwSU2Aq4E9gCnAS5Luj4iyOhnT57sbss8Om9Nvu860\naN6U1qu34IZzfsjUGZ9z7zNJqfc98wbXnpGE5cC9uvO7W54F4L2pn/CfabPZfKM2jH2jcbQErP61\nb9+BKVMmL34/deoUOnTokGNFK4nSqUKUsmW5LTApIt6LiK+B24D9S7i/OvnV8H/Q6cDf0/WgPzDo\n/L/z1Pj3GXLh3Tzw3Jvs3H0TAPp225hJk2cBMPnjz9il56YArLvW6nTZoA3vfzg7t/qt8vXaZhsm\nTXqH/7z/Pl9//TV33n4be++zX95llZzI3qps0C1LoAMwueD9FKD30itJGgYMA6DFGiUsp3b+95bn\nGHHujzjxJ32Y++XXHPvb5IT7JSOfZvhZA3jpxuMQcPY1jzPrsy/zLbaMDDrsYJ59+ilmzpzJZht3\n5Nxfnc/gIUPzLqusNW3alMuvuIp9996TRYsWcfjgIWyx5ZZ5l7VSlEMIZqVSXZyQdCDQLyKOTN8P\nBHpHxAnL+0xVq/bRovtRJamnoZr9xPl5l2CNwA69ezFu3Nh6Tbam62warfe6MPP6s/966LiI6FWf\nNdRGKVuWU4ENCt53TOeZmQGV1bIs5TnLl4DOkjaR1Bz4KXB/CfdnZpVEtZyybFJqIullSQ+m7zeR\nNCbtkXN7mkVIapG+n5Qu37jYtksWlhGxEDgBeBR4A7gjIl4r1f7MrLIIUVVVlXnK6CSSvKl2KXB5\nRHQCZgPVJ9CHArPT+Zen69WopP0sI+KhiOgSEZtFxEWl3JeZVZ76vBouqSOwN/CX9L2AXYG/p6uM\nBAakr/dP35Mu301FduI7eMwsP7U7DG8jaWzBNGyprf0B+CVQff/xOsCn6VEuJD1yqjuwLu6tky7/\nLF1/uTxEm5nlQ7W+wDNzeVfDJe0DTI+IcZJ2qY/yluawNLPc1OPV8B2A/STtBawCtAauANaU1DRt\nPRb2yKnurTNFUlNgDWBWTTvwYbiZ5aa+zllGxJkR0TEiNibpefNERBwKPAkcmK52OFA9nNP96XvS\n5U9EkU7nDkszy8VKut3xdOBkSZNIzklen86/HlgnnX8yUHSgHx+Gm1l+StAnPSKeAp5KX79HMk7F\n0uvMA35cm+06LM0sH7W/wJMrh6WZ5cZhaWaWQTk8Wycrh6WZ5cYtSzOzIsplUN+sHJZmlhuHpZlZ\nBg5LM7MsKicrHZZmlh+3LM3MinGndDOz4gRUUFY6LM0sL6LKndLNzIrzYbiZWTHyYbiZWVECH4ab\nmWXhlqWZWQY+Z2lmVozPWZqZFZf0s6yctHRYmllOPESbmVkmFZSVDkszy4ncdcjMrCifszQzy6iC\nstJhaWb5ccvSzCyDCsrK8grL7l3a8/wT5+ddRkVZa8+L8y6h4sx6+Iy8S6g4UYqNevBfM7PiPPiv\nmVkm7pRuZpZJBWWlw9LMcuJO6WZmxblTuplZRg5LM7MMKigrHZZmlh+3LM3MivFI6WZmxcn9LM3M\nsqmgrKQq7wLMrPGqkjJPNZG0iqQXJb0i6TVJ56fzN5E0RtIkSbdLap7Ob5G+n5Qu37horfXwfc3M\n6kTKPhUxH9g1IrYGugH9JG0HXApcHhGdgNnA0HT9ocDsdP7l6Xo1cliaWS4kaFKlzFNNIvFF+rZZ\nOgWwK/D3dP5IYED6ev/0Peny3VTkBKrD0sxyIynzBLSRNLZgGrbUtppImgBMBx4H3gU+jYiF6SpT\ngA7p6w7AZIB0+WfAOjXVutwLPJJa1/TBiPi8puVmZsXU8gLPzIjotbyFEbEI6CZpTeAeoOuKVbek\nmq6Gv0bSjC38OtXvA9iwPgsxs8ZFJN2H6ltEfCrpSaAPsKakpmnrsSMwNV1tKrABMEVSU2ANYFZN\n211uWEbEBvVSuZnZctTXoEOS2gIL0qBcFdiD5KLNk8CBwG3A4cB96UfuT9+PTpc/ERE1DgifqZ+l\npJ8Cm0bEbyR1BNpFxLg6fCczs4TqtVP6+sBISU1IrsXcEREPSnoduE3ShcDLwPXp+tcDN0uaBHwC\n/LTYDoqGpaSrSK4s7QT8BvgSuAbYpvbfx8zsW/WVlRExEei+jPnvAdsuY/484Me12UeWluX2EdFD\n0svpTj6p7thpZlZXgqKdzctJlrBcIKmK9AFvktYBvilpVWbWKFRQVmbqZ3k1cBfQNr2F6Dky9HY3\nMyumlv0sc1W0ZRkRN0kaB+yezvpxRLxa2rLMrKGrvoOnUmQddagJsIDkUNx3/ZhZvaicqMwQfJLO\nBm4F2pN06vybpDNLXZiZNXwN6jAcGAR0j4gvASRdRNJf6eJSFmZmDVtyNTzvKrLLEpbTllqvaTrP\nzKzuyqTFmFVNA2lcTnKO8hPgNUmPpu9/ALy0csozs4asgrKyxpZl9RXv14BRBfNfKF05ZtaYNIiW\nZURcv7xlZmYrqtLOWWa5Gr6ZpNskTZT0dvW0MorL29FHDmHD9uvSs9tWeZdSlqqqxOhrjuCui769\nxfa8ITsxceTRvHzDURx3wLdDD/7u+D149aZjePG6oXTr3C6PcsveokWL6LNtD340YN+8S1lpKulq\neJY+kzcCI0j+IegP3AHcXsKaysbAwwdz34OP5F1G2Trhh71467/fDgE4cM/v0rFta7YefC3dh1zH\nnU++DsCe227GZh3XYqtB13DC7x/mypP65VVyWbv6j1ewedfv5F3GSiNBEynzlLcsYblaRDwKEBHv\nRsQ5JKHZ4O3YdyfWXnvtvMsoSx3atKJf706MeOiVxfOG7deD39z8PNWjAs749EsA9tmhM397LDkF\n/uIbH7JGyxast/bqK73mcjZ1yhQeefghBh8xtPjKDUg9PrCs5LJ0HZqfDqTxrqRjSEYYblXasqzc\nXXb87pw9/ElarvbtAFSbtF+LA3f5Dvvt2IWZn33JKVc9zrtTZ9O+TSumzPj2KSRTZ8yhfZtWfPTJ\n3DxKL0u/PPUXXHTxpcyZMyfvUlaqcji8zipLy/IXwOrAz4AdgKOAIcU+JOkGSdMl+T7yBqb/dp2Y\nPvtLXn7noyXmt2jWhPkLFrLjcTcyYtQErj1t75wqrCwPj3qQtm3b0r1Hz7xLWekaVMsyIsakL+cA\nA2ux7RuBq4Cbal+WlbM+W3Zgn+070a/3prRo3pTWq7XghjP3ZeqMOdz77FsA3Pfc24vD8sOZc+jY\n9tvn33Vo24oPZzauFlRNRo9+nlGjHuDRRx9m3rx5zPn8c4YMHsgNN96cd2klJVRR41kut2Up6R5J\ndy9vKrbhiHiGpEO7NTC/uv5pOv30aroe+mcGXXgfT034gCEXP8ADz7/Nzt02AqDv1hsyacpsAEb9\n6x0O+UHSo2Db77Tn87nzfQhe4IILL+ad9ybzxtvvM/LmW9l5l10bfFACUItWZTlkak0ty6tWRgHp\ns3+HAWywYXk9MHLQYQfz7NNPMXPmTDbbuCPn/up8Bg9pXCfga+N/bx3NiLP248QfbcPceQs49ncP\nAfDImHfZs/dmvHbzMXw5bwFHXzaqyJassaikc5Yq8kCzFdu4tDHwYERk6qjYs2eveH7M2JLV0xCt\ntafHM6mtWQ+fkXcJFWfHPtswftzYek22dTttFQdddmfm9a/64RbjanpueKllHc/SzKxeicpqWTos\nzSw3Dep2x2qSWtRmw5JuJXmA+eaSpkjyyT4zW6z6sRJZp7xleW74tiQPJF8D2FDS1sCREXFiTZ+L\niIPrp0Qza6jKIAMzy9KyvBLYB5gFEBGvAN8vZVFm1jg0lK5D1aoi4oOlTsQuKlE9ZtZIJEO0lUEK\nZpQlLCenh+IhqQlwItAohmgzs9KqpEfFZgnLY0kOxTcEPgb+kc4zM1shFdSwzHRv+HTgpyuhFjNr\nRKTKujc8y9Xw60geVLaEiBhWkorMrNGooKzMdBj+j4LXqwAHAJNLU46ZNSaV1HUoy2H4Eo+QkHQz\n8FzJKjKzRkFQFp3Ns6rL7Y6bAH7ilJmtGDWwlqWk2Xx7zrKKZIxKD9tiZitMVE5a1hiWSnqib03y\n3B2Ab6KUY7qZWaPRoJ4bngbjQxGxKJ0clGZWb6qUfcpblg70EyR1L3klZtboSMo85a2mZ/BUH6J3\nB16S9Jak8ZJeljR+5ZRnZg1V9WF4fbQsJW0g6UlJr0t6TdJJ6fy1JT0u6Z30z7XS+ZJ0paRJkiZK\n6lGs3prOWb4I9AD2y/rlzcwyq9/RhBYCp0TEeEmtgHGSHgcGA/+MiEsknUFycfp0oD/QOZ16A39O\n/1yumsJSABHx7op+CzOzZamv2x0jYhowLX09R9IbQAdgf2CXdLWRwFMkYbk/cFN6HeYFSWtKWj/d\nzjLVFJZtJZ1cQ3G/r8V3MTNbQqmuhqcPSuwOjAHaFQTgR3zbR7wDS96JOCWdV6ewbAK0hArqCGVm\nFUQ0qV3Lso2kwse/Do+I4UtsUWoJ3AX8PCI+L7wwFBEhqc49emoKy2kRcUFdN2xmVpPk6Y61+sjM\nmh6FK6kZSVDeEhF3p7M/rj68lrQ+MD2dPxXYoODjHfm2P/ky1dR1yC1KMyudWlwJz3A1XCTPCntj\nqVOE9wOHp68PB+4rmD8ovSq+HfBZTecroeaW5W41l2dmtmLqcTzLHYCBwL8lTUjnnQVcAtyRPl32\nA+An6bKHgL2AScCXwBHFdrDcsIyIT+pet5lZzepwGL5cEfEcyz8a/n8Nv/Qq+PG12UddRh0yM6sX\nDWqkdDOzUqmgrHRYmlk+RMN7uqOZWf0TZTFARlYOSzPLTeVEpcPSzHIiqO0dPLlyWJpZbiooKx2W\nZpaX8hjUNyuHpZnlwlfDzcwycsvSzCyDyonKMgvLhd8Es774Ou8yKsrU+0/Lu4SKs07vE/MuoeLM\nf+u/9b9R97M0MyvO5yzNzDJyy9LMLINSPIOnVByWZpaL5DC8ctLSYWlmuamgo3CHpZnlRcgtSzOz\n4tyyNDMrwucszcyykFuWZmaZOCzNzDLwBR4zsyKEO6WbmWXi54abmWXgw3AzsyJ8GG5mlonv4DEz\nK879LM3MsqmgrHRYmlk+knOWlROXDkszy03lRKXD0szyVEFp6bA0s9z4MNzMLIPKiUqHpZnlqYLS\n0mFpZrkQvt3RzKy4CuuUXpV3AWbWeKkWU9FtSTdImi7p1YJ5a0t6XNI76Z9rpfMl6UpJkyRNlNSj\n2PYdlmaWn/pMS7gR6LfUvDOAf0ZEZ+Cf6XuA/kDndBoG/LnYxh2WZpYT1ep/xUTEM8AnS83eHxiZ\nvh4JDCiYf1MkXgDWlLR+Tdv3OUszy00tz1m2kTS24P3wiBhe5DPtImJa+vojoF36ugMwuWC9Kem8\naSyHW5bLsGjRIvbcaVsOPyj5RygiuPR/fkXfXluyS+/vcf21V+VcYXmZOmUy+/ffnT49v8f2vbbm\n2quvBODSiy5gy84bsXOfnuzcpyePP/pwzpWWh6oqMfrW07nrimMA2HmbLvzrb6cz9s6zuO6CgTRp\nkvy17NuzMx89cxkv3HYGL9x2BmcOW/oIs7LV5gg8zdSZEdGrYCoWlEuIiACirvW6ZbkM11/zRzp1\n6coXcz4H4I6/3cSHU6fw9Iv/pqqqipkzpudcYXlp0rQpF1z8W7bu1oM5c+awW9/e7Lzr7gAce8JJ\nnHDSyTlXWF5OOOT7vPX+x7RafRUk8ZcLBtL/6D8y6b/TOffYvTls396MvHc0AM+//C4/OumanCsu\nHZX+cvjHktaPiGnpYXb1X96pwAYF63VM5y2XW5ZL+XDqFP752MMcMuiIxfNuumE4P//lWVRVJT9X\nm7br5lVeWVpvvfXZultyMbFVq1Z03rwr06Z9mHNV5anDumvSb8ctGXHPvwBYZ83V+XrBQib9N/k7\n/MQLbzJgt255lrhSSdmnOrofODx9fThwX8H8QelV8e2AzwoO15fJYbmU8846lbPPvxhVffvTfPD+\nezxw99/Z6/t9OOzAfXnv3XdyrLC8/feD//DvVybQs9e2APzl2j/Rt3d3Tjz2SD6dPTvn6vJ32Wk/\n4uwr7uWbb5KjwZmzv6Bp0yb02GJDAA7YvRsd2621eP3e39uEMbefwb1XHct3Nl0vl5pLqZ67Dt0K\njAY2lzRF0lDgEmAPSe8Au6fvAR4C3gMmAdcBxxXbfsnCUtIGkp6U9Lqk1ySdVKp91Zd/PDKKNm3a\n8r1uS3a5+vrr+bRYpQUPPTmaQw4fyqknHJ1TheXtiy++YPChP+GiS39H69atOeLIoxn377d4evQ4\n2rVbn3PPOi3vEnPVv+9WTP9kDi+/MXmJ+YPOGMFvT/khz958KnPmzmfRN98AMOHNyWy+17n0PugS\n/nzb09xx+bA8yi6dOpy0rElEHBwR60dEs4joGBHXR8SsiNgtIjpHxO4R8Um6bkTE8RGxWUR8NyLG\nFtt+Kc9ZLgROiYjxkloB4yQ9HhGvl3CfK+SlMaN57JFRPPH4o8yfP485cz7nxGGDWb99B/rvm1zs\n6b/P/pxy/FE5V1p+FixYwOBDf8KBBx3MvvsfAMC67dotXj7oiKEcfOCA5X28UejTbVP22fm79Ntx\nS1o0b0br1VfhhgsHMeScm9h96B8A2G27rnTeKDnNM2fuvMWfffS517nizCass+bqzPp0bi71l0Il\n3e5YspZlREyLiPHp6znAGySX5svWmb++kLGvvccLE9/m6utvZoe+u/DH4Tey51778a9nnwZg9PPP\nsGmnzjlXWl4igp8ddxRdNu/KcSf+YvH8jz769hTQqAfu5TtbbJlHeWXjV3+8n079zqXr3r9m0Bkj\neOqltxlyzk20XaslAM2bNeWUwXtw3d+fA6DdOq0Wf7bXlhtRJTWwoFwp5yzrzUq5Gi5pY6A7MGZl\n7K++Hf+L0zjxqMO57k9XsnrLllx2RcO9OlkXY0Y/zx233sIWW27Fzn16AnDOeRdy15238erEV5DE\nhhttzO+u/FPOlZanXxy+O/37bkVVlbjuzmd5+qW3AThg9+4c9eO+LFy0iHnzFjDozBE5V1r/yiAD\nM1PS9aiEO5BaAk8DF0XE3ctYPozkdiM6dNyw55h/++JJbazazNfoaqvDjj/Pu4SKM/+tO/jmy+n1\nmm1bbd0j7nzk2czrb9G+5biI6FWfNdRGSf+mSWoG3AXcsqygBIiI4dWdTNdp06aU5ZhZmanP2x1L\nrWSH4Up6m14PvBERvy/VfsysclXln4GZlbJluQMwENhV0oR02quE+zOzSlO/ow6VVMlalhHxHGXx\nFc2sHHmkdDOzLMqkS1BWDkszy00FZaXD0sxyVEFp6bA0s5yUR5egrByWZpYbn7M0MyuiTHoEZeaw\nNLP8VFBaOizNLDdVFXQc7rA0s9xUTlQ6LM0sL+6UbmaWVeWkpcPSzHJRPVJ6pXBYmlluKigrHZZm\nlh+3LM3MMvDtjmZmWVROVjoszSw/FZSVDkszy4fkO3jMzLKpnKx0WJpZfiooKx2WZpafCjoKd1ia\nWV48UrqZWVGVdrtjVd4FmJlVArcszSw3ldSydFiaWW58ztLMrIikU3reVWTnsDSz/DgszcyK82G4\nmVkGlXSBx12HzCw3qsVUdFtSP0lvSZok6Yz6rtVhaWb5qae0lNQEuBroD2wBHCxpi/os1WFpZrlR\nLf5XxLbApIh4LyK+Bm4D9q/PWsvqnOXECeNndlyrxQd517EMbYCZeRdRYfyb1U25/m4b1fcGXx4/\n7tHVmqtNLT6yiqSxBe+HR8Tw9HUHYHLBsilA7xWtsVBZhWVEtM27hmWRNDYieuVdRyXxb1Y3jel3\ni4h+eddQGz4MN7OGYCqwQcH7jum8euOwNLOG4CWgs6RNJDUHfgrcX587KKvD8DI2vPgqthT/ZnXj\n360OImKhpBOAR4EmwA0R8Vp97kMRUZ/bMzNrkHwYbmaWgcPSzCwDh6WZWQYOy+WQtLmkPpKapbdS\nWQb+rWpHUidJvSS1yLsWq5kv8CyDpB8CvyHppzUVGAvcGBGf51pYGZPUJSLeTl83iYhFeddU7iTt\nQ/Lf2SzgI+DX1b+hlR+3LJciqRlwEDA0InYD7iPp7Hq6pNa5Flem0r/0EyT9DSAiFrmFWTNJ2wOX\nAYdHxPeB2UC9j5Rj9cdhuWytgc7p63uAB4FmwCFSJY3AV3qSVgdOAH4OfC3pr+DAzOjSiHg5ff1r\nYG0fjpcvh+VSImIB8Hvgh5L6RsQ3wHPABGDHXIsrQxExFxgC/A04lWSwg8WBmWdtZW4McDcsPs/b\ngmSwitbpvHXyK82WxWG5bM8CjwEDJe0UEYsi4m9Ae2DrfEsrPxHxYUR8EREzgaOBVasDU1IPSV3z\nrbD8pP9NVZ8DF/Ap8ElEzJB0KHChpFXzq9CW5tsdlyEi5km6BQjgzPQv+3ygHTAt1+LKXETMknQ0\ncJmkN0luPft+zmWVtYhYCHwa/qvGAAADrElEQVQhabKki4EfAIMj4qucS7MCDsvliIjZkq4DXidp\nLc0DDouIj/OtrPxFxExJE0lGrd4jIqbkXVM5S8+DNwP6pn/uFhHv5FuVLc1dhzJIzylFev7SipC0\nFnAHcEpETMy7nkohaTDwUn0PAGH1w2FpJSFplYiYl3cdlUSSwn8hy5bD0swsA18NNzPLwGFpZpaB\nw9LMLAOHpZlZBg7LBkLSIkkTJL0q6U5Jq63AtnaR9GD6ej9Jyx3gQdKako6rwz7Ok3Rq1vlLrXOj\npANrsa+NJb1a2xrNCjksG46vIqJbRGwFfA0cU7hQiVr//x0R90fEJTWssiZQ67A0qzQOy4bpWaBT\n2qJ6S9JNwKvABpJ+IGm0pPFpC7QlgKR+kt6UNB74YfWGJA2WdFX6up2keyS9kk7bA5cAm6Wt2svS\n9U6T9JKkiZLOL9jW2ZLelvQcsHmxLyHpqHQ7r0i6a6nW8u6Sxqbb2yddv4mkywr2ffSK/pBm1RyW\nDYykpiS3Gf47ndUZ+FNEbAnMBc4Bdo+IHiSDGp8saRXgOmBfoCew3nI2fyXwdERsDfQAXiMZg/Hd\ntFV7mqQfpPvcFugG9JS0k6SeJM9y7gbsBWyT4evcHRHbpPt7AxhasGzjdB97A9ek32Eo8FlEbJNu\n/yhJm2TYj1lRvje84VhV0oT09bPA9SSjJH0QES+k87cDtgCeT4flbA6MBroC71ffj5yOGDRsGfvY\nFRgEi4df+yy9tbHQD9KpepzGliTh2Qq4JyK+TPdxf4bvtJWkC0kO9VuSPBO62h3p7afvSHov/Q4/\nAL5XcD5zjXTfHn3cVpjDsuH4KiK6Fc5IA3Fu4Szg8Yg4eKn1lvjcChJwcURcu9Q+fl6Hbd0IDIiI\nV9L7pncpWLb0rWeR7vvEiCgMVSRtXId9my3Bh+GNywvADpI6QTLKuaQuwJvAxpI2S9c7eDmf/ydw\nbPrZJpLWAOaQtBqrPQoMKTgX2kHSusAzwABJq0pqRXLIX0wrYJqSR30cutSyH0uqSmveFHgr3fex\n6fpI6qJkJHezFeaWZSOSDiw7GLhV3z6+4JyIeFvSMGCUpC9JDuNbLWMTJwHDJQ0FFgHHRsRoSc+n\nXXMeTs9bfgcYnbZsvyAZ2m68pNuBV4DpwEsZSj6XZETxGemfhTX9F3iRZGTxY9IxSP9Cci5zfDrs\n2QxgQLZfx6xmHkjDzCwDH4abmWXgsDQzy8BhaWaWgcPSzCwDh6WZWQYOSzOzDByWZmYZ/B93ctPR\n3CKZmQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10366f240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm = confusion_matrix(y_pred, y_valid)\n",
    "plot_confusion_matrix(cm,[0,1,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fancy Accuracies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      0.91      0.95       515\n",
      "          1       0.99      0.95      0.97       486\n",
      "          2       0.87      0.99      0.93       499\n",
      "\n",
      "avg / total       0.95      0.95      0.95      1500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#print('Area Under Curve: {}'.format(round(roc_auc_score(y_valid, y_pred),3)))\n",
    "print(classification_report(y_valid, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Stanford University CS231n](http://cs231n.github.io/neural-networks-case-study/): Convolutional Neural Networks for Visual Recognition by Andrej Karpathy\n",
    "* Olah, Christopher. “Calculus on Computational Graphs: Backpropagation.” [Calculus on Computational Graphs: Backpropagation -- Colah's Blog](colah.github.io/posts/2015-08-Backprop/), 31 Aug. 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}