COSMOS_ML_04_CNN_LeNet.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sungjuGit/COSMOS_Ju/blob/main/COSMOS_ML_04_CNN_LeNet.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J3VsmcZU4O8I"
      },
      "source": [
        "# LeNet\n",
        "\n",
        "In this notebook, we will be implementing a modified version of [LeNet-5](http://yann.lecun.com/exdb/lenet/), a neural network model that uses convolutional neural network (CNN) layers and was designed for [classifying handwritten characters](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf).\n",
        "\n",
        "CNNs are ubiquitous in modern computer vision. Almost every neural network model for computer vision contains many CNN layers, also referred to as convolutional layers, thus they're an important concept to understand. We will explain how they work when we implement the model.\n",
        "\n",
        "The image below shows the architecture of LeNet-5. It consists of two convolutional layers, each followed by a subsampling layer, and then three fully connected linear layers.\n",
        "\n",
        "![](https://github.com/bentrevett/pytorch-image-classification/blob/master/assets/lenet5.png?raw=1)\n",
        "\n",
        "\n",
        "Most of this notebook will be similar to the previous one.\n",
        "\n",
        "This notebook was adapted from (https://github.com/bentrevett/pytorch-image-classification)."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "## Set Things Up"
      ],
      "metadata": {
        "id": "PACH3MN6Zx31"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CHadwpvmREvE",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "# @title Import Modules\n",
        "\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "import torch.utils.data as data\n",
        "\n",
        "import torchvision.transforms as transforms\n",
        "import torchvision.datasets as datasets\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "import time\n",
        "import random\n",
        "\n",
        "SEED = 1234\n",
        "\n",
        "torch.manual_seed(SEED)\n",
        "torch.cuda.manual_seed(SEED)\n",
        "torch.backends.cudnn.deterministic = True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yDMpvXwB4O8Q"
      },
      "source": [
        "The first thing we'll do is load our dataset. This will automatically download the training set for the MNIST dataset and save it in a folder called `.data`. It will create the folder if it does not exist."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UOIQVy-cREvQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "82584f4b-d617-406e-9779-de6fa7b33c7c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
            "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to .data/MNIST/raw/train-images-idx3-ubyte.gz\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 9912422/9912422 [00:00<00:00, 86077364.25it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extracting .data/MNIST/raw/train-images-idx3-ubyte.gz to .data/MNIST/raw\n",
            "\n",
            "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
            "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to .data/MNIST/raw/train-labels-idx1-ubyte.gz\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 28881/28881 [00:00<00:00, 115477305.84it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extracting .data/MNIST/raw/train-labels-idx1-ubyte.gz to .data/MNIST/raw\n",
            "\n",
            "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
            "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to .data/MNIST/raw/t10k-images-idx3-ubyte.gz\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1648877/1648877 [00:00<00:00, 40508955.32it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extracting .data/MNIST/raw/t10k-images-idx3-ubyte.gz to .data/MNIST/raw\n",
            "\n",
            "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
            "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to .data/MNIST/raw/t10k-labels-idx1-ubyte.gz\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 4542/4542 [00:00<00:00, 1167311.81it/s]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extracting .data/MNIST/raw/t10k-labels-idx1-ubyte.gz to .data/MNIST/raw\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "ROOT = '.data'\n",
        "\n",
        "train_data = datasets.MNIST(root=ROOT,\n",
        "                            train=True,\n",
        "                            download=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gMThB8Vs4O8V"
      },
      "source": [
        "Next, we want to *normalize* our data. This means we want it to have a mean of zero and a standard deviation of one.\n",
        "\n",
        "Why do we want to do this? Normalizing our data allows our models to train faster and to also help them avoid local minima, i.e. train more reliably.\n",
        "\n",
        "The image data has values between 0-255, which we want to scale between 0-1, so we divide by 255."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FhM076c7Zzrv"
      },
      "outputs": [],
      "source": [
        "mean = train_data.data.float().mean() / 255\n",
        "std = train_data.data.float().std() / 255"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U50VOMAW4O8d"
      },
      "source": [
        "Now we've calculated our mean and standard deviation how do we actually use them? We use Torchvision's `transforms`.\n",
        "\n",
        "A `transform` states how our data should be augmented and processed. Data augmentation involves manipulating the available training data in a way that artificially creates more training examples. We use `transforms.Compose` to built a list of transformations that will be applied to the image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WuYq4cRdREvV",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "# @title Transform data\n",
        "train_transforms = transforms.Compose([\n",
        "                            transforms.RandomRotation(5, fill=(0,)),\n",
        "                            transforms.RandomCrop(28, padding=2),\n",
        "                            transforms.ToTensor(),\n",
        "                            transforms.Normalize(mean=[mean], std=[std])\n",
        "                                      ])\n",
        "\n",
        "test_transforms = transforms.Compose([\n",
        "                           transforms.ToTensor(),\n",
        "                           transforms.Normalize(mean=[mean], std=[std])\n",
        "                                     ])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "73S5zyox4O8h"
      },
      "source": [
        "Now we have defined our transforms we can then load the train and test data with the relevant transforms defined."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GPvVD4VLREva"
      },
      "outputs": [],
      "source": [
        "train_data = datasets.MNIST(root=ROOT,\n",
        "                            train=True,\n",
        "                            download=True,\n",
        "                            transform=train_transforms)\n",
        "\n",
        "test_data = datasets.MNIST(root=ROOT,\n",
        "                           train=False,\n",
        "                           download=True,\n",
        "                           transform=test_transforms)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "## Check Out Data"
      ],
      "metadata": {
        "id": "rFweArLKaRgk"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ayyHLZjg4O8o"
      },
      "source": [
        "We can simply check the `len` of the datasets to see how many examples are within each."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fPo3pfgciRo3",
        "outputId": "9cf3a5d3-acae-4c78-f44f-d8626c97dfa9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of training examples: 60000\n",
            "Number of testing examples: 10000\n"
          ]
        }
      ],
      "source": [
        "print(f'Number of training examples: {len(train_data)}')\n",
        "print(f'Number of testing examples: {len(test_data)}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KZuPIyvN4O9b"
      },
      "source": [
        "The LeNet architectures, and almost every modern neural network computer vision architecture, makes use of convolutional neural network (CNN) layers.\n",
        "\n",
        "CNNs are inspired by classic computer vision techniques, like [Sobel filters](https://en.wikipedia.org/wiki/Sobel_operator). Let's try manually choosing weights of a 3x3 filter to make Sobel filters and apply them to some MNIST digits to see what type of things our CNN layers can learn.\n",
        "\n",
        "The `plot_filter` function takes in a batch of images and a two-dimensional filter and plots the output of that filter applied to all of the images."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# @title Plot Filter function\n",
        "def plot_filter(images, filter):\n",
        "\n",
        "    images = images = torch.cat([i.unsqueeze(0) for i in images],\n",
        "                                dim=0).cpu()\n",
        "    filter = torch.FloatTensor(filter).unsqueeze(0).unsqueeze(0).cpu()\n",
        "\n",
        "    n_images = images.shape[0]\n",
        "\n",
        "    filtered_images = F.conv2d(images, filter)\n",
        "\n",
        "    fig = plt.figure(figsize=(20, 5))\n",
        "\n",
        "    for i in range(n_images):\n",
        "\n",
        "        ax = fig.add_subplot(2, n_images, i+1)\n",
        "        ax.imshow(images[i].squeeze(0), cmap='bone')\n",
        "        ax.set_title('Original')\n",
        "        ax.axis('off')\n",
        "\n",
        "        image = filtered_images[i].squeeze(0)\n",
        "\n",
        "        ax = fig.add_subplot(2, n_images, n_images+i+1)\n",
        "        ax.imshow(image, cmap='bone')\n",
        "        ax.set_title('Filtered')\n",
        "        ax.axis('off')"
      ],
      "metadata": {
        "cellView": "form",
        "id": "JaipAC0PRFrI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll then get a batch of images to test our handwritten Sobel filters on. We get the images from the test set, as they're easier to examine without the transforms applied to them."
      ],
      "metadata": {
        "id": "feOn8_8nRJjC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "N_IMAGES = 5\n",
        "\n",
        "images = [image for image, label in [test_data[i] for i in range(N_IMAGES)]]"
      ],
      "metadata": {
        "id": "HuiDTrTJRJPQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first filter is for detecting horizontal lines.\n",
        "\n",
        "We can see on the filtered images that the highest values (the whitest pixels) of the filtered image are where there is a horizontal line that is black on top and white below, e.g. the top of the 7 digit. The lowest values (the blackest pixels) of the filtered image are where there is a horizontal line that goes from white to black, e.g. the bottoms of all the digits."
      ],
      "metadata": {
        "id": "vcfsCmrbRSok"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "horizontal_filter = [[-1, -2, -1],\n",
        "                     [ 0,  0,  0],\n",
        "                     [ 1,  2,  1]]\n",
        "\n",
        "plot_filter(images, horizontal_filter)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 444
        },
        "id": "SpkKU5QfRT6o",
        "outputId": "e528c110-ab75-4646-d3f2-32c60f8ed63a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 2000x500 with 10 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABcUAAAGrCAYAAAAB5v8sAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABANklEQVR4nO3deXzfVZ0v/vPNvjVN0qRbutCWUqDsVcANRZFFGcVxGRX5gaKgXrfxqtcFBXQGHfWKd9yuKILiPi6MjgKKIosioKW2UCjdW7o3bZMu6ZLk+/vjznRAzwn9lqRJep7Px2P+8PXK+XxO0ubw/b7zmaZQLBaLAQAAAAAAMlA21BsAAAAAAIBDxVAcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABkw1B8kFx11VWhUCgc1Nobb7wxFAqFsGLFioHd1BOsWLEiFAqFcOONNw7aPYB8OQOBXDn/gJw5A4GcOQNHFkPxiIcffji84Q1vCO3t7aG6ujpMnDgxXHjhheHhhx8e6q0BDDpnIJAr5x+QM2cgkDNnYH4KxWKxONSbGE5+8pOfhNe97nWhpaUlXHrppWHatGlhxYoV4frrrw8dHR3h+9//fnjFK17xlNfp6ekJPT09oaampuQ99Pb2hn379oXq6uqD/gnTU1mxYkWYNm1auOGGG8Ill1wyKPcARh5nIJAr5x+QM2cgkDNnYJ4qhnoDw8nSpUvDRRddFKZPnx7uuuuu0NbWtr9797vfHZ73vOeFiy66KMyfPz9Mnz49eo2dO3eG+vr6UFFRESoqDu7LW15eHsrLyw9qLcDBcgYCuXL+ATlzBgI5cwbmyz+f8gSf+cxnwq5du8J11133pG+CEEJobW0NX/3qV8POnTvDpz/96RDCf/9bQQsXLgyvf/3rQ3Nzc3juc5/7pO6Juru7w7ve9a7Q2toaRo0aFV72speFNWvWhEKhEK666qr9Hxf7d4SOOOKIcP7554d77rknnHrqqaGmpiZMnz49fOtb33rSPbZs2RLe9773heOPPz40NDSExsbGcN5554W//OUvA/iVAg5HzkAgV84/IGfOQCBnzsB8eVL8CX7+85+HI444Ijzvec+L9meccUY44ogjwi9+8Ysn5a9+9avDzJkzwzXXXBP6+9doLrnkkvDDH/4wXHTRReH0008Pd955Z3jpS196wPtbsmRJeNWrXhUuvfTScPHFF4dvfOMb4ZJLLglz5swJs2fPDiGEsGzZsnDzzTeHV7/61WHatGlhw4YN4atf/Wp4/vOfHxYuXBgmTpx4wPcD8uIMBHLl/ANy5gwEcuYMzJeh+H/q7OwMa9euDS9/+cv7/bgTTjgh/OxnPwvbt2/fn5144onhu9/9br/r5s6dG374wx+G97znPeHaa68NIYTw9re/PbzxjW884J/cLFq0KNx11137v1Ff85rXhMmTJ4cbbrghfPaznw0hhHD88ceHxx57LJSV/ff/E8BFF10Ujj766HD99deHj370owd0LyAvzkAgV84/IGfOQCBnzsC8+edT/tN//cUeNWpUvx/3X31XV9f+7K1vfetTXv/WW28NIfy/v/xP9M53vvOA93jsscc+6SdXbW1tYdasWWHZsmX7s+rq6v3fBL29vaGjoyM0NDSEWbNmhblz5x7wvYC8OAOBXDn/gJw5A4GcOQPzZij+n/7rL/gTf+oTE/uGmTZt2lNef+XKlaGsrOxvPvbII4884D1OmTLlb7Lm5uawdevW/f+7r68vXHvttWHmzJmhuro6tLa2hra2tjB//vzQ2dl5wPcC8uIMBHLl/ANy5gwEcuYMzJuh+H8aPXp0mDBhQpg/f36/Hzd//vzQ3t4eGhsb92e1tbWDvb0QQkj+Fton/ttF11xzTXjve98bzjjjjPDtb3873HbbbeHXv/51mD17dujr6zsk+wRGHmcgkCvnH5AzZyCQM2dg3vyb4k9w/vnnh6997Wvhnnvu2f+bY5/o7rvvDitWrAiXX355ydeeOnVq6OvrC8uXLw8zZ87cny9ZsuRp7fmv/ehHPwpnnnlmuP7665+Ub9u2LbS2tg7ovYDDizMQyJXzD8iZMxDImTMwX54Uf4L3v//9oba2Nlx++eWho6PjSd2WLVvCW9/61lBXVxfe//73l3ztc845J4QQwpe//OUn5V/4whcOfsMR5eXlf/Nbb//t3/4trFmzZkDvAxx+nIFArpx/QM6cgUDOnIH58qT4E8ycOTN885vfDBdeeGE4/vjjw6WXXhqmTZsWVqxYEa6//vqwefPm8L3vfS/MmDGj5GvPmTMnvPKVrwyf//znQ0dHRzj99NPDnXfeGR577LEQQgiFQmFAPofzzz8/fPzjHw9vfOMbw7Of/eywYMGC8J3vfCdMnz59QK4PHL6cgUCunH9AzpyBQM6cgfkyFP8rr371q8PRRx8dPvnJT+7/yz9mzJhw5plnhg9/+MPhuOOOO+hrf+tb3wrjx48P3/ve98JPf/rTcNZZZ4Uf/OAHYdasWaGmpmZA9v/hD3847Ny5M3z3u98NP/jBD8Ipp5wSfvGLX4QPfvCDA3J94PDmDARy5fwDcuYMBHLmDMxTofjXz9dzSM2bNy+cfPLJ4dvf/na48MILh3o7AIeUMxDIlfMPyJkzEMiZM3B48G+KH0Ld3d1/k33+858PZWVl4YwzzhiCHQEcOs5AIFfOPyBnzkAgZ87A4cs/n3IIffrTnw5//vOfw5lnnhkqKirCLbfcEm655ZZw2WWXhcmTJw/19gAGlTMQyJXzD8iZMxDImTNw+PLPpxxCv/71r8PVV18dFi5cGHbs2BGmTJkSLrroovCRj3wkVFT4+QRweHMGArly/gE5cwYCOXMGDl+G4gAAAAAAZMO/KQ4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2Tjgf9G9UCgM5j5g0Phn8xkIzkBGKmcgT5fzj5HK+cdAcAYyUjkDGQjOQEaypzoHPSkOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMhGxVBvAIDh6y3v/OdkV1NXE82PPv3oaP72C15S8v3/6cvfjuZzb58bzX/602tLvgcAAACQF0+KAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkoFIvF4gF9YKEw2HuBQXGAf8WhX4f7GfiFH/08mr/tFS85xDs5MPNWrozmL3vu2dF87dolg7mdYc0ZyNN1uJ9/I820aSdE8yVL50XzN112dfJa3/x6ujscOP8YCM7AA1dbOyqaf/izX47mH3rr65PX+tWCBdH8LS99dTRfs2bxU+wuP85ABoIzkJHsqc5BT4oDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2agY6g0AcOh84Uc/j+Zve8VLBuwe9y1dGs1v+dld0XzKMZOT13rTuWdF85OmTo3mF7z+smj+5c9+IHkPgJFk9uznRPPevr5ovn7144O5HYD9xo07Ipp/4LLXRvPUuRVCCGcff3w0f+E58Wvd9I1P9L85gBLNnv3caP6DW78XzY+bnH5fOxw973mvTnZLlsyN5uvWxd/rj1SeFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyUTHUGwBg4B1//POj+Vtedk5J1/n9Y48luwvPenk037J1fTTftasrmldWVifvMf3BP0fzFxxzTDRvamtKXgvgcHDsacdF885du6L5bbddP5jbATLU0jIhml/7oxsO8U4ABs8Z5/5dNK+rrjrEOxkcZ73m/GT3mvEXRfN3vvplg7WdIeFJcQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGSjYqg38HScd95lye61/+vCaL5xxYZovqd7TzS/+frvJu+xadOqaL5y5cPJNQCHwtixU6J5oVCI5r9/7LFo/qozzkneI3UGluot7/5EsnvWkUeWdK3f/vQXT3c7AMPCrFmnRvMPvfeSaP7FG38yiLsBcnPJW65Kdme/8exo/pITTxyk3fy3U18SPxvLyq6M5o/M/1PyWvff73UjEEJ5eXw0+qJ/OPMQ7+TQWnDn/GT3zk+8OZrX1o6K5t3d2wdkT4eaJ8UBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbMR/xeoI8amvXpHsjps0KV6cUdo9PnT5hcluy44d0fwPixeXdpNhavWKddH8q1d9JpovWHDnYG4HKMFvfnNTNJ9+xB+j+Y4d26J5Z+emgdpS0mvedH6yq66sHPT7AwxH06efFM0ba2uj+X986weDuBsgN1//vx9Ndr19fYdwJ0922cvPLSlfuGZN8loXnb89mj/00F2lbwwYsZ71rAui+ctOOSWaf+Sarw7ibg6d0WObkt2zZs6M5rW1DdG8uzt+ng53nhQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAslEx1Bt4Ot722ncnu2NOiv+W2CUPPRTNjzzuuGg++zmzk/d4yZmnR/PzTz45mj+6dm00P3rixOQ9SrWvpyear96yJblm+tix8SLxeaxdEv88FvyvO/vfHDDk1qxZPGT3vuxd10TzZ0yfVvK1fjFvXjSfN+83JV8LYDh65yffHs0Xrnk8ms+f73UYULobbr8jmpeXDe3zc2u3bo3mW3ftiuaz29uj+fGTJyfvMe8v8c+9orz8KXYHjDSzZp2a7H7ws+ui+V9WrozmX/rUhwZkT0PtxRedNdRbGHKeFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyUTHUG3g6/vCHnx5UF3PnXT+IF19Or7mqsTWaz5793Gi+YMGd0fyEE17Q39ZKsmdP/LdxL18+P7lmwZKHo/n4pqZovuqRVSXvC8jHC1/4hmj+uU//YzSvqaxKXmtVx+Zofs3br4zmu3fveIrdAQwfk9qPSnbnnHBCNP/TsmXRvLt7+4DsCTg8nXba+dH85JnTo3lvX1/yWv11pfjkV76T7O7+999E8+3bt0TzZ7/4rGj+L1e9o+R9XXjxR6L5d775zyVfCxge3vFPVyS70XW10fy8F/x9NN+1q2tA9nSojB7dFs3/4fTTk2sG6pwf7jwpDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGxUDPUGRrKurs3R/N57by7pOn/4w08HYDf9O/fctyS7caNHR/O7H300mv/8p18dkD0Bh6cTnjUnmtdUVpV8rRu//R/R/L774jnASHLa6eeXvGb1li2DsBPgcDGp/aho/r2fXx/N25tbBuzeC9esiebf/+4t0fz/fPz9yWvt3r2jpHuvWfNYNH/Hu16fXDO5Jf65f+FLH4rm1XXV0fxbX/un5D16evYmO2DgnXfeZdH89eeemVwzd8XKaD5//u8GYktD7h1XXBPNe/v6kmt+cO8fo3lXV8eA7Gm48KQ4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkI2Kod4AA2vMmInR/P/eFP9tsyGEUF4W/9nIp957bTTv7NxU+saAw851v7gtmr/2hWeUdJ1rb/pxsvvsFf9Y0rUARpJZpx5d8pqvfOgLg7AT4HBRUVkdzdubWwbsHv/2x/ui+Ttf9qpovnXr+gG7d8qaNYuj+Sc+8uXkmuu+ckU0b6ytjeZf+dcPRfPbf/H95D1WrVqY7ICB95JL/y6aj0p8X4cQwnWf+vZgbeeQmtR+VDR/88Uvj+Y9vb3Ja33x/Z+Jr+nZW/rGhjFPigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZqBjqDTCwLnrbB6L55DFjkmvWd3ZG82XL5g3EloARrK1tSrI797Q50byuqiqaP75lSzT/wseuSt5j166u9OYARoiTT35xNH/HZa9OrvnNww9H83vu/tGA7AmgP7fNn5/sPvAPl0TzrVvXD9JuDt7tt3w32f1qwcuj+XknnjhY2wEGyKhRLdH8haefXPK1vn3jPz3d7QwLr7v83dG8vSX+tfrjkiXJa917780DsaVhz5PiAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYqhnoDHJw5c86J5td89G0lX+vlZ18YzR977IGSrwUcXr5+60+T3cTm5pKu9bUbb47mq1YtLOk6ACPNc14cf902oakpueaHt94Zzffs7R6ILQGZKS8r7Xm4l5588iDt5NAqhEKyS31NSv1a/eOn/jndvf4VJV0LODBVVTXRfOb48dH8Sz/8+WBuZ1iYcuyUkj5+3vxFg7STkcOT4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMhGxVBvgINz5sv+LppXV1ZG8+/e8/vktebO/fWA7AkYuc466/+L57Nnl3ytH9z7x2h+7ZX/s+RrARwOjnnWMdG8r1hMrvnlDT8brO0Ah7HXv+2d0by3r+8Q72R4eNF5r0t3ide5qa9VKr/2gx8pfWPA07Jjx7ZofsfChdH85JNmJa81enRbNO/s3FTyvg6FMWMmRvO3veIlJV3ngVv/NBDbGdE8KQ4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABko2KoN0D/qqvrovnZf//8aL57795ofu17PpG8R09PfA1w+GlqGhfN3/3p90Tz6srKku+x8N74b/zetaur5GsBjCRtrZOj+Uue88xo/sCypclr/epX3xiQPQF5ecWrXjTUWxhULS0TovnMmXOi+ZXXvH3A7r16y5Zo7v00HHp79uyK5o89tjKaX37Beclr3XTHrdH8/15xXekbK9FRpxwTzaceOzW5ZvKR7dG8r1gs6d7FvtI+/nDkSXEAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgGxVDvQH6d/l7/ymav2j27Gj+rd/eGc3nzv3VgO0JGLne8t6PRfPzTjyx5Gt97T/i58q1V/7Pkq8FcDh41UX/I5pPaW2N5j/65e8GcTcAh593XBF/f/yRd/5/A3aPhx5/PJq/5VVvi+Zr1y4ZsHsDT8/n/1f8/W6hUEiuuei8F0bzc3/2pQHZU3/WbNkSzYuhmFzT3twyIPf+t+99bkCuM5J5UhwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgzFAQAAAADIRsVQb4AQzjzzwmT3mY+/M5p3bN8ezf/1/dcMyJ6Aw9NVH7h0wK71sTdfHs137eoasHsAjCTtR7WX9PGdm52XADE33H5HND955vRBv/efFy2J5g888MtBvzfw9Cxd+mA0f8crz0+u+fKxz4nmU6fOHpA99eeWW64rec1nb/pRNH/X6y4o6Tp79uwq+d6HG0+KAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBsVQ72BnDQ1jYvm1954TXJNRXl5NP/erb+L5g8++OuS9wVwMJqaxkbznp69g37v7du3lHTvioqq5LVGjWop6d6jR7dF80vf/4GSrtOf3t7eaP6pD7wjuWb37h0Ddn/g4LzmgrNK+vjf/Pg/BmknQK4KhUI0Ly8r7Xm45z//tSXf+2vfvzaaTx8bf83Yn9R+e/v6Sr5WqS49+0WDfg9g+Fi48Pcl5UNt9SOrBuQ6s2admuwWLbp/QO4x3HlSHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMhGxVBv4HBUVlYezb9356+i+fGTJyevNW/lymj+v9//odI3BjCAHnn0viG791d+8otovnHVxmjeOqktea3/8arzB2RPh8Km1ZuS3Zc+8/5DuBPI2zOf+ZJoPrW19RDvBODJvvyZb0fzr/xrae8ff/Pb7yS73r6+kq5V6scfqmt98ivpzxFguCoUCtG8LJGnLFp0/0BsZ0TzpDgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQjYqh3sDhaOrU2dH87OOPL/laH3jzx6L5qlULS74WwE2/+l00f/NLX3xoN/I0ve3vXzro99jb0xPN+4p9JV/rhv+4PZovuGtBSdd54O47Sr43MPDOfs0ronlFeXk0/83DD0fz++77+YDtCSCEEG772U3RfPVVl0fzyS0tg7mdAbd6y5Zoft/iJdH8Q69/S/JaGzeuGpA9ARxKxWIxmvclctI8KQ4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABko2KoNzCStbfPjOY/v+sXJV3n8ndek+x+85tvl3QtgP689WXnRvMH3/3JaF5RNXD/mTjqGUdF8//xqvMH7B6f+cYPo/njjz1e8rVu/cl3o/nSpQ+WfC1g5KmpaUh2f/eKF5R0rZ/e+Mto3tfXW9J1AJ7KmjWLo/nr/+5N0fzs17wiml/1j28csD0NpKs/9MVoftM3PnGIdwIwNKprq0v6+J179gzSTkY+T4oDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4VisVg8oA8sFAZ7LyPO+67+YjT/l4++vaTrnHTimcluwYI7S7oWf+sA/4pDv5yBjFTOQJ6uXM+/ioqqZPfj++6N5hvWb47m73rlK6P57t07St8YB8z5x0DI9Qx8/hn/kOxe9743RvM3nveiaP6NX94ezb/3v29I3qMQ4l/3xUv+HM3Xrl2SvFaunIEMhFzPwOFs+caN0byqoiKaX/HBf43mN1x31UBtadh6qnPQk+IAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgrFA/yVxLn+xtlnPvMlye7W3/4gmjfV15d0j5NOPDPZLVhwZ0nX4m/5rdsMhFzPQEY+ZyBPl/OPkcr5x0BwBjJSOQMZCM7A4efrt94eza+/+ovR/N57bx7E3QxvT3UOelIcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyEbFUG9guHvGGc9Pdk319SVda97KldF8187Okq4DAAAAAOTlzeeeNdRbOGx4UhwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgzFAQAAAADIRsVQb+Bw9LtHFkbzv3/2mdG8s3PTYG4HAAAAAID/5ElxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZKNQLBaLB/SBhcJg7wUGxQH+FYd+OQMZqZyBPF3OP0Yq5x8DwRnISOUMZCA4AxnJnuoc9KQ4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkI1C0a8kBgAAAAAgE54UBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ/ESrFixIhQKhXDjjTfuz6666qpQKBSGblMlGmn7BYYPZyCQM2cgkCvnH5AzZ+Dhy1D8r9x4442hUChE/++DH/zgAV3jmmuuCTfffPPgbhRgEDgDgZw5A4FcOf+AnDkD81Qx1BsYrj7+8Y+HadOmPSmbPXt2uPHGG0NlZWW/a6+55prwqle9KlxwwQWDuEOAweMMBHLmDARy5fwDcuYMzIuheMJ5550XnvGMZwz1NvbbvXt3qKqqCmVlHu4HBp8zEMiZMxDIlfMPyJkzMC++qiWI/TtCf61QKISdO3eGb37zm/v/Xy0uueSS/f2aNWvCm970pjBu3LhQXV0dZs+eHb7xjW886Rq/+93vQqFQCN///vfDFVdcEdrb20NdXV3o6uoKIYRw3333hXPPPTeMHj061NXVhec///nh97///d/s5Z577gnPfOYzQ01NTZgxY0b46le/OiBfByBPzkAgZ85AIFfOPyBnzsDDlyfFEzo7O8PmzZtLXnfTTTeFN7/5zeHUU08Nl112WQghhBkzZoQQQtiwYUM4/fTTQ6FQCO94xztCW1tbuOWWW8Kll14aurq6wnve854nXesTn/hEqKqqCu973/vCnj17QlVVVfjtb38bzjvvvDBnzpxw5ZVXhrKysnDDDTeEF77wheHuu+8Op556agghhAULFoSzzz47tLW1hauuuir09PSEK6+8MowbN+7pfWGALDgDgZw5A4FcOf+AnDkDM1PkSW644YZiCCH6f8uXLy+GEIo33HDD/o+/8sori3/9Zayvry9efPHFf3PtSy+9tDhhwoTi5s2bn5S/9rWvLY4ePbq4a9euYrFYLN5xxx3FEEJx+vTp+7NisVjs6+srzpw5s3jOOecU+/r69ue7du0qTps2rfjiF794f3bBBRcUa2pqiitXrtyfLVy4sFheXv43+wX4L85AIGfOQCBXzj8gZ87APHlSPOFLX/pSOOqoowbsesViMfz4xz8Or3nNa0KxWHzST57OOeec8P3vfz/MnTs3POc5z9mfX3zxxaG2tnb//543b15YvHhxuOKKK0JHR8eTrv+iF70o3HTTTaGvry8Ui8Vw2223hQsuuCBMmTJl/8ccc8wx4Zxzzgm//OUvB+zzAg5PzkAgZ85AIFfOPyBnzsC8GIonnHrqqX/zj+uvWLHioK+3adOmsG3btnDdddeF6667LvoxGzdufNL//uvfeLt48eIQwv/7Bknp7OwMe/bsCd3d3WHmzJl/08+aNcs3AvCUnIFAzpyBQK6cf0DOnIF5MRQ/RPr6+kIIIbzhDW9I/kU+4YQTnvS/n/iToSde4zOf+Uw46aSTotdoaGgIe/bseZq7BRhYzkAgZ85AIFfOPyBnzsDhzVB8EBQKhb/J2trawqhRo0Jvb28466yzDuq6//WP9Dc2NvZ7jba2tlBbW7v/p0lPtGjRooO6N8CBcgYCOXMGArly/gE5cwaOPGVDvYHDUX19fdi2bduTsvLy8vDKV74y/PjHPw4PPfTQ36zZtGnTU153zpw5YcaMGeGzn/1s2LFjR/Ia5eXl4Zxzzgk333xzWLVq1f7+kUceCbfddluJnw1AaZyBQM6cgUCunH9AzpyBI48nxQfBnDlzwu233x4+97nPhYkTJ4Zp06aF0047LXzqU58Kd9xxRzjttNPCW97ylnDssceGLVu2hLlz54bbb789bNmypd/rlpWVha9//evhvPPOC7Nnzw5vfOMbQ3t7e1izZk244447QmNjY/j5z38eQgjh6quvDrfeemt43vOeF97+9reHnp6e8IUvfCHMnj07zJ8//1B8GYBMOQOBnDkDgVw5/4CcOQNHHkPxQfC5z30uXHbZZeGKK64I3d3d4eKLLw6nnXZaGDduXLj//vvDxz/+8fCTn/wkfPnLXw5jxowJs2fPDv/yL/9yQNd+wQteEO69997wiU98Inzxi18MO3bsCOPHjw+nnXZauPzyy/d/3AknnBBuu+228N73vjd87GMfC5MmTQpXX311WLdunW8EYFA5A4GcOQOBXDn/gJw5A0eeQrFYLA71JgAAAAAA4FDwb4oDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABko+JAP/Afr/j8IG4DBte1//Seod4CI5wzkJHK+cfT9dH/ff1QbwEOyif+56VDvQUOA+/5yLVDvQU4KJ//538c6i1wGPA+mJHsqd4Le1IcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2agY6g0AMHQamkclu/aZ7dG8bcKY5JppbW3Jbub48dG8p7c3uWb+qlXJ7qFHliW7NYvXRPPu7buSawAAAIA8eFIcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyEbFUG8AgMHVMmFMsnvGc09Ids856qho3lhb+7T3dKCeNXNmsmuoqUl2d+3eG80ff+zx5Jq+nt4D3xjAIKusrkx2Jz37uGR33oknRvNH1q5Nrvn1Hfclu22bOpMdkJdCoZDs+nu9eeKzZkfzZ0ybllyzsqMj2f3hD/OS3dpl66J5776e5BoA8uRJcQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGSjYqg3AMDT1zJhTLJ7xnNPSHZHTZiQ7OatXBnNl69dn1zz2J8WJ7tH73s0mtc11iXXzDl7TrKbOnNSsmsc0xjN6/u51/Yt25MdwKFWU1+T7Noa42dcCCFUlpdH8+3d3ck1PT29B74xIFvllenxQfO4pmQ3K/F6c3xTek1/lhwxLtlt3bgtmu/Y6nUe8N8qq6uS3ZRjpyS71vb4++6VC1cl16xftu7ANzaIquuqk1396IZk17N3XzTf0bkzuaZvhLy29KQ4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAslEx1BsA4MBVVldF89ZJrck1NZWVye439z+Y7r7z22h+372/SK5Zt25pstu3b280Hzt2SnJNZ8e2ZHfymScnu8YxjdG8r6+YXAMwnBQKhWQ37+HFye7ff/jraL760dXJNS0TxiS7prFNyQ44PJWVl0fztsltyTVj2tOvRe96cEE0/96S25JryhN7CCGE8sp017OvJ9kB/Jea+upk1zyuOdmNrq2L5v29bhsuahpqk90Rxx+R7FLn8dJ56ff+W9Z1HPC+hpInxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJCNiqHeQExNQ22ym3LMlGR35IzJ0by6Iv1pbujsTHbrVm5Idh1rNkfznV27kmt69/UkO4ADUVYR/1nm9i3bk2vuvWdespv323R37z3/Hs03bVqdXDO6sS3ZTZ5yTDSfNu245Jq2yenrbVq9Kdk9vnhNNK+uqUquqa6rTnYAg6G6Nn3ujJs6Nr2wrJCsujZ3RfO9e/Yl11RUlqfvBRyWGppHJbtpx0+L5scfeURyTWd3d7Jb8dCKaL5tw9bkmv7OxzHtrclu8qz4TGBn587kmtR7+xBC2NO9J9kBI1d5ZXpO2LG2I9kt3bg0mm9dv+Vp72mw9TczWLd0XbJrmdASzcv7e/1YSL9WDcViujvEPCkOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZCP961aH0OjW0cnulONmJruTpx4RzasqDu7T7Ds1/RtRd+zeHc237kz/Vuvd+/Yd1D4OpZ7e3mS3qiP+G3gfeSj+23dDCGH9svXJbt+evQe+MSCEEEJXR1c0f/gP85NrFiy4K9l1dKxNduPGHRHNzz3vLck1E6dPTHaTZk2K50dPTq7Zum5rslvy4JJk17u9O5r3VQ2f33QNUFlT2U9Xlew2rd6U7LZt6ozmjS2jkmsamtMdcHhqaku/555z9JHRvLK8PLnmgb88muy2rk+/nkupaahNdjOOm5bsnj0zPi/Y2BV/DR1CCHff/5dkt3LhqmTn/SwMb9V11cluVD+vi3bvjM/7Qghh3dL4++eefelZWkNTfbI7lPbs2pPs1i9Pz+66dyTeW/ekP+dCoZDsisXh857ck+IAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgzFAQAAAADIRsVQbyBm7ZK1ye5rD34/2XVu7ozmPXt7kmsmTBuf7E54wYnJ7lknHRvNp7a2Jte0lZcnu8qK9B9FZT/rUvb2pD/n3r6+ZFfRz73GjR4dzXft3Ztc07kp/mcSQgj79qTXAXF9PfHv3127tifX7NnTnezGjp2S7E466cxoPmHGxOSaprFNya5lwphovnt7en/rV6xPdqkzvz9lFX4WDAwf9Y11ya5QSK9bt2xdstu1fWc0bxzTeMD7Ag4Pre1tyW7yrMnJbunGjdF85WOrk2s2rtyQ7FLv+/p6i8k1XR1dyW7JgmXJrliMX/O4SZOSa5558jHJbvfO3cnu8cceT3bAoVNRVRnNJx2V/r6fMDU9C7z/tgeS3aK/PBTNx45P36uhqT7ZHUqjW+MzvRBCGN2W7nr27ovmnZ3x15whpM/i4cZ0AAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANmoGOoNxGxctTHZ/em+25LdipUPRfPe3p6D2kfZ/ylPds3N46J5S8vE5JrKiqpkV1Vdk15Xme6KxWI037dvd3JNY2Nrsjvp2acnu6nHTonme7r3JNd0b+9OdkDpahvi58GxzzgpuebI445NdmVlhWRXWR0/s+pG1yXXTJ09Nd1NGh/NH31kWXLNqoUrk93KZYuSXeqMrm9M7x1gsFRUxV9y1zc1JNfUVVUnu5r69GvD1va2xL3qk2uAkau6Nn1WtExsSXbFEH8fGUIIi/+yNJqvX77uwDf2BDu27YzmKx9bnFyzevWjya6nZ1+ymzp1djR/7t+9MLnmhOccl+yax6e/hls3bIvmOzt3JNcAA69xTGM0Hzd5bHLNulUbkt38389Ldnv3xmdctQ21yTWHUqEs/exzf3vs77Xllu27ovne3XsPfGPDlCfFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkI2Kod5ATPO4pmR38ikvSnYzjjyl5Hvt2tWV7DZsWJHsNm5cGc1Xrnyo5D2EEEJPz75k19fXW/L1Ghqak92sWacmu461m5PdptWbonn3zl3JNWMmjEl2jWMakx1QmorK8n662gG9V3VNVbIrFovJbtWaDdF85cJVyTXr16xOdvv27Ul29aPi50tVbXVyDcBgqRtVF83LygrJNY+vWp/s9uxKn3/N4+KvAav6ObuB4a+8Mv7WfcKMick1k6aMT3br1sTf24UQwtb1W6L5vr09yTWdG7cluyWL5kfzxYv/nFyzc2dnsquoqEx2IfFadPRdbcklNfU1ye7k02Ynu+q6+OvKxx5YlFyzfcv2ZNff62jIXU1D+j3t2Cljo3n37vTrpYfvSc/utm/vSHYT2qdH88bW4THfGtUyKtm1TGxJdtv6OcPXLV0Xzft6+5Jravv58yqvGD7PZw+fnQAAAAAAwCAzFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQjfivsB5i/f3250mzJvWzsr+udL09z0p23Tu6o/me7vRvt+3P3u69yW7fnn3JrryyPJpPP2Facs3MOUclu8VzFye7P952ZzSvqkr/ebWWpX/LNzC8jZnYGs2Pfkb6DOnP3DvmRfN5d9+fXLNp0+pkN2nSrGTXMr45mpeVFZJrAAZL6uzZt7cnuWbVI6uS3ZI/p1+vtU6Kv/Zqm+w1GYxkhUL8HOndlz5Htm7rSnZdHeluZ9euaL50waLkmvnzf5fsduzYGs0nTzo6uWbS5HRXUVGZ7Lq6OqL5tm0bk2s2rtyQ7LpPnpnsZkwYH7/XlPjnG0L6axtC/3+WkLu6UXXJrqq2Kpo//mj6veTmxzcnuwkT0vO01OupQ/0+s7wyPs5tGtuUXNPQ3JDslsxdkuyWPvxING9pjZ+BIYRQ21Cb7IYTT4oDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgGxVDvYHhrLwi/TODhqb6kvLB0jimMZpPO2F6ck1ZP5/Xzm07k11NTfxza2kbm1xTN6ou2QFDr6yiPNnVJ86z5vr0Obe2Y0uyW79sXTTfuHFVeg918TMuhBDGtk9IdtV11ckOYDDUNtQku7ap8ddKVdWVyTXbNmxLdps2PZ7sRrWkz01g5Ord1xPN1yZeX4UQwvoVG5Jd947uZLdi4dJovmjR/ck1vb3x/YUQwrHHPjuaHzX75OSa5nHN6Xv19Ca7dUvjr1PXrVuWXLNsQbqrb2pIduMSZ/v2LduTa4B+FAoHtSz1Pff44jXJNR0d65PduAmTk111TdWBb2wQ1Y8ufSbZuXFbstu8ZnOy6+2Nn7kVlemRcln5wf1ZHmqeFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2KoZ6Azy1qpqqZDdu2vhoPqqxPrnmgdv/nOzm//H+ZFdbOyp+r5Z4HkII5RV+7gLD2ZiJY5Jd2+S2aL546erkmoV/WJjs1ixfGc1bmuPnWAghTDpiRrIb3TY62QEcarWj6pLd2OamaL65s+ug7jVx0vRk1zQufi9gZCsWi9G8d19Pck1foZDsNixfn+weeeTeaF5dXZtc88zTzk1246aOi+YH+16xorI82bVOao3m/b2nLu/nels3bE2vK4/vv3FMY3JNfWP6vxVdHQf33wQ4XBT6ObN69u5LdhtWbIjm2zZsS65paGhOdvVN6XnacLerqzvZbVgZ/zqFEMK+3emv77gJk6P56Nb0Wdffn+VwYmIJAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZKNiqDfAU6upr0l2ldWV0XztinXJNUvnLU12XV2bk11j45hoXlHlrxEMZ9X9nCGtE+Pf1yGEUF8bX7dq4crkmmUPLU52HVvWRvNJk2Yl1zSNS/9W8LKykfEbrYE89PX1JbsNm7dE8+UPrUiu6eroSnaj20Ynu/rR9ckOGLnqRzfE86b09/zObTuTXWVNVbKbPOnoaD6mtT25ZuyUscmuvOLQPYvXPD7+2nHq7KnJNX096fO7c3NnyV1/a4C0vt7eZNexLv5aKoQQ9u7eG80bmuPnZgghjJ3SluzKysuTXbGf13sDrVCWPjvLK+J77OnpSa7ZvXP3Qe0j9XXsb1Y5UnhSHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANmoGOoN8J8KhWRVXpn+Y9rVuSuaL/rTouSaDetWJrtJk2Ylu8kzp0bzqurK5Brg0CiUpX/GOWbCmGRXN7o+2T2+fG08X/R4cs2+fXuTXXPzuHg+tiW5prLKf6aA4aO8sjzZVddUJbu9u+Nn44bl65NrNm9Yl+zqGuuSXaGf15TAyFXbUBvNm8c1J9f07utNdm2T25Ld2Cnp7lApK0+ft6mvRQghtB/VHs3HjE+/3lzXz1m8Zf2WZLf58c3RfPu27ck1LePSr8tHtzUmO8hBX28x2e3q3JnsUjOz4557XHLNjFlTkt2WrvT38I6tO5JdSn/vaRtb0t/3E5qakl1vX180X7Rk1QHv64nqGtPn6uE88/OkOAAAAAAA2TAUBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJRMdQb4P8Z1dyQ7MZNHZfsdu/cHc23rtuaXFMoFJJdY3NTsquqrUp2wNDq/wwZm+zKytI/G92wcmM0X/XYiuSa3bt3JLv2KTOieWNrY3INwHDSOCZ9XrUfMSHZbVzXEc1379yTXNM8Jn1293fmA4enHZ3x11i9vb3JNd07upNdP28Jh4VRLaOS3QnPOS7ZHTdpUjR/6PHHk2s2r9mc7FYtXJnsNm5cHc3b2iYn19Q31Sc7yF2hLH0w7d2zL9ltWh7/Pq1rrEuuecYpxyS7c044IdlVlpcnu5Tevr5kt2df+vPq6WfdI2vXRPPOzZ3JNdu3bE923Tvis8UQQihLfM7VddXJNSOFJ8UBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQjYqh3kBOqutrkt3kY6Yku0kTxya7e3/zp2i+cf3jyTWjR6evN7p1dLIrFArJDhhadaPrk111bXWy27y2I9mteGhFNF+16pH0PupGlbyPsjJnCzAy9PdaqHvP3mTX1dEVzWsba5Nrmsc3J7v6fs584PC0e0d3SflI0NreluxOOP3YZHfkuHHJbvmmTdF8yYJlyTVb+nk93NW1Jdnt2L41mjc3j0+uAdL6e1/Y32uw9RuWR/O5X/t1cs3dt96S7I4/5VnJrnlcU7JL6docfx0YQgjLH3ss2e3evTPZTZ56VDQfOyU97ysWi+mur5+un3UjnSfFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGxUDPUGDjeFsvTPGdraW5PdpInp3xD7+NqNyW7RA/HfVNvTsze5prFldLKrrqtOdsDwtWPrjmS3YuHKg1pXVV0ZzWfNekZyTfOElmTXMr45mvd3bhb7+pLdvr09ya6vN76urDx9r8qq9H8SyyvSXdPYpmjePrM9uaZlVEOy68/Grdui+cqH0n/GwMDp6+lNdru270p2O7Zuj+a7t3cn11RWxc9ggCcqFAoHta66vibZNbU1RfOJMyYk15w6a2aymzFuXDSvLC9Prlnf2Znsbp+/INktun9RNE+dwyGEsGX91mS3vasj2Y2fMD2aT5w+MbmmojL9OQNpja2Nye6oo+ZE85596bnYokX3JbsHH/z1gW/sABQK/bwHraxKdqMb25JdWeI99Kjm9PvM5vHp9+qtk9Lzyp7E++7efen34yOFJ8UBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQjYqh3sDhpm1yW7I79sSZyW7nnj3J7oFbHkh2q1YsiuatrROTaxpbRiU7YGTasXX7QXX9mXXa0dH8RS94ZnLNiVOnJrvK8vKS99DV3Z3slm3cmOw6duyI5mMaGpJrjmhLn99NdXXJ7lDa2NUZzX9bXXmIdwKHt9qGmmjeNK45uaaqtirZFcrjz6FU9PO9W5ZYA+Rpx7ad0XxnIg8hhPqm+mTXPrM92Z37otOj+bHtk5JrDsbyTenXcr/7y8PJbsWC5clu1/Zd0Xz71vhrwxBC6Ni0LtmVV6TP6Za21mheXVedXAMcnKp+XjNNmhU/m9qmvjy55uRNL0h2XVu7kl1v775kl1JXl34P2jJxTLKr7Odz7t3XE80rqkpf81Td4cyrbQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGSjYqg3MFKVV8a/dA1N9ck1u/ftS3bz71mQ7B760wPJrljsjebNY1uTa6rrqpMdMDJ179id7HbvTHc19TXJrq+3L5rv2rs3uWZ7d3eyqygvT3Yp5WWFZDd70qR+1sV/5tvbF/+cQghhb0/6jN68fXuyW7ttWzT/86IlyTVL5qa7jrUdya68Mv41HNXckFzz2tOfleyAuIbmUdF8dCIPIYS1K9Ynuw3L413vvp7kmrJyz64A/617+65ovmnDmuSanp5xyW5H585kt2hd/MxatnFTcs3CuY8luw0rN0Tz1GvNEELo7Ym/zw0hhO4d6debjy9ZHs0XPnJvck1lZVWymzPnxcmudVL6PTcw9Kpr0t/bbZPbDqobLiqrSh/n7t2Tfr+7r5+usroymlcl8pHEq20AAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABko2KoNzCclVWUJ7v2me3RfMLU8ck1C//0aLK7899/ley2b9+S7GbMOCmaN45pTK4BDj8dazYnu4UP35vs+op9yW7UH1qi+c1f/05yTUVFVbIbMyZ+bo4/YkJ6zcQxya6hqT7Zbd+yPZqvemR1cs3Sx+Ynu+XLFyS7rVvXR/Oenr3JNQercVT863Hssc9OL/rw2wZ8H3A4KCsrJLuKqvhL5PKy9PMkW9anX6+tXrwymtfWNyTXNI1rTnZAfmoaaqN5b29vcs2jC+9Pdh2b1iW71Y/GXy+Nak6fWQejkD6Gw46tO5LdQw/+Mdk9+mi6S5k167RkVzc6/XoTYMgkDtBiX/r9fXl5esZZ0ZAeD/f3mnmk86Q4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkI30rxclNPTzm6bHThkbzasr01/SzWs6kl1Pz75kN2nSrGQ3/ogJ0byyyh8t5KS6vibZlZWnz4Oli/+U7Pbs2fW09nR4Sf/G7ULiN3/X1TUm14wfPy3ZtbZOSnYNDc3RfExb/L8FQFqxWEx22zZui+Yd/byW+8sdf0l2mzatjuZHj5+TXFPbUJvsgPykzoTGxjHJNUuWzE12S5c+mOxWrZwRzWcdfVpyzZjx8ffHIYSwed2GaL5y5cPJNV1d6fO2tzf93nnq1OOi+axZpybXjD9iXLKrrKlKdgCDqb6pIdnVjYr/N2H1o/HXnCGEsHbZ2mTXMi7935KmsaOT3UjnSXEAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABko2KoNzDUqmqqkl3blLHJrqKiPJovXrgiuWbfnn3J7ugTTkp2/e2xuq462QH5aJvUmuxOrTsr2bW3z0x2XV0d0bxYLCbX9PTsTXYdHWui+fr1y5Nruru3J7u6usZkN2HCjGje0jIhuWbUqJZkN27i5GTXPL45mldVVybXAEOvt6cv2T1090PR/PbbvpNcs2nTqmR3yilnR/OG5obkmkKhkOyA/FRUxt9/Tpw+MbmmvPy8ZLd48Z+T3fLlf4nmS5fNS64ZaI2N6de2xx9/RrKblXhfPaqf8xZgONqza0+y27BiQzT/0+9/m1yzY0dnsjup5sxk1zR2dLIb6TwpDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGxUDPUGhlp1XXWyq6hMf3m2btwWzTet3pRcs2/PvvQ+atP7qKypSnaFQiHZAYQQQmPLqH664w7hTgCGj0JZ+jVURVX8NWBLy/jkmtbW9mR3/CnPiuZNbU3JNQAHor/3s1OOmZzsxk4Zm+yOWDo7mj/66P3JNZs3P57sWlsnxe9zRPw+IYTQ1NSW7Bqa069taxtqkx3ASLJy4cpk9+cHfhXNt3dtSa6ZdfRpya5xTOOBb+ww4klxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZKNiqDcw1Pbs2pPs1i1dm+x2du2K5ov+tDC5Zs3ji5PdhIkzkt3UY6Ynu+qaqmQHAEBcoVBIdlOPnZrILxus7QAcUjX11cluxgkzS8oBGHhV/cz7pkyZHc1HNTQn17QfOSnZ1dTXHPjGDiOeFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyUTHUGxhqm9dsTnYL581Ndw/fE8137upMrmlvPyrZTZ5yTLIrFArJDgAAAAA4fLRNaj2ojgPnSXEAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABko1AsFotDvQkAAAAAADgUPCkOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2/n8QuZGxx6K1+wAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also design filters that detect vertical lines.\n",
        "\n",
        "Here's one that detects vertical lines that are black on the left and white on the right."
      ],
      "metadata": {
        "id": "XQWV7rvcRhpf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "vertical_filter = [[-1, 0, 1],\n",
        "                   [-2, 0, 2],\n",
        "                   [-1, 0, 1]]\n",
        "\n",
        "plot_filter(images, vertical_filter)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 444
        },
        "id": "miBwtyAtRqFn",
        "outputId": "8c75ea39-1ea8-4d56-d1ed-87d17ba8ee49"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 2000x500 with 10 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABcUAAAGrCAYAAAAB5v8sAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABApklEQVR4nO3deXyeVZk38PMkT7a2Sbok3feFbpSt7CIIooAyiiM6KjKgKKjjNr7q68II6LzoqK/4jooDiqAogqIwIgIKslsQKKWFAqWULhQKbdMtTbcs7x/jdEDPCX1K0iQ93+/nwx9cv5xzn4Tm6v1cuclT6Ojo6AgAAAAAAJCBsp4+AAAAAAAA7CmG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4t3k/PPPD4VCYbfWXnHFFaFQKISlS5d27aFeYunSpaFQKIQrrrii264B5EsPBHKl/wE50wOBnOmBfYuheMRjjz0W3vve94ZRo0aFqqqqMHLkyHDaaaeFxx57rKePBtDt9EAgV/ofkDM9EMiZHpifQkdHR0dPH6I3+fWvfx3e/e53h8GDB4ezzjorTJgwISxdujRcdtllYe3ateHqq68Ob3vb215xn9bW1tDa2hqqq6tLPkNbW1vYsWNHqKqq2u2fML2SpUuXhgkTJoTLL788nHnmmd1yDaDv0QOBXOl/QM70QCBnemCeij19gN7k6aefDqeffnqYOHFiuOuuu0JjY+PO7BOf+ER47WtfG04//fQwf/78MHHixOgemzdvDv379w/FYjEUi7v35S0vLw/l5eW7tRZgd+mBQK70PyBneiCQMz0wX359ykt84xvfCC0tLeHSSy992TdBCCE0NDSESy65JGzevDl8/etfDyH8z+8KWrhwYXjPe94TBg0aFI466qiXZS+1ZcuW8PGPfzw0NDSE2tra8Ja3vCWsXLkyFAqFcP755+/8uNjvERo/fnw4+eSTwz333BMOPfTQUF1dHSZOnBh+8pOfvOwaTU1N4dOf/nSYNWtWGDBgQKirqwsnnXRSeOSRR7rwKwXsjfRAIFf6H5AzPRDImR6YL0+Kv8QNN9wQxo8fH1772tdG86OPPjqMHz8+3HjjjS+rv+Md7whTpkwJF154Yejst9GceeaZ4Re/+EU4/fTTw+GHHx7uvPPO8OY3v3mXz7d48eJw6qmnhrPOOiucccYZ4Uc/+lE488wzw+zZs8PMmTNDCCEsWbIkXH/99eEd73hHmDBhQnjhhRfCJZdcEo455piwcOHCMHLkyF2+HpAXPRDIlf4H5EwPBHKmB+bLUPwvNmzYEJ577rnw1re+tdOP22+//cJvfvObsGnTpp21/fffP1x11VWdrps7d274xS9+ET75yU+Giy66KIQQwkc+8pHwvve9b5d/cvPkk0+Gu+66a+c36jvf+c4wZsyYcPnll4dvfvObIYQQZs2aFRYtWhTKyv7nfwI4/fTTw7Rp08Jll10W/uVf/mWXrgXkRQ8EcqX/ATnTA4Gc6YF58+tT/uK//2DX1tZ2+nH/nW/cuHFn7UMf+tAr7n/zzTeHEP7rD/9LfexjH9vlM86YMeNlP7lqbGwMU6dODUuWLNlZq6qq2vlN0NbWFtauXRsGDBgQpk6dGubOnbvL1wLyogcCudL/gJzpgUDO9MC8GYr/xX//AX/pT31iYt8wEyZMeMX9ly1bFsrKyv7mYydPnrzLZxw7duzf1AYNGhTWrVu389/b29vDRRddFKZMmRKqqqpCQ0NDaGxsDPPnzw8bNmzY5WsBedEDgVzpf0DO9EAgZ3pg3gzF/6K+vj6MGDEizJ8/v9OPmz9/fhg1alSoq6vbWaupqenu44UQQvJdaF/6u4suvPDC8KlPfSocffTR4ac//Wm45ZZbwh/+8Icwc+bM0N7evkfOCfQ9eiCQK/0PyJkeCORMD8yb3yn+EieffHL4wQ9+EO65556d7xz7UnfffXdYunRpOOecc0ree9y4caG9vT0888wzYcqUKTvrixcvflVn/mvXXnttOPbYY8Nll132svr69etDQ0NDl14L2LvogUCu9D8gZ3ogkDM9MF+eFH+Jz3zmM6Gmpiacc845Ye3atS/Lmpqawoc+9KHQr1+/8JnPfKbkvU844YQQQggXX3zxy+rf+c53dv/AEeXl5X/zrre//OUvw8qVK7v0OsDeRw8EcqX/ATnTA4Gc6YH58qT4S0yZMiX8+Mc/DqeddlqYNWtWOOuss8KECRPC0qVLw2WXXRbWrFkTfv7zn4dJkyaVvPfs2bPD29/+9vDtb387rF27Nhx++OHhzjvvDIsWLQohhFAoFLrkczj55JPDl7/85fC+970vHHnkkWHBggXhZz/7WZg4cWKX7A/svfRAIFf6H5AzPRDImR6YL0Pxv/KOd7wjTJs2LXz1q1/d+Yd/yJAh4dhjjw1f+MIXwr777rvbe//kJz8Jw4cPDz//+c/DddddF44//vhwzTXXhKlTp4bq6uouOf8XvvCFsHnz5nDVVVeFa665Jhx00EHhxhtvDJ/73Oe6ZH9g76YHArnS/4Cc6YFAzvTAPBU6/vr5evaoefPmhQMPPDD89Kc/DaeddlpPHwdgj9IDgVzpf0DO9EAgZ3pg7+B3iu9BW7Zs+Zvat7/97VBWVhaOPvroHjgRwJ6jBwK50v+AnOmBQM70wN7Lr0/Zg77+9a+Hhx56KBx77LGhWCyGm266Kdx0003h7LPPDmPGjOnp4wF0Kz0QyJX+B+RMDwRypgf2Xn59yh70hz/8IVxwwQVh4cKFobm5OYwdOzacfvrp4Ytf/GIoFv18Ati76YFArvQ/IGd6IJAzPbD3MhQHAAAAACAbfqc4AAAAAADZMBQHAAAAACAbhuIAAAAAAGRjl3+je6FQ6M5zQLfxa/PpCnogfZUeyKul/9FX6X90BT2QvkoPpCvogfRlr9QHPSkOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMhGsacPAEDv9cGP/Z9kVt2vOlqfdvi0aP0jp7yp5Ov/68U/jdbn3jo3Wr/uuotKvgYAAACQF0+KAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkodHR0dOzSBxYK3X0W6Ba7+EccOrW398DvXHtDtP7ht71pD59k18xbtixaf8tRb4zWn3tucXcep1fTA3m19vb+19dMmLBftL746XnR+vvPviC5149/mM72BvofXUEP3HU1NbXR+he+eXG0/vkPvSe51+8XLIjWP/jmd0TrK1c+9Qqny48eSFfQA+nLXqkPelIcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyEaxpw8AwJ7znWtviNY//LY3ddk17n/66Wj9pt/cFa2PnT4mudf7Tzw+Wj9g3Lho/ZT3nB2tX/zNzyavAdCXzJz5mmi9rb09Wl+14tnuPA7ATsOGjY/WP3v2u6L1VN8KIYQ3zpoVrR93QnyvK3/0lc4PB1CimTOPitavufnn0fq+Y9Kva3uj1772Hcls8eK50frzz8df6/dVnhQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAslHs6QMA0PVmzTomWv/gW04oaZ97Fy1KZqcd/9ZovWndqmi9pWVjtF5RUZW8xsSHH4rWXzd9erQ+sHFgci+AvcGMw/aN1je0tETrt9xyWXceB8jQ4MEjovWLrr18D58EoPscfeLfRev9qir38Em6x/HvPDmZvXP46dH6x97xlu46To/wpDgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyUezpA7waJ510djJ71/8+LVp/cekL0fq2Ldui9esvuyp5jdWrl0fry5Y9llwDsCcMHTo2Wi8UCtH6vYsWReunHn1C8hqpHliqD37iK8nsiMmTS9rrj9fd+GqPA9ArTJ16aLT++U+dGa1/94pfd+NpgNyc+cHzk9kb3/fGaP1N++/fTaf5H4e+Kd4by8rOi9Yfn/9gcq8//9l9IxBCeXl8NPr6fzh2D59kz1pw5/xk9rGvfCBar6mpjda3bNnUJWfa0zwpDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGQj/harfcTXLjk3me07enQ8OLq0a3z+nNOSWVNzc7T+p6eeKu0ivdSKpc9H65ec/41ofcGCO7vzOEAJbrvtymh94vj7ovXm5vXR+oYNq7vqSEnvfP/JyayqoqLbrw/QG02ceEC0XldTE63/9ifXdONpgNz88D/+JZm1tbfvwZO83NlvPbGk+sKVK5N7nX7ypmj90UfvKv1gQJ91xBGnROtvOeigaP2LF17SjafZc+qHDkxmR0yZEq3X1AyI1rdsiffT3s6T4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2ij19gFfjw+/6RDKbfkD8XWIXP/potD55332j9ZmvmZm8xpuOPTxaP/nAA6P1J557LlqfNnJk8hql2tHaGq2vaGpKrpk4dGg8SHwezy2Ofx4L/vednR8O6HErVz7VY9c+++MXRusHT5xQ8l43zpsXrc+bd1vJewH0Rh/76kei9YUrn43W5893HwaU7vJbb4/Wy8t69vm559ati9bXtbRE6zNHjYrWZ40Zk7zGvEfin3uxvPwVTgf0NVOnHprMrvnNpdH6I8uWRevf+9rnu+RMPe0Npx/f00focZ4UBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJR7OkDvBp/+tN1u5XF3HnXNfHg4vSa8+saovWZM4+K1hcsuDNa32+/13V2tJJs2xZ/N+5nnpmfXLNg8WPR+vCBA6P15Y8vL/lcQD6OO+690fq3vv7P0Xp1RWVyr+Vr10TrF37kvGh969bmVzgdQO8xetQ+yeyE/faL1h9csiRa37JlU5ecCdg7HXbYydH6gVMmRutt7e3JvTrLSvHV7/8smd39n7dF65s2NUXrR77h+Gj9387/aMnnOu2ML0brP/vx/yl5L6B3+Oi/npvM6vvVROsnve7vo/WWlo1dcqY9pb6+MVr/h8MPT67pqj7f23lSHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANko9vQB+rKNG9dE63PmXF/SPn/603VdcJrOnXjiB5PZsPr6aP3uJ56I1m+47pIuOROwd9rviNnRenVFZcl7XfHT30br998frwP0JYcdfnLJa1Y0NXXDSYC9xehR+0TrP7/hsmh91KDBXXbthStXRutXX3VTtP7/vvyZ5F5btzaXdO2VKxdF6x/9+HuSa8YMjn/u3/ne56P1qn5V0fpPfvCvyWu0tm5PZkDXO+mks6P195x4bHLN3KXLovX58+/oiiP1uI+ee2G03tbenlxzzZz7ovWNG9d2yZl6C0+KAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANko9vQB6FpDhoyM1v/jyvi7zYYQQnlZ/GcjX/vURdH6hg2rSz8YsNe59MZbovV3HXd0SftcdOWvktk3z/3nkvYC6EumHjqt5DXf//x3uuEkwN6iWFEVrY8aNLjLrvHL++6P1j/2llOj9XXrVnXZtVNWrnwqWv/KFy9Orrn0++dG63U1NdH69//989H6rTdenbzG8uULkxnQ9d501t9F67WJ7+sQQrj0az/truPsUaNH7ROtf+CMt0brrW1tyb2++5lvxNe0bi/9YL2YJ8UBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBR7+gB0rdM//NlofcyQIck1qzZsiNaXLJnXFUcC+rDGxrHJ7MTDZkfr/Soro/Vnm5qi9e986fzkNVpaNqYPB9BHHHjgG6L1j579juSa2x57LFq/5+5ru+RMAJ25Zf78ZPbZfzgzWl+3blU3nWb33XrTVcns9wveGq2ftP/+3XUcoIvU1g6O1o87/MCS9/rpFf/6ao/TK7z7nE9E66MGx79W9y1enNxrzpzru+JIvZ4nxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsFHv6AOye2bNPiNYv/JcPl7zXW994WrS+aNEDJe8F7F1+ePN1yWzkoEEl7fWDK66P1pcvX1jSPgB9zWveEL9vGzFwYHLNL26+M1rftn1LVxwJyEx5WWnPw735wAO76SR7ViEUklnqa1Lq1+qfv/Z/0tl73lbSXsCuqaysjtanDB8erX/vFzd053F6hbEzxpb08fPmP9lNJ+k7PCkOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBR7+gDsnmPf8nfRelVFRbR+1T33JveaO/cPXXImoO86/vh/jNdnzix5r2vm3BetX3Te/yp5L4C9wfQjpkfr7R0dyTW/u/w33XUcYC/2ng9/LFpva2/fwyfpHV5/0rvTWeI+N/W1StUv+twXSz8Y8Ko0N6+P1m9fuDBaP/CAqcm96usbo/UNG1aXfK49YciQkdH6h9/2ppL2eeDmB7viOH2aJ8UBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBR7+gB0rqqqX7T+xr8/Jlrfun17tH7RJ7+SvEZra3wNsPcZOHBYtP6Jr38yWq+qqCj5GgvnxN/xu6VlY8l7AfQljQ1jovU3veaQaP2BJU8n9/r973/UJWcC8vK2U1/f00foVoMHj4jWp0yZHa2fd+FHuuzaK5qaonWvp2HP27atJVpftGhZtH7OKScl97ry9puj9f8499LSD1aifQ6aHq2PmzEuuWbM5FHRentHR0nX7mgv7eP3Rp4UBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJR7OkD0LlzPvWv0frrZ86M1n/yxzuj9blzf99lZwL6rg9+6kvR+kn771/yXj/4bbyvXHTe/yp5L4C9wamn/1O0PrahIVq/9nd3dONpAPY+Hz03/vr4ix/7xy67xqPPPhutf/DUD0frzz23uMuuDbw63/7f8de7hUIhueb0k46L1k/8zfe65EydWdnUFK13hI7kmlGDBnfJtX/58291yT59mSfFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwUe/oAhHDssacls298+WPR+tpNm6L1f//MhV1yJmDvdP5nz+qyvb70gXOi9ZaWjV12DYC+ZNQ+o0r6+A1r9EuAmMtvvT1aP3DKxG6/9kNPLo7WH3jgd91+beDVefrph6P1j7795OSai2e8JlofN25ml5ypMzfddGnJa7555bXR+sfffUpJ+2zb1lLytfc2nhQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANoo9fYCcDBw4LFq/6IoLk2uK5eXR+s9vviNaf/jhP5R8LoDdMXDg0Gi9tXV7t19706amkq5dLFYm96qtHVzStevrG6P1sz7z2ZL26UxbW1u0/rXPfjS5ZuvW5i67PrB73nnK8SV9/G2/+m03nQTIVaFQiNbLy0p7Hu6YY95V8rV/cPVF0frEofF7xs6kztvW3l7yXqU6642v7/ZrAL3HwoX3llTvaSseX94l+0ydemgye/LJP3fJNXo7T4oDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2Sj29AH2RmVl5dH6z+/8fbQ+a8yY5F7zli2L1v/vZz5f+sEAutDjT9zfY9f+/q9vjNZfXP5itN4wujG51z+denKXnGlPWL1idTL73jc+swdPAnk75JA3RevjGhr28EkAXu7ib/w0Wv/+v5f2+vG2P/4smbW1t5e0V6kfv6f2+ur3058jQG9VKBSi9bJEPeXJJ//cFcfp0zwpDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGSj2NMH2BuNGzczWn/jrFkl7/XZD3wpWl++fGHJewFc+fs7ovUPvPkNe/Ygr9KH//7N3X6N7a2t0Xp7R3vJe13+21uj9QV3LShpnwfuvr3kawNd743vfFu0Xiwvj9Zve+yxaP3++2/osjMBhBDCLb+5Mlpfcf450fqYwYO78zhdbkVTU7R+/1OLo/XPv+eDyb1efHF5l5wJYE/q6OiI1tsTddI8KQ4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABko9jTB+jLRo2aEq3fcNeNJe1zzscuTGa33fbTkvYC6MyH3nJitP7wJ74arRcru+6viX0O3ida/6dTT+6ya3zjR7+I1p9d9GzJe93866ui9aeffrjkvYC+p7p6QDL7u7e9rqS9rrvid9F6e3tbSfsAvJKVK5+K1t/zd++P1t/4zrdF6+f/8/u67Exd6YLPfzdav/JHX9nDJwHoGVU1VSV9/OZt27rpJH2fJ8UBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQjUJHR0fHLn1godDdZ+lzPn3Bd6P1f/uXj5S0zwH7H5vMFiy4s6S9+Fu7+EccOqUH0lfpgbxaufa/YrEymf3q/jnR+gur1kTrH3/726P1rVubSz8Yu0z/oyvk2gOPOfofktm7P/2+aP19J70+Wv/R726N1n/+fy9PXqMQ4l/3pxY/FK0/99zi5F650gPpCrn2wN7smRdfjNYri8Vo/dzP/Xu0fvml53fVkXqtV+qDnhQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAslHo2MW3JM71HWcPOeRNyezmP14TrQ/s37+kaxyw/7HJbMGCO0vai7/lXbfpCrn2QPo+PZBXS/+jr9L/6Ap6IH2VHkhX0AN7nx/efGu0ftkF343W58y5vhtP07u9Uh/0pDgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQjWJPH6C3O/joY5LZwP79S9pr3rJl0XrL5g0l7QMAAAAA5OUDJx7f00fYa3hSHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMhGsacPsDe64/GF0frfH3lstL5hw+ruPA4AAAAAAH/hSXEAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABko9DR0dGxSx9YKHT3WaBb7OIfceiUHkhfpQfyaul/9FX6H11BD6Sv0gPpCnogfdkr9UFPigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZKHR4S2IAAAAAADLhSXEAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUL8HSpUtDoVAIV1xxxc7a+eefHwqFQs8dqkR97bxA76EHAjnTA4Fc6X9AzvTAvZeh+F+54oorQqFQiP7zuc99bpf2uPDCC8P111/fvQcF6AZ6IJAzPRDIlf4H5EwPzFOxpw/QW335y18OEyZMeFlt5syZ4YorrggVFRWdrr3wwgvDqaeeGk455ZRuPCFA99EDgZzpgUCu9D8gZ3pgXgzFE0466aRw8MEH9/Qxdtq6dWuorKwMZWUe7ge6nx4I5EwPBHKl/wE50wPz4qtagtjvEfprhUIhbN68Ofz4xz/e+b9anHnmmTvzlStXhve///1h2LBhoaqqKsycOTP86Ec/etked9xxRygUCuHqq68O5557bhg1alTo169f2LhxYwghhPvvvz+ceOKJob6+PvTr1y8cc8wx4d577/2bs9xzzz3hkEMOCdXV1WHSpEnhkksu6ZKvA5AnPRDImR4I5Er/A3KmB+69PCmesGHDhrBmzZqS11155ZXhAx/4QDj00EPD2WefHUIIYdKkSSGEEF544YVw+OGHh0KhED760Y+GxsbGcNNNN4WzzjorbNy4MXzyk5982V5f+cpXQmVlZfj0pz8dtm3bFiorK8Mf//jHcNJJJ4XZs2eH8847L5SVlYXLL788HHfcceHuu+8Ohx56aAghhAULFoQ3vvGNobGxMZx//vmhtbU1nHfeeWHYsGGv7gsDZEEPBHKmBwK50v+AnOmBmengZS6//PKOEEL0n2eeeaYjhNBx+eWX7/z48847r+Ovv4z9+/fvOOOMM/5m77POOqtjxIgRHWvWrHlZ/V3veldHfX19R0tLS0dHR0fH7bff3hFC6Jg4ceLOWkdHR0d7e3vHlClTOk444YSO9vb2nfWWlpaOCRMmdLzhDW/YWTvllFM6qqurO5YtW7aztnDhwo7y8vK/OS/Af9MDgZzpgUCu9D8gZ3pgnjwpnvC9730v7LPPPl22X0dHR/jVr34V3vnOd4aOjo6X/eTphBNOCFdffXWYO3dueM1rXrOzfsYZZ4Sampqd/z5v3rzw1FNPhXPPPTesXbv2Zfu//vWvD1deeWVob28PHR0d4ZZbbgmnnHJKGDt27M6PmT59ejjhhBPC7373uy77vIC9kx4I5EwPBHKl/wE50wPzYiiecOihh/7NL9dfunTpbu+3evXqsH79+nDppZeGSy+9NPoxL7744sv+/a/f8fapp54KIfzXN0jKhg0bwrZt28KWLVvClClT/iafOnWqbwTgFemBQM70QCBX+h+QMz0wL4bie0h7e3sIIYT3vve9yT/I++2338v+/aU/GXrpHt/4xjfCAQccEN1jwIABYdu2ba/ytABdSw8EcqYHArnS/4Cc6YG9m6F4NygUCn9Ta2xsDLW1taGtrS0cf/zxu7Xvf/+S/rq6uk73aGxsDDU1NTt/mvRSTz755G5dG2BX6YFAzvRAIFf6H5AzPbDvKevpA+yN+vfvH9avX/+yWnl5eXj7298efvWrX4VHH330b9asXr36FfedPXt2mDRpUvjmN78Zmpubk3uUl5eHE044IVx//fVh+fLlO/PHH3883HLLLSV+NgCl0QOBnOmBQK70PyBnemDf40nxbjB79uxw6623hm9961th5MiRYcKECeGwww4LX/va18Ltt98eDjvssPDBD34wzJgxIzQ1NYW5c+eGW2+9NTQ1NXW6b1lZWfjhD38YTjrppDBz5szwvve9L4waNSqsXLky3H777aGuri7ccMMNIYQQLrjggnDzzTeH1772teEjH/lIaG1tDd/5znfCzJkzw/z58/fElwHIlB4I5EwPBHKl/wE50wP7HkPxbvCtb30rnH322eHcc88NW7ZsCWeccUY47LDDwrBhw8Kf//zn8OUvfzn8+te/DhdffHEYMmRImDlzZvi3f/u3Xdr7da97XZgzZ074yle+Er773e+G5ubmMHz48HDYYYeFc845Z+fH7bfffuGWW24Jn/rUp8KXvvSlMHr06HDBBReE559/3jcC0K30QCBneiCQK/0PyJke2PcUOjo6Onr6EAAAAAAAsCf4neIAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZKO7qB/7zud/uxmNA97roXz/Z00egj9MD6av0P14t/Y++Sv+jK+iB9FV6IF1BD6Qve6U+6ElxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZKPY0wcAoOcMGFSbzAqFeH3r5m3JNTUDqpPZ9m07ovXVK1an12zdnszKytI/162sqYzW+9f1S64BAAAA8uBJcQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbxZ4+AADda/CIIcls0gGTktm6F9ZF688veT65pm5IXTJr2bQlWt+8fnP6DGtXJ7OmpvQ5amsHR+szDj4guaZYUZ7MAPa08or0bfqICcOT2epn433z6YefTq6pHzowmdUOGpDMgMwUCsloYCd9pF9tv2h99Yr0fd6yhcuSWXkxfc82ZGT8vtd9HgB/zZPiAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyEaxpw8AwKs3eMSQZHbwUfsls8ba2mR274rV0frmDZuTa1o2tiSz8mJ5tD58wrDkmsqaymT2xBP3JbNnn30yWh8xemxyTeOYxmQGsKfVDalLZiccd1gy++W1t0br9825IbnmoNknJLPaQQOSGZCXyqqKZLb/kfsms5EDB0brv7nu9uSaJxb+OZlt3Zq+Fz30yHg/axiVvlcG8lNRlX6dOaSTfrFs4bJofe3Ktck1IyeN2PWDdaOqflXJbMf21mTWknj9X6xM/51QrIi/9u9tPCkOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBR7+gAA7LqKqspo/eCj9kuuOXD8+GR2w733J7N5f5wXra9fvT65prw8/bPWIaMaovUj3nJEcs22lm3J7NGH70tmjzzyx2j9uRXLkmsaxzQmM4A9rWF0vGeGEML0kaOS2QtLX4jWVz63OLnmkEPfvOsHA/Z6hbL4/dzwiSOSa46cMjmZPdu0LlrfsXV7cs3WrZuT2ZIljySziRMPiNYbRg1JrgHyM6STnnD00bOT2bf/MDdaf+iBPyTXjJz0j7t+sG40fEK6h2/bkn7dPec3f4rWq2qqk2tGTkpfqzfxpDgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyUezpA8TU1PZLZgMG9k9mm9Y1R+vNiXoIIWzbsi2Ztbe1J7PKmspovby8PLmmWJHOAHZF3ZC6aP2IKVOSa17YsCGZ3XrlbcnskYfviNYbG8cm14ybNDWZDRo+KFo/dFr67M1btyazAQMGJrO2ttZoffv2Lck1AHtasbIimU2cODqZPdvUlMwennNPtF47IN6DQwihcUxjMgP2Tv0HDkhmqfvNKdPHJ9c0NW9OZn96cEG0vn3bjuSagQOHJrOqqppktmXLpmg9NSsIIYTKqnQvrupXlcyAvmvM1DHJbNqIEclsxbJF0fpzzy1+1WfqbkNGDUlmg2rTfyfceMWvovVNm9Ym1wyfcHIyKysrJLM9zZPiAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDaKPX2AmIn7T0xnY9PvAjvnroej9Q0vrk+u2bw+/S7Unb0bdnW/6mi9s3enrh2UfjfX3qKttT2ZbWqKv5N3W1tbck3t4Npk1tm7fANx5RXl0foLGzYk1zy05Jlk9szTjyazjo6OaH348HHJNQOHDkxmMw6fHq2Pb2hIrvn9o+nzrVmzMpml1Nc3lrwGoLvUDalLZgeMS/famx+M3/OGEMITT9wfrY8dNzO5pl9tTTID9k7TE/dlIYQwpmFItL5izdrkmtsfeCSZLX98xa4f7C+GDR2fzNraWpNZTU389efTCx/vZE36dfr4GZOSWVV1ZTIDel5V//jcLoQQ9h83NpmtXNeUzJYvfyxa79+/ftcP1kNSr+9DCGH/semvR7EY73XLl6f7auuOE5NZb5oFelIcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2Sj29AFixo4alswmNDYms7u2t0br5RXpT7NxTHq/9raOZLZ96/Zofce2eP2VdHbGth3xz6szba3tnVyrPJl1dKQ/5xVLno7WV69ekVyzz7SDk9noKaOSGRC3Yc2GaP339zyQXLP+xfiaEEIYOXJyMhs3bnq0PnziiOSagUMHJrPDpu0TrW/eti25ZsF9C5PZmjXPJrOKiqpovX7w4OQagD1t1D7pe6FB/fols7t+cWcya2nZGK1PmnTALp8L2Ds0jG5IZq+ftW8yW7tpU7R+281zkmu2Nm9JZqnXyO3t6deejWPSM4GGkenX8KmXs4/PfzC5ZvFT6ay9vS2ZTT84/TUE9pxiZUW0PnHWhOSaIbW1yez7P/pVMmtqWhWt77ffMck1vUVNdfw1cgghDK2rS2ZtbTui9Y0b16bX7Ej3zlAV/+/VEzwpDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGTDUBwAAAAAgGwUe/oAMUueWZnM1mzclMxaNrRE68WK8uSahtENyayiqjKZte5ojdZXPpU+e2dqBtQks+Z16c+5vb0jWt/asjW5ZvzMccmsqn91Mnvwzjuj9UWLHkyuGTRoeDIbPWVUMgPitm2Of28vfWxZck1Hok+EEMKYqWOTWaFQiNb71fdLrpl22LRkNqh/fN0fFy5Mrpl/1/xktnnz+mQ2atQ+0Xrt4NrkGoDuUl4Rv+WePXVycs2GLfH72hBC+NOdv0tmlZXxe7lh49L3ZEDfVdWvKpkddMSsZFZfk379edMDc6P19S+uT64pK4vfN4YQwqZ1zdH65vWbk2s6+7xGTh6ZzOqG1JW83+OX35fM5s79QzIbPWl8tF47aEByDdD1ho8fFq0fM3u/5JrbFjyazK79wWXJrKFhdLS+z8wDk2v2pEJZ+tnn6aPSvbNYnp6bPv/8kmi9tXV7ck1HR3oG0Zt4UhwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZKPb0AWKWPrY0mVVWVSSzHdt3lHytlo1b0teqaUtm1f2ro/WKyt37kpaVd+3PJ6pqqpLZYUcdsFt7XrF5Q7Te2ro9fY6qfrt1LaA07a3pftWZYkV5yWuGjRuWzI7bd2Yye7ZpXbQ+54Y5yTUPP3RbMqusrElm++57VLSe6t0A3aluSF20vs/w4ck181csT2bPPDM/mfXvPzBa1/+gbyuviL/OHL/vhOSaWWPGJLMHn3kmmS1dEM/aOrnfbG5Ov65e8vgT0frWLZuSa8ZPSt9TVvVLv9Y94jX7R+ujp45Ornno3ruS2RNP3JfMnnp0QbQ+eea+yTW1gwYks0KhkMwgd9UD0q/9Dpg9PVqvrU7f+1zz9auT2ZJO7rOOP/70aH3QsEHJNXvS4OGDk9mkoenX8U8+/3wyW7VqSbReWZn++paV9Y1+5klxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBvxt7DuYds2b92tbHe0bNyczLZ08g7aqWx33zF6ayfX6kx5sTxan354/N13Qwhh9oT0O5Rf9fs7ktnKlYui9eHDxifXjBw3NpkBvduQkQ3R+tQZ6R6ybceOZHbfvIXR+gN33J1cs3btc8ls8uSDktmYKeOi9b7yLtjA3mXo2KHRek1lZXLNAw8/nsy2b0/fD48ZMyJa1/+gb6uqqYrWX3fwfsk1xfL4a8UQQpjzp0eS2drnm6L15U8sS65pbl6XzDZuXBOtDxgwMLmm/8D+yax1e2sya6itjdbHNgxJrnn4zScls85e3z/xxP3Remtr+n74gCOPSGaVVRXJDHI3ep/RyezAcfHXfjf8+cHkmnvuuTaZDRo0LJmNmTo+Wt/T91nlFfFx7vRDpibXdPZ3wrXX3ZbMNm/eEK2PH79v+lqVfaOfeVIcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2Sj29AF6s4729mS2bfPWLr3W9q3bd2vd4OGDo/U3HH5Qcs2GlpZk9p8XX5vMNm5cE60fcMDrk2vqGuqSGdDzyorlyWz2a2ZF67PGjEmuueWBh5PZbVfeFq0vWvRAcs3o0VOT2bR9Zyezqn5VyQygO1RWVyazSRNHR+tNzc3JNXdcfUcyq6hI97gpUw5OZkDfVVFdEa1PGjYsuWbNpk3JbMUTK5LZ0wsWReuLFv05uWbIkFHJbOLE+D3lgEG1yTX96/sns5ZN6dez85Yui9b/bnb69fERbzosmW1qSn8NH398TrT+2GP3JNdMPyB9jsqq+H9jyEahkIz2nT4xmdXW1ETrv/vBTck169a9kMxe97p3J7OqTu739qTBI+KzwCMmT06ueWrVqmR2w4+vTmbl5fHR8YwZR6bXFPvGM9h945QAAAAAANAFDMUBAAAAAMiGoTgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBR7+gC8sqp+Vcls+iFTo/Xh9XXJNZdc+7tk9vDDtyazxsax0froCROTa8rKCskM6HkjJgxPZodMjH9vr9qwPrnm4dseTmZz5vxntN7W1ppcM2Pfw5PZ4BGDkxnAnjZkVEMy23f06Gj9/qefTq6ZM+c3yay+vjGZjZkyPpkBe5+K8vJk1tHRkcwWz38ymS1YcEe0XleX7nPjxk1PZiMmjYzWy4u794xee2tbMlu2cFm03jxzRnLNgePHJ7PHpoxKZgMHDovWn332ieSapuebklntoAHJDHJQUVmRzKaOjPeREEJY29wcrd9793XJNTU16e+3STPS/ay3GDs9Pp8bVl+fXHPDPX9OZo89dk8yq62Nv+4ePTl+hhBCKBT6xizQk+IAAAAAAGTDUBwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANoo9fQBeWePoxmR29NSp0frjK59LrvnFdy5PZi1bNiazo446NVqvG1KXXAP0vKr+1cnsiCP2T2blZfGfm/7hrgeTax6Zc38y27BhdbQ+c+ZRyTWNY9L9r6ysb7yjNZCHkZNHJrNh9fXR+n23z02uef75Jclsxowjk1nNgJpkBvRdA+r7l7xm3ebNyeyhh25JZmvXrozWjz42/nowhBCGjBySzMqLvftZvN29payrS3/OKRs3rukkHbt7B4G9RFl5+puxf1VVMntq1apofd26F5Jr6uvTrzP71fZLZntSIfF6PIQQZowZXfJ+D9/2cDLbsmVTMhs7dnq0Xt3JnKGv6N1/OwEAAAAAQBcyFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyUezpA/AXhUIyGj99XDIbUlsbrX//J9cn1zzyyB3JbNSoKclswvTJ0XpFpT9G0NMKZemfcY6fOT6ZzRozJpnNW7YsWn/o9w8l1zz22D3JbODAodH6tFmzk2v0F6A3KSuWJ7MpI4cns63bt0fr992U7pllnfX18ft2si59Twn0XRNmjC95zcPPLE1mq1cvL3m/YePi93LdobN728rqimQ2Zlr83raymL6nnLN4cTLbuHZjOtuwOlovFNJnr69vTGaQu7bW9mTW1NyczGqrq6P1SZMOTK5Zt+6FZLZhzYZkNnj4oGSWUqxM96xiJ693yzu576xK9LQnn38+ueaJh+cls84MHjxyt9b1BZ4UBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDaKPX0A/kt9Y30y23/s2GT29AsvROs3X3Vdck2hUEhmM2celcyq+1cnM6BnDRo2KJkdvv/0ZLZu8+Zkds9dc6P1RQsfTq7ZujW937Rph0frnZ0doDcZ2DgwmU1sHJrM5i1fHq/Puy25pr6+MZmNmpS+NwT2TvuNGVPymmeffLYbTrJnDBk5OJk1jk73x9ENQ6L1+SvifTiEEB67//FktuKJFclsw8Y10frw4ROSawaPSH9ekLvWHa3J7LFn0/3soAnx77k3vftdyTX3/i59D9ayqSWZDR5e+mvX8mL6eeTK6spkVqxIj2wfenpJtN68rjm55tmVi5JZZwYPHr5b6/oCT4oDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG8WePkBOqgfUJLOZh01PZvU16XXf/8n10fqjj96VXDN69NRkNmrK6GQG9F4zDp2WzKaOGJHMfv3nB5LZvf8Z7yOLF89NrpkwYb9kNnn6rGi9rKyQXAPQm4zaZ1QyGzxgQDKbc/3vo/XVq1ck10yadGAy61ebvjcE9k61nbwmTNnSvKUbTtJ1GkY3JLNps9OvWQf375/MVqxeE68/vjy5Znkn2fx5dyazYrEyWt9//+OSa6r7VyczyF5HRzJa+MhTyaysLP68735Hp1+bFgrp16DrX1iXzHbHuhfWJ7NiZXosO3H/icls2+at0frqFauTa158cVky68ywMel5Ql/nSXEAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG+m3OWW3lBXLk9mEfccns9dOm5bM7nryyWT26x9eEa13dPKuvTNmHJnMqqrj76AN9G6vnbpPMqupTH9fL3og3V/mz78zWt+4cU1yzetff1oyGzRsUDJLaW9P97KOTrLyYvxnvoXEO5OH0Hnf7OzdyWsHDYjWOzt787pNu3WODas3lrwG6DqTRo9IZuWd9JdH73ksWu/oaE+uGTIkfS2AV6uqql/JawZ2ci9XXp5+HbzuxXXResOoxuSafp3cvy5etjKZrVqyKlpven5tcs3TCx9PZkufWZDMpk0/PFqfMH1Sck2xIv11AtJeWPZiMtvSvDVar2+sT64ZMmJwMuvstdW2zfFrdWbz+uaS14QQwrBxw5LZwZMmRut/6OR827a17NY5qvpX79a6vsCT4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMhGsacP0GcVCtHyiAnDk0uOPmhWMntx48Zk9p//cUMyW7TowWh90qQDk2tGTh6VzIC+aciA2t1at2ldczIbMGBgtD5mzPTkmobRDcmsrCzeNzvTvD59vrYdbcls4ND6aL2+oW639tvSvCWZ1Q2J79namt6ved2mZNaZ5U8vjtbXr39xt/YD4iqrK6P1KcPT93kt27Yls4WPPFDyGYYMcb8G/I+m5vg90fD6+D1PCCEMGDggmU2efFDJZxg8fFAyq6iK980QQti4Nv5ad83K1ck161evT2brVjUlsx3bW6P1zu55Fyy4I5kVK9Kf174HHBmtV/WrSq4Bdk/r9h3JrOn5tdH6pqb0a67yivJOrhXvI7urfye9uNN1NdXJbNrIkdH6g8MHJ9eUl1d0crWtyST19ahK3C/3JZ4UBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDaKPX2AvqqisiJaf81rDkyuGTNkcDK78P9dmcxu//0vk9ngwcOj9ekzD0uuqaquTGZA37Rp69ZkNrBfv2RW3b86mR1w0Oui9ba2tuSaiqqu7S/bNqc/r21btyezgUPro/X+df2Ta7Zv3ZbMmtc3J7ONazdG65s3tiTXbFi9IZlV19Yks+3b41+PVauWJNcApRs6fli03lBbm1wzb9myZLZixRPRekVFVXJN/eD0fSOQn4efWRqtTx0xIrlm4szxyWz24ccls7bW+L3e2uebkmuKxfRoIXXvuGbl2vSaHen7zR3b0veAqfuyF59fmVxTLKbvX/fZZ2YyaxzdkMyAntdZr9iRfunX5fp18vquM9tbW0teU6woT2Y7dvOTrqiKzz/3Bp4UBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDaKPX2A3qy8Iv3lGTNtdLS+35h4PYQQ7nz8iWT2259dlcxaWjYks4MPPilaHzZuWHINsPe5b/HiZHbAuHHJbOjYoclsy6aWks9RrKxIZv1qa6L1Hdtbk2v6DxyQzNrWbExmGxNZ9YC1yTVbm7cksw2r09favnV7tN68rjm5ZvP6dFZWLE9mw4fH/1u2taW/hkBCoZCMpsyYEK0Xy9Pfn7ff+WAya2p6PlpvaEjfN9YNqUtmQH6emf9MtP7UxHi/CiGECY2NyWzfo/ZNZhvWxF9/rlu1LrmmrCzdU5M66cNNz6fv2V58/rlktmljfF2xoiq55pBD4q+pQwhhYMOQZAbQnco66ZEp7W0dyay1Nf669b+kr1VZlX6N39d5UhwAAAAAgGwYigMAAAAAkA1DcQAAAAAAsmEoDgAAAABANgzFAQAAAADIRrGnD9CbDR4xOJkdc/iB0frqjZuSa376f3+RzJ588s/JbNasY5LZhOlTovViRXlyDbD3efCe+cmsaVNzMitWdvLXQOrdrjvS72jd2X51Q+qi9S3NW5NrttS2JLPmdel+u3Htxmi9ql9Ver/1m5PZ1uYtyayypjJarxlQk1xTXp7+mXRqvxBCaBzTGK3XNdYn1wBxZZ18H04aOjRa39CS7kn3Xnd3MisU4teaPPmg5Jrq/ul+BeRn3YvrovXbH3gkuWbwsEHJbOTkkcls4LCB0fqqJauSa9Y+tyaZrVm5Nlrv7F5p7Yurk9mKFU8ks2FDx0XroydMTq4ZOHRgMisvel0N9IwhtQOSWVt7e7S+4skVyTWp+9EQQhg4MH7vu7fzpDgAAAAAANkwFAcAAAAAIBuG4gAAAAAAZMNQHAAAAACAbBiKAwAAAACQDUNxAAAAAACyUezpA/S0qn5VyWzG7KnJbHxDQ7T+27kPJ9fcd++NyWzbtpZkNnbs9GRWM6A6mQH5WPvcmmS2bcu2ZFYodLJpR0fJ52jd3prMNq7dGK3v6GRNZ/rV9U9mVTXx3l5ekf5rr19dv2RWWV2RzIqV8T0rq9Jrurp3d/Z3GRDX0Z7ucSvXrYvWl61J99qmdc8ns8mTD4rWJ02bmVxT6LRBA7lpb22L1pcueCa55oVlLySzxtGNyayuoS6+Zkx6TWf3omtWrYrWBzcOTZ9h4KBkNmrUlGQ2Ysy4aL1h1JDkGoDeaGQnfXDJiy9G6/ded3dyzdChY5NZ6l51b+dJcQAAAAAAsmEoDgAAAABANgzFAQAAAADIhqE4AAAAAADZMBQHAAAAACAbhuIAAAAAAGSj2NMH6GkNoxqS2ZFTJiez59ati9Yfv+/x5JrmTfE1IYRQXp7+T1HdvyqZAbyS5nWb9ti1WrfvSGYb16az3TFgYP8u3a+isrO/Emu69FpAz+tob09mD/5pfrTe1ppeM37StPTFCoVoua6hLr0GYBds37p9t7Ktm7cmswGrBryqM/21huHDo/WaAen7q8qaymQ2aNigZFZRVbHrBwPoxZ5ZvTqZLV++Klpf2/Rccs3++x+XzEZPGr/L59qbeFIcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2Sj29AF6WsPoxmQ2tK4+mf3pqcXR+oK75yfXtHe0JbOxY2cks6qaqmQGAEDXWr1idclrGsek7ykBepttm7fuVpZSKBSSWePohpL361RVRdfuB9ALPXTvgmTWvH5TtD5hSnq2WDtoQDKr7l+96wfbi3hSHAAAAACAbBiKAwAAAACQDUNxAAAAAACyYSgOAAAAAEA2DMUBAAAAAMhGsacP0NO2tWxLZgtXrkxmK5bEsxdWPZNcM2zYhGQ2Y8bhyaxfff9kBgAAAADsPdY+t6bkNY2jG7rhJHsvT4oDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG4WOjo6Onj4EAAAAAADsCZ4UBwAAAAAgG4biAAAAAABkw1AcAAAAAIBsGIoDAAAAAJANQ3EAAAAAALJhKA4AAAAAQDYMxQEAAAAAyIahOAAAAAAA2TAUBwAAAAAgG/8fdchmNyYFMOsAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C7vOeo1a4O9W"
      },
      "source": [
        "Next, we'll define a `DataLoader` for each of the training and test sets. We can iterate over these, and they will yield batches of images and labels which we can use to train our model.\n",
        "\n",
        "We only need to shuffle our training set as it will be used for stochastic gradient descent, and we want each batch to be different between epochs.\n",
        "\n",
        "Ideally, we want to use the biggest batch size that we can. The 64 here is relatively small and can be increased if our hardware can handle it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sGdKSORpREvr"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 64\n",
        "\n",
        "train_loader = data.DataLoader(train_data,\n",
        "                                 shuffle=True,\n",
        "                                 batch_size=BATCH_SIZE)\n",
        "test_loader = data.DataLoader(test_data,\n",
        "                                batch_size=BATCH_SIZE)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "## Define the Model"
      ],
      "metadata": {
        "id": "Es1Cm2gzaWtq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we've gone through all the concepts, we can implement our model.\n",
        "\n",
        "![](https://github.com/bentrevett/pytorch-image-classification/blob/master/assets/lenet5.png?raw=1)\n",
        "\n",
        "Our actual implementation is going to slightly differ from the actual LeNet architecture, as it was built to handle 32x32 images, whereas the MNIST dataset consists of 28x28 images. We're also not going to use the Gaussian connections at the end, instead we'll just use a standard linear layer.\n",
        "\n",
        "The first layer in our model is a convolutional layer with 6 filters (PyTorch calls them `out_channels`) and a kernel size of 5. This turns our `[1, 28, 28]` image into `[6, 24, 24]`. We then downsample our image with a max pooling layer that has a filter size of 2 to get a `[6, 12, 12]` image. This is then passed through an activation function, ReLU in this case, which is applied elementwise and does not change the of the image.\n",
        "\n",
        "Afterwards, we pass the image to the second convolutional layer with 16 filters that are 5x5x6, a height and width of 5 and a depth of 6 as our previous convolutional layer had 6 filters. This gives us an image size of `[16, 8, 8]` which we then max pool to half the height and width to `[16, 4, 4]` and then pass through another ReLU function.\n",
        "\n",
        "We then flatten our `[16, 4, 4]` image to `[256]` and pass this through three linear layers. Each of the linear layers are followed by another ReLU, except for the last.\n",
        "\n",
        "We return the results from the final linear layer as well as from the flattened result of the second convolutional layer, which we can plot in lower dimensions later."
      ],
      "metadata": {
        "id": "p8Hjd-7LSmgj"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lAqzcW9XREvu"
      },
      "outputs": [],
      "source": [
        "OUTPUT_DIM = 10\n",
        "\n",
        "model = nn.Sequential(\n",
        "\n",
        "            # x = [batch size, 1, 28, 28]\n",
        "\n",
        "            nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5),\n",
        "            # x = [batch size, 6, 24, 24]\n",
        "            nn.MaxPool2d(kernel_size=2),\n",
        "            # x = [batch size, 6, 12, 12]\n",
        "            nn.ReLU(),\n",
        "\n",
        "            nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5),\n",
        "            # x = [batch size, 16, 8, 8]\n",
        "            nn.MaxPool2d(kernel_size=2),\n",
        "            # x = [batch size, 16, 4, 4]\n",
        "            nn.ReLU(),\n",
        "\n",
        "            nn.Flatten(),\n",
        "            # x = [batch size, 16*4*4 = 256]\n",
        "\n",
        "            nn.Linear(16 * 4 * 4, 120),\n",
        "            # x = [batch size, 120]\n",
        "            nn.ReLU(),\n",
        "\n",
        "            nn.Linear(120, 84),\n",
        "            # x = [batch size, 84]\n",
        "            nn.ReLU(),\n",
        "\n",
        "            nn.Linear(84, OUTPUT_DIM),\n",
        "            # x = [batch size, output dim]\n",
        "\n",
        "        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3bacWHd4O9j"
      },
      "source": [
        "Let's calculate the number of trainable parameters (weights and biases) in our model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1gF2fmTBREv8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4350b064-18af-4547-f2cf-36b3fc6355ff"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The model has 44,426 trainable parameters\n"
          ]
        }
      ],
      "source": [
        "# @title Number of parameters\n",
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "print(f'The model has {count_parameters(model):,} trainable parameters')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "---\n",
        "## Train and Test the Model"
      ],
      "metadata": {
        "id": "z_gSytVladoE"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IiL9hZqX4O9x"
      },
      "source": [
        "- pass a batch of data through your model\n",
        "- calculate the loss of your batch by comparing your model's predictions against the actual labels\n",
        "- calculate the gradient of each of your parameters with respect to the loss\n",
        "- update each of your parameters by subtracting their gradient multiplied by a small *learning rate* parameter\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eC2P8WJuREv_"
      },
      "outputs": [],
      "source": [
        "optimizer = optim.Adam(model.parameters())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gBvdk2RR4O95"
      },
      "source": [
        "We use as our loss `CrossEntropyLoss`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OO2BL-qBREwD"
      },
      "outputs": [],
      "source": [
        "loss_fn = nn.CrossEntropyLoss()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7QgyFD6gREwH",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "# @title Use GPU if available\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "model = model.to(device)\n",
        "loss_fn = loss_fn.to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bd0F_7i_4O-M"
      },
      "source": [
        "Next, we'll define a function to calculate the accuracy of our model. This takes the index of the highest value for your prediction and compares it against the actual class label. We then divide how many our model got correct by the amount in the batch to calculate accuracy across the batch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VkttGrmkREwP",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "# @title Function to calculate accuracy\n",
        "def calculate_accuracy(y_pred, y):\n",
        "    top_pred = y_pred.argmax(1, keepdim=True)\n",
        "    correct = top_pred.eq(y.view_as(top_pred)).sum()\n",
        "    acc = correct.float() / y.shape[0]\n",
        "    return acc"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0kCtKBMfREwS"
      },
      "outputs": [],
      "source": [
        "def train(model, loader, optimizer, loss_fn, device):\n",
        "\n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "\n",
        "    model.train()\n",
        "\n",
        "    for (x, y) in loader:\n",
        "\n",
        "        x = x.to(device)\n",
        "        y = y.to(device)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        y_pred = model(x)\n",
        "\n",
        "        loss = loss_fn(y_pred, y)\n",
        "\n",
        "        acc = calculate_accuracy(y_pred, y)\n",
        "\n",
        "        loss.backward()\n",
        "\n",
        "        optimizer.step()\n",
        "\n",
        "        epoch_loss += loss.item()\n",
        "        epoch_acc += acc.item()\n",
        "\n",
        "    return epoch_loss / len(loader), epoch_acc / len(loader)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dySDirym4O-c"
      },
      "source": [
        "The final step before training is to define a small function to tell us how long an epoch took."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ENrsvnEZREwZ"
      },
      "outputs": [],
      "source": [
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHn-rDvA4O-h"
      },
      "source": [
        "We're finally ready to train!\n",
        "\n",
        "During each epoch we calculate the training loss and accuracy, followed by the validation loss and accuracy. We then check if the validation loss achieved is the best validation loss we have seen. If so, we save our model's parameters (called a `state_dict`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p-gtfzafREwc",
        "outputId": "e7c34252-ff6f-4afe-8c67-138aa2c96911"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.377 | Train Acc: 88.16%\n",
            "Epoch: 02 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.123 | Train Acc: 96.21%\n",
            "Epoch: 03 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.091 | Train Acc: 97.17%\n",
            "Epoch: 04 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.078 | Train Acc: 97.52%\n",
            "Epoch: 05 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.069 | Train Acc: 97.82%\n",
            "Epoch: 06 | Epoch Time: 0m 27s\n",
            "\tTrain Loss: 0.062 | Train Acc: 98.03%\n",
            "Epoch: 07 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.058 | Train Acc: 98.19%\n",
            "Epoch: 08 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.053 | Train Acc: 98.38%\n",
            "Epoch: 09 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.049 | Train Acc: 98.44%\n",
            "Epoch: 10 | Epoch Time: 0m 28s\n",
            "\tTrain Loss: 0.047 | Train Acc: 98.52%\n"
          ]
        }
      ],
      "source": [
        "EPOCHS = 10\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "\n",
        "    start_time = time.monotonic()\n",
        "\n",
        "    train_loss, train_acc = train(model, train_loader, optimizer, loss_fn, device)\n",
        "\n",
        "    end_time = time.monotonic()\n",
        "\n",
        "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "\n",
        "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "test_idx = random.randint(0,10000-1) # out of 10000\n",
        "\n",
        "model.eval()\n",
        "\n",
        "with torch.no_grad():\n",
        "  test_pred = model(test_data[test_idx][0].unsqueeze_(0).to(device))\n",
        "  prob = torch.softmax(test_pred, dim=-1)\n",
        "  print(torch.argmax(prob, 1))\n",
        "\n",
        "plt.imshow(test_data[test_idx][0].view(28, 28).cpu().numpy(), cmap='bone')\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 466
        },
        "id": "w8UEQ1X9onpr",
        "outputId": "642b553d-24ed-4dd3-8358-5f0cbae6078d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([8], device='cuda:0')\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.image.AxesImage at 0x7f95cfeef9d0>"
            ]
          },
          "metadata": {},
          "execution_count": 30
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAawklEQVR4nO3df2zUdb7v8dfwoyNoO1hKOx0p2KKCR6B7RKg9KotLT39sLgvCycUfuYE9rFzcYha7rt5uFMTdpLtsjutxw2JyYuiaK+iSCERyFo9UW6IWPFRYwnG3hzZdQWnLyl5mSoFC6Of+QRwdKeJ3mPbdmT4fyTehM99P5+3Xb/rk206/+JxzTgAADLBh1gMAAIYmAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEyMsB7gq3p7e3Xs2DGlp6fL5/NZjwMA8Mg5p66uLoVCIQ0bdvnrnEEXoGPHjikvL896DADAVTp69KjGjx9/2ecH3bfg0tPTrUcAACTAlb6e91uA1q9frxtvvFHXXHONioqK9MEHH3yjdXzbDQBSw5W+nvdLgF577TVVVVVpzZo1+vDDD1VYWKiysjIdP368P14OAJCMXD+YNWuWq6ysjH584cIFFwqFXE1NzRXXhsNhJ4mNjY2NLcm3cDj8tV/vE34FdO7cOTU1NamkpCT62LBhw1RSUqLGxsZL9u/p6VEkEonZAACpL+EB+uyzz3ThwgXl5OTEPJ6Tk6OOjo5L9q+pqVEgEIhuvAMOAIYG83fBVVdXKxwOR7ejR49ajwQAGAAJ/z2grKwsDR8+XJ2dnTGPd3Z2KhgMXrK/3++X3+9P9BgAgEEu4VdAaWlpmjFjhurq6qKP9fb2qq6uTsXFxYl+OQBAkuqXOyFUVVVpyZIluuOOOzRr1iw9//zz6u7u1ve///3+eDkAQBLqlwAtXrxYf/3rX7V69Wp1dHToW9/6lnbu3HnJGxMAAEOXzznnrIf4skgkokAgYD0GAOAqhcNhZWRkXPZ583fBAQCGJgIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYCLhAXrmmWfk8/litilTpiT6ZQAASW5Ef3zS2267Tbt27friRUb0y8sAAJJYv5RhxIgRCgaD/fGpAQApol9+BnT48GGFQiEVFBTooYce0pEjRy67b09PjyKRSMwGAEh9CQ9QUVGRamtrtXPnTm3YsEFtbW2655571NXV1ef+NTU1CgQC0S0vLy/RIwEABiGfc8715wucPHlSEydO1HPPPadly5Zd8nxPT496enqiH0ciESIEACkgHA4rIyPjss/3+7sDxowZo1tuuUUtLS19Pu/3++X3+/t7DADAINPvvwd06tQptba2Kjc3t79fCgCQRBIeoMcff1wNDQ36y1/+ovfff1/33Xefhg8frgceeCDRLwUASGIJ/xbcJ598ogceeEAnTpzQuHHjdPfdd2vPnj0aN25col8KAJDE+v1NCF5FIhEFAgHrMfANzJnzoOc1/2PZQs9rHv9f/+R5DZLDH/74R89r/m3t7zyvef315zyvwdW70psQuBccAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCm5EibtubmjyvqSgs9LzmW9Nme17z0Z/e97wGA++l/6jzvMb1ev+S9YPyEs9rcPW4GSkAYFAiQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACAiRHWAyB5/VdTs+c137v9ds9rxo+f7HkNd8NODukZ13peMyFrbD9MAgtcAQEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJrgZKeLWsr9lQF6n8hcrPa9p3LM9rtfq6vpbXOsg/f3f/6PnNQvuuMPzml2HDnleg8GJKyAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQ3I0XcJhUWDMjr3HD99Z7X5OdPj+u1Dh6sj2sdpNGj0z2vGTl8uOc1f9hc53kNBieugAAAJggQAMCE5wDt3r1b8+bNUygUks/n07Zt22Ked85p9erVys3N1ahRo1RSUqLDhw8nal4AQIrwHKDu7m4VFhZq/fr1fT6/bt06vfDCC3rxxRe1d+9eXXvttSorK9PZs2evelgAQOrw/CaEiooKVVRU9Pmcc07PP/+8nnrqKc2fP1+S9PLLLysnJ0fbtm3T/ffff3XTAgBSRkJ/BtTW1qaOjg6VlJREHwsEAioqKlJjY2Ofa3p6ehSJRGI2AEDqS2iAOjo6JEk5OTkxj+fk5ESf+6qamhoFAoHolpeXl8iRAACDlPm74KqrqxUOh6Pb0aNHrUcCAAyAhAYoGAxKkjo7O2Me7+zsjD73VX6/XxkZGTEbACD1JTRA+fn5CgaDqqv74jeVI5GI9u7dq+Li4kS+FAAgyXl+F9ypU6fU0tIS/bitrU0HDhxQZmamJkyYoFWrVunnP/+5br75ZuXn5+vpp59WKBTSggULEjk3ACDJeQ7Qvn37dO+990Y/rqqqkiQtWbJEtbW1euKJJ9Td3a3ly5fr5MmTuvvuu7Vz505dc801iZsaAJD0PAdozpw5cs5d9nmfz6dnn31Wzz777FUNhoFz553z41q34qH41nn1+1f+4HkNNxVNXZ8e/tR6BCSI+bvgAABDEwECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEx4vhs2Us+//t9/iWtd5nXXJXiSvh3YvW9AXgfJ4Yabb7AeAQnCFRAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIKbkWLQy76Bm0/iCx+++571CEgQroAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABM+55yzHuLLIpGIAoGA9RhDSjBYENe6un27Pa/5uzhuLNpz/rznNf/v9GnPaySpdssfPK85ceyE5zX/3dTsec2nn/635zXx+tvf2j2vefJf13le87/nlXlec+ONUz2v+fjj//K8BlcvHA4rIyPjss9zBQQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmBhhPQDsjRge32mQP25cgifpm3/kSM9rgnHe0Pb//OD+uNalmsiZM57XZIwa5XlN19mzntecO+d9DQYnroAAACYIEADAhOcA7d69W/PmzVMoFJLP59O2bdtinl+6dKl8Pl/MVl5enqh5AQApwnOAuru7VVhYqPXr1192n/LycrW3t0e3zZs3X9WQAIDU4/mnzxUVFaqoqPjaffx+v4LBYNxDAQBSX7/8DKi+vl7Z2dmaPHmyHnnkEZ04cfl/srinp0eRSCRmAwCkvoQHqLy8XC+//LLq6ur0y1/+Ug0NDaqoqNCFCxf63L+mpkaBQCC65eXlJXokAMAglPDfA7r//i9+j2LatGmaPn26Jk2apPr6es2dO/eS/aurq1VVVRX9OBKJECEAGAL6/W3YBQUFysrKUktLS5/P+/1+ZWRkxGwAgNTX7wH65JNPdOLECeXm5vb3SwEAkojnb8GdOnUq5mqmra1NBw4cUGZmpjIzM7V27VotWrRIwWBQra2teuKJJ3TTTTeprKwsoYMDAJKb5wDt27dP9957b/Tjz39+s2TJEm3YsEEHDx7U7373O508eVKhUEilpaX62c9+Jr/fn7ipAQBJz3OA5syZI+fcZZ9/8803r2ogDLzvPfCDuNaNSkvzvKb3a86dy3n9P//T85od//bvntdIUuPu+NYNhLQ07zf7rPifi+N6rYLCAs9rVnzP+x1P3m1u9rymvb3V8xoMTtwLDgBgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYS/k9yI/nceuetca3rCIc9r3nuhVc8r/nV6krPa3DRoUO741pXWvrPntfEczdsDG1cAQEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJrgZKfToP82Lb12C5wAwtHAFBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCY4GakAMy8v2uf9QgwxBUQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCm5ECMPPH+v3WI8AQV0AAABMECABgwlOAampqNHPmTKWnpys7O1sLFixQc3NzzD5nz55VZWWlxo4dq+uuu06LFi1SZ2dnQocGACQ/TwFqaGhQZWWl9uzZo7feekvnz59XaWmpuru7o/s89thjeuONN7RlyxY1NDTo2LFjWrhwYcIHBwAkN09vQti5c2fMx7W1tcrOzlZTU5Nmz56tcDisl156SZs2bdJ3vvMdSdLGjRt16623as+ePbrzzjsTNzkAIKld1c+AwuGwJCkzM1OS1NTUpPPnz6ukpCS6z5QpUzRhwgQ1Njb2+Tl6enoUiURiNgBA6os7QL29vVq1apXuuusuTZ06VZLU0dGhtLQ0jRkzJmbfnJwcdXR09Pl5ampqFAgEolteXl68IwEAkkjcAaqsrNShQ4f06quvXtUA1dXVCofD0e3o0aNX9fkAAMkhrl9EXblypXbs2KHdu3dr/Pjx0ceDwaDOnTunkydPxlwFdXZ2KhgM9vm5/H6//H5/PGMAAJKYpysg55xWrlyprVu36u2331Z+fn7M8zNmzNDIkSNVV1cXfay5uVlHjhxRcXFxYiYGAKQET1dAlZWV2rRpk7Zv36709PToz3UCgYBGjRqlQCCgZcuWqaqqSpmZmcrIyNCjjz6q4uJi3gEHAIjhKUAbNmyQJM2ZMyfm8Y0bN2rp0qWSpF//+tcaNmyYFi1apJ6eHpWVlem3v/1tQoYFAKQOn3POWQ/xZZFIRIFAwHoMYEgrLf1nz2vefPMlz2uysm7wvObEiWOe18BGOBxWRkbGZZ/nXnAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwEde/iAogtf3dzGnWI2AI4AoIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADDBzUgBXOLmO27xvOajTz/1vObMmVOe1yB1cAUEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJjgZqRACvP54vs75s03TfC8Zszo0Z7XjBg+0vMapA6ugAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAE9yMFEhh//APC+Ja949Tp3pe88q773leE+k64XkNUgdXQAAAEwQIAGDCU4Bqamo0c+ZMpaenKzs7WwsWLFBzc3PMPnPmzJHP54vZVqxYkdChAQDJz1OAGhoaVFlZqT179uitt97S+fPnVVpaqu7u7pj9Hn74YbW3t0e3devWJXRoAEDy8/QmhJ07d8Z8XFtbq+zsbDU1NWn27NnRx0ePHq1gMJiYCQEAKemqfgYUDoclSZmZmTGPv/LKK8rKytLUqVNVXV2t06dPX/Zz9PT0KBKJxGwAgNQX99uwe3t7tWrVKt11112a+qW3bD744IOaOHGiQqGQDh48qCeffFLNzc16/fXX+/w8NTU1Wrt2bbxjAACSVNwBqqys1KFDh/Tuu+/GPL58+fLon6dNm6bc3FzNnTtXra2tmjRp0iWfp7q6WlVVVdGPI5GI8vLy4h0LAJAk4grQypUrtWPHDu3evVvjx4//2n2LiookSS0tLX0GyO/3y+/3xzMGACCJeQqQc06PPvqotm7dqvr6euXn519xzYEDByRJubm5cQ0IAEhNngJUWVmpTZs2afv27UpPT1dHR4ckKRAIaNSoUWptbdWmTZv03e9+V2PHjtXBgwf12GOPafbs2Zo+fXq//AcAAJKTpwBt2LBB0sVfNv2yjRs3aunSpUpLS9OuXbv0/PPPq7u7W3l5eVq0aJGeeuqphA0MAEgNnr8F93Xy8vLU0NBwVQMBAIYGn7tSVQZYJBJRIBCwHgMAcJXC4bAyMjIu+zw3IwUAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMDEoAuQc856BABAAlzp6/mgC1BXV5f1CACABLjS13OfG2SXHL29vTp27JjS09Pl8/linotEIsrLy9PRo0eVkZFhNKE9jsNFHIeLOA4XcRwuGgzHwTmnrq4uhUIhDRt2+eucEQM40zcybNgwjR8//mv3ycjIGNIn2Oc4DhdxHC7iOFzEcbjI+jgEAoEr7jPovgUHABgaCBAAwERSBcjv92vNmjXy+/3Wo5jiOFzEcbiI43ARx+GiZDoOg+5NCACAoSGproAAAKmDAAEATBAgAIAJAgQAMJE0AVq/fr1uvPFGXXPNNSoqKtIHH3xgPdKAe+aZZ+Tz+WK2KVOmWI/V73bv3q158+YpFArJ5/Np27ZtMc8757R69Wrl5uZq1KhRKikp0eHDh22G7UdXOg5Lly695PwoLy+3Gbaf1NTUaObMmUpPT1d2drYWLFig5ubmmH3Onj2ryspKjR07Vtddd50WLVqkzs5Oo4n7xzc5DnPmzLnkfFixYoXRxH1LigC99tprqqqq0po1a/Thhx+qsLBQZWVlOn78uPVoA+62225Te3t7dHv33XetR+p33d3dKiws1Pr16/t8ft26dXrhhRf04osvau/evbr22mtVVlams2fPDvCk/etKx0GSysvLY86PzZs3D+CE/a+hoUGVlZXas2eP3nrrLZ0/f16lpaXq7u6O7vPYY4/pjTfe0JYtW9TQ0KBjx45p4cKFhlMn3jc5DpL08MMPx5wP69atM5r4MlwSmDVrlqusrIx+fOHCBRcKhVxNTY3hVANvzZo1rrCw0HoMU5Lc1q1box/39va6YDDofvWrX0UfO3nypPP7/W7z5s0GEw6Mrx4H55xbsmSJmz9/vsk8Vo4fP+4kuYaGBufcxf/3I0eOdFu2bInu86c//clJco2NjVZj9ruvHgfnnPv2t7/tfvSjH9kN9Q0M+iugc+fOqampSSUlJdHHhg0bppKSEjU2NhpOZuPw4cMKhUIqKCjQQw89pCNHjliPZKqtrU0dHR0x50cgEFBRUdGQPD/q6+uVnZ2tyZMn65FHHtGJEyesR+pX4XBYkpSZmSlJampq0vnz52POhylTpmjChAkpfT589Th87pVXXlFWVpamTp2q6upqnT592mK8yxp0NyP9qs8++0wXLlxQTk5OzOM5OTn685//bDSVjaKiItXW1mry5Mlqb2/X2rVrdc899+jQoUNKT0+3Hs9ER0eHJPV5fnz+3FBRXl6uhQsXKj8/X62trfrpT3+qiooKNTY2avjw4dbjJVxvb69WrVqlu+66S1OnTpV08XxIS0vTmDFjYvZN5fOhr+MgSQ8++KAmTpyoUCikgwcP6sknn1Rzc7Nef/11w2ljDfoA4QsVFRXRP0+fPl1FRUWaOHGifv/732vZsmWGk2EwuP/++6N/njZtmqZPn65Jkyapvr5ec+fONZysf1RWVurQoUND4uegX+dyx2H58uXRP0+bNk25ubmaO3euWltbNWnSpIEes0+D/ltwWVlZGj58+CXvYuns7FQwGDSaanAYM2aMbrnlFrW0tFiPYubzc4Dz41IFBQXKyspKyfNj5cqV2rFjh955552Yf74lGAzq3LlzOnnyZMz+qXo+XO449KWoqEiSBtX5MOgDlJaWphkzZqiuri76WG9vr+rq6lRcXGw4mb1Tp06ptbVVubm51qOYyc/PVzAYjDk/IpGI9u7dO+TPj08++UQnTpxIqfPDOaeVK1dq69atevvtt5Wfnx/z/IwZMzRy5MiY86G5uVlHjhxJqfPhSsehLwcOHJCkwXU+WL8L4pt49dVXnd/vd7W1te6jjz5yy5cvd2PGjHEdHR3Wow2oH//4x66+vt61tbW59957z5WUlLisrCx3/Phx69H6VVdXl9u/f7/bv3+/k+See+45t3//fvfxxx8755z7xS9+4caMGeO2b9/uDh486ObPn+/y8/PdmTNnjCdPrK87Dl1dXe7xxx93jY2Nrq2tze3atcvdfvvt7uabb3Znz561Hj1hHnnkERcIBFx9fb1rb2+PbqdPn47us2LFCjdhwgT39ttvu3379rni4mJXXFxsOHXiXek4tLS0uGeffdbt27fPtbW1ue3bt7uCggI3e/Zs48ljJUWAnHPuN7/5jZswYYJLS0tzs2bNcnv27LEeacAtXrzY5ebmurS0NHfDDTe4xYsXu5aWFuux+t0777zjJF2yLVmyxDl38a3YTz/9tMvJyXF+v9/NnTvXNTc32w7dD77uOJw+fdqVlpa6cePGuZEjR7qJEye6hx9+OOX+ktbXf78kt3Hjxug+Z86ccT/84Q/d9ddf70aPHu3uu+8+197ebjd0P7jScThy5IibPXu2y8zMdH6/3910003uJz/5iQuHw7aDfwX/HAMAwMSg/xkQACA1ESAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAm/j+srngj2lZj7gAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9kuF4-Cx4O-X"
      },
      "source": [
        "The evaluation loop is similar to the training loop. The differences are:\n",
        "- we put our model into evaluation mode with `model.eval()`\n",
        "- we wrap the iterations inside a `with torch.no_grad()`\n",
        "- we do not zero gradients as we are not calculating any\n",
        "- we do not calculate gradients as we are not updating parameters\n",
        "- we do not take an optimizer step as we are not calculating gradients\n",
        "\n",
        "`torch.no_grad()` ensures that gradients are not calculated for whatever is inside the `with` block. As our model will not have to calculate gradients, it will be faster and use less memory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3XL4sRI8REwV"
      },
      "outputs": [],
      "source": [
        "def evaluate(model, loader, loss_fn, device):\n",
        "\n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "\n",
        "    model.eval()\n",
        "\n",
        "    with torch.no_grad():\n",
        "\n",
        "        for (x, y) in loader:\n",
        "\n",
        "            x = x.to(device)\n",
        "            y = y.to(device)\n",
        "\n",
        "            y_pred = model(x)\n",
        "\n",
        "            loss = loss_fn(y_pred, y)\n",
        "\n",
        "            acc = calculate_accuracy(y_pred, y)\n",
        "\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_acc += acc.item()\n",
        "\n",
        "    return epoch_loss / len(loader), epoch_acc / len(loader)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VGDeNGLhREwf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a5f2dc9b-cde2-45f7-e560-3c68a6f49de4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test Loss: 0.030 | Test Acc: 99.23%\n"
          ]
        }
      ],
      "source": [
        "test_loss, test_acc = evaluate(model, test_loader, loss_fn, device)\n",
        "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "machine_shape": "hm",
      "provenance": [],
      "collapsed_sections": [
        "PACH3MN6Zx31",
        "rFweArLKaRgk",
        "Es1Cm2gzaWtq",
        "z_gSytVladoE"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}