Part3_CNNs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\newcommand{\\mat}[1]{\\boldsymbol {#1}}\n",
    "\\newcommand{\\mattr}[1]{\\boldsymbol {#1}^\\top}\n",
    "\\newcommand{\\matinv}[1]{\\boldsymbol {#1}^{-1}}\n",
    "\\newcommand{\\vec}[1]{\\boldsymbol {#1}}\n",
    "\\newcommand{\\vectr}[1]{\\boldsymbol {#1}^\\top}\n",
    "\\newcommand{\\rvar}[1]{\\mathrm {#1}}\n",
    "\\newcommand{\\rvec}[1]{\\boldsymbol{\\mathrm{#1}}}\n",
    "\\newcommand{\\diag}{\\mathop{\\mathrm {diag}}}\n",
    "\\newcommand{\\set}[1]{\\mathbb {#1}}\n",
    "\\newcommand{\\norm}[1]{\\left\\lVert#1\\right\\rVert}\n",
    "\\newcommand{\\pderiv}[2]{\\frac{\\partial #1}{\\partial #2}}\n",
    "\\newcommand{\\bb}[1]{\\boldsymbol{#1}}\n",
    "$$\n",
    "# Part 3: Convolutional Architectures\n",
    "<a id=part3></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part we will explore convolution networks and the effects of their architecture on accuracy. We'll implement a common block-based deep CNN pattern and we'll perform various experiments on it while varying the architecture. Then we'll implement our own custom architecture to see whether we can get high classification results on a large subset of CIFAR-10.\n",
    "\n",
    "Training will be performed on GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import sys\n",
    "import glob\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import unittest\n",
    "import torch\n",
    "import torchvision\n",
    "import torchvision.transforms as tvtf\n",
    "\n",
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "seed = 42\n",
    "#device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "device = torch.device('cpu')\n",
    "\n",
    "plt.rcParams.update({'font.size': 12})\n",
    "test = unittest.TestCase()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional layers and networks\n",
    "<a id=part3_1></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convolutional layers are the most essential building blocks of the state of the art deep learning image classification models and also play an important role in many other tasks.\n",
    "As we saw in the tutorial, convolutional layers operate on and produce volumes (3D tensors) of activations.\n",
    "\n",
    "\n",
    "One way to think about them is as if the neurons are organized in a 3D grid,\n",
    "where neurons at the same depth share weights (represented here as colors).\n",
    "Contrary to fully connected (affine) layers, neurons in convolutional layers are **not** connected to each of the activations of the previous layer.\n",
    "Instead, each neuron is connected only to a small region of the input volume e.g. a 5x5x$C_{\\mathrm{in}}$ slice\n",
    "(where $C_{\\mathrm{in}}$ is the input volume's depth).\n",
    "\n",
    "<img src=\"imgs/depthcol.jpeg\" />\n",
    "\n",
    "Another way to interpret convolutional layers is as a collection of 3D learnable filters,\n",
    "each of which operates on a small spatial region of the input volume.\n",
    "Each filter is convolved with the input volume (\"slides over it\"),\n",
    "and a dot product is computed at each location followed by a non-linearity which produces one activation.\n",
    "All these activations produce a 2D plane known as a **feature map**.\n",
    "Multiple feature maps (one for each filter) comprise the output volume.\n",
    "\n",
    "<img src=\"imgs/cnn_filters.png\" width=\"600\" />\n",
    "\n",
    "A crucial property of convolutional layers is their translation invariance,\n",
    "i.e. their ability to detect features regardelss of their spatial location in the input.\n",
    "\n",
    "Convolutional network architectures usually follow a pattern basic repeating blocks: one or more convolution layers, each followed by a non-linearity (generally ReLU) and then a pooling layer to reduce spatial dimensions. Usually, the number of convolutional filters increases the deeper they are in the network.\n",
    "These layers are meant to extract features from the input.\n",
    "Then, one or more fully-connected layers is used to combine the extracted features into the required number of output class scores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building convolutional networks with PyTorch\n",
    "<a id=part3_2></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch provides all the basic building blocks needed for creating a convolutional arcitecture within the [`torch.nn`](https://pytorch.org/docs/stable/nn.html) package.\n",
    "Let's use them to create a basic convolutional network with the following architecture pattern:\n",
    "\n",
    "    [(CONV -> ReLU)*P -> MaxPool]*(N/P) -> (Linear -> ReLU)*M -> Linear\n",
    "\n",
    "Here $N$ is the total number of convolutional layers,\n",
    "$P$ specifies how many convolutions to perform before each pooling layer\n",
    "and $M$ specifies the number of hidden fully-connected layers before the final output layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Complete the implementaion of the `ConvClassifier` class in the `hw2/models.py` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ConvClassifier(\n",
      "  (feature_extractor): Sequential(\n",
      "    (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (1): ReLU()\n",
      "    (2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (3): ReLU()\n",
      "    (4): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)\n",
      "    (5): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (6): ReLU()\n",
      "    (7): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (8): ReLU()\n",
      "    (9): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)\n",
      "  )\n",
      "  (classifier): Sequential(\n",
      "    (0): Linear(in_features=20000, out_features=100, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=100, out_features=100, bias=True)\n",
      "    (3): ReLU()\n",
      "    (4): Linear(in_features=100, out_features=10, bias=True)\n",
      "  )\n",
      ")\n",
      "out = tensor([[-0.0868, -0.3790, -0.4341, -0.1236, -0.2160,  0.1683,  0.4739,  0.0750,\n",
      "          0.1151, -0.1606]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "import hw2.models as models\n",
    "torch.manual_seed(seed)\n",
    "\n",
    "net = models.ConvClassifier((3,100,100), 10, filters=[32]*4, pool_every=2, hidden_dims=[100]*2).to(device)\n",
    "print(net)\n",
    "\n",
    "test_image = torch.randint(low=0, high=256, size=(3, 100, 100), dtype=torch.float).to(device)\n",
    "test_out = net(test_image.unsqueeze(0))\n",
    "print('out =', test_out)\n",
    "\n",
    "expected_out = torch.load('tests/assets/expected_conv_out.pt').to(device)\n",
    "test.assertLess(torch.norm(test_out - expected_out).item(), 1e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note about running on GPUs**.\n",
    "\n",
    "Notice how we called `.to(device)` on **both** the model and the input tensor.\n",
    "Here the `device` is a `torch.device` object that we created above. If an nvidia GPU is available on the machine you're running this on, the `device` will be `'cuda'`. When you run `.to(device)` on a model, it recursively goes over all the model parameter tensors and copies their memory to the GPU. Similarly, calling `.to(device)` on the input image also copies it.\n",
    "\n",
    "In order to train on a GPU, you need to make sure to move **all** your tensors to it. You'll get errors if you try to mix CPU and GPU tensors in a computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This notebook is running with device=cpu\n",
      "The model parameter tensors are therefore also on device=cpu\n",
      "The test image is therefore also on device=cpu\n"
     ]
    }
   ],
   "source": [
    "print(f'This notebook is running with device={device}')\n",
    "print(f'The model parameter tensors are therefore also on device={next(net.parameters()).device}')\n",
    "print(f'The test image is therefore also on device={test_image.device}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load CIFAR-10 again to use as our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n",
      "Train: 50000 samples\n",
      "Test: 10000 samples\n",
      "input image size = torch.Size([3, 32, 32])\n"
     ]
    }
   ],
   "source": [
    "data_dir = os.path.expanduser('~/.pytorch-datasets')\n",
    "ds_train = torchvision.datasets.CIFAR10(root=data_dir, download=True, train=True, transform=tvtf.ToTensor())\n",
    "ds_test = torchvision.datasets.CIFAR10(root=data_dir, download=True, train=False, transform=tvtf.ToTensor())\n",
    "\n",
    "print(f'Train: {len(ds_train)} samples')\n",
    "print(f'Test: {len(ds_test)} samples')\n",
    "\n",
    "x0,_ = ds_train[0]\n",
    "in_size = x0.shape\n",
    "num_classes = 10\n",
    "print('input image size =', in_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now as usual, as a sanity test let's make sure we can overfit a tiny dataset with our model. But first we need to adapt our `Trainer` for PyTorch models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Complete the implementaion of the `TorchTrainer` class in the `hw2/training.py` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_batch (Avg. Loss 2.371, Accuracy 6.0): 100%|██████████| 25/25 [00:00<00:00, 31.05it/s]\n",
      "train_batch (Avg. Loss 2.238, Accuracy 16.0): 100%|██████████| 25/25 [00:00<00:00, 71.27it/s]\n",
      "train_batch (Avg. Loss 2.131, Accuracy 22.0): 100%|██████████| 25/25 [00:00<00:00, 82.79it/s]\n",
      "train_batch (Avg. Loss 1.829, Accuracy 34.0): 100%|██████████| 25/25 [00:00<00:00, 82.20it/s]\n",
      "train_batch (Avg. Loss 1.065, Accuracy 58.0): 100%|██████████| 25/25 [00:00<00:00, 57.53it/s]\n",
      "train_batch (Avg. Loss 1.149, Accuracy 56.0): 100%|██████████| 25/25 [00:00<00:00, 63.82it/s]\n",
      "train_batch (Avg. Loss 0.781, Accuracy 70.0): 100%|██████████| 25/25 [00:00<00:00, 60.88it/s]\n",
      "train_batch (Avg. Loss 0.975, Accuracy 78.0): 100%|██████████| 25/25 [00:00<00:00, 61.56it/s]\n",
      "train_batch (Avg. Loss 0.379, Accuracy 88.0): 100%|██████████| 25/25 [00:00<00:00, 72.15it/s]\n",
      "train_batch (Avg. Loss 0.353, Accuracy 88.0): 100%|██████████| 25/25 [00:00<00:00, 64.62it/s]\n",
      "train_batch (Avg. Loss 0.475, Accuracy 92.0): 100%|██████████| 25/25 [00:00<00:00, 70.73it/s]\n"
     ]
    },
    {
     "ename": "AssertionError",
     "evalue": "tensor(94) not greater than or equal to 95",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-a83f9a7e7819>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     21\u001b[0m \u001b[0;31m# Test overfitting\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m \u001b[0mtest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0massertGreaterEqual\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbest_acc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m95\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/miniconda3/envs/cs236605-hw/lib/python3.7/unittest/case.py\u001b[0m in \u001b[0;36massertGreaterEqual\u001b[0;34m(self, a, b, msg)\u001b[0m\n\u001b[1;32m   1242\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0ma\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1243\u001b[0m             \u001b[0mstandardMsg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'%s not greater than or equal to %s'\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msafe_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msafe_repr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1244\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfail\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_formatMessage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstandardMsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1245\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1246\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0massertIsNone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/cs236605-hw/lib/python3.7/unittest/case.py\u001b[0m in \u001b[0;36mfail\u001b[0;34m(self, msg)\u001b[0m\n\u001b[1;32m    678\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mfail\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    679\u001b[0m         \u001b[0;34m\"\"\"Fail immediately, with the given message.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 680\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfailureException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    681\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    682\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0massertFalse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexpr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAssertionError\u001b[0m: tensor(94) not greater than or equal to 95"
     ]
    }
   ],
   "source": [
    "import hw2.training as training\n",
    "torch.manual_seed(seed)\n",
    "\n",
    "# Define a tiny part of the CIFAR-10 dataset to overfit it\n",
    "batch_size = 2\n",
    "max_batches = 25\n",
    "dl_train = torch.utils.data.DataLoader(ds_train, batch_size, shuffle=False)\n",
    "\n",
    "# Create model, loss and optimizer instances\n",
    "model = models.ConvClassifier(in_size, num_classes, filters=[32], pool_every=1, hidden_dims=[100])\n",
    "loss_fn = torch.nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9,)\n",
    "\n",
    "# Use TorchTrainer to run only the training loop a few times.\n",
    "trainer = training.TorchTrainer(model, loss_fn, optimizer, device)\n",
    "best_acc = 0\n",
    "for i in range(22):\n",
    "    res = trainer.train_epoch(dl_train, max_batches=max_batches, verbose=(i%2==0))\n",
    "    best_acc = res.accuracy if res.accuracy > best_acc else best_acc\n",
    "    \n",
    "# Test overfitting\n",
    "test.assertGreaterEqual(best_acc, 95)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experimenting with model architectures\n",
    "<a id=part3_3></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will now perform a series of experiments that train various model configurations on a much larger part of the CIFAR-10 dataset.\n",
    "\n",
    "To perform the experiments, you'll need to use a machine with a GPU since training time might be too long otherwise.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes on using course servers\n",
    "\n",
    "First, please read the [course servers guide](https://vistalab-technion.github.io/cs236605/assignments/hpc-servers/) carefully.\n",
    "\n",
    "To run the experiments on the course servers, you can use the `py-sbatch.sh` script directly to perform a single experiment run in batch mode (since it runs python once), or use the `srun` command to do a single run in interactive mode. For example, running a single run of experiment 1 interactively (after `conda activate` of course):\n",
    "\n",
    "```shell\n",
    "srun -c 2 --gres=gpu:1 --pty python -m hw2.experiments run-exp -n test -K 32 64 -L 2 -P 2 -H 100\n",
    "```\n",
    "\n",
    "To perform multiple runs in batch mode with `sbatch` (e.g. for running all the configurations of an experiments), you can create your own script based on `py-sbatch.sh` and invoke whatever commands you need within it.\n",
    "\n",
    "Please don't request more than **2 CPU** cores and **1 GPU** device for your runs. The code won't be able to utilize more than that anyway, so you'll see no performance gain if you do. It will only cause delays for other students using the servers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General notes for running experiments\n",
    "\n",
    "- You can run the experiments on a different machine (e.g. the course servers) and copy the results (files)\n",
    "to the `results` folder on your local machine.\n",
    "This notebook will only display the results, not run the actual experiment code (except for a demo run).\n",
    "\n",
    "- It's important to give each experiment run a name as specified by the notebook instructions later on. The each run has a `run_name` parameter that will also be the name of the results file which this notebook will expect to load.\n",
    "\n",
    "- You will implement the code to run the experiments in the `hw2/experiments.py` module. This module has a CLI parser so that you can invoke it as a script and pass in all the configuration parameters for a single experiment run.\n",
    "\n",
    "- You should use `python -m hw2.experiments run-exp` to run an experiment, and **not** `python hw2/experiments.py run-exp`, regardless of how/where you run it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment 1 - Network depth and number of filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part we will test some different architecture configurations based on our `ConvClassifier`.\n",
    "Specifically, we want to try different depths and number of features to see the effects these parameters have on the model's performance.\n",
    "\n",
    "To do this, we'll define two extra hyperparameters for our model, `K` (`filters_per_layer`) and `L` (`layers_per_block`).\n",
    "- `K` is a list, containing the number of filters we want to have in our conv layers.\n",
    "- `L` is the number of consecutive layers with the same number of filters to use.\n",
    "\n",
    "For example, if `K=[32, 64]` and `L=2` it means we want two conv layers with 32 filters followed by two conv layers with 64 filters. The feature-extraction part of our model will therefore be:\n",
    "\n",
    "    Conv(X,32)->ReLu->Conv(32,32)->ReLU->MaxPool->Conv(32,64)->ReLU->Conv(64,64)->ReLU->MaxPool\n",
    "    \n",
    "We'll try various values of the `K` and `L` parameters in combination and see how each architecture trains. **All other hyperparameters are up to you**, including the choice of the optimization algorithm, the learning rate, regularization and architecture hyperparams such as `pad_every` and `hidden_dims`. You can try some manual runs to determine some good values for the hyperparameters or implement cross-validation to do it.\n",
    "However, the **dataset size** you test on should be large. Use at least ~12000 training images and ~3000 validation images. The **batch size** parameter however is also up to you.\n",
    "\n",
    "The important thing is that you state what you used, how you decided on it, and explain your results based on that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to write some code to run the experiment.\n",
    "\n",
    "**TODO**:\n",
    "1. Implement the `run_experiment()` function in the `hw2/experiments.py` module.\n",
    "1. If you haven't done so already, it would be an excellent idea to implement the **early stopping** feature of the `Trainer` class.\n",
    "\n",
    "The following block tests that your implementation works. It's also meant to show you that each experiment run creates a result file containing the parameters to reproduce and the `FitResult` object for plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hw2.experiments as experiments\n",
    "from hw2.experiments import load_experiment\n",
    "from cs236605.plot import plot_fit\n",
    "\n",
    "# Test experiment1 implementation on a few data samples and with a small model\n",
    "experiments.run_experiment('test_run', seed=seed, bs_train=50, batches=10, epochs=10, early_stopping=5,\n",
    "                           filters_per_layer=[32], layers_per_block=1, pool_every=1, hidden_dims=[100])\n",
    "\n",
    "# There should now be a file 'test_run.json' in your `results/` folder.\n",
    "# We can use it to load the results of the experiment.\n",
    "cfg, fit_res = load_experiment('results/test_run.json')\n",
    "_, _ = plot_fit(fit_res)\n",
    "\n",
    "# And `cfg` contains the exact parameters to reproduce it\n",
    "print('experiment config: ', cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the following function to load multiple experiment results and plot them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_exp_results(filename_pattern, results_dir='results'):\n",
    "    fig = None\n",
    "    result_files = glob.glob(os.path.join(results_dir, filename_pattern))\n",
    "    result_files.sort()\n",
    "    if len(result_files) == 0:\n",
    "        print(f'No results found for pattern {filename_pattern}.', file=sys.stderr)\n",
    "        return\n",
    "    for filepath in result_files:\n",
    "        m = re.match('exp\\d_(\\d_)?(.*)\\.json', os.path.basename(filepath))\n",
    "        cfg, fit_res = load_experiment(filepath)\n",
    "        fig, axes = plot_fit(fit_res, fig, legend=m[2],log_loss=True)\n",
    "    del cfg['filters_per_layer']\n",
    "    del cfg['layers_per_block']\n",
    "    print('common config: ', cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment 1.1: Varying the network depth (`L`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll test the effect of the network depth on training.\n",
    "\n",
    "**Configuratons**:\n",
    "- `K=32` fixed, with `L=2,4,8,16` varying per run\n",
    "- `K=64` fixed, with `L=2,4,8,16` varying per run\n",
    "\n",
    "So 8 different runs in total.\n",
    "\n",
    "**Naming runs**:\n",
    "Each run should be named `exp1_1_K{}_L{}` where the braces are placeholders for the values. For example, the first run should be named `exp1_1_K32_L2`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Run the experiment on the above configuration. Make sure the result file names are as expected. Use the following blocks to display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_exp_results('exp1_1_K32*.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_exp_results('exp1_1_K64*.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment 1.2: Varying the number of filters per layer (`K`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll test the effect of the number of convolutional filters in each layer.\n",
    "\n",
    "**Configuratons**:\n",
    "- `L=2` fixed, with `K=[32],[64],[128],[258]` varying per run.\n",
    "- `L=4` fixed, with `K=[32],[64],[128],[258]` varying per run.\n",
    "- `L=8` fixed, with `K=[32],[64],[128],[258]` varying per run.\n",
    "\n",
    "So 12 different runs in total. To clarify, each run `K` takes the value of a list with a single element.\n",
    "\n",
    "**Naming runs**:\n",
    "Each run should be named `exp1_2_L{}_K{}` where the braces are placeholders for the values. For example, the first run should be named `exp1_2_L2_K32`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Run the experiment on the above configuration. Make sure the result file names are as expected. Use the following blocks to display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_exp_results('exp1_2_L2*.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_exp_results('exp1_2_L4*.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_exp_results('exp1_2_L8*.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment 1.3: Varying both the number of filters (`K`) and network depth (`L`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll test the effect of the number of convolutional filters in each layer.\n",
    "\n",
    "**Configuratons**:\n",
    "- `K=[64, 128, 256]` fixed with `L=1,2,3,4` varying per run.\n",
    "\n",
    "So 4 different runs in total. To clarify, each run `K` takes the value of an array with a three elements.\n",
    "\n",
    "**Naming runs**:\n",
    "Each run should be named `exp1_3_L{}_K{}-{}-{}` where the braces are placeholders for the values. For example, the first run should be named `exp1_3_L1_K64-128-256`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Run the experiment on the above configuration. Make sure the result file names are as expected. Use the following blocks to display the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_exp_results('exp1_3*.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment 2 - Custom network architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part you will create your own custom network architecture based on the `ConvClassifier` you've implemented.\n",
    "\n",
    "Try to overcome some of the limitations your experiment 1 results, using what you learned in the course.\n",
    "\n",
    "You are free to add whatever you like to the model, for instance \n",
    "- Batch normalization\n",
    "- Dropout layers\n",
    "- Skip connections\n",
    "- Change kernel spatial sizes and strides\n",
    "- Custom blocks or ideas from known architectures (e.g. inception module)\n",
    "\n",
    "Just make sure to keep the model's `init` API identical (or maybe just add parameters)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Implement your custom architecture in the `YourCodeNet` class within the `hw2/models.py` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "YourCodeNet(\n",
      "  (feature_extractor): Sequential(\n",
      "    (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (2): ReLU()\n",
      "    (3): Dropout(p=0.0)\n",
      "    (4): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (5): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (6): ReLU()\n",
      "    (7): Dropout(p=0.0)\n",
      "    (8): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)\n",
      "    (9): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (10): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (11): ReLU()\n",
      "    (12): Dropout(p=0.0)\n",
      "    (13): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "    (14): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (15): ReLU()\n",
      "    (16): Dropout(p=0.0)\n",
      "    (17): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)\n",
      "  )\n",
      "  (classifier): Sequential(\n",
      "    (0): Linear(in_features=20000, out_features=100, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=100, out_features=100, bias=True)\n",
      "    (3): ReLU()\n",
      "    (4): Linear(in_features=100, out_features=10, bias=True)\n",
      "  )\n",
      ")\n",
      "out = tensor([[ 0.0595,  0.1474,  0.0284,  0.0845, -0.2110, -0.0991, -0.1282, -0.1282,\n",
      "         -0.1251,  0.0493]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "net = models.YourCodeNet((3,100,100), 10, filters=[32]*4, pool_every=2, hidden_dims=[100]*2).to(device)\n",
    "print(net)\n",
    "\n",
    "test_image = torch.randint(low=0, high=256, size=(3, 100, 100), dtype=torch.float).to(device)\n",
    "test_out = net(test_image.unsqueeze(0))\n",
    "print('out =', test_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment 2 Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run your custom model on at least the following:\n",
    "\n",
    "**Configuratons**:\n",
    "- `K=[64, 128, 256, 512]` fixed with `L=1,2,3,4` varying per run.\n",
    "\n",
    "So 4 different runs in total. To clarify, each run `K` takes the value of an array with a four elements.\n",
    "\n",
    "If you want, you can add some extra following the same pattern.\n",
    "Try to see how deep a model you can train.\n",
    "\n",
    "**Naming runs**:\n",
    "Each run should be named `exp2_L{}_K{}-{}-{}-{}` where the braces are placeholders for the values. For example, the first run should be named `exp2_L1_K64-128-256-512`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO**: Run the experiment on the above configuration. Make sure the result file names are as expected. Use the following blocks to display the results. To make the experiment runner use your custom network model instead of the previous one, use the `--ycn` argument on the command line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_exp_results('exp2*.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions\n",
    "<a id=part3_4></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO** Answer the following questions. Write your answers in the appropriate variables in the module `hw2/answers.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cs236605.answers import display_answer\n",
    "import hw2.answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1 \n",
    "\n",
    "Analyze your results from experiment 1.1. In particular,\n",
    "1.  Explain the effect of depth on the accuracy. What depth produces the best results and why do you think that's the case?\n",
    "1. Were there values of `L` for which the network wasn't trainable? what causes this? Suggest two things which may be done to resolve it at least partially."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_answer(hw2.answers.part3_q1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2 \n",
    "\n",
    "Analyze your results from experiment 1.2. In particular, compare to the results of experiment 1.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_answer(hw2.answers.part3_q2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3 \n",
    "\n",
    "Analyze your results from experiment 1.3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_answer(hw2.answers.part3_q3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4 \n",
    "\n",
    "1. Explain your modifications to the architecture which you implemented in the `YourCodeNet` class.\n",
    "2. Analyze the results of experiment 2. Compare to experiment 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_answer(hw2.answers.part3_q4)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}