algorithmics-optuna-baselines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will be tuning a deep fully-connected neural network by experimenting with its number of layers & number of neurons. To this end, we will be using Optuna, a framework for tuning the hyperparameters of machine learning models. We will be applying RandomSearch & Bayesian Optimization methods to create a baseline, so to compare its results to the hyperparameter tuning results of the differential evolution algorithm we implemented.\n",
    "\n",
    "Let's start by importing the necessary packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loading & Transforms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enabling notebook extension jupyter-js-widgets/extension...\n",
      "      - Validating: \u001b[32mOK\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!jupyter nbextension enable --py widgetsnbextension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import time\n",
    "\n",
    "import torch\n",
    "import torchvision.datasets as datasets\n",
    "from torchvision import transforms\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import torch.utils.data\n",
    "\n",
    "# !pip install optuna\n",
    "# Kaggle already has it installed in its notebooks btw.\n",
    "import optuna\n",
    "import optuna\n",
    "from optuna.trial import TrialState\n",
    "\n",
    "import os\n",
    "\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are in /home/dick\n"
     ]
    }
   ],
   "source": [
    "print('You are in', os.getcwd())\n",
    "data_dir = os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DEVICE = torch.device(\"cpu\")\n",
    "classes = 10\n",
    "epochs = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now download the datasets and transform them into pytorch tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/torchvision/datasets/mnist.py:498: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:180.)\n",
      "  return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)\n"
     ]
    }
   ],
   "source": [
    "mnist_trainset = datasets.MNIST(root=data_dir, train=True, download=True, transform=transforms.ToTensor())\n",
    "mnist_testset = datasets.MNIST(root=data_dir, train=False, download=True, transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train dataset has a size of 60000 samples and the test has 10000 hand-written digit images in it. The class labels change between 0 and 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset MNIST\n",
      "    Number of datapoints: 60000\n",
      "    Root location: /home/dick\n",
      "    Split: Train\n",
      "    StandardTransform\n",
      "Transform: ToTensor()\n",
      "\n",
      "Dataset MNIST\n",
      "    Number of datapoints: 10000\n",
      "    Root location: /home/dick\n",
      "    Split: Test\n",
      "    StandardTransform\n",
      "Transform: ToTensor()\n"
     ]
    }
   ],
   "source": [
    "print(mnist_trainset)\n",
    "print()\n",
    "print(mnist_testset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create the dataloader objects for the datasets we downloaded. In order to make a fair comparison, we will be using the default hyperparameters of the notebook that uses the optimization via differential evolution algorithm. But in case you'd like to increase the batch_size or shuffle the rows to prevent any overfitting, feel free to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_loader = torch.utils.data.DataLoader(mnist_trainset, batch_size=10, shuffle=False)\n",
    "test_loader = torch.utils.data.DataLoader(mnist_testset, batch_size=10, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before creating the model, let's print the shapes of the training instances in one batch. We won't be checking the test instances since the same results apply to them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of the input features in one batch: torch.Size([10, 1, 28, 28])\n",
      "Shape of the targets in one batch: torch.Size([10])\n"
     ]
    }
   ],
   "source": [
    "print('Shape of the input features in one batch:', next(iter(train_loader))[0].shape)\n",
    "print('Shape of the targets in one batch:', next(iter(train_loader))[1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make it clear, the numbers for the shape of input tensors above correspond the **[Batch_Size, RGB Channels, Height, Width]**. MNIST is black and white so its **channel count is 1**. The rest self-explanatory, our batches are comprised of **10 image instances** in the form of **28 x 28 pixels**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've finished creating the data loaders and now we can start defining the model. Remember that we will be tuning only the number of layers&neurons, other hyperparameters will be the same as that of DE.ipynb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def baseline_model(trial):\n",
    "    \n",
    "    # In the default setting; the number of layers were in the range of 5 and 70, while the number of hidden neurons\n",
    "    # were changing between 4 and 10.\n",
    "    n_layers = trial.suggest_int(\"n_layers\", 5, 70)\n",
    "    layers = []\n",
    "    \n",
    "    in_features = 28 * 28\n",
    "    for i in range(n_layers):\n",
    "        out_features = trial.suggest_int(\"n_units_l{}\".format(i), 4, 10)\n",
    "        layers.append(nn.Linear(in_features, out_features))\n",
    "        layers.append(nn.ReLU())\n",
    "\n",
    "        in_features = out_features\n",
    "        \n",
    "    layers.append(nn.Linear(in_features, 10))\n",
    "    layers.append(nn.LogSoftmax(dim = 1))\n",
    "\n",
    "    return nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create our objective function to tune with Optuna."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optuna requires an objective function to start its experiments. Below we will be creating it by instantiating the model and fetching the datasets, followed by training and testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Reduce these numbers for faster training.\n",
    "BATCHSIZE = 10\n",
    "N_TRAIN_EXAMPLES = BATCHSIZE * 6000\n",
    "N_VALID_EXAMPLES = BATCHSIZE * 1000\n",
    "early_stopping_rounds = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def objective(trial):\n",
    "\n",
    "    model = baseline_model(trial).to(DEVICE)\n",
    "\n",
    "    # Tune the specific hyperparameters of the gradient descent if you wish.\n",
    "    # optimizer_name = trial.suggest_categorical(\"optimizer\", [\"Adam\", \"RMSprop\", \"SGD\"])\n",
    "    # lr = trial.suggest_float(\"lr\", 1e-5, 1e-1, log=True)\n",
    "    \n",
    "    optimizer_name = 'Adam'\n",
    "    lr = 0.001\n",
    "    \n",
    "    optimizer = getattr(optim, optimizer_name)(model.parameters(), lr=lr)\n",
    "    loss_fn = torch.nn.NLLLoss()\n",
    "    \n",
    "    # Get the MNIST dataloaders.\n",
    "    #train_loader, valid_loader = train_loader, test_loader\n",
    "    valid_loader = test_loader\n",
    "    \n",
    "    train_batch_count = min(len(train_loader.dataset), N_TRAIN_EXAMPLES) / BATCHSIZE\n",
    "    valid_batch_count = min(len(valid_loader.dataset), N_VALID_EXAMPLES) / BATCHSIZE\n",
    "    # Training of the model.\n",
    "    stop_counter = 0\n",
    "    best_val_loss = float('inf')\n",
    "    for epoch in range(epochs):\n",
    "        if stop_counter > early_stopping_rounds:\n",
    "            print()\n",
    "            print('No improvement in validation data for the specified number of early stopping rounds. Stopping training.')\n",
    "            break\n",
    "                \n",
    "        model.train()\n",
    "        trainloss = 0\n",
    "        for batch_idx, (data, target) in enumerate(train_loader):\n",
    "            # Limiting training data for faster epochs.\n",
    "            if batch_idx * BATCHSIZE >= N_TRAIN_EXAMPLES:\n",
    "                break\n",
    "            \n",
    "            \n",
    "            data, target = data.view(data.size(0), -1).to(DEVICE), target.to(DEVICE)\n",
    "            \n",
    "            optimizer.zero_grad()\n",
    "            output = model(data)\n",
    "\n",
    "            loss = loss_fn(output, target)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            trainloss += loss.item()\n",
    "            \n",
    "        trainloss = trainloss / train_batch_count\n",
    "        # Validation of the model.\n",
    "        model.eval()\n",
    "        validationloss = 0\n",
    "        correct = 0\n",
    "        with torch.no_grad():\n",
    "            for batch_idx, (data, target) in enumerate(valid_loader):\n",
    "                # Limiting validation data.\n",
    "                if batch_idx * BATCHSIZE >= N_VALID_EXAMPLES:\n",
    "                    break\n",
    "                    \n",
    "                data, target = data.view(data.size(0), -1).to(DEVICE), target.to(DEVICE)\n",
    "                \n",
    "                output = model(data)\n",
    "                loss = loss_fn(output, target)\n",
    "                validationloss += loss.item()\n",
    "                \n",
    "                # Get the index of the max probability.\n",
    "                pred = output.argmax(dim=1, keepdim=True)\n",
    "                correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "                \n",
    "        validationloss = validationloss / valid_batch_count\n",
    "        \n",
    "        if validationloss < best_val_loss:\n",
    "            best_val_loss = validationloss\n",
    "            stop_counter = 0\n",
    "        else:\n",
    "            stop_counter += 1\n",
    "        \n",
    "        accuracy = correct /  min(len(valid_loader.dataset), N_VALID_EXAMPLES)\n",
    "        \n",
    "        print('Epoch {} => Training_Loss:{:.4f}, Validation_Loss:{:.8f}, Validation_Acc:{:.8f}, Stop_counter:{}'.format(epoch, trainloss, validationloss, accuracy, stop_counter))\n",
    "        trial.report(accuracy, epoch)\n",
    "\n",
    "        # Handle pruning based on the intermediate value. Doing this will stop tuning experiments which doesn't seem very promising.\n",
    "        # In case you'd like to get the worst-case runtime of optuna, feel free to comment the lines below.\n",
    "        if trial.should_prune():\n",
    "            raise optuna.exceptions.TrialPruned()\n",
    "\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create the study object and start our experiments. One can add a time limit to the experiments by defining the 'timeout' paramater to the optimize attribute of the study.\n",
    "\n",
    "Here, we will be using the TPE sampler(default sampler of optuna), which results in optimizing the hyperparameters using bayesian statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/optuna/progress_bar.py:47: ExperimentalWarning: Progress bar is experimental (supported from v1.2.0). The interface can change in the future.\n",
      "  self._init_valid()\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09d7be5eca7544f99380d55882b5fa62",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 => Training_Loss:2.3022, Validation_Loss:2.30153231, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 1 => Training_Loss:2.3018, Validation_Loss:2.30150786, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 2 => Training_Loss:2.3018, Validation_Loss:2.30147860, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 3 => Training_Loss:2.3017, Validation_Loss:2.30146436, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 4 => Training_Loss:2.3017, Validation_Loss:2.30145326, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 5 => Training_Loss:2.3017, Validation_Loss:2.30144351, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 6 => Training_Loss:2.3016, Validation_Loss:2.30143410, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 7 => Training_Loss:2.3016, Validation_Loss:2.30142711, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 8 => Training_Loss:2.3016, Validation_Loss:2.30142132, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 9 => Training_Loss:2.3016, Validation_Loss:2.30141070, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 10 => Training_Loss:2.3015, Validation_Loss:2.30140659, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 11 => Training_Loss:2.3015, Validation_Loss:2.30140408, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 12 => Training_Loss:2.3015, Validation_Loss:2.30144241, Validation_Acc:0.10280000, Stop_counter:1\n",
      "Epoch 13 => Training_Loss:2.3015, Validation_Loss:2.30138467, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 14 => Training_Loss:2.3015, Validation_Loss:2.30138003, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 15 => Training_Loss:2.3015, Validation_Loss:2.30137566, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 16 => Training_Loss:2.3015, Validation_Loss:2.30137103, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 17 => Training_Loss:2.3015, Validation_Loss:2.30136679, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 18 => Training_Loss:2.3014, Validation_Loss:2.30136274, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 19 => Training_Loss:2.3014, Validation_Loss:2.30135901, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 20 => Training_Loss:2.3014, Validation_Loss:2.30135458, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 21 => Training_Loss:2.3014, Validation_Loss:2.30136748, Validation_Acc:0.10280000, Stop_counter:1\n",
      "Epoch 22 => Training_Loss:2.3014, Validation_Loss:2.30137827, Validation_Acc:0.10280000, Stop_counter:2\n",
      "Epoch 23 => Training_Loss:2.3014, Validation_Loss:2.30135868, Validation_Acc:0.10280000, Stop_counter:3\n",
      "Epoch 24 => Training_Loss:2.3014, Validation_Loss:2.30135940, Validation_Acc:0.10280000, Stop_counter:4\n",
      "\n",
      "No improvement in validation data for the specified number of early stopping rounds. Stopping training.\n",
      "Epoch 0 => Training_Loss:2.3025, Validation_Loss:2.30142015, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 1 => Training_Loss:2.3018, Validation_Loss:2.30141614, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 2 => Training_Loss:2.3017, Validation_Loss:2.30142054, Validation_Acc:0.10280000, Stop_counter:1\n",
      "Epoch 3 => Training_Loss:2.3017, Validation_Loss:2.30138582, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 4 => Training_Loss:2.3016, Validation_Loss:2.30142097, Validation_Acc:0.10280000, Stop_counter:1\n",
      "Epoch 5 => Training_Loss:2.3016, Validation_Loss:2.30139151, Validation_Acc:0.10280000, Stop_counter:2\n",
      "Epoch 6 => Training_Loss:2.3015, Validation_Loss:2.30139723, Validation_Acc:0.10280000, Stop_counter:3\n",
      "Epoch 7 => Training_Loss:2.3015, Validation_Loss:2.30143026, Validation_Acc:0.10280000, Stop_counter:4\n",
      "\n",
      "No improvement in validation data for the specified number of early stopping rounds. Stopping training.\n",
      "Epoch 0 => Training_Loss:2.3027, Validation_Loss:2.30146327, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 1 => Training_Loss:2.3017, Validation_Loss:2.30145327, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 2 => Training_Loss:2.3016, Validation_Loss:2.30144307, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 3 => Training_Loss:2.3016, Validation_Loss:2.30143016, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 4 => Training_Loss:2.3016, Validation_Loss:2.30142460, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 5 => Training_Loss:2.3016, Validation_Loss:2.30141866, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 6 => Training_Loss:2.3015, Validation_Loss:2.30141159, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 7 => Training_Loss:2.3015, Validation_Loss:2.30140994, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 8 => Training_Loss:2.3015, Validation_Loss:2.30141933, Validation_Acc:0.10280000, Stop_counter:1\n",
      "Epoch 9 => Training_Loss:2.3015, Validation_Loss:2.30141435, Validation_Acc:0.10280000, Stop_counter:2\n",
      "Epoch 10 => Training_Loss:2.3014, Validation_Loss:2.30134828, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 11 => Training_Loss:2.3014, Validation_Loss:2.30134444, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 12 => Training_Loss:2.3014, Validation_Loss:2.30134086, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 13 => Training_Loss:2.3014, Validation_Loss:2.30133671, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 14 => Training_Loss:2.3014, Validation_Loss:2.30133314, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 15 => Training_Loss:2.3014, Validation_Loss:2.30132952, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 16 => Training_Loss:2.3014, Validation_Loss:2.30132500, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 17 => Training_Loss:2.3014, Validation_Loss:2.30132065, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 18 => Training_Loss:2.3014, Validation_Loss:2.30131628, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 19 => Training_Loss:2.3014, Validation_Loss:2.30131145, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 20 => Training_Loss:2.3014, Validation_Loss:2.30130677, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 21 => Training_Loss:2.3014, Validation_Loss:2.30130248, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 22 => Training_Loss:2.3014, Validation_Loss:2.30130255, Validation_Acc:0.11350000, Stop_counter:1\n",
      "Epoch 23 => Training_Loss:2.3014, Validation_Loss:2.30130255, Validation_Acc:0.11350000, Stop_counter:2\n",
      "Epoch 24 => Training_Loss:2.3014, Validation_Loss:2.30130255, Validation_Acc:0.11350000, Stop_counter:3\n",
      "Epoch 25 => Training_Loss:2.3014, Validation_Loss:2.30130255, Validation_Acc:0.11350000, Stop_counter:4\n",
      "\n",
      "No improvement in validation data for the specified number of early stopping rounds. Stopping training.\n",
      "Epoch 0 => Training_Loss:1.4464, Validation_Loss:1.06241752, Validation_Acc:0.63320000, Stop_counter:0\n",
      "Epoch 1 => Training_Loss:0.8856, Validation_Loss:0.82830207, Validation_Acc:0.75300000, Stop_counter:0\n",
      "Epoch 2 => Training_Loss:0.7565, Validation_Loss:0.78108937, Validation_Acc:0.77370000, Stop_counter:0\n",
      "Epoch 3 => Training_Loss:0.7033, Validation_Loss:0.74803427, Validation_Acc:0.78360000, Stop_counter:0\n",
      "Epoch 4 => Training_Loss:0.6746, Validation_Loss:0.73684299, Validation_Acc:0.79050000, Stop_counter:0\n",
      "Epoch 5 => Training_Loss:0.6545, Validation_Loss:0.71536457, Validation_Acc:0.79470000, Stop_counter:0\n",
      "Epoch 6 => Training_Loss:0.6415, Validation_Loss:0.71902679, Validation_Acc:0.79630000, Stop_counter:1\n",
      "Epoch 7 => Training_Loss:0.6297, Validation_Loss:0.70900866, Validation_Acc:0.79410000, Stop_counter:0\n",
      "Epoch 8 => Training_Loss:0.6234, Validation_Loss:0.67491900, Validation_Acc:0.80560000, Stop_counter:0\n",
      "Epoch 9 => Training_Loss:0.6124, Validation_Loss:0.66512462, Validation_Acc:0.80720000, Stop_counter:0\n",
      "Epoch 10 => Training_Loss:0.6097, Validation_Loss:0.64524757, Validation_Acc:0.81530000, Stop_counter:0\n",
      "Epoch 11 => Training_Loss:0.5990, Validation_Loss:0.66585063, Validation_Acc:0.81020000, Stop_counter:1\n",
      "Epoch 12 => Training_Loss:0.5935, Validation_Loss:0.65811793, Validation_Acc:0.80850000, Stop_counter:2\n",
      "Epoch 13 => Training_Loss:0.5892, Validation_Loss:0.66493096, Validation_Acc:0.81070000, Stop_counter:3\n",
      "Epoch 14 => Training_Loss:0.5794, Validation_Loss:0.63175374, Validation_Acc:0.81610000, Stop_counter:0\n",
      "Epoch 15 => Training_Loss:0.5758, Validation_Loss:0.62131079, Validation_Acc:0.82060000, Stop_counter:0\n",
      "Epoch 16 => Training_Loss:0.5730, Validation_Loss:0.62496835, Validation_Acc:0.82030000, Stop_counter:1\n",
      "Epoch 17 => Training_Loss:0.5677, Validation_Loss:0.62694598, Validation_Acc:0.82220000, Stop_counter:2\n",
      "Epoch 18 => Training_Loss:0.5641, Validation_Loss:0.61720733, Validation_Acc:0.82490000, Stop_counter:0\n",
      "Epoch 19 => Training_Loss:0.5511, Validation_Loss:0.59754877, Validation_Acc:0.83360000, Stop_counter:0\n",
      "Epoch 20 => Training_Loss:0.5603, Validation_Loss:0.61961096, Validation_Acc:0.83460000, Stop_counter:1\n",
      "Epoch 21 => Training_Loss:0.5388, Validation_Loss:0.59722863, Validation_Acc:0.83750000, Stop_counter:0\n",
      "Epoch 22 => Training_Loss:0.5246, Validation_Loss:0.58099256, Validation_Acc:0.84520000, Stop_counter:0\n",
      "Epoch 23 => Training_Loss:0.5169, Validation_Loss:0.56237481, Validation_Acc:0.84880000, Stop_counter:0\n",
      "Epoch 24 => Training_Loss:0.5124, Validation_Loss:0.58074112, Validation_Acc:0.84590000, Stop_counter:1\n",
      "Epoch 25 => Training_Loss:0.5084, Validation_Loss:0.59369684, Validation_Acc:0.84550000, Stop_counter:2\n",
      "Epoch 26 => Training_Loss:0.5054, Validation_Loss:0.57716806, Validation_Acc:0.85010000, Stop_counter:3\n",
      "Epoch 27 => Training_Loss:0.5000, Validation_Loss:0.58847515, Validation_Acc:0.83940000, Stop_counter:4\n",
      "\n",
      "No improvement in validation data for the specified number of early stopping rounds. Stopping training.\n",
      "Study statistics: \n",
      "  Number of finished trials:  4\n",
      "  Number of pruned trials:  0\n",
      "  Number of complete trials:  4\n",
      "Best trial:\n",
      "  Value:  0.8394\n",
      "  Params: \n",
      "    n_layers: 12\n",
      "    n_units_l0: 6\n",
      "    n_units_l1: 9\n",
      "    n_units_l2: 4\n",
      "    n_units_l3: 10\n",
      "    n_units_l4: 8\n",
      "    n_units_l5: 4\n",
      "    n_units_l6: 5\n",
      "    n_units_l7: 8\n",
      "    n_units_l8: 4\n",
      "    n_units_l9: 4\n",
      "    n_units_l10: 9\n",
      "    n_units_l11: 7\n"
     ]
    }
   ],
   "source": [
    "# You can change the logging type if you'd like to see more verbosity in trials. For now I just want to see the progress and no other details\n",
    "# regarding the trials.\n",
    "optuna.logging.set_verbosity(optuna.logging.WARNING)\n",
    "\n",
    "study = optuna.create_study(direction=\"maximize\")\n",
    "study.optimize(objective, n_trials=10, timeout=1500, show_progress_bar = True)\n",
    "\n",
    "pruned_trials = study.get_trials(deepcopy=False, states=[TrialState.PRUNED])\n",
    "complete_trials = study.get_trials(deepcopy=False, states=[TrialState.COMPLETE])\n",
    "\n",
    "print(\"Study statistics: \")\n",
    "print(\"  Number of finished trials: \", len(study.trials))\n",
    "print(\"  Number of pruned trials: \", len(pruned_trials))\n",
    "print(\"  Number of complete trials: \", len(complete_trials))\n",
    "\n",
    "print(\"Best trial:\")\n",
    "best_trial = study.best_trial\n",
    "\n",
    "print(\"  Value: \", best_trial.value)\n",
    "\n",
    "print(\"  Params: \")\n",
    "for key, value in best_trial.params.items():\n",
    "    print(\"    {}: {}\".format(key, value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the bayesian optimization baseline took around 25 minutes to complete its run on the CPU, and came up with an accuracy value of 84% and a validation loss of 0.59. It proposes 12 layers, with varying number of neurons.\n",
    "\n",
    "Let's save the results of the trials for later usage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘studies’: File exists\n"
     ]
    }
   ],
   "source": [
    "!mkdir studies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "study_path = './studies'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number</th>\n",
       "      <th>value</th>\n",
       "      <th>datetime_start</th>\n",
       "      <th>datetime_complete</th>\n",
       "      <th>duration</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>2022-01-21 20:45:35.656644</td>\n",
       "      <td>2022-01-21 20:45:35.656696</td>\n",
       "      <td>00:00:00.000052</td>\n",
       "      <td>FAIL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   number value             datetime_start          datetime_complete  \\\n",
       "0       0  None 2022-01-21 20:45:35.656644 2022-01-21 20:45:35.656696   \n",
       "\n",
       "         duration state  \n",
       "0 00:00:00.000052  FAIL  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(study, study_path + '/mnist_optuna_tpe_10trials2.pkl')\n",
    "study = joblib.load(study_path + '/mnist_optuna_tpe_10trials2.pkl')\n",
    "df_bayes = study.trials_dataframe()\n",
    "df_bayes.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the tuning results obtained by Random Search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/optuna/progress_bar.py:47: ExperimentalWarning: Progress bar is experimental (supported from v1.2.0). The interface can change in the future.\n",
      "  self._init_valid()\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4781d3650d754f93b8f96cb1047feba8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 => Training_Loss:2.3023, Validation_Loss:2.30151453, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 1 => Training_Loss:2.3017, Validation_Loss:2.30152230, Validation_Acc:0.10280000, Stop_counter:1\n",
      "Epoch 2 => Training_Loss:2.3017, Validation_Loss:2.30151260, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 3 => Training_Loss:2.3016, Validation_Loss:2.30150102, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 4 => Training_Loss:2.3016, Validation_Loss:2.30149903, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 5 => Training_Loss:2.3015, Validation_Loss:2.30149046, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 6 => Training_Loss:2.3015, Validation_Loss:2.30148129, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 7 => Training_Loss:2.3015, Validation_Loss:2.30147279, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 8 => Training_Loss:2.3015, Validation_Loss:2.30146418, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 9 => Training_Loss:2.3015, Validation_Loss:2.30145525, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 10 => Training_Loss:2.3015, Validation_Loss:2.30144764, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 11 => Training_Loss:2.3015, Validation_Loss:2.30143878, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 12 => Training_Loss:2.3015, Validation_Loss:2.30138993, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 13 => Training_Loss:2.3014, Validation_Loss:2.30138159, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 14 => Training_Loss:2.3014, Validation_Loss:2.30134518, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 15 => Training_Loss:2.3014, Validation_Loss:2.30134108, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 16 => Training_Loss:2.3014, Validation_Loss:2.30133694, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 17 => Training_Loss:2.3014, Validation_Loss:2.30133300, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 18 => Training_Loss:2.3014, Validation_Loss:2.30132892, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 19 => Training_Loss:2.3014, Validation_Loss:2.30132499, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 20 => Training_Loss:2.3014, Validation_Loss:2.30132068, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 21 => Training_Loss:2.3014, Validation_Loss:2.30131724, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 22 => Training_Loss:2.3014, Validation_Loss:2.30133455, Validation_Acc:0.10280000, Stop_counter:1\n",
      "Epoch 23 => Training_Loss:2.3014, Validation_Loss:2.30130256, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 24 => Training_Loss:2.3014, Validation_Loss:2.30130261, Validation_Acc:0.11350000, Stop_counter:1\n",
      "Epoch 25 => Training_Loss:2.3014, Validation_Loss:2.30130261, Validation_Acc:0.11350000, Stop_counter:2\n",
      "Epoch 26 => Training_Loss:2.3014, Validation_Loss:2.30130261, Validation_Acc:0.11350000, Stop_counter:3\n",
      "Epoch 27 => Training_Loss:2.3014, Validation_Loss:2.30130261, Validation_Acc:0.11350000, Stop_counter:4\n",
      "\n",
      "No improvement in validation data for the specified number of early stopping rounds. Stopping training.\n",
      "Epoch 0 => Training_Loss:2.3031, Validation_Loss:2.30151501, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 1 => Training_Loss:2.3018, Validation_Loss:2.30150525, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 2 => Training_Loss:2.3017, Validation_Loss:2.30150051, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 3 => Training_Loss:2.3017, Validation_Loss:2.30149270, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 4 => Training_Loss:2.3017, Validation_Loss:2.30148597, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 5 => Training_Loss:2.3016, Validation_Loss:2.30147788, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 6 => Training_Loss:2.3016, Validation_Loss:2.30146462, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 7 => Training_Loss:2.3016, Validation_Loss:2.30146247, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 8 => Training_Loss:2.3016, Validation_Loss:2.30144994, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 9 => Training_Loss:2.3015, Validation_Loss:2.30144507, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 10 => Training_Loss:2.3015, Validation_Loss:2.30143856, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 11 => Training_Loss:2.3015, Validation_Loss:2.30143385, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 12 => Training_Loss:2.3015, Validation_Loss:2.30142823, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 13 => Training_Loss:2.3015, Validation_Loss:2.30142296, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 14 => Training_Loss:2.3015, Validation_Loss:2.30141842, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 15 => Training_Loss:2.3015, Validation_Loss:2.30141365, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 16 => Training_Loss:2.3015, Validation_Loss:2.30140963, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 17 => Training_Loss:2.3015, Validation_Loss:2.30133882, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 18 => Training_Loss:2.3014, Validation_Loss:2.30133631, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 19 => Training_Loss:2.3014, Validation_Loss:2.30133274, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 20 => Training_Loss:2.3014, Validation_Loss:2.30132970, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 21 => Training_Loss:2.3014, Validation_Loss:2.30132621, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 22 => Training_Loss:2.3014, Validation_Loss:2.30132282, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 23 => Training_Loss:2.3014, Validation_Loss:2.30131957, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 24 => Training_Loss:2.3014, Validation_Loss:2.30131624, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 25 => Training_Loss:2.3014, Validation_Loss:2.30131291, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 26 => Training_Loss:2.3014, Validation_Loss:2.30130953, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 27 => Training_Loss:2.3014, Validation_Loss:2.30130614, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 28 => Training_Loss:2.3014, Validation_Loss:2.30130271, Validation_Acc:0.10280000, Stop_counter:0\n",
      "Epoch 29 => Training_Loss:2.3014, Validation_Loss:2.30129942, Validation_Acc:0.11350000, Stop_counter:0\n",
      "Epoch 30 => Training_Loss:2.3014, Validation_Loss:2.30132656, Validation_Acc:0.11350000, Stop_counter:1\n",
      "Epoch 31 => Training_Loss:2.3014, Validation_Loss:2.30130248, Validation_Acc:0.11350000, Stop_counter:2\n",
      "Epoch 32 => Training_Loss:2.3014, Validation_Loss:2.30130256, Validation_Acc:0.11350000, Stop_counter:3\n",
      "Epoch 33 => Training_Loss:2.3014, Validation_Loss:2.30130256, Validation_Acc:0.11350000, Stop_counter:4\n",
      "\n",
      "No improvement in validation data for the specified number of early stopping rounds. Stopping training.\n",
      "Study statistics: \n",
      "  Number of finished trials:  2\n",
      "  Number of pruned trials:  0\n",
      "  Number of complete trials:  2\n",
      "Best trial:\n",
      "  Value:  0.1135\n",
      "  Params: \n",
      "    n_layers: 55\n",
      "    n_units_l0: 4\n",
      "    n_units_l1: 8\n",
      "    n_units_l2: 9\n",
      "    n_units_l3: 7\n",
      "    n_units_l4: 5\n",
      "    n_units_l5: 5\n",
      "    n_units_l6: 9\n",
      "    n_units_l7: 5\n",
      "    n_units_l8: 4\n",
      "    n_units_l9: 8\n",
      "    n_units_l10: 10\n",
      "    n_units_l11: 4\n",
      "    n_units_l12: 7\n",
      "    n_units_l13: 9\n",
      "    n_units_l14: 8\n",
      "    n_units_l15: 9\n",
      "    n_units_l16: 6\n",
      "    n_units_l17: 10\n",
      "    n_units_l18: 9\n",
      "    n_units_l19: 7\n",
      "    n_units_l20: 4\n",
      "    n_units_l21: 6\n",
      "    n_units_l22: 8\n",
      "    n_units_l23: 7\n",
      "    n_units_l24: 7\n",
      "    n_units_l25: 8\n",
      "    n_units_l26: 7\n",
      "    n_units_l27: 8\n",
      "    n_units_l28: 8\n",
      "    n_units_l29: 9\n",
      "    n_units_l30: 7\n",
      "    n_units_l31: 10\n",
      "    n_units_l32: 6\n",
      "    n_units_l33: 4\n",
      "    n_units_l34: 6\n",
      "    n_units_l35: 4\n",
      "    n_units_l36: 9\n",
      "    n_units_l37: 4\n",
      "    n_units_l38: 8\n",
      "    n_units_l39: 7\n",
      "    n_units_l40: 9\n",
      "    n_units_l41: 5\n",
      "    n_units_l42: 9\n",
      "    n_units_l43: 6\n",
      "    n_units_l44: 9\n",
      "    n_units_l45: 6\n",
      "    n_units_l46: 10\n",
      "    n_units_l47: 6\n",
      "    n_units_l48: 5\n",
      "    n_units_l49: 6\n",
      "    n_units_l50: 4\n",
      "    n_units_l51: 9\n",
      "    n_units_l52: 5\n",
      "    n_units_l53: 6\n",
      "    n_units_l54: 10\n"
     ]
    }
   ],
   "source": [
    "optuna.logging.set_verbosity(optuna.logging.WARNING)\n",
    "\n",
    "random_sampler=optuna.samplers.RandomSampler(seed = 10)\n",
    "\n",
    "study = optuna.create_study(sampler = random_sampler, direction=\"maximize\")\n",
    "study.optimize(objective, n_trials=10, timeout=1500, show_progress_bar = True)\n",
    "\n",
    "pruned_trials = study.get_trials(deepcopy=False, states=[TrialState.PRUNED])\n",
    "complete_trials = study.get_trials(deepcopy=False, states=[TrialState.COMPLETE])\n",
    "\n",
    "print(\"Study statistics: \")\n",
    "print(\"  Number of finished trials: \", len(study.trials))\n",
    "print(\"  Number of pruned trials: \", len(pruned_trials))\n",
    "print(\"  Number of complete trials: \", len(complete_trials))\n",
    "\n",
    "print(\"Best trial:\")\n",
    "trial = study.best_trial\n",
    "\n",
    "print(\"  Value: \", trial.value)\n",
    "\n",
    "print(\"  Params: \")\n",
    "for key, value in trial.params.items():\n",
    "    print(\"    {}: {}\".format(key, value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a validation loss of 2.30 and an accuracy of 11%, random search proved much worse than bayesian search. The model it proposes also have a lot more parameters than the one proposed by the bayesian approach. This is not too surprising since random search is generally a more primitive method for finding the optimal points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-21T06:47:41.684581Z",
     "iopub.status.busy": "2022-01-21T06:47:41.684254Z",
     "iopub.status.idle": "2022-01-21T06:47:41.725515Z",
     "shell.execute_reply": "2022-01-21T06:47:41.724666Z",
     "shell.execute_reply.started": "2022-01-21T06:47:41.684539Z"
    }
   },
   "outputs": [],
   "source": [
    "joblib.dump(study, study_path + '/mnist_optuna_random_10trials2.pkl')\n",
    "study = joblib.load(study_path + '/mnist_optuna_random_10trials2.pkl')\n",
    "df_random = study.trials_dataframe()\n",
    "df_random.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our simplest baseline, random search, returned a weak accuracy value and a considerably higher valid loss. This may also be due to the timeout parameter, which we set it to 1500 seconds, causing only 2 hyperparameter tuning rounds to take place. The bayesian approach however, returned a pretty good accuracy score and a low valdation loss in the given time limit; probably its output would get better if we increase the timeout value so that the search process will include more trials."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}