pytorch_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b3abcffa",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# An Introduction to PyTorch\n",
    "by Tobias Jülg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaf25d63",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is PyTorch?\n",
    "- Extended version of NumPy for deeplearning\n",
    "- `np.array` $\\rightarrow$ `torch.tensor`\n",
    "    - GPU accelorated calculations\n",
    "    - Automatic gradient tracking\n",
    "- Common optimizer for weight updates\n",
    "- Define networks with python classes\n",
    "- Preimplemented common layers and loss functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc85d44",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Content\n",
    "- Tensors: `torch.tensor`\n",
    "- Datasets & Dataloaders: `torch.utils.data`\n",
    "- Transformers: `torchvision.transforms`\n",
    "- Building Networks: `torch.nn`\n",
    "- Optimizer: `torch.optim`\n",
    "- Training Loop\n",
    "- Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "411cedc8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `torch.tensor`\n",
    "- Very similar to `np.array`: manipulation exactly as in numpy\n",
    "- Can be executed on the GPU\n",
    "    - $\\rightarrow$ `.to(device)` method\n",
    "- Compatible to `np.array` if on CPU\n",
    "- Can track gradients of calculations\n",
    "    - $\\rightarrow$ `requires_grad=True` attribute\n",
    "- `b = a` always results in reference\n",
    "    - $\\rightarrow$ use `b = a.clone()` to get a separate memory copy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4f800052",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 2],\n",
       "        [3, 4]])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "tensor = torch.tensor([[1, 2], [3, 4]])\n",
    "tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2d231abc",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 2])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# access dimensions\n",
    "tensor.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbfa8919",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Numpy compatiblity\n",
    "`.numpy()` and `torch.from_numpy(...)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "733343f9",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2],\n",
       "       [3, 4]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np_array = tensor.numpy()\n",
    "np_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "62b0ae2c",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 2],\n",
       "        [3, 4]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.from_numpy(np_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1d5b40a",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Data types\n",
    "- The whole tensor has the same data type: `dtype`\n",
    "- Set it with `.type(...)`\n",
    "- Possible types: `torch.int`, `torch.float32`, `torch.float64`\n",
    "- $\\rightarrow$ Standard for DL is float32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "398784fc",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# access data type\n",
    "tensor.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "923fede4",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# cast to new data type\n",
    "tensor.type(torch.float32).dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d63cca8d",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Tensors on the GPU\n",
    "- `.device` gives us the current data location\n",
    "- `.to(...)` moves data between devices\n",
    "- Tensors need to be on the same device for calculations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c95f3790",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# move our tensor to the GPU if available\n",
    "if torch.cuda.is_available():\n",
    "    tensor = tensor.to(\"cuda\")\n",
    "    # torch also supports AMDs ROCm in beta\n",
    "tensor.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "25210790",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2., 3.],\n",
       "        [4., 5.]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# error if GPU available\n",
    "tensor + torch.ones((2, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21d9afac",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Tensor/Matrix Calculations:\n",
    "- All common operations default to **element-wise**\n",
    "- Special methods/operators matrix calculations\n",
    "- e.g. `@` for matrix multiplication\n",
    "\n",
    "$$\\begin{bmatrix}\n",
    "1 & 2\\\\\n",
    "3 & 4\n",
    "\\end{bmatrix}\n",
    "\\begin{pmatrix}\n",
    "1\\\\\n",
    "1\n",
    "\\end{pmatrix}+\n",
    "\\begin{pmatrix}\n",
    "1\\\\\n",
    "1\n",
    "\\end{pmatrix} = \n",
    "\\begin{pmatrix}\n",
    "4\\\\\n",
    "8\n",
    "\\end{pmatrix}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5375b8bf",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([4, 8])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = torch.tensor([[1, 2], [3, 4]])\n",
    "tensor @ torch.tensor([1, 1]) + torch.tensor([1, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e0ea602e",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([0, 1, 1]), tensor([1, 0, 1]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# advanced indexing\n",
    "torch.where(tensor > 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f65f616",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Squeeze and Unsqueeze Dimensions\n",
    "- Important to match dimensions\n",
    "- `.squeeze(dim)` can be used to remove a dimension with 1 entry\n",
    "- `.unsqueeze(dim)` can be used to add a dimension with 1 entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "da4928f1",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 4])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = torch.tensor([[1, 2, 3, 4]])\n",
    "tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8adceb0a",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 4, 1])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = tensor.unsqueeze(-1)\n",
    "tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "59013330",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 1])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor.squeeze(0).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "382b84fb",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Automatic Differentiation\n",
    "- Tensors can track calculations: `requires_grad` attribute\n",
    "- Computation graph:\n",
    "    - Dynamic graph (PyTorch): good for debugging\n",
    "    - Static graph (Tensorflow): good for production\n",
    "- `backward()`: traverse computational graph in reverse order, calculates gradients using the chain rule\n",
    "- Gradients accessible with `grad` attribute\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf7227de",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "![](graph.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a21e7915",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Example:\n",
    "\n",
    "$x=1, w=1, y=2$\n",
    "\n",
    "$z = x\\cdot w = 1$\n",
    "\n",
    "$loss = (z - y)^2 = 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3792636f",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "![](graph2.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6f9a360f",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x = torch.ones(1)\n",
    "y = torch.ones(1)*2\n",
    "w = torch.ones(1, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "74886e1c",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.], grad_fn=<PowBackward0>)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = w * x\n",
    "loss = (z-y)**2\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b32e4215",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$\\frac{\\partial loss}{\\partial w} = \\frac{\\partial loss}{\\partial z}\\cdot\\frac{\\partial z}{\\partial w} = 2(z-y) \\cdot x = -2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1fdd03f9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-2.])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# .grad needs to be None before calling .backward()\n",
    "loss.backward()\n",
    "w.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "703e657a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Toy Problem\n",
    "Learn the function $f(x) = x^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dac1be99",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `torch.utils.data`\n",
    "Module for data representation, loading and splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3bb86e3a",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader, random_split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72baf590",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Dataset\n",
    "- Represents data in form of a python class\n",
    "- Map-Style\n",
    "    - Index maps to sample\n",
    "    - Inherit from `Dataset`\n",
    "    - Implement `__getitem__` and `__len__` methods\n",
    "- Iterator-Syle: No fix index\n",
    "    - No index mapping given e.g. data is randomly generated\n",
    "    - Inherit from `IterableDataset`\n",
    "    - Implement `__iter__` method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e3a2d8",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Learn $x^2$ function for $x\\in [-50, 50]$. \n",
    "Dataset should contain 10000 sample:\n",
    "\n",
    "Map Index $\\{0, ..., 9999\\} \\rightarrow [-50, 50]$\n",
    "\n",
    "$\\rightarrow$ **Map-Style dataset**\n",
    "\n",
    "$$f(i) = (\\frac{i}{10000}\\cdot 100 - 50)^2$$\n",
    "\n",
    "Normalize input to $[-1, 1]$\n",
    "\n",
    "$$norm(i) = \\frac{i}{10000}\\cdot 2 - 1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "c7218da9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class MyDataset(Dataset):\n",
    "    def __getitem__(self, i):\n",
    "        if i < 0 or i >= len(self):\n",
    "            raise IndexError\n",
    "        # normilize input to [0, 1]\n",
    "        # always return vectors, otherwise nasty broadcast errors might follow\n",
    "        return torch.tensor([i/10_000*2-1], dtype=torch.float32), torch.tensor([(i/10_000*100-50)**2], dtype=torch.float32)\n",
    "    def __len__(self):\n",
    "        return int(10_000.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97b3b47f",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "class MyDataset(Dataset):\n",
    "    def __getitem__(self, i):\n",
    "        if i < 0 or i >= len(self):\n",
    "            raise IndexError\n",
    "        # normilize input to [0, 1]\n",
    "        return torch.tensor([i/10_000*100-50], dtype=torch.float32), torch.tensor([(i/10_000*100-50)**2], dtype=torch.float32)\n",
    "        #return torch.tensor([i/10_000*2-1], dtype=torch.float32), torch.tensor([(i/10_000*100-50)**2], dtype=torch.float32)\n",
    "    def __len__(self):\n",
    "        return 10_000"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b8dfde5",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Splitting\n",
    "`random_split` takes\n",
    "- dataset to split\n",
    "- number of elements in each split\n",
    "- (optionally) generator to fix random seed\n",
    "\n",
    "and returns sub-datasets\n",
    "\n",
    "Example 0.6/0.2/0.2 split:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "a095b865",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dataset = MyDataset()\n",
    "train_size = int(0.6 * len(dataset))\n",
    "val_size = int(0.2 * len(dataset))\n",
    "test_size = len(dataset) - train_size - val_size\n",
    "train_data, val_data, test_data = random_split(dataset,\n",
    "                                               [train_size, val_size, test_size],\n",
    "                                               generator=torch.Generator().manual_seed(42))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3565b129",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Dataloader\n",
    "- Fetches data from dataset\n",
    "- Can use several fetch workers\n",
    "- Prepares batches: Iterator of batches\n",
    "- Optionally reshuffels batch data in each epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "c88bac21",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "train = DataLoader(train_data, batch_size=64, shuffle=True, num_workers=0, drop_last=False)\n",
    "val = DataLoader(val_data, batch_size=64, shuffle=True)\n",
    "test = DataLoader(test_data, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80ba8e49",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `torch.nn`: Building Models\n",
    "- Define networks as a python class\n",
    "- Recursive network construtction with `nn.module`\n",
    "- $\\rightarrow$ Every network unit is module and can consist of modules itself\n",
    "- Defines layers, calculation flow through the layers and parameters\n",
    "- Can be called like a function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "1795563d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "from torch.nn import Module, Linear, MSELoss, ReLU, L1Loss\n",
    "class MyModel(Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.layer1 = Linear(1, 100, bias=True)\n",
    "        self.act1 = ReLU()\n",
    "        self.layer2 = Linear(100, 100)\n",
    "        self.act2 = ReLU()\n",
    "        self.layer3 = Linear(100, 1)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.layer3(self.act2(self.layer2(self.act1(self.layer1(x)))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "154f5685",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.2906],\n",
       "        [0.2494]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# already initializes weights\n",
    "model = MyModel()\n",
    "# acts like a function, input always needs to be batch matrix (single sample: vector)\n",
    "model(torch.tensor([[10.0], [2.0]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d6b0262",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- `parameters()`: Iterates over all learnable weights\n",
    "- `state_dict()`: Python dictionary which contains the current weight values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "41ae58ed",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.9592],\n",
       "        [ 0.0726],\n",
       "        [-0.9228],\n",
       "        [ 0.6859],\n",
       "        [-0.7280],\n",
       "        [ 0.7114],\n",
       "        [-0.6910],\n",
       "        [-0.5108],\n",
       "        [-0.9658],\n",
       "        [ 0.2034],\n",
       "        [ 0.1898],\n",
       "        [-0.5138],\n",
       "        [ 0.2864],\n",
       "        [-0.8050],\n",
       "        [-0.5771],\n",
       "        [-0.9390],\n",
       "        [ 0.1982],\n",
       "        [ 0.3592],\n",
       "        [ 0.7640],\n",
       "        [ 0.3410],\n",
       "        [ 0.1092],\n",
       "        [-0.1244],\n",
       "        [-0.6911],\n",
       "        [ 0.0131],\n",
       "        [-0.6193],\n",
       "        [ 0.4877],\n",
       "        [ 0.5649],\n",
       "        [-0.2504],\n",
       "        [ 0.6222],\n",
       "        [ 0.0133],\n",
       "        [ 0.4826],\n",
       "        [-0.3736],\n",
       "        [ 0.2075],\n",
       "        [ 0.6168],\n",
       "        [ 0.5092],\n",
       "        [-0.4441],\n",
       "        [-0.6959],\n",
       "        [-0.7803],\n",
       "        [-0.3449],\n",
       "        [-0.1461],\n",
       "        [-0.8836],\n",
       "        [-0.2711],\n",
       "        [ 0.3792],\n",
       "        [ 0.2807],\n",
       "        [-0.6859],\n",
       "        [-0.8743],\n",
       "        [ 0.6515],\n",
       "        [-0.6292],\n",
       "        [ 0.2332],\n",
       "        [ 0.0559],\n",
       "        [ 0.7979],\n",
       "        [-0.8890],\n",
       "        [-0.5918],\n",
       "        [ 0.6524],\n",
       "        [ 0.0057],\n",
       "        [ 0.7422],\n",
       "        [-0.1767],\n",
       "        [-0.7494],\n",
       "        [ 0.7105],\n",
       "        [ 0.3834],\n",
       "        [ 0.3200],\n",
       "        [-0.5391],\n",
       "        [ 0.2289],\n",
       "        [ 0.6570],\n",
       "        [ 0.7407],\n",
       "        [-0.4248],\n",
       "        [-0.8660],\n",
       "        [ 0.1203],\n",
       "        [ 0.2631],\n",
       "        [-0.3658],\n",
       "        [ 0.7171],\n",
       "        [-0.1185],\n",
       "        [-0.9091],\n",
       "        [ 0.8048],\n",
       "        [ 0.4602],\n",
       "        [ 0.5065],\n",
       "        [ 0.8024],\n",
       "        [-0.1908],\n",
       "        [ 0.2757],\n",
       "        [-0.0188],\n",
       "        [-0.7182],\n",
       "        [ 0.0566],\n",
       "        [-0.8035],\n",
       "        [ 0.8188],\n",
       "        [ 0.4962],\n",
       "        [ 0.4726],\n",
       "        [ 0.9833],\n",
       "        [ 0.7031],\n",
       "        [-0.2274],\n",
       "        [-0.6178],\n",
       "        [-0.0881],\n",
       "        [-0.6924],\n",
       "        [-0.5587],\n",
       "        [ 0.6295],\n",
       "        [-0.6190],\n",
       "        [ 0.1547],\n",
       "        [-0.3035],\n",
       "        [-0.9886],\n",
       "        [ 0.2831],\n",
       "        [-0.4860]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Model's state: weight values\n",
    "model.state_dict()[\"layer1.weight\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f53ef7c",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Exkurs Weight Initialization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6e11d7b",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Loss Functions\n",
    "- Defines our optimization problem\n",
    "- Loss function depends on the problem:\n",
    "    - Regression:\n",
    "        - Mean squared error: `MSELoss`\n",
    "        - Mean absolut error: `L1Loss`\n",
    "        - Combination: `SmoothL1Loss` (less sensitive to outliers)\n",
    "    - Classification:\n",
    "        - Binary cross entropy: `BCELoss`\n",
    "        - Cross entropy: `CrossEntropyLoss` (for n classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "a70b1e8d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# we want to learn a function -> regression, usually MSE is a good pick\n",
    "loss_fn = MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62c430a6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `torch.optim`: Optimizer\n",
    "- Optimizer: Strategy to update the weights given the gradients\n",
    "- Takes parameters to optimize\n",
    "- Additional optimizer specific parateters\n",
    "- `.step()` function updates weights\n",
    "- `.zero_grad()` empties gradients -> needed before every backward pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09aba2a3",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- SGD:\n",
    "    - Stochastic Gradient Desent (GD with mini-batches)\n",
    "    - first order momentum possible\n",
    "    - weight decay possible\n",
    "- Adam:\n",
    "    - Adaptive Moment Estimation\n",
    "    - First and second order momentum\n",
    "    - weight decay possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "ba975bb3",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e80729c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training Loop\n",
    "- Epoch: Iterate over all batches\n",
    "- For each batch:\n",
    "    - Pass the batch through the model\n",
    "    - Calculate the loss\n",
    "    - Backpropagate and calculate gradients\n",
    "    - Update weights using the optimizer\n",
    "    - Log Metrics\n",
    "- Forward validation dataset and log metrics\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "b8aa6185",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 1249254.5000000000, val loss: 1193008.0000000000, train loss: 1210606.5000000000\n",
      "Loss: 101630.0000000000, val loss: 82924.8203125000, train loss: 84228.3203125000\n",
      "Loss: 6741.1694335938, val loss: 6251.2939453125, train loss: 6212.5278320312\n",
      "Loss: 1552.9964599609, val loss: 1409.7972412109, train loss: 1447.4603271484\n",
      "Loss: 404.7177429199, val loss: 357.6051330566, train loss: 376.3672180176\n",
      "Loss: 126.3145370483, val loss: 110.0412445068, train loss: 119.5708084106\n",
      "Loss: 48.8971405029, val loss: 42.1058464050, train loss: 46.0612678528\n",
      "Loss: 19.7660541534, val loss: 16.2278652191, train loss: 18.1943893433\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Input \u001b[0;32mIn [45]\u001b[0m, in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      7\u001b[0m loss \u001b[38;5;241m=\u001b[39m loss_fn(y_hat, y)\n\u001b[1;32m      8\u001b[0m optimizer\u001b[38;5;241m.\u001b[39mzero_grad()\n\u001b[0;32m----> 9\u001b[0m \u001b[43mloss\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbackward\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     10\u001b[0m optimizer\u001b[38;5;241m.\u001b[39mstep()\n\u001b[1;32m     11\u001b[0m losses\u001b[38;5;241m.\u001b[39mappend(loss)\n",
      "File \u001b[0;32m~/cloud/Uni/sem-3/idp/venv/lib/python3.9/site-packages/torch/_tensor.py:307\u001b[0m, in \u001b[0;36mTensor.backward\u001b[0;34m(self, gradient, retain_graph, create_graph, inputs)\u001b[0m\n\u001b[1;32m    298\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m has_torch_function_unary(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m    299\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m handle_torch_function(\n\u001b[1;32m    300\u001b[0m         Tensor\u001b[38;5;241m.\u001b[39mbackward,\n\u001b[1;32m    301\u001b[0m         (\u001b[38;5;28mself\u001b[39m,),\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    305\u001b[0m         create_graph\u001b[38;5;241m=\u001b[39mcreate_graph,\n\u001b[1;32m    306\u001b[0m         inputs\u001b[38;5;241m=\u001b[39minputs)\n\u001b[0;32m--> 307\u001b[0m \u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mautograd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbackward\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgradient\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mretain_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcreate_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minputs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/cloud/Uni/sem-3/idp/venv/lib/python3.9/site-packages/torch/autograd/__init__.py:154\u001b[0m, in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)\u001b[0m\n\u001b[1;32m    151\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m retain_graph \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    152\u001b[0m     retain_graph \u001b[38;5;241m=\u001b[39m create_graph\n\u001b[0;32m--> 154\u001b[0m \u001b[43mVariable\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_execution_engine\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun_backward\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    155\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtensors\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgrad_tensors_\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mretain_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcreate_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    156\u001b[0m \u001b[43m    \u001b[49m\u001b[43mallow_unreachable\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43maccumulate_grad\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "for epoch in range(150):\n",
    "    losses = []\n",
    "    model.train()\n",
    "    for batch in train:\n",
    "        X, y = batch\n",
    "        y_hat = model(X)\n",
    "        loss = loss_fn(y_hat, y)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        losses.append(loss)\n",
    "        \n",
    "    # evaluation\n",
    "    losses_val2 = []\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for batch in train:\n",
    "            X, y = batch\n",
    "            y_hat = model(X)\n",
    "            losses_val2.append(loss_fn(y_hat, y))\n",
    "    \n",
    "    # evaluation\n",
    "    losses_val = []\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for batch in val:\n",
    "            X, y = batch\n",
    "            y_hat = model(X)\n",
    "            losses_val.append(loss_fn(y_hat, y))\n",
    "    l, l_val, l_ = torch.stack(losses).mean().item(), torch.stack(losses_val).mean().item(), torch.stack(losses_val2).mean().item()\n",
    "    if epoch % 10 == 0:\n",
    "        print(f\"Loss: {l:.10f}, val loss: {l_val:.10f}, train loss: {l_:.10f}\")\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81cc9bcd",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "X, y = iter(train).next()\n",
    "X.shape, y.shape, model(X).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf2317ac",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "for epoch in range(5):\n",
    "    losses = []\n",
    "    model.train()\n",
    "    for batch in train:\n",
    "        X, y = batch\n",
    "        y_hat = model(X)\n",
    "        # shapes must match!\n",
    "        # y_hat.shape: [64, 1], y.shape: [64]\n",
    "        y = y.unsqueeze(-1)\n",
    "        loss = loss_fn(y_hat, y)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        losses.append(loss)\n",
    "    \n",
    "    # evaluation\n",
    "    losses_val = []\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for batch in train:\n",
    "            X, y = batch\n",
    "            y_hat = model(X)\n",
    "            losses_val.append(loss_fn(y_hat, y.unsqueeze(-1)))\n",
    "    l, l_val = torch.stack(losses).mean().item(), torch.stack(losses_val).mean().item()\n",
    "    print(f\"Loss: {l:.4f}, val loss: {l_val:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a47b4034",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch Lightning\n",
    "- High level framework based on PyTorch (what Keras is to TF)\n",
    "- Phylosophy:\n",
    "    - No room for errors in code that you dont write\n",
    "    - Concentrate on the important part of DL projects\n",
    "    - Still able to access deep components if you want to\n",
    "- Implements base functionality used in most projects\n",
    "    - Training loop\n",
    "    - Logging\n",
    "    - Takes care of pushing data to GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "00c1363c",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import pytorch_lightning as pl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce652a63",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lightning Module\n",
    "- PyTorch module which also defines how training/validation/test steps are performed e.g. `training_step`\n",
    "- Defines optimizer\n",
    "- Defines dataloader\n",
    "- Easy logging\n",
    "- Possible to hook into other steps in the training pipeline e.g. `validation_epoch_end`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "26125102",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class PLModule(pl.LightningModule):\n",
    "    def __init__(self, model, data, hparms):\n",
    "        super().__init__()\n",
    "        self.model = model\n",
    "        self.crit = MSELoss()\n",
    "        self.hparams.update(hparams)\n",
    "        self.data = data\n",
    "        \n",
    "    def training_step(self, batch, batch_idx):\n",
    "        x, y = batch[0], batch[1]\n",
    "        y_hat = self.model(x)\n",
    "        loss = self.crit(y_hat, y)\n",
    "        self.log(\"loss\", loss)\n",
    "        return loss\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        # no need for torch.no_grad() here as lighting handles this\n",
    "        x, y = batch[0], batch[1]\n",
    "        y_hat = self.model(x)\n",
    "        loss = self.crit(y_hat, y)\n",
    "        self.log(\"val_loss\", loss)\n",
    "        \n",
    "    def train_dataloader(self):\n",
    "        return DataLoader(self.data[0], batch_size=64)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return DataLoader(self.data[1], batch_size=64)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return DataLoader(self.data[2], batch_size=64)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        return torch.optim.Adam(self.model.parameters(),\n",
    "                                self.hparams[\"learning_rate\"],\n",
    "                                weight_decay=self.hparams[\"weight_decay\"])\n",
    "\n",
    "hparams = {\"learning_rate\": 1e-3, \"weight_decay\": 1e-6, \"epochs\": 1000}\n",
    "model = MyModel()\n",
    "pl_module = PLModule(model, (train_data, val_data, test_data), hparams)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8601470",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lightning Trainer\n",
    "- Object that defines the training\n",
    "- `.fit(pl_module)` runs the training\n",
    "- Callbacks for\n",
    "    - early stopping\n",
    "    - checkpoints\n",
    "    - logging (default: Tensorboard)\n",
    "    - ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "86b01680",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "\n",
      "  | Name  | Type    | Params\n",
      "----------------------------------\n",
      "0 | model | MyModel | 10.4 K\n",
      "1 | crit  | MSELoss | 0     \n",
      "----------------------------------\n",
      "10.4 K    Trainable params\n",
      "0         Non-trainable params\n",
      "10.4 K    Total params\n",
      "0.042     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation sanity check: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tobi/cloud/Uni/sem-3/idp/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/data_loading.py:132: UserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n",
      "/home/tobi/cloud/Uni/sem-3/idp/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/data_loading.py:132: UserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 4 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f68bb58566d46d29c364b8946426538",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved. New best score: 1201541.625\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 191042.688 >= min_delta = 0.0. New best score: 1010498.938\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 323908.062 >= min_delta = 0.0. New best score: 686590.875\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 212505.344 >= min_delta = 0.0. New best score: 474085.531\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 57293.125 >= min_delta = 0.0. New best score: 416792.406\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 38066.844 >= min_delta = 0.0. New best score: 378725.562\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 41106.969 >= min_delta = 0.0. New best score: 337618.594\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 48782.750 >= min_delta = 0.0. New best score: 288835.844\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 51358.156 >= min_delta = 0.0. New best score: 237477.688\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 50097.703 >= min_delta = 0.0. New best score: 187379.984\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 46222.656 >= min_delta = 0.0. New best score: 141157.328\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 38537.312 >= min_delta = 0.0. New best score: 102620.016\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 29803.023 >= min_delta = 0.0. New best score: 72816.992\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 21646.746 >= min_delta = 0.0. New best score: 51170.246\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 14749.637 >= min_delta = 0.0. New best score: 36420.609\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 9488.611 >= min_delta = 0.0. New best score: 26931.998\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 5945.668 >= min_delta = 0.0. New best score: 20986.330\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 3826.965 >= min_delta = 0.0. New best score: 17159.365\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 2685.286 >= min_delta = 0.0. New best score: 14474.079\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 2073.055 >= min_delta = 0.0. New best score: 12401.024\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1711.419 >= min_delta = 0.0. New best score: 10689.605\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1459.699 >= min_delta = 0.0. New best score: 9229.906\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1254.346 >= min_delta = 0.0. New best score: 7975.561\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1080.885 >= min_delta = 0.0. New best score: 6894.675\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 928.627 >= min_delta = 0.0. New best score: 5966.048\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 796.655 >= min_delta = 0.0. New best score: 5169.393\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 684.326 >= min_delta = 0.0. New best score: 4485.066\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 586.546 >= min_delta = 0.0. New best score: 3898.520\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 503.922 >= min_delta = 0.0. New best score: 3394.599\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 433.166 >= min_delta = 0.0. New best score: 2961.433\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 371.414 >= min_delta = 0.0. New best score: 2590.019\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 320.179 >= min_delta = 0.0. New best score: 2269.841\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 276.378 >= min_delta = 0.0. New best score: 1993.463\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 239.399 >= min_delta = 0.0. New best score: 1754.064\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 208.131 >= min_delta = 0.0. New best score: 1545.933\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 182.356 >= min_delta = 0.0. New best score: 1363.576\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 160.020 >= min_delta = 0.0. New best score: 1203.557\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 140.949 >= min_delta = 0.0. New best score: 1062.607\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 124.231 >= min_delta = 0.0. New best score: 938.376\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 109.744 >= min_delta = 0.0. New best score: 828.632\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 96.887 >= min_delta = 0.0. New best score: 731.745\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 85.562 >= min_delta = 0.0. New best score: 646.183\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 75.309 >= min_delta = 0.0. New best score: 570.874\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 66.150 >= min_delta = 0.0. New best score: 504.724\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 57.980 >= min_delta = 0.0. New best score: 446.744\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 51.067 >= min_delta = 0.0. New best score: 395.676\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 44.898 >= min_delta = 0.0. New best score: 350.778\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 39.303 >= min_delta = 0.0. New best score: 311.475\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 34.344 >= min_delta = 0.0. New best score: 277.132\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 30.030 >= min_delta = 0.0. New best score: 247.101\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 26.458 >= min_delta = 0.0. New best score: 220.643\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 23.230 >= min_delta = 0.0. New best score: 197.413\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 20.387 >= min_delta = 0.0. New best score: 177.026\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 17.978 >= min_delta = 0.0. New best score: 159.048\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 15.916 >= min_delta = 0.0. New best score: 143.131\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 14.115 >= min_delta = 0.0. New best score: 129.016\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 12.531 >= min_delta = 0.0. New best score: 116.485\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 11.153 >= min_delta = 0.0. New best score: 105.332\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 9.988 >= min_delta = 0.0. New best score: 95.344\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 8.833 >= min_delta = 0.0. New best score: 86.511\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 7.961 >= min_delta = 0.0. New best score: 78.549\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 7.138 >= min_delta = 0.0. New best score: 71.411\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 6.446 >= min_delta = 0.0. New best score: 64.965\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 5.906 >= min_delta = 0.0. New best score: 59.059\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 5.356 >= min_delta = 0.0. New best score: 53.703\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 4.880 >= min_delta = 0.0. New best score: 48.823\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 4.452 >= min_delta = 0.0. New best score: 44.371\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 4.017 >= min_delta = 0.0. New best score: 40.354\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 3.642 >= min_delta = 0.0. New best score: 36.713\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 3.318 >= min_delta = 0.0. New best score: 33.394\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 2.923 >= min_delta = 0.0. New best score: 30.471\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 2.645 >= min_delta = 0.0. New best score: 27.827\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 2.324 >= min_delta = 0.0. New best score: 25.502\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 2.098 >= min_delta = 0.0. New best score: 23.405\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1.865 >= min_delta = 0.0. New best score: 21.539\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1.683 >= min_delta = 0.0. New best score: 19.856\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1.537 >= min_delta = 0.0. New best score: 18.319\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1.360 >= min_delta = 0.0. New best score: 16.959\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1.248 >= min_delta = 0.0. New best score: 15.711\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1.144 >= min_delta = 0.0. New best score: 14.567\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1.071 >= min_delta = 0.0. New best score: 13.496\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 1.024 >= min_delta = 0.0. New best score: 12.472\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.921 >= min_delta = 0.0. New best score: 11.551\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.819 >= min_delta = 0.0. New best score: 10.732\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.763 >= min_delta = 0.0. New best score: 9.969\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.650 >= min_delta = 0.0. New best score: 9.319\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.617 >= min_delta = 0.0. New best score: 8.702\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.566 >= min_delta = 0.0. New best score: 8.136\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.545 >= min_delta = 0.0. New best score: 7.591\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.569 >= min_delta = 0.0. New best score: 7.023\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.535 >= min_delta = 0.0. New best score: 6.487\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.469 >= min_delta = 0.0. New best score: 6.018\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.410 >= min_delta = 0.0. New best score: 5.608\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.358 >= min_delta = 0.0. New best score: 5.250\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.319 >= min_delta = 0.0. New best score: 4.931\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.303 >= min_delta = 0.0. New best score: 4.628\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.283 >= min_delta = 0.0. New best score: 4.344\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.265 >= min_delta = 0.0. New best score: 4.079\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.244 >= min_delta = 0.0. New best score: 3.836\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.204 >= min_delta = 0.0. New best score: 3.632\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.156 >= min_delta = 0.0. New best score: 3.476\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.146 >= min_delta = 0.0. New best score: 3.330\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.224 >= min_delta = 0.0. New best score: 3.106\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.305 >= min_delta = 0.0. New best score: 2.800\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.355 >= min_delta = 0.0. New best score: 2.446\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.309 >= min_delta = 0.0. New best score: 2.137\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.211 >= min_delta = 0.0. New best score: 1.926\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.116 >= min_delta = 0.0. New best score: 1.810\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.101 >= min_delta = 0.0. New best score: 1.709\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.094 >= min_delta = 0.0. New best score: 1.614\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.127 >= min_delta = 0.0. New best score: 1.488\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.149 >= min_delta = 0.0. New best score: 1.338\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.135 >= min_delta = 0.0. New best score: 1.204\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.086 >= min_delta = 0.0. New best score: 1.118\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.055 >= min_delta = 0.0. New best score: 1.063\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.043 >= min_delta = 0.0. New best score: 1.019\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.037 >= min_delta = 0.0. New best score: 0.982\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.034 >= min_delta = 0.0. New best score: 0.948\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Metric val_loss improved by 0.018 >= min_delta = 0.0. New best score: 0.931\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5dc1d47824744462ba68ea7f5963f640",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validating: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tobi/cloud/Uni/sem-3/idp/venv/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:688: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...\n",
      "  rank_zero_warn(\"Detected KeyboardInterrupt, attempting graceful shutdown...\")\n"
     ]
    }
   ],
   "source": [
    "from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "\n",
    "\n",
    "early_stop_callback = EarlyStopping(\n",
    "    monitor='val_loss',\n",
    "     =10,\n",
    "    strict=True,\n",
    "    verbose=True,\n",
    "    mode='min'\n",
    ")  \n",
    "model_checkpoint = ModelCheckpoint(\n",
    "    \"ckpts\",\n",
    "    monitor='val_loss',\n",
    "    save_last=True,\n",
    "    save_top_k=1,\n",
    ")\n",
    "callbacks = [early_stop_callback, model_checkpoint]\n",
    "\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    callbacks=callbacks,\n",
    "    max_epochs=hparams[\"epochs\"],\n",
    "    deterministic=True,\n",
    "    #gpus=[0],\n",
    "    #profiler=\"simple\",\n",
    ")\n",
    "trainer.fit(pl_module)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9bf2041",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tensorboard\n",
    "- Website to visualize all sorts of metrics\n",
    "- Live training progress\n",
    "- Can also be opened in Jupyter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "c0276bb9",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-5fd90841f32462df\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-5fd90841f32462df\");\n",
       "          const url = new URL(\"/\", window.location);\n",
       "          const port = 6006;\n",
       "          if (port) {\n",
       "            url.port = port;\n",
       "          }\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%load_ext tensorboard\n",
    "%tensorboard --logdir lightning_logs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf359071",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "f67e9759",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# forward whole dataset\n",
    "with torch.no_grad():\n",
    "    y_hat = [model(i[0]).item() for i in dataset]\n",
    "y = [i[1] for i in dataset]\n",
    "x = [i[0]*50 for i in dataset]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "6986a79a",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA16ElEQVR4nO3deVyU5f7/8deHHUFFEZdEBRVRcAHF3SyzXNKyPetkm2WZVp5f1rGz1fnW+WZ7WWbHTqW2mWWWrWZuZa4ouKOiIoKKqKCisl+/Pxj9UqmgMlwzw+f5ePBg5pp75n4Po7y5dzHGoJRSSp2Ll+0ASimlXJ+WhVJKqQppWSillKqQloVSSqkKaVkopZSqkI/tAM7QoEEDExERYTuGUkq5lTVr1hw0xoSd6TGPLIuIiAgSExNtx1BKKbciIrvP9piuhlJKKVUhLQullFIV0rJQSilVIS0LpZRSFdKyUEopVSGnlYWINBORRSKyWUQ2icijjvGnRSRTRJIdX1eXe86TIpIqIltFZGC58UGOsVQRmeCszEoppc7MmbvOFgOPGWPWikhtYI2IzHc89qox5qXyE4tIDDAciAUuAX4SkTaOhycDVwEZwGoRmWuM2ezE7Eoppcpx2pKFMWafMWat4/YxYAvQ9BxPGQbMNMYUGGN2AalAN8dXqjFmpzGmEJjpmLbK5Z4oZOq3y9i2J8sZL6+UUk61MHk7P23ajzMuPVEt2yxEJAKIB1Y6hsaKyHoReU9E6jnGmgJ7yj0twzF2tvHfz2OUiCSKSGJ2dvYF5fTK2ck9q4ay9cepF/R8pZSyxRiD9zeP0PTL6xEnvL7Ty0JEgoHZwDhjzFFgCtAKiAP2AS9XxXyMMVONMQnGmISwsDMerV6hOpdEsz+gJdF7ZpFfWFwVsZRSqlokb95Kr6KVeDXrClL1deHUshARX8qK4iNjzBcAxpgsY0yJMaYUeIey1UwAmUCzck8Pd4ydbdwZgSmIv4c2pLPq52+dMgullHKG/Yv/g6+U0HzAGKe8vjP3hhLgXWCLMeaVcuNNyk12PbDRcXsuMFxE/EUkEogCVgGrgSgRiRQRP8o2gs91Vu6W/e7mGLXwTnzXWbNQSqkqdeT4SeIOfElqcAKBjaOdMg9n7g3VGxgBbBCRZMfYX4HbRCQOMEAa8ACAMWaTiMwCNlO2J9UYY0wJgIiMBeYB3sB7xphNzgrt5R/EzvDr6LrnU3an7aRFREtnzUopparE2p9m0k8Ok97jOafNQ5yx1dy2hIQEczFnnT2UtpHQab1ZEv4Al933QhUmU0qpqmWMYc2/ryCiJI0Gf98O3he+DCAia4wxCWd6TI/gPoPQiPZsDuxMdMZsCgsLbcdRSqmz2rZlPQnFa8lsectFFUVFtCzOorjLvTTmIBsWzbIdRSmlzipr0dsUGy9aDnLOhu1TtCzOIvby4RygPn5J79mOopRSZ3T8eB4dDnzN5jq9qR3W3Knz0rI4C28fX7Y3u5kO+WvYv3NjxU9QSqlqtmH+DOrJMQJ63u/0eWlZnEPEgNEUGW8yf5psO4pSSv1ByKYZZHg1IarHUKfPS8viHJo2i2RNrT5E7Z1LScFx23GUUuq0tM0raVu0hcyWwxEvb6fPT8uiIl1HUoc8ti2cbjuJUkqdlr3obQqML1EDH6iW+WlZVKDzpUPZQTiBydNsR1FKKQAKjucSk/0dSXX7UT+sScVPqAJaFhXw8/UmtcVwIgq2cnj7cttxlFKKlPnvEUR+tWzYPkXLohKiBtzHceNPlm7oVkrZZgwhG2ewTSLp2O3KaputlkUltGzahGVBV9Iyax4leYdsx1FK1WAZ6xfTongXe6Nux8u7+n6Fa1lUkl/PUfhTyK6f/mM7ilKqBju0ZAp5JpD2g0ZW63y1LCqpZ8++rKUddTbOgNJS23GUUjXQydwDtDu8gLX1BtKgfmi1zlvLopL8fLzIjLqdhsX7OLj+e9txlFI10LZ5U/CjmJC+D1b7vLUszkP8wDs5aOqQu2SK7ShKqZqmtJSGWz9hvXcMHeJ7VPvstSzOQ3iDEFaEDKVlzlKKDqXZjqOUqkHSVn1Dk9J95LQbgTjhGtsV0bI4T/X6jsIY2D1Pd6NVSlWf48v+wyFTh/hBd1qZv5bFeeoRH8ev3l0JS/0Uigtsx1FK1QDHDqTR9sivrA+7hjrBwVYyaFmcJ28vITf2TuqWHmH/Cr0wklLK+Xb+MBkBmvR/yFoGLYsL0POqm0gzjSlcPtV2FKWUhzPFhYTv+py1fl1o2669tRxaFhcgrE4gaxteT/Pj6ynIWGc7jlLKg6Uu/YxQc5iTne62mkPL4gI1u+J+8o0vGT++YTuKUsqDla76L5mEkXDlrVZzaFlcoIS2LVns15dL0r+G/CO24yilPNDh9M1En1hLyiU3EBjgZzWLlsUFEhEK4+8lkHz2/jzNdhyllAdK//FNiow3LQdUzwWOzkXL4iJc1m8g600rvNe8C8bYjqOU8iDF+cdpmfEVibV6ExnRynYcLYuLUTfQl+3Nb6VRwW6OpiyyHUcp5UG2/PQ+dcjDu9t9tqMAWhYXrdOge8gxwRxY8KbtKEopT2EMweveY4c0p0vfa2ynAbQsLlrrpg35tfYgIg4uojg303YcpZQH2J28kMiiHWRE3YF3NV7g6FxcI4Wbq3vpA/hQyq55b9mOopTyALmLJ3PEBNFx8CjbUU7TsqgCvbp2ZYVXPKFbP4GSIttxlFJu7GhWOjG5i1nXYCj16tWzHec0LYsqcOp8UfVLD5GxYrbtOEopN5b6/SS8KaXJVQ/bjvIbTisLEWkmIotEZLOIbBKRRx3j9UVkvohsd3yv5xgXEZkkIqkisl5EOpd7rbsc028Xkbuclfli9Bh4G5mmAfnL9BrdSqkLU1KYT0TaLJICuhHVtoPtOL/hzCWLYuAxY0wM0AMYIyIxwARggTEmCljguA8wGIhyfI0CpkBZuQBPAd2BbsBTpwrGlYQEB7KhyQ20Pr6Wo3s22o6jlHJDWxbMoD5HKE6433aUP3BaWRhj9hlj1jpuHwO2AE2BYcB0x2TTgesct4cBM0yZFUCIiDQBBgLzjTGHjTE5wHxgkLNyX4xWA0dTaLzZPU93o1VKnb/ApP+SRlM697vedpQ/qJZtFiISAcQDK4FGxph9jof2A40ct5sCe8o9LcMxdrbx389jlIgkikhidnZ21b6BSoqKbMmqwL5EZnxFSX6elQxKKfe0Z8MvtCrcyu5Wt+Pr42M7zh84vSxEJBiYDYwzxhwt/5gxxgBVcp4MY8xUY0yCMSYhLCysKl7ygnh3v59gTpAy/z1rGZRS7ufQwjfIM4G0HzLadpQzcmpZiIgvZUXxkTHmC8dwlmP1Eo7vBxzjmUCzck8Pd4ydbdwldb10ENulBUHrp+n5opRSlXLsUCaxh38iKXQwofVDbcc5I2fuDSXAu8AWY8wr5R6aC5zao+ku4Kty43c69orqARxxrK6aBwwQkXqODdsDHGMuycfHm8yoPxFRtIO0dYttx1FKuYGt376Br5TQsP8jtqOclTOXLHoDI4ArRCTZ8XU1MBG4SkS2A1c67gN8B+wEUoF3gIcAjDGHgWeA1Y6v/3GMuay4q+/nmAkkd9Ek21GUUi6uuDCfiJ0zSfJPIDo23nacs3LaVhRjzFJAzvJw/zNMb4AxZ3mt9wC32QgQElKfpQ2vo8eBTzmUuYPQpvZPL6yUck3r539IZ3JI7+o6p/Y4Ez2C20maDxoHwM7vXrUbRCnluoyhdtJ/yJAmdOp3k+0056Rl4STNW7VlbXBf2mbOJj8v13YcpZQL2rrqR6KKt5HR9l68vb1txzknLQsnCuz7KLU5wabvptiOopRyQfk/v04Otekw5EHbUSqkZeFE7bv1Y7NPOy7Z8j6mpNh2HKWUC9m3Yz0d8paxqektBAXXsR2nQloWTiQiHIsfRROTxaZFn9iOo5RyIXt/eJkifIgaOs52lErRsnCy+KtGkElDfFe9bTuKUspF5B3eR+yBb1kTMpBGTZrbjlMpWhZO5ufny67WI4gu3EjauiW24yilXMDWr18jQIoIverPtqNUmpZFNWg/ZCzHTCA5C163HUUpZVlx/nFa7vqYRP/uRLdPsB2n0rQsqkFIvfqsa3gdHY4s4mBmqu04SimLtsybSj2OUtpzrO0o50XLopo0G1y2uJn27SsVTKmU8lSmtIT6698hxas1XS4dajvOedGyqCYtWkazJvhy2u79ghNHD9mOo5SyYNsvn9O0JJMD7e/H29u9fv26V1o3F3zFYwRzks1z9RQgStVIy95gL2F0G3KP7STnTcuiGsV26UOSXxciUz+gKP+47ThKqWq0e90Sogs2kNpyBAH+/rbjnDcti2pW2mscoeSy8Ts97kKpmiTnp1c4amrR8ZqHbUe5IFoW1Sz+0qFs8W5Do41TMSVFtuMoparB/t0pdDi6hA2NbySkXn3bcS6IlkU18/L2Iid+DJeU7mfzgg9tx1FKVYPd375MCV60vOb/2Y5ywbQsLEgYeAe7pSm1Vr2h1+lWysMdOZRFh6yvWBdyJU3CW9qOc8G0LCzw8/Vhd9v7iCzewfYVc23HUUo50ZavXqKWFNBgwHjbUS6KloUlXYY+QBb1KVmiB+kp5alOHsulXfrHJAX2JDK2m+04F0XLwpKgoCC2tBhB2/xk9mz4xXYcpZQTbPrmdeqSh+/lj9uOctG0LCzqMOxRjpggcue/YDuKUqqKFRecIGLb+6zzjSO22xW241w0LQuLQuuHktT4JmKP/EL2ro224yilqtCm76bQwORQ0HMcImI7zkXTsrCs9dDxFOJDxrcTbUdRSlWR0uIiGq1/my3ebUi4bJjtOFVCy8Ky8GbNSaw/lNjs78jZl2Y7jlKqCmyY9x6NzQGOJjyCl5udMPBsPONduLnwIU/ghWH7XN12oZS7M6UlhKx9kx3Sgi5X3WY7TpXRsnABEa1jSKrTj9i9szlyaL/tOEqpi7Bx4Se0KEnnQKcx+Pj42I5TZbQsXETooAkEST4pc563HUUpdYFMaSmBK14jQxrT5Wr3Ow35uWhZuIiWsd1IDOpLbMYnHMs5YDuOUuoCbP51Lq2Lt7On3Sj8/Pxsx6lSWhYupM7AvxHMSV26UMpNyS8vk0V94q8ZbTtKldOycCFtOvZgdWAf2qZ/zIkjB23HUUqdh5QV3xNTuJ6dUfcSEFjLdpwq57SyEJH3ROSAiGwsN/a0iGSKSLLj6+pyjz0pIqkislVEBpYbH+QYSxWRCc7K6yqCBjxJbU6Q8qXuGaWUOyldPJFsQuh03TjbUZzCmUsW04BBZxh/1RgT5/j6DkBEYoDhQKzjOW+JiLeIeAOTgcFADHCbY1qPFRPfh9UBvYja9QH5xw7bjqOUqoQdq38kJj+Zra3upVZQbdtxnMJpZWGM+Rmo7G+7YcBMY0yBMWYXkAp0c3ylGmN2GmMKgZmOaT2aX/8JunShlBspXPC/HKSuxy5VgJ1tFmNFZL1jNVU9x1hTYE+5aTIcY2cb/wMRGSUiiSKSmJ2d7Yzc1aZjQl9W+/eg5Y4ZFOTl2I6jlDqHnYk/0i4/iU2R91K7dl3bcZymustiCtAKiAP2AS9X1QsbY6YaYxKMMQlhYWFV9bJWiAg+/SZQh+Ns+fJF23GUUudQ8FPZUkX8DX+2HcWpqrUsjDFZxpgSY0wp8A5lq5kAMoFm5SYNd4ydbdzjxXW/nET/bkSmTidfly6Uckmpq8uWKra0vJc6HrxUAdVcFiLSpNzd64FTe0rNBYaLiL+IRAJRwCpgNRAlIpEi4kfZRvAacR1SEcGv/1+pSx4b57xkO45S6gwKFpxaqvh/tqM4nTN3nf0EWA5Ei0iGiIwEXhCRDSKyHugH/BnAGLMJmAVsBn4AxjiWQIqBscA8YAswyzFtjdCxWz+S/LvResc0Th7LtR1HKVXO1pXziM1PYmurkQQH17Edx+nEGGM7Q5VLSEgwiYmJtmNUic2rFxLz7fWsbPkw3e981nYcpZTDxucup0nBTgLGbyTIQ8pCRNYYYxLO9Jgewe3iYrpewbqABNrsnMZxXbpQyiVsXjGP9gVJbG890mOKoiJaFm4g4Mq/UY9jbJit54xSyjpjkIXPcJAQOl3v+dsqTtGycAPRCVeQFNiTmLRpekZapSzb8ssc2hVuYHv0gwR66NHaZ6Jl4SaCBz9NsDlJyux/246iVI1lSkvw//lZ9tKQ+OvH2Y5TrbQs3ERUxx6srn0F7TM+5uiBDNtxlKqRtiz4gJbFO9jR/hECAgJtx6lWWhZuJHToU/iaYrbPftp2FKVqnNLiIuosf4Gd0oxu1z5gO06107JwI63bdmJlvSF02P8F2Xu22Y6jVI2y/tsphJdmkpXwOP4edhW8ytCycDMR1z+NwYv0L/5pO4pSNUZRwQkuSX6NLd7RdBs0wnYcK7Qs3EzTFq1JbHQTcYd/IGNbku04StUI6+e8QkNziJOX/hVv75r5a7Nmvms3F33jP8nHn+y5/7AdRSmPd/JYLi1T/sM6v3jiL/P4y+mclZaFG2rQ6BLWN7uD+LxfSF272HYcpTzaxtn/Sz2O4nvVPxER23Gs0bJwU+1v/huHqEvRD38HDzy/l1Ku4OjB/bRLm8GaWn2I6XqF7ThWaVm4qdp165MS/RDtCjewefEs23GU8kgps/9FoMknZOi/bEexrsKyEJGHy13+VLmQLtePI10uIXjpM5iSIttxlPIo2Xu2Ebd3FmvqDqBVzBlPxFqjVGbJohGwWkRmicggqckr7VxMQEAAGV3+QvOSPWz45i3bcZTyKJmfT6AEL5re+L+2o7iECsvCGPN3yq5c9y5wN7BdRP5XRFo5OZuqhO6D72SjdzuaJr9KwYmjtuMo5RF2JS8h7sgCEi+5naYtWtuO4xIqtc3ClF0hab/jqxioB3wuIi84MZuqBG9vL4qu+BehJoeNn+lJBpW6WKa0lMLvnuQgdelwy1O247iMymyzeFRE1gAvAL8CHYwxo4EuwI1OzqcqIb73QFYHXkq7Xe+ToycZVOqibFr4IdGFm0hp+zAh9erbjuMyKrNkUR+4wRgz0BjzmTGmCMAYUwoMdWo6VWkNrvs3vqaY1Fl/tx1FKbdVXJhPvV//zS5pTrcbHrUdx6VUZpvFU8aY3Wd5bEvVR1IXIjK6E4kNhhGf/RXp25Jtx1HKLa374iWamv0c6v0P/GrgyQLPRY+z8CBRtzzDSfzJnfOE7ShKuZ283Gxap0wh2a8zXfrfbDuOy9Gy8CANGoWzvvUDdDy5ks1LPrcdRym3kvLpPwg2xwkc8lyNPq3H2WhZeJguN08gXZoQvOQpSooKbcdRyi1kpW2m495ZrAy5muhOPWzHcUlaFh4mICCQvd3+TvPSDNbPecl2HKXcwv7P/0IRPkTcrLufn42WhQfqPvB2knw702rzm+TlZNmOo5RL27psLp3yfiapxT1cEh5pO47L0rLwQOLlRcDQ56llTrLtkwm24yjlskqKCgj86UkyaEyX4Xr1yXPRsvBQ7Tp1Y0XodXTKmkP6ltW24yjlkpJnP0/z0gz29XyKwFpBtuO4NC0LD9Z2+HMcI4ijXz6OKS21HUcpl3LkQDptUyaz1r8bCQNusx3H5WlZeLAGDRuzue1Y2hckkfzTx7bjKOVSdn3yOD6mmLo3vKS7ylaCloWH63rTY6R5Nafx8qfJP3HMdhylXEJa0gLicn5gZePbaRXdyXYct6Bl4eF8ff3Iu/J5mphs1n/8D9txlLLOlBRjvn2c/YTS8bb/sR3HbTitLETkPRE5ICIby43VF5H5IrLd8b2eY1xEZJKIpIrIehHpXO45dzmm3y4idzkrrydr3+tqVtQeQNyeGezfsc52HKWs2jD3dSKLd7A9bgIhIXoR0Mpy5pLFNGDQ78YmAAuMMVHAAsd9gMGUXWApChgFTIGycgGeAroD3YCn9BKvF6bFbS9zkgByP38UjLEdRykrjuVk0WLdK6zz6Uiva+6zHcetOK0sjDE/A4d/NzwMmO64PR24rtz4DFNmBRAiIk2AgcB8Y8xhY0wOMJ8/FpCqhCaXNGdt1MO0PZnEph/fsx1HKSu2fvQEQeYEAde8hLe3roU/H9X902pkjNnnuL2fsut7AzQF9pSbLsMxdrbxPxCRUSKSKCKJ2dnZVZvaQ/S65TFSvKJotPwZTh7NsR1HqWq1PfkXOmd/xeqGNxLdqbvtOG7HWrU6LtVaZetDjDFTjTEJxpiEsLCwqnpZj+Lv50fR4Jeob3LZ9JGexlzVHCXFRXh9/SiHpS6xf5poO45bqu6yyHKsXsLx/YBjPBNoVm66cMfY2cbVBerQ9XKWh15H/P7PSNuwzHYcpapF0qf/plXJDtK6PU2dkAa247il6i6LucCpPZruAr4qN36nY6+oHsARx+qqecAAEann2LA9wDGmLkLMn14kV+pQ+NU4SktKbMdRyqkOpqcQu20yawJ60mWQ7lB5oZy56+wnwHIgWkQyRGQkMBG4SkS2A1c67gN8B+wEUoF3gIcAjDGHgWeA1Y6v/3GMqYtQLzSM1LgJtCneSuKcV23HUcp5jOHgzIcowZuGw99AvHSj9oUS44G7USYkJJjExETbMVyaKS1l08TLaVG4jcJRKwi9JMJ2JKWqXMoPU2m74nEWt3qCy0f8zXYclycia4wxCWd6TGu2hhIvL4JvfgsfU8K+jx7UYy+Uxzmes5/GK/6HTV7R9ByuO3RcLC2LGiwiqj0rI8fQ/vhyNv34ru04SlWp1BkPU8ucgGsn4e/razuO29OyqOF63v5XNntH03T50xw7tNd2HKWqxNZf59Ap50eWN7mT2Di9pnZV0LKo4fz9/JBhbxJoTrJrxhjbcZS6aPnHj1LnpydIk6YkjNBralcVLQtFu47dWNFsJB2PLGTLwo9sx1Hqomz48C80MQc4cuXLBAXp1e+qipaFAqD7Hf/Ddq9IGv78V47n6ulSlHtKTf6Zzns/YXm9a+nUe7DtOB5Fy0IBEBAQQMGQN6ljjrF9xiO24yh13oryj+M39yEOSj1iRujxQ1VNy0Kd1r5LH5Y3uYO4w9+xbekXtuModV42zBhP89I9pF/6EnXr6yk9qpqWhfqNLiOeY5eEU3fB45w4pgfLK/ewY9X3xGV+wtJ619G1/42243gkLQv1G0FBQeQNfJ2w0kNsnDbOdhylKpSfl0PQ94+QKY3pcPfrtuN4LC0L9QcdelzJqsbD6XboK9YvnGk7jlLntGXaWMJKszk04HXq1g2xHcdjaVmoM4q7+xV2eEXS7OfHyc3aU/ETlLJgy+JPiT/4Db82HkFcr4G243g0LQt1RgGBtTA3/pdAc5LM6fdgSvVU5sq15B3eR6PFj5PqFUnXu1+wHcfjaVmos2odm8Cq6PHEnljN+i/0P6NyIcaw6/1RBJnjFF37FoGBgbYTeTwtC3VOvW99gkT/7rTb8BJZ2/S078o1JH87lQ7HfmZFiwdpF9fLdpwaQctCnZO3txdNRvyXIwRTOOseivOP246karh9OzcSlfhPNvvE0mvE07bj1BhaFqpCTcObs63XCzQrTmfjtEdtx1E1WHHBCU5+PIIi40PIiOn46qnHq42WhaqU3gNvZUn9W4jb/xlblsyyHUfVUBvee5iWxTvZ2vMFLmkRZTtOjaJloSotYeRr7PCKoPGixziclW47jqphti78gPisz1kSeivdB/3JdpwaR8tCVVpQUBDmxnfLdqd9/x5KS3R3WlU9cjO20vTnJ9js1Yau9+lR2jZoWajz0jo2gXUxj9MhP5HEj56yHUfVAKWF+eTOuIMSI3jf8j61dDdZK7Qs1HnrdvPjrA6+nC473mT7im9sx1EebsO0R4ko3Mba+GeJbtvedpwaS8tCnTfx8qLNfdNI9wqnwQ+jydm7w3Yk5aG2LPyITntnsjjkRi4fdo/tODWaloW6IHVD6lF00wx8TRGH3x9OccFJ25GUhzmQvpWmPz/ONq/WdL3/DUTEdqQaTctCXbA2sZ1Zl/AcrYq2seHd0bbjKA9SWJBPzowRiCnF77YZei1tF6BloS5K72vuYUnYn4g/MIcN30y2HUd5iOR3RhNdvJVtPZ4jIirWdhyFloWqAj3uf5V1vp1os/op9mxebjuOcnNJX7xMt4NfsKzRbXQZrNspXIWWhbpo/n7+NLznI3KkDj6f3cmxnAO2Iyk3lbrqe9qv+zdr/buSMPIN23FUOVoWqko0uaQZBwa9Q/3Sw6RNvZ2S4mLbkZSbOZieQth395Hh1YSIUZ/g56fnfXIlWhaqynTs0Z+k2CfpcHI1a955yHYc5Uby83I4Pv0WMIaSWz+mfmiY7Ujqd6yUhYikicgGEUkWkUTHWH0RmS8i2x3f6znGRUQmiUiqiKwXkc42MqvK6XHLeJaF3Uq3rE9Z+9lE23GUGzAlxaS+fTtNi/ewte9kWrftZDuSOgObSxb9jDFxxpgEx/0JwAJjTBSwwHEfYDAQ5fgaBUyp9qTqvHQbNZnEgJ502jiRbT9/ZjuOcnHJ08fTPm8ZS1uPp1v/623HUWfhSquhhgHTHbenA9eVG59hyqwAQkSkiYV8qpJ8fH2JenAmqd4tCV84lv0pK21HUi4q6ZupxKe/zy91r+GyPz1pO446B1tlYYAfRWSNiIxyjDUyxuxz3N4PNHLcbgrsKffcDMfYb4jIKBFJFJHE7OxsZ+VWlVQ3JISAOz/jCEH4fDqcI1lptiMpF7NlxffErP4rG3070nX0fxEvV/rbVf2erU+njzGmM2WrmMaISN/yDxpjDGWFUmnGmKnGmARjTEJYmG4ccwUtIlqRfc0HBJSeJOed68jPy7UdSbmIPSmrCf/hXvZ7NSL8gc8ICAiwHUlVwEpZGGMyHd8PAHOAbkDWqdVLju+ndtbPBJqVe3q4Y0y5gY5derOpzyTCi3az/a2bKSkush1JWXYoYzuBn97MCQLwuWsOIQ0a246kKqHay0JEgkSk9qnbwABgIzAXuMsx2V3AV47bc4E7HXtF9QCOlFtdpdxA96tuYWXMX+lwYhWJU+7HlJbajqQsOZ6TRf77w/AtLSTnhk9pGtHGdiRVSTaWLBoBS0VkHbAK+NYY8wMwEbhKRLYDVzruA3wH7ARSgXcA3YHfDfW+9XGWN7mD7ofmsGr6hIqfoDxOwYmj7J9yDaHFB9h2xTu07djNdiR1Hnyqe4bGmJ3AH3akNsYcAvqfYdwAY6ohmnKy7vdNYtXr2XTf/R/WzAymy/B/2o6kqklxYQHb37iRdgXbWNFtEr0vG2I7kjpPuvuBqjZe3t7Ejf2Q1bX60iXlZZJmv2Q7kqoGpSUlJE8eQfuTq1ge8w96D7nTdiR1AbQsVLXy8/OjwyOzWBvQnfgNz7DuGz3G0pOZ0hKSptxNwpF5LGv+IH1ufcx2JHWBtCxUtQsICCT64S9Y5xdP+9VPsvHH6RU/SbkdU1rC+rdH0uXgXH5pfBc9737OdiR1EbQslBVBQcFEjv2SFN92RP/6ZzYv+tR2JFWFTGkJ6/4zkk4H5rAo7A563/+aHnTn5vTTU9bUqRNCk4e+YadPS6IWj2bTfF3C8ASmtITkt+8jLmsOixvewWUPvoGXt/6qcXf6CSqr6tcPpeGYH9jqG03bpY+y6bu3bUdSF8GUlpL09v3EH/iCJWF/ou8DWhSeotp3nbWlqKiIjIwM8vPzbUep8QICAggPD8fXt+ziNvXqN8Br7PdsmDyMuFV/YVPRCWKH/T/LKdX5MqWlrH17JF0OfMHPDW/n0gfe1KLwIDWmLDIyMqhduzYRERGIiO04NZYxhkOHDpGRkUFkZOTp8bohIbR89BtWT7qRrkn/YmPhCdrf/HeLSdX5KC4sIPmtu0jI/Z6lDW/n0gcn6zYKD1NjPs38/HxCQ0O1KCwTEUJDQ8+4hFcnuDbtxn3FsoDLaL/pRdbO+AuY8zqfpLIg//gRtr46hITc71nWbBS9tSg8Uo36RLUoXMO5PofgWoF0/vPnLKs9kM4732bl1DF6LikXdvzwXva+1p/oE2tYGvMUvUa+qEXhofRTVS4nwN+P7uM+YUXo9XTf9xErJo2gqLDAdiz1O4d2b+bYm5fTuDCd1T0m0+cW3c7kybQsapCnn36al1764yk2vvzySzZv3nzer5eWlsbHH398+v60adMYO3bsRWU8xdvbm+5j3mN1s3vomfsN214ewLGcAxU/UVWLnclL8Hp/IH4lJ9g84CN6Dr7ddiTlZFoWLqa4uLja53musjhXnt+XRVUTLy+6jnyNFR2fpXX+Ro6+0ZfM7clOm5+qnHULZtJ4zs0cJ5BDw78hofdVtiOpalBj9oYq719fb2Lz3qNV+poxl9ThqWtizznNM888w4cffkhYWBjNmjWjS5cujB8/nssvv5y4uDiWLl3KbbfdRlxcHOPHj6e4uJiuXbsyZcoU/P39iYiIIDExkQYNGpCYmMj48eNZvHgxTz/9NOnp6ezcuZP09HTGjRvHI488AsC///1vpk+fTsOGDU/Ps7xly5Yxd+5clixZwrPPPsvs2bMZOXLkb/Js2LCBoUOHctNNNwEQHBxMXl4eEyZMYMuWLcTFxXHXXXdRr1499u7dy6BBg9ixYwfXX389L7zwwkX/bHvc8DAbGkfR9Mf78f1oMJv7TyHm0usu+nXV+TGlJaz5+F/Eb5/ETp9WhNw3h/AmzW3HUtWkRpaFDatXr2b27NmsW7eOoqIiOnfu/Jtf3IWFhSQmJpKfn09UVBQLFiygTZs23HnnnUyZMoVx48ad8/VTUlJYtGgRx44dIzo6mtGjR7N+/XpmzpxJcnIyxcXFf5gnQK9evbj22mt/Uwbl8wDcfffdZ5znxIkTeemll/jmm2+AstVQycnJJCUl4e/vT3R0NA8//DDNmjU74/PPR4deg9jT+CcKP7yF6J/uZs3eJ+hyy5OgOy1Ui5NHD7Nj6h0k5P3K6uDLiR09nVrBIbZjqWpUI8uioiUAZ/j1118ZNmwYAQEBBAQEcM011/zm8VtvvRWArVu3EhkZSZs2ZVcQu+uuu5g8eXKFZTFkyBD8/f3x9/enYcOGZGVl8csvv3D99ddTq1YtAK699tpK5z2V53z179+funXrAhATE8Pu3burpCwAmrWM5ui4xSRPuZ0uW55n3asraXPfNALr1KuS11dnti9lJcy6k+iSbBa3eoy+f/q7HmxXA+kn7iKCgoIqnMbHx4dSx26kvz9Owd/f//Rtb2/vi972UT5P+fmWlpZSWFh41udVdY7fq1OnHnHjv+HnFg8Te+Rncl/rTsbGpVU6D/V/Ur6dTP2ZQ/AqLWT9VZ9w+Z3/1KKoofRTrya9e/fm66+/Jj8/n7y8vNOrbn4vOjqatLQ0UlNTAfjggw+47LLLAIiIiGDNmjUAzJ49u8J59u3bly+//JKTJ09y7Ngxvv766zNOV7t2bY4dO3bW1yk/37lz51JUVFSp5zmLt7c3fe95lvUDZkJpMY0+u5YNsyfqAXxVqODkMZLfuJ22q//KJp9Yiu5bQpc+A23HUhZpWVSTrl27cu2119KxY0cGDx5Mhw4dTq+uKS8gIID333+fm2++mQ4dOuDl5cWDDz4IwFNPPcWjjz5KQkIC3t7eFc6zc+fO3HrrrXTq1InBgwfTtWvXM043fPhwXnzxReLj49mxY8cfHr///vtZsmQJnTp1Yvny5aeXOjp27Ii3tzedOnXi1VdfPZ8fR5Xo3HsgPLiUpICudNjwHBteuYajudnVnsPT7E5ZS8ZLlxJ36FsWNbqbmMfnEx6uG7JrOjEe+NdYQkKCObVx9pQtW7bQrl07S4nK5OXlERwczIkTJ+jbty9Tp06lc+fOVjPZUpWfR1FxCUs/fIbeuyZxWOpz+Irniel7Y5W8dk1iSopI+vQZYre+xQkJIO3SV4jvf4vtWKoaicgaY0zCmR7TJYtqNGrUKOLi4ujcuTM33nhjjS2Kqubr402/u59m17VfUCD+xCy8l/WTbuVEbpbtaG4jc+sadk7sRedtr7OuVneKH1imRaF+o0buDWWLMw9gUxDd5XJOtFvFohl/o/e+GRx/LYG0Hv8kZuB9uovtWRQXFpA082k67ZhKHoEs7fwSva+9T8+jpv5AlyyUR6lVK4h+D75GyrXfsM+rMTErxrPpxQFkZ2y3Hc3lpKz6kbTne9J151usC+5D0YMr6DPsfi0KdUZaFsojdezSi1YTfmVJy/FEHF9H0Du9WT79b+SfyLMdzbrs3Skkv3wNbb+7mTolOSR2f52E8V/SqEm47WjKhWlZKI/l7+fHZXf+g8N3/8LWoC703PUmR1/owNovX6e0uMh2vGqXl5tN0jujCXmvF1FHV/Jr+CiCHksmYfDdujShKqRloTxes8ho4p/4no0DZnLYJ4zOyf8k47l4kn/8AFNaYjue0508lsPq6U9S+lonOmV8wso6A8gZuZLe971IUO0/7r6t1JloWbipxYsXM3ToUKDsQLmJEyeeddrc3Fzeeuut0/f37t37m/NA1RTtew2mzZMrWNH1dUxpKXHLxrLn2Y5smPs6pYUnbcercseOHGb1B3+j4OX2dN31FqmBHdh2/Xf0eWwm4c0jK34BpcrR4yxcTElJSaUOuFu8ePFvTuJ3LmlpaQwdOpSNGzdWRcQqYfvzKC4qZO337xOS/B/alO4gR+qyu+XttB46juB6ja3lqgoHdiSRPu9N2mZ9S7CcZI1/d/z7P0n7bv1sR1Mu7lzHWdTMXWe/nwD7N1TtazbuAIPP/tc9lP3SHjRoEF26dGHt2rXExsYyY8YMYmJiuPXWW5k/fz5PPPEE9evX56mnnqKgoIBWrVrx/vvvExwczA8//MC4ceOoVasWffr0Of2606ZNIzExkTfffJOsrCwefPBBdu7cCcCUKVOYNGkSO3bsIC4ujquuuooxY8acLo/8/HxGjx5NYmIiPj4+vPLKK/Tr149p06Yxd+5cTpw48ZvTjZeUlDBy5EgSExMREe69917+/Oc/V+3Pshr4+PrR7doHKBlyP78umovfyjfpumMK+a/9l8SwITTo9xARMd3cZpfb4sJ8UhZ/jN/a92mTv566xpfkuldQv99YusT3tR1PeYCaWRYWbd26lXfffZfevXtz7733nl49FBoaytq1azl48CA33HADP/30E0FBQTz//PO88sorPPHEE9x///0sXLiQ1q1bn/WssI888giXXXYZc+bMoaSkhLy8PCZOnMjGjRtJTk4GykrrlMmTJyMibNiwgZSUFAYMGMC2bdsAzni68QMHDpCZmXl6KSU3N9dpP6vq4O3tRe8rr4Mrr2PLhtXkLniNztnf4v/Zl6R7NSO7xRAiLr+L0BYxtqP+gSktYee6Xzi46nNa7/ua9uSSQUMWNBtL20EP0r1p1ZztVymoqWVRwRKAMzVr1ozevXsDcMcddzBp0iTg/04JvmLFCjZv3nx6msLCQnr27ElKSgqRkZFERUWdfu7UqVP/8PoLFy5kxowZQNkJ9+rWrUtOTs5Z8yxdupSHH34YgLZt29KiRYvTZXGm043Hxsayc+dOHn74YYYMGcKAAQMu+mfiKtp16AodPuLQgb2sWfghtVPnEr/zP3jtepsdPq3JajaYS+IH0CK2J+LtayVjYf5JdiX+wPF1X9H84GJamRxaGC821urGrs730qnfjYT71Mz/1sq53OZflYgMAl4HvIH/GmPs/ca/CL/fRfHU/VMn5zPGcNVVV/HJJ5/8ZrpTSwXV6UynG69Xrx7r1q1j3rx5vP3228yaNYv33nuv2rM5U2jDS+g1/AngCdJ2bSdtyUdcsudbeu16A3a9wfEvAtkd1IH8pj2pH9OP5rG98PL1r/B1L8SJQxlkblpK3o4V+O1PIiI/hWjJ57jxZ3OtbmxvM5joPjcSF+be21mU63OLshARb2AycBWQAawWkbnGmDNfONqFpaens3z5cnr27MnHH39Mnz59SEpKOv14jx49GDNmDKmpqbRu3Zrjx4+TmZlJ27ZtSUtLY8eOHbRq1eoPZXJK//79T19Z79RqqHOdSvzSSy/lo48+4oorrmDbtm2kp6cTHR3N2rVrzzj9wYMH8fPz48YbbyQ6Opo77rjj4n8oLiwiMoqIyKeBp8neu5ttq+ZRsusXwo+sJWbb67DtdU7M8SfdN5K8oBaY+q3wbdiaWmEtqFe/IfVDw/CuFQI+AWfc/mGM4XjeEY7t38WxrDTysnZRlJOOX+5Omh7fRENzkCigyHiz0zuCjWFX4xM9kNY9htC1du1q/mmomswtygLoBqQaY3YCiMhMYBjgdmURHR3N5MmTuffee4mJiWH06NG88cYbpx8PCwtj2rRp3HbbbRQUFADw7LPP0qZNG6ZOncqQIUOoVasWl1566RkL4PXXX2fUqFG8++67eHt7M2XKFHr27Env3r1p3749gwcPZsyYMaenf+ihhxg9ejQdOnTAx8eHadOm/WaJ4vcyMzO55557Tl8M6bnnnquqH43LC7ukBWHXjQJGYYwhI2MPe5Ln45W+jOCj22l2JJFGR+bBrj8+txAf8ggiXwLwoRhfU4QvRfiZIoKlmGCgiWPaIuNNllcYu2p1YEvDOIJadqd5bA+i6+sVAZU9brHrrIjcBAwyxtznuD8C6G6MGVtumrL/xUDz5s277N69+zevYXtXTXDNXVhtcYXPwxlyj+RyMD2FY9kZ5B05REFeDl4FR/AqPIpv4VH8Sk9SIr6UevlR6u2Lt28AXoEhlNZthl9oCxo0bUmTppH4+dnZJqJqthqx66wxZiowFcqOs7AcR9VQIXVDCOnQw3YMpaqcuxzBnQmU3w8w3DHmViIiInSpQinlltylLFYDUSISKSJ+wHBg7vm+iDuscqsJ9HNQyv24RVkYY4qBscA8YAswyxiz6XxeIyAggEOHDukvKsuMMRw6dIiAgADbUZRS58FttlkYY74DvrvQ54eHh5ORkUF2dnYVplIXIiAggPBwvXaCUu7EbcriYvn6+hIZqWfaVEqpC+EWq6GUUkrZpWWhlFKqQloWSimlKuQWR3CfLxHJBnZXOKHraQActB2imul7rhn0PbuHFsaYsDM94JFl4a5EJPFsh9p7Kn3PNYO+Z/enq6GUUkpVSMtCKaVUhbQsXMsfL33n+fQ91wz6nt2cbrNQSilVIV2yUEopVSEtC6WUUhXSsnAhIvKYiBgRaeC4LyIySURSRWS9iHS2nbGqiMiLIpLieF9zRCSk3GNPOt7zVhEZaDFmlRKRQY73lCoiE2zncQYRaSYii0Rks4hsEpFHHeP1RWS+iGx3fPe4a8SKiLeIJInIN477kSKy0vF5f+q4vILb0rJwESLSDBgApJcbHgxEOb5GAVMsRHOW+UB7Y0xHYBvwJICIxFB2vZJYYBDwloh4W0tZRRzvYTJln2kMcJvjvXqaYuAxY0wM0AMY43ifE4AFxpgoYIHjvqd5lLJLKJzyPPCqMaY1kAOMtJKqimhZuI5XgSeA8nscDANmmDIrgBARaWIlXRUzxvzouE4JwArKrn4IZe95pjGmwBizC0gFutnIWMW6AanGmJ3GmEJgJmXv1aMYY/YZY9Y6bh+j7JdnU8re63THZNOB66wEdBIRCQeGAP913BfgCuBzxyRu/561LFyAiAwDMo0x6373UFNgT7n7GY4xT3Mv8L3jtqe+Z099X2clIhFAPLASaGSM2ed4aD/QyFYuJ3mNsj/2Sh33Q4Hccn8Quf3nXWOuZ2GbiPwEND7DQ38D/krZKiiPcq73bIz5yjHN3yhbdfFRdWZTziUiwcBsYJwx5mjZH9pljDFGRDxmn30RGQocMMasEZHLLcdxGi2LamKMufJM4yLSAYgE1jn+Q4UDa0WkG5AJNCs3ebhjzC2c7T2fIiJ3A0OB/ub/Dvhx6/d8Dp76vv5ARHwpK4qPjDFfOIazRKSJMWafY1XqAXsJq1xv4FoRuRoIAOoAr1O22tjHsXTh9p+3roayzBizwRjT0BgTYYyJoGxxtbMxZj8wF7jTsVdUD+BIuUV5tyYigyhbbL/WGHOi3ENzgeEi4i8ikZRt3F9lI2MVWw1EOfaQ8aNsI/5cy5mqnGNd/bvAFmPMK+Uemgvc5bh9F/BVdWdzFmPMk8aYcMf/3+HAQmPMn4BFwE2Oydz+PeuShWv7Driaso28J4B77MapUm8C/sB8xxLVCmPMg8aYTSIyC9hM2eqpMcaYEos5q4QxplhExgLzAG/gPWPMJsuxnKE3MALYICLJjrG/AhOBWSIykrLLB9xiJ161+gswU0SeBZIoK1G3paf7UEopVSFdDaWUUqpCWhZKKaUqpGWhlFKqQloWSimlKqRloZRSqkJaFkoppSqkZaGUUqpCWhZKVQMR6eq4dkeAiAQ5rvXQ3nYupSpLD8pTqpo4juQNAAKBDGPMc5YjKVVpWhZKVRPHOaFWA/lAL084jYmqOXQ1lFLVJxQIBmpTtoShlNvQJQulqomIzKXsCnmRQBNjzFjLkZSqND3rrFLVQETuBIqMMR87rse9TESuMMYstJ1NqcrQJQullFIV0m0WSimlKqRloZRSqkJaFkoppSqkZaGUUqpCWhZKKaUqpGWhlFKqQloWSimlKvT/AX2668mnGrqqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(x, y, label=\"ground truth\")\n",
    "plt.plot(x, y_hat, label=\"predictions\")\n",
    "plt.ylabel('y')\n",
    "plt.xlabel('x')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "121c04c9",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEKCAYAAAAVaT4rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA7mUlEQVR4nO3dd3RU1d7G8e+e9E4IIQQIJJSEFggh1FClC4JSRWmiYu/i1eu1i4IFyyugCIoCoiJFQEQpoRdpoXcIJdQECOlt9vvHBEFMqJmczMzvs1ZWkpmTc54MrIfNnnP2UVprhBBCOA6T0QGEEEKULCl+IYRwMFL8QgjhYKT4hRDCwUjxCyGEg5HiF0IIB2O14ldKhSil4pRSu5RSO5VSzxQ8/qZSKlEpFV/wcae1MgghhPg3Za3z+JVSwUCw1nqzUsoH2ATcDfQD0rTWH1nlwEIIIa7J2Vo71lqfBE4WfJ2qlNoNVLLW8YQQQtwYq434/3EQpUKBFUA94HlgKHAR2Ai8oLU+X8jPDAeGAzh5+DSKCjJBfg6Ur2P1vEIIYctys9JxObePTSfNSVrrwKuft3rxK6W8geXASK31LKVUEJAEaOAdLNNBw661D7fgmnrfD69QdflzMHQBhMZaNbMQQtiyjRMep37ij7i9fX6T1jrm6uetelaPUsoFmAlM01rPAtBan9Za52utzcDXQJPr7cekFJOSI8HNF7ZMtWZkIYSwafn5+YScWMhOj8ZFbmPNs3oUMAnYrbUec8XjwVdsdg+w43r78vNwYea2ZHJr3wO75kB2arHnFUIIe7B97R8EkQyRvYvcxpoj/lhgEHDHVadufqCU2q6U2ga0A5673o7KermQnpPPcs9OkJsBO2dbMbYQQtiuixumk4krddr1L3Iba57VswpQhTy14Gb35enqTMXy3ozb70SHchGwZRpED779kMJQubm5HD9+nKysLKOjODR3d3cqV66Mi4uL0VHEbUq+mE69C3EcKtuKup5+RW5nteIvbvc2DuHd33Zztm0fAteNhKT9UK6m0bHEbTh+/Dg+Pj6EhoZimRkUJU1rTXJyMsePHycsLMzoOOI2/bV0Nl1VKplNB1xzO5tZsuGehpVwcVJMyWwOygnipxkdSdymrKwsAgICpPQNpJQiICBA/tdlB7TWOO2cSbryolJMj2tuazPFH+DtRqc6FZiyPZP8Gh0hfjrk5xkdS9wmKX3jyZ+Bfdhy6BTNctZyumIHcHa75rY2U/wA/RuHcD4jl81lu0HaKTi41OhIQghRKmyN+wVflUnFloOuu61NFX/LGuWoVMaDL45XB89ysGWK0ZGEuC1vvvkmH33072Wr5syZw65du256fwkJCfzwww9/fz958mSefPLJ28ooSr+UjFwqHJ1PmnMZ3MPbXXd7myp+k0nRv3EIyw9eIDWiN+z9HdKTjY4l7FxeXslPKV6r+K+V5+riF45h/sZ9tFWbyQ7vAU7XP2fHpoofoG9MZUwKZprbgjkXtv9sdCRhw9555x0iIiJo2bIlAwYM+Hv03bZtW5599lliYmL47LPPWLJkCQ0bNiQyMpJhw4aRnZ0NQGhoKElJSQBs3LiRtm3bApaR/LBhw2jbti3VqlXj888///uYI0eOJDw8nJYtW7J3795/ZVqzZg1z585lxIgRREVFcfDgwX/lGTp0KL/88svfP+Pt7Q3Ayy+/zMqVK4mKiuKTTz4B4MSJE3Tp0oWaNWvy0ksvFf+LKAyltebY2pl4qBwCmt53Qz9jM6dzXhLs50HbiPJ8ufsiQypGo7ZMhaaPgrxBZdPemreTXScuFus+61T05Y276hb5/IYNG5g5cyZbt24lNzeX6OhoGjVq9PfzOTk5bNy4kaysLGrWrMmSJUsIDw9n8ODBjB8/nmefffaax9+zZw9xcXGkpqYSERHBY489xrZt2/jxxx+Jj48nLy/vX8cEaNGiBT169KB79+706dPnX3kAhg4dWugxR40axUcffcT8+fMBy1RPfHw8W7Zswc3NjYiICJ566ilCQkKumV3Yjs1HLxCTtpR0rwp4hTS9oZ+xuRE/WN7kPXUxi73BPeH0Dji51ehIwgatXr2anj174u7ujo+PD3fdddc/nu/f33Ll4969ewkLCyM8PByAIUOGsGLFiuvuv1u3bri5uVGuXDnKly/P6dOnWblyJffccw+enp74+vrSo8e1T7srLM/Nat++PX5+fri7u1OnTh2OHDlyS/sRpdOcNTtoY9qGS4M+YLqxSre5ET/AHbXKU87bjbFno/g/Z3fLwm0Vo4yOJW7DtUbmRvHy8rruNs7OzpjNZoB/nQvv5nb5lDonJ6fbfq/gyjxXHtdsNpOTk1PkzxV3DlF6pGTmwu5fcTHlQ4O+N/xzNjnid3Ey0S+mMr/tTyezxp2wfQbkygUo4ubExsYyb948srKySEtL+3t65GoREREkJCRw4MABAKZMmUKbNm0Ayxz/pk2bAJg5c+Z1j9m6dWvmzJlDZmYmqampzJs3r9DtfHx8SE0tejHCK487d+5ccnNzb+jnhH2ZsyWRLnoN2X7VILjBDf+cTRY/wIAmVdDAb053QNYF2Pub0ZGEjWncuDE9evSgfv36dO3alcjISPz8/r2+ibu7O99++y19+/YlMjISk8nEo48+CsAbb7zBM888Q0xMDE5OTtc9ZnR0NP3796dBgwZ07dqVxo0LXzr33nvv5cMPP6Rhw4YcPHjwX88//PDDLF++nAYNGrB27dq//zdQv359nJycaNCgwd9v7gr7pLXmj3XxNHfahVtU35t6n7NE7sB1u2JiYvSlN7Wu9MC3f7Er8QLrvJ5DlYuAQbMMSCdu1e7du6ldu7ahGdLS0vD29iYjI4PWrVszYcIEoqOjDc1khNLwZyFuzuaj55n31Wu84TIFntgAgeH/2kYpVfI3YrG2+5tW5XRaLgcr9rRcxZty3OhIwsYMHz6cqKgooqOj6d27t0OWvrBN09cfpafzOvKDIgst/WuxyTd3L2lXqzwV/dwZd6EJYxgLW6dD6xFGxxI2RC52ErYoJTOXzdvi+dBpP0S+edM/b9MjfieTYkCTKsw67EJmpVjL2T0FZzoIIYS9+jU+kc7m1ZZv6hV9p62i2HTxg+WcfieTYpFbBzifAEfXGB1JCCGsRmvND+uP0td9PYQ0hTJVbnofNl/85X3d6VQniPcP10S7+VjuziWEEHZq89Hz5J3eTVh+wi2N9sEOih8sb/KezDSRUKGL3IxdCGHXpqw9Ql/XdWhlgrr33NI+7KL4W1QPIKycFxNSW8jN2IVhli1bRvfu3QHLRVWjRo0qctsLFy4wbty4v78/ceLEP9blEaIwSWnZxG1PYKDLUlT19uBd/pb2YxfFbzIp7mtSheknypPtX9PyJq8QxSQ/P/+mf6ZHjx68/PLLRT5/dfFXrFjxH6ttClGYnzceYwAL8cq7AG1ufaVVuyh+gN6NKuPq7MQyj05wbL3lZuxCXEdCQgK1atXi/vvvp3bt2vTp04eMjAxCQ0P5z3/+Q3R0NDNmzODPP/+kefPmREdH07dvX9LS0gBYuHAhtWrVIjo6mlmzLl9AeOUNUE6fPs0999xDgwYNaNCgAWvWrOHll1/m4MGDREVFMWLECBISEqhXrx5gWfPngQceIDIykoYNGxIXF/f3Pnv16vWvJZbz8/MZOnQo9erVIzIyUq7YtVP5Zs3stXt4wvU3qNERQprc8r5s+jz+K5X1cqVbZDDv7WpAJ5OTZbnmjm8ZHUvcqN9fhlPbi3efFSKha9HTLZfs3buXSZMmERsby7Bhw/4eiQcEBLB582aSkpLo1asXixcvxsvLi9GjRzNmzBheeuklHn74YZYuXUqNGjWKXD3z6aefpk2bNsyePZv8/HzS0tIYNWoUO3bsID4+HrD8A3TJ2LFjUUqxfft29uzZQ6dOndi3bx9AoUssnzlzhsTERHbs2AFY/jch7E/cnjN0TpuDj0sqtPvvbe3Lbkb8AAObVeFItjeJga1g649yM3ZxQ0JCQoiNjQVg4MCBrFq1Cri8DPK6devYtWsXsbGxREVF8d1333HkyBH27NlDWFgYNWvWRCnFwIEDC93/0qVLeeyxxwDL6piFrQd0pVWrVv29r1q1alG1atW/i7+wJZarVavGoUOHeOqpp1i4cCG+vr63/6KIUmfmmp0Md1mAObwrVLq9K8ztZsQPEF3Fn1oVfPg2I5bX0pbBwSUQ3tnoWOJG3MDI3FrUVYtbXfr+0sJnWms6duzI9OnT/7HdpdF6SSpsiWV/f3+2bt3KH3/8wZdffsnPP//MN998U+LZhPUcSU6nVsL3+Dqn3/ZoH+xsxK+U4v6mVfguKYJc93LyJq+4IUePHmXt2rWAZQmHli1b/uP5Zs2asXr16r+XZU5PT2ffvn3UqlWLhISEv1fPvPofhkvat2/P+PHjAct8fEpKyjWXT27VqhXTplmuR9m3bx9Hjx4lIiKiyPxJSUmYzWZ69+7Nu+++y+bNm2/itxe2YNaq7QxzWkhWze4QXP+292dXxQ9wT3Rl3N3cWeVxR8HN2JOMjiRKuYiICMaOHUvt2rU5f/7839MylwQGBjJ58mQGDBhA/fr1ad68OXv27MHd3Z0JEybQrVs3oqOjKV++8FPrPvvsM+Li4oiMjKRRo0bs2rWLgIAAYmNjqVevHiNG/HN9qccffxyz2UxkZCT9+/dn8uTJ/xjpXy0xMZG2bdsSFRXFwIEDef/992//RRGlRlZuPr5bxuOlsnDv+L9i2adNL8tclDfn7uSv9atY4PISdH4fmj9uxXTiVpWGpYATEhLo3r3732+MOqrS8GchCjd3dTwd/uxEelgnAofe3CyGXS7LXJTBzauyK78yp7zrWKZ7bOAfNyGEKEz+qk9xU7mU6/Z6se3TLou/WqA3rcMDmZzZEs7shJPxRkcSpVRoaKjDj/ZF6bVz7166ZszncHA31E2uuX8tdln8AEOaV+WH9Mbkm9xk4bZSzBamGu2d/BmUXuf+GI0z+QT1KL7RPthx8beNKE+ZsoGscW0B23+Wm7GXQu7u7iQnJ0vxGEhrTXJyMu7u7kZHEVe5cCqBpsm/siXgTnyCi2+0D3Z2Hv+VnEyKQc2qMn5hc1q5xlluxn6LS5gK66hcuTLHjx/n7NmzRkdxaO7u7lSuXNnoGOIqiXPfoSYa/66vFvu+7bb4AfrFhDBmUT3Ou1TAf8tUKf5SxsXFhbCwMKNjCFHq5CUnEH5iDsu8u9KxZp1i37/dTvUA+Hm6cHfDEKZlx6IPxsnN2IUQNuHkvHcwa4VrW+vcQ9xqxa+UClFKxSmldimldiqlnil4vKxSapFSan/BZ39rZQAY3DyUH3NbodAQX/iVlUIIUWokH6RiwizmunSiZaMGVjmENUf8ecALWus6QDPgCaVUHeBlYInWuiawpOB7q6kd7EvF0FpsMkWi4+Vm7EKI0u38wpHkaGdymj2Lk0ld/wdugdWKX2t9Umu9ueDrVGA3UAnoCXxXsNl3wN3WynDJ0BahTMlqhZKbsQshSrOz+/DbP5sf6UT32IZWO0yJzPErpUKBhsB6IEhrfbLgqVNAUBE/M1wptVEptfF2z/roWCeILV6tyFCesnCbEKLUyl7yHpnahVORj+Ln4WK141i9+JVS3sBM4Fmt9cUrn9OWE7gLPYlbaz1Bax2jtY4JDAy8rQwuTib6Nq/JnNxmmHfOgayL1/0ZIYQoUad34bpnDt/ld6Zva+uN9sHKxa+UcsFS+tO01pfuS3daKRVc8HwwcMaaGS4Z0KQKs2iHKS9TbsYuhCh1zHHvk447O6oOoUZ5b6sey5pn9ShgErBbaz3miqfmAkMKvh4C/GqtDFcK8HajRlQbDuhK5G2aUhKHFEKIG3NyG6Y9c5mU15W+rW9/vf3rseaIPxYYBNyhlIov+LgTGAV0VErtBzoUfF8ihrWqxk95bXA+sQHO7iupwwohxLUte5805c1iv960qXl7U9s3wmpX7mqtVwFFnYvU3lrHvZbwIB9OVO1J3okfUVum4tTpbSNiCCHEZYmbYO8CvsztS5+O9TBZ6RTOK9n1lbuF6dsmmrj8huRumiY3YxdCGC/ufdJNvsxw7k7vRiWzZpLDFX+b8EBWenfGPTsJfWCx0XGEEI7s2F9wYBHjcrvRLSYcb7eSWT7N4YpfKUXt1n04q305v/obo+MIIRzZ0ndJd/Fncl5HhrYILbHDOlzxA9wTE8rvqg2+R5fIzdiFEMZIWAWHlzMutwet64ZSJcCzxA7tkMXv7uJEfoP7cCaPc+vkSl4hRAnTGuLeI8M1kIlZ7XioVbUSPbxDFj9At/Z3sNVcndwN38vN2IUQJevQMjiymgn6bupWKU+jqlZdpPhfHLb4y/u6s69iT4KyDpKasNHoOEIIR1Ew2s/0qMD41FiGty7Z0T44cPED1O30AFnahaOLvzI6ihDCURxYDMf/4jvnPlQIKEPHOhVKPIJDF3+dalXY6NGSkMQF5GVnGB1HCGHvtIa4kWR7V+bjs00YFhtmtTX3r8Whix/Ao+kQfEknfvE0o6MIIezd3t/hxBZ+8hyAp4cHfWOMucm9wxd/w9Y9OKXKo7ZMQ8ubvEIIazGbIe49cv3CePtYfe5vWgVP15K5YOtqDl/8Jicnkmr0pmFuPBu3bjc6jhDCXu2ZB6e3M7fMIJTJmSEleMHW1Ry++AFqdn4Ek9IcWTrR6ChCCHtkzoe498kvW5PXD9WmR4NKBPm6GxZHih9wKxfG8TKNaXzhd3Ymnjc6jhDC3uycDWd3szhoGOm5modahRkaR4q/QNmWw6hqOsPShXJ3LiFEMcrPg2XvYw6szev7a9CqZjlqB/saGkmKv4Bn/bvJcvKiUsIsjp2TUzuFEMVk+wxIPsDakOGcTsvl4RJenqEwUvyXuHqSX6c3XU3rmbp8h9FphBD2ID8Xlo9CV6jPa3vDqFvRl1Y1yxmdSor/Sl5Nh+ChcsjYMoPz6TlGxxFC2Lqt0+F8ApuqPcah5Awea1sdy+3IjSXFf6VKjcj2D+du4piy7ojRaYQQtiwvB5Z/iK7UiLf3hlA1wJOu9YKNTgVI8f+TUrjFDKKRaT/LV68mKzff6ERCCFu15XtIOcquiCfZlniRR1pXN2R5hsJI8V+twb1o5UTHnMX8sum40WmEELYoNwtWfAwhTXl/b0UCfdzoFV3J6FR/k+K/mnd5CO9EP9dVfLNiP/lmWcZBCHGTNk2G1BMcjHyGVQeTebBlGO4uTkan+psUfyFUw0GUNZ8n9MJaFmw/aXQcIYQtycmAVWMgtBUf76+Aj7sz9zetYnSqf5DiL0zNTmivQB7wXM3YuAOyeJsQ4sZtnARppznR8Dl+33GKQc2q4uPuYnSqf5DiL4yTC6p+f2LzN3DmVCJLdp8xOpEQwhZkp8GqT6BaOz4/EIiLk4kHYo1dnqEwUvxFaTgQk87jAZ/1fCGjfiHEjfhrAmQkk9zkRWZuPk6/mMoE+rgZnepfpPiLUr42VGrEQPdVxB87z5qDyUYnEkKUZlkXYc3nULMTXx0KwKzhkdbVjU5VKCn+a2k4EP/U/bTxPs4XSw8YnUYIUZqtGw+Z50lp9hJT1x2hW2QwIWU9jU5VKCn+a6nXG5zd+U/QJtYeSmbTEVmyWQhRiMzzsHYs1OrO1wd8ycjJ58k7ahidqkhS/Nfi7ge1e1A7+Q+CPDRj42TUL4QoxNqxkJ1CavMRfLcmga71KhAe5GN0qiJJ8V9Pw/tRWSm8FZ7A0j1n2HkixehEQojSJD3ZMs1T526+2e9FanZeqR7tgxT/9YW2Br8qtM9ehI+bM+PiDhqdSAhRmqz5HHLSSW8xgm9WH6ZD7SDqVvQzOtU1SfFfj8kEUffhkrCcpxqaWLDjJAfOpBqdSghRGqSdsZzCGdmHyfvdScnM5en2pXu0D1L8NybmAXD1YmjaRNycTYxbJqN+IQSw+jPIyyKzxYtMWnWYNuGB1K9cxuhU12W14ldKfaOUOqOU2nHFY28qpRKVUvEFH3da6/jFyqcCtHoB1wML+V+tU/waf4KEpHSjUwkhjHTxJGyYCPXvZep+V86l5/B0+5pGp7oh1hzxTwa6FPL4J1rrqIKPBVY8fvFq/gT4h3Fv8jjcTfn8n5zXL4RjWzUGzHlkx77IVysOEVsjgEZV/Y1OdUOsVvxa6xXAOWvtv8Q5u0GX93E+t48xYRuYE58oo34hHFXKccvSy1H3M32/iaS0bJ66wzZG+2DMHP+TSqltBVNBRf7zqJQarpTaqJTaePbs2ZLMV7TwLlC9PR3PfEt5p1Q+X7rf6ERCCCOs+Ai0JrvF83y5/BBNQsvSrFqA0aluWEkX/3igOhAFnAQ+LmpDrfUErXWM1jomMDCwhOJdh1LQZRSm3AzGBv3GnC2JHJZRvxCO5XwCbJkCjYYw44Di1MUsm5nbv6REi19rfVprna+1NgNfA01K8vjFIjAcmjxCw6S5RDkf4f+WyKhfCIey4kNQTmQ1e5axcQeIrlKG2Bq2M9qHEi5+pdSVt5i/B9hR1LalWpuXUJ4BfOY3nTnxxzl0Ns3oREKIkpB8EOKnQ8wwftyTx8mULF7oFIFSpeMm6jfKmqdzTgfWAhFKqeNKqQeBD5RS25VS24B2wHPWOr5VeZSB9q8TkraNXi5r5QwfIRzF8tHg5Epm06cZu+wgTcPK0qK6bY32AZyttWOt9YBCHp5kreOVuIYDYeMkXk/6iebx0Ry8owbVA72NTiWEsJaze2H7DGj+JFN3ZHE2NZux90Xb3Ggf5MrdW2dygq4f4Jt7lqdc5slcvxD2btkocPYgvfETjF9+kFY1y9EkrKzRqW6JFP/tqNIMIvvxkNNvbNm6hQNnZK5fCLt0eifsnAXNHmVyfBrn0nN4rmO40alumRT/7er4Fk7OzvzP9Qc+XbzP6DRCCGtY9j64+ZIa/SgTVhyiXUQg0VVs4yrdwkjx3y7fiqhWL9BRbeDcjkXsOnHR6ERCiOJ0civsngfNHuebTSmkZObyfMcIo1PdFin+4tD8SfLLhPK26xQ++WOn0WmEEMUp7n1w9yMlajgTVx2iU50gIiuX7vX2r0eKvzi4uOPUeSQ1OEalAz+wMcF+ligSwqEd3wT7focWT/H1X0mkZuXZ9Nz+JVL8xaVWN/JD2/C8y0zGL1iP1troREKI2xU3EjzKci7yQb5dfZhu9YOpHexrdKrbJsVfXJTC6c4P8FZZ3HHia1bsTzI6kRDidhxdBweXQMtn+WLVKTJz83mug22tyVMUKf7iVL4WuvHDDHBeyi+/LZBRvxC2LG4keAWSWPN+pq47Qp9GlalR3sfoVMVCir+YObV7hVzXMgw8P47ft580Oo4Q4lYcXgmHV0DL5xmzLBEUPNvB9uf2L5HiL24eZXDp+AZNTXvYtGASeflmoxMJIW6G1hD3HvgEsy+kD7O2HGdoi1AqlvEwOlmxkeK3AlOjwaSUqcOwzG/5dYMs5SCETTkUB0fXQKsX+GDJEbzdnHm8bXWjUxUrKX5rMDnhe/dHVFLJXFz0Idl5+UYnEkLcCK1h6UjwrczGgO4s3n2GR9tUp4ynq9HJipUUv5Wo0FjOVu3OgLw5zF661ug4QogbsX8RJG5Etx7BqD8PE+jjxgOxoUanKnZS/FYU2Gs0ymQiYM3bpGTkGh1HCHEtWlvO5ClTlaXuHdh45DzPtK+Jp6vVVq83zHWLXyn11LVuii6uwa8yKTFP0ZH1zJ/7k9FphBDXsncBnIzH3PolRi86SFg5L/o3DjE6lVXcyIg/CNiglPpZKdVF2eJdBwxUvtOLnHOpQMzuURxLkgXchCiVzGbLmTxlqzM7vyX7TqfxQqdwXJzsc1Lkur+V1vp/QE0sd88aCuxXSr2nlLKvt7mtxcUD1XkkEeoY62d8ZHQaIURhdv8Kp3eQ0/IlPlx0kMhKftxZL/j6P2ejbuifM225BPVUwUce4A/8opT6wIrZ7IZ/o94c8Y2hw6mJ7DqQYHQcIcSVzPmWu2uVi2DCuShOXczif91qYzLZ7+TGjczxP6OU2gR8AKwGIrXWjwGNgN5WzmcflCKg7yd4q0yOz/qfLOUgRGmyYxac3UNKsxcYtyKBznWDaFrN9m6gfjNuZMRfFuilte6stZ6htc4F0Fqbge5WTWdHvEPqsz+kH+3T57Nh3XKj4wghAPLzLHfXCqrH6CMR5OSZeblrbaNTWd2NzPG/obU+UsRzu4s/kv2q3u890pQ3botfJU8u6hLCeNt/hnMHOR71DD9uTGRw81DCynkZncrq7PMt61LK1SeA41HP0yB/B+t++8boOEI4tvxcWD4aghvwys4q+Li78HT7GkanKhFS/CWszl1Pc9i5GtW3jCI1NcXoOEI4rvhpcD6BbTWfYOWBZJ66o4bdLc1QFCn+EqacnDF3HkUwScT/+LbRcYRwTHnZsOIjdKUYXowPIjTAk8HNQ41OVWKk+A1QvXFn4n3bEXP8e44e3mt0HCEcz+bvIeUYS4MfYt+ZdF7uWgtXZ8epQ8f5TUuZyv0sF3Od/uUlg5MI4WByM2Hlx+RVbsZ/tgTQONSfznUrGJ2qREnxG6Rc5RpsD3uAxunL2LZqntFxhHAcmyZD6kl+8BxEckYub9xVF0dbiUaK30D1+73GKRWI19L/kZebY3QcIexfTgasHENGxRa8vaMs9zauQr1KfkanKnFS/AZy9/ThZNNXqW5OYOOsT42OI4T92zAR0s/wYW5fPF2deLGT/dxH92ZI8RssqtMQdrnWp9buz0hJPm10HCHsV3YqrP6UpKCWfHssiOc7hhPg7WZ0KkNI8RtMmUx49PwIH53OnumvGB1HCPu1/ivISOaVC3cRHuTNwGZVjU5kGCn+UiCsblM2BN5No7OzObTzL6PjCGF/slJgzf9xuGwrFqWE8OZddXG207X2b4Tj/ualTO0Bo0hXnqT/+iLmfLPRcYSwL+vGQ9YFXjh7J3dGVqBFjXJGJzKUFH8p4RdQgYN1nyEyZyvrFkw2Oo4Q9iPjHKwdS7x3K3YRxn/vtP/VN6/HasWvlPpGKXVGKbXjisfKKqUWKaX2F3yWe/leIeqe50hwCiV003ukpMhtGoUoFmvHQvZF/pPcjcfa1KCyv6fRiQxnzRH/ZKDLVY+9DCzRWtcElhR8LwqYnF3QXUdRkbNsmP6W0XGEsH3pyej1XxLnHEumfy0eaVPN6ESlgtWKX2u9Ajh31cM9ge8Kvv4OuNtax7dVYTFd2VWmLbEnv2f3nl1GxxHCtq3+FJ2Twbvpd/N2z7q4uzgZnahUKOk5/iCt9cmCr08BQUVtqJQarpTaqJTaePbs2ZJJV0qE3DsGpeDsrP9gNsttGoW4JWlnMP81gXnmFtSqF0PbiPJGJyo1DHtzt+AG7kW2mtZ6gtY6RmsdExgYWILJjOdToToJEQ/SOmcFS/6YY3QcIWySXjkGnZfDV6ovr3WvY3ScUqWki/+0UioYoODzmRI+vs2I6P0aSaZAQta/SfLFDKPjCGFbLp7EvGESM/Na0bdTGyr4uRudqFQp6eKfCwwp+HoI8GsJH99mKFcvctq/RS0SWPLDR0bHEcKm5Cz7EG3OZ2HAIAY58BW6RbHm6ZzTgbVAhFLquFLqQWAU0FEptR/oUPC9KELFFvdxzLchHU5+xZodB4yOI4RtuHAM05bv+Tm/DU/36ejQV+gWxZpn9QzQWgdrrV201pW11pO01sla6/Za65pa6w5a66vP+hFXUoryfT/FT2WQOOd1MnLyjE4kRKmXvPA98s2aE5FPEBVSxug4pZL8U1jKuYVEkRTen3tyf2fq3D+MjiNEqZabdAi/PT/xq1MHHr6rjdFxSi0pfhsQ1HMkuc6e1Nn2HjuOXzA6jhCl1oEZr5OnTZS/87/4ebgYHafUkuK3BV4B0Pa/tDTtYPaPE8iTRdyE+Jcj+7YRfmo+q/170jamgdFxSjUpfhvh0WI4qb41GXJxApOX7zE6jhClitmsOTLrdXKUK/XvfcPoOKWeFL+tcHLBu+dHVDGdJXXZpxxNlnP7hbhk/pKltMxcxtHq9xNYoYrRcUo9KX4boqq3JavGnTyi5vDBjCWynIMQQOKFTFxWfUi2yZ3wXq8aHccmSPHbGPdu7+PmpOmQOI5pfx01Oo4QhtJaM/6nX+mq1pITMxzl5dg3WLlRUvy2xj8UU+zT3O20hoULZsuUj3BoszYn0vL4RLKdvfG74zmj49gMKX4bpFo9T753MK+qybw0Y7NM+QiHdOJCJj/Nm0cXpw24xD4FHnJfpxslxW+LXL1w6vQOddRhqh6bw/drE4xOJESJ0lrzn5nbeFz/TL5bGUzNHzM6kk2R4rdVkX3QIc141W0GYxduIiEp3ehEQpSYqeuPknpgHW3VZpxaPg3ufkZHsilS/LZKKVTX0fiYU3jKaRYvzthKvkz5CAdwJDmd937bzdt+v6I9A6DJI0ZHsjlS/LasYhQqejAD1R+cP7qDb1cfNjqREFaVb9a88PNWGjvtpX7WJlTss+DmbXQsmyPFb+vav45y9eTzMj/xwcI97D550ehEQljNpFWH2HjkPB+Vmw9e5aHxQ0ZHsklS/LbOqxyq7SvUzdxId7etPPtjPFm5+UanEqLY7Tudykd/7uOpsBOUT/oLWj0Prp5Gx7JJUvz2oMnDUC6CkZ4/cPj0OUYvlLV8hH3Jys3n6elb8HVz4mn1M/hUhEYPGB3LZknx2wMnF+jyPh5pRxlXbS3frk5g+b6zRqcSotiM+n0Pe06lMrFVKi4n/oLWL4CL3Ef3Vknx24sa7SHiTtqfnUKzwGxenLGV5LRso1MJcduW7jnN5DUJDGsRStS+/wO/EGg4yOhYNk2K3550Hoky5/JlhfmkZOTyn5nb0VpO8RS268zFLEbM2EbtYF9eCd0DJ7ZA21fA2c3oaDZNit+elK0GzZ+kzP6ZfNQ8i8W7T/ODLOQmbJTZrHlhxlbSc/KY0DYPl/nPQFA9aHCv0dFsnhS/vWn1AvgEc9eJz2hdoyxvz9slp3gKm/TN6sOs3J/E5y0yCfltIHiXh/tngMnJ6Gg2T4rf3rh5Q4e3UCe2MLbOHnw9XHhi2mbSsvOMTibEDduRmMLohXt4KuwEHTc/Ab4V4YEFls/itknx26P6/aByE3xWj+SLXtVJSE7n1dky3y9sQ0pmLo9P20xXzz08f/Y1VJmqMPQ38KlgdDS7IcVvj5SCrqMhPYmmRyfybIdwfo0/wU8bjhmdTIhr0lozYsZWqqes41PzKFTZajB0vmWaRxQbKX57VSkaGg6EdeN40uMPWlYP4I25O2W+X5Rqk1YdJnfPQia6fowpMByGzAO5q1axk+K3Z11GQcSdmP58lYm+Ewh0N8t8vyi1Nh05x8aFU/na9RNMFerC4LngFWB0LLskxW/P3Lyh3xRo9z/cd89ioe9IcpMTeGWWzPeL0iU5LZtfpozjC5dPoWID1OBfwbOs0bHslhS/vTOZoM0IuO8nvNOP8afX6yRtX8SkVbKEsygd8s2aH779jHdyPyY7KArnwXPAo4zRseyaFL+jCO8MD8fh7h/MNNf3Of3Hx6w5IOv5COMt+OFzHk96j3Nlo/Aa9iu4+xodye5J8TuScjVQDy3GHN6VV52ncmHaUE6cTTY6lXBgW+aN4879b5DgHUX5x+aDm4/RkRyCFL+jcfPB+d6pJDd9iS7m1WR+2YHsszLtI0reibgJNNj4X3a4RVH5yXng6mV0JIchxe+ITCYCur7K1tZfEZh3irwv26APxhmdSjiQ9NVfU3H5CNaZogh+ZDZuHjLSL0lS/A6sYfv+zGw0hcRcb/SUXrDm/0DO9hFWlr/uK7wWvUicuSGeg6ZTPsDf6EgOR4rfwQ3p3p7/q/Ylf+THwJ//g5kPQU6G0bGEvVo3HqeFL/FnfiOSu00iqlqw0YkckiHFr5RKUEptV0rFK6U2GpFBWJhMitH3teCLcq/xmb4XvWMmTOoE5xOMjibszerPYeHL/J7fmPWNxtCnaXWjEzksI0f87bTWUVrrGAMzCMDT1ZmJQxszzbUvL7q8ivnCUZjQFmTeXxSXlR/Dotf4Lb8Zs6q9w397NDA6kUOTqR4BQLCfBxOHxPBbVj2e9PoYs3cQTO1lGaXJvL+4HctGw5K3ma9bMi7gFT65rzFOJmV0KodmVPFr4E+l1Cal1PDCNlBKDVdKbVRKbTx7Vi40Kgn1K5fhk35RLDjhyUtlxqBr3wWLXoOZD0JOutHxhK3RGpaOhGXvscDUjpGuTzPxgaZ4uzkbnczhGVX8LbXW0UBX4AmlVOurN9BaT9Bax2itYwIDA0s+oYPqGhnMiM4R/LL9Ah/4vALtX4cds2TeX9wcrWHJ27DiAxa5deLF3OF8PbQZwX4eRicTGFT8WuvEgs9ngNlAEyNyiMI93rY69zetwvjlh/jW1Avu/wVSjhXM+y81Op4o7bS2/E9x1RiWeXfnkYuD+WxAI+pV8jM6mShQ4sWvlPJSSvlc+hroBOwo6RyiaEop3u5Zj051gnh7/i7mZ9aBh+PAJxim9oZVn8q8vyic1rDwFVjzf6zyv4ehSQN4u2d9OtYJMjqZuIIRI/4gYJVSaivwF/Cb1nqhATnENTiZFJ8PaEhMVX+e/2kra877wYOLoHYPWPwG/DJM5v3FP5nNsGAErB/PX0H3MvBkH57rEMHAZlWNTiauUuLFr7U+pLVuUPBRV2s9sqQziBvj7uLExMGNCS3nyfApm9iZnA99J0OHN2HnbJjYEc7JOj8CS+n/9hxs+JqtVQbT78hdDG4eytPtaxidTBRCTucU1+Tn6cLkB5rg4+7MkG82cDApHVo+BwN/gYvHLfP+B5YYHVMYyWyGeU/BpsnsqfEwPfd1plv9irxxV12UktM2SyMpfnFdFct4MOXBpmituf/r9RxNzoAaHWD4MvCtBNP6wKpPZN7fEZnz4dfHYctUDtR+nG672hFboxxj+jWQc/VLMSl+cUNqlPdm6kNNycrL576J6zhxIRPKVoOHFkGdnrD4TZgxFLLTjI4qSkp+Hsx+BLZO52DdZ+i6rRUNKpfhq0ExuDk7GZ1OXIMUv7hhtYN9+X5YE1Iycrl/4nrOXMyyrKHe51vo8Bbsnms53//cIaOjCmvLz4VZD8H2GRxq8CJd45tTJ9iXycOayAVaNkCKX9yU+pXLMHlYY05fzOL+ietJTssGpaDls5bz/S8mwoR2cGCx0VGFteTlwC8PwM7ZHI5+ha6bYqhR3pvvhzXF193F6HTiBkjxi5vWqGpZJg6J4ei5DAZ8vY6zqdmWJ2q0vzzvP7UPrBwj8/72Ji8bZgyB3fNIaPwad26IIqycF1Mfaoqfp5S+rZDiF7ekRfVyfDO0McfOZdJ/wlpOpWRZnigbZpn3r3sPLHnLUhIy728fcrPgp0GwdwH7Y96g89q6VPb3YOpDTSnr5Wp0OnETpPjFLYutUY7vhjXhzMVs+n21luPnC27g4uoFfb6Bjm/D7nkwqSMkHzQ2rLg9uZnw432w/w92NnqbbmtrUz3Qmx+HN6Oct5vR6cRNkuIXt6VJWFmmPNiE8xk59P9qHUeSC67mVQpin4GBMyH1JHzdDvbLvL9NysmA6ffCwaVsafgOPdbWpE5FX6Y/3IwAKX2bJMUvblvDKv5Mf7gZ6Tl59P1yLbtPXrz8ZPU7LPP+fiGW8/1Xfizz/rYkOw1+6AeHV7C+wbv0WledmKr+Mqdv46T4RbGoV8mPnx9pjkkp+n21lnWHki8/6R8KD/4J9XpZlur9ebDM+9uC7FSY1gd9ZDV/RrxN//VhtKoZyOQH5JRNWyfFL4pNeJAPMx9vQZCvO4O/+Yvft5+8/KSrF/SeBJ3ehT3zYWIHmfcvzbIuwpRe6GN/8VOVNxgeX417GlZi4uAYPFzl4ixbJ8UvilWlMh788mhz6lX05fEfNjNlbcLlJ5WCFk/BwFmQdspyvv/+RYZlFUXIvABT7kaf2MxXgf/j5b01ebRNdT7u2wBXZ6kMeyB/iqLYlfF0ZdpDzbgjojyv/bqTd+fvIt98xbx+9XaWeX//KjCtL6z4SOb9S4uMc/B9T/TJbYz2+S+jj0XwVo+6vNy1FiZZe8duSPELq/BwdeKrQY0Y2iKUiasO89B3G0jNyr28gX8oDPsT6vWGpe/Az4Msc8rCOOnJ8H0PzKd3McLpJb5NrsO4+6IZ0iLU6GSimEnxC6txdjLxZo+6vHt3PVbsT6L3+DWWlT0vcfWE3hOh00jY85vM+xsp7Sx8dxf5Z/bxSN4LrFTR/PxIc7pGBhudTFiBFL+wuoHNqjJlWBNOX8zm7nGr/3nGj1LQ4kkYNBvSzljm/ff9aVxYR5R2Bv1dd3KTDjA463nOBLVi3pMtaRBSxuhkwkqk+EWJaFGjHHOeiMXf04X7vl7H+GUHMV8571+t7eV5/x/6wYoPLTf4ENaVeor8b+4kJymBQVkjKN+gCz8Nb0Z5X3ejkwkrkuIXJSasnBe/PtmSrpHBjF64h+FTNpGSceW8f1XLvH9kX1j6rsz7W1tKItlfdyH73DGG5PyHDl17M6ZfA9xd5HRNeyfFL0qUt5szXwxoyJt31WH5vjN0/2Il24+nXN7A1RN6TYDO78He3+Hr9pB0wLjAdkpfOErql53ITTnFsy6vM2L4AzzUqprcKtFBSPGLEqeUYmhsGD890pz8fE2v8av5cvnBy6d8KgXNn7DM+6eftazzs3ehsaHtyMWTB0j+oiM64xwfB3/AqGcfplFVf6NjiRIkxS8ME13Fn9+ebkX7WkGM+n0PA75ed3mFT4BqbeCR5ZZTP6ffC8s/kHn/27R+00bSv+qMc24qf8RM4LXhg2RJZQckxS8M5e/lyviB0XzYpz67Tlyk66crmbX5OPrSBV1lqljW+anfD+JGWub9sy5ee6fiXzJy8vj0p9+pMrcvniqbpF4z6HvXXXJRloOS4heGU0rRNyaE359pRUQFH57/eSsPfbeRxAuZlg1cPOCer6DLKMu8/8T2kLTf2NA2ZPWBJB79ZDoDdj2Kr4sZ9wcXUKNBrNGxhIGUtoFL5WNiYvTGjRuNjiFKQL5Z8+3qw3z85z6Ughc7RTCkRShOl0amh1fAjKGWm333mgARXQ3NW5qdS8/h3d92sX3Len50fw8fN2dch82H8rWNjiZKiFJqk9Y65urHZcQvShUnk+KhVtX487nWNAkry9vzd9Fr3Grij12wbBDW2nK+f9kwy7z/stEy738Vs1nz88ZjtP94GXu3rmWO1/uU9XLH9cHfpfQFICN+UYpprZm/7SRvzdtFUlo2vRpW4qUutajg5265FeC8Z2HbjxDRDe75Etx9jY5suL8On+Od+bvYnphC74rnGJ3xGs4uHjB0PgRUNzqeKGFFjfil+EWpl5adx7i4A0xcdRgnpXisbXUeahWGp4sTrP8K/vivpdT6T4PAcKPjGuJocgbv/76b33ecItjPnfea5dF2/XCUqzcMnQdlqxkdURhAil/YvGPnLOW2YPspArxceaxtde5vWhWPxDWWef+8bMu8f607jY5aYo6fz2Bs3AFmbDyOi5OJx9pWZ3j187hP7wMefjBknuV0WOGQpPiF3dh05DyfLt7Hyv1JBPq48Xjb6gwIN+E+azCcjIe2r0Drl8Bkv29hJV7ILCj8YygU/RuH8OQdNQhK2QZTe4NnWRgyH8qEGB1VGEiKX9id9YeSGbNoH+sPn8Pf04WhjYMYnvoFHrt+hog7C+b9/YyOWay2HrvApFWHWbD9JEpBv5gQnmhXg4plPODIGsuNbbyDLCN9v0pGxxUGk+IXduuvw+eYuPIQi3afxsWkGB2yjrtPfwFlq6Hu/cHm5/1z8sws3n2ab1cfZkPCebzdnOkXE8KwlqFU9ve0bHR4pWVVU99KltL3lXX0hRS/cACHk9L5dvVhZmw8Tv28HXzl/jmepjyy7hqHb9TdRse7abtOXGTGpmP8Gn+Cc+k5VPb3YGiLUPo3DsHH3eXyhgcWw48DLaubDp4LPkHGhRalihS/cBipWbnM33aSJes281TSWzQwHWK270CyYkfQsW4w5bzdSjaQOR9yMyynoP7r8+WvdU46Z89d4OCJMxw5nUxGWipephzC/EyE+SnKuZlR/9pHBmSeh6C6MHA2eAeW7O8mSjUpfuGQ9ieeJXP2M9RP+o1F+dG8mPc4EaGVaRMeSGz1AOoFueNszvpXCRda0DnpRZZ24Y8VbJ+fc9O5c3FGO7vj7OaFydUTXDwtS1a7eFi+vvKzZwA0eww8ZIVN8U+lqviVUl2AzwAnYKLWetS1tpfiF7dFa/RfE2Dhf8kxuZFpdsbJnIMH2TirW7jq19m9oHQvFXAhZeziAa5e/3hMO3uQnOPEkYuagxfM7Dyby44zuaSaXckzuVGnagVa1alCu7pVKO/vU/yvg3A4RRW/swFBnICxQEfgOLBBKTVXa72rpLMIB6EUqukjENwAt63TcVMmMrQLh9LgcIqZ/efMJKZDpnYjEzc8vLzx9/MjwL8M5cuWoZy/P76+vvj7+eHv54enmwuuTqZ/rGyZl28mJ99MalYeSWnZJKflkJSWzdFzGRw5lcHhpHQOnU3jYlYeAC5OiroV/WjRKoAW1cvRqKo/Hq5y5ytRMkq8+IEmwAGt9SEApdSPQE9Ail9YV5Vmlg/AEwgv+OgMpGTmsjMxhe2JKew+eZEtyRkkHEznwvYc4HTBxz+5OCmcTIrcfH35JjKFqOjnTmg5L7o3qEi9in5EVvIjvII3bs5S9MIYRhR/JeDYFd8fB5pevZFSajgwvODbNKXU3hLIdi3lgCSDM5QW8lpcdt3X4giwtmSyGE3+XlxWWl6LqoU9aETx3xCt9QRggtE5LlFKbSxsrswRyWtxmbwWl8lrcVlpfy2MuKY9EbjyOvLKBY8JIYQoAUYU/wagplIqTCnlCtwLzDUghxBCOKQSn+rRWucppZ4E/sByOuc3WuudJZ3jFpSaaadSQF6Ly+S1uExei8tK9WthExdwCSGEKD72u26tEEKIQknxCyGEg5HivwVKqReUUlopVc7oLEZRSn2olNqjlNqmlJqtlCpjdKaSppTqopTaq5Q6oJR62eg8RlFKhSil4pRSu5RSO5VSzxidyWhKKSel1Bal1HyjsxRGiv8mKaVCgE7AUaOzGGwRUE9rXR/YB7xicJ4SdcXSI12BOsAApVQdY1MZJg94QWtdB2gGPOHAr8UlzwC7jQ5RFCn+m/cJ8BLg0O+Ka63/1FrnFXy7Dsv1GI7k76VHtNY5wKWlRxyO1vqk1npzwdepWArPYW//pZSqDHQDJhqdpShS/DdBKdUTSNRabzU6SykzDPjd6BAlrLClRxy27C5RSoUCDYH1Bkcx0qdYBoe3sPRrySi1SzYYRSm1GKhQyFOvAv/FMs3jEK71Wmitfy3Y5lUs/9WfVpLZROmjlPIGZgLPaq0vGp3HCEqp7sAZrfUmpVRbg+MUSYr/KlrrDoU9rpSKBMKArUopsExtbFZKNdFanyrBiCWmqNfiEqXUUKA70F473gUhsvTIFZRSLlhKf5rWepbReQwUC/RQSt0JuAO+SqmpWuuBBuf6B7mA6xYppRKAGK11aViBr8QV3ExnDNBGa33W6DwlTSnljOVN7fZYCn8DcJ+NXIVerJRlJPQdcE5r/azBcUqNghH/i1rr7gZH+ReZ4xe36gvAB1iklIpXSn1pdKCSVPDG9qWlR3YDPzti6ReIBQYBdxT8XYgvGPGKUkpG/EII4WBkxC+EEA5Gil8IIRyMFL8QQjgYKX4hhHAwUvxCCOFgpPiFKIWUUpOVUn2MziHskxS/EEI4GCl+YdeUUo0L7hngrpTyKlgvvt5V2zx6xYVHh5VScQWPD1BKbVdK7VBKjb5i+6IeTyu4T8FOpdRipVQTpdQypdQhpVSPgm2cCrbZUJDrkYLHlVLqi4L1/RcD5UvkBRIOSS7gEnZPKfUulnVTPIDjWuv3i9jOBVgKfABswrLcdCPgPPAn8DnwV2GPa63nKKU0cKfW+nel1GzAC8vyvHWA77TWUUqp4UB5rfW7Sik3YDXQF8uKlo8BXYAgYBfwkNb6l2J/QYTDk0XahCN4G8taOlnA09fY7jNgqdZ6XsES3MsurUOklJoGtMZyH4bCHp8D5AALC/a1HcjWWucqpbYDoQWPdwLqXzF/7wfULNjHdK11PnBCKbX0tn9rIYogxS8cQQDgDbhgGfmnX71BwUqjVbGsv3Orcq9YpdQMZANorc0Fi7oBKOAprfUfVx1f1rYRJUbm+IUj+Ap4Dcs9A0Zf/aRSqhHwIjBQa33p5hl/AW2UUuUKbrM4AFh+jcdv1B/AYwXTSiilwpVSXsAKoH/BewDBQLtb+UWFuBEy4hd2TSk1GMtI/IeCol6jlLpDa33lVMqTQFkgruBeCxu11g8V3EA9Dsso/bcrbj5T6OM3aCKWaZ/NBcsZnwXuBmYDd2CZ2z8KrL3FX1mI65I3d4UQwsHIVI8QQjgYKX4hhHAwUvxCCOFgpPiFEMLBSPELIYSDkeIXQggHI8UvhBAO5v8Bl1WBviTnZdIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, y, label=\"ground truth\")\n",
    "plt.plot(x, y_hat, label=\"predictions\")\n",
    "plt.xlim([-5, 5])\n",
    "plt.ylim([0, 25])\n",
    "plt.ylabel('y')\n",
    "plt.xlabel('x zoomed')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c143c4ce",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "((torch.tensor(y)-torch.tensor(y_hat))**2).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17fb3818",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tips\n",
    "- Start small\n",
    "    - Small model\n",
    "    - Small dataset $\\rightarrow$ overfit a single batch to check that the pipeline is working\n",
    "- Always check monitor the training progress e.g. Tensorboard\n",
    "- Visualize the results as low loss does not always imply good results (e.g. loss weakly choosen for the problem)\n",
    "- Don't be too naive when approaching a problem\n",
    "- Don't force to use DL on every problem"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d644be5",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "![](meme.jpg)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}