CS224W_Colab3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XuXWJLEm2UWS"
   },
   "source": [
    "# **CS224W - Colab 3**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8gzsP50bF6Gb"
   },
   "source": [
    "In Colab 2 we constructed GNN models by using PyTorch Geometric's built in GCN layer, `GCNConv`. In this Colab we will go a step deeper and implement the **GraphSAGE** ([Hamilton et al. (2017)](https://arxiv.org/abs/1706.02216)) layer directly. Then we will run our models on the CORA dataset, which is a standard citation network benchmark dataset.\n",
    "\n",
    "**Note**: Make sure to **sequentially run all the cells in each section** so that the intermediate variables / packages will carry over to the next cell\n",
    "\n",
    "Have fun and good luck on Colab 3 :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MSaetj53YnT6"
   },
   "source": [
    "# Device\n",
    "We recommend using a GPU for this Colab.\n",
    "\n",
    "Please click `Runtime` and then `Change runtime type`. Then set the `hardware accelerator` to **GPU**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "67gOQITlCNQi"
   },
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "J_m9l6OYCQZP",
    "outputId": "c64fadc7-2007-4c57-c49e-16d8be113d71"
   },
   "outputs": [],
   "source": [
    "# Install torch geometric\n",
    "import os\n",
    "# if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "#     !pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.9.0+cu111.html\n",
    "#     !pip install torch-sparse -f https://pytorch-geometric.com/whl/torch-1.9.0+cu111.html\n",
    "#     !pip install torch-geometric\n",
    "#     !pip install -q git+https://github.com/snap-stanford/deepsnap.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "PRfgbfTjCRD_",
    "outputId": "7d78402e-5e4d-405b-9980-8592b2f49744"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/arc/project/st-jiaruid-1/miniconda3/envs/gnn/lib/python3.8/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch_geometric\n",
    "torch_geometric.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZoXlf4MtYrbz"
   },
   "source": [
    "# 1) GNN Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TQy2RBfgYut4"
   },
   "source": [
    "## Implementing Layer Modules\n",
    "\n",
    "In Colab 2, we implemented a GCN model for node and graph classification tasks. However, for that notebook we took advantage of PyG's built in GCN module. For Colab 3, we provide a build upon a general Graph Neural Network Stack, into which we will be able to plugin our own module implementations: GraphSAGE and GAT.\n",
    "\n",
    "We will then use our layer implemenations to complete node classification on the CORA dataset, a standard citation network benchmark. In this dataset, nodes correspond to documents and edges correspond to undirected citations. Each node or document in the graph is assigned a class label and features based on the documents binarized bag-of-words representation. Specifically, the Cora graph has 2708 nodes, 5429 edges, 7 prediction classes, and 1433 features per node. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H4ne6Gw-CT5G"
   },
   "source": [
    "## GNN Stack Module\n",
    "\n",
    "Below is the implementation of a general GNN stack, where we can plugin any GNN layer, such as **GraphSage**, **GAT**, etc. This module is provided for you. Your implementations of the **GraphSage** and **GAT** (Colab 4) layers will function as components in the GNNStack Module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "Ys8vZAFPCWWe"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch_scatter\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import torch_geometric.nn as pyg_nn\n",
    "import torch_geometric.utils as pyg_utils\n",
    "\n",
    "from torch import Tensor\n",
    "from typing import Union, Tuple, Optional\n",
    "from torch_geometric.typing import (OptPairTensor, Adj, Size, NoneType,\n",
    "                                    OptTensor)\n",
    "\n",
    "from torch.nn import Parameter, Linear\n",
    "from torch_sparse import SparseTensor, set_diag\n",
    "from torch_geometric.nn.conv import MessagePassing\n",
    "from torch_geometric.utils import remove_self_loops, add_self_loops, softmax\n",
    "\n",
    "class GNNStack(torch.nn.Module):\n",
    "    def __init__(self, input_dim, hidden_dim, output_dim, args, emb=False):\n",
    "        super(GNNStack, self).__init__()\n",
    "        conv_model = self.build_conv_model(args.model_type)\n",
    "        self.convs = nn.ModuleList()\n",
    "        self.convs.append(conv_model(input_dim, hidden_dim))\n",
    "        assert (args.num_layers >= 1), 'Number of layers is not >=1'\n",
    "        for l in range(args.num_layers-1):\n",
    "            self.convs.append(conv_model(args.heads * hidden_dim, hidden_dim))\n",
    "\n",
    "        # post-message-passing\n",
    "        self.post_mp = nn.Sequential(\n",
    "            nn.Linear(args.heads * hidden_dim, hidden_dim), nn.Dropout(args.dropout), \n",
    "            nn.Linear(hidden_dim, output_dim))\n",
    "\n",
    "        self.dropout = args.dropout\n",
    "        self.num_layers = args.num_layers\n",
    "\n",
    "        self.emb = emb\n",
    "\n",
    "    def build_conv_model(self, model_type):\n",
    "        if model_type == 'GraphSage':\n",
    "            return GraphSage\n",
    "        elif model_type == 'GAT':\n",
    "            # When applying GAT with num heads > 1, you need to modify the \n",
    "            # input and output dimension of the conv layers (self.convs),\n",
    "            # to ensure that the input dim of the next layer is num heads\n",
    "            # multiplied by the output dim of the previous layer.\n",
    "            # HINT: In case you want to play with multiheads, you need to change the for-loop that builds up self.convs to be\n",
    "            # self.convs.append(conv_model(hidden_dim * num_heads, hidden_dim)), \n",
    "            # and also the first nn.Linear(hidden_dim * num_heads, hidden_dim) in post-message-passing.\n",
    "            return GAT\n",
    "\n",
    "    def forward(self, data):\n",
    "        x, edge_index, batch = data.x, data.edge_index, data.batch\n",
    "          \n",
    "        for i in range(self.num_layers):\n",
    "            x = self.convs[i](x, edge_index)\n",
    "            x = F.relu(x)\n",
    "            x = F.dropout(x, p=self.dropout,training=self.training)\n",
    "\n",
    "        x = self.post_mp(x)\n",
    "\n",
    "        if self.emb == True:\n",
    "            return x\n",
    "\n",
    "        return F.log_softmax(x, dim=1)\n",
    "\n",
    "    def loss(self, pred, label):\n",
    "        return F.nll_loss(pred, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5nW_XpEwASNZ"
   },
   "source": [
    "## Creating Our Own Message Passing Layer\n",
    "\n",
    "Now let's start implementing our own message passing layers! Working through this part will help us become acutely familiar with the behind the scenes work of implementing Pytorch Message Passing Layers, allowing us to build our own GNN models. To do so, we will work with and implement 3 critcal functions needed to define a PyG Message Passing Layer: `forward`, `message`, and `aggregate`.\n",
    "\n",
    "Before diving head first into the coding details, let us quickly review the key components of the message passing process. To do so, we will focus on a single round of messsage passing with respect to a single central node $x$. Before message passing, $x$ is associated with a feature vector $x^{l-1}$, and the goal of message passing is to update this feature vector as $x^l$. To do so, we implement the following steps: 1) each neighboring node $v$ passes its current message $v^{l-1}$ across the edge $(x, v)$ - 2) for the node $x$, we aggregate all of the messages of the neighboring nodes (for example through a sum or mean) - and 3) we transform the aggregated information by for example applying linear and non-linear transformations. Altogether, the message passing process is applied such that every node $u$ in our graph updates its embedding by acting as the central node $x$ in step 1-3 described above. \n",
    "\n",
    "Now, we extending this process to that of a single message passing layer, the job of a message passing layer is to update the current feature representation or embedding of each node in a graph by propagating and transforming information within the graph. Overall, the general paradigm of a message passing layers is: 1) pre-processing -> 2) **message passing** / propagation -> 3) post-processing. \n",
    "\n",
    "The `forward` fuction that we will implement for our message passing layer captures this execution logic. Namely, the `forward` function handles the pre and post-processing of node features / embeddings, as well as initiates message passing by calling the `propagate` function. \n",
    "\n",
    "\n",
    "The `propagate` function encapsulates the message passing process! It does so by calling three important functions: 1) `message`, 2) `aggregate`, and 3) `update`. Our implementation will vary slightly from this, as we will not explicitly implement `update`, but instead place the logic for updating node embeddings after message passing and within the `forward` function. To be more specific, after information is propagated (message passing), we can further transform the node embeddings outputed by `propagate`. Therefore, the output of `forward` is exactly the node embeddings after one GNN layer.\n",
    "\n",
    "Lastly, before starting to implement our own layer, let us dig a bit deeper into each of the functions described above:\n",
    "\n",
    "1. \n",
    "\n",
    "```\n",
    "def propagate(edge_index, x=(x_i, x_j), extra=(extra_i, extra_j), size=size):\n",
    "```\n",
    "Calling `propagate` initiates the message passing process. Looking at the function parameters, we highlight a couple of key parameters. \n",
    "\n",
    "  - `edge_index` is passed to the forward function and captures the edge structure of the graph.\n",
    "  - `x=(x_i, x_j)` represents the node features that will be used in message passing. In order to explain why we pass the tuple `(x_i, x_j)`, we first look at how our edges are represented. For every edge $(i, j) \\in \\mathcal{E}$, we can differentiate $i$ as the source or central node ($x_{central}$) and j as the neighboring node ($x_{neighbor}$). \n",
    "  \n",
    "    Taking the example of message passing above, for a central node $u$ we will aggregate and transform all of the messages associated with the nodes $v$ s.t. $(u, v) \\in \\mathcal{E}$ (i.e. $v \\in \\mathcal{N}_{u}$). Thus we see, the subscripts `_i` and `_j` allow us to specifcally differenciate features associated with central nodes (i.e. nodes  recieving message information) and neighboring nodes (i.e. nodes passing messages). \n",
    "\n",
    "    This is definitely a somewhat confusing concept; however, one key thing to remember / wrap your head around is that depending on the perspective, a node $x$ acts as a central node or a neighboring node. In fact, in undirected graphs we store both edge directions (i.e. $(i, j)$ and $(j, i)$). From the central node perspective, `x_i`, x is collecting neighboring information to update its embedding. From a neighboring node perspective, `x_j`, x is passing its message information along the edge connecting it to a different central node.\n",
    "\n",
    "  - `extra=(extra_i, extra_j)` represents additional information that we can associate with each node beyond its current feature embedding. In fact, we can include as many additional parameters of the form `param=(param_i, param_j)` as we would like. Again, we highlight that indexing with `_i` and `_j` allows us to differentiate central and neighboring nodes. \n",
    "\n",
    "  The output of the `propagate` function is a matrix of node embeddings after the message passing process and has shape $[N, d]$.\n",
    "\n",
    "2. \n",
    "```\n",
    "def message(x_j, ...):\n",
    "```\n",
    "The `message` function is called by propagate and constructs the messages from\n",
    "neighboring nodes $j$ to central nodes $i$ for each edge $(i, j)$ in *edge_index*. This function can take any argument that was initially passed to `propagate`. Furthermore, we can again differentiate central nodes and neighboring nodes by appending `_i` or `_j` to the variable name, .e.g. `x_i` and `x_j`. Looking more specifically at the variables, we have:\n",
    "\n",
    "  - `x_j` represents a matrix of feature embeddings for all neighboring nodes passing their messages along their respective edge (i.e. all nodes $j$ for edges $(i, j) \\in \\mathcal{E}$). Thus, its shape is $[|\\mathcal{E}|, d]$!\n",
    "  - In implementing GAT we will see how to access additional variables passed to propagate\n",
    "\n",
    "  Critically, we see that the output of the `message` function is a matrix of neighboring node embeddings ready to be aggregated, having shape $[|\\mathcal{E}|, d]$.\n",
    "\n",
    "3. \n",
    "```\n",
    "def aggregate(self, inputs, index, dim_size = None):\n",
    "```\n",
    "Lastly, the `aggregate` function is used to aggregate the messages from neighboring nodes. Looking at the parameters we highlight:\n",
    "\n",
    "  - `inputs` represents a matrix of the messages passed from neighboring nodes (i.e. the output of the `message` function).\n",
    "  - `index` has the same shape as `inputs` and tells us the central node that corresponding to each of the rows / messages $j$ in the `inputs` matrix. Thus, `index` tells us which rows / messages to aggregate for each central node.\n",
    "\n",
    "  The output of `aggregate` is of shape $[N, d]$.\n",
    "\n",
    "\n",
    "For additional resources refer to the PyG documentation for implementing custom message passing layers: https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "syDtxjxoCZgq"
   },
   "source": [
    "## GraphSage Implementation\n",
    "\n",
    "For our first GNN layer, we will implement the well known GraphSage ([Hamilton et al. (2017)](https://arxiv.org/abs/1706.02216)) layer! \n",
    "\n",
    "For a given *central* node $v$ with current embedding $h_v^{l-1}$, the message passing update rule to tranform $h_v^{l-1} \\rightarrow h_v^l$ is as follows: \n",
    "\n",
    "\\begin{equation}\n",
    "h_v^{(l)} = W_l\\cdot h_v^{(l-1)} + W_r \\cdot AGG(\\{h_u^{(l-1)}, \\forall u \\in N(v) \\})\n",
    "\\end{equation}\n",
    "\n",
    "where $W_1$ and $W_2$ are learanble weight matrices and the nodes $u$ are *neighboring* nodes. Additionally, we use mean aggregation for simplicity:\n",
    "\n",
    "\\begin{equation}\n",
    "AGG(\\{h_u^{(l-1)}, \\forall u \\in N(v) \\}) = \\frac{1}{|N(v)|} \\sum_{u\\in N(v)} h_u^{(l-1)}\n",
    "\\end{equation}\n",
    "\n",
    "One thing to note is that we're adding a **skip connection** to our GraphSage implementation through the term $W_l\\cdot h_v^{(l-1)}$. \n",
    "\n",
    "Before implementing this update rule, we encourage you to think about how different parts of the formulas above correspond with the functions outlined earlier: 1) `forward`, 2) `message`, and 3) `aggregate`. As a hint, we are given what the aggregation function is (i.e. mean aggregation)! Now the question remains, what are the messages passed by each neighbor nodes and when do we call the `propagate` function? \n",
    "\n",
    "Note: in this case the message function or messages are actually quite simple. Additionally, remember that the `propagate` function encapsulates the operations of / the outputs of the combined `message` and `aggregate` functions.\n",
    "\n",
    "\n",
    "Lastly, $\\ell$-2 normalization of the node embeddings is applied after each iteration.\n",
    "\n",
    "\n",
    "<font color='red'>For the following questions, DON'T refer to any existing implementations online.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "RwG4HqCFCaOD"
   },
   "outputs": [],
   "source": [
    "class GraphSage(MessagePassing):\n",
    "    \n",
    "    def __init__(self, in_channels, out_channels, normalize = True,\n",
    "                 bias = False, **kwargs):  \n",
    "        super(GraphSage, self).__init__(**kwargs)\n",
    "\n",
    "        self.in_channels = in_channels\n",
    "        self.out_channels = out_channels\n",
    "        self.normalize = normalize\n",
    "\n",
    "        self.lin_l = None\n",
    "        self.lin_r = None\n",
    "\n",
    "        ############################################################################\n",
    "        # TODO: Your code here! \n",
    "        # Define the layers needed for the message and update functions below.\n",
    "        # self.lin_l is the linear transformation that you apply to embedding \n",
    "        #            for central node.\n",
    "        # self.lin_r is the linear transformation that you apply to aggregated \n",
    "        #            message from neighbors.\n",
    "        # Don't forget the bias!\n",
    "        # Our implementation is ~2 lines, but don't worry if you deviate from this.\n",
    "        self.lin_l = nn.Linear(in_channels, out_channels, bias=False)\n",
    "        self.lin_r = nn.Linear(in_channels, out_channels, bias=False)\n",
    "\n",
    "        ############################################################################\n",
    "\n",
    "        self.reset_parameters()\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        self.lin_l.reset_parameters()\n",
    "        self.lin_r.reset_parameters()\n",
    "\n",
    "    def forward(self, x, edge_index, size = None):\n",
    "        \"\"\"\"\"\"\n",
    "\n",
    "        out = None\n",
    "\n",
    "        ############################################################################\n",
    "        # TODO: Your code here! \n",
    "        # Implement message passing, as well as any post-processing (our update rule).\n",
    "        # 1. Call the propagate function to conduct the message passing.\n",
    "        #    1.1 See the description of propagate above or the following link for more information: \n",
    "        #        https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html\n",
    "        #    1.2 We will only use the representation for neighbor nodes (x_j), so by default\n",
    "        #        we pass the same representation for central and neighbor nodes as x=(x, x). \n",
    "        # 2. Update our node embedding with skip connection from the previous layer.\n",
    "        # 3. If normalize is set, do L-2 normalization (defined in \n",
    "        #    torch.nn.functional)\n",
    "        #\n",
    "        # Our implementation is ~5 lines, but don't worry if you deviate from this.\n",
    "        out = self.propagate(edge_index, x=(x, x), size=size)\n",
    "        out = out + self.lin_l(x) # skip connection\n",
    "        if self.normalize:\n",
    "            out = F.normalize(out, p=2.0, dim=1) # l2-norm\n",
    "        \n",
    "        ############################################################################\n",
    "\n",
    "        return out\n",
    "\n",
    "    def message(self, x_j):\n",
    "\n",
    "        out = None\n",
    "\n",
    "        ############################################################################\n",
    "        # TODO: Your code here! \n",
    "        # Implement your message function here.\n",
    "        # Hint: Look at the formulation of the mean aggregation function, focusing on \n",
    "        # what message each neighboring node passes.\n",
    "        #\n",
    "        # Our implementation is ~1 lines, but don't worry if you deviate from this.\n",
    "        out = self.lin_r(x_j)\n",
    "\n",
    "        ############################################################################\n",
    "\n",
    "        return out\n",
    "\n",
    "    def aggregate(self, inputs, index, dim_size = None):\n",
    "\n",
    "        out = None\n",
    "\n",
    "        # The axis along which to index number of nodes.\n",
    "        node_dim = self.node_dim\n",
    "\n",
    "        ############################################################################\n",
    "        # TODO: Your code here! \n",
    "        # Implement your aggregate function here.\n",
    "        # See here as how to use torch_scatter.scatter: \n",
    "        # https://pytorch-scatter.readthedocs.io/en/latest/functions/scatter.html#torch_scatter.scatter\n",
    "        #\n",
    "        # Our implementation is ~1 lines, but don't worry if you deviate from this.\n",
    "        out = torch_scatter.scatter(inputs, index, dim=node_dim, reduce='mean')\n",
    "\n",
    "        ############################################################################\n",
    "\n",
    "        return out\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V2dkgSuWCheU"
   },
   "source": [
    "## Building Optimizers\n",
    "\n",
    "This function has been implemented for you. **For grading purposes please use the default Adam optimizer**, but feel free to play with other types of optimizers on your own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "f_TIQ8NPCjBP"
   },
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "def build_optimizer(args, params):\n",
    "    weight_decay = args.weight_decay\n",
    "    filter_fn = filter(lambda p : p.requires_grad, params)\n",
    "    if args.opt == 'adam':\n",
    "        optimizer = optim.Adam(filter_fn, lr=args.lr, weight_decay=weight_decay)\n",
    "    elif args.opt == 'sgd':\n",
    "        optimizer = optim.SGD(filter_fn, lr=args.lr, momentum=0.95, weight_decay=weight_decay)\n",
    "    elif args.opt == 'rmsprop':\n",
    "        optimizer = optim.RMSprop(filter_fn, lr=args.lr, weight_decay=weight_decay)\n",
    "    elif args.opt == 'adagrad':\n",
    "        optimizer = optim.Adagrad(filter_fn, lr=args.lr, weight_decay=weight_decay)\n",
    "    if args.opt_scheduler == 'none':\n",
    "        return None, optimizer\n",
    "    elif args.opt_scheduler == 'step':\n",
    "        scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=args.opt_decay_step, gamma=args.opt_decay_rate)\n",
    "    elif args.opt_scheduler == 'cos':\n",
    "        scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=args.opt_restart)\n",
    "    return scheduler, optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hBYdWFwYCkwY"
   },
   "source": [
    "## Training and Testing\n",
    "\n",
    "Here we provide you with the functions to train and test. **Please do not modify this part for grading purposes.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "_tZMWRc8CmGg"
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "import networkx as nx\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.optim as optim\n",
    "from tqdm import trange\n",
    "import pandas as pd\n",
    "import copy\n",
    "\n",
    "from torch_geometric.datasets import TUDataset\n",
    "from torch_geometric.datasets import Planetoid\n",
    "from torch_geometric.data import DataLoader\n",
    "\n",
    "import torch_geometric.nn as pyg_nn\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def train(dataset, args):\n",
    "    \n",
    "    print(\"Node task. test set size:\", np.sum(dataset[0]['test_mask'].numpy()))\n",
    "    print()\n",
    "    test_loader = loader = DataLoader(dataset, batch_size=args.batch_size, shuffle=False)\n",
    "\n",
    "    # build model\n",
    "    model = GNNStack(dataset.num_node_features, args.hidden_dim, dataset.num_classes, \n",
    "                            args)\n",
    "    scheduler, opt = build_optimizer(args, model.parameters())\n",
    "\n",
    "    # train\n",
    "    losses = []\n",
    "    test_accs = []\n",
    "    best_acc = 0\n",
    "    best_model = None\n",
    "    for epoch in trange(args.epochs, desc=\"Training\", unit=\"Epochs\"):\n",
    "        total_loss = 0\n",
    "        model.train()\n",
    "        for batch in loader:\n",
    "            opt.zero_grad()\n",
    "            pred = model(batch)\n",
    "            label = batch.y\n",
    "            pred = pred[batch.train_mask]\n",
    "            label = label[batch.train_mask]\n",
    "            loss = model.loss(pred, label)\n",
    "            loss.backward()\n",
    "            opt.step()\n",
    "            total_loss += loss.item() * batch.num_graphs\n",
    "        total_loss /= len(loader.dataset)\n",
    "        losses.append(total_loss)\n",
    "\n",
    "        if epoch % 10 == 0:\n",
    "          test_acc = test(test_loader, model)\n",
    "          test_accs.append(test_acc)\n",
    "          if test_acc > best_acc:\n",
    "            best_acc = test_acc\n",
    "            best_model = copy.deepcopy(model)\n",
    "        else:\n",
    "          test_accs.append(test_accs[-1])\n",
    "    \n",
    "    return test_accs, losses, best_model, best_acc, test_loader\n",
    "\n",
    "def test(loader, test_model, is_validation=False, save_model_preds=False, model_type=None):\n",
    "    test_model.eval()\n",
    "\n",
    "    correct = 0\n",
    "    # Note that Cora is only one graph!\n",
    "    for data in loader:\n",
    "        with torch.no_grad():\n",
    "            # max(dim=1) returns values, indices tuple; only need indices\n",
    "            pred = test_model(data).max(dim=1)[1]\n",
    "            label = data.y\n",
    "\n",
    "        mask = data.val_mask if is_validation else data.test_mask\n",
    "        # node classification: only evaluate on nodes in test set\n",
    "        pred = pred[mask]\n",
    "        label = label[mask]\n",
    "\n",
    "        if save_model_preds:\n",
    "          print (\"Saving Model Predictions for Model Type\", model_type)\n",
    "\n",
    "          data = {}\n",
    "          data['pred'] = pred.view(-1).cpu().detach().numpy()\n",
    "          data['label'] = label.view(-1).cpu().detach().numpy()\n",
    "\n",
    "          df = pd.DataFrame(data=data)\n",
    "          # Save locally as csv\n",
    "          df.to_csv('CORA-Node-' + model_type + '.csv', sep=',', index=False)\n",
    "            \n",
    "        correct += pred.eq(label).sum().item()\n",
    "\n",
    "    total = 0\n",
    "    for data in loader.dataset:\n",
    "        total += torch.sum(data.val_mask if is_validation else data.test_mask).item()\n",
    "\n",
    "    return correct / total\n",
    "  \n",
    "class objectview(object):\n",
    "    def __init__(self, d):\n",
    "        self.__dict__ = d\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L7-h7jIsCns4"
   },
   "source": [
    "## Let's Start the Training!\n",
    "\n",
    "We will be working on the CORA dataset on node-level classification.\n",
    "\n",
    "This part is implemented for you. **For grading purposes, please do not modify the default parameters.** However, feel free to play with different configurations just for fun!\n",
    "\n",
    "**Submit your best accuracy and loss on Gradescope.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 723
    },
    "id": "qe9B45l9Cpz2",
    "outputId": "f2dfdf11-af74-41d1-9359-3cf9f7fbf554"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node task. test set size: 1000\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training: 100%|██████████| 500/500 [00:49<00:00, 10.11Epochs/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Maximum test set accuracy: 0.81\n",
      "Minimum loss: 0.09034553915262222\n",
      "Saving Model Predictions for Model Type GraphSage\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAsTAAALEwEAmpwYAABQX0lEQVR4nO2dd3hUVfrHPye991ADhN4CBAhNRJGOBdsqYu/+2HVddddd2FVUdl1ddbGuBRWsa68rKCiCgNKRXiT0hJJOej+/P+7MZWoyCQmBmffzPPNk5txz7z13cud73vue97xHaa0RBEEQvBe/lm6AIAiC0LyI0AuCIHg5IvSCIAhejgi9IAiClyNCLwiC4OWI0AuCIHg5IvSCIAhejgi94PMopa5VSq1XShUrpY4qpb5RSp3b0u0ShKZChF7waZRS9wPPAv8EWgMdgZeASxt4nIAmb5wgNBEi9ILPopSKBmYDv9Naf6a1LtFaV2mt/6e1fkApFayUelYpdcTyelYpFWzZd7RSKkMp9Rel1DFgvlIqVin1tVIqWymVb3mf1KIXKQiI0Au+zQggBPjczfa/AcOBVGAAMBR40GZ7GyAO6ATcifF7mm/53BEoA15shnYLQoNQkutG8FWUUtcB/9Zat3GzfS/we631QsvnicCrWutkpdRoYDEQpbUud7N/KrBUax3bDM0XBI8Rv6Lgy+QCCUqpAK11tYvt7YCDNp8PWsqsZNuKvFIqDHgGmARYxT1SKeWvta5p2qYLgueI60bwZVYBFcBlbrYfwXDDWOloKbPi+Dj8R6AnMExrHQWcZylXp9xSQTgFxKIXfBat9Qml1CzgP0qpagxXTBUwDrgAeB94UCm1DkPUZwHv1nHISAy/fIFSKg54uDnbLwieIha94NNorf8N3I8xyJoNHAbuBr4A/gGsB7YAW4GNljJ3PAuEAjnAauDbZmq2IDQIGYwVBEHwcsSiFwRB8HJE6AVBELwcEXpBEAQvR4ReEATByzkjwysTEhJ0cnJySzdDEAThrGHDhg05WutEV9vOSKFPTk5m/fr1Ld0MQRCEswal1EF32+p13SilOiilliqldiiltiul/uCijlJKPa+USldKbVFKDbLZdpNSao/ldVPjL0MQBEFoDJ5Y9NXAH7XWG5VSkcAGpdR3WusdNnUmA90tr2HAy8Awm9mBaRgzCzcopb7SWuc36VUIgiAIbqnXotdaH9Vab7S8LwJ2Au0dql0KvK0NVgMxSqm2wETgO611nkXcv8NI+CQIgiCcJhrko1dKJQMDgTUOm9pjTB23kmEpc1fu6th3YuT0pmPHjg1pliA0OVVVVWRkZFBe7jIDsSC0GCEhISQlJREYGOjxPh4LvVIqAvgUuFdrXdiI9tWJ1nouMBcgLS1N8jIILUpGRgaRkZEkJyejlCSfFM4MtNbk5uaSkZFB586dPd7Pozh6pVQghsi/p7X+zEWVTKCDzeckS5m7ckE4oykvLyc+Pl5EXjijUEoRHx/f4CdNT6JuFPAGsFNrPcdNta+AGy3RN8OBE1rro8AiYIJlLc1YYIKlTBDOeETkhTORxtyXnrhuRgI3AFuVUpssZX/FWIQBrfUrwELgQiAdKAVusWzLU0r9HVhn2W+21jqvwa30gPKqGt5ZdZC+7aI4p1tCc5xCEAThrMSTqJuVWmulte6vtU61vBZqrV+xiDyWaJvfaa27aq37aa3X2+w/T2vdzfKa31wXEuCnmLtiH/N/PtBcpxCE00ZBQQEvvfRSo/a98MILKSgoqLPOrFmz+P777xt1fEeSk5PJyclpkmM1hDlz5tCrVy/69evHgAEDuP/++6mqqmqSYz/yyCM8/fTTLrc99thj9O3bl/79+5OamsqaNY6xKWceZ+TM2MYQ4O/HpQPa8daqAxSUVhITFtTSTRKERmMV+t/+9rdO26qrqwkIcP/TXbhwYb3Hnz179im1r6V55ZVXWLx4MatXryYmJobKykrmzJlDWVmZUzRKTU0N/v7+TXLeVatW8fXXX7Nx40aCg4PJycmhsrKySY7dnHhVUrMxvVtRVaPZnHGipZsiCKfEjBkz2Lt3L6mpqTzwwAMsW7aMUaNGMWXKFPr06QPAZZddxuDBg+nbty9z584197Va2AcOHKB3797ccccd9O3blwkTJlBWVgbAzTffzCeffGLWf/jhhxk0aBD9+vVj165dAGRnZzN+/Hj69u3L7bffTqdOneq13OfMmUNKSgopKSk8++yzAJSUlHDRRRcxYMAAUlJS+PDDD81r7NOnD/379+dPf/pTg76fxx57jJdffpmYmBgAgoKCmDFjBlFRUQBERETwxz/+kQEDBrBq1Spmz57NkCFDSElJ4c4778S64NLo0aP5wx/+QGpqKikpKaxdu9Y8x44dOxg9ejRdunTh+eefB+Do0aMkJCQQHBwMQEJCAu3aGevFuzvHunXrTOv/gQceICUlBTA6oAceeIAhQ4bQv39/Xn311QZ9Bw3Bayx6gL5towHYfuQE5/dwmdtHEBrMo//bzo4jTRtR3KddFA9f0tft9ieeeIJt27axadMmAJYtW8bGjRvZtm2bGVY3b9484uLiKCsrY8iQIVx55ZXEx8fbHWfPnj28//77vPbaa1x99dV8+umnXH/99U7nS0hIYOPGjbz00ks8/fTTvP766zz66KOMGTOGmTNn8u233/LGG2/UeU0bNmxg/vz5rFmzBq01w4YN4/zzz2ffvn20a9eOBQsWAHDixAlyc3P5/PPP2bVrF0qpel1NthQWFlJcXFxneGFJSQnDhg3j3//+NwB9+vRh1qxZANxwww18/fXXXHLJJQCUlpayadMmli9fzq233sq2bdsA2LVrF0uXLqWoqIiePXsyffp0JkyYwOzZs+nRowfjxo1j6tSpnH/++QDcfffdLs9xyy238NprrzFixAhmzJhhtvGNN94gOjqadevWUVFRwciRI5kwYUKDwiY9xass+uiwQJJiQ9me2eRh/oLQ4gwdOtROBJ5//nkGDBjA8OHDOXz4MHv27HHap3PnzqSmpgIwePBgDhw44PLYV1xxhVOdlStXcs011wAwadIkYmNj62zfypUrufzyywkPDyciIoIrrriCFStW0K9fP7777jv+8pe/sGLFCqKjo4mOjiYkJITbbruNzz77jLCwsAZ+GydZtGgRqampJCcn8/PPPwPg7+/PlVdeadZZunQpw4YNo1+/fvzwww9s377d3DZt2jQAzjvvPAoLC81O56KLLiI4OJiEhARatWrF8ePHiYiIYMOGDcydO5fExESmTp3Km2++6fYcBQUFFBUVMWLECACuvfZa87yLFy/m7bffJjU1lWHDhpGbm+vyf9gUeJVFDzCqeyKfbDhMelYx3VpFtHRzBC+gLsv7dBIeHm6+X7ZsGd9//z2rVq0iLCyM0aNHu4yttroYwBA/q+vGXT1/f3+qq6ubtN09evRg48aNLFy4kAcffJCxY8cya9Ys1q5dy5IlS/jkk0948cUX+eGHH+z2mzhxIsePHyctLY3XX3/dLI+KiiIiIoL9+/fTuXNnJk6cyMSJE7n44otNf3lISIjply8vL+e3v/0t69evp0OHDjzyyCN235VjuKL1s+N3Z/1e/P39GT16NKNHj6Zfv3689dZbXHPNNXWewxVaa1544QUmTpzY0K+0wXiVRQ9w//geVNVovtl6tKWbIgiNJjIykqKiIrfbT5w4QWxsLGFhYezatYvVq1c3eRtGjhzJRx99BBjWZ35+3bkIR40axRdffEFpaSklJSV8/vnnjBo1iiNHjhAWFsb111/PAw88wMaNGykuLubEiRNceOGFPPPMM2zevNnpeIsWLWLTpk12Im9l5syZTJ8+3bS+tdZuhdVanpCQQHFxsTk2YcU6ZrBy5UrzacMdu3fvtrO6N23aRKdOndyeIyYmhsjISDMy54MPPjD3nThxIi+//LIZKfTrr79SUlLi9tyngtdZ9ImRwXRvFcHGQ5IgUzh7iY+PZ+TIkaSkpDB58mQuuugiu+2TJk3ilVdeoXfv3vTs2ZPhw4c3eRsefvhhpk2bxjvvvMOIESNo06YNkZGRbusPGjSIm2++maFDhwJw++23M3DgQBYtWsQDDzyAn58fgYGBvPzyyxQVFXHppZdSXl6O1po5c9zNxXTN9OnTTT98cHAwERERjBw5koEDBzrVjYmJ4Y477iAlJYU2bdowZMgQu+0hISEMHDiQqqoq5s2bV+d5i4uL+f3vf09BQQEBAQF069aNuXPn1nmON954gzvuuAM/Pz/OP/98syO5/fbbOXDgAIMGDUJrTWJiIl988UWDvgdPUdaR4TOJtLQ0fSoLj/zlky0s2nGMXx4aL7MbhUaxc+dOevfu3dLNaFEqKirw9/cnICCAVatWMX36dHNw2FsYPXo0Tz/9NGlpac12juLiYiIiDDfyE088wdGjR3nuuedO6Ziu7k+l1AattcsL8TqLHmBgxxg+XH+YfTkldE0UP70gNIZDhw5x9dVXU1tbS1BQEK+99lpLN+msZMGCBTz++ONUV1fTqVMnc/D2dOKVQj+okxEdsPFgvgi9IDSS7t2788svv7R0M5qVZcuWNfs5pk6dytSpU5v9PHXhdYOxAN0SI4gMCWDT4YKWboogCEKL45VC7+en6N0mil3H3EctCIIg+ApeKfQAPdtE8uuxIs7EwWZBEITTiVcLfVFFNZkFrieICIIg+ApeK/TWWbH7sptnAoIgNCenkqYY4Nlnn6W0tLQJW3R2UF1dzV//+le6d+9OamoqqampPPbYY012fNtkcLbU1tZyzz33kJKSQr9+/RgyZAj79+9vsvOeKl4r9J3ijdwZh/J872YXzn68QeibOpWCJzz44IMcOXKErVu3smnTJlasWOEyR73Wmtra2iY774cffsiRI0fYsmULW7du5fPPPzcza54JeK3Qt44MISjAT4ReOCtxTFMM8NRTT5kpbR9++GHAdQrg559/niNHjnDBBRdwwQUXOB3bXTrd9PR0xo0bx4ABAxg0aBB79+4F4F//+pe5uIc1++Lo0aOxTmrMyckhOTkZgDfffJMpU6YwZswYxo4dS3FxMWPHjjVTIH/55ZdmO95++2369+/PgAEDuOGGGygqKqJz586mMBcWFtp9ro/S0lJee+01XnjhBUJCQgAjlcQjjzwCwIEDB+jZsyc33ngjKSkpHD58mOnTp5OWlkbfvn3N7xSM1M1//vOf6devH0OHDiU9Pd3ctnz5cs455xy6dOliWvdHjx6lbdu2+PkZkpqUlGQmgXN3joULF9KrVy8GDx7MPffcw8UXX2z+T2+99VaGDh3KwIED7b6zxuKVcfRgRN50iA3lUK4IvXCKfDMDjm1t2mO26QeTn3C72TFN8eLFi9mzZw9r165Fa82UKVNYvnw52dnZTimAo6OjmTNnDkuXLiUhwXlZTXfpdK+77jpmzJjB5ZdfTnl5ObW1tXzzzTd8+eWXrFmzhrCwMPLy6l8JdOPGjWzZsoW4uDiqq6v5/PPPiYqKIicnh+HDhzNlyhR27NjBP/7xD37++WcSEhLIy8sjMjKS0aNHs2DBAi677DI++OADrrjiCqeFRNyRnp5Ox44d60zTsGfPHt566y0zZcRjjz1GXFwcNTU1jB07li1bttC/f38AoqOj2bp1K2+//Tb33nsvX3/9NWCI+sqVK9m1axdTpkzhN7/5DVdffTXnnnsuK1asYOzYsVx//fVmOgZX5+jRowd33XUXy5cvp3PnzmYGTWv9MWPGMG/ePAoKChg6dCjjxo2zS2rXUDxZHHyeUipLKbXNzfYHlFKbLK9tSqkapVScZdsBpdRWy7bG5zRoJJ3iwzmQKz564exn8eLFLF68mIEDBzJo0CB27drFnj17XKYArg9X6XSLiorIzMzk8ssvB4z8L2FhYXz//ffccsstZhrhuLi4eo8/fvx4s57Wmr/+9a/079+fcePGkZmZyfHjx/nhhx+46qqrzI7IWv/2229n/nxjxdH58+dzyy23NPzLsjB//nxSU1Pp0KEDhw8fBqBTp052eYE++ugjBg0axMCBA9m+fTs7duwwt1nFd9q0aaxatcosv+yyy/Dz86NPnz4cP34cMCz43bt38/jjj+Pn58fYsWNZsmSJ23Ps2rWLLl26mGmnbYV+8eLFPPHEE6SmpppZSQ8dOtTo7wE8s+jfBF4E3na1UWv9FPAUgFLqEuA+hwXAL9Ban/4FJYHurSJYuSeHqppaAv291kslNDd1WN6nC601M2fO5K677nLa5ioFsDvqS9nrKQEBAaaP23F/W8vzvffeIzs7mw0bNhAYGEhycnKd5xs5ciQHDhxg2bJl1NTUmKsxWampqWHw4MEATJkyxW5JxG7dunHo0CGKioqIjIzklltu4ZZbbiElJYWamhqntu3fv5+nn36adevWERsby8033+w2fbHte9v0xbbh28HBwUyePJnJkyfTunVrvvjiC7p06VLnOVyhtebTTz+lZ8+eddZrCJ4sDr4cqP95zWAa8P4ptagJ6dU2ksqaWg7kiFUvnF04pimeOHEi8+bNo7i4GIDMzEyysrJcpgB2tb8Vd+l0IyMjSUpKMrMnVlRUUFpayvjx45k/f745sGt13SQnJ7NhwwYAl1EoVk6cOEGrVq0IDAxk6dKlHDx4EIAxY8bw8ccfk5uba3dcgBtvvJFrr73WpTXv7+/Ppk2b2LRpk9O6t2FhYdx2223cfffd5nXW1NS4XdO1sLCQ8PBwoqOjOX78ON98843ddmv64g8//NBcOMQdGzdu5MiRI4ARgbNlyxY6derk9hw9e/Zk37595iIv1nOB8b9+4YUXzE6kKdJQNJmPXikVBkwC7rYp1sBipZQGXtVaz3W5s7H/ncCdAB07dmySNvVsbawfuetYEd1bu/fbCcKZhmOa4qeeeoqdO3eaghMREcG7775Lenq6UwpggDvvvJNJkybRrl07li5dah63rnS677zzDnfddRezZs0iMDCQjz/+mEmTJrFp0ybS0tIICgriwgsv5J///Cd/+tOfuPrqq5k7d65TCmVbrrvuOi655BL69etHWloavXr1AqBv37787W9/4/zzz8ff35+BAweayb6uu+46HnzwQTt3hqc89thjPPTQQ6SkpBAZGUloaCg33XQT7dq1M4XYyoABAxg4cCC9evWiQ4cOjBw50m57fn4+/fv3Jzg4mPffr9t+zcrK4o477qCiogIwVgO7++67zRTIjucIDQ3lpZdeYtKkSYSHh9v9Hx566CHuvfde+vfvT21tLZ07dzbHBxqN1rreF5AMbKunzlTgfw5l7S1/WwGbgfM8Od/gwYN1U1BSUaU7/eVr/dLS9CY5nuA77Nixo6Wb4LN8/PHH+vrrr2/RNnTq1ElnZ2c36zmKioq01lrX1tbq6dOn6zlz5ni8r6v7E1iv3WhqU0bdXIOD20ZrnWn5m6WU+hwYCixvwnPWSVhQACGBfuSVVJyuUwqCcAr8/ve/55tvvmHhwoUt3ZRm57XXXuOtt96isrKSgQMHuhx/aSqaROiVUtHA+cD1NmXhgJ/WusjyfgIw280hmo348GByS1z76ARBOLN44YUXWroJAG4XUW9K7rvvPu67775mPw94IPRKqfeB0UCCUioDeBgIBNBav2KpdjmwWGttO+rZGvjcMlodAPxXa/1t0zXdM+LCg8gToRcagdZaVigTzjh0IxI11iv0Wut6R0S01m9ihGHalu0DBjS4RU2MCL3QGEJCQsjNzSU+Pl7EXjhj0FqTm5trzvz1FK+dGWslPiKI9Kzilm6GcJaRlJRERkYG2dnZLd0UQbAjJCSEpKSkBu3j/UIvFr3QCAIDA81Zi4JwtuP100UTIoIpq6qhsNyzxEiCIAjehtcLfecEY8rzfslLLwiCj+L1Qt/VsgCJ+OkFQfBVvF7oO8aFEeCn2JstQi8Igm/i9UIf6O9H+9hQDufL2rGCIPgmXi/0AJEhAZRUnP5lzQRBEM4EfELow4MCKC4XoRcEwTfxCaGPDAmgWCx6QRB8FJ8Q+vDgAEoqRegFQfBNfEboxXUjCIKv4hNCHxksrhtBEHwXnxD68OAAKqprqaqpbemmCIIgnHZ8Qugjgo3cbRJiKQiCL+JTQi/uG0EQfBGfEPpw06KvaeGWCIIgnH7qFXql1DylVJZSapub7aOVUieUUpssr1k22yYppXYrpdKVUjOasuENISLEatFLqmJBEHwPTyz6N4FJ9dRZobVOtbxmAyil/IH/AJOBPsA0pVSfU2lsY4kNCwQgr0SEXhAE36NeoddaLwfyGnHsoUC61nqf1roS+AC4tBHHOWVaRRrrK2YVlbfE6QVBEFqUpvLRj1BKbVZKfaOU6mspaw8ctqmTYSlziVLqTqXUeqXU+qZepzMhIgil4HhhRZMeVxAE4WygKYR+I9BJaz0AeAH4ojEH0VrP1Vqnaa3TEhMTm6BZJwnw9yMhIpisQrHoBUHwPU5Z6LXWhVrrYsv7hUCgUioByAQ62FRNspS1CK2jgskqEoteEATf45SFXinVRimlLO+HWo6ZC6wDuiulOiulgoBrgK9O9XyNpVVkCMdOiEUvCILvEVBfBaXU+8BoIEEplQE8DAQCaK1fAX4DTFdKVQNlwDVaaw1UK6XuBhYB/sA8rfX2ZrkKD+jVJpIff80mq6jcHJwVBEHwBeoVeq31tHq2vwi86GbbQmBh45rWtFw5OImXlu1l4Zaj3Dyyc0s3RxAE4bThEzNjAbomRhAVEsDe7JKWboogCMJpxWeEHiApNozMAlkkXBAE38KnhL59bCgZ+aUt3QxBEITTik8JfVJsKJn5ZRhjxYIgCL6BTwl9+5hQSiprOFEmOW8EQfAdfEroo0KN5GZFsn6sIAg+hE8JvXUBktJKyUsvCILv4FNCHxbkD8hKU4Ig+BY+JfThpkUvQi8Igu/gW0IfJEsKCoLge/iW0AcbrpsScd0IguBD+JTQhwWJ60YQBN/Dp4TetOgl6kYQBB/Cp4Q+NNAfpaBUXDeCIPgQPiX0SinCgwIolsFYQRB8CJ8SejBi6cVHLwiCL+FzQh8RHCA+ekEQfArfE/qQAAolqZkgCD5EvUKvlJqnlMpSSm1zs/06pdQWpdRWpdTPSqkBNtsOWMo3KaXWN2XDG0tCRDA5xRUt3QxBEITThicW/ZvApDq27wfO11r3A/4OzHXYfoHWOlVrnda4JjYtiRHBZBeJ0AuC4DvUK/Ra6+VAXh3bf9Za51s+rgaSmqhtzUKrqGBySyqpqZXFRwRB8A2a2kd/G/CNzWcNLFZKbVBK3VnXjkqpO5VS65VS67Ozs5u4WSdJjAymplaTX1rZbOcQBEE4k2gyoVdKXYAh9H+xKT5Xaz0ImAz8Til1nrv9tdZztdZpWuu0xMTEpmqWE4kRwQDivhEEwWdoEqFXSvUHXgcu1VrnWsu11pmWv1nA58DQpjjfqZAYaQh9lgi9IAg+wikLvVKqI/AZcIPW+leb8nClVKT1PTABcBm5czppGxMKwNGCshZuiSAIwukhoL4KSqn3gdFAglIqA3gYCATQWr8CzALigZeUUgDVlgib1sDnlrIA4L9a62+b4RoaROvIYPz9FBn5IvSCIPgG9Qq91npaPdtvB253Ub4PGOC8R8sS4O9H2+gQMvJLW7opgiAIpwWfmxkLkBQbymGx6AVB8BF8Uujbx4SRKUIvCIKP4JNCnxARRJ7E0QuC4CP4pNBHhQZSWV1LeZVksRQEwfvxSaGPCQsEoKBUslgKguD9+KTQR4caQn9C0hULguAD+KTQx4QGASL0giD4Bj4p9GLRC4LgS/i00BdI5I0gCD6Abwp9mFj0giD4Dj4p9JHBAfgpZO1YQRB8Ap8Uej8/RXhQAMUVEkcvCIL345NCDxAa5E9pZXVLN0MQBKHZ8VmhDw8OoKRSLHpBELwfnxX6sCB/SivEohcEwfvxWaEPDwqgRFw3giD4AD4r9GHB/pSK60YQBB/AI6FXSs1TSmUppVyu+aoMnldKpSultiilBtlsu0kptcfyuqmpGn6qhAcFiNALguATeGrRvwlMqmP7ZKC75XUn8DKAUioOY43ZYcBQ4GGlVGxjG9uUiI9eEARfwSOh11ovB/LqqHIp8LY2WA3EKKXaAhOB77TWeVrrfOA76u4wThsSdSMIgq/QVD769sBhm88ZljJ35U4ope5USq1XSq3Pzs5uoma5J0zi6AVB8BHOmMFYrfVcrXWa1jotMTGx2c8XHhxAVY2msrq22c8lCILQkjSV0GcCHWw+J1nK3JW3OGFB/gBi1QuC4PU0ldB/Bdxoib4ZDpzQWh8FFgETlFKxlkHYCZayFic8KACAonIRekEQvJsATyoppd4HRgMJSqkMjEiaQACt9SvAQuBCIB0oBW6xbMtTSv0dWGc51GytdV2DuqeNrq3CAdh+pJAOcWEt3BpBEITmwyOh11pPq2e7Bn7nZts8YF7Dm9a8pLSPJjjAj3UH8piU0qalmyMIgtBsnDGDsaeb4AB/+raLYlvmiZZuiiAIQrPis0IP0CY6hNwSWU5QEATvxqeFPj48mNziipZuhiAIQrPi00IfFx5EfmkV1TUSSy8Igvfi00KfEBEEQF6puG8EQfBefFro4yOCAcgTP70gCF6Mbwt9uGHR5xaL0AuC4L34ttBbXDc5MiArCIIX49NCHxUaCEChpEEQBMGL8W2hD7EIfVlVC7dEEASh+fBpoQ8J9Cc4wE+EXhAEr8anhR4M901huQi9IAjei88LfXRoIIVl4qMXBMF78Sh7pTcTFRLACXHdCJ5QWwv5+0E7zKRWfhDbGfx83m4SzlBE6EMDZcJUY8j+FX5+zhA/W/z8YMgd0C61RZrVrKydC9/+xfW2MQ/CeQ+c3vYIgoeI0IcEciCnpKWbcfbxy9uw6b8QlWRfXpYHO/8HCT2c9+k0ElqnOJe3GwgJ3Tw/d1k+lLpYvyYsDkJjPT/OL+/BhvnO5f7BMOV5iO9qX37wJ4hqD+Nn25cv/SccWHl2CH1NFRQcci4PDIOotqe/PXVRW2O01xH/QPDzP/Xj11TDiUOgtfPxozuAUqd2/KJj8M7lUF7ovC2qnevvOyQaprxwaud1gc8LfXRoYPPG0ZcXur5Zw+JO/UYCQ/AKXSzDGxZv3EzNxfEdhmj/3wr78qydsOTvUF1mX15+An561v3xQmKcv4/O58FVb9mXVxTB84OMDsWR4Gh4YA8EBHt2DetegxMZ0KafffmBlbD8aRjxW/vyzA3QcTj0+41z/W2fwq+L7cuVgo4jIDjCs/acDv53L2x61/W20TOh3SDPjqMUJKV53rFWlhqdarmL9R86DINuY+3LaqrghcFQcNC5fnQHuGcT+J+ifH19L/zyjuttl70CqQ7rLR3ZBP+7x+ggHOl9MbRPsy/bvRCydsCAaaBsOyYN2buNlyOhcQ24AM/xeaFvFRlMXkklheVVZly9W/IPwq4Fxk2u/C1//QzrQvlZXjbvs7bDymcB7XysEXfDxMc8b+jBVVB01L6sugK++TNUuLAYAkLhvm0QnuD5OVyhtSGGjn7p49uh6wXO9Vv1hmn/dX2sExlQVW5fVlUKO750voYTGUb56+MMC8tKWYEh8hP/CWE213ZkI6x5BfL2GW2oj+oKOLYNRvwOxj9qv+3T22Hzf42XI0lDnMs6nWOI2H+vct7WfrDxv7ZF10L2Lqgqc1F/EPSe4lzuF+DaMCjJgdx05/LQWEjs6VyeucFo09C77Ms3vQfLHneuXx/+QZ7Vq60BXeN6W3CUca8G2izpefAnQ+QH3QSxySfLs3bA1o+NsZKE7p6de8ls+NmFlVxTCSlXQveJ9uWL/wZ7lzgL/eb3IWsXdB9vX16aBz/+y/W5u4yGy1/xrJ3NiKdrxk4CngP8gde11k84bH8GsP7qw4BWWusYy7YaYKtl2yGttYu7uOXo3yEGgG0ZJzinm0U4qisMwXBk4QNwYIVzeV10Gw/dJ9iXbXwb9i3z/BilefDmhc5iC4bl/pt59j+40jzD8vj0dvsfibX+6JnO1lDefsMydWTXAvj1G9ftcuWGqYvoJNflbfs7l1VXwJd3Q/Ex+/KIRMP6G+GwcmViT0Poc/Y4C/2PTxruJFtqKqG2ynAbOTL5SehzGU4dtF+A8cN1JOU3kNjL+cktYx18OwM+ucV5H+VndMa26BpYVe5cFyC+O5zze2M/2/rfP+r66Ub5wd3r7d1PNdVGpzDidzBgqsM1XAnHtji7MdxRUWiIcW0DnoY7n+9sHBxcBfMnwRMdnev7Bxsduu0T0ZFfDKHP2um50O/+1vgd9LrI+fjDp0NojEP9hZD+vfH041jeeRRc8559udZGe1x13J62sZmpV+iVUv7Af4DxQAawTin1ldZ6h7WO1vo+m/q/B2x/PWVa69Qma3ETMyApGoBfDhecFPqv73f/eDt2Fgy+xfjn6lrjx6ZrjVetzXurKMd3c7bESrJgxRyjM/F3cDNEtLK3YMGop2vhojmGn9uWqLaGX8+RfUvhwE+G5W1F10BprjFQ2utim3JtiNGRX1xf84i7oVUf+zL/QOh5oev6TUFAMFz5muf14y0+/v3L7V1W1eWGGyausxEZY0ubfq6fSsLijEdxT/Hzc91ZJQ02jlNR7Lwtqh2ERNmX1dYYlnXxcefy1S8bnbcjITFw9Tv2YlhRBB/daLjKutlYnyXZRufmavzEP8B4mmgIrr67htJxOEx50blDB8OQcHR7JfQEFGz+wPkJN6qdcV/b/t5qqiDnVzjnbhj3iGdt6ncVHF5jGDm2KD9Ivc65vlLQuo9z+RmEJxb9UCBda70PQCn1AXApsMNN/WnAw03TvOYnJiyI7q0iWL0vl99dYBGLzPXGI7qj1egfZFjnjkLcUNoPNkT3eRfWZO9LYKpDJ5O33/jbaSS06uXZOa5607msphqe6QMfXu96n/F/h76X25cFhp66++d0EBxhCPm614yXI1e85lqMmxt3TzGu8POHQTe63nbO710PQIfGuh4D6DDceHLc+LbztjMpIkopGHSD5/WDwoynsN0LjJcjabdBdPuTn8vyjc7N0VCpi94XN6yjPwvwROjbA4dtPmcAw1xVVEp1AjoDP9gUhyil1gPVwBNa6y/c7HsncCdAx44uHuOakVHdE3lvzUHKq2oIUdXG4/+59zqLXlPRbTxc8brhn7Zl5/8gfQlUV0KAjSsm/4DxN7bTqZ3XPwB+M9+1+ykwDIbd5flA5pnIDZ+78VfHtYzINyVB4cbLU6772HV0TXCEszvvbOO2xcZTiy1aw+d3wfo3nOsHhLoeW/Ehmnow9hrgE63tRl06aa0zlVJdgB+UUlu11nsdd9RazwXmAqSlpXnoKDxFNn8Iu77mrsIK0lQ+Ze++RUhglWFtN8QCaCj+AdDfxcBdWBykfwevj4GAkJPl+Qchsq1hXZ8qySONlzcS19l4CYZbqE0Dx1DOFvwDjd+KI9d9bIy9OKL8Tz1C5yzHk6vPBDrYfE6ylLniGsDO36G1zrT83aeUWobhv3cS+hZh+VNQkkVsWBu6qhL88vIgJNAIk0oedfrb03UM9L0Cygvsy9ukQNexLncRBMGCUmf3E2kz4onQrwO6K6U6Ywj8NcC1jpWUUr2AWGCVTVksUKq1rlBKJQAjgSebouGnjDWyZtT9cN5fmfzQN9zdrxv3T3ARkna6CAqHq1xM4BEEQTgF6hV6rXW1UupuYBFGeOU8rfV2pdRsYL3W+itL1WuAD7S2i8/qDbyqlKrFSKD2hG20zmmjtsbZp5e923DRJPYiKMCPpNgw9ueWut5fEAThLMYjx5XWeiGw0KFslsPnR1zs9zPQz7H8tPPOZUbYnSta9wWgR+sIVu7J5khBGe1imsAXLgiCcIbgGyMU+QehbSr0d5gkEp5gTHQB7h3Xg4tfWMmSXVncMPwUo1sEQRDOIHxD6HWtEUXjmLvEho7xxvTr8ko307QFQRDOUnwjgbaurTdXeEiAkXSovEqEXhAE78I3hL62xj5HiAsC/RV+CiqqXeSTEQRBOIvxDaHXNQ5pQp1RShES6C8WvSAIXodvCH1tjUcLFQQH+FFeLUIvCIJ34RtCr2vrtegBi0UvrhtBELwLHxL6+i81JNBffPSCIHgdviH0DXHdiI9eEAQvwzeEvgEW/d7sYk6UuljjVRAE4SzFR4S+/vBKMCz6fdklXPh8A5cLFARBOIPxDaH30HUTEmjUySxwsfajIAjCWYpvCL2HUTe1ni6MLAiCcBbh/UKvNaA9ct2UVDRgRXtBEISzBO8X+lpLFI0HrpuSCom4EQTB+/B+obcuX+uJRV8pFr0gCN6HDwi9ZQKUBxZ9maQoFgTBC/FI6JVSk5RSu5VS6UqpGS6236yUylZKbbK8brfZdpNSao/ldVNTNt4jaj236JVSzdwYQRCE00+96qeU8gf+A0wG+gDTlFJ9XFT9UGudanm9btk3DngYGAYMBR62LBh++jBdN/Vb9O/ePpSgAOMrqZDkZoIgeAmeWPRDgXSt9T6tdSXwAXCph8efCHyntc7TWucD3wGTGtfURtIA102vNlHMnGwsLSgDs4IgeAueCH174LDN5wxLmSNXKqW2KKU+UUp1aOC+KKXuVEqtV0qtz87O9qBZHlJrEXoPXDcA4cHG6ooSaikIgrfQVIOx/wOStdb9Maz2txp6AK31XK11mtY6LTExsYmaRYOibgAiLEJfLEIvCIKX4In6ZQIdbD4nWcpMtNa5WusKy8fXgcGe7tvsNCCOHkToBUHwPjwR+nVAd6VUZ6VUEHAN8JVtBaVUW5uPU4CdlveLgAlKqVjLIOwES9npQzfMdZMYGQxAdlFFPTUFQRDODgLqq6C1rlZK3Y0h0P7APK31dqXUbGC91vor4B6l1BSgGsgDbrbsm6eU+jtGZwEwW2ud1wzXUccFeB51A9A6KgSAYyfKm6tFgiAIp5V6hR5Aa70QWOhQNsvm/Uxgppt95wHzTqGNp0YDXTexYYEE+ftxvEiEXhAE78B3ZsZ6aNErpWgVFcxxsegFQfASfEjoPb/U1lEh7D5ezLfbjjZTowRBEE4f3i/0puvG80ttEx3CzqOF/N+7G8nIL22mhgmCIJwevF/oG+i6AejdJtJ8/+vxogadbuWeHM791w+yyLggCGcMPiD0DZswBZDSPtp8v/Now4T+0f9tJyO/jIO58iQgCMKZgfcLfQOjbsBe6H85lN+g09VYliP09/5vVhCEswTvl6MGxtEDJEQEs+Cec7n5nGSW7s7m6AnPFwuXZWcFQTjT8AGhtyhvA1w3AH3bRXPdsI7U1GpW/Jrj8X41tcb5qmrqV/zyqhq2HznRoHY1B+VVNby8bC9VNbUt3RRBEJoB7xf6RkTdWOmaGEFkSACbMwo83scq9JXV9YvmzM+2ctHzK8ktbtl0Cy8v28u/vt3Fh+sO119ZEISzDu8X+ka4bqz4+Sn6J0Wz7kAeVTW1vLBkT705cLS2WvT1C/3KdONJoayFI3SKyo0EbhIpJAjeiQ8IvecLj7jikv7t+PV4Mf/4egf//u5XLnlhZZ31rYOxnlj0VmFt6bVqNTKwIAjejPcLfQPWjHXF1CEdiAwOYMUew/o+VljOwdwS96ezaGalBxZ9haUzKG1pobe02U/WzBUEr8T7hf4UXDdg5L5JjAxmX85JcT//qWVsOlzg+nTa88HYyjNG6I22toTO7zpWSHpWw+Yq+BobDuZRWytPXULj8QGhPzXXDUCCJUe9LZ9vzACcffGmRe+B68bK6n25pti2BNYz2+r8zqOFp2XxlUnPrmDcnOXNdvyK6hrWHTi9mbGbkhV7srny5VXM+2l/SzdFOIvxfqFv4JqxrkiMcBb6TYcLWLori+5/+4YdRwrN8pPhlZ4L/XNL9vBBC0S8fLkpkw/WHjJdN7ad1OTnVnD3fzee9jY1NY8v3MVVr6xi97Gz46lBa21nvVtnWO/NLm6pJglegPcLfSNSIDhiXXUqNiyQt28dyqCOMRSVV/O/LUcA2JZ5Mha+1sPwSkcLfn+Oe7+/lbLKGiqqm87N84cPNjHjs63UWtpiHTPItoR7bjjYsFnBZwoV1TXsOmZ0vnssbqE1+xv21LRyT06LLD7z3JI9dPnrQnOg3vq/kfET4VTwfqFvRAoER6xCHxYUwHk9EunZJpLC8mpKK4xjhwadPLb1h1lZU8vs/+3g572uJ1sVObhFggPq/1f0nvUtl//n50ZdQ11YDUhrJ5JVaAicdf3c5qI53FVzl++l54PfMunZFWQXVRBkyUUx68vtvLHScH9UVtfWee7aWs31b6zhqlcb/l2v2ZdL8owFZBZ4Ppvalrd+PgBAieX+sD4h+vuJ0AuNx/uFvhH56B1Jjg8HMH+8kSGB5BRX8O32Y4C9m8YaXlleVcO8n/Zz7WtrXB7ziIMQeCL0ADuOFtZfyQNshc769GG16LMscwXCm1noS2wGoT11dZ0orWJfHW6Mfy7cZb4vLK8i0Cbp0PoD+WQVltPjwW/479pD7s9RVgXA4byGi7X1uGv25TZ4XzhpuVvnVliF3pct+uozfMb2D7uOs6eBWW5PNx6pi1JqklJqt1IqXSk1w8X2+5VSO5RSW5RSS5RSnWy21SilNlleXznu2+ycYtQNwOSUNlw7rCMzJ/cCINJBAPNKKk3Xi9U6zi2prPOYGQ4iour5IZdWnnwCaApLuLD85PGsboKKqtMr9FZBdXxfF5f+ZyVj/v2jR3Urq2sJDjz5f/fzg3RLJ/HlpiNu98uxuK4C/Y3/yf6cEi5+YQV59fxP4eSAdmP/RdbbwBqJVX2KFv13O46TPGMBh/POzmyqh/NK6fa3b/jMEvxwJnLrm+sZ/0zzBRQ0BfUKvVLKH/gPMBnoA0xTSvVxqPYLkKa17g98Ajxps61Ma51qeU1ponZ7ThO4bvz8FP+8vB93nd8VgMgQewH8x4KdXPD0Mt5edcC0jvOKnUWhqqbW3O74aG8r5K7IKjw5I/fIiXI2Hy7gcF4pK/fkcKKsivFzfmSLQ6qGiuoaispdC2i2zZq4uSXGscstrptsi+smzEYkSyurSc8q4t3VB+tspycczislu6iCE6Un21ZQ6pnQH2hA+ueSimrTdQNGZ2rtzOp6grKOUVg7ujX7ctmWWcj+nPoHRK0ddmO7Yuv+VqEvtbhwGmvPf7TeGOTfmll3TiWtNckzFjBn8e5GnsnoIFfsyW70/q6wrgdRV8fsjo2H8hsU/XbxCyt4aVl6g87RktFyDcETi34okK613qe1rgQ+AC61raC1Xqq1tv4CVwNJTdvMU8BMatZ4oXckMiTQZfmsL7eb73Nc5K+Z9OxyBjy6GICM/FI7sSmpqGHp7ixW7rH36X+34zi7jhWaVjbADa+v4dL//MSoJ5dy/RtruPC5FezJKuaZ73612/fWN9fR75HFLttq23FYj11RZXREyy1tsPrsP92QQZ9Zixg3ZzkPfrHN9B83llFPLmXE40soLLe16Ou3lm1x9QN2bFdRRTVBASclUnFybCQk0P39kGvppMODDKHPyDc6ZU/mO5y06D0XgDd/2s+gv3+H1hqr4W4V+GLLOFB5IwfhzTkS9dQrLDPO9/wPDRM6W254Yy03vLG2SZPj+Vm+kNoGCuqvx4u44qWfefLbk668uv4nWmu2ZRby5LcN6+hs3Y8XPrei3vrHTpSf8u+nMXgi9O0B29i/DEuZO24DvrH5HKKUWq+UWq2UuszdTkqpOy311mdnN6FVYLpums7H6WjRuyLH5jHfOiFob3aJ6Xvdl11Ch7gws86hvFJumb+O699Yw77sYvNmuOPt9Ux6dgXHC09a4PscInSsTwcRDh3QT+mGn9hxJq/WmueW7DE/Z1tEv6K6hrX788zJYFZh+2qzvTV1qAncANW12s5dk1/imUVvxTFtxIfrDtH34UV2ZcZ3ePL/rjUUlBr/l71ZxW6janJMi97oDKzLSXqSqsJqkTdEmB753w7ySirZmnkCZWnvta+v4ef0HPNJr6yyceJpdSXW5xo7YknFbXVXNYadlvEjT5/OPMG/Ed8nYP5etlieZL7ZepTOMxdyxUs/8fqKfU7168s39c7qg9z1znqncuv9BJ6Nnw1/fAlXv7qq3npNTZMOxiqlrgfSgKdsijtprdOAa4FnlVJdXe2rtZ6rtU7TWqclJiY2XaOawHXjiNWi75oYbpa1jwkFYGpaByKCA8ixscDHzVnOOzYuj5mfbWHJriwGd4wlPjwIgB92ZZnbx/z7R/o+vMjuMdoq1gM6xLhtV5DDaicBFmvIKvgAz3z3Kze8sZY1+/MY17s1cNLKXbs/j+vfMAaPz+kaT0llNTnFFU6Lr0x+bgX3vP+LXVlWUTn5HviwbV1UdkJv84PZeCif5BkLnGbMXv3KyR9IaZW9VbR2v3MoaHF5tZ3lX1heZVrr+3JKOPdfP5jbnv3+V657fTVwUuiDLE9cVovek+RzVnvCk05h8fZjXP7ST+bnKS/+xDGbDv3a19eYk9asFr35uaqGq19ZxbLdWdTWah78YitbM5zdM9bB3LxS9/+bgtJK00USGRJIeVWNnWHhCS8v22u+b+jTWV1Y5b2mnpnB2zJPcPP8teb9ZR1P8VPw0brD3POBcb9uPFTAPxbspLZW2x2zvs7poS+2sWj7cb7bcdwu+Z+nY0twMvR6+xHnDuFIQRnJMxbwU7rnKdEbgidCnwl0sPmcZCmzQyk1DvgbMEVrbaqc1jrT8ncfsAwYeArtbTiNWDO2PqzhlLaP/6kdYwBoGxNCoL8y/d5WHvpim/n+/bXGA9KgTjFseGg8CRFBLs9j+xj9zuqDtI4KZmpaB5d1AT7dmMHMz7aaN6LVx2w72ea5JXtYmZ5DgJ/i1pHJdvvbuoc6xYdzOK+MtH98bzdwa+WrzUfsbvihjy1h4N+/47sdx92270BOid32P3+yxXxvO3i9YMtRAJbsPNn5Aay1meFaUmEvpK7SKBRXVNvNOygsr7brUKptfujPfr+Hn9JzySosZ/NhQzC3ZRYy6dnlJ4W+Aa6bUjedQn5JJe+sOkBWUTl3vrOBXw4V1Hk8a8dUXlnD/pwSUh5exOsr9pGRX8baA3ncPH8dWUUVvLv6EJe86Jxwzyp8dXXCqbO/4w8fbAKMkNrfvbeRYf9c0iD3079sXCT5TWjRWzvq2joeaLZlnuDiF1aybHc2S3cZ3gBrR+WnFH/+dItTSpLLX/qJy/5zspO1FfobLMbO4bxSp87zjrfX0+uhb5ny4kreWXXASeirampZsvM4ry13fmoodDNeBifF/+9f73B/oaeAJ0K/DuiulOqslAoCrgHsomeUUgOBVzFEPsumPFYpFWx5nwCMBJrnStzRBBOmHAm3CH0/myUHkywWPRiWYHlV3Y/aU9M6MCmlLeDamuho49YBOF5YwaCOsYzv07pOL9T7aw/R/9HFDHnse/MmtLoebMcNbhjRiVZRzjN+rUQE198x9nroW7snFTB+CBn5pS5THo9+epkpKCfPE0BIoB8Hc0t4aVk6BaWVhFo60CU7s9ymYbjznfX88aPNgOGKSs8q5rLUdsSFn+w0/7FgpymUAEVlVU4T0xwtxaH/XGKmjwbYdayI45aBa0989NbD2XYKS3YeNx/xH/hkCw99ud1se32ssoRp7j5exAsWd9s/Fuy0S5e9x6aT6//IIrtZwDmW689z4xpz7LxCAv1YYnm6LKmsYeOhfB76Ypsp+p9tzKizM4fGuW52HSt0uZKbtaOuqaPTmWrjCvnJMm/l2Anj+3H3FLY54wRbM0+YYc62LhhrAsNRTy7lkhdX8uE651DcLRkneOjL7azeax9Gm19ayW1vreexhTud9rHtAB07UWuHvOtYUb1PL42hXvXTWlcDdwOLgJ3AR1rr7Uqp2UopaxTNU0AE8LFDGGVvYL1SajOwFHhCa316hb4ZXDfdW0fy9q1DeWRKX7NsZLcEwHCt2MZuu7LWlYJ//aY/0aGGC6jaxT922tCOTmWDO8WSGBnMgt+PcprMFGUzblBZXWsnBBn5Zby7+iDnPG64Kv44vgd/u7A3UaGuB5V7to40B8EcufmcZP44vof5+clvd5E8Y4FdnXP/tZReD33L/72zoV6rUCmIDw/m/bWHefLb3dz65jrziWntgTwzLbRjLPW+7BI+tYTcZRdVUFJZw6BOsfz7qgF29VbZxLPvyykxf8RWDueV1ttG62Zb0UjPKuL+DzeZs4dX78tl3YE808Vi7RS2ZBRw21vrmf31DrTWZkTPij05nNM1vs7z2pKRX8Znv5x8kLYdd9liY3UWllebuX1yiitMIcsrqeCPH202o3CsrNlvL1S2T0oFpZVc8dLPvLP6oPlUd/9Hm7nj7fVmxNT+nBK+3mI/hlNQh5vIHZOeXcGIx3+wK3vg4838YHmqq0v8bDuBDQfyqaqpNaN/6ls/Yt2BPA7llnLt6/bzXR78Yqv5/i+fbnXczWS9w+zxpTYuWMdBV9unyc4zF5I8YwGV1bVUVNeYQRgvXjuw0RFWdeFRoLTWeiGw0KFsls37cW72+xnodyoNPGWawXUDcF4PYxzhwn5t2HW0iPN6JLLmr2NpHRViPuoDzJzcmz9+bG+9OcbhWxnXuxVhQQF0jAuje6sIp+2XDTTGwPu0i6JdTAi/Hj/pkrkqrYM589OWoAA/DuWV8sx3v1JZU0uftlHcem5nAvz9SAh3tujX/m0sYUEB/PUz1zf3hf3a2vmRi1y4dax8u/0Y248UMvXVVcyZmuqyTlVNLQkRQeaA8sHcUipsBHV/TgmZBWWEuAmHrKqp5YtNhgC2jQ61m6VsZWjnOIYmx/HiUueIkge/2Ma947o7lbeNDuGow2CtrfX72cZMPvslk5ySSt6+dSjXzDX8+2N7tQJOCr1VWI+dKOfu939hs03W06lDOrBqX269MfdJsaFk5Jfx8CV9iAwJ5E8fb2aXjdVuTcHx84wxjP33j+ZTywtL9qC1sf/6A/kUVVTz6cYMokICmNi3DRXVtfzp4y125yq0cUUcsgllLSyrwrbv/3lvDpP7teXNn/bztsNTXVMMxuYWV/DxhpOx846Dsd9uO8qbPx/g8oHtSYoNIz2rmCHJsWw4mM9Xm46wJ8v4bdj+Fl1xvLCcV5fvdSp/d7X7CXW2/Oxg0duGgR4rLKdr4snfsasO8HhhOa/8uNe81gtT2ro1sk6F5p0RcybQBNkr6+Kl6wab71tHhdhte3RKXzN9QmRIAG/eMoQrX17lNjzz9ZuGmO9tE6W9eO1AarWxaLmVlPbR/Hq8mNZRwbxx0xB6t43i/vE9zMiTBfecyys/7qNjXCj/WWrcyC9MG8glA9qZx7C9oS7u35YBSTG0ijSuIcxGMP97+zBySiopLKtiaOc4Vllu7siQgDqFHuDpxbspqaxhxqf2grL4vvOY8Mxyqmo08TbXlVtSyfFCeytszb5c83t0ZPq7G/l+p+FKaO3GFRUc4EfbmJP/m5iwQFOMVqbn2LlqAP5+WQr7s0ucMkZaLfrDeaXmfAJH37fV7VFmeRS3RjAdKShzEoUOcWEeTayyjp0MSY4zxzJ22kR4bMk4gVLQKjKY5IRwvt12jPSsYtYdyGNC39b0aRdlFzb4f+9u5K7zutAxPoyc4go6xoWZkVTFNoPltlZuQWmV3TjK0RPlHM4rJbOgzOkaPt2Ywddbj3LbuZ2ZYnO/WamqqeXR/23n/87vyvNL9tgZLK8t38cd53UhPct+zoKjRf/yj/vYfLiANfvz0BpuGN6J83okcsfb603D6vPfnsPlL9WdxmJrZiGLLDPcAV67MY073naOrrFl198nUVRezZDHvnfaZvs//nxjJgVlldx1Xld+OVzgMiQ40+G+aA6RB18QenPhkdM3hXz66K74K8VN5ySbESupHWLoGGdE6Vw20P7m/+6+85xCFru2OhnRM7FvGzt3EMB1wzrx2cZMnriyPymWsQLbmay920TxwrSB5JdUsuNIIUmxYVzUr63bNt84IpmhnePMz3+9qDfndEtw+UONt7ijIoIDmH1pX+770L2/edlu4xHacYDOGqXUITbUjDyyss1mwfSYsEA+Xp9h54KxxSryYHS0thPEwoP8KamsIcjfj7bRJ4W+pp61AgZ3jLXraMFIaFdaWYPWmgnPLDdFf292MetdpEEurazhka+2sy3TOM5BFyGprSKDuXVkZ9Kzi8kqLLez0m2xCkS3VhEoy8D6jiOFRAYHUFRRTWZBGYmRwQT4+9ElIZwFW4+aT0iTU9rSIS6UJ9nNxf3bUlOr+WbbMV61DBaGBPpxzdAOZkfgruN5bsmvfG8zOD776x3MdjNwaL2Oe97/hYv6tcXfT/HN1qOEBPlzQc9WrNufx7urD5GRX2beH1YeW7iTyppa/u0wcau4opp3Vx+kV5tI1h3IZ3vmCXq2jmS3JVooISKYYV3i7PYZkBTDyG7xdlFnjny95QgK6NUmkl3HiuiccPJ398zUAU73dqC/IiTQn5BAfy5LbcfK9FwW3nMu76w+yAuW4Ilrh3Xkv2sOmU+QP6Xnsj+nhN5to5zOf6Sg7LTkMfJ+oW+CFAgN5S+Tepnv+yfFcM+YbtwwIpnEyGBWzxxLKwfrtHvrSLq3jrQrCw7wZ9Os8ezLKXESeTD89dseneg28ZjVMogND2L+LUPrbbNjm6JCAl2KPEAbi2jeOrIzlw9M4rEFO81BvxFd4t2K8qjuCazYk8OfJvQgPDiA565JZXCnWH78NZuPN2SQGBlMdlGFGYFw/fCORAQH8sqPzo/WrogPDzLHO+LCg7jrvC48/s0uqms1baNPDpZbV/+6b1wPfs0qMqN8rHRJDLcbTL5jVGcWbTfC6lam55gib23vb15xjoven1PCYsug5TVDOrhMQ50YGcysS05OMn9q0S7+s3Qvgf6KF6YN4s+fbOavF/YmISKYVftyCQn0N1NmF1VU0z8p2vTPW6OxbIUKDDdfcnwYz12TygW9WhEVEsh5Ty41DYu+7aJpHWn/JGpLp/gwDuaWmiJ/Ttd4dhwtdOmeeevWoazel2sXann1q6uYfWlfpr9npLx+/Ip+5jW4m7X61CLnSUsHc0t50CZyDeDyQe154hsj2icxMpiokEAev6IfMy1uRz8/xS3ndHYr9NZru7B/W56dmkppRQ3hwf7cMLwTaclG4APYC/1n00ea7+dcnUqN1gT6+3H7uV1Mob/rvC78d81J14/VlbbTRZz9Pxee/O00Jz4g9M3ruqkPfz/F/RN6mp/bRLv/UTkSExbEoI6uQy/BdXbJt24d2qDY3vYxoWQWlNUZgeNIVEgg6Y9NNi2RAD+jI4oODWTO1AFsPlzA/717Mpf9FQPb0yEujNtHdSY00N/c79JUY8zh2qEdKSyrZkhyLDfNW0tJZQ2DO8Xyj8v6UVxRzYGcEjOB3Pf3n8+4OT9ybrcEEiKCWJmeY/5QAvz97GYbW11dReVVtLMReuv57zq/C0H+fqbQB/n7ccOIToQE+ptREM9MHcDlA5NY/qsxeen1FftpFRnMir9cwPyfDphCAzBjci/z8x4b18PdY7rx8YYMJ/dDcID9PdmvfQwAw7vEMymlDZNS2pjbxvUx5jzYRhU9cUV/3ltzkKTYMKaPNqanOAp9x7gwlFLmdw3YhZx2SQjH8u9jQFI0mx3CCT++awRD/7kEgLk3DOaCXq3o/rdvLO2MY/W+PPyUEW0UGxbI78d0sxP6DQfzuej5k2GfMz/bSjfL+JOriJjYsEA6xYfz50k93SYEBJjQpzXThna0E3qANg7u03F9WvPfO4a5PNbADjEczC1l+vldCfT3IzrM+CL+flmKWScuPMiMyf/23lH0anPSKvfzU/hZhk6jwwKZf8sQ2kaH0Ck+nNBAf4/mXdiK/LXDnAMwmgrvEvqnukGVw+BLjeWLPI0WfUtyfo+GTTZ7/47hrNqXQ1hQw26FAJunDKtwvnf7MNpGhzq5PUZ0jeeqOuL/lVKmUJ3fM5GFW4+ZYwQRwQG8csNg0rOKeGPlATonhBMbFkhqhxj+OKEH5VW1jHpyqRk6GmUZ//jzxJ7mymCF5dVEhRrX17ttFE9e2Z9vtx8150H8ZVIv2seGmm4GgDvP68rPe3M5v4cxuBoS5M+i7YaFfseozgQH+NuJrr+fYlzvVnbCf16PRN6+1Xia+vYPo5j99Q6nqB9bzulmROFcP7yT2zrWzmx0z0T6tIviscvtYx06J9oLvSu3wPAu8eagYXhwgPnE2D8phuuGdeLPNuMptpFZ5/dMJNDfjyB/PyprapnYtw2r9+VxQc9WDOgQQ5+2UXb3hSPWJzqr//2gQ96if17ej2lDO5iziy/s14aFW485HWfzwxPMiDUrQ5MNt40rgyXeRdDBZantuG54J3q3jTJdn65oHxNKXkkl790+zE7kXXFBz1bm+4iQAMqqapwEPzI4gIrqWmr0yQlb53SN5+XrBztdU1PiXUI/8HqocWHNxiZDoOeWtC/RMT6MjvGnZklYp81bhbNPO+MH8fAlffjlUAGT6xgbcOTi/u1YuPWY00Sibq0iefwKQ9S+uvtcEiKCUUoRGuTPjw+MNkPsggL8OPDERQBst/j6i8qrUErx3X3n0SoyhOiwQPolnfxxWzsZWwZ3imXrIxPNz7bRMtbZydanh0l92/DKDYPNMM3ebaN46OLepNrMYu7eOpKnrxrAMIt17IqokECz7XWx7dGJZmoAR7rYWPTPXZPqss7jV/QjOT6c55bsIbVDDJP6tuHecd259dzO5uD6JQPaMf38rnaTAq1PIJ//7hz2ZpeQYOnoereN4p6xzpFLVjrEhXLzOZ25dWQyH6/PMDsSx2ygo7on2GVxdeeWtBXE347uSmJkMNFhRpmjRQ/OaR2uG9aRv1+agp+fYkhynFN9W9rFhLA180SDRbhXm0iyiyoY06sVC7YepXNCODFhgTw6pS/9k2IAw5Xzf+9u4LHL+zWryIO3Cf24R1q6BT7JfeN78IcPNpkDnm2jQ03BumVkXXs6M97iopjYt43bOh0cJpO5S6ds+rMt4uU4DtIQEiKCzMfsAZYfqlVUrJa4tTNJjAwmJszZ5WZ9AujbLoo3PRg3cYerMRsrMWFB3DuuO+N6t3ZrqYYFBXDf+B5GRE7bKJRS3DvOmBvhSWfTt100fdsZx379xjRG9Uios/6KP48x3189pIPdE8N943rw8YbDZOSXERNmL3YRwcZnpWDeTUNoFxPqtMLan23GwwBiLd+7bZRWx7gwJvRpTaC/Hwu2HqV1VIjH0S3tY4x7LcpNpJw7np2aysJtx+jeKoIFW4+SXVTB0j+NtqvTu20UPz5wQYOO21i8S+iFFuHS1PZ2PuBTIdDfj22PTnQbN98QrMLax0W0Q0P5dPo5bM44wb7sYpJiDX//sC7xLLxnFL3bnuxA6upMAv39+P7+82gXE9pgV1lDsIp2fVjFuj7m3zzEKXTYinXswJbF953HibIqrnpllcsJg0v/NJolO4+zel8ev72gK9OGdWDV3lynsGNr8sCOcWFc0KuV03Fc4eeneOvWoXbzUAL8/Zh7YxpllTW0jQ7h9lGdPToWQM82EYQE+hHnJk2JO+IjgrlheCdzwLmuiLfTgToT8ymnpaXp9evrjmUVBE/YfLiA5Phw89FeOH1kFZUTHODfaLfE6yv28Y8FO2kfE8pPM8bUv0MzUFOrySoqt4vaaigFpZV2YyHNhVJqgyWBpBNi0QteTV3ZPoXmpVUdYZueYLXoGxJF1tT4+6lTEnnApRvvdOP9a8YKgnBWYs0f5S6xneA5YtELgnBGkhQbxjNTB5jZTIXGI0IvCMIZy+UDz5xVSc9mxHUjCILg5YjQC4IgeDki9IIgCF6OCL0gCIKX45HQK6UmKaV2K6XSlVIzXGwPVkp9aNm+RimVbLNtpqV8t1JqouO+giAIQvNSr9ArpfyB/wCTgT7ANKVUH4dqtwH5WutuwDPAvyz79sFYTLwvMAl4yXI8QRAE4TThiUU/FEjXWu/TWlcCHwCXOtS5FHjL8v4TYKwy0tBdCnygta7QWu8H0i3HEwRBEE4Tngh9e8B2eZwMS5nLOlrrauAEEO/hvgAope5USq1XSq3Pzs52VUUQBEFoBGfMhCmt9VxgLoBSKlspdbCeXdyRALhf3cE7kWv2DeSafYPGXrPbFWs8EfpMwHZ5oCRLmas6GUqpACAayPVwXye01g1bJskGpdR6dxncvBW5Zt9Artk3aI5r9sR1sw7orpTqrJQKwhhc/cqhzlfATZb3vwF+0Eb+46+AayxROZ2B7sDapmm6IAiC4An1WvRa62ql1N3AIsAfmKe13q6Umg2s11p/BbwBvKOUSgfyMDoDLPU+AnYA1cDvtNb1r5grCIIgNBke+ei11guBhQ5ls2zelwNXudn3MeCxU2hjQ5l7Gs91piDX7BvINfsGTX7NZ+QKU4IgCELTISkQBEEQvBwRekEQBC/Ha4S+vnw8ZytKqXlKqSyl1Dabsjil1HdKqT2Wv7GWcqWUet7yHWxRSg1quZY3HqVUB6XUUqXUDqXUdqXUHyzlXnvdSqkQpdRapdRmyzU/ainvbMkflW7JJxVkKXebX+psQynlr5T6RSn1teWzV1+zUuqAUmqrUmqTUmq9paxZ722vEHoP8/GcrbyJkSfIlhnAEq11d2CJ5TMY19/d8roTePk0tbGpqQb+qLXuAwwHfmf5f3rzdVcAY7TWA4BUYJJSajhG3qhnLHmk8jHySoGb/FJnKX8Adtp89oVrvkBrnWoTL9+897bW+qx/ASOARTafZwIzW7pdTXh9ycA2m8+7gbaW922B3Zb3rwLTXNU7m1/Al8B4X7luIAzYCAzDmCEZYCk373OMcOcRlvcBlnqqpdveiGtNsgjbGOBrQPnANR8AEhzKmvXe9gqLngbk1PESWmutj1reHwNaW9573fdgeTwfCKzBy6/b4sLYBGQB3wF7gQJt5I8C++tyl1/qbONZ4M9AreVzPN5/zRpYrJTaoJS601LWrPf2GZPrRmgcWmutlPLKGFmlVATwKXCv1rrQSIhq4I3XrY3JhKlKqRjgc6BXy7aoeVFKXQxkaa03KKVGt3BzTifnaq0zlVKtgO+UUrtsNzbHve0tFn2jcuqcxRxXSrUFsPzNspR7zfeglArEEPn3tNafWYq9/roBtNYFwFIMt0WMJX8U2F+Xec0O+aXOJkYCU5RSBzDSn48BnsO7rxmtdablbxZGhz6UZr63vUXoPcnH403Y5ha6CcOHbS2/0TJSPxw4YfM4eNagDNP9DWCn1nqOzSavvW6lVKLFkkcpFYoxJrETQ/B/Y6nmeM2u8kudNWitZ2qtk7TWyRi/2R+01tfhxdeslApXSkVa3wMTgG00973d0gMTTTjAcSHwK4Zf828t3Z4mvK73gaNAFYZ/7jYMv+QSYA/wPRBnqaswoo/2AluBtJZufyOv+VwMP+YWYJPldaE3XzfQH/jFcs3bgFmW8i4YiQDTgY+BYEt5iOVzumV7l5a+hlO8/tHA195+zZZr22x5bbdqVXPf25ICQRAEwcvxFteNIAiC4AYRekEQBC9HhF4QBMHLEaEXBEHwckToBUEQvBwRekEQBC9HhF4QBMHL+X+3npggZ8KK3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    for args in [\n",
    "        {'model_type': 'GraphSage', 'dataset': 'cora', 'num_layers': 2, 'heads': 1, 'batch_size': 32, 'hidden_dim': 32, 'dropout': 0.5, 'epochs': 500, 'opt': 'adam', 'opt_scheduler': 'none', 'opt_restart': 0, 'weight_decay': 5e-3, 'lr': 0.01},\n",
    "    ]:\n",
    "        args = objectview(args)\n",
    "        for model in ['GraphSage']:\n",
    "            args.model_type = model\n",
    "\n",
    "            # Match the dimension.\n",
    "            if model == 'GAT':\n",
    "              args.heads = 2\n",
    "            else:\n",
    "              args.heads = 1\n",
    "\n",
    "            if args.dataset == 'cora':\n",
    "                dataset = Planetoid(root='cora/', name='Cora')\n",
    "            else:\n",
    "                raise NotImplementedError(\"Unknown dataset\") \n",
    "            test_accs, losses, best_model, best_acc, test_loader = train(dataset, args) \n",
    "\n",
    "            print(\"Maximum test set accuracy: {0}\".format(max(test_accs)))\n",
    "            print(\"Minimum loss: {0}\".format(min(losses)))\n",
    "\n",
    "            # Run test for our best model to save the predictions!\n",
    "            test(test_loader, best_model, is_validation=False, save_model_preds=True, model_type=model)\n",
    "            print()\n",
    "\n",
    "            plt.title(dataset.name)\n",
    "            plt.plot(losses, label=\"training loss\" + \" - \" + args.model_type)\n",
    "            plt.plot(test_accs, label=\"test accuracy\" + \" - \" + args.model_type)\n",
    "        plt.legend()\n",
    "        plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gHELqjARZ1W5"
   },
   "source": [
    "## Question 1.1: What is the maximum accuracy obtained on the test set for GraphSage? (10 points)\n",
    "\n",
    "Running the cell above will show the results of your best model and save your best model's predictions to a file named *CORA-Node-GraphSage.csv*.  \n",
    "\n",
    "As we have seen before you can view this file by clicking on the *Folder* icon on the left side pannel. When you sumbit your assignment, you will have to download this file and attatch it to your submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "gnn",
   "language": "python",
   "name": "gnn"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}