CS_280_Lesson_1_Bayesian_Classifier.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/gabosantos/1920cs129.15/blob/master/CS_280_Lesson_1_Bayesian_Classifier.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Overview\n",
        "In this lesson you will learn:\n",
        "* What is the fundamental definition of AI\n",
        "* What are the applications and types of machine learning\n",
        "* How to model uncertainty using Naive Bayes\n"
      ],
      "metadata": {
        "id": "2yEr_Uiribdv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Concepts"
      ],
      "metadata": {
        "id": "UIY84Sgcirvj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## A. Artificial Intelligence\n",
        "\n",
        "<font color=\"#9980FA\">Artificial intelligence</font> is the machine's ability to <font color=\"#9980FA\">perform tasks that require any form of intelligence</font>—human or otherwise."
      ],
      "metadata": {
        "id": "JMtsVU74jK6v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Approaches to AI\n",
        "\n",
        "There are four main approaches to AI."
      ],
      "metadata": {
        "id": "NesyZI1ah2pw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The <font color=\"#9980FA\">Acting Humanly or Turing Test approach</font> aims to develop AI that can mimic human behavior. For an AI to pass a Turing Test, it should be able to engage in a conversation so that a human cannot reliably distinguish whether they are communicating with a machine or another person. This feat can be achieved using natural language processing (NLP), knowledge representation, automated reasoning, and machine learning. For a Total Turing Test, computer vision, speech recognition, and robotics might be necessary to incorporate into the AI.\n",
        "\n",
        "Try this: [Cleverbot](https://www.cleverbot.com/)\n",
        "\n",
        "\n",
        "<table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1s0dbQnetj61IBDWAv4r_T53SCOZWWa4C\" height=\"360\"/></td></tr></table>"
      ],
      "metadata": {
        "id": "o_YXJXuhshLT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The <font color=\"#9980FA\">Thinking Humanly or Cognitive Modeling approach</font> focuses more on making an AI that simulates the human thought process. This approach touches on neuroscience and psychology, which study human thoughts through introspection, psychological experiments, and brain imaging. Cognitive science is a relatively young field that combines all  \"disciplines of the brain,\" including computer models from AI.\n",
        "\n",
        "Try this: [Akinator](https://en.akinator.com/game)\n",
        "\n",
        "<table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1_C6AIo1dJIvR8eZfPBCdTErIOzp3K0zd\" height=\"360\"/></td></tr></table>"
      ],
      "metadata": {
        "id": "De-VPbHrV4pf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<div class=\"black\">\n",
        "    This markdown cell is blue.\n",
        "</div>\n"
      ],
      "metadata": {
        "id": "mc3q8CRmdF50"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The <font color=\"#9980FA\">Thinking Rationally or Laws of Thought approach</font> attempts a more universal approach to modeling the operations of the mind. Logicism is the belief that all human reasoning and thought processes can be reduced to and explained by formal logic, which follows established rules and principles (e.g., syllogisms). This approach assumes that humans fundamentally think rationally and logically, and seeks to capture and replicate these logical processes in AI systems. However, given the uncertainties and limitless possibilities in real life, rational thinking may be challenging in practical situations.\n",
        "\n",
        "Try this: [Symbolab](https://www.symbolab.com/)\n",
        "\n",
        "<table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1zMTQ3ZpVHffgbfl8zK-RyPbWULKQ-zTt\" height=\"360\"/></td></tr></table>\n",
        "\n"
      ],
      "metadata": {
        "id": "VafXtUskV4eB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "The <font color=\"#9980FA\">Acting Rationally or Rational Agent approach</font> tackles the practical challenge with the Thinking Rationally approach. This perspective prioritizes AI systems that can make decisions and take actions that maximize their chances of achieving their goals effectively and efficiently. Rational behavior requires rational thinking, but the converse is not always true. In some instances, humans rely on quick and innate reflexes, like when touching a hot surface or hearing a speeding vehicle. These automatic reactions occur spontaneously and are not always guided by deliberate, rational thought processes.\n",
        "\n",
        "Try this: [Google Search](https://blog.google/products/search/25-biggest-google-search-updates/)\n",
        "\n",
        "<table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1S228xWinBYtIfXzIJp8WxcQzczkz91x4\" height=\"360\"/></td></tr></table>\n",
        "\n",
        "The Rational Agent approach also requires skills from the other two approaches. Many instances of rational behavior in the world are exemplified by actions akin to how humans act (Acting Humanly) and are guided by cognitive processes similar to human thinking (Thinking Humanly). Nevertheless, there are also instances where rational behaviors can be inspired by non-human actors (e.g., intelligent animals or automated systems), highlighting the versatility of this approach.\n",
        "\n"
      ],
      "metadata": {
        "id": "FdcXIEjAV4TQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Foundational Disciplines\n",
        "AI is a multidisciplinary field that draws upon and integrates various disciplines to develop intelligent systems capable of mimicking human-like or rational functions."
      ],
      "metadata": {
        "id": "9p-aiPnLh8Pa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<font color=\"#9980FA\">Philosophy</font> delves into fundamental questions about the nature of intelligence and consciousness. It also grounds us to the ethical implications of creating machines with the ability to reason. One notable concept is the notion of utility (from utilitarianism).\n",
        "\n",
        "In AI, utility measures the desirability or goodness of an outcome or decision. AI systems often employ utility functions to make choices that maximize desirable outcomes, reflecting the utilitarian idea of maximizing overall happiness or well-being.\n",
        "\n",
        "Additionally, the concept of \"consequences\" in AI aligns with philosophical considerations surrounding ethics and morality. Philosophers have long explored the moral implications of actions and their consequences. In AI ethics, understanding the potential consequences of algorithmic decisions is crucial. For example, AI developers should examine the impact of automated systems on society, considering potential biases, discrimination, and unintended outcomes.\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1RKN-wVzbWpY0fpeJ5eUNrfF1NzS1Yugb\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From datasciencecentral.com</small></center>\n"
      ],
      "metadata": {
        "id": "n7tbVxtEktDj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<font color=\"#9980FA\">Math</font> is at the core of AI (and computer science in general), serving as the fundamental framework for formulating complex models and algorithms. Fields of mathematics such as linear algebra, calculus, probability theory, and statistics provide the foundational tools for modeling data, learning patterns, and making informed decisions in AI systems.\n",
        "\n",
        "Linear algebra, for instance, is indispensable for representing and manipulating data in multi-dimensional spaces, while calculus plays a vital role in optimization algorithms. Probability theory and statistics enable AI to make predictions, estimate uncertainties, and perform data-driven inference.\n",
        "\n",
        "Additionally, advanced mathematical concepts like differential equations and graph theory are employed in specialized AI applications, further emphasizing the deep connection between mathematics and AI.\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=11a45VV1wco1sKuW5CptH6F7RaawWoL9y\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From Shafi @medium.com</small></center>"
      ],
      "metadata": {
        "id": "EpjQR0aJks6G"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "AI is also inspired by <font color=\"#9980FA\">economics</font>, particularly in the way it models decision-making and resource allocation. Concepts from economics, such as utility theory and game theory, have a significant influence on AI.\n",
        "\n",
        "Utility theory, for example, is used to represent and quantify preferences and values in AI systems, helping them make choices that maximize expected benefits or outcomes (similar to philosophy).\n",
        "\n",
        "Game theory, on the other hand, is employed in scenarios where multiple agents or entities interact strategically, mirroring real-world situations like negotiation, competition, and cooperation.\n",
        "\n",
        "Furthermore, economic principles guide AI in optimizing resource allocation, as seen in the allocation of computational resources, energy, or bandwidth in distributed systems. AI systems often use economic models to allocate resources efficiently, balancing cost and performance.\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1qMbt36V4itLPoQJLIdv16YwDlex8N4pE\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From codertime.org</small></center>"
      ],
      "metadata": {
        "id": "6Mvzkxqvksxz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<font color=\"#9980FA\">Neuroscience</font> and psychology play pivotal roles in shaping the development and understanding of AI. From the perspective of neuroscience, AI draws inspiration from the structure and function of the human brain. Artificial neural networks, for instance, are computational models directly inspired by the neural connections and information-processing mechanisms in the brain.\n",
        "\n",
        "<font color=\"#9980FA\">Psychology</font> contributes by helping AI systems better understand and mimic human behavior and cognition. Cognitive psychology, in particular, provides insights into how humans perceive, learn, and reason, which has influenced the design of intelligent algorithms. For example, cognitive psychology informs the development of recommendation systems, chatbots, and virtual assistants that aim to interact with users in a way that feels natural and empathetic.\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1wJ5oUQ1Q8NvPWy-s1ZHCHIKWGu2kUAQu\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From Naveen @nomidl.com</small></center>"
      ],
      "metadata": {
        "id": "ymcRLHYXksnS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Several other fields have influenced the development of AI. <font color=\"#9980FA\">Computer engineering</font> provides the essential hardware infrastructure needed for computationally heavy AI training. <font color=\"#9980FA\">Control theory</font> enables autonomous decision-making in robotics and automation, while <font color=\"#9980FA\">cybernetics</font> contributes concepts related to self-regulation and feedback loops. <font color=\"#9980FA\">Linguistics</font> contributes to natural language understanding and processing in AI, allowing for effective human-computer communication.\n",
        "\n",
        "Together, these disciplines synergize to form the foundation of AI, facilitating the development of intelligent systems capable of learning, reasoning, and adapting in complex environments.\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1IJoUShIUeckeKPLvMBebCzheOJE9KIL_\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From wikimedia.org</small></center>"
      ],
      "metadata": {
        "id": "K80x6R2sksba"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## B. Machine Learning\n",
        "\n",
        "<font color=\"#9980FA\">Machine learning</font> is the science (and art) of teaching computers to <font color=\"#9980FA\">learn from data</font> without explicitly programming them.\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1BoM4v_3vVuJSD7LudLCtOuU2cMaSawZ2\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From r/programmerhumor</small></center>"
      ],
      "metadata": {
        "id": "kZ3QkLSGdiMq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ML Applications\n",
        "Some example applications of machine learning include <font color=\"#9980FA\">image recognition</font>, where machine learning models can classify and identify objects within images, enabling applications like facial recognition, content moderation, and autonomous vehicles' perception systems.\n",
        "\n",
        "In <font color=\"#9980FA\">natural language processing</font>, machine learning algorithms analyze and understand human language, enabling chatbots, language translation, sentiment analysis, and voice assistants like Siri and Alexa.\n",
        "\n",
        "Machine learning also powers <font color=\"#9980FA\">recommendation systems</font> used by platforms like Netflix and Amazon to suggest movies, products, or content tailored to individual preferences.\n",
        "\n",
        "In <font color=\"#9980FA\">healthcare</font>, machine learning aids in medical image analysis, disease prediction, and drug discovery. These diverse applications highlight the versatility of machine learning in solving complex problems and making data-driven decisions across various domains.\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1CHqQzRjR5pQ78KA281tIBUZ-kX1CDUha\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From Abdul Rahid</small></center>"
      ],
      "metadata": {
        "id": "JvmycI9gvFa3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Learning Systems\n",
        "Machine learning can be broadly categorized into three main types: supervised learning, unsupervised learning, and reinforcement learning, each with distinct approaches and applications.\n"
      ],
      "metadata": {
        "id": "niwB1UfZvFGv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In <font color=\"#9980FA\">supervised learning</font>, the algorithm learns from labeled data, where input data is paired with corresponding output labels. The goal is to predict the correct label or outcome for new, unseen data based on patterns and relationships learned from the training data.\n",
        "\n",
        "> *In ML, we call input variables as <font color=\"#9980FA\">features/predictors</font>, and output variables as <font color=\"#9980FA\">target/predictions</font>.*\n",
        "\n",
        "Supervised learning is commonly used in tasks such as image classification, spam email detection, and language translation. It is well-suited for problems where there is a clear distinction between input and output and abundant labeled training data is available."
      ],
      "metadata": {
        "id": "ADGZNACyvEkf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<font color=\"#9980FA\">Unsupervised learning</font> operates on unlabeled data, seeking to discover patterns, structures, or relationships within the data itself. It doesn't rely on predefined output labels but rather identifies inherent structures or clusters. Examples include clustering similar documents, segmenting customer groups, and reducing data dimensionality.\n",
        "\n",
        "Unsupervised learning is valuable when dealing with large datasets and can unveil hidden insights and patterns that may not be apparent through human observation alone."
      ],
      "metadata": {
        "id": "zQGI7ESrvEam"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<font color=\"#9980FA\">Reinforcement learning</font> is centered around agents that interact with an environment to achieve specific goals. The agent takes action and receives feedback in the form of rewards or penalties based on its decisions. Over time, the agent learns to optimize its actions to maximize cumulative rewards.\n",
        "\n",
        "Applications of reinforcement learning include game-playing algorithms (e.g., AlphaGo), robotics control, autonomous navigation, and recommendation systems that adapt to user preferences. Reinforcement learning is particularly powerful in situations where the optimal action may not be immediately clear, and the agent must explore and learn from trial and error.\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1T9D7Q4zHs1X_HGWUJ6hkUYX5kX9zlhYe\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From Mathworks</small></center>"
      ],
      "metadata": {
        "id": "CEeBmr0HvPuZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## C. Uncertainty and Naive Bayes\n",
        "\n",
        "An indicator of intelligence lies in the capacity to <font color=\"#9980FA\">make decisions amidst uncertainties</font> and effectively adapt to changes in the environment. In problem-solving AI systems, dealing with uncertainties involves keeping track of various possible scenarios (belief states) and creating plans for each of them. However, this can become computationally intensive because the number of scenarios can grow rapidly. To address this challenge, we can employ <font color=\"#9980FA\">probabilistic modeling techniques</font> like Bayesian inference.\n",
        "\n",
        "> *When faced with uncertainties, the <font color=\"#9980FA\">rational decision</font> depends on the relative importance of various goals and the likelihood that they will be achieved.* [AIMA]\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1NMVVXDtrIz9pBmPzMZeRuEhg2B6hwoEK\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From brainkart.com</small></center>"
      ],
      "metadata": {
        "id": "WV2rH7nodnQE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Bayesian Inference\n",
        "\n",
        "<font color=\"#9980FA\">Bayesian inference</font> is one of the most common statistical techniques in probabilistic AI that involves updating beliefs or probabilities based on new evidence or data.\n",
        "\n",
        "To learn more about how Bayesian Inference works, visit this interactive lesson on Brown University: \\[[webpage](https://seeing-theory.brown.edu/bayesian-inference/index.html)\\]\n",
        "\n",
        "<br><table width=\"100%\"><tr><td align=\"center\" bgcolor=\"white\"><img src=\"https://drive.google.com/uc?export=view&id=1er3gOwdiXaXyWsErtkxT_qsrCUdQk__F\" height=\"360\"/></td></tr></table>\n",
        "<center><small>From Brown Univeristy</small></center>"
      ],
      "metadata": {
        "id": "w2HKZxdT8DP_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Naive Bayes Classifier\n",
        "\n",
        "We can expand Bayesian inference to include additional features. For instance, consider a scenario where a patient presents a range of symptoms, and our objective is to predict the disease they are most likely to be afflicted by.\n",
        "\n",
        "<br>\n",
        "Let $D_k$ be the disease class, $S$ be the set of symptoms. The <font color=\"#9980FA\">Naive Bayes model</font> can be formulated as:\n",
        "\n",
        "<br>\n",
        "\\begin{equation}\n",
        "\\begin{split}\n",
        "P(D_k, S) &= P(D_k) \\prod_i{P(S_i|D_k)} \\\\\n",
        "&= P(D_k) \\cdot P(S_1|D_k) \\ldots P(S_n|D_k)  \n",
        "\\end{split}\n",
        "\\end{equation}\n",
        "\n",
        "<br>\n",
        "\n",
        "We call it \"naive\" because we treat the features as if they are independent of each other, which is a convenient yet simplistic assumption. Although this assumption may not hold true in many real-world scenarios where features are correlated or dependent on each other, Naive Bayes can perform surprisingly well in various classification tasks.\n",
        "\n",
        "<br>\n",
        "\n",
        "To use Naive Bayes for classification:\n",
        "\n",
        "<br>\n",
        "\\begin{equation}\n",
        "P(D_k|S) = \\frac{P(S_1, S_2, S_3, \\ldots, y | D_k) \\cdot P(D_k)}{P(S_1, S_2, S_3, \\ldots, y)}\n",
        "\\end{equation}\n",
        "\n",
        "<br>\n",
        "\n",
        "where the marginal probability is the product of the probabilities of the symptoms, including the unobserved ones ($y$).\n",
        "\n",
        "<br>\n",
        "\n",
        "We can improve this model further by introducing a smoothing parameter to address situations where marginal probabilities are zero:\n",
        "\n",
        "<br>\n",
        "\\begin{equation}\n",
        "P(S_i|D_k) = \\frac{N_{S_i|D_k} + \\alpha}{N_{D_k} + \\alpha \\cdot |V|}\n",
        "\\end{equation}\n",
        "\n",
        "<br>\n",
        "\n",
        "where\n",
        "* $N_{S_i|D_k}$ is the number of patients with $D_k$ that had symptom $S_i$\n",
        "* $N_{D_k}$ is the number of patients with $D_k$\n",
        "* $\\alpha$ is smoothing parameter\n",
        "* $|V|$ is the number of all symptoms for all $D$"
      ],
      "metadata": {
        "id": "yWB2bwJf8C73"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The Naive Bayes Classifier is valuable for <font color=\"#9980FA\">text classification</font> because of its effectiveness in handling high-dimensional data, such as text documents. By adapting the formulation above, we can align $D_k$ with text categories (e.g., spam, sports, negative review) and $S_i$ with sets of words or terms within those documents. This transformation enables the model to analyze the likelihood of a given category being associated with a specific set of words, making it a powerful tool for tasks like email spam detection, sentiment analysis, and topic categorization."
      ],
      "metadata": {
        "id": "Kj9ijoJ6RkwI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Codes"
      ],
      "metadata": {
        "id": "nul_Uu1bjPtP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's take a look at how we can do text classification using the Naive Bayes Classifier. For this demonstration, we'll use the <font color=\"#9980FA\">`Rotten Tomatoes Reviews dataset`</font>, which includes both positive and negative text reviews. Our aim is to create a classifier that can determine the sentiment of these reviews."
      ],
      "metadata": {
        "id": "NjzhwDHTSsLy"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-i6L_s3ihvK-"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load the data set"
      ],
      "metadata": {
        "id": "yK2c6sw_Z-6t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset\n",
        "from pprint import pprint\n",
        "\n",
        "# Data set: https://huggingface.co/datasets/rotten_tomatoes\n",
        "dataset = load_dataset('rotten_tomatoes', split='train')\n",
        "print(dataset.info.description)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Fw-jJZTEgccG",
        "outputId": "5fc4c2cb-724f-4895-ac70-21d11b23f1a8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Movie Review Dataset.\n",
            "This is a dataset of containing 5,331 positive and 5,331 negative processed\n",
            "sentences from Rotten Tomatoes movie reviews. This data was first used in Bo\n",
            "Pang and Lillian Lee, ``Seeing stars: Exploiting class relationships for\n",
            "sentiment categorization with respect to rating scales.'', Proceedings of the\n",
            "ACL, 2005.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Select feature and target variables"
      ],
      "metadata": {
        "id": "FQEv-ClpaD5I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "df = pd.DataFrame({'text':dataset['text'], 'label':dataset['label']}, columns=['text', 'label'])\n",
        "\n",
        "X_train = df['text']  # features\n",
        "y_train = df['label'] # target (positive=1, negative=0)\n",
        "\n",
        "print(\"Sample positive reviews\")\n",
        "print(df[df['label']==1][:5])\n",
        "\n",
        "print(\"\\nSample negative reviews\")\n",
        "print(df[df['label']==0][:5])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sACXTIwqhASf",
        "outputId": "4c78067c-e53e-42ca-8dd3-d3225597d44e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sample positive reviews\n",
            "                                                text  label\n",
            "0  the rock is destined to be the 21st century's ...      1\n",
            "1  the gorgeously elaborate continuation of \" the...      1\n",
            "2                     effective but too-tepid biopic      1\n",
            "3  if you sometimes like to go to the movies to h...      1\n",
            "4  emerges as something rare , an issue movie tha...      1\n",
            "\n",
            "Sample negative reviews\n",
            "                                                   text  label\n",
            "4265                   simplistic , silly and tedious .      0\n",
            "4266  it's so laddish and juvenile , only teenage bo...      0\n",
            "4267  exploitative and largely devoid of the depth o...      0\n",
            "4268  [garbus] discards the potential for pathologic...      0\n",
            "4269  a visually flashy but narratively opaque and e...      0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Convert text to numerical features\n",
        "\n",
        "We will use <font color=\"#9980FA\">`CountVectorizer`</font> for bag-of-words approach. Another option is the <font color=\"#9980FA\">`TfidfVectorizer`</font>, which offers an alternative representation method that considers the importance of words not only within individual documents but also in the context of the entire corpus."
      ],
      "metadata": {
        "id": "Fy-8FeWHaH45"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "vectorizer = CountVectorizer()\n",
        "# vectorizer = CountVectorizer(stop_words='english')  # remove stop words\n",
        "X_train_vec = vectorizer.fit_transform(X_train)\n",
        "\n",
        "words = vectorizer.get_feature_names_out()\n",
        "counts = X_train_vec.toarray().sum(axis=0)\n",
        "\n",
        "df_words = pd.DataFrame({'words':words, 'count':counts}, columns=['words', 'count'])\n",
        "df_words.sort_values(by=['count'], ascending=False).head(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "id": "v_VKyjMnXkeJ",
        "outputId": "1088aefb-7786-4881-e56a-d5c891c1b514"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      words  count\n",
              "14636   the   8115\n",
              "661     and   4971\n",
              "9992     of   4886\n",
              "14825    to   3447\n",
              "7828     it   2743\n",
              "7814     is   2713\n",
              "7342     in   2146\n",
              "14634  that   2130\n",
              "918      as   1412\n",
              "2009    but   1326"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-80c2b31e-9904-4e88-8be0-d6ecdd41b224\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>words</th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>14636</th>\n",
              "      <td>the</td>\n",
              "      <td>8115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>661</th>\n",
              "      <td>and</td>\n",
              "      <td>4971</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9992</th>\n",
              "      <td>of</td>\n",
              "      <td>4886</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14825</th>\n",
              "      <td>to</td>\n",
              "      <td>3447</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7828</th>\n",
              "      <td>it</td>\n",
              "      <td>2743</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7814</th>\n",
              "      <td>is</td>\n",
              "      <td>2713</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7342</th>\n",
              "      <td>in</td>\n",
              "      <td>2146</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14634</th>\n",
              "      <td>that</td>\n",
              "      <td>2130</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>918</th>\n",
              "      <td>as</td>\n",
              "      <td>1412</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2009</th>\n",
              "      <td>but</td>\n",
              "      <td>1326</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-80c2b31e-9904-4e88-8be0-d6ecdd41b224')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-80c2b31e-9904-4e88-8be0-d6ecdd41b224 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-80c2b31e-9904-4e88-8be0-d6ecdd41b224');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-59359314-a09f-4e6d-9a9b-1f91f45b3795\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-59359314-a09f-4e6d-9a9b-1f91f45b3795')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-59359314-a09f-4e6d-9a9b-1f91f45b3795 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Fit the Naive Bayes Classifier\n",
        "\n",
        "Scikit-learn provides pre-built Naive Bayes classification models. Choosing which type of model mostly depends on the feature types:\n",
        "\n",
        "* <font color=\"#9980FA\">`BernoulliNB`</font>:   binary/boolean features\n",
        "* <font color=\"#9980FA\">`MultinomialNB`</font>: discrete features\n",
        "* <font color=\"#9980FA\">`GaussianNB`</font>:    continuous features"
      ],
      "metadata": {
        "id": "ktAJV-p2as4v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.naive_bayes import MultinomialNB\n",
        "\n",
        "model = MultinomialNB()\n",
        "model.fit(X_train_vec, y_train)\n",
        "y_pred = model.predict(X_train_vec) # Note: this should be done a separate test set\n",
        "y_true = y_train                    # Note: this should be done a separate test set\n",
        "\n",
        "# Print some of the prediction results\n",
        "prediction_results = pd.DataFrame({'Target':y_true, 'Prediction':y_pred}, columns=['Target', 'Prediction'])\n",
        "print(prediction_results.sample(n=20, random_state=280))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TjXVJUmuS74r",
        "outputId": "3caa1c20-e583-4f70-d0c1-5286fa7f8e9f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "      Target  Prediction\n",
            "8211       0           0\n",
            "6479       0           0\n",
            "446        1           1\n",
            "7572       0           0\n",
            "7135       0           0\n",
            "7799       0           0\n",
            "5768       0           0\n",
            "6286       0           0\n",
            "6695       0           0\n",
            "4738       0           0\n",
            "1682       1           1\n",
            "4608       0           0\n",
            "8488       0           0\n",
            "703        1           1\n",
            "8073       0           0\n",
            "469        1           1\n",
            "4621       0           0\n",
            "5227       0           0\n",
            "5616       0           0\n",
            "1699       1           1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Analyze the model performance\n"
      ],
      "metadata": {
        "id": "6MmiGq9JbL4d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Compute confusion matrix\n",
        "from sklearn.metrics import confusion_matrix\n",
        "\n",
        "cm = confusion_matrix(y_true, y_pred, labels=[1, 0])\n",
        "\n",
        "# Plot confusion matrix\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "ax = plt.subplot()\n",
        "sns.heatmap(cm, annot=True, fmt='g', ax=ax)\n",
        "\n",
        "ax.set_xlabel('Prediction')\n",
        "ax.set_ylabel('Target')\n",
        "ax.set_title('Confusion Matrix')\n",
        "ax.xaxis.set_ticklabels(['positive', 'negative'])\n",
        "ax.yaxis.set_ticklabels(['positive', 'negative'])\n",
        "\n",
        "plt.show()\n",
        "\n",
        "# Print performance statistics\n",
        "from sklearn.metrics import classification_report\n",
        "\n",
        "print(\"\\n\",classification_report(y_true, y_pred, labels=[1, 0], target_names=['Positive', 'Negative']))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 651
        },
        "id": "3HuHJ6TwWOV2",
        "outputId": "76c2fda1-3007-4a8c-ca94-dfc8aed26020"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAHHCAYAAACPy0PBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABdnElEQVR4nO3dd1hU17oG8HcQGJozgEizIIgF7KjRsRsLKkaNJkYlAsYSPFixEBIrFozGmkQx0SNG0WhirKiIKFbsQawEFSVRig0Q6cy+f3iZkwmYYXTGGfH9nWc/l1l77bW/PVfjx2pbJAiCACIiIiIdMtB1AERERERMSIiIiEjnmJAQERGRzjEhISIiIp1jQkJEREQ6x4SEiIiIdI4JCREREekcExIiIiLSOSYkREREpHNMSIi0KCkpCT179oRUKoVIJMKuXbs02v7du3chEokQHh6u0XbfZl26dEGXLl10HQYRqYkJCVV6t2/fxueffw4XFxeYmJhAIpGgffv2WLlyJfLy8rR6b19fX1y5cgULFizApk2b0KpVK63e703y8/ODSCSCRCIp93tMSkqCSCSCSCTCN998o3b7Dx48wJw5cxAfH6+BaIlI3xnqOgAibYqMjMTHH38MsVgMHx8fNG7cGIWFhTh58iSmTZuGa9eu4YcfftDKvfPy8hAXF4evvvoK48aN08o9nJyckJeXByMjI620r4qhoSFyc3Oxd+9eDB48WOlcREQETExMkJ+f/0ptP3jwAHPnzkWdOnXQvHnzCl936NChV7ofEekWExKqtJKTkzFkyBA4OTnhyJEjcHBwUJwLCAjArVu3EBkZqbX7P3z4EABgaWmptXuIRCKYmJhorX1VxGIx2rdvj61bt5ZJSLZs2QIvLy/s2LHjjcSSm5sLMzMzGBsbv5H7EZFmcciGKq3FixcjJycH69evV0pGSrm6umLixImKz8XFxZg3bx7q1q0LsViMOnXq4Msvv0RBQYHSdXXq1EHfvn1x8uRJvPfeezAxMYGLiwt++uknRZ05c+bAyckJADBt2jSIRCLUqVMHwIuhjtKf/27OnDkQiURKZdHR0ejQoQMsLS1hYWGBBg0a4Msvv1Scf9kckiNHjqBjx44wNzeHpaUl+vfvjxs3bpR7v1u3bsHPzw+WlpaQSqUYMWIEcnNzX/7F/sOwYcNw4MABZGZmKsrOnz+PpKQkDBs2rEz9J0+eYOrUqWjSpAksLCwgkUjQu3dvXL58WVEnNjYWrVu3BgCMGDFCMfRT+pxdunRB48aNcfHiRXTq1AlmZmaK7+Wfc0h8fX1hYmJS5vk9PT1hZWWFBw8eVPhZiUh7mJBQpbV37164uLigXbt2Fao/atQozJo1Cx4eHli+fDk6d+6M0NBQDBkypEzdW7du4aOPPkKPHj2wdOlSWFlZwc/PD9euXQMADBw4EMuXLwcADB06FJs2bcKKFSvUiv/atWvo27cvCgoKEBISgqVLl6Jfv344derUv153+PBheHp6IiMjA3PmzEFgYCBOnz6N9u3b4+7du2XqDx48GM+ePUNoaCgGDx6M8PBwzJ07t8JxDhw4ECKRCL/99puibMuWLWjYsCE8PDzK1L9z5w527dqFvn37YtmyZZg2bRquXLmCzp07K5IDNzc3hISEAADGjBmDTZs2YdOmTejUqZOincePH6N3795o3rw5VqxYga5du5Yb38qVK1G9enX4+vqipKQEALB27VocOnQI3377LRwdHSv8rESkRQJRJZSVlSUAEPr371+h+vHx8QIAYdSoUUrlU6dOFQAIR44cUZQ5OTkJAITjx48ryjIyMgSxWCxMmTJFUZacnCwAEJYsWaLUpq+vr+Dk5FQmhtmzZwt//yu5fPlyAYDw8OHDl8Zdeo8NGzYoypo3by7Y2toKjx8/VpRdvnxZMDAwEHx8fMrc77PPPlNq88MPPxSqVav20nv+/TnMzc0FQRCEjz76SOjWrZsgCIJQUlIi2NvbC3Pnzi33O8jPzxdKSkrKPIdYLBZCQkIUZefPny/zbKU6d+4sABDCwsLKPde5c2elsqioKAGAMH/+fOHOnTuChYWFMGDAAJXPSERvDntIqFLKzs4GAFStWrVC9ffv3w8ACAwMVCqfMmUKAJSZa+Lu7o6OHTsqPlevXh0NGjTAnTt3Xjnmfyqde7J7927I5fIKXZOamor4+Hj4+fnB2tpaUd60aVP06NFD8Zx/5+/vr/S5Y8eOePz4seI7rIhhw4YhNjYWaWlpOHLkCNLS0sodrgFezDsxMHjxn56SkhI8fvxYMRx16dKlCt9TLBZjxIgRFarbs2dPfP755wgJCcHAgQNhYmKCtWvXVvheRKR9TEioUpJIJACAZ8+eVaj+vXv3YGBgAFdXV6Vye3t7WFpa4t69e0rltWvXLtOGlZUVnj59+ooRl/XJJ5+gffv2GDVqFOzs7DBkyBBs3779X5OT0jgbNGhQ5pybmxsePXqE58+fK5X/81msrKwAQK1n6dOnD6pWrYpt27YhIiICrVu3LvNdlpLL5Vi+fDnq1asHsVgMGxsbVK9eHQkJCcjKyqrwPWvUqKHWBNZvvvkG1tbWiI+Px6pVq2Bra1vha4lI+5iQUKUkkUjg6OiIq1evqnXdPyeVvkyVKlXKLRcE4ZXvUTq/oZSpqSmOHz+Ow4cPY/jw4UhISMAnn3yCHj16lKn7Ol7nWUqJxWIMHDgQGzduxM6dO1/aOwIACxcuRGBgIDp16oTNmzcjKioK0dHRaNSoUYV7goAX3486fv/9d2RkZAAArly5ota1RKR9TEio0urbty9u376NuLg4lXWdnJwgl8uRlJSkVJ6eno7MzEzFihlNsLKyUlqRUuqfvTAAYGBggG7dumHZsmW4fv06FixYgCNHjuDo0aPltl0aZ2JiYplzN2/ehI2NDczNzV/vAV5i2LBh+P333/Hs2bNyJwKX+vXXX9G1a1esX78eQ4YMQc+ePdG9e/cy30lFk8OKeP78OUaMGAF3d3eMGTMGixcvxvnz5zXWPhG9PiYkVGlNnz4d5ubmGDVqFNLT08ucv337NlauXAngxZADgDIrYZYtWwYA8PLy0lhcdevWRVZWFhISEhRlqamp2Llzp1K9J0+elLm2dIOwfy5FLuXg4IDmzZtj48aNSv/AX716FYcOHVI8pzZ07doV8+bNw3fffQd7e/uX1qtSpUqZ3pdffvkF9+/fVyorTZzKS97UFRQUhJSUFGzcuBHLli1DnTp14Ovr+9LvkYjePG6MRpVW3bp1sWXLFnzyySdwc3NT2qn19OnT+OWXX+Dn5wcAaNasGXx9ffHDDz8gMzMTnTt3xrlz57Bx40YMGDDgpUtKX8WQIUMQFBSEDz/8EBMmTEBubi7WrFmD+vXrK03qDAkJwfHjx+Hl5QUnJydkZGRg9erVqFmzJjp06PDS9pcsWYLevXtDJpNh5MiRyMvLw7fffgupVIo5c+Zo7Dn+ycDAADNmzFBZr2/fvggJCcGIESPQrl07XLlyBREREXBxcVGqV7duXVhaWiIsLAxVq1aFubk52rRpA2dnZ7XiOnLkCFavXo3Zs2crliFv2LABXbp0wcyZM7F48WK12iMiLdHxKh8irfvjjz+E0aNHC3Xq1BGMjY2FqlWrCu3btxe+/fZbIT8/X1GvqKhImDt3ruDs7CwYGRkJtWrVEoKDg5XqCMKLZb9eXl5l7vPP5aYvW/YrCIJw6NAhoXHjxoKxsbHQoEEDYfPmzWWW/cbExAj9+/cXHB0dBWNjY8HR0VEYOnSo8Mcff5S5xz+Xxh4+fFho3769YGpqKkgkEuGDDz4Qrl+/rlSn9H7/XFa8YcMGAYCQnJz80u9UEJSX/b7My5b9TpkyRXBwcBBMTU2F9u3bC3FxceUu1929e7fg7u4uGBoaKj1n586dhUaNGpV7z7+3k52dLTg5OQkeHh5CUVGRUr3JkycLBgYGQlxc3L8+AxG9GSJBUGPmGhEREZEWcA4JERER6RwTEiIiItI5JiRERESkc0xIiIiISOeYkBAREZHOMSEhIiIinWNCQkRE9A5YtGgRRCIRJk2apCjLz89HQEAAqlWrBgsLCwwaNKjMztYpKSnw8vKCmZkZbG1tMW3aNBQXFyvViY2NhYeHB8RiMVxdXREeHq52fJVyp9bCuxd0HQKRXrJpNFjXIRDpneznd7R+j6JHmrmHkY2L6krlOH/+PNauXYumTZsqlU+ePBmRkZH45ZdfIJVKMW7cOAwcOBCnTp0C8OKln15eXrC3t8fp06eRmpoKHx8fGBkZYeHChQCA5ORkeHl5wd/fHxEREYiJicGoUaPg4OAAT0/PCsdYKTdGY0JCVD4mJERlVfaEJCcnBx4eHli9ejXmz5+P5s2bY8WKFcjKykL16tWxZcsWfPTRRwBevITTzc0NcXFxaNu2LQ4cOIC+ffviwYMHsLOzAwCEhYUhKCgIDx8+hLGxMYKCghAZGan0dvUhQ4YgMzMTBw8erHCcHLIhIiLSNnmJRo6CggJkZ2crHapeEhkQEAAvLy90795dqfzixYsoKipSKm/YsCFq166teEt6XFwcmjRpokhGAMDT0xPZ2dm4du2aos4/2/b09KzQm9b/jgkJERGRtglyjRyhoaGQSqVKR2ho6Etv+/PPP+PSpUvl1klLS4OxsTEsLS2Vyu3s7JCWlqao8/dkpPR86bl/q5OdnY28vLwKf0WVcg4JERGRXpHLNdJMcHAwAgMDlcrEYnG5df/8809MnDgR0dHRMDEx0cj9tYk9JERERG8JsVgMiUSidLwsIbl48SIyMjLg4eEBQ0NDGBoa4tixY1i1ahUMDQ1hZ2eHwsJCZGZmKl2Xnp4Oe3t7AIC9vX2ZVTeln1XVkUgkMDU1rfCzMSEhIiLSMkGQa+RQR7du3XDlyhXEx8crjlatWsHb21vxs5GREWJiYhTXJCYmIiUlBTKZDAAgk8lw5coVZGRkKOpER0dDIpHA3d1dUefvbZTWKW2jojhkQ0REpG0aGrJRR9WqVdG4cWOlMnNzc1SrVk1RPnLkSAQGBsLa2hoSiQTjx4+HTCZD27ZtAQA9e/aEu7s7hg8fjsWLFyMtLQ0zZsxAQECAomfG398f3333HaZPn47PPvsMR44cwfbt2xEZGalWvExIiIiI3lHLly+HgYEBBg0ahIKCAnh6emL16tWK81WqVMG+ffswduxYyGQymJubw9fXFyEhIYo6zs7OiIyMxOTJk7Fy5UrUrFkT69atU2sPEoD7kBC9U7gPCVFZb2IfksI/L2ukHeNazTTSjj5iDwkREZG2yUt0HYHe46RWIiIi0jn2kBAREWmbmitk3kVMSIiIiLRNB6ts3jYcsiEiIiKdYw8JERGRlqm7qdm7iAkJERGRtnHIRiUmJERERNrGHhKVOIeEiIiIdI49JERERNrGjdFUYkJCRESkbRyyUYlDNkRERKRz7CEhIiLSNq6yUYkJCRERkbZxyEYlDtkQERGRzrGHhIiISNs4ZKMSExIiIiItEwQu+1WFQzZERESkc+whISIi0jZOalWJCQkREZG2cQ6JSkxIiIiItI09JCpxDgkRERHpHHtIiIiItI0v11OJCQkREZG2cchGJQ7ZEBERkc6xh4SIiEjbuMpGJSYkRERE2sYhG5U4ZENEREQ6xx4SIiIibeOQjUpMSIiIiLSNCYlKHLIhIiIinWMPCRERkZYJAjdGU4UJCRERkbZxyEYlJiRERETaxmW/KnEOCREREekce0iIiIi0jUM2KjEhISIi0jYO2ajEIRsiIqJKaM2aNWjatCkkEgkkEglkMhkOHDigON+lSxeIRCKlw9/fX6mNlJQUeHl5wczMDLa2tpg2bRqKi4uV6sTGxsLDwwNisRiurq4IDw9/pXjZQ0JERKRtOhiyqVmzJhYtWoR69epBEARs3LgR/fv3x++//45GjRoBAEaPHo2QkBDFNWZmZoqfS0pK4OXlBXt7e5w+fRqpqanw8fGBkZERFi5cCABITk6Gl5cX/P39ERERgZiYGIwaNQoODg7w9PRUK16RIAiCBp5brxTevaDrEIj0kk2jwboOgUjvZD+/o/V75EV9p5F2TD3Hvdb11tbWWLJkCUaOHIkuXbqgefPmWLFiRbl1Dxw4gL59++LBgwews7MDAISFhSEoKAgPHz6EsbExgoKCEBkZiatXryquGzJkCDIzM3Hw4EG1YuOQDRERUSVXUlKCn3/+Gc+fP4dMJlOUR0REwMbGBo0bN0ZwcDByc3MV5+Li4tCkSRNFMgIAnp6eyM7OxrVr1xR1unfvrnQvT09PxMXFqR0jh2yIiIi0TUNDNgUFBSgoKFAqE4vFEIvF5da/cuUKZDIZ8vPzYWFhgZ07d8Ld3R0AMGzYMDg5OcHR0REJCQkICgpCYmIifvvtNwBAWlqaUjICQPE5LS3tX+tkZ2cjLy8PpqamFX42JiRERETapqGEJDQ0FHPnzlUqmz17NubMmVNu/QYNGiA+Ph5ZWVn49ddf4evri2PHjsHd3R1jxoxR1GvSpAkcHBzQrVs33L59G3Xr1tVIvOpgQkJERPSWCA4ORmBgoFLZy3pHAMDY2Biurq4AgJYtW+L8+fNYuXIl1q5dW6ZumzZtAAC3bt1C3bp1YW9vj3PnzinVSU9PBwDY29sr/m9p2d/rSCQStXpHAM4hISIi0j5BrpFDLBYrlvGWHv+WkPyTXC4vM+RTKj4+HgDg4OAAAJDJZLhy5QoyMjIUdaKjoyGRSBTDPjKZDDExMUrtREdHK81TqSj2kBAREWmbDpb9BgcHo3fv3qhduzaePXuGLVu2IDY2FlFRUbh9+za2bNmCPn36oFq1akhISMDkyZPRqVMnNG3aFADQs2dPuLu7Y/jw4Vi8eDHS0tIwY8YMBAQEKJIgf39/fPfdd5g+fTo+++wzHDlyBNu3b0dkZKTa8TIhISIi0jYd7NSakZEBHx8fpKamQiqVomnTpoiKikKPHj3w559/4vDhw1ixYgWeP3+OWrVqYdCgQZgxY4bi+ipVqmDfvn0YO3YsZDIZzM3N4evrq7RvibOzMyIjIzF58mSsXLkSNWvWxLp169TegwTgPiRE7xTuQ0JU1hvZh2T3Yo20Y9p/ukba0UfsISEiItI2vlxPJSYkRERE2saX66nEVTZERESkc+whISIi0jYO2ajEhISIiEjbmJCoxCEbIiIi0jn2kBAREWlb5dthQ+OYkBAREWkbh2xU4pANERER6Rx7SIiIiLSNPSQqMSEhIiLSNm6MppLeDNmcOHECn376KWQyGe7fvw8A2LRpE06ePKnjyIiIiF6TXK6ZoxLTi4Rkx44d8PT0hKmpKX7//XcUFBQAALKysrBw4UIdR0dERETaphcJyfz58xEWFoYff/wRRkZGivL27dvj0qVLOoyMiIhIAwRBM0clphdzSBITE9GpU6cy5VKpFJmZmW8+ICIiIk2q5MMtmqAXPST29va4detWmfKTJ0/CxcVFBxERERHRm6QXCcno0aMxceJEnD17FiKRCA8ePEBERASmTp2KsWPH6jo8IiKi18NJrSrpxZDNF198Ablcjm7duiE3NxedOnWCWCzG1KlTMX78eF2HR0RE9Hq47FclvUhIRCIRvvrqK0ybNg23bt1CTk4O3N3dYWFhoevQiIiI6A3Qi4Rk8+bNGDhwIMzMzODu7q7rcIiIiDRKkFfuFTKaoBdzSCZPngxbW1sMGzYM+/fvR0lJia5DIiIi0hzOIVFJLxKS1NRU/PzzzxCJRBg8eDAcHBwQEBCA06dP6zo0IiIiegP0IiExNDRE3759ERERgYyMDCxfvhx3795F165dUbduXV2HR0RE9HoEuWaOSkwv5pD8nZmZGTw9PfH06VPcu3cPN27c0HVIREREr4dzSFTSm4QkNzcXO3fuREREBGJiYlCrVi0MHToUv/76q65DIyIiej2VfP6HJuhFQjJkyBDs27cPZmZmGDx4MGbOnAmZTKbrsIiIiOgN0YuEpEqVKti+fTs8PT1RpUoVXYdDRESkWewhUUkvEpKIiAhdh0BERKQ9lfxNvZqgs4Rk1apVGDNmDExMTLBq1ap/rTthwoQ3FBURERHpgkgQdJO2OTs748KFC6hWrRqcnZ1fWk8kEuHOnTtqtV1498Lrhkf/b9vew9gWeRgP0h8CAOo61YS/94fo2Lo5AODPB+n45sct+P1aIgqLitC+ZTMEB/jCxkqqaCMrOwcLV2/EsbOXYCAyQPcOrfHFWB+YmZoAAFZv2oE1m38rc29TsRjn9vxX+w/5DrFpNFjXIVRaI0d5Y+Rob9SuXQMAcPNGEr5e9C2iDx0DAEQe2IKOndoqXbN+3RZMnjhD8dnDoynmzJuO5s0bQxAEXLp4GTNnLMLVKzff3IO8g7Kfq/dvzKvIXTZaI+2YBf6okXb0kc4SEm1iQqI5sWcuwcDAAE417CEIAvZEn8CGX/fhl+8XwtHeBoP8g9HApTb+M3wQAOC7jb/i4eOniFg5FwYGL7a58f/qazx6kolZE0eiuLgEM5euRaP6LlgcPA4AkJuXj9y8fKX7jgpaiEYNXLBgqv+bfeBKjgmJ9vTq/T7kcjlu37oLkUiEod4DMXHSaHRo9wFu3khC5IEtuHUrGQvmL1dck5ebj2fPcgAA5uZmuHbjBPbvj8GypWtgaGiIL7+aBFm7VnCr3x7FxcW6erRK740kJN+M0kg7ZlPXaaQdfaQXG6OFhIQgNze3THleXh5CQkJ0EBGV6tLWA53eaw6nGvaoU9MBE0YMhpmJCRJu3kL8tT/wIP0h5k/5HPWda6O+c20smOaPa0nJOBt/HQBwJ+U+Tl1IwNzJo9G0oSs8GjdA8H98cfDYGWQ8fgoAMDM1gY21peJ4/DQLt1PuY6BnFx0+OZF6Dh44gkNRsbh9+y5u3UrGvLlL8TwnF61bt1DUycvLR0b6I8VRmowAQP36dWFdzQoL5i3HraRk3LyRhEWhq2BnV13R60JUmelFQjJ37lzk5OSUKc/NzcXcuXN1EBGVp6REjgOxccgrKEAzN1cUFhVDBBGMjYwUdcRGRjAQifD7tUQAwOUbSahqYYZG9V0Uddp6NIaBSIQrN2+Ve58dB2NRp6YDWjZpqN0HItISAwMDDPqoL8zMTXHu3CVF+eDB/ZB87wLOnD+A2XOnwfT/hy0BICnpDh4/egIf38EwMjKCiYkYPr6DcfNGEu7d+0sXj0GaxJ1aVdKLVTaCIEAkEpUpv3z5MqytrXUQEf3dH8kp+HTSHBQWFsHM1AQrZk1GXaeasJJKYGoixvL1P2PCiMEQIGDF+m0okcvx8EkmAODRkyxUs5QqtWdYpQqkVS3w6ElWmXsVFBYi8sgpjPzkgzfxaEQa5d6oAQ4f+RUmJmLk5OTCe+hYJP5/4v3L9j3488/7SE3NQOPGDTF33nTUq+eCT4eNBQDk5DxHn97DsPXnMEz/4sVw5u1bd/Fhf1++cLQy4E6tKuk0IbGysoJIJIJIJEL9+vWVkpKSkhLk5OTA3//f5xAUFBSgoKBAqUxUUAix2FgrMb+LnGs64tfVC/EsNw/RJ85ixjdh2LBkBuo61cTSGRMw79sNiNgdBQORCL27yuDmWgcG5SSYFRFz6gJy8/LRr0cnDT8FkfYl/XEHHWR9IZFURf8PeyNs7RL07jUUiTdvIXzDz4p6168lIi0tA/v2R8DZuTaSk1NgYiLG96sX4cyZi/jMbxKqVDHA+Imj8ctv69Gl4wDk5xf8y52J3n46TUhWrFgBQRDw2WefYe7cuZBK//ebtLGxMerUqaNyx9bQ0NAywzozJo7GzEljtBLzu8jIyBC1a9gDABrVc8bVxDvYvCsKsyeORLuWTXEgfDmeZj1DlSoGkFiYo8uQ/6Cmgy0AwMZaiseZyj0hxSUlyHqWAxtraZl7/XYwFp3atFBapUP0tigqKsKdO/cAAPHxV+HRsinG/scPkybMKFP3wvl4AIBLXSckJ6fg48H9UdupJrp1HYTStQYjR0xCyv3f4dW3B3b8uu+NPQdpnsCN0VTSaULi6+sL4MUS4Hbt2sHob3MRKio4OBiBgYFKZaLUqxqJj8onCAIKi4qUyqykVQEAZ+Ov4UlmNrq09QAANHOrh2c5ubiWlIxG9V4s7z4Xfw1yQUCThq5KbfyVloFzl6/j2znK//8kelsZGIhe2lvbpKk7ACAt7cWSejMzE8jlcvx94WPp59IVa/QW45CNSjpLSLKzsyGRSAAALVq0QF5eHvLy8sqtW1qvPGKxGGKxWKms8AmHazRlxX9/RofWzeBQ3QbP8/Kw/+hpnE+4gbAFQQCAnVHH4FLbEdZSCeJvJOHrNZsw/MNecK7lCABwqV0D7Vs1xdwV6zBz/GcoLinGwu83olfntrCtZqV0r51Rx1Dd2hId/n+PE6K3yey50xB9KBZ//fkAFlUt8PHgfujYsS0+7O8HZ+fa+HhwPxyKisWTJ0/RqHFDLPp6Bk6eOItrV1/sMXLkyEnMWxCMZctDEBa2EQYGBgic4o/i4hIcPx6n46ej11bJJ6Rqgs7SbisrK2RkZAAALC0tYWVlVeYoLSfdeZKZja+WhOGDUVMxOigUVxPvIGxBENq1bAIAuPtXKibOXY5+o6dhbcROjB7aH1PHeCu18XVQAOrUdMCoLxbiPzOWoEWjBpgzSXlNvlwux+5Dx9G/RydUqcLfBuntU716Naz9cSkuxh/G3shN8GjZFB/298PRIydRWFiELl3bY9eejbjw+2EsDP0Ku3cfxCcf/2+zrKQ/7uCTj0ejUeOGOHxkB6IObYO9gx0GDfBD+v/3ohCpY82aNWjatCkkEgkkEglkMhkOHDigOJ+fn4+AgABUq1YNFhYWGDRoENLT05XaSElJgZeXF8zMzGBra4tp06aV2RMnNjYWHh4eEIvFcHV1RXh4+CvFq7ON0Y4dO4b27dvD0NAQx44d+9e6nTt3VqttboxGVD5ujEZU1pvYGO15iLfqShVgPqvi737bu3cvqlSpgnr16kEQBGzcuBFLlizB77//jkaNGmHs2LGIjIxEeHg4pFIpxo0bBwMDA5w6dQrAi8UlzZs3h729PZYsWYLU1FT4+Phg9OjRWLhwIQAgOTkZjRs3hr+/P0aNGoWYmBhMmjQJkZGR8PT0VOvZuFMr0TuECQlRWW8kIZkzVCPtmM/Z+lrXW1tbY8mSJfjoo49QvXp1bNmyBR999BEA4ObNm3Bzc0NcXBzatm2LAwcOoG/fvnjw4AHs7OwAAGFhYQgKCsLDhw9hbGyMoKAgREZG4urV/83dHDJkCDIzM3Hw4EG1YtOLvvGDBw/i5MmTis/ff/89mjdvjmHDhuHp06c6jIyIiEh/FBQUIDs7W+n459YX5SkpKcHPP/+M58+fQyaT4eLFiygqKkL37t0VdRo2bIjatWsjLu7FnKW4uDg0adJEkYwAgKenJ7Kzs3Ht2jVFnb+3UVqntA116EVCMm3aNGRnZwMArly5gsDAQPTp0wfJycllVtAQERG9deSCRo7Q0FBIpVKlIzQ09KW3vXLlCiwsLCAWi+Hv74+dO3fC3d0daWlpMDY2hqWlpVJ9Ozs7pKWlAQDS0tKUkpHS86Xn/q1Odnb2SxeqvIxe7NSanJwMd/cXS+B27NiBDz74AAsXLsSlS5fQp08fHUdHRET0mjS0yqa8rS7+udL07xo0aID4+HhkZWXh119/ha+vr8p5m7qiFwmJsbGx4uV6hw8fho+PD4AXY12lPSdERETvuvK2uvg3xsbGcHV9sedTy5Ytcf78eaxcuRKffPIJCgsLkZmZqdRLkp6eDnv7Fxth2tvb49y5c0rtla7C+Xudf67MSU9Ph0QigampqVrPphdDNh06dEBgYCDmzZuHc+fOwcvLCwDwxx9/oGbNmjqOjoiI6DVpaMjmtcOQy1FQUICWLVvCyMgIMTExinOJiYlISUlR7JAuk8lw5coVxRYdABAdHQ2JRKIY1ZDJZEptlNZRtct6efQiIfnuu+9gaGiIX3/9FWvWrEGNGi9etX3gwAH06tVLx9ERERG9HkEu18ihjuDgYBw/fhx3797FlStXEBwcjNjYWHh7e0MqlWLkyJEIDAzE0aNHcfHiRYwYMQIymQxt27YFAPTs2RPu7u4YPnw4Ll++jKioKMyYMQMBAQGKXhp/f3/cuXMH06dPx82bN7F69Wps374dkydPVvs70oshm9q1a2PfvrLvaVi+fLkOoiEiInr7ZWRkwMfHB6mpqZBKpWjatCmioqLQo0cPAC/+jTUwMMCgQYNQUFAAT09PrF69WnF9lSpVsG/fPowdOxYymQzm5ubw9fVFSEiIoo6zszMiIyMxefJkrFy5EjVr1sS6devU3oME0KN9SEpKSrBr1y7cuHEDANCoUSP069cPVapUUbst7kNCVD7uQ0JU1pvYhyQnaKBG2rH4+jeNtKOP9KKH5NatW+jTpw/u37+PBg0aAHjxFt9atWohMjISdevW1XGEREREr4Ev11NJL+aQTJgwAXXr1sWff/6JS5cu4dKlS0hJSYGzszMmTJig6/CIiIhejyDXzFGJ6UUPybFjx3DmzBlYW1sryqpVq4ZFixahffv2OoyMiIiI3gS9SEjEYjGePXtWpjwnJwfGxsY6iIiIiEiDOGSjkl4M2fTt2xdjxozB2bNnIQgCBEHAmTNn4O/vj379+uk6PCIiotciyAWNHJWZXiQkq1atQt26dSGTyWBiYgITExO0a9cOrq6uWLlypa7DIyIiIi3TiyEbS0tL7N69G7du3cL169cBAO7u7ortbomIiN5qlbx3QxP0IiEBgPXr12P58uVISkoCANSrVw+TJk3CqFGjdBwZERHRa1Jzl9V3kV4kJLNmzcKyZcswfvx4xf73cXFxmDx5MlJSUpR2hSMiIqLKRy8SkjVr1uDHH3/E0KFDFWX9+vVD06ZNMX78eCYkRET0duOQjUp6kZAUFRWhVatWZcpbtmyJ4uJiHURERESkQUxIVNKLVTbDhw/HmjVrypT/8MMP8Pb21kFERERE9CbpRQ8J8GJS66FDhxSvPT579ixSUlLg4+ODwMBARb1ly5bpKkQiIqJXoifvsdVrepGQXL16FR4eHgCA27dvAwBsbGxgY2ODq1evKuqJRCKdxEdERPRaOGSjkl4kJEePHtV1CERERNrDhEQlvZhDQkRERO82veghISIiqswq+3toNIEJCRERkbYxIVGJQzZERESkc+whISIi0ja+ykYlJiRERERaxjkkqnHIhoiIiHSOPSRERETaxh4SlZiQEBERaRvnkKjEIRsiIiLSOfaQEBERaRkntarGhISIiEjbOGSjEhMSIiIiLWMPiWqcQ0JEREQ6xx4SIiIibeOQjUpMSIiIiLRMYEKiEodsiIiISOfYQ0JERKRt7CFRiQkJERGRlnHIRjUO2RAREZHOsYeEiIhI29hDohITEiIiIi3jkI1qTEiIiIi0jAmJapxDQkREVAmFhoaidevWqFq1KmxtbTFgwAAkJiYq1enSpQtEIpHS4e/vr1QnJSUFXl5eMDMzg62tLaZNm4bi4mKlOrGxsfDw8IBYLIarqyvCw8PVjpcJCRERkZYJcs0c6jh27BgCAgJw5swZREdHo6ioCD179sTz58+V6o0ePRqpqamKY/HixYpzJSUl8PLyQmFhIU6fPo2NGzciPDwcs2bNUtRJTk6Gl5cXunbtivj4eEyaNAmjRo1CVFSUWvGKBEGodG/8Kbx7QdchEOklm0aDdR0Ckd7Jfn5H6/dI79JFI+3Yxca+8rUPHz6Era0tjh07hk6dOgF40UPSvHlzrFixotxrDhw4gL59++LBgwews7MDAISFhSEoKAgPHz6EsbExgoKCEBkZiatXryquGzJkCDIzM3Hw4MEKx8ceEiIiordEQUEBsrOzlY6CgoIKXZuVlQUAsLa2ViqPiIiAjY0NGjdujODgYOTm5irOxcXFoUmTJopkBAA8PT2RnZ2Na9euKep0795dqU1PT0/ExcWp9WxMSIiIiLRMU0M2oaGhkEqlSkdoaKjK+8vlckyaNAnt27dH48aNFeXDhg3D5s2bcfToUQQHB2PTpk349NNPFefT0tKUkhEAis9paWn/Wic7Oxt5eXkV/o64yoaIiEjLBLlII+0EBwcjMDBQqUwsFqu8LiAgAFevXsXJkyeVyseMGaP4uUmTJnBwcEC3bt1w+/Zt1K1bVyMxVxR7SIiIiN4SYrEYEolE6VCVkIwbNw779u3D0aNHUbNmzX+t26ZNGwDArVu3AAD29vZIT09XqlP62d7e/l/rSCQSmJqaVvjZmJAQERFpmS5W2QiCgHHjxmHnzp04cuQInJ2dVV4THx8PAHBwcAAAyGQyXLlyBRkZGYo60dHRkEgkcHd3V9SJiYlRaic6OhoymUyteJmQEBERaZkgiDRyqCMgIACbN2/Gli1bULVqVaSlpSEtLU0xr+P27duYN28eLl68iLt372LPnj3w8fFBp06d0LRpUwBAz5494e7ujuHDh+Py5cuIiorCjBkzEBAQoOiZ8ff3x507dzB9+nTcvHkTq1evxvbt2zF58mS14uWyX6J3CJf9EpX1Jpb93pe9r5F2asQdqXBdkaj8BGbDhg3w8/PDn3/+iU8//RRXr17F8+fPUatWLXz44YeYMWMGJBKJov69e/cwduxYxMbGwtzcHL6+vli0aBEMDf83DTU2NhaTJ0/G9evXUbNmTcycORN+fn5qPRsTEqJ3CBMSorLeRELyVxvNJCQ1z1Y8IXnbcJUNERGRlmlqlU1lxoSEiIhIyyrfWITmcVIrERER6Rx7SIiIiLSMQzaqMSEhIiLSMiYkqnHIhoiIiHSOPSRERERaxkmtqjEhISIi0jIO2ajGIRsiIiLSObUTEhcXFzx+/LhMeWZmJlxcXDQSFBERUWWii3fZvG3UHrK5e/cuSkpKypQXFBTg/v37GgmKiIioMlH3Tb3vogonJHv27FH8HBUVBalUqvhcUlKCmJgY1KlTR6PBERER0buhwgnJgAEDALx4e6Cvr6/SOSMjI9SpUwdLly7VaHBERESVgbySD7doQoUTErn8RX+Ts7Mzzp8/DxsbG60FRUREVJlU9vkfmqD2HJLk5GTFz/n5+TAxMdFoQERERJUNl/2qpvYqG7lcjnnz5qFGjRqwsLDAnTt3AAAzZ87E+vXrNR4gERERVX5qJyTz589HeHg4Fi9eDGNjY0V548aNsW7dOo0GR0REVBkIgmaOykzthOSnn37CDz/8AG9vb1SpUkVR3qxZM9y8eVOjwREREVUGglykkaMyUzshuX//PlxdXcuUy+VyFBUVaSQoIiIiereonZC4u7vjxIkTZcp//fVXtGjRQiNBERERVSZyQaSRozJTe5XNrFmz4Ovri/v370Mul+O3335DYmIifvrpJ+zbt08bMRIREb3VuOxXNbV7SPr374+9e/fi8OHDMDc3x6xZs3Djxg3s3bsXPXr00EaMREREVMmp3UMCAB07dkR0dLSmYyEiIqqUKvsKGU14pYSEiIiIKq6yz//QBLUTEisrK4hEZb9YkUgEExMTuLq6ws/PDyNGjNBIgERERFT5vdKk1gULFqB379547733AADnzp3DwYMHERAQgOTkZIwdOxbFxcUYPXq0xgMmIiJ623BSq2pqJyQnT57E/Pnz4e/vr1S+du1aHDp0CDt27EDTpk2xatUqJiRERETgHJKKUHuVTVRUFLp3716mvFu3boiKigIA9OnTR/GOGyIioncd9yFRTe2ExNraGnv37i1TvnfvXlhbWwMAnj9/jqpVq75+dERERPROUHvIZubMmRg7diyOHj2qmENy/vx57N+/H2FhYQCA6OhodO7cWbORqsGsfn+d3ZtIn+U9KLvLMhFpH+eQqCYSBPVHtk6dOoXvvvsOiYmJAIAGDRpg/PjxaNeuncYDfBWGxjV0HQKRXmJCQlSWkY2L1u9x1nGgRtpp8+A3jbSjj9TqISkqKsLnn3+OmTNnYuvWrdqKiYiIiN4xas0hMTIywo4dO7QVCxERUaUkaOiozNSe1DpgwADs2rVLC6EQERFVTlxlo5rak1rr1auHkJAQnDp1Ci1btoS5ubnS+QkTJmgsOCIiIno3qD2p1dnZ+eWNiUR6sf8IJ7USlY+TWonKehOTWk/Zf6SRdtqn/aqRdvSR2j0kycnJ2oiDiIio0pLrOoC3gNpzSIiIiIg07ZUSkr/++gurV6/GF198gcDAQKWDiIiIlAkQaeRQR2hoKFq3bo2qVavC1tYWAwYMUOwfVio/Px8BAQGoVq0aLCwsMGjQIKSnpyvVSUlJgZeXF8zMzGBra4tp06ahuLhYqU5sbCw8PDwgFovh6uqK8PBwtb8jtYdsYmJi0K9fP7i4uODmzZto3Lgx7t69C0EQ4OHhoXYARERElZ1cB2t2jx07hoCAALRu3RrFxcX48ssv0bNnT1y/fl2xIGXy5MmIjIzEL7/8AqlUinHjxmHgwIE4deoUAKCkpAReXl6wt7fH6dOnkZqaCh8fHxgZGWHhwoUAXkzl8PLygr+/PyIiIhATE4NRo0bBwcEBnp6eFY5X7Umt7733Hnr37o25c+eiatWquHz5MmxtbeHt7Y1evXph7Nix6jSnFZzUSlQ+TmolKutNTGo9YjdYI+28n779la99+PAhbG1tcezYMXTq1AlZWVmoXr06tmzZgo8+ejHp9ubNm3Bzc0NcXBzatm2LAwcOoG/fvnjw4AHs7OwAAGFhYQgKCsLDhw9hbGyMoKAgREZG4urVq4p7DRkyBJmZmTh48GCF41N7yObGjRvw8fEBABgaGiIvLw8WFhYICQnB119/rW5zREREVEEFBQXIzs5WOgoKCip0bVZWFgAoXoR78eJFFBUVoXv37oo6DRs2RO3atREXFwcAiIuLQ5MmTRTJCAB4enoiOzsb165dU9T5exuldUrbqCi1ExJzc3MUFhYCABwcHHD79m3FuUePHqnbHBERUaWnqTkkoaGhkEqlSkdoaKjK+8vlckyaNAnt27dH48aNAQBpaWkwNjaGpaWlUl07OzukpaUp6vw9GSk9X3ru3+pkZ2cjLy+vwt9RheeQhISEYMqUKWjbti1OnjwJNzc39OnTB1OmTMGVK1fw22+/oW3bthW+MRER0btCU8t+g4ODyywgEYvFKq8LCAjA1atXcfLkSQ1FonkV7iGZO3cunj9/jmXLlqFNmzaKsm7dumHbtm2oU6cO1q9fr7VAiYiI3nVisRgSiUTpUJWQjBs3Dvv27cPRo0dRs2ZNRbm9vT0KCwuRmZmpVD89PR329vaKOv9cdVP6WVUdiUQCU1PTCj9bhXtISue+urj8b/KPubk5wsLCKnwzIiKid5G6S3Y1ck9BwPjx47Fz507ExsaW2Wm9ZcuWMDIyQkxMDAYNGgQASExMREpKCmQyGQBAJpNhwYIFyMjIgK2tLQAgOjoaEokE7u7uijr79+9Xajs6OlrRRkWptexXJKrcL/YhIiLSBl3s1BoQEIAtW7Zg9+7dqFq1qmLOh1QqhampKaRSKUaOHInAwEBYW1tDIpFg/PjxkMlkiikYPXv2hLu7O4YPH47FixcjLS0NM2bMQEBAgKJnxt/fH9999x2mT5+Ozz77DEeOHMH27dsRGRmpVrwVXvZrYGAAqVSqMil58uSJWgFoA5f9EpWPy36JynoTy34P2g3RSDu90n+ucN2X/Xu9YcMG+Pn5AXixMdqUKVOwdetWFBQUwNPTE6tXr1YMxwDAvXv3MHbsWMTGxsLc3By+vr5YtGgRDA3/16cRGxuLyZMn4/r166hZsyZmzpypuEeF41UnIVmxYgWkUum/1vP19VUrAG1gQkJUPiYkRGW9iYRkv4YSkj5qJCRvG7WGbIYMGaIYQyIiIqKK0cUckrdNhVfZcP4IERERaYvaq2yIiIhIPXL+Tq9ShRMSuVwXc4SJiIjefnIO2aik9tt+iYiISD0cY1BN7XfZEBEREWkae0iIiIi0jJMeVGNCQkREpGVyrlRViUM2REREpHPsISEiItIyTmpVjQkJERGRlnEOiWocsiEiIiKdYw8JERGRlnGnVtWYkBAREWkZd2pVjUM2REREpHPsISEiItIyrrJRjQkJERGRlnEOiWpMSIiIiLSMy35V4xwSIiIi0jn2kBAREWkZ55CoxoSEiIhIyziHRDUO2RAREZHOsYeEiIhIyzipVTUmJERERFrGhEQ1DtkQERGRzrGHhIiISMsETmpViQkJERGRlnHIRjUO2RAREZHOsYeEiIhIy9hDohoTEiIiIi3jTq2qMSEhIiLSMu7UqhrnkBAREZHOsYeEiIhIyziHRDUmJERERFrGhEQ1DtkQERGRzrGHhIiISMu4ykY1JiRERERaxlU2qnHIhoiIqJI6fvw4PvjgAzg6OkIkEmHXrl1K5/38/CASiZSOXr16KdV58uQJvL29IZFIYGlpiZEjRyInJ0epTkJCAjp27AgTExPUqlULixcvVjtWJiRERERaJtfQoa7nz5+jWbNm+P77719ap1evXkhNTVUcW7duVTrv7e2Na9euITo6Gvv27cPx48cxZswYxfns7Gz07NkTTk5OuHjxIpYsWYI5c+bghx9+UCtWDtkQERFpma7mkPTu3Ru9e/f+1zpisRj29vblnrtx4wYOHjyI8+fPo1WrVgCAb7/9Fn369ME333wDR0dHREREoLCwEP/9739hbGyMRo0aIT4+HsuWLVNKXFRhDwkREdE7LDY2Fra2tmjQoAHGjh2Lx48fK87FxcXB0tJSkYwAQPfu3WFgYICzZ88q6nTq1AnGxsaKOp6enkhMTMTTp08rHAd7SIiIiLRMrqE+koKCAhQUFCiVicViiMXiV2qvV69eGDhwIJydnXH79m18+eWX6N27N+Li4lClShWkpaXB1tZW6RpDQ0NYW1sjLS0NAJCWlgZnZ2elOnZ2dopzVlZWFYqFPSRERERapqk5JKGhoZBKpUpHaGjoK8c1ZMgQ9OvXD02aNMGAAQOwb98+nD9/HrGxsa/c5qtiQkJERKRlgoaO4OBgZGVlKR3BwcEai9PFxQU2Nja4desWAMDe3h4ZGRlKdYqLi/HkyRPFvBN7e3ukp6cr1Sn9/LK5KeVhQkJERPSWEIvFkEgkSserDteU56+//sLjx4/h4OAAAJDJZMjMzMTFixcVdY4cOQK5XI42bdoo6hw/fhxFRUWKOtHR0WjQoEGFh2sAPUtICgsLkZiYiOLiYl2HQkREpDG6Wvabk5OD+Ph4xMfHAwCSk5MRHx+PlJQU5OTkYNq0aThz5gzu3r2LmJgY9O/fH66urvD09AQAuLm5oVevXhg9ejTOnTuHU6dOYdy4cRgyZAgcHR0BAMOGDYOxsTFGjhyJa9euYdu2bVi5ciUCAwPVilUvEpLc3FyMHDkSZmZmaNSoEVJSUgAA48ePx6JFi3QcHRER0euRizRzqOvChQto0aIFWrRoAQAIDAxEixYtMGvWLFSpUgUJCQno168f6tevj5EjR6Jly5Y4ceKEUq9LREQEGjZsiG7duqFPnz7o0KGD0h4jUqkUhw4dQnJyMlq2bIkpU6Zg1qxZai35BQCRIAg632J/4sSJOHXqFFasWIFevXohISEBLi4u2L17N+bMmYPff/9drfYMjWtoKVKit1vegxO6DoFI7xjZuGj9HrPqeGuknZC7ERppRx/pxbLfXbt2Ydu2bWjbti1Eov+lgI0aNcLt27d1GBkREdHr09Sy38pMLxKShw8fllnnDLzY8vbvCQoREdHbiOmIanoxh6RVq1aIjIxUfC5NQtatWweZTKarsIiIiOgN0YsekoULF6J37964fv06iouLsXLlSly/fh2nT5/GsWPHdB0eERHRa3mVFTLvGr3oIenQoQPi4+NRXFyMJk2a4NChQ7C1tUVcXBxatmyp6/CIiIheixyCRo7KTC96SACgbt26+PHHH3UdBhEREemAXvSQdO/eHeHh4cjOztZ1KERERBqnqa3jKzO9SEgaNWqE4OBg2Nvb4+OPP8bu3buVtqAlIiJ6m+lqp9a3iV4kJCtXrsT9+/exa9cumJubw8fHB3Z2dhgzZgwntRIR0VuPc0hU04uEBAAMDAzQs2dPhIeHIz09HWvXrsW5c+fw/vvv6zo0IiIi0jK9mdRaKi0tDT///DM2b96MhIQEvPfee7oOiYiI6LVU7r4NzdCLHpLs7Gxs2LABPXr0QK1atbBmzRr069cPSUlJOHPmjK7DIyIiei2cQ6KaXvSQ2NnZwcrKCp988glCQ0PRqlUrXYdEREREb5BeJCR79uxBt27dYGCgFx02REREGiVw0EYlvUhIevTooesQiIiItKayD7dogs4SEg8PD8TExMDKygotWrT417f6Xrp06Q1GRkRERG+azhKS/v37QywWK37+t4SEiIjobVbZ9xDRBJEgCJXuWzI0rqHrEIj0Ut6DE7oOgUjvGNm4aP0eY+sM1kg7a+5u10g7+kgvZpG6uLjg8ePHZcozMzPh4qL9PyhERESkW3qRkNy9exclJSVlygsKCvDXX3/pICJ6maDp4xB3OhJPHyfiwV+XsePX9ahfv65SndXff43EG6fwLOsWUu8n4Lcd/0WDBsp1igvvlzkGD+73Jh+FSGPWbdqOxu17Y9GKMEVZQUEh5i/9Hu17D0br7h9i0pfz8ejJU8X5zKxsfB44A137eaNFlw/Q7cPhWLB0NXKeP1dq+9ylBHw8YhxadPkAvQd/hl2R0W/suUhzuHW8ajpdZbNnzx7Fz1FRUZBKpYrPJSUliImJgbOzsy5Co5fo1LEt1qzZiAsX42FoaIj5IV/gQOQWNGnWBbm5eQCAS5cSsHXrb0j58z6srSwxa9YUHIjcCtf6bSGX/2+u+WcjJyPq0FHF58xMvu2Z3j5XbiTil937Ud9V+b9VX69ai+Nx57Fs/pewMDfHwmWrMenL+dgcthQAIBKJ0LVjW4wf7QNrKylS/nqABUtXI2vJMyyeEwQA+OtBGgKmzcLgAV5YNHs6zl6Ix+yvV6C6jTXat2n5xp+VXh1X2aim04RkwIABAF78xfT19VU6Z2RkhDp16mDp0qU6iIxexuuDT5U+fzZqEtIeXEFLj6Y4cfIsAGDd+gjF+Xv3/sKs2Yvx+8XDqFOnFu7cuac4l5mVhfT0h28mcCItyM3Nwxdzl2BO0ESs3bhVUf4s5zl+23cIi+dMR5uWzQEA874KRL9hY3D56g00a+wGqaQqhnzYV3GNo70dPhnYFxu2/Koo274rEjUc7DFt/GgAQN06tXEp4Rp+2raTCclbhvuQqKbTIRu5XA65XI7atWsjIyND8Vkul6OgoACJiYno27ev6oZIZ6RSCQDgydPMcs+bmZnCz+cT3LlzD3/++UDp3LcrFyDtwRXEndoHP99PtB0qkcbNX/o9OslaQ9a6hVL59cQkFBcXo22r/5W7ONWCg50tLl+9WW5bGQ8f4/CxU2jVvImi7PLVm2jbqrlSvfZtWuLy1RuaewgiPaEXG6MlJye/8rUFBQUoKChQKhMEgcuI3wCRSIRl38zFqVPncO1aotI5/899sSj0K1hYmONm4i306jMURUVFivOz5yzB0aMnkZuXhx7dO+O7bxfCwsIc333/3zf9GESvZP/hWNz44zZ+XreyzLlHj5/CyMgQkqoWSuXVrC3x6MkTpbJpsxfh6IkzyC8oQJf2bRDyxaT/tfPkKapZWym3YWWJnOe5yC8ogMn/b51A+o9DNqrpRUICAM+fP8exY8eQkpKCwsJCpXMTJkx46XWhoaGYO3euUpnIwAKiKhKtxEn/8+2qhWjUqAE6d/2wzLktW3/D4ZjjcLC3RWCgP7ZuCUOnzgMUyeOChSsUdePjr8Hc3AxTAscyIaG3Qmr6QyxasRY/rlgIsdj4tdoKmjAGYz/zxr2U+1gRtgGLv/0BM6eO01CkpC84ZKOaXiQkv//+O/r06YPc3Fw8f/4c1tbWePToEczMzGBra/uvCUlwcDACAwOVyqyqNdR2yO+8lSvmw6tPd3TtNhD376eWOZ+d/QzZ2c9w61Yyzpy9hEcZ1zFgQC9s27a73PbOnfsdM76aDGNj4zIJKZG+uZ6YhCdPMzH4s/8lDiUlclyMv4qtv+3F2mXzUVRUjOxnOUq9JI+fZMLG2lqpLZtq1rCpZg0Xp1qQSizg859p8Pcbhuo21rCxtsLjv63MAYDHTzNhYW7G3hGqdPQiIZk8eTI++OADhIWFQSqV4syZMzAyMsKnn36KiRMn/uu1YrFYseNrKQ7XaNfKFfMxoH8vdOvxMe7e/VNlfZFIBJFIBLHxy/8D2qxZIzx58pTJCL0V2rZsjp2b1iiVzViwDM5OtTDy049hb1sdhoaGOHshHj26dgAAJN/7C6npGWjW+OW/MMn/f5/Kwv8f3mzWuCFOxF1QqhN3/nc0a+ymycehN4BDNqrpRUISHx+PtWvXwsDAAFWqVEFBQQFcXFywePFi+Pr6YuDAgboOkf7ft6sWYuiQARg46DM8e5YDO7vqAICsrGfIz8+Hs3NtDP64H6Kjj+Hho8eoWcMR06cHIC8vHwcOxgAA+nr1gK2tDc6eu4T8/AJ079YJXwSNx7LlYf92ayK9YW5uhnoudZTKTE1NYCmpqigf2LcnFn/7I6SSqjA3N8PC5WvQrLGbIpk4fvocHj/NRGO3+jAzNcWt5HtY+v06tGjqjhoOdgCAwQO8sHXHXiz9fj0+7NsT5y5eRtSR41i9JORNPi5pgLzybYqucXqRkBgZGcHA4MWCH1tbW6SkpMDNzQ1SqRR//qn6N3B6c8b6v1iefSRmh1L5ZyMn46dN25GfX4AO7d/DhPGjYGUlRXr6I5w4eQYdO/fHw4cvduMtKirC2LF+WPrNHIhEIty6fRdTp81VWi5M9LYLmvA5DAwMMOmr+SgqKkK791pi5tQAxXkTsRi/7jmIxat+QGFhEeztqqN753YY+en/thiv6WiP75eEYPGqtdj8yy7YVbfB3KBJXPJLlZJevMumZ8+e8PPzw7BhwzB69GgkJCRgwoQJ2LRpE54+fYqzZ8+q1R7fZUNUPr7LhqisN/Eum0+dNNPTv/nebxppRx/pxdbxCxcuhIODAwBgwYIFsLKywtixY/Hw4UP88MMPOo6OiIjo9XDreNX0YsimVatWip9tbW1x8OBBHUZDREREb5peJCRERESVGfchUU0vEpIWLVqUu1RXJBLBxMQErq6u8PPzQ9euXXUQHRER0evhsl/V9GIOSa9evXDnzh2Ym5uja9eu6Nq1KywsLHD79m20bt0aqamp6N69O3bvLn9TLSIiIn3GOSSq6UUPyaNHjzBlyhTMnDlTqXz+/Pm4d+8eDh06hNmzZ2PevHno37+/jqIkIiIibdGLHpLt27dj6NChZcqHDBmC7du3AwCGDh2KxMTEMnWIiIj0naCh/1VmepGQmJiY4PTp02XKT58+DRMTEwCAXC5X/ExERPQ2kWvoqMz0IiEZP348/P39MXHiRGzevBmbN2/GxIkTMXbsWMWL9aKiotC8eXPdBkpERPQWOX78OD744AM4OjpCJBJh165dSucFQcCsWbPg4OAAU1NTdO/eHUlJSUp1njx5Am9vb0gkElhaWmLkyJHIyclRqpOQkICOHTvCxMQEtWrVwuLFi9WOVS8SkhkzZuDHH3/EuXPnMGHCBEyYMAHnzp3Djz/+iK+++goA4O/vj7179+o4UiIiIvUJgqCRQ13Pnz9Hs2bN8P3335d7fvHixVi1ahXCwsJw9uxZmJubw9PTE/n5+Yo63t7euHbtGqKjo7Fv3z4cP34cY8aMUZzPzs5Gz5494eTkhIsXL2LJkiWYM2eO2hub6sXW8ZrGreOJyset44nKehNbx/ev3Vcj7exO2ffK14pEIuzcuRMDBgwA8CJJcnR0xJQpUzB16lQAQFZWFuzs7BAeHo4hQ4bgxo0bcHd3x/nz5xWbmB48eBB9+vTBX3/9BUdHR6xZswZfffUV0tLSYGxsDAD44osvsGvXLty8ebPC8elFDwkAZGZmYt26dfjyyy/x5MkTAMClS5dw//59HUdGRESkHwoKCpCdna10FBQUvFJbycnJSEtLQ/fu3RVlUqkUbdq0QVxcHAAgLi4OlpaWSjuqd+/eHQYGBor3zMXFxaFTp06KZAQAPD09kZiYiKdPn1Y4Hr1ISBISElC/fn18/fXXWLJkCTIzMwEAv/32G4KDg3UbHBER0WvS1KTW0NBQSKVSpSM0NPSVYkpLSwMA2NnZKZXb2dkpzqWlpcHW1lbpvKGhIaytrZXqlNfG3+9REXqRkAQGBsLPzw9JSUlKK2n69OmD48eP6zAyIiKi16epZb/BwcHIyspSOirLL+56sTHa+fPnsXbt2jLlNWrUUCu7IiIiqszEYjHEYrFG2rK3twcApKenw8HBQVGenp6uWNVqb2+PjIwMpeuKi4vx5MkTxfX29vZIT09XqlP6ubRORehFD4lYLEZ2dnaZ8j/++APVq1fXQURERESao49bxzs7O8Pe3h4xMTGKsuzsbJw9exYymQwAIJPJkJmZiYsXLyrqHDlyBHK5HG3atFHUOX78OIqKihR1oqOj0aBBA1hZWVU4Hr1ISPr164eQkBDFw4hEIqSkpCAoKAiDBg3ScXRERESvR1fLfnNychAfH4/4+HgALyayxsfHIyUlBSKRCJMmTcL8+fOxZ88eXLlyBT4+PnB0dFSsxHFzc0OvXr0wevRonDt3DqdOncK4ceMwZMgQODo6AgCGDRsGY2NjjBw5EteuXcO2bduwcuVKBAYGqhWrXiz7zcrKwkcffYQLFy7g2bNncHR0RFpaGtq2bYsDBw7A3Nxcrfa47JeofFz2S1TWm1j261mrt0baifrzgFr1Y2Nj0bVr1zLlvr6+CA8PhyAImD17Nn744QdkZmaiQ4cOWL16NerXr6+o++TJE4wbNw579+6FgYEBBg0ahFWrVsHCwkJRJyEhAQEBATh//jxsbGwwfvx4BAUFqRWrXiQkpU6dOoXLly8jJycHHh4eSkuR1MGEhKh8TEiIyqrMCcnbRC8mtQJATEwMYmJikJGRAblcjps3b2LLli0AgP/+9786jo6IiOjVVfYX42mCXiQkc+fORUhICFq1agUHBweIRCJdh0RERKQxmp6QWhnpRUISFhaG8PBwDB8+XNehEBERkQ7oRUJSWFiIdu3a6ToMIiIirdCj6Zp6Sy+W/Y4aNUoxX4SIiKiy0cd9SPSNXvSQ5Ofn44cffsDhw4fRtGlTGBkZKZ1ftmyZjiIjIiKiN0EvEpKEhATFNrVXr15VOscJrkRE9LbjKhvV9CIhOXr0qK5DICIi0ho555CopBdzSIiIiOjdphc9JERERJUZ+0dUY0JCRESkZZV9hYwmMCEhIiLSMiYkqnEOCREREekce0iIiIi0jDu1qsaEhIiISMs4ZKMah2yIiIhI59hDQkREpGXcqVU1JiRERERaxjkkqnHIhoiIiHSOPSRERERaxkmtqjEhISIi0jIO2ajGIRsiIiLSOfaQEBERaRmHbFRjQkJERKRlXParGhMSIiIiLZNzDolKnENCREREOsceEiIiIi3jkI1qTEiIiIi0jEM2qnHIhoiIiHSOPSRERERaxiEb1ZiQEBERaRmHbFTjkA0RERHpHHtIiIiItIxDNqoxISEiItIyDtmoxiEbIiIi0jn2kBAREWkZh2xUY0JCRESkZYIg13UIeo8JCRERkZbJ2UOiEueQEBERVUJz5syBSCRSOho2bKg4n5+fj4CAAFSrVg0WFhYYNGgQ0tPTldpISUmBl5cXzMzMYGtri2nTpqG4uFgr8bKHhIiISMsEHa2yadSoEQ4fPqz4bGj4v3/2J0+ejMjISPzyyy+QSqUYN24cBg4ciFOnTgEASkpK4OXlBXt7e5w+fRqpqanw8fGBkZERFi5cqPFYmZAQERFpma6GbAwNDWFvb1+mPCsrC+vXr8eWLVvw/vvvAwA2bNgANzc3nDlzBm3btsWhQ4dw/fp1HD58GHZ2dmjevDnmzZuHoKAgzJkzB8bGxhqNlUM2REREb4mCggJkZ2crHQUFBS+tn5SUBEdHR7i4uMDb2xspKSkAgIsXL6KoqAjdu3dX1G3YsCFq166NuLg4AEBcXByaNGkCOzs7RR1PT09kZ2fj2rVrGn82JiRERERaJgiCRo7Q0FBIpVKlIzQ0tNx7tmnTBuHh4Th48CDWrFmD5ORkdOzYEc+ePUNaWhqMjY1haWmpdI2dnR3S0tIAAGlpaUrJSOn50nOaxiEbIiIiLdPUTq3BwcEIDAxUKhOLxeXW7d27t+Lnpk2bok2bNnBycsL27dthamqqkXg0iT0kREREbwmxWAyJRKJ0vCwh+SdLS0vUr18ft27dgr29PQoLC5GZmalUJz09XTHnxN7evsyqm9LP5c1LeV1MSIiIiLRM0ND/XkdOTg5u374NBwcHtGzZEkZGRoiJiVGcT0xMREpKCmQyGQBAJpPhypUryMjIUNSJjo6GRCKBu7v7a8VSHg7ZEBERaZkulv1OnToVH3zwAZycnPDgwQPMnj0bVapUwdChQyGVSjFy5EgEBgbC2toaEokE48ePh0wmQ9u2bQEAPXv2hLu7O4YPH47FixcjLS0NM2bMQEBAQIV7ZdTBhISIiKgS+uuvvzB06FA8fvwY1atXR4cOHXDmzBlUr14dALB8+XIYGBhg0KBBKCgogKenJ1avXq24vkqVKti3bx/Gjh0LmUwGc3Nz+Pr6IiQkRCvxigRd7daiRYbGNXQdApFeyntwQtchEOkdIxsXrd+jurSBRtp5mJWokXb0EXtIiIiItKwS/u6vcUxIiIiItExTy34rM66yISIiIp1jDwkREZGWcchGNSYkREREWqarl+u9TThkQ0RERDrHHhIiIiIt45CNakxIiIiItIyrbFTjkA0RERHpHHtIiIiItOx1X4z3LmBCQkREpGUcslGNQzZERESkc+whISIi0jKuslGNCQkREZGWcQ6JakxIiIiItIw9JKpxDgkRERHpHHtIiIiItIw9JKoxISEiItIypiOqcciGiIiIdE4ksB+JtKSgoAChoaEIDg6GWCzWdThEeoN/N4jKYkJCWpOdnQ2pVIqsrCxIJBJdh0OkN/h3g6gsDtkQERGRzjEhISIiIp1jQkJEREQ6x4SEtEYsFmP27NmctEf0D/y7QVQWJ7USERGRzrGHhIiIiHSOCQkRERHpHBMSIiIi0jkmJKRxsbGxEIlEyMzM/Nd6derUwYoVK95ITERvqzlz5qB58+a6DoNI6ziplTSusLAQT548gZ2dHUQiEcLDwzFp0qQyCcrDhw9hbm4OMzMz3QRKpGdEIhF27tyJAQMGKMpycnJQUFCAatWq6S4wojeAb/sljTM2Noa9vb3KetWrV38D0RC93SwsLGBhYaHrMIi0jkM276guXbpg3LhxGDduHKRSKWxsbDBz5kyUdpg9ffoUPj4+sLKygpmZGXr37o2kpCTF9ffu3cMHH3wAKysrmJubo1GjRti/fz8A5SGb2NhYjBgxAllZWRCJRBCJRJgzZw4A5SGbYcOG4ZNPPlGKsaioCDY2Nvjpp58AAHK5HKGhoXB2doapqSmaNWuGX3/9VcvfFL0LunTpggkTJmD69OmwtraGvb294s8pAGRmZmLUqFGoXr06JBIJ3n//fVy+fFmpjfnz58PW1hZVq1bFqFGj8MUXXygNtZw/fx49evSAjY0NpFIpOnfujEuXLinO16lTBwDw4YcfQiQSKT7/fcjm0KFDMDExKdPbOHHiRLz//vuKzydPnkTHjh1hamqKWrVqYcKECXj+/Plrf09E2sSE5B22ceNGGBoa4ty5c1i5ciWWLVuGdevWAQD8/Pxw4cIF7NmzB3FxcRAEAX369EFRUREAICAgAAUFBTh+/DiuXLmCr7/+utzf4tq1a4cVK1ZAIpEgNTUVqampmDp1apl63t7e2Lt3L3JychRlUVFRyM3NxYcffggACA0NxU8//YSwsDBcu3YNkydPxqeffopjx45p4+uhd8zGjRthbm6Os2fPYvHixQgJCUF0dDQA4OOPP0ZGRgYOHDiAixcvwsPDA926dcOTJ08AABEREViwYAG+/vprXLx4EbVr18aaNWuU2n/27Bl8fX1x8uRJnDlzBvXq1UOfPn3w7NkzAC8SFgDYsGEDUlNTFZ//rlu3brC0tMSOHTsUZSUlJdi2bRu8vb0BALdv30avXr0waNAgJCQkYNu2bTh58iTGjRun+S+NSJMEeid17txZcHNzE+RyuaIsKChIcHNzE/744w8BgHDq1CnFuUePHgmmpqbC9u3bBUEQhCZNmghz5swpt+2jR48KAISnT58KgiAIGzZsEKRSaZl6Tk5OwvLlywVBEISioiLBxsZG+OmnnxTnhw4dKnzyySeCIAhCfn6+YGZmJpw+fVqpjZEjRwpDhw5V+/mJ/q5z585Chw4dlMpat24tBAUFCSdOnBAkEomQn5+vdL5u3brC2rVrBUEQhDZt2ggBAQFK59u3by80a9bspfcsKSkRqlatKuzdu1dRBkDYuXOnUr3Zs2crtTNx4kTh/fffV3yOiooSxGKx4u/byJEjhTFjxii1ceLECcHAwEDIy8t7aTxEusYekndY27ZtIRKJFJ9lMhmSkpJw/fp1GBoaok2bNopz1apVQ4MGDXDjxg0AwIQJEzB//ny0b98es2fPRkJCwmvFYmhoiMGDByMiIgIA8Pz5c+zevVvxW9+tW7eQm5uLHj16KMbULSws8NNPP+H27duvdW8iAGjatKnSZwcHB2RkZODy5cvIyclBtWrVlP7sJScnK/7sJSYm4r333lO6/p+f09PTMXr0aNSrVw9SqRQSiQQ5OTlISUlRK05vb2/ExsbiwYMHAF70znh5ecHS0hIAcPnyZYSHhyvF6unpCblcjuTkZLXuRfQmcVIrvZJRo0bB09MTkZGROHToEEJDQ7F06VKMHz/+ldv09vZG586dkZGRgejoaJiamqJXr14AoBjKiYyMRI0aNZSu4/tASBOMjIyUPotEIsjlcuTk5MDBwQGxsbFlrilNAirC19cXjx8/xsqVK+Hk5ASxWAyZTIbCwkK14mzdujXq1q2Ln3/+GWPHjsXOnTsRHh6uOJ+Tk4PPP/8cEyZMKHNt7dq11boX0ZvEhOQddvbsWaXPpePa7u7uKC4uxtmzZ9GuXTsAwOPHj5GYmAh3d3dF/Vq1asHf3x/+/v4IDg7Gjz/+WG5CYmxsjJKSEpXxtGvXDrVq1cK2bdtw4MABfPzxx4p/JNzd3SEWi5GSkoLOnTu/zmMTqcXDwwNpaWkwNDRUTDT9pwYNGuD8+fPw8fFRlP1zDsipU6ewevVq9OnTBwDw559/4tGjR0p1jIyMKvR3xdvbGxEREahZsyYMDAzg5eWlFO/169fh6upa0Uck0gscsnmHpaSkIDAwEImJidi6dSu+/fZbTJw4EfXq1UP//v0xevRonDx5EpcvX8ann36KGjVqoH///gCASZMmISoqCsnJybh06RKOHj0KNze3cu9Tp04d5OTkICYmBo8ePUJubu5LYxo2bBjCwsIQHR2tGK4BgKpVq2Lq1KmYPHkyNm7ciNu3b+PSpUv49ttvsXHjRs1+MUR/0717d8hkMgwYMACHDh3C3bt3cfr0aXz11Ve4cOECAGD8+PFYv349Nm7ciKSkJMyfPx8JCQlKQ6L16tXDpk2bcOPGDZw9exbe3t4wNTVVuledOnUQExODtLQ0PH369KUxeXt749KlS1iwYAE++ugjpV7CoKAgnD59GuPGjUN8fDySkpKwe/duTmolvceE5B3m4+ODvLw8vPfeewgICMDEiRMxZswYAC9m+rds2RJ9+/aFTCaDIAjYv3+/oseipKQEAQEBcHNzQ69evVC/fn2sXr263Pu0a9cO/v7++OSTT1C9enUsXrz4pTF5e3vj+vXrqFGjBtq3b690bt68eZg5cyZCQ0MV942MjISzs7OGvhGiskQiEfbv349OnTphxIgRqF+/PoYMGYJ79+7Bzs4OwIs/t8HBwZg6dSo8PDyQnJwMPz8/mJiYKNpZv349nj59Cg8PDwwfPhwTJkyAra2t0r2WLl2K6Oho1KpVCy1atHhpTK6urnjvvfeQkJCglLgDL+bCHDt2DH/88Qc6duyIFi1aYNasWXB0dNTgt0Kkedyp9R3VpUsXNG/enFu3E2lJjx49YG9vj02bNuk6FKK3AueQEBG9ptzcXISFhcHT0xNVqlTB1q1bcfjwYcU+JkSkGhMSIqLXVDqss2DBAuTn56NBgwbYsWMHunfvruvQiN4aHLIhIiIineOkViIiItI5JiRERESkc0xIiIiISOeYkBAREZHOMSEhqkT8/PwwYMAAxecuXbpg0qRJr9WmJtogIlKFCQnRG+Dn5weRSASRSARjY2O4uroiJCQExcXFWr3vb7/9hnnz5lWobmxsLEQiETIzM1+5DSKiV8V9SIjekF69emHDhg0oKCjA/v37ERAQACMjIwQHByvVKywshLGxsUbuaW1trRdtEBGpwh4SojdELBbD3t4eTk5OGDt2LLp37449e/YohlkWLFgAR0dHNGjQAMCLt8EOHjwYlpaWsLa2Rv/+/XH37l1FeyUlJQgMDISlpSWqVauG6dOn45/bCv1zuKWgoABBQUGoVasWxGIxXF1dsX79ety9exddu3YFAFhZWUEkEsHPz6/cNp4+fQofHx9YWVnBzMwMvXv3RlJSkuJ8eHg4LC0tERUVBTc3N1hYWKBXr15ITU3V7BdKRJUKExIiHTE1NUVhYSEAICYmBomJiYiOjsa+fftQVFQET09PVK1aFSdOnMCpU6cU/7CXXrN06VKEh4fjv//9L06ePIknT55g586d/3pPHx8fbN26FatWrcKNGzewdu1aWFhYoFatWtixYwcAIDExEampqVi5cmW5bfj5+eHChQvYs2cP4uLiIAgC+vTpg6KiIkWd3NxcfPPNN9i0aROOHz+OlJQUTJ06VRNfGxFVUhyyIXrDBEFATEwMoqKiMH78eDx8+BDm5uZYt26dYqhm8+bNkMvlWLduneIV9hs2bIClpSViY2PRs2dPrFixAsHBwRg4cCAAICwsDFFRUS+97x9//IHt27cjOjpasaW5i4uL4nzp0IytrS0sLS3LbSMpKQl79uzBqVOn0K5dOwBAREQEatWqhV27duHjjz8GABQVFSEsLAx169YFAIwbNw4hISGv+pUR0TuACQnRG7Jv3z5YWFigqKgIcrkcw4YNw5w5cxAQEIAmTZoozRu5fPkybt26hapVqyq1kZ+fj9u3byMrKwupqalo06aN4pyhoSFatWpVZtimVHx8PKpUqYLOnTu/8jPcuHEDhoaGSvetVq0aGjRogBs3bijKzMzMFMkIADg4OCAjI+OV70tElR8TEqI3pGvXrlizZg2MjY3h6OgIQ8P//fUzNzdXqpuTk4OWLVsiIiKiTDvVq1d/pfubmpq+0nWvwsjISOmzSCR6aaJERARwDgnRG2Nubg5XV1fUrl1bKRkpj4eHB5KSkmBrawtXV1elQyqVQiqVwsHBAWfPnlVcU1xcjIsXL760zSZNmkAul+PYsWPlni/toSkpKXlpG25ubiguLla67+PHj5GYmAh3d/d/fSYion/DhIRID3l7e8PGxgb9+/fHiRMnkJycjNjYWEyYMAF//fUXAGDixIlYtGgRdu3ahZs3b+I///lPmT1E/q5OnTrw9fXFZ599hl27dina3L59OwDAyckJIpEI+/btw8OHD5GTk1OmjXr16qF///4YPXo0Tp48icuXL+PTTz9FjRo10L9/f618F0T0bmBCQqSHzMzMcPz4cdSuXRsDBw6Em5sbRo4cifz8fEgkEgDAlClTMHz4cPj6+kImk6Fq1ar48MMP/7XdNWvW4KOPPsJ//vMfNGzYEKNHj8bz588BADVq1MDcuXPxxRdfwM7ODuPGjSu3jQ0bNqBly5bo27cvZDIZBEHA/v37ywzTEBGpQyRwYJeIiIh0jD0kREREpHNMSIiIiEjnmJAQERGRzjEhISIiIp1jQkJEREQ6x4SEiIiIdI4JCREREekcExIiIiLSOSYkREREpHNMSIiIiEjnmJAQERGRzjEhISIiIp37P4G0MmfF7EOiAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "               precision    recall  f1-score   support\n",
            "\n",
            "    Positive       0.94      0.92      0.93      4265\n",
            "    Negative       0.92      0.94      0.93      4265\n",
            "\n",
            "    accuracy                           0.93      8530\n",
            "   macro avg       0.93      0.93      0.93      8530\n",
            "weighted avg       0.93      0.93      0.93      8530\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Challenges"
      ],
      "metadata": {
        "id": "QOY73ej_jTlJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Our Naive Bayes Classifier achieved an impressively high accuracy rate (>90%), but there's a catch! This accuracy figure represents the training accuracy, and it has limitations. Training accuracy merely informs us about how well the model has learned from the training data. It doesn't tell us how the model will perform when faced with new, unseen data. To measure the model's generalization ability, we need to evaluate its predictive performance on a separate test dataset.\n",
        "\n",
        "For your challenge, you'll be working with the Rotten Tomatoes Reviews test dataset to evaluate how well the model performs on unseen data. Let's see if you can achieve a high test accuracy. You got this!"
      ],
      "metadata": {
        "id": "Ve52wt091boL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Q1 [1 pt] Loading test data\n",
        "Load the Rotten Tomatoes Reviews Test data set from HuggingFace."
      ],
      "metadata": {
        "id": "Yxsc_fLTlChw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_test, y_test = pd.DataFrame(columns=['text']), pd.DataFrame(columns=['label'])\n",
        "\n",
        "# --------------------------------------------\n",
        "# TODO: INSERT CODE HERE\n",
        "# Assign the test features to X_test\n",
        "# and the test labels to y_test.\n",
        "# --------------------------------------------\n",
        "\n",
        "print(\"Train set size:\", len(X_train))\n",
        "print(\"Test set size:\", len(X_test))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7srWn5I80KHe",
        "outputId": "ebcfffaf-1fb9-4fc2-a35a-baba9f59ead6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train set size: 8530\n",
            "Test set size: 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Q2 [1 pt] Transforming features\n",
        "Transform the test data with the previously computed bag-of-words (CountVectorizer)."
      ],
      "metadata": {
        "id": "rKlPo-0_lKpv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "words = vectorizer.get_feature_names_out()\n",
        "counts_test = counts.copy()\n",
        "counts_test[:] = 0\n",
        "\n",
        "# --------------------------------------------\n",
        "# TODO: INSERT CODE HERE\n",
        "# Store the test data word counts to\n",
        "# counts_test.\n",
        "# --------------------------------------------\n",
        "\n",
        "df_words_test = pd.DataFrame({'words':words, 'count':counts_test}, columns=['words', 'count'])\n",
        "df_words_test.sort_values(by=['count'], ascending=False).head(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "id": "_ua0IcE_21Jn",
        "outputId": "2514ac39-c908-4fcc-db0e-5ef72141a46f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "               words  count\n",
              "0                 00      0\n",
              "10940           pork      0\n",
              "10974      possessed      0\n",
              "10975      possesses      0\n",
              "10976     possession      0\n",
              "10977  possibilities      0\n",
              "10978    possibility      0\n",
              "10979       possible      0\n",
              "10980       possibly      0\n",
              "10981         possui      0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c78ea851-1bcf-424b-b826-16eb1979ca01\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>words</th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>00</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10940</th>\n",
              "      <td>pork</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10974</th>\n",
              "      <td>possessed</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10975</th>\n",
              "      <td>possesses</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10976</th>\n",
              "      <td>possession</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10977</th>\n",
              "      <td>possibilities</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10978</th>\n",
              "      <td>possibility</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10979</th>\n",
              "      <td>possible</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10980</th>\n",
              "      <td>possibly</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10981</th>\n",
              "      <td>possui</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c78ea851-1bcf-424b-b826-16eb1979ca01')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c78ea851-1bcf-424b-b826-16eb1979ca01 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c78ea851-1bcf-424b-b826-16eb1979ca01');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-9fd004af-045b-4c60-882b-4c0e632cd354\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-9fd004af-045b-4c60-882b-4c0e632cd354')\"\n",
              "            title=\"Suggest charts.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-9fd004af-045b-4c60-882b-4c0e632cd354 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Q3 [1 pt] Predicting test data\n",
        "Predict the vectorized test data using the previously fitted Multinomial Naive Bayes model."
      ],
      "metadata": {
        "id": "xgi4k-lzlPHC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_pred, y_true = [0]*10, [1]*10\n",
        "\n",
        "# --------------------------------------------\n",
        "# TODO: INSERT CODE HERE\n",
        "# Predict the review sentiments on the\n",
        "# test data set.\n",
        "#\n",
        "# y_pred should have the predicted values\n",
        "# y_true should have the target/actual labels\n",
        "# --------------------------------------------\n",
        "\n",
        "prediction_results = pd.DataFrame({'Target':y_true, 'Prediction':y_pred}, columns=['Target', 'Prediction'])\n",
        "print(prediction_results.sample(n=10, random_state=280))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qtpVQ5Oh21CS",
        "outputId": "5c4f1644-5fe8-4f8c-806b-5cf24e0683e9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   Target  Prediction\n",
            "7       1           0\n",
            "2       1           0\n",
            "3       1           0\n",
            "9       1           0\n",
            "1       1           0\n",
            "6       1           0\n",
            "8       1           0\n",
            "0       1           0\n",
            "4       1           0\n",
            "5       1           0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Q4 [2 pts] Analyzing test performance\n",
        "Compute the model performance, and display the confusion matrix, accuracy, precision, recall, and f1-score."
      ],
      "metadata": {
        "id": "7WvLz3LDlQOa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# --------------------------------------------\n",
        "# TODO: INSERT CODE HERE\n",
        "# Plot the confusion matrix based on the\n",
        "# predicted values in Q3.\n",
        "#\n",
        "# Then print performance statistics.\n",
        "# --------------------------------------------"
      ],
      "metadata": {
        "id": "WzzikRqXjWVF",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 633
        },
        "outputId": "898e9d85-341d-4d0c-a321-6451db1c6c52"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhsAAAHHCAYAAAAWM5p0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOeklEQVR4nO3dd1QU19sH8O/SlroUlWYBAUPE3qJIFI0oKiYau6KCsUSDlWgM+VmxkJhYU8REI/YSew0iCkbBEo1iJUgwGAWxAQJK23n/8LBvNqCyssMu7PeTM+ewd+7ceWYTw+NtIxEEQQARERGRSPQ0HQARERFVb0w2iIiISFRMNoiIiEhUTDaIiIhIVEw2iIiISFRMNoiIiEhUTDaIiIhIVEw2iIiISFRMNoiIiEhUTDaIRJSUlIRu3brB0tISEokEe/fuVWv7t2/fhkQiQUREhFrbrco6deqETp06aToMIvoXJhtU7SUnJ+Pjjz+Gi4sLjI2NIZPJ4OXlhRUrVuDZs2ei3jsgIABXrlzBwoULsXHjRrRu3VrU+1WmwMBASCQSyGSyMr/HpKQkSCQSSCQSfPPNNyq3f+/ePcydOxeXLl1SQ7REpEkGmg6ASEyHDh3CgAEDIJVKMWLECDRu3BgFBQU4deoUpk+fjmvXruHHH38U5d7Pnj1DfHw8/ve//2HChAmi3MPJyQnPnj2DoaGhKO2/joGBAfLy8nDgwAEMHDhQ6dzmzZthbGyM58+fv1Hb9+7dw7x58+Ds7IzmzZuX+7qjR4++0f2ISDxMNqjaSklJweDBg+Hk5ITjx4/DwcFBcS4oKAi3bt3CoUOHRLv/gwcPAABWVlai3UMikcDY2Fi09l9HKpXCy8sLW7duLZVsbNmyBX5+fti1a1elxJKXlwdTU1MYGRlVyv2IqPw4jELV1uLFi5GTk4O1a9cqJRol3NzcMHnyZMXnoqIizJ8/H66urpBKpXB2dsYXX3yB/Px8peucnZ3Rq1cvnDp1Cu+88w6MjY3h4uKCDRs2KOrMnTsXTk5OAIDp06dDIpHA2dkZwIvhh5Kf/23u3LmQSCRKZVFRUXj33XdhZWUFc3NzuLu744svvlCcf9mcjePHj6NDhw4wMzODlZUVevfujRs3bpR5v1u3biEwMBBWVlawtLTEyJEjkZeX9/Iv9j+GDh2KI0eOIDMzU1F2/vx5JCUlYejQoaXqP378GNOmTUOTJk1gbm4OmUyGHj164PLly4o6MTExaNOmDQBg5MiRiuGYkufs1KkTGjdujAsXLqBjx44wNTVVfC//nbMREBAAY2PjUs/v6+sLa2tr3Lt3r9zPSkRvhskGVVsHDhyAi4sL2rdvX676o0ePxuzZs9GyZUssW7YM3t7eCAsLw+DBg0vVvXXrFvr374+uXbtiyZIlsLa2RmBgIK5duwYA6Nu3L5YtWwYAGDJkCDZu3Ijly5erFP+1a9fQq1cv5OfnIzQ0FEuWLMEHH3yA06dPv/K6Y8eOwdfXFxkZGZg7dy6Cg4MRFxcHLy8v3L59u1T9gQMH4unTpwgLC8PAgQMRERGBefPmlTvOvn37QiKRYPfu3YqyLVu24O2330bLli1L1f/rr7+wd+9e9OrVC0uXLsX06dNx5coVeHt7K37xN2zYEKGhoQCAsWPHYuPGjdi4cSM6duyoaOfRo0fo0aMHmjdvjuXLl6Nz585lxrdixQrUqlULAQEBKC4uBgCsXr0aR48exbfffgtHR8dyPysRvSGBqBrKysoSAAi9e/cuV/1Lly4JAITRo0crlU+bNk0AIBw/flxR5uTkJAAQTp48qSjLyMgQpFKp8OmnnyrKUlJSBADC119/rdRmQECA4OTkVCqGOXPmCP/+I7ls2TIBgPDgwYOXxl1yj3Xr1inKmjdvLtja2gqPHj1SlF2+fFnQ09MTRowYUep+H330kVKbH374oVCjRo2X3vPfz2FmZiYIgiD0799f6NKliyAIglBcXCzY29sL8+bNK/M7eP78uVBcXFzqOaRSqRAaGqooO3/+fKlnK+Ht7S0AEMLDw8s85+3trVQWGRkpABAWLFgg/PXXX4K5ubnQp0+f1z4jEakHezaoWsrOzgYAWFhYlKv+4cOHAQDBwcFK5Z9++ikAlJrb4eHhgQ4dOig+16pVC+7u7vjrr7/eOOb/KpnrsW/fPsjl8nJdk5aWhkuXLiEwMBA2NjaK8qZNm6Jr166K5/y3cePGKX3u0KEDHj16pPgOy2Po0KGIiYlBeno6jh8/jvT09DKHUIAX8zz09F78r6e4uBiPHj1SDBFdvHix3PeUSqUYOXJkuep269YNH3/8MUJDQ9G3b18YGxtj9erV5b4XEVUMkw2qlmQyGQDg6dOn5ar/999/Q09PD25ubkrl9vb2sLKywt9//61UXq9evVJtWFtb48mTJ28YcWmDBg2Cl5cXRo8eDTs7OwwePBg7dux4ZeJREqe7u3upcw0bNsTDhw+Rm5urVP7fZ7G2tgYAlZ6lZ8+esLCwwPbt27F582a0adOm1HdZQi6XY9myZWjQoAGkUilq1qyJWrVqISEhAVlZWeW+Z+3atVWaDPrNN9/AxsYGly5dwsqVK2Fra1vua4moYphsULUkk8ng6OiIq1evqnTdfydovoy+vn6Z5YIgvPE9SuYTlDAxMcHJkydx7NgxDB8+HAkJCRg0aBC6du1aqm5FVORZSkilUvTt2xfr16/Hnj17XtqrAQCLFi1CcHAwOnbsiE2bNiEyMhJRUVFo1KhRuXtwgBffjyr++OMPZGRkAACuXLmi0rVEVDFMNqja6tWrF5KTkxEfH//auk5OTpDL5UhKSlIqv3//PjIzMxUrS9TB2tpaaeVGif/2ngCAnp4eunTpgqVLl+L69etYuHAhjh8/jhMnTpTZdkmciYmJpc7dvHkTNWvWhJmZWcUe4CWGDh2KP/74A0+fPi1zUm2JnTt3onPnzli7di0GDx6Mbt26wcfHp9R3Ut7Erzxyc3MxcuRIeHh4YOzYsVi8eDHOnz+vtvaJ6NWYbFC19dlnn8HMzAyjR4/G/fv3S51PTk7GihUrALwYBgBQasXI0qVLAQB+fn5qi8vV1RVZWVlISEhQlKWlpWHPnj1K9R4/flzq2pLNrf67HLeEg4MDmjdvjvXr1yv98r569SqOHj2qeE4xdO7cGfPnz8d3330He3v7l9bT19cv1Wvyyy+/4O7du0plJUlRWYmZqmbMmIHU1FSsX78eS5cuhbOzMwICAl76PRKRenFTL6q2XF1dsWXLFgwaNAgNGzZU2kE0Li4Ov/zyCwIDAwEAzZo1Q0BAAH788UdkZmbC29sb586dw/r169GnT5+XLqt8E4MHD8aMGTPw4YcfYtKkScjLy8OqVavw1ltvKU2QDA0NxcmTJ+Hn5wcnJydkZGTghx9+QJ06dfDuu+++tP2vv/4aPXr0gKenJ0aNGoVnz57h22+/haWlJebOnau25/gvPT09zJw587X1evXqhdDQUIwcORLt27fHlStXsHnzZri4uCjVc3V1hZWVFcLDw2FhYQEzMzO0bdsW9evXVymu48eP44cffsCcOXMUS3HXrVuHTp06YdasWVi8eLFK7RHRG9Dwahgi0f3555/CmDFjBGdnZ8HIyEiwsLAQvLy8hG+//VZ4/vy5ol5hYaEwb948oX79+oKhoaFQt25dISQkRKmOILxY+urn51fqPv9dcvmypa+CIAhHjx4VGjduLBgZGQnu7u7Cpk2bSi19jY6OFnr37i04OjoKRkZGgqOjozBkyBDhzz//LHWP/y4PPXbsmODl5SWYmJgIMplMeP/994Xr168r1Sm533+X1q5bt04AIKSkpLz0OxUE5aWvL/Oypa+ffvqp4ODgIJiYmAheXl5CfHx8mUtW9+3bJ3h4eAgGBgZKz+nt7S00atSozHv+u53s7GzByclJaNmypVBYWKhUb+rUqYKenp4QHx//ymcgooqTCIIKs8CIiIiIVMQ5G0RERCQqJhtEREQkKiYbREREJComG0RERCQqJhtEREQkKiYbREREJComG0RERCSqarmDaOFD9b3mm6g6MXHsoOkQiLROUcHd11eqIHX9XjKs6fL6SlqIPRtEREQkqmrZs0FERKRV5MWajkCjmGwQERGJTZBrOgKNYrJBREQkNrluJxucs0FERESiYs8GERGRyAQOoxAREZGoOIxCREREJB72bBAREYlNx4dR2LNBREQkNnmxeo4K+PLLLyGRSDBlyhRF2fPnzxEUFIQaNWrA3Nwc/fr1w/3795WuS01NhZ+fH0xNTWFra4vp06ejqKhIpXsz2SAiIqrmzp8/j9WrV6Np06ZK5VOnTsWBAwfwyy+/IDY2Fvfu3UPfvn0V54uLi+Hn54eCggLExcVh/fr1iIiIwOzZs1W6P5MNIiIisQly9RxvICcnB/7+/vjpp59gbW2tKM/KysLatWuxdOlSvPfee2jVqhXWrVuHuLg4nDlzBgBw9OhRXL9+HZs2bULz5s3Ro0cPzJ8/H99//z0KCgrKHQOTDSIiIrHJ5Wo58vPzkZ2drXTk5+e/8tZBQUHw8/ODj4+PUvmFCxdQWFioVP7222+jXr16iI+PBwDEx8ejSZMmsLOzU9Tx9fVFdnY2rl27Vu7HZ7JBRERURYSFhcHS0lLpCAsLe2n9bdu24eLFi2XWSU9Ph5GREaysrJTK7ezskJ6erqjz70Sj5HzJufLiahQiIiKRqWtTr5CQEAQHByuVSaXSMuveuXMHkydPRlRUFIyNjdVy/zfFng0iIiKxqWkYRSqVQiaTKR0vSzYuXLiAjIwMtGzZEgYGBjAwMEBsbCxWrlwJAwMD2NnZoaCgAJmZmUrX3b9/H/b29gAAe3v7UqtTSj6X1CkPJhtERERi08AE0S5duuDKlSu4dOmS4mjdujX8/f0VPxsaGiI6OlpxTWJiIlJTU+Hp6QkA8PT0xJUrV5CRkaGoExUVBZlMBg8Pj3LHwmEUIiKiasjCwgKNGzdWKjMzM0ONGjUU5aNGjUJwcDBsbGwgk8kwceJEeHp6ol27dgCAbt26wcPDA8OHD8fixYuRnp6OmTNnIigo6KU9KmVhskFERCS2Cm7IJZZly5ZBT08P/fr1Q35+Pnx9ffHDDz8ozuvr6+PgwYMYP348PD09YWZmhoCAAISGhqp0H4kgCIK6g9e0wod/aToEIq1k4thB0yEQaZ2igrui3yP/xgm1tCNt2Fkt7VQ2ztkgIiIiUXEYhYiISGw6/op5JhtERERi41tfiYiIiMTDng0iIiKxcRiFiIiIxCQI2rn0tbJwGIWIiIhExZ4NIiIisen4BFEmG0RERGLjnA0iIiISlY73bHDOBhEREYmKPRtERERi09IXsVUWJhtERERi4zAKERERkXjYs0FERCQ2rkYhIiIiUXEYhYiIiEg87NkgIiISG4dRiIiISFQ6nmxwGIWIiIhExZ4NIiIiken6K+aZbBAREYlNx4dRmGwQERGJjUtfiYiIiMTDng0iIiKxcRiFiIiIRMVhFCIiIiLxsGeDiIhIbBxGISIiIlFxGIWIiIhIPOzZICIiEhuHUYiIiEhUOp5scBiFiIiIRMWeDSIiIrHp+ARRJhtERERi0/FhFCYbREREYtPxng3O2SAiIiJRsWeDiIhIbBxGISIiIlFxGIWIiIhIPOzZICIiEhuHUYiIiEhUOp5scBiFiIiIRMWeDSIiIrEJgqYj0CgmG0RERGLjMAoRERGReNizQUREJDYd79lgskFERCQ2buqlHX777TcMGzYMnp6euHv3LgBg48aNOHXqlIYjIyIiqiC5XD1HFaUVycauXbvg6+sLExMT/PHHH8jPzwcAZGVlYdGiRRqOjoiIiCpCK5KNBQsWIDw8HD/99BMMDQ0V5V5eXrh48aIGIyMiIlIDQVDPUUVpxZyNxMREdOzYsVS5paUlMjMzKz8gIiIidarCQyDqoBU9G/b29rh161ap8lOnTsHFxUUDEREREZG6aEWyMWbMGEyePBlnz56FRCLBvXv3sHnzZkybNg3jx4/XdHhEREQVo4EJoqtWrULTpk0hk8kgk8ng6emJI0eOKM536tQJEolE6Rg3bpxSG6mpqfDz84OpqSlsbW0xffp0FBUVqfz4WjGM8vnnn0Mul6NLly7Iy8tDx44dIZVKMW3aNEycOFHT4REREVWMBpa+1qlTB19++SUaNGgAQRCwfv169O7dG3/88QcaNWoE4MVf9kNDQxXXmJqaKn4uLi6Gn58f7O3tERcXh7S0NIwYMQKGhoYqL96QCIL2zDgpKCjArVu3kJOTAw8PD5ibm79RO4UP/1JzZETVg4ljB02HQKR1igruin6PZ2uC1dKOyeilFbrexsYGX3/9NUaNGoVOnTqhefPmWL58eZl1jxw5gl69euHevXuws7MDAISHh2PGjBl48OABjIyMyn1frRhG2bRpE/Ly8mBkZAQPDw+88847b5xoEBERaRtBLqjlyM/PR3Z2ttJRsl3EqxQXF2Pbtm3Izc2Fp6enonzz5s2oWbMmGjdujJCQEOTl5SnOxcfHo0mTJopEAwB8fX2RnZ2Na9euqfT8WpFsTJ06Fba2thg6dCgOHz6M4uJiTYdERESkPmqasxEWFgZLS0ulIyws7KW3vXLlCszNzSGVSjFu3Djs2bMHHh4eAIChQ4di06ZNOHHiBEJCQrBx40YMGzZMcW16erpSogFA8Tk9PV2lx9eKORtpaWn49ddfsXXrVgwcOBCmpqYYMGAA/P390b59e02HR0REpBVCQkIQHKw8JCOVSl9a393dHZcuXUJWVhZ27tyJgIAAxMbGwsPDA2PHjlXUa9KkCRwcHNClSxckJyfD1dVVrXFrRc+GgYEBevXqhc2bNyMjIwPLli3D7du30blzZ7U/MBERUaUT5Go5pFKpYnVJyfGqZMPIyAhubm5o1aoVwsLC0KxZM6xYsaLMum3btgUAxVYU9vb2uH//vlKdks/29vYqPb5WJBv/ZmpqCl9fX/To0QMNGjTA7du3NR0SERFRxcgF9RwVDUMuf+kcj0uXLgEAHBwcAACenp64cuUKMjIyFHWioqIgk8kUQzHlpRXDKACQl5eHPXv2YPPmzYiOjkbdunUxZMgQ7Ny5U9OhERERVYwGdhANCQlBjx49UK9ePTx9+hRbtmxBTEwMIiMjkZycjC1btqBnz56oUaMGEhISMHXqVHTs2BFNmzYFAHTr1g0eHh4YPnw4Fi9ejPT0dMycORNBQUGv7E0pi1YkG4MHD8bBgwdhamqKgQMHYtasWUqzZYmIiEg1GRkZGDFiBNLS0mBpaYmmTZsiMjISXbt2xZ07d3Ds2DEsX74cubm5qFu3Lvr164eZM2cqrtfX18fBgwcxfvx4eHp6wszMDAEBAUr7cpSXViQb+vr62LFjB3x9faGvr6/pcIiIiNRLAz0ba9eufem5unXrIjY29rVtODk54fDhwxWORSuSjc2bN2s6BCIiIvFoz/6ZGqGxZGPlypUYO3YsjI2NsXLlylfWnTRpUiVFRUREROqmse3K69evj99//x01atRA/fr1X1pPIpHgr79U236c25VXnjUbd2B5+DoMG9Abn0958QKfX/YdxqGoGNxIvIXcvGeI+/UXyCyUd4RdvX4rTsadR2LSXzA0NEB8JCcCVwZuVy6eDu+2xaefjkfLFk3g6GiPvv0/wv79kQBeLO+fH/oZund/Dy71nZCVlY3o46fwxf8WIS3txVJCJ6c6+N8XU9C5kxfs7Wvh3r372LJ1NxaFrURhYaEmH63aq4ztyvOWjlFLO6bBP6mlncqmsZ6NlJSUMn+mquPKjUT8su8w3nJTThafP8/Hu21b4922rbE8fF2Z1xYWFsG3cwc0b9wQuw9GVka4RKIyMzNFQsJ1rIvYhl2/KI+Vm5qaoEXzJli4aAUSEq7D2soSy5bOw57d69DOsycA4G13N+jp6eGToBm4lXwbjRq5Y/Wqr2FmaorPPp+viUcidVLDstWqTCvmbISGhmLatGlKb5sDgGfPnuHrr7/G7NmzNRQZvUxe3jN8Pu9rzJ0xGavXb1U6N3zQhwCAcxcTXnr9hNHDAQB7D0WJFyRRJfo18gR+jTxR5rns7Kfo3nOIUtmkyTNxJv4w6tZ1xJ079xB5NAaRR2MU51NSUrH0rXB8PHYEkw2q8rRiU6958+YhJyenVHleXh7mzZungYjodRYs+R4dPdvAs00LTYdCVCVZWsogl8uRmZn9yjqPn2RWXlAkHjXtIFpVaUWyIQgCJBJJqfLLly/DxsZGAxHRqxw+FoMbfyZjyriRmg6FqEqSSqVYtOgLbNu+F0+flv6LFgC4ujoj6JOR+OmnTZUcHYlCS3YQ1RSNDqNYW1tDIpFAIpHgrbfeUko4iouLkZOTg3Hjxr2yjfz8/FJbr+rl56u8uxmVT9r9B/hy+Wr8tHwRpFIjTYdDVOUYGBhg29ZwSCQSBE0IKbOOo6M9Dh3YhJ27DmLtz1sqOUIi9dNosrF8+XIIgoCPPvoI8+bNg6WlpeKckZERnJ2dX7uTaFhYWKmhlpnTJ2H2Z5NFiVnXXU9MwuMnmRj40QRFWXGxHBcuXcXW3Qdw8cR+bsxG9BIliUa9enXQtdvAMns1HBzscCzqF8SfuYBx4z/TQJQkBkEDm3ppE40mGwEBAQBeLINt3749DA0NVW6jrNft6j0VfxmTrmrXqjn2bFylVDZz4VLUd6qLUcMGMNEgeomSRMPNrT58ug7A48dPStVxdLTHsahfcPFiAkaNngoN7UxAYqjCQyDqoLFkIzs7GzKZDADQokULPHv2DM+ePSuzbkm9skil0lJDJoUFD9UXKCkxMzNFAxdnpTITE2NYySwU5Q8fPcbDR0+Q+s89AEBS8m2YmZrAwd4WljILAEBaegaysp8i7X4GiovluPlnMgCgXh1HmJqaVNrzEKmLmZkp3P61DLy+cz00a9YIjx8/QVpaBnZs/xEtmjdB7w8DoK+vDzu7WgCAx48zUVhYCEdHe0RH7URq6j/4bMZ81KpVQ9HW/fsPKv15SM2q8OROddBYsmFtbY20tDTY2trCysqqzAmiJRNHi4uLNRAhvantew9j1c//vwV9QNB0AMCCL4LRx68rAOC7NRux78gxRZ3+I18My/z87Vd4p2XTSoyWSD1at2qG6GP/vzndkm/mAgDWb9iB0PlL8MH7vgCAi78rL/fu4tMfsSfj4dOlIxo0qI8GDeoj9fYFpToGRrXFDZ5IZBrbQTQ2NhZeXl4wMDB47ctgvL29VWqbO4gSlY07iBKVVhk7iOaG+qulHbPZVfNdYhrr2fh3AqFqMkFERFSl6PgEUa3YZ+PXX3/FqVOnFJ+///57NG/eHEOHDsWTJ6UnUREREVHVoRXJxvTp05Gd/WIXvStXriA4OBg9e/ZESkpKqZUmREREVQ439dK8lJQUeHh4AAB27dqF999/H4sWLcLFixfRs2dPDUdHRERUQTq+GkUrejaMjIyQl5cHADh27Bi6desGALCxsVH0eBAREVHVpBU9G++++y6Cg4Ph5eWFc+fOYfv27QCAP//8E3Xq1NFwdERERBVUhYdA1EEreja+++47GBgYYOfOnVi1ahVq136xpvzIkSPo3r27hqMjIiKqGEEuV8tRVWlsnw0xcZ8NorJxnw2i0ipjn42ckH5qacc8bJda2qlsWjGMArx4y+vevXtx48YNAECjRo3wwQcf8F0bRERU9en4MIpWJBu3bt1Cz549cffuXbi7uwN48TbXunXr4tChQ3B1ddVwhERERBWg48mGVszZmDRpElxdXXHnzh1cvHgRFy9eRGpqKurXr49JkyZpOjwiIqKKEeTqOaoorejZiI2NxZkzZ2BjY6Moq1GjBr788kt4eXlpMDIiIiKqKK1INqRSKZ4+fVqqPCcnB0ZGRhqIiIiISI04jKJ5vXr1wtixY3H27FkIggBBEHDmzBmMGzcOH3zwgabDIyIiqhBBLqjlqKq0ItlYuXIlXF1d4enpCWNjYxgbG6N9+/Zwc3PDihUrNB0eERERVYBWDKNYWVlh3759uHXrFq5fvw4A8PDwgJubm4YjIyIiUoMq3CuhDlqRbADA2rVrsWzZMiQlJQEAGjRogClTpmD06NEajoyIiKiCqvDun+qgFcnG7NmzsXTpUkycOBGenp4AgPj4eEydOhWpqakIDQ3VcIRERET0prRiu/JatWph5cqVGDJkiFL51q1bMXHiRDx8+FCl9rhdOVHZuF05UWmVsV350096qKUdix+OqKWdyqYVPRuFhYVo3bp1qfJWrVqhqKhIAxERERGpkY7P2dCK1SjDhw/HqlWrSpX/+OOP8Pf310BEREREpC5a0bMBvJggevToUbRr1w4AcPbsWaSmpmLEiBEIDg5W1Fu6dKmmQiQiInojWjBjQaO0Itm4evUqWrZsCQBITk4GANSsWRM1a9bE1atXFfUkEolG4iMiIqoQHR9G0Ypk48SJE5oOgYiISDw6nmxoxZwNIiIiqr60omeDiIioOqvK7zVRByYbREREYtPxZIPDKERERCQq9mwQERGJTbdfjcJkg4iISGy6PmeDwyhEREQkKvZsEBERiU3HezaYbBAREYlNx+dscBiFiIiIRMWeDSIiIpHp+gRRJhtERERi0/FhFCYbREREItP1ng3O2SAiIiJRsWeDiIhIbBxGISIiIjEJOp5scBiFiIioGlq1ahWaNm0KmUwGmUwGT09PHDlyRHH++fPnCAoKQo0aNWBubo5+/frh/v37Sm2kpqbCz88PpqamsLW1xfTp01FUVKRyLEw2iIiIxCZX06GCOnXq4Msvv8SFCxfw+++/47333kPv3r1x7do1AMDUqVNx4MAB/PLLL4iNjcW9e/fQt29fxfXFxcXw8/NDQUEB4uLisH79ekRERGD27NkqP75EEIRqN0W28OFfmg6BSCuZOHbQdAhEWqeo4K7o93jYw1st7dQ8Eluh621sbPD111+jf//+qFWrFrZs2YL+/fsDAG7evImGDRsiPj4e7dq1w5EjR9CrVy/cu3cPdnZ2AIDw8HDMmDEDDx48gJGRUbnvy54NIiKiKiI/Px/Z2dlKR35+/muvKy4uxrZt25CbmwtPT09cuHABhYWF8PHxUdR5++23Ua9ePcTHxwMA4uPj0aRJE0WiAQC+vr7Izs5W9I6UF5MNIiIisalpGCUsLAyWlpZKR1hY2Etve+XKFZibm0MqlWLcuHHYs2cPPDw8kJ6eDiMjI1hZWSnVt7OzQ3p6OgAgPT1dKdEoOV9yThVcjUJERCQyda1GCQkJQXBwsFKZVCp9aX13d3dcunQJWVlZ2LlzJwICAhAbW7GhmDfBZIOIiEhk6ko2pFLpK5OL/zIyMoKbmxsAoFWrVjh//jxWrFiBQYMGoaCgAJmZmUq9G/fv34e9vT0AwN7eHufOnVNqr2S1Skmd8uIwChERkY6Qy+XIz89Hq1atYGhoiOjoaMW5xMREpKamwtPTEwDg6emJK1euICMjQ1EnKioKMpkMHh4eKt2XPRtEREQi08SmXiEhIejRowfq1auHp0+fYsuWLYiJiUFkZCQsLS0xatQoBAcHw8bGBjKZDBMnToSnpyfatWsHAOjWrRs8PDwwfPhwLF68GOnp6Zg5cyaCgoJU6l0BmGwQERGJT5BU+i0zMjIwYsQIpKWlwdLSEk2bNkVkZCS6du0KAFi2bBn09PTQr18/5Ofnw9fXFz/88IPien19fRw8eBDjx4+Hp6cnzMzMEBAQgNDQUJVj4T4bRDqE+2wQlVYZ+2zc79RJLe3YxcSopZ3Kxp4NIiIiken6u1GYbBAREYlMkFf+MIo24WoUIiIiEhV7NoiIiETGYRQiIiISlaCB1SjahMMoREREJCr2bBAREYmMwyhEREQkKl1fjcJkg4iISGTVb/tM1XDOBhEREYmKPRtEREQi4zAKERERiUrXkw0OoxAREZGo2LNBREQkMl2fIMpkg4iISGQcRiEiIiISkcrJhouLCx49elSqPDMzEy4uLmoJioiIqDoRBIlajqpK5WGU27dvo7i4uFR5fn4+7t69q5agiIiIqhNuV15O+/fvV/wcGRkJS0tLxefi4mJER0fD2dlZrcERERFR1VfuZKNPnz4AAIlEgoCAAKVzhoaGcHZ2xpIlS9QaHBERUXUgr8JDIOpQ7mRDLn/RB1S/fn2cP38eNWvWFC0oIiKi6qQqz7dQB5XnbKSkpCh+fv78OYyNjdUaEBERUXXDpa8qksvlmD9/PmrXrg1zc3P89ddfAIBZs2Zh7dq1ag+QiIiIqjaVk40FCxYgIiICixcvhpGRkaK8cePGWLNmjVqDIyIiqg4EQT1HVaVysrFhwwb8+OOP8Pf3h76+vqK8WbNmuHnzplqDIyIiqg4EuUQtR1WlcrJx9+5duLm5lSqXy+UoLCxUS1BERERUfaicbHh4eOC3334rVb5z5060aNFCLUERERFVJ3JBopajqlJ5Ncrs2bMREBCAu3fvQi6XY/fu3UhMTMSGDRtw8OBBMWIkIiKq0nR96avKPRu9e/fGgQMHcOzYMZiZmWH27Nm4ceMGDhw4gK5du4oRIxEREVVhb/SK+Q4dOiAqKkrdsRAREVVLVXkliTq8UbJBRERE5VeV51uog8rJhrW1NSSS0l+aRCKBsbEx3NzcEBgYiJEjR6olQCIiIqra3miC6MKFC9GjRw+88847AIBz587h119/RVBQEFJSUjB+/HgUFRVhzJgxag+YiIioqtH1CaIqJxunTp3CggULMG7cOKXy1atX4+jRo9i1axeaNm2KlStXMtkgIiIC52yovBolMjISPj4+pcq7dOmCyMhIAEDPnj0V70whIiLSdbq+z4bKyYaNjQ0OHDhQqvzAgQOwsbEBAOTm5sLCwqLi0REREVGVp/IwyqxZszB+/HicOHFCMWfj/PnzOHz4MMLDwwEAUVFR8Pb2Vm+kKjBx7KCxexNps2f3Su/+S0Ti0/U5GxJBUH0k6fTp0/juu++QmJgIAHB3d8fEiRPRvn17tQf4JgyMams6BCKtxGSDqDTDmi6i3+OsY1+1tNP23m61tFPZVOrZKCwsxMcff4xZs2Zh69atYsVERERE1YhKczYMDQ2xa9cusWIhIiKqlgQ1HVWVyhNE+/Tpg71794oQChERUfWk66tRVJ4g2qBBA4SGhuL06dNo1aoVzMzMlM5PmjRJbcERERFR1afyBNH69eu/vDGJRCv21+AEUaKycYIoUWmVMUH0tH1/tbTjlb5TLe1UNpV7NlJSUsSIg4iIqNqSazoADVN5zgYRERGRKt7oFfP//PMP9u/fj9TUVBQUFCidW7p0qVoCIyIiqi4EVN3JneqgcrIRHR2NDz74AC4uLrh58yYaN26M27dvQxAEtGzZUowYiYiIqjR5VV63qgYqD6OEhIRg2rRpuHLlCoyNjbFr1y7cuXMH3t7eGDBggBgxEhERVWlySNRyVFUqJxs3btzAiBEjAAAGBgZ49uwZzM3NERoaiq+++krtARIREVHVpnKyYWZmppin4eDggOTkZMW5hw8fqi8yIiKiakKARC1HVVXuZCM0NBS5ublo164dTp06BQDo2bMnPv30UyxcuBAfffQR2rVrJ1qgREREVZVcTUdVVe5NvfT19ZGWloacnBzk5OSgadOmyM3Nxaeffoq4uDg0aNAAS5cuhZOTk9gxvxY39SIqGzf1IiqtMjb1irIbpJZ2ut7frpZ2Klu5V6OU5CQuLv//L8XMzAzh4eHqj4qIiKgaqcpDIOqg0pwNiUS3vywiIqI3oYlhlLCwMLRp0wYWFhawtbVFnz59kJiYqFSnU6dOkEgkSse4ceOU6qSmpsLPzw+mpqawtbXF9OnTUVRUpFIsKu2z8dZbb7024Xj8+LFKARAREZH6xcbGIigoCG3atEFRURG++OILdOvWDdevX1d6ieqYMWMQGhqq+Gxqaqr4ubi4GH5+frC3t0dcXBzS0tIwYsQIGBoaYtGiReWORaVkY968ebC0tFTlEiIiIp2nicmdv/76q9LniIgI2Nra4sKFC+jYsaOi3NTUFPb29mW2cfToUVy/fh3Hjh2DnZ0dmjdvjvnz52PGjBmYO3cujIyMyhWLSsnG4MGDYWtrq8olREREOk9dczby8/ORn5+vVCaVSiGVSl97bVZWFgDAxsZGqXzz5s3YtGkT7O3t8f7772PWrFmK3o34+Hg0adIEdnZ2ivq+vr4YP348rl27hhYtWpQr7nLP2eB8DSIiIs0KCwuDpaWl0hEWFvba6+RyOaZMmQIvLy80btxYUT506FBs2rQJJ06cQEhICDZu3Ihhw4YpzqenpyslGgAUn9PT08sdt8qrUYiIiEg1cjX9fT0kJATBwcFKZeXp1QgKCsLVq1cV+2SVGDt2rOLnJk2awMHBAV26dEFycjJcXV3VEzRUSDbk8qq8nQgREZHmqOu9JuUdMvm3CRMm4ODBgzh58iTq1Knzyrpt27YFANy6dQuurq6wt7fHuXPnlOrcv38fAF46z6MsKm9XTkRERKoR1HSodE9BwIQJE7Bnzx4cP34c9evXf+01ly5dAvDidSQA4OnpiStXriAjI0NRJyoqCjKZDB4eHuWOReVXzBMREZH2CwoKwpYtW7Bv3z5YWFgo5lhYWlrCxMQEycnJ2LJlC3r27IkaNWogISEBU6dORceOHdG0aVMAQLdu3eDh4YHhw4dj8eLFSE9Px8yZMxEUFKRSD0u5tyuvSrhdOVHZuF05UWmVsV35bvuhammnb/qWctd92cKOdevWITAwEHfu3MGwYcNw9epV5Obmom7duvjwww8xc+ZMyGQyRf2///4b48ePR0xMDMzMzBAQEIAvv/wSBgbl769gskGkQ5hsEJVWGcnGTgd/tbTTP22zWtqpbJyzQURERKLinA0iIiKRVbshBBUx2SAiIhKZrm8ewWEUIiIiEhV7NoiIiESmrh1EqyomG0RERCJT1w6iVRWHUYiIiEhU7NkgIiISGVejEBERkag4Z4OIiIhExaWvRERERCJizwYREZHIOGeDiIiIRKXrczY4jEJERESiYs8GERGRyHR9giiTDSIiIpHperLBYRQiIiISFXs2iIiIRCbo+ARRJhtEREQi4zAKERERkYjYs0FERCQyXe/ZYLJBREQkMu4gSkRERKLiDqJEREREImLPBhERkcg4Z4OIiIhEpevJBodRiIiISFTs2SAiIhIZV6MQERGRqLgahYiIiEhE7NkgIiISma5PEGWyQUREJDJdn7PBYRQiIiISFXs2iIiIRCbX8b4NJhtEREQi45wNIiIiEpVu92twzgYRERGJTKuSjYKCAiQmJqKoqEjToRAREamNXE1HVaUVyUZeXh5GjRoFU1NTNGrUCKmpqQCAiRMn4ssvv9RwdERERBUjl6jnqKq0ItkICQnB5cuXERMTA2NjY0W5j48Ptm/frsHIiIiIqKK0YoLo3r17sX37drRr1w4Syf+nbo0aNUJycrIGIyMiIqo4Ln3VAg8ePICtrW2p8tzcXKXkg4iIqCrS7VRDS4ZRWrdujUOHDik+lyQYa9asgaenp6bCIiIiIjXQip6NRYsWoUePHrh+/TqKioqwYsUKXL9+HXFxcYiNjdV0eERERBVSlVeSqINW9Gy8++67uHTpEoqKitCkSRMcPXoUtra2iI+PR6tWrTQdHhERUYXIIajlqKq0omcDAFxdXfHTTz9pOgwiIiJSM63o2fDx8UFERASys7M1HQoREZHaCWo6qiqtSDYaNWqEkJAQ2NvbY8CAAdi3bx8KCws1HRYREZFacAdRLbBixQrcvXsXe/fuhZmZGUaMGAE7OzuMHTuWE0SJiKjK0/U5G1qRbACAnp4eunXrhoiICNy/fx+rV6/GuXPn8N5772k6NCIiIqoArZkgWiI9PR3btm3Dpk2bkJCQgHfeeUfTIREREVVI1e2TUA+t6NnIzs7GunXr0LVrV9StWxerVq3CBx98gKSkJJw5c0bT4REREVWIrs/Z0IqeDTs7O1hbW2PQoEEICwtD69atNR0SERERqYlW9Gzs378f//zzD5YtW8ZEg4iIqh1BTf+oIiwsDG3atIGFhQVsbW3Rp08fJCYmKtV5/vw5goKCUKNGDZibm6Nfv364f/++Up3U1FT4+fnB1NQUtra2mD59OoqKilSKRSuSja5du0JPTytCISIiUjtNDKPExsYiKCgIZ86cQVRUFAoLC9GtWzfk5uYq6kydOhUHDhzAL7/8gtjYWNy7dw99+/ZVnC8uLoafnx8KCgoQFxeH9evXIyIiArNnz1YpFokgCBqZt9KyZUtER0fD2toaLVq0eOXbXS9evKhS2wZGtSsaHlG19Ozeb5oOgUjrGNZ0Ef0eE5wHqaWd725vf+NrS96wHhsbi44dOyIrKwu1atXCli1b0L9/fwDAzZs30bBhQ8THx6Ndu3Y4cuQIevXqhXv37sHOzg4AEB4ejhkzZuDBgwcwMjIq1701Nmejd+/ekEqlip/5KnkiIqqu1LVHRn5+PvLz85XKpFKp4vfpq2RlZQEAbGxsAAAXLlxAYWEhfHx8FHXefvtt1KtXT5FsxMfHo0mTJopEAwB8fX0xfvx4XLt2DS1atChX3BpLNubMmaP4ee7cuZoKg4iISHTqGkIICwvDvHnzlMrmzJnz2t+jcrkcU6ZMgZeXFxo3bgzgxVYTRkZGsLKyUqprZ2eH9PR0RZ1/Jxol50vOlZdWTJRwcXHBo0ePSpVnZmbCxUX87i0iIqKqICQkBFlZWUpHSEjIa68LCgrC1atXsW3btkqIsjStSDZu376N4uLiUuX5+fn4559/NBARvUqHd9ti754IpN6+gKKCu/jgA1/FOQMDA4Qt+gJ/XDyGrCdJSL19Aet+XgEHB+XMeM/udfjr1jnkZCfjzt8XEbFuZak6RFXVmo070NirB75cHq4o+2XfYQRO+Axtu/ZFY68eyH6aU+q6rOynmDH3K7Tt2heevv0xK2wZ8vKeVWboJBJ1bVculUohk8mUjtcNoUyYMAEHDx7EiRMnUKdOHUW5vb09CgoKkJmZqVT//v37sLe3V9T57+qUks8ldcpDo8nG/v37sX//fgBAZGSk4vP+/fuxZ88ezJ8/H/Xr19dkiFQGMzNTJCRcx8TJ/yt1ztTUBC2aN8HCRSvQpm13DBg4Bu5vuWDP7nVK9WJi4jBk6Dh4NO6IgYPGwtXFCTu2/VhZj0Akmis3EvHLvsN4y035/13Pn+fj3batMWbE4JdeO2PeYtxKScVPyxfh+8VzceHSVcxdvFLskKkSaGI1iiAImDBhAvbs2YPjx4+X+n3aqlUrGBoaIjo6WlGWmJiI1NRUeHp6AgA8PT1x5coVZGRkKOpERUVBJpPBw8Oj3LFobDUKAMVyV4lEgv+GYWhoCGdnZyxZsgS9evVSqV2uRqk8RQV30bf/R9i/P/KldVq3aoYz8YdR37UN7ty5V2adXr26YvfOn2FqXl/l9dtUflyNIq68vGcY8NFEzPw0CKvXb8Xbbi74fMo4pTrnLibgo4kzEPfrL5BZmCvKk2+norf/x9i2ZgUaN3wLAHDqzO8YP202ovdshG2tGpX6LLqkMlajjHbur5Z21tzeWe66n3zyCbZs2YJ9+/bB3d1dUW5paQkTExMAwPjx43H48GFERERAJpNh4sSJAIC4uDgAL5a+Nm/eHI6Ojli8eDHS09MxfPhwjB49GosWLSp3LBrt2ZDL5ZDL5ahXrx4yMjIUn+VyOfLz85GYmKhyokHax9JSBrlcjszM7DLPW1tbYeiQvoiP/52JBlVpC5Z8j46ebeDZpnwz9P/t8tUbkFmYKxINAGjXugX09CRIuH5TnWGSjli1ahWysrLQqVMnODg4KI7t2/9/+eyyZcvQq1cv9OvXDx07doS9vT12796tOK+vr4+DBw9CX18fnp6eGDZsGEaMGIHQ0FCVYtGK7cpTUlLe+NqylgEJgsCltFpCKpVi0aIvsG37Xjz9zxh12KIv8Mn4kTAzM8WZMxfwQZ8ADUVJVHGHj8Xgxp/J2LZmxRtd//DRE9hYWSqVGRjow9LCAg8fP1FHiKRBmnivSXkGLoyNjfH999/j+++/f2kdJycnHD58uEKxaEWyAQC5ubmIjY1FamoqCgoKlM5NmjTppdeVtQxIomcOib5MlDip/AwMDLBtazgkEgmCJpSeLf3NklX4ed02ONWrjVkzgxHx8wp80GeEBiIlqpi0+w/w5fLV+Gn5Ikil5dvkiHSLqluNVzdakWz88ccf6NmzJ/Ly8pCbmwsbGxs8fPhQsQ/7q5KNkJAQBAcHK5VZ13hb7JDpNUoSjXr16qBrt4GlejUA4NGjJ3j06AmSkv7CjZu38HfK72jXthXOnL2ggYiJ3tz1xCQ8fpKJgR9NUJQVF8tx4dJVbN19ABdP7Ie+vv4r26hZwxqPM7OUyoqKipH19Clq2liLEjdRZdGKZGPq1Kl4//33ER4eDktLS5w5cwaGhoYYNmwYJk+e/Mpry9o5jUMomlWSaLi51YdP1wF4XI4uYD29F//O+LdCqoratWqOPRtXKZXNXLgU9Z3qYtSwAa9NNACgWeOGyH6ag2s3k9Do7QYAgLMXLkEuF9DUg3+Bquqq8uvh1UErko1Lly5h9erV0NPTg76+PvLz8+Hi4oLFixcjICBA6aUwpHlmZqZw+9eyvvrO9dCsWSM8fvwEaWkZ2LH9R7Ro3gS9PwyAvr4+7OxqAQAeP85EYWEh3mnTAq1bN8PpuPN48iQTri7OmDd3Om7dSkH8GfZqUNVjZmaKBi7OSmUmJsawklkoyh8+eoyHj54g9Z8XK7KSkm/DzNQEDva2sJRZwNW5Ht5t1xpzv1qB2dMnorCoCIuWrUIPH2+uRKkG5Jpb+KkVtCLZMDQ0VCyDtbW1RWpqKho2bAhLS0vcuXNHw9HRf7Vu1QzRx/5/+dWSb+YCANZv2IHQ+UvwwfsvNvm6+HuU0nVdfPoj9mQ88p49w4d9emLO7GkwMzNBWloGIo/GYFHYilLzdYiqi+17D2PVz5sVnwOCpgMAFnwRjD5+XQEAX835DAuX/oBRk0KgpyeBTycvfDFlvEbiJVInje6zUaJbt24IDAzE0KFDMWbMGCQkJGDSpEnYuHEjnjx5grNnz6rUHvfZICob99kgKq0y9tkY5qSeHvpNf+9+fSUtpBXblS9atAgODg4AgIULF8La2hrjx4/HgwcP8OOP3FWSiIiqNnVtV15VacUwSuvWrRU/29ra4tdff9VgNERERKROWpFsEBERVWfcZ0MLtGjRoszlqhKJBMbGxnBzc0NgYCA6d+6sgeiIiIgqRteXvmrFnI3u3bvjr7/+gpmZGTp37ozOnTvD3NwcycnJaNOmDdLS0uDj44N9+/ZpOlQiIiKVcc6GFnj48CE+/fRTzJo1S6l8wYIF+Pvvv3H06FHMmTMH8+fPR+/evTUUJREREb0JrejZ2LFjB4YMGVKqfPDgwdixYwcAYMiQIUhMTKzs0IiIiCpMUNM/VZVWJBvGxsaIi4srVR4XFwdjY2MAL15HX/IzERFRVSJX01FVacUwysSJEzFu3DhcuHABbdq0AQCcP38ea9aswRdffAEAiIyMRPPmzTUYJREREb0JrdhBFAA2b96M7777TjFU4u7ujokTJ2Lo0KEAgGfPnilWp7wOdxAlKht3ECUqrTJ2EP2w3vtqaWdP6gG1tFPZtCbZUCcmG0RlY7JBVFplJBu96/VSSzv7Ug+qpZ3KphVzNgAgMzNTMWzy+PFjAMDFixdx9+5dDUdGREREFaEVczYSEhLg4+MDS0tL3L59G6NHj4aNjQ12796N1NRUbNiwQdMhEhERvbGqPLlTHbSiZyM4OBiBgYFISkpSmpPRs2dPnDx5UoORERERVRyXvmqB8+fP4+OPPy5VXrt2baSnp2sgIiIiIlIXrRhGkUqlyM7OLlX+559/olatWhqIiIiISH2q8lbj6qAVPRsffPABQkNDUVhYCODFC9hSU1MxY8YM9OvXT8PRERERVYwgCGo5qiqtSDaWLFmCnJwc2Nra4tmzZ/D29oabmxvMzc2xcOFCTYdHRERUIdxBVAtYWloiKioKp0+fxuXLl5GTk4OWLVvCx8dH06ERERFRBWlFsgEA0dHRiI6ORkZGBuRyOW7evIktW7YAAH7++WcNR0dERPTmqvJKEnXQimRj3rx5CA0NRevWreHg4ACJRKLpkIiIiNRG1yeIakWyER4ejoiICAwfPlzToRAREZGaaUWyUVBQgPbt22s6DCIiIlFU5ZUk6qAVq1FGjx6tmJ9BRERU3cghqOWoqrSiZ+P58+f48ccfcezYMTRt2hSGhoZK55cuXaqhyIiIiKiitCLZSEhIQPPmzQEAV69eVTrHyaJERFTVcTWKFjhx4oSmQyAiIhKNnHM2iIiIiMSjFT0bRERE1Zlu92sw2SAiIhJdVV5Jog5MNoiIiESm68kG52wQERGRqNizQUREJDJd30GUyQYREZHIOIxCREREJCL2bBAREYmMO4gSERGRqHR9zgaHUYiIiEhU7NkgIiISma5PEGWyQUREJDIOoxARERGJiD0bREREIuMwChEREYmKS1+JiIhIVHLO2SAiIiISD3s2iIiIRMZhFCIiIhIVh1GIiIioWjp58iTef/99ODo6QiKRYO/evUrnAwMDIZFIlI7u3bsr1Xn8+DH8/f0hk8lgZWWFUaNGIScnR6U4mGwQERGJTFDTP6rKzc1Fs2bN8P3337+0Tvfu3ZGWlqY4tm7dqnTe398f165dQ1RUFA4ePIiTJ09i7NixKsXBYRQiIiKRaWoYpUePHujRo8cr60ilUtjb25d57saNG/j1119x/vx5tG7dGgDw7bffomfPnvjmm2/g6OhYrjjYs0FERFRF5OfnIzs7W+nIz8+vUJsxMTGwtbWFu7s7xo8fj0ePHinOxcfHw8rKSpFoAICPjw/09PRw9uzZct+DyQYREZHI1DWMEhYWBktLS6UjLCzsjePq3r07NmzYgOjoaHz11VeIjY1Fjx49UFxcDABIT0+Hra2t0jUGBgawsbFBenp6ue/DYRQiIiKRqWsYJSQkBMHBwUplUqn0jdsbPHiw4ucmTZqgadOmcHV1RUxMDLp06fLG7f4XezaIiIiqCKlUCplMpnRUJNn4LxcXF9SsWRO3bt0CANjb2yMjI0OpTlFRER4/fvzSeR5lYbJBREQkMk2tRlHVP//8g0ePHsHBwQEA4OnpiczMTFy4cEFR5/jx45DL5Wjbtm252+UwChERkcgEQa6R++bk5Ch6KQAgJSUFly5dgo2NDWxsbDBv3jz069cP9vb2SE5OxmeffQY3Nzf4+voCABo2bIju3btjzJgxCA8PR2FhISZMmIDBgweXeyUKAEgEofpta2ZgVFvTIRBppWf3ftN0CERax7Cmi+j3cKrRVC3t/P0oQaX6MTEx6Ny5c6nygIAArFq1Cn369MEff/yBzMxMODo6olu3bpg/fz7s7OwUdR8/fowJEybgwIED0NPTQ79+/bBy5UqYm5uXOw4mG0Q6hMkGUWnVOdnQFhxGISIiElk1/Hu9SphsEBERiUyu42995WoUIiIiEhV7NoiIiETGYRQiIiISlaZexKYtOIxCREREomLPBhERkcgqY/dPbcZkg4iISGS6PmeDwyhEREQkKvZsEBERiUzX99lgskFERCQyXR9GYbJBREQkMi59JSIiIhIRezaIiIhExmEUIiIiEpWuTxDlMAoRERGJij0bREREIuMwChEREYmKq1GIiIiIRMSeDSIiIpHxRWxEREQkKg6jEBEREYmIPRtEREQi42oUIiIiEhXnbBAREZGodL1ng3M2iIiISFTs2SAiIhKZrvdsMNkgIiISmW6nGhxGISIiIpFJBF3v2yHR5OfnIywsDCEhIZBKpZoOh0hr8M8G6RomGySa7OxsWFpaIisrCzKZTNPhEGkN/tkgXcNhFCIiIhIVkw0iIiISFZMNIiIiEhWTDRKNVCrFnDlzOAGO6D/4Z4N0DSeIEhERkajYs0FERESiYrJBREREomKyQURERKJiskFqFxMTA4lEgszMzFfWc3Z2xvLlyyslJqKqau7cuWjevLmmwyCqEE4QJbUrKCjA48ePYWdnB4lEgoiICEyZMqVU8vHgwQOYmZnB1NRUM4ESaRmJRII9e/agT58+irKcnBzk5+ejRo0amguMqIL41ldSOyMjI9jb27+2Xq1atSohGqKqzdzcHObm5poOg6hCOIyiozp16oQJEyZgwoQJsLS0RM2aNTFr1iyUdHQ9efIEI0aMgLW1NUxNTdGjRw8kJSUprv/777/x/vvvw9raGmZmZmjUqBEOHz4MQHkYJSYmBiNHjkRWVhYkEgkkEgnmzp0LQHkYZejQoRg0aJBSjIWFhahZsyY2bNgAAJDL5QgLC0P9+vVhYmKCZs2aYefOnSJ/U6QLOnXqhEmTJuGzzz6DjY0N7O3tFf+dAkBmZiZGjx6NWrVqQSaT4b333sPly5eV2liwYAFsbW1hYWGB0aNH4/PPP1ca/jh//jy6du2KmjVrwtLSEt7e3rh48aLivLOzMwDgww8/hEQiUXz+9zDK0aNHYWxsXKqXcPLkyXjvvfcUn0+dOoUOHTrAxMQEdevWxaRJk5Cbm1vh74noTTHZ0GHr16+HgYEBzp07hxUrVmDp0qVYs2YNACAwMBC///479u/fj/j4eAiCgJ49e6KwsBAAEBQUhPz8fJw8eRJXrlzBV199Vebfvtq3b4/ly5dDJpMhLS0NaWlpmDZtWql6/v7+OHDgAHJychRlkZGRyMvLw4cffggACAsLw4YNGxAeHo5r165h6tSpGDZsGGJjY8X4ekjHrF+/HmZmZjh79iwWL16M0NBQREVFAQAGDBiAjIwMHDlyBBcuXEDLli3RpUsXPH78GACwefNmLFy4EF999RUuXLiAevXqYdWqVUrtP336FAEBATh16hTOnDmDBg0aoGfPnnj69CmAF8kIAKxbtw5paWmKz//WpUsXWFlZYdeuXYqy4uJibN++Hf7+/gCA5ORkdO/eHf369UNCQgK2b9+OU6dOYcKECer/0ojKSyCd5O3tLTRs2FCQy+WKshkzZggNGzYU/vzzTwGAcPr0acW5hw8fCiYmJsKOHTsEQRCEJk2aCHPnzi2z7RMnTggAhCdPngiCIAjr1q0TLC0tS9VzcnISli1bJgiCIBQWFgo1a9YUNmzYoDg/ZMgQYdCgQYIgCMLz588FU1NTIS4uTqmNUaNGCUOGDFH5+Yn+zdvbW3j33XeVytq0aSPMmDFD+O233wSZTCY8f/5c6byrq6uwevVqQRAEoW3btkJQUJDSeS8vL6FZs2YvvWdxcbFgYWEhHDhwQFEGQNizZ49SvTlz5ii1M3nyZOG9995TfI6MjBSkUqniz9uoUaOEsWPHKrXx22+/CXp6esKzZ89eGg+RmNizocPatWsHiUSi+Ozp6YmkpCRcv34dBgYGaNu2reJcjRo14O7ujhs3bgAAJk2ahAULFsDLywtz5sxBQkJChWIxMDDAwIEDsXnzZgBAbm4u9u3bp/jb2q1bt5CXl4euXbsqxrDNzc2xYcMGJCcnV+jeRADQtGlTpc8ODg7IyMjA5cuXkZOTgxo1aij9t5eSkqL4by8xMRHvvPOO0vX//Xz//n2MGTMGDRo0gKWlJWQyGXJycpCamqpSnP7+/oiJicG9e/cAvOhV8fPzg5WVFQDg8uXLiIiIUIrV19cXcrkcKSkpKt2LSF04QZTeyOjRo+Hr64tDhw7h6NGjCAsLw5IlSzBx4sQ3btPf3x/e3t7IyMhAVFQUTExM0L17dwBQDK8cOnQItWvXVrqO75cgdTA0NFT6LJFIIJfLkZOTAwcHB8TExJS6puQXfHkEBATg0aNHWLFiBZycnCCVSuHp6YmCggKV4mzTpg1cXV2xbds2jB8/Hnv27EFERITifE5ODj7++GNMmjSp1LX16tVT6V5E6sJkQ4edPXtW6XPJOLKHhweKiopw9uxZtG/fHgDw6NEjJCYmwsPDQ1G/bt26GDduHMaNG4eQkBD89NNPZSYbRkZGKC4ufm087du3R926dbF9+3YcOXIEAwYMUPwC8PDwgFQqRWpqKry9vSvy2EQqadmyJdLT02FgYKCYtPlf7u7uOH/+PEaMGKEo+++ci9OnT+OHH35Az549AQB37tzBw4cPleoYGhqW68+Kv78/Nm/ejDp16kBPTw9+fn5K8V6/fh1ubm7lfUQi0XEYRYelpqYiODgYiYmJ2Lp1K7799ltMnjwZDRo0QO/evTFmzBicOnUKly9fxrBhw1C7dm307t0bADBlyhRERkYiJSUFFy9exIkTJ9CwYcMy7+Ps7IycnBxER0fj4cOHyMvLe2lMQ4cORXh4OKKiohRDKABgYWGBadOmYerUqVi/fj2Sk5Nx8eJFfPvtt1i/fr16vxiif/Hx8YGnpyf69OmDo0eP4vbt24iLi8P//vc//P777wCAiRMnYu3atVi/fj2SkpKwYMECJCQkKA1TNmjQABs3bsSNGzdw9uxZ+Pv7w8TEROlezs7OiI6ORnp6Op48efLSmPz9/XHx4kUsXLgQ/fv3V+rdmzFjBuLi4jBhwgRcunQJSUlJ2LdvHyeIkkYx2dBhI0aMwLNnz/DOO+8gKCgIkydPxtixYwG8mBHfqlUr9OrVC56enhAEAYcPH1b0NBQXFyMoKAgNGzZE9+7d8dZbb+GHH34o8z7t27fHuHHjMGjQINSqVQuLFy9+aUz+/v64fv06ateuDS8vL6Vz8+fPx6xZsxAWFqa476FDh1C/fn01fSNEpUkkEhw+fBgdO3bEyJEj8dZbb2Hw4MH4+++/YWdnB+DFf7chISGYNm0aWrZsiZSUFAQGBsLY2FjRztq1a/HkyRO0bNkSw4cPx6RJk2Bra6t0ryVLliAqKgp169ZFixYtXhqTm5sb3nnnHSQkJCgl5cCLuSexsbH4888/0aFDB7Ro0QKzZ8+Go6OjGr8VItVwB1Ed1alTJzRv3pzbhROJpGvXrrC3t8fGjRs1HQqRxnHOBhFRBeXl5SE8PBy+vr7Q19fH1q1bcezYMcU+HUS6jskGEVEFlQy1LFy4EM+fP4e7uzt27doFHx8fTYdGpBU4jEJERESi4gRRIiIiEhWTDSIiIhIVkw0iIiISFZMNIiIiEhWTDaJqJDAwEH369FF87tSpE6ZMmVKhNtXRBhHpNiYbRJUgMDAQEokEEokERkZGcHNzQ2hoKIqKikS97+7duzF//vxy1Y2JiYFEIkFmZuYbt0FEVBbus0FUSbp3745169YhPz8fhw8fRlBQEAwNDRESEqJUr6CgAEZGRmq5p42NjVa0QUS6jT0bRJVEKpXC3t4eTk5OGD9+PHx8fLB//37F0MfChQvh6OgId3d3AC/eCjpw4EBYWVnBxsYGvXv3xu3btxXtFRcXIzg4GFZWVqhRowY+++wz/HfbnP8OgeTn52PGjBmoW7cupFIp3NzcsHbtWty+fRudO3cGAFhbW0MikSAwMLDMNp48eYIRI0bA2toapqam6NGjB5KSkhTnIyIiYGVlhcjISDRs2BDm5ubo3r070tLS1PuFElGVwWSDSENMTExQUFAAAIiOjkZiYiKioqJw8OBBFBYWwtfXFxYWFvjtt99w+vRpxS/tkmuWLFmCiIgI/Pzzzzh16hQeP36MPXv2vPKeI0aMwNatW7Fy5UrcuHEDq1evhrm5OerWrYtdu3YBABITE5GWloYVK1aU2UZgYCB+//137N+/H/Hx8RAEAT179kRhYaGiTl5eHr755hts3LgRJ0+eRGpqKqZNm6aOr42IqiAOoxBVMkEQEB0djcjISEycOBEPHjyAmZkZ1qxZoxg+2bRpE+RyOdasWaN4Tfm6detgZWWFmJgYdOvWDcuXL0dISAj69u0LAAgPD0dkZORL7/vnn39ix44diIqKUmyj7eLiojhfMlxia2sLKyurMttISkrC/v37cfr0abRv3x4AsHnzZtStWxd79+7FgAEDAACFhYUIDw+Hq6srAGDChAkIDQ1906+MiKo4JhtEleTgwYMwNzdHYWEh5HI5hg4dirlz5yIoKAhNmjRRmqdx+fJl3Lp1CxYWFkptPH/+HMnJycjKykJaWhratm2rOGdgYIDWrVuXGkopcenSJejr68Pb2/uNn+HGjRswMDBQum+NGjXg7u6OGzduKMpMTU0ViQYAODg4ICMj443vS0RVG5MNokrSuXNnrFq1CkZGRnB0dISBwf//8TMzM1Oqm5OTg1atWmHz5s2l2qlVq9Yb3d/ExOSNrnsThoaGSp8lEslLkyAiqv44Z4OokpiZmcHNzQ316tVTSjTK0rJlSyQlJcHW1hZubm5Kh6WlJSwtLeHg4ICzZ88qrikqKsKFCxde2maTJk0gl8sRGxtb5vmSnpXi4uKXttGwYUMUFRUp3ffRo0dITEyEh4fHK5+JiHQXkw0iLeTv74+aNWuid+/e+O2335CSkoKYmBhMmjQJ//zzDwBg8uTJ+PLLL7F3717cvHkTn3zySak9Mv7N2dkZAQEB+Oijj7B3715Fmzt27AAAODk5QSKR4ODBg3jw4AFycnJKtdGgQQP07t0bY8aMwalTp3D58mUMGzYMtWvXRu/evUX5Loio6mOyQaSFTE1NcfLkSdSrVw99+/ZFw4YNMWrUKDx//hwymQwA8Omnn2L48OEICAiAp6cnLCws8OGHH76y3VWrVqF///745JNP8Pbbb2PMmDHIzc0FANSuXRvz5s3D559/Djs7O0yYMKHMNtatW4dWrVqhV69e8PT0hCAIOHz4cKmhEyKiEhKBA6lEREQkIvZsEBERkaiYbBAREZGomGwQERGRqJhsEBERkaiYbBAREZGomGwQERGRqJhsEBERkaiYbBAREZGomGwQERGRqJhsEBERkaiYbBAREZGomGwQERGRqP4P8imsycfMg8cAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "    Positive       0.77      0.77      0.77       533\n",
            "    Negative       0.77      0.77      0.77       533\n",
            "\n",
            "    accuracy                           0.77      1066\n",
            "   macro avg       0.77      0.77      0.77      1066\n",
            "weighted avg       0.77      0.77      0.77      1066\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Q5 [3 pts] Improving accuracy\n",
        "\n",
        "Try to improve the test accuracy of your Naive Bayes Classifier.\n",
        "\n",
        "* 1 pt: ≥ 78%\n",
        "* 2 pts: ≥ 79%\n",
        "* 3 pts: ≥ 80%\n",
        "\n",
        "<br>\n",
        "\n",
        "Here are the rules:\n",
        "* You may only use an NB model, but you may explore NB model variants such as Gaussian, etc.\n",
        "* You cannot use other classifier models (e.g., ensemble learning).\n",
        "* You may generate/extract new features, including alternative vectorizers\n",
        "* You may perform additional preprocessing steps\n",
        "* You must separate the training and testing, and you are not allowed to mix these data sets. However, you may perform validation using the validation set (see HuggingFace)\n",
        "\n",
        "<br>\n",
        "\n",
        "Tips:\n",
        "1. Run this locally for faster / more efficient processing\n",
        "2. Look out for the distinct words on each review class\n",
        "3. Play around with the function parameters (check Sklearn API)\n",
        "4. Reaching an 80% accuracy might be very challenging for Naive Bayes, so don't be too hard on yourself."
      ],
      "metadata": {
        "id": "8veCE28N230o"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# ------------------------------------------------------------------------\n",
        "# TODO: INSERT CODE HERE\n",
        "# Feel free to install and import new packages.\n",
        "# You may break apart your code to muliple blocks.\n",
        "#\n",
        "# Store your new NB model to my_model.\n",
        "# If needed, use new variable names so they won't override the variables\n",
        "# used in the previous questions. Otherwise, this might affect the variables'\n",
        "# values, which would impact your scores.\n",
        "#\n",
        "# For checking purposes, export your model (e.g., via pickle).\n",
        "# Be prepared to present this in case of any clarifications.\n",
        "#\n",
        "# Make sure to document each step of your ML process.\n",
        "#\n",
        "# Finally, print the classification_report of your model.\n",
        "# We will only check the accuracy, but the other metrics might still be\n",
        "# helpful in this task.\n",
        "# ------------------------------------------------------------------------"
      ],
      "metadata": {
        "id": "7ZJMUq-_23jr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Q6 [2 pts] Explaining the model\n",
        "\n",
        "Check the misclassified reviews and find patterns that could explain why the model struggles to classify them correctly. Write a short essay explaining your findings. Don't forget to cite your sources.\n",
        "\n",
        "<br>\n",
        "\n",
        "Suggested questions:\n",
        "1. Are misclassified reviews more likely to contain specific words or phrases that could indicate confusion or ambiguity?\n",
        "2. Is there a common length or structure to misclassified reviews, such as unusually short or long ones?\n",
        "3. Do misclassified reviews tend to focus on specific topics or themes that might pose challenges for classification?\n",
        "4. Are there any recurring sentiment-related patterns in the misclassified reviews, such as mixed sentiments or sarcasm, that make the model struggle to identify?\n",
        "5. Do reviews that include slang, abbreviations, or informal language have a higher likelihood of being misclassified?\n",
        "\n",
        "\n",
        "Suggested references:\n",
        "* [Rizal et al.](https://ieeexplore.ieee.org/document/10030009), Sentiment Analysis on Movie Review from Rotten Tomatoes Using Word2Vec and Naive Bayes, 2022\n",
        "* [Liang](https://cs229.stanford.edu/proj2006/Liang-SentimentClassification.pdf), Rotten Tomatoes: Sentiment Classification in Movie Reviews, 2006\n",
        "* [Hung](https://cseweb.ucsd.edu/classes/sp15/cse190-c/reports/sp15/049.pdf), Sentiment Analysis Classification for Rotten Tomatoes Phrases on Kaggle"
      ],
      "metadata": {
        "id": "7gTSDxYfZ2Xn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# ------------------------------------------------------------------------\n",
        "# TODO: INSERT CODE HERE\n",
        "# You are not restricted to use Python for this task.\n",
        "# You may use other software/tools such as Matlab, R, etc.\n",
        "# Excel/Google Sheets might also be useful.\n",
        "#\n",
        "# However, you still need to present all results in this notebook.\n",
        "# For figures and tables, you can embed them on markdown cell blocks.\n",
        "# ------------------------------------------------------------------------"
      ],
      "metadata": {
        "id": "oZr1GiRwljcW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Machine Problem\n",
        "\n",
        "Once you're done with the challenges, head over to UVLE and submit your machine problem."
      ],
      "metadata": {
        "id": "bsrW_BaVbXhL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# References\n",
        "\n",
        "[AIMA] Russell and Norvig, *Artificial Intelligence: A Modern Approach*, 4th ed, 2020\n",
        "\n",
        "[HML] Géron, *Hands-on Machine Learning with Scikit-Learn*, Keras, and TensorFlow, 2017\n",
        "\n",
        "[PRML] Bishop, *Pattern Recognition and Machine Learning*, 2006"
      ],
      "metadata": {
        "id": "q3oeqczSjZaW"
      }
    }
  ]
}