2 - 情感分析改进.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2-Updated Sentiment Analysis情感分析改进\n",
    "\n",
    "在上一本笔记本中，我们进行了情感分析的基础知识。在此笔记本中，我们实际上将获得不错的结果。\n",
    "\n",
    "我们将使用：\n",
    "\n",
    "- 打包填充序列(padded sequences\n",
    "- 预训练词嵌入(pre-trained word embeddings\n",
    "- 不同的RNN架构(different RNN architecture\n",
    "- 双向RNN(bidirectional RNN\n",
    "- 多层RNN(multi-layer RNN\n",
    "- 正则化 regularization\n",
    "- 不同的优化器 optimizer\n",
    "\n",
    "这将使我们达到〜84％的测试精度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据\n",
    "\n",
    "### 使用*打包的填充序列 packed padded sequences*\n",
    "\n",
    "和以前一样，我们将设置种子，定义`Fields`并获得train/valid/test分割。\n",
    "\n",
    "我们将使用*打包的填充序列 packed padded sequences*，这将使我们的RNN仅处理序列的非填充元素，对于任何填充元素，`output`将为零张量。\n",
    "\n",
    "要使用打包的填充序列，我们必须告诉RNN实际序列有多长时间。为此，我们为`TEXT`字段设置`include_lengths = True`。这将使`batch.text`现在成为一个元组，第一个元素是我们的句子（已填充的数字化张量），第二个元素是我们句子的实际长度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torchtext import data\n",
    "from torchtext import datasets\n",
    "\n",
    "SEED = 1234 # 随机种子,可重复性\n",
    "\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True\n",
    "\n",
    "# 先处理数据，处理标签\n",
    "TEXT = data.Field(tokenize = 'spacy', include_lengths = True) # 打包的填充序列 \n",
    "# 将数据拆分为离散的字符,tokenize无参数，默认用空格来拆分\n",
    "LABEL = data.LabelField(dtype = torch.float) # 处理标签"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们加载IMDb数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext import datasets\n",
    "\n",
    "train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后根据我们的训练集创建验证集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "train_data, valid_data = train_data.split(random_state = random.seed(SEED))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用**预训练的单词嵌入**。\n",
    "\n",
    "- 使用`\"glove.6B.100d\" 向量`（`glove`是用于计算向量的算法）\n",
    "\n",
    "现在，我们不再使用随机初始化单词嵌入的方法，而是使用这些经过预先训练的向量进行初始化。我们只需指定所需的向量（前25000个单词）并将其作为参数传递给`build_vocab`即可获得这些向量。 `TorchText`负责下载矢量并将它们与我们词汇表中的正确单词相关联。\n",
    "\n",
    "在这里，我们将使用`\"glove.6B.100d\" 向量\"`。`GloVe`是用于计算向量的算法，在 [这里](https://nlp.stanford.edu/projects/glove/) 可以查看更多信息。`6B`表示这些向量在60亿个令牌上训练，而`100d`表示这些向量是100尺寸。\n",
    "\n",
    "您可以在[此处](https://github.com/pytorch/text/blob/master/torchtext/vocab.py#L113)查看其他可用的向量。\n",
    "\n",
    "从理论上讲，这些经过预训练的向量在向量空间中已经具有接近语义的词，它们在向量空间中靠得很近。\"terrible\", \"awful\", \"dreadful\" are nearby。这为我们的嵌入层提供了很好的初始化方法，因为它不必从头开始学习这些关系。\n",
    "\n",
    "**注意**：这些向量约为862MB，因此，如果您的网络连接有限，请当心。\n",
    "\n",
    "默认情况下，TorchText会将词汇表中的单词初始化，但不会将您预训练的嵌入词中的单词初始化为零。我们不希望这样，而是通过将`unk_init`设置为`torch.Tensor.normal_`来随机初始化它们。现在，这将通过高斯分布来初始化这些单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_VOCAB_SIZE = 25_000 # 仅保留前25,000个单词\n",
    "\n",
    "TEXT.build_vocab(train_data, \n",
    "                 max_size = MAX_VOCAB_SIZE, \n",
    "                 vectors = \"glove.6B.100d\", \n",
    "                 unk_init = torch.Tensor.normal_)\n",
    "\n",
    "LABEL.build_vocab(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建迭代器\n",
    "\n",
    "和以前一样，我们创建迭代器，将张量放置在GPU上（如果可用）。\n",
    "\n",
    "打包填充序列的另一件事是，一批中的所有张量都需要按其长度排序。通过设置`sort_within_batch = True`，可以在迭代器中进行处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data), \n",
    "    batch_size = BATCH_SIZE,\n",
    "    sort_within_batch = True, # 打包填充序列：一批中的所有张量都需要按长度排序\n",
    "    device = device)"
   ]
  },
  {
   "attachments": {
    "sentiment2.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAACCCAYAAADok9QKAAAbUElEQVR4Xu2dX4hVVd/Hl2HplFEwRoUh5c1UFFPSRTHjzVDaE0Fg5FyMkiRh4p8YqueVZPybRaYNqWMN5pUKTSQEgTU33jhDV1kGT+lFIZKlpRGUf6LMl7Wfd8273bPPWb99ztm/vfacz4HImVl7/db5fM7Z+3vWXnufSVeuXLlieEAAAhCAAAQgAAEIFEZgEoGsMPYUhgAEIAABCEAAAhEBAhkvBAhAAAIQgAAEIFAwAQJZwQIoDwEIQAACEIAABAhkvAYyE9i/f7/5/vvvTV9fn9m0aVO0vf130Y/jx4+bPXv2mA0bNpiWlpaih6NWHx9qqEWF8CHCpNYIH2qoKVQnAQJZnQCbbfNk6AkpkFkXdudrHz09PU2hBh9hacYHPrIQaLb9VRY2zdiWQNaM1ut4zjaAdXV1mY6OjqgX+/OpU6fMiRMnzPDwcPS7kZGRsb/XUSrqe+3atVEX8+bNi8JWa2tr1S7PnTtnVqxYYdavX2/a2trqKV+KbfERliZ84IP9VVivgTKNhkBWJlsFj9V++rdBZ+fOnWPByB6ADhw4YIaGhqIAZEPT3r17ReGp2tOx/Rw+fNj09/dHpx+TP1fb1o5p1qxZE36WDB8FvyES5fGBD/ZXYb0GyjYaAlnZjBU43tHR0ShsuZ2OmyGz/3dryOxBadWqVWb79u1XzVDF13H4noKd5bKnHG2fbiYubZu0A6BtlyW8+cYS8t9r8VHrrGM1H3YcnZ2dY6iSM6T4qPz+sGwWLlwYsdu4caNoLabv/WHfF93d3ebo0aOpM8v48O+vfIzj+4UsbdOWeDSLj5D3paGMjUAWiokSjCMtVCV3MGmBzB10pAecSqEujsgddG677bZxs3E2INhxSU5xlgB7xSFm9XH27Flz6NChsYO+dP1fNR/2YLRmzRqzefPmaNY0jT0+0gOAFesuQrH/7u3tNXPmzPHO7FbzcfHiRbNu3TqzZMmSsRnr+EyzrYMPfyBzH1wkyy8k+yvH3X5wSe4Hm8VHmfe1WmMnkCmR/vbTbqVKtZe5519DVTdOO4D7AplbYzZ79mxz5swZ0QyAbwdnd2DLly+Prqb84IMPrjqFap9ApZmz+JNrVh9xBtIDgc+Hb7YAH/4A4GZ23dXL1d6IWXykscdHdR9u5tmui/XN0rv9TdpZgeT7wn5wsY8ZM2ZctR+U+Fi6dGntO3alLQcHB5UqTdwyBDIltzYA3NP5b6Vq2ct8O7LF+AJZ1hmZ+KJ6zVOWkqDR7D5cAEjOnqS9crKckkkLC/jwBzI7syWdIcviI+10GD4q+3AXBdkLg+yHSUkgk/hw61pt4LaP+G2CJD5sILNLRUJ92Ncugax+OwSy+hmKepgIAaCWNUsOTpZAlhYW0nZa9awha3YfWWZZpD4qhQrJGplm9uHW4EmvJJb4cCHBXvlcy5q+ZvXhbkPx+OOPi9axxvdv8Q838f2VXSrgTk1v3bp1XCCTvD8IZKLDbOkbEciUFE6EHVylq8jin/iki/olnwrjC9Db29vHruR0yioFMslVls3sw62/GxgYGLtool4fLowlT8dYV/jwz5BZTnEHx44d866D9L0/bJ9pszf48K/pu3DhwlWBrNb3x8yZM69a01dp2YfvqnACmdKBuuAyBDIlARMhAEgPrmlI02bIdu/ebebPn++9t1glRWmBLLnIvNK2zerDrb9ztymJ86nVhzvoL1q0aNyCdHzIdzDJDzO1+ohXTM5a4qOyj/gVr/FWboaxFh/xK17jfbqF/VIfBDL5+6jMLQlkSvYmSgBI3olcii8ZyGw/Bw8ejNbN1PpIC2TSO183o49qpylr9eFb+4SPyq/u5BKA+KmrkydP1vT+SN4YOekcH7K9TXJmsdb3R7JacoZM6oNAJvNW9lYEMiWDEyUAWFxZ14PVuo1PTTKQZQmLzegjbQYgy7qlNB+VZgDsrML06dPF3y3ajD7c+8Ldh6xeF85P0omb4eH9If++XclCfd/+Ke3v8UCWxQeBrBba5duGQKbkbCIdcJSQ5VoGH7nizdw5PjIjy3UDfOSKN3PnBLLMyEq5AYFMSRs7OCXQwjL4EIJSaoYPJdDCMvgQglJqRiBTAl1wGQKZkgB2cEqghWXwIQSl1AwfSqCFZfAhBKXUjECmBLrgMgQyJQHs4JRAC8vgQwhKqRk+lEALy+BDCEqpGYFMCXTBZQhkSgLYwSmBFpbBhxCUUjN8KIEWlsGHEJRSMwKZEuiCyxDIlASwg1MCLSyDDyEopWb4UAItLIMPISilZgQyJdAFlyGQKQlgB6cEWlgGH0JQSs3woQRaWAYfQlBKzQhkSqALLkMgUxLADk4JtLAMPoSglJrhQwm0sAw+hKCUmhHIlEB7yrgbYbsvUndeWlpaGjLAioEsfvPPtO/fakj1GjrJcjO9GrrPbRN2cLmhraljfNSELbeN8JEb2po6xkdN2HLbiECWG1pxx8kw5jZsZChLDWTJ0BNSILMQpF83ISZdZ8M//vjDbN++3Tz77LPGfrly2oMdXJ2QM2yOjwywFJriQwFyhhL4yABLoanEB4Gssoi//vrLfPTRR+aZZ54xkydPzs2Y/aqzzs5O475R4+zZs6a7uzuql/bdwLUMJDWQ2QDW1dVlOjo6oj7tz6dOnTInTpwww8PD0e/c13HUUjS+je177dq10a+kXx2S/L62esfQiO3vuusu8+OPP5onn3wyCmfJYEYgawRleR/4kLPSaIkPDcryGviQs9Jo6fNBIKtuwR5vT58+bVavXm02b96cizKXVdwXw7tsZPPLvn37TE9PT911xwWytC9stgM5cODAWAq0M1R79+6NZqpaW1trHkT8y3TtOdjkz9U6tmOaNWtWQyDU/ARiG3766afGfifdr7/+aqZOnWqeeOKJq4IZgawRlOV94EPOSqMlPjQoy2vgQ85Ko6XPB4GsugU7Q7Z48eKo0fnz56NJng0bNjRUnQtk8fCVFtLqKToukNlpORu2+vv7jVuoljxlaUPbqlWrosDR1tZm3Bex2tkz6flUyZe3uifb3t4+bkowGd5++eUX8/PPP9fDou5tn3rqKfPdd9+N9eOC2bZt28zFb//H3NP577pr5NXBtyNbzD8z/ztTOVEe+AjLJD7w0SgCzba/euONN6JjcqiP3t7eKBM06nHNNddEXdn/S/+7//77zU8//RRtd8MNN5hLly5Foezll182U6ZMqWto8fVj8UBmc4idiInPmtVTaFwgiy/mdx1XC2QzZ840VsacOXOi2SrperNkqEs+iXjgOnLkSNRvfEbOBsf473bt2mXsf0U+bMi0wfDy5ctjw5g2bZp5/vnnzfOPnQo+kD3zyn+KxNfw2vhoONK6OsRHXfgavjE+Go60rg6r+bCzPqEHMntMbtTjvvvuM19//bX5559/xP/Z0GRDmN3GPeykiA1Mu3fvrmtohQWytEBVLZBNnz7drFixwqxfvz6aLUsGpUoUfIEsfkoybTYt7dRqXcQbsPF1111n7AJD+7jlllvMHXfcYbZs2WIeffRRwynLBgDO2AU+MgLLuTk+cgacsXt8ZASWc/NqPjhl6Yd/4403GnuBhH3Ys3vXXnutee2118yyZcvqXuxfKZDlfsoy6wyZffI2jO3cuTNaT+YLWg5rtVOW7skvWrQourDA/exm4Wwf0uDn19iYFla8nba8+eabrwpirncCWWM4S3vBh5SUTjt86HCWVsGHlJROO58PAll1D++884559dVXzZUrV6Iw9vrrr0fLpxr5KGRRf9Y1ZPbSz/ipQ2kgs6CS68BcyNqxY4dZuXKl6evrqxjIslwA0Egplfq69dZbze233262bt0azYglHwQyDQv/XwMfurx91fDhI6T7d3zo8vZV8/kgkFUneNNNN0UzYm+++aZZsmSJD3dNf0/e9sJ2Ypdp2as7c7vtRaWrLG1xG5DsIx667M+VZsiSYS2NQvy2F27xvluXVm2GLLSrLL/55htz7733VhRNIKvpPVDzRvioGV0uG+IjF6w1d4qPmtHlsqHPB4GsOvbPP//cPPLII7m4cZ0WdmPYLGEneU+w5KlEu5hu/vz5mW+PUW0Nma25Zs2a6H4j9dx2I1d7ic4JZJq0/bXw4Wek2QIfmrT9tfDhZ6TZgkCmSbtyrUK+OinL1xMl13fFLwCw/Rw8eDC6CjPro9pVlqHdqV/y3NjBSSjptcGHHmtJJXxIKOm1wYcea0klApmEUvnbiL7L0vc0a7kPma9P+/e0+5BlCYuSGlpt2MFpkZbVwYeMk1YrfGiRltXBh4yTVisCmRbpYutUDGTFDmviVWcHF5ZTfOAjCwF7I9J7/jWUZZNSt+X9EZY+AllYPvIaDYEsL7KJftnBKYEWlsGHEJRSM3wogRaWwYcQlFIzApkS6ILLEMiUBLCDUwItLIMPISilZvhQAi0sgw8hKKVmBDIl0AWXIZApCWAHpwRaWAYfQlBKzfChBFpYBh9CUErNCGRKoAsuQyBTEsAOTgm0sAw+hKCUmuFDCbSwDD6EoJSaEciUQBdchkCmJIAdnBJoYRl8CEEpNcOHEmhhGXwIQSk1I5ApgS64DIFMSQA7OCXQwjL4EIJSaoYPJdDCMvgQglJqRiBTAl1wGQKZkgB2cEqghWXwIQSl1AwfSqCFZfAhBKXUjECmBLrgMgQyJQHs4JRAC8vgQwhKqRk+lEALy+BDCEqpGYFMCXTBZQhkSgLYwSmBFpbBhxCUUjN8KIEWlsGHEJRSMwKZEuiCyxDIlASwg1MCLSyDDyEopWb4UAItLIMPISilZgQyJdAFlyGQKQlgB6cEWlgGH0JQSs3woQRaWAYfQlBKzQhkSqALLkMgUxJgd3ChP5rtu/rwEQ4B3h/huLAjwUdYPmwgC/0xODgY+hCDHx+BLHhF4wd4+fJlM3v2bPPFF1+YyZMnl/AZTKwh4yMsn/jAR1gEGA0EZAQIZDJOQbV66aWXzK5du8yyZcvM22+/HdTYmnEw+AjLOj7wERYBRgMBGQECmYxTMK3+/PNPM23aNPP3339Hs2O///67mTp1ajDja7aB4CMs4/jAR1gEGA0E5AQIZHJWQbS0n/4HBgaMPfDYIMYsWbFa8FEs/2R1fOAjLAKMBgJyAgQyOavCW8Y//bvBMEtWnBZ8FMc+rTI+8BEWAUYDgWwECGTZeBXaOv7pf8qUKcySFWrDGHwULCBRHh/4CIsAo4FANgIEsmy8CmttP/1ff/31pqWlxTz44INmZGTEdHZ2miNHjphLly6Z8+fPs5ZM0Q4+FGELSuFDAEmxCT4UYVNqwhAgkJVE5ejoqHnxxRfN+++/bx544AEzadIkc+XKFfPVV1+Z5557zuzYscN0dHSU5NmUf5j4CMshPvARFgFGA4HsBAhk2ZkFsYULZEEMhkGMBWRQhEGA90cYHtwo8BGWD0YTJgECWZhevKNiB+dFpNoAH6q4vcXw4UWk2gAfqrgpVlICBLKyivu/U5YlHf6EGzYHnLCU4gMfYRFgNBDwEyCQ+RkF2YIDTlha8IGPsAiENRreH2H5YDRhEiCQhenFOyp2cF5Eqg3woYrbWwwfXkSqDfChiptiJSVAICurOE5ZBmWOA05QOrjIIiwd+AjMB8MJkwCBLEwv3lERALyIVBvgQxW3txg+vIhUG+BDFTfFSkqAQFZWccyQBWWOA05QOpiRCUsHPgLzwXDCJEAgC9OLd1QEAC8i1Qb4UMXtLYYPLyLVBvhQxU2xkhIgkJVVHDNkQZnjgBOUDmZkwtKBj8B8MJwwCRDIwvTiHRUBwItItQE+VHF7i+HDi0i1AT5UcVOspAQIZGUVxwxZUOY44ASlgxmZsHTgIzAfDCdMAgSyML14R0UA8CJSbYAPVdzeYvjwIlJtgA9V3BQrKQECWVnFMUMWlDkOOEHpYEYmLB34CMwHwwmTAIEsTC/eUREAvIhUG+BDFbe3GD68iFQb4EMVN8VKSoBAVlZxzJAFZY4DTlA6mJEJSwc+AvPBcMIkQCAL04t3VAQALyLVBvhQxe0thg8vItUG+FDFTbGSEiCQlVUcM2RBmeOAE5QOZmTC0oGPwHwwnDAJEMjC9OIdFQHAi0i1AT5UcXuL4cOLSLUBPlRxU6ykBAhkZRXHDFlQ5jjgBKWDGZmwdOAjMB8MJ0wCBLIwvXhHRQDwIlJtgA9V3N5i+PAiUm2AD1XcFCspAQJZWcUxQxaUOQ44QelgRiYsHfgIzAfDCZMAgSxML95REQC8iFQb4EMVt7cYPryIVBvgQxU3xUpKgEBWVnHMkAVljgNOUDqYkQlLBz4C88FwwiRAIAvTi3dUBAAvItUG+FDF7S2GDy8i1Qb4UMVNsZISIJCVVRwzZEGZ44ATlA5mZMLSgY/AfDCcMAkQyML04h0VAcCLSLUBPlRxe4vhw4tItQE+VHFTrKQECGRlFccMWVDmOOAEpYMZmbB04CMwHwwnTAIEsjC9eEdFAPAiUm2AD1Xc3mL48CJSbYAPVdwUKykBAllZxTFDFpQ5DjhB6WBGJiwd+AjMB8MJkwCBLEwv3lERALyIVBvgQxW3txg+vIhUG+BDFTfFSkqAQFZScRs2bDDr1q0r6egn3rDxEZZTfOAjLAKMBgJ+AgQyPyNaQAACEIAABCAAgVwJEMhyxUvnEIAABCAAAQhAwE+AQOZnRAsIQAACEIAABCCQKwECWa54G9/5uXPnTE9Pjzl9+rQZGhoybW1tjS/SpD3u37/fLFy40CxdutT09/eblpaWukgcP37c7Nmzx9j1TPX2VddAJsjGmzZtMl1dXaajo8O498Hw8LAZGRmJfpf2sA66u7ujP/F+adwLIc7f9rp69Wrz5ZdfmriPRr+fGjd6eoJAmAQIZGF6qTgqAll+whp5ABkdHTWdnZ0NC3f5Pevwe7548aLp7e01g4ODY+GLQFasNxuO165dOzaId99913z88ccEsmK1UL3kBAhkJRNIIMtPGIEsP7b19FxPIKunLttWJuAC2b59+6IZ+7RHI99PuIBAMxAgkJXMMoEsP2HxA8gLL7xgFi9ebI4ePWra29vHne5yM2BuNPE2rp/4SN2BK3mqp1GnR/OjUmzPSV52NPPmzTM7duwwK1euHJuROXTo0NiMzcaNG01fX1808EqnLJMzPPFtin3GYVdP82Ff+++9955Zv3591RmyCxcujC23GBgYMNaBPcXp3jvTp0+P/h7/HUsywn49MLrGEiCQNZZn7r0RyPJDnBakXDUbAuzfW1tbTTKMJdt89tln0Vq0ZCB76KGHovVMNuTFH/G+83t25exZEsjSnpkLwGmBLBnG3PaEMv9rpBGBzAau5GPu3LnmzJkzV703eF/4fdBiYhEgkJXMJ4EsP2HxQOYWirsDug1R7nfugJ5sY0fmFo6nrSFLO80jOfWT3zMuR8++U5bxWUbH0/3u5MmTVy3qnzlz5rj1aM4VAUD+eki+btPW9CVPWboZMhvIXPiNf7hJ/i5tZlo+QlpCoHwECGQlc0Ygy09Y2pqXaovH40HBjip+AEkGsvjBKO0ZMDtT2asvkMWvskw6TAYyewosOUNW7SrN/F5t5e653kDmmKe9v9jHlfu1wehrJ0Agq51dIVuys8oPuzSQVTq1WU8gYy2ZXiBLO+1mq+NA/t6qJ5DFb9mTtj9jHyf3QMuJRYBAVjKf7KzyEyYJZHfffffYwmM3q5W2TqnaDBkzMtkcNnqGLF49OcuJG5kbApmME60gkIUAgSwLrQDaEsjykyAJZPZKsOSNRt12aTNk8XVJyfVN9pm4+2tVu31Afs+4HD3HQ1Pa1apZTlnGr+Rz26X1Xw4yxY2SQFYceypPXAIEspK5JZDlJ0wSyOIzZMmRpAUy18bOpi1YsICrLGvQl7ZWL+02C7ZryRoyySnnGobZVJsQyJpKN09WiQCBTAl0o8oQyBpFcnw/kkBmv6InfuWlC2EffvhhdB+s+ExXfPF48vSmu/UFV/bJfCaZ1xPI4sHNVceDzINrRSDLxovWEJAQIJBJKNEGAhCAAAQgAAEI5EiAQJYjXLqGAAQgAAEIQAACEgIEMgkl2kAAAhCAAAQgAIEcCRDIcoRL1xCAAAQgAAEIQEBCgEAmoUQbCEAAAhCAAAQgkCMBAlmOcOkaAhCAAAQgAAEISAgQyCSUaAMBCEAAAhCAAARyJEAgyxEuXUMAAhCAAAQgAAEJAQKZhBJtIAABCEAAAhCAQI4ECGQ5wqVrCEAAAhCAAAQgICFAIJNQKqCN+6qYp59+2vT19UUjcF+bdOedd5r+/n7T0tJSwMiatyROwnKPD3yERYDRQKA+AgSy+vjluvXo6KhZvny5GRoaMm1tbcZ+f5z9nf3OxdbW1lxr03k6AZyE9crABz7CIsBoIFA7AQJZ7exUtrQh7NSpU8bOlM2dO9eMjIwY+wXXPIojgJPi2KdVxgc+wiLAaCBQGwECWW3c1LZyp2WOHj1qNm7cOHb6Mj6Aixcvmt7eXrNo0SLCmoKZak7if7NDqeRMYZhNU6KaD3eaf3h4OOKxb98+09PT0zRsiniikn2WHZed3bRhmhn/IixRM0QCBLIQrSTGZHdaBw4cGDt1mRbGBgcHmT1TdJnmxB387Zo/O4vpfrZBmRCQr5w0H+6Dypw5cyL+LigMDAzwwSVfHVHQqrTPsqXde8P+m0CWswy6Lw0BAlngquynyM7OzmiUS5cuvWoxvzvAPPzww+bEiRPR7BmnM/MXWs1Jsro9MNmHuzAj/9E1XwWpj2RAaz5SOs9Y4sOGsE8++cT89ttvBDIdLVQpAQECWcCS3KdIG7IWLFhguru7zSuvvDI22/LDDz9Eo7dXW9oZAAJZ/jJ9Tghk+TuIV8jiw36AWbVqldm+fXt0kQyPxhOQ+LAe9uzZYx577DGzbds2AlnjNdBjSQkQyAIWZz9F7t27d2yHZX9+6623xp26TJ4qC/gplX5oUif2iXKKLH/dEh9uZsye1mdNX75OJD7srHFXV1c0ENaQ5euD3stFgEAWqC837R9fhOwOLPb0ZHzdBYFMR2ItTuzsJqcr8/GTxYcdgXv/zJgxAyc5KJH4OHbsmDl06FDEn0X9OUigy1ITIJCVWt9/B08gC0ti/LQNYSwsN4SA4nzYQLxu3TqzZMmS6JQxLopzQeUwCRDIwvSSaVQEsky4cm3MlZW54q27czuzfPjwYb7pom6S2TtI3hLG9dDe3p56BXn2CmwBgXITIJCV2x8zZAH545RYQDJipyjdbS8Iy2H5YYYsLB+MpngCBLLiHdQ9AmbI6kbYkA4qzQAkb1fSkGJ0IiKQvDEsi/pF2FQaEchUMFOkRAQIZCWSxVAhAAEIQAACEJiYBP4XULiTEHzZRVAAAAAASUVORK5CYII="
    },
    "sentiment3.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAADVCAYAAAA7HtUwAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAFiUAABYlAUlSJPAAAAYLelRYdG14R3JhcGhNb2RlbAAATVbHsoM6Ev2aWzWzuLfIYQmYYMA2yaQdyeQkMl8/YvymajaidU6r1WodaH5wod0/ZZP9YMgyZYDLs27+wW8/GPboz7Jpoh9MIv8QSP/LK7u03yZoPh04oAiEcR5akKCIr7lTxL/hgxuGJvOyWCvnaz1O/+HUFUJTnIf+gwnQbsr62lTOkrq/lggF6FuISBT1h/zhOMn+oTgLCTv6RKD8vzAYBNcMTGXffTNl/7A/+otnaTn34Atv2/aXgmj7K/svOR9D9qXSbC2T7EJx8QcX0jLKQdRClzL9OiQExUQMQf7iJI39xhiL/SI4if4mcZShaJYSCJp+g3ZR+09QI8qzX/QblA7nNKYehHQPYea3MDu5fh+SuWlfYALnuwdp5KWL25MEx9G8nD2Qw6EpNpFbWGvese6m8fSQcfb5KcCNF0PSQVEM+ezgkLZncYFh7wbxuoUTcjIRoT8L7KBEiOrJ1in3Wy6FiyBXNUlFGmH71A1ZKi3C6eORDsC+V6ocW6ClEVtV84p1p1JSj1H6xLVeZdRGSIvsEox7S+aVSK8tzwcw11jP4Jklws7vbgJRb2F2HZ0qWnSFw8EsyBU5gmF2suKliuppc5NL7aMoqpNeGeeYyQE7r/SKjhUpoKSI9OMkT1nKzue9RvdtJAeO38hbF9osQbJ0PpB2mev2m7MlWGTeR8cPlKAEBCKfOsME0ROikXKRkrgHUG987Vk7nEnMJSdTkb2HAqhhHNWa1iefOQPb0Ti237c3260uomLT0BpIdQNueNbbHsLk9pP0tGMgqYn3NlWk7nwze5z7KVOCOFVpNSzG6d87UUgA7uGD+9Z476cJumQyBoWhPwJCfxB/V8qJkSdqvQ4fmFZ+wCctcVcRuafvzXjaHuXLPcz2KU/6557762TMWmTUPEW99deGJk9ciKDq+eDRbQxNZ5iBmTdG4NCs0ewmy5R3Wb6e97db2VXsWJRIE7PXfMZxVgFFcbsx4m9jfgJd9hoU37M5HGYvO2sXZGGKyOBpv9abpjttIQzZSNvRxKqjzi6I1xjWWA0wVYy2ouBGRsCVWq4J8RhgPJqjaqvt7ZKh+fHxeLUpWrZshTRo+tOz+JFA4E1eNxbDgVivIz8RfTXS2VzT5CDBPb2gmcVzGTnv7FGcxfl5PrHqpJNTetK5srboaQFSaT2MiWIqhc6bDqhLSjxPPy65fYKbKF/vRNoih7/gmAeuT1bnmV1guWwlp7FOPuhNMe4iHUBmRL2covmE4e/y1mJvQkHpwC8mypiW9qO+NLYpnvywU0MwtIs+KMAE4WhpYrA3Hp+lgkkn4WPSi5ttoyUNtcizmTiaWUY6LeiW2s02/Int+stS8s8R+54333QGCJckraXxdG1adkohP6h+Hx17bdPlOa7eiRzzzlXbIqNBOwoHUwge1bvqYCrcWzXg4jehhlXcEQ5zzy95V7Say9BormqMzMKxNr8OgINQOd1Zx2nryZnvtpOdfbO+M+2mYwoqgiyurcEg+mVQeaIs1gXgBmlmCYuimjnyeFYwk+gz3ejmWFTvzYuM+xS3VU8/cqMVilCOPmzC+aAu1002Gu2ejjUVmX3IFp65uPqdUKZGsdtajrr93ZcUs+Ynb1NB7MPc0NfOnEVI6leeTXMbotsB9nPH/cSPsngmK2TsgkjDD5z4cAxibiU1iR/o7dJBF8TlcLwLf1FbBW+lsoJ3L1Xci3mwULDQ1oDaTZ6WgLdK43jdUxH6MIZpENmjWzrkPKHP4vXUY1a80HhY/L7Pz0Xvw/LBlDVNVFjHhdtrGUXualDSROl0Idi8cy7mEIW7KxZKWPbPziTCB29sk8okQo2stWslLrGdFhWUah90sMjvbnP3mHcQshNk9nqHtzdmaFcr00p2MqlVWTKdVESa2Y/NMwxc1ESkgq1MOvEdtLigXEqPUT/eaQPFx4MQsJEZvAP/ABa3skVwmt7PJxQ3p9MK8/FFl43v+LzAv1g5RshKA8qbElHlzV06ZVLjMM7bcajXRZCmdh1wS672hcOeLP2vD/63KcL5P78GuPgfCVuq7QAAGhFJREFUeF7t3U9oG3me9/GPn84fuTsHB2aCfFNuMrtg5YEFNzwH+zBtBfbBaraHtWFAahqedg7ddhhYOwybdmbpWEkfrO55wM7DQGwYkHb6aWzzLMQmAevwQAR7sAy7SLfoZqVpiA+hrST9oKeq9JMtJ7Yju2ynVH6/hhr96ifJtL5R1Ue/X5VUbVWLAMBHPv/8c9Pyrnv37pkWjhrBBsB37GCbmpoya95z/fp1gu0Y/RdzCwCALxBsAABfIdgAAL5CsAEAfIVgAwD4CsEGAPAVgg0A4CsEGwDAVwg2AICvEGwAAF8h2AAAvkKwAQB8hWADAByL4tKMYtGoYrGYotGEMvmyued4EWwAgCOXn42p6+o1LS4va3FxUcvLcxq60qlk9vjDjWAD0HIePnxoWjiMY69fJa/xTxfNSlzp9LS6zdqNvlkdd7QRbABazieffKJAIKBkMml6cBDHXb+NYlbLpp0uzGhwcFgL8yOmJ6PiMScbwQag5Xz33Xd67733NDExofb2dgLugI67fuV81rTiioQDTisU7nFupTWVNiqmfTwINgAtJx6P6/3339eLFy9UqVR27KBfvXplHoW9HHv9Ah2mYalnWCisAdM8bgSbR926dUttbW3OYrfr6K+hv+Y09/f29ppebe2gb968qc8++8z0elv9dbyr5aeffjL/JcdYv/5eBWsDNivsAqrHXf6Y5yLbqhbThgfYG+5XX31l1gDs5de//vXWzvncuXPOztredn7/+9/riy++0NTUlHOfF12/fl337t0za+/GfvWz190oZhLqGpqzWgNa3VxQxA63jawiF/u0ZjXThU0NminK48CIzWPsKQEA+5ubm9PPP//s7IDPnz/vbDf2iOPGjRuud8qnwUnWzxqo1ZTLTqjZjvcIG8EGoAV9+eWX+uWXX3bskNG8465fOBI1rUXNZIpOKztbP0GlX5HQ8Y3WbASbxzBiA97u+++/d44LEWiHc+z1C0c1Zb649u1QlzPN2XfHjNdGxmtTk8eIYPMYjq8Bb/fRRx+ZFg7j+OvXodFcQWP1b2XXxdNaT22f9HNcCDYAwNELhJXMV7W+/sRZnqw/U3V2UEFz93Ei2Dym8dRmAGh1wWDIWULBhu+2HTOCzWM4xgYA7hBsAABfIdg8hhEbALhDsHkMZ0UCgDsEGwDAVwg2j+GsSABwh2DzGI6xAYA7BBsAwFcINo9hxAYA7hBsHsNZkQDgDsEGAPAVgs1jOCsSANxpq1pMGx5gX7eIfxLAnc8//9y0vOvevXumhaNGsHkMwQZ4y6tXrxSPxzU3N6ezZ8+aXngZU5Eew1mRgLd88803+uGHH3T37l3TA69jxAYA+wgEAnrx4oXOnTvn3ML7GLEBwB5u375tWrXDBF9//bVZg5cxYvMY+6xIvssGeEN9tFbHqK01MGLzGI6xAd7QOFqrY9TWGhixeQxnRQLecP78eZ05c0YXLlzQjz/+qEuXLun58+fOWZIvX740j4IXMWLzGEZswLv36NEjtbe3O981e/r0qdNn39rrH3zwgR4+fOj0wZsYsQHAWzCT0loYsQEAfIVg8xh+KxIA3GEq0mOY8gC8h+2ytTBiAwD4CsHmMZwVCQDuMBUJAG/BVGRrYcQGAPAVgs1jOCsSANxhKtJjmPIAvIftsrUwYgMA+AojNpf+5d8+Ny3v+ue/v2da3kP93KF+7jRbv5v//X/pj//nf5i1k+Xl+nkVweaSvWH8w999ada854d//87zOxbqd3jUz51m6/c3wb/Vf5b/w6ydHK/Xz6uYigQA+ArBBgDwFYINAOArBBsAwFcINgCArxBsAABfIdgAAL5CsAEATlBFpWJeueySZpKjSi2VTP/RIdgAACdoQ/lsRsN9V3XtxrfqCAVN/x7KeRXLpt0kgg0AcIKCig1PaLjfbsfVEwo4vbvayCnaOaFKh1lvEsEGADhZVmDNLFu38aj2yrVyPqO2ix+q90FSwY2ySqXSrkt5o2KesY1gAwCcqHI+pzXrNh7t0VauNeZTeUmdV4ac5o2rXers7NTly5d3XRILbx6jI9gAACequJRxbhPRkCrFJcXa2tTW3qZIMuv0KxhVtbqu6X4r/NIF51p4ey1LiXDtOQ0INgDACSpr6Y41XuuflPIzau+6qt7707IPua1lSw0Dt6CGlzYVne1SpvjmdON+CLZT7tGjR6aFw6B+7lA/d1qyfuWc7ti3yzfU17eglWdVjcZCsg+5KRzcnpp0BDS4VFUsvM8JJrsg2E6p27dv6/z58/rtb39renAQ1M8d6udOK9evnMuZlpQuLKi3Y3tqMt7z5rSi7WCxRrCdOvYGEQgE9Mc//lFnzpzRn/70J3MPmkH93KF+7vihfrkFZ7wmjaxo0BmJVZTPzDld0Z6Qc+sWwXYKvHr1ascG8eLFC2e5cOGCfve735lHYS/Uzx3q546/6lfUgpNhA1pN9jo9qpSUWbQbk4o25prdP5NSMjmjbHHDdDaHYDsF4vG4JiYmtjaIut7eXrXZZyNZy61bt0yvnPZJ9bcC6ucO9XPHy/Vr7G9GpZiXk2sjw4qY+cVKKSc71wbSMVVyM8rkK/YDlWi/rFwoptHhsEa7LiqVaz7c2qr2+ZI4tH/5t8/1D3/3pVnznh/+/TuN9/9P3b171/m0Z3v58qVze+nSJT19+tRpvyvUzx3q506z9fub4N/qP8v/YdZOjtfrd1DFTEJdQ3OafPxM4z21nxMpZmJW36IGRuJa/HZDq5sL0mxEV64Na7M67BxfK2fH1dknrVeTessPcDkYsZ0CZ8+e1R/+8Afn097Nmzd17tw5Z3n+/Ln+8pe/mEdhL9TPHernjp/qV8rZ47UBRSONv5FVa9uhtrK+YI3kNpS7tqb++9tf3g4GI9b/31G2ydP+CbZTpnEDsefuv/jiC3MPmkH93KF+7rR6/aIp+0vVdniZDkt4cFabm5tOf689HKuUtWTdBBrOhazHWbPfZiPYTil7A7GnNP7617+aHhwE9XOH+rnjt/rZJ8bsJxAMqdu0m0GwnXK/+c1vTAuHQf3coX7u+LJ+gaCiVopVGrKuUi45vy3ZLIINAOAhAXUEpeX89kXYagO6boWD+4/s6gg2AICHBNQ7OiLdSal+rkh2ZkgaGNeOc072QbABADwlGE3p8XRFXT2jmklG1XdnTIXMYMPpJPsj2AAAntMzvKTN7IRiwxlVq0kd5HeQ+YL2HgoP/tG0vKv6X2+a1t7sL3j+89/fM2snh/q5Q/3coX6nG8G2B3vD6Ppv/2TWvKfwf+96fsdC/Q6P+rlD/U43piIBAL5CsAEAfIVgAwD4CsEGAPAVgg0A4CsEGwDAVwg2AICvvBZsFZWKeeWyS5pJjiq1VDL9AAC0hte+oF3WwkxKE9fuOJcIuF/YVOIgv2NiPT9fDCgSbvKXKj2sFb7g+b//X3N15guyb6J+7lA/d7xev1a3yy+PVDQTbde15bgKm7MH+n0uaUOpyEV1zD5TotmfYfaoVtgwuq7+q1nzHurnDvVzh/qdbm8eY9vIaWbZuo1HFdot1CobKpVKuy/ligZn0/r0ykUlF/JNX8YbAICj8kawlfM5ZxoyHu3ZvkRAQ0JVSgu6fPny7ktnpzqvDDmPu/HxFU1kty8UBwDASXgj2IpLGec2EQ2pUlxSrK1Nbe1tiiSzTn8gnJA9e7nXsrl633pUXI/Xq0r2Bp3nAABwUl4LtrKW7ljjtf5JKT+j9q6r6r0/rX7rnrVsqYmpxaIGr2RVqM6qh0wDALwDO4OtnNMd+3b5hvr6FrTyrKrRWEj2ITeFg01cvTSsjBVqYbPmZQ8fPjQtHAb1c4f6uUP9sJ8dwVbO5UxLShcW1NuxPTUZ72kurg50EuU79MknnygQCCiZTJoeHAT1c4f6uUP9sJ8dwZZbcMZr0siKBp3z/CvKZ+acrmhPyLn1i++++07vvfeeJiYm1N7ezgZyQNTPHernDvXDfhqCragFJ8MGtJrsdXpUKSmzaDcmFW3MNbt/JmW9mWaULW6YztYSj8f1/vvv68WLF6pUKjs2kFevXplHYS/Uzx3q5w71w362gq1SzMvJtZFhRcx8YqWUk51rA+mYKrkZZfIV+4FKtF9WLhTT6HBYo10XlcodPNxu3bqlNvuMy3e4/PTTT+a/RlsbyM2bN/XZZ5+ZXm/b7TWd5EL93C3Uz93S6vXD8dkKtlJ+ybmdHOxxbm2l/EKtkUuq88MlhcMB5WcHrQCcVtIawgU6erW0MqbrHyZ10G+sffXVV7t+XeAkl1/96lfmv0Y6d+6czp8/7wTun//8Z9Prbbu9ppNcqJ+7hfq5W1q9fjg+28GWs8drA4ru+CmsWnvx2w2trC9YI7kN5a6tqf/+9pe3g8GI9f93lC221u+MzM3N6eeff97aIOypDPsT340bN5w+7I/6uUP93KF+2M9WsEVT9qcgO7xMhyU8OKvNzU2n3/mudaUse1wXaDj3sR5nrRVr0pdffqlffvllxwaB5lE/d6ifO9QP+2k4eWR39im1+wkEQ+o27Vby/fffO/PybBCHQ/3coX7uUD/s563BtkMgqKiVYpWGrKuUS85vS7aajz76yLRwGNTPHernDvXDfg4WbAqoIygt57dPFakN6LoVDu4/sgMA4CQcONh6R0ekOynVzxXJzgxJA+Nq8cuvAQB84oDBJgWjKT2erqirZ1Qzyaj67oypkBlsOJ0EAIB358DBZusZXtJmdkKx4Yyq1eQBr7INAMDxOVSw2QIdHQpaCwAAXnLoYAMAwIsINgCArxBsAABfIdgAAL5CsAEAfIVgAwD4CsEGAPAVgg0A4CsEGwDAV9qq9jXW8YbCg380Le/quvqvpuU91M8d6ucO9TvdCLYT8OrVK8Xjcedy9mfPnjW9aBb1c4f6uUP9Wg9TkSfgm2++0Q8//KC7d++aHhwE9XOH+rlD/VoPI7YTEAgEnMvYnzt3zrnFwVA/d6ifO9Sv9TBiO2a3b982LetTRFubvv76a7OGZlA/d6ifO9SvNTFiO2b1T3t1fOo7GOrnDvVzh/q1JkZsx6jx014dn/qaR/3coX7uUL/WxYjtGJ0/f15nzpzRhQsX9OOPP+rSpUt6/vy5c5bVy5cvzaOwF+rnDvVzh/q1LkZsx+TRo0dqb2/XvXv39PTpU6fPvrXXP/jgAz18+NDpw+6onzvUzx3q19oYsZ0QewqDUh8e9XOH+rlD/VoLIzYAgK8QbAAAXyHYAAC+QrABAHyFYAMA+ArBBgDwFYINAOArBBsAwFcINgCArxBsAABfIdgAAL5CsAEAfIVgAwD4CsEGAPAVgg0A4CsEGwDAVwg2AICvEGwAAF8h2AAAvkKwAQB8hWADAPgKwQYA8BWCDQDgKwQbAMBXCDYAgK8QbAAAXyHYAAC+QrABAHyFYAMA+ArBBgDwlfcmLKaNY9TW1qbe3l6zhoOifu5QP3eoX2tpq1pMGwCAlsdUJADAVwg2AKdUxfof/IhgO3YVZRJtzhx9WyJzijekokbtGlhLpui+ChvlssrWsmHWT5XKhvPay+XtV1/KjDbxHrPeizHzXoyd5vdiRdlUzKpDu9qtWkQzJamcqdWlLaHa23P7/Tpq34+WQrCdhPr+p2xuT6n67sH1DrWS12BnpzqtJZM/fbvnYmbQee2d0YWtWlbqb7K3vcfqTyidvrptKS6o7/qiWXldebtE9dvKKa5ViyLYcGI6zK0UMLeHFOhQ0DQ7Ai7/VgsKBMyrDwV2reTeFQkoMj6vdDqt+VSv23+FllWxRrw13UqvPtHSYMgqTUTzVl3S8xN2WR3b71e0GoINJ65SziozHlMkGlU0ElUyk9/6lGyrlPNKjdpTRW2KRCKKxkaVyZrHbBSVSU5oznmklJlNaXYhb9YqymaS1t+sPS8STWjGus9Pn7eL2YwmkubVlzJKzcwq3zhKC2woZz0mZtU1GrVql0gqV96uQKU+hbuxPY3p1DsRNVNxVu2sus1mi+Zef9nIW7W5cs2srSmfyypbtGthjXk3KqpYddnr/VK26jo+nlQqk1U+O2vV2HqPWctoKmuNlyvK2e89+z1tLclMzlfvu5Zjn+6P47RZTQ/I/kpFVf1pa+20KlTjdg32WAbuF2oPW3+w6/320j31uLpZmN7lvmmrrs+q0/2v95tl7EHtb7e8zer0Lq9vanWzWkjH3+jfXgaq1kOc56frNeq/X3svrq+89tjtZevfxEcK091vvtZp63Wup816v6nV9vu1XodCemD7OU0sA2n/1a9VMGLDO9Cv+ysFFVbT6jY9i+YTbnEhVetwponWtfmsoDHzoLUlaxQRTuhJw/Mm51e1vj6oSnZG15ZrfSP3V6y+gtKTA7WOO1flj+P/ASWePVG6XpD+Sa0WnigR3jmp2D+SVsF+/WP9pmdR+foJO1sPrTWK1sijplsPnmxaH3TXdd+UbfHTrO9GHaHBJa0+mDRr/ZpfLehZIrx9HNyqy95TtNuTk/HpFa0+3n4f2uy+QkPf4iyjtneFYMOJ67+fUqI3rHBkUMkR02mOlYViM1pdXVE6nVQ4UFYum1NxzbnLYj8moJA9BVTrUCgcVjDYoXwuY3q61RMJyp5pC/ds/1JEJuePqbVAR8h6febVB0KKhEPq2LEnHlAyNahwMKzB0WHTZ9lrb711vGlNV2PWv8dsVsHEvB5bgflsM7HPTr41BTqCioRC9TXrPRh26newAJpSarhXkZ5BpeofMrqnnb6w1TduPhjU39M4eQQbTlzMCp66UE+81tgaUJQ1m+jT0NBVXem6or6PP7XGG3urPa2icraefmsautKlrq4uXem7bvqk0oZ/PjtvvRKr8car6o+qvtuWFW6munsKR4e3Rx1ri7rx6ZCufvyxPuy6rJklf57mvqNmh3lbDAS3Aj9Y/5AR6jB9DX/wMH8bR4Jgw4kL7PlJtqLM8If61smofk2nV1RYX9eDyfqU2pvqf6l+oqA9YllZXbVGfTuXTCxs7veR3coYqO9gmxSMKvfsiVYepDU9ObJjau3Gx+PmO13YYbeaUCdPIdjgSd2TExoe7FU4uKGlG+bg2S577KW8PaoIWCOP+tikpLI9RWd9kg6pqInxCY0nxpXfPgnQPxaX3vw62oF2sBXNRtrUfvGy+lIVDY6nlK9WtXp/e5zH/hqtiGDDiTlItqzd+FDRWEyRti59a/rU8EXZ+t+aG+pyfkUjFB22xni2NQ11XXROW794ZUiLy4tatkaA4fqXk3xh69Wrq93NL7kEFEmY0fDyp7rYFnFOVb/yaf3LFGEF/VS2XexWuXqfHz8LnRYE20mon0zl853E2+xbBqczoFjqgQkoa1+7uKi1gUlN189utHa+C85OPKzx6foReosdeB09Wlh/rK0TAeu6x/R4fUkRH9U+HB1Vw6vfuXPe43Vuddf/EUxiRUYzmq/X1/pQsLxsRsfdI1p5MrH1RXh/2m3adruvXqqdJ+dY3nxSQ1/APqen3sQ7wmVr4EEVbZQ3VAkEFOzYPsV6NxUr1F4/ZlfZKKt2roj1/OD+z29p1mu3Xv3RnHxn//5k/QQb+5dd3tibA62DYAMA+ApTkQAAXyHYAAC+QrABAHyFYAMA+ArBBgDwFYINAOArBBsAwFcINgCArxBsR6ak1GhCo6Mp5Rt+4yifSVl9o5rNNl6/H2+ifu5QP3eon6/YvzyCo7E63V+7LHx8vtZRqF9ufqRacC43j/1QP3eonzvUzz8ItiO1Xp3qtjcEVSfnH1Qn+2vtNFtFk6ifO9TPHernF/xW5FErLynSeVX16zkP3F/VQsJcZbdBKTujrGJK9Pr799MP7G312yhqdsaqXVkKBiMaTAwq4vdrqxzEW+pXKWU1M7OgsgIKRXoVi0V9f2maA2ly+7UvapMZn1AgMaFY2Mc/tN2qnHjDkVoZqX3Ss6cwnpi+Rpvrj6sD1v3d9wumB432rN+zx9V+p7+7OjY5Vu122nyift1e9dvcmlobqI6NDJh2nGm217xt+7WtTtfq1z+9anrgJQTbUVtfMTvf2hJPN4ZXwQm0+n0DBNub9qlf4b59DCReXa3viDcL1bj1mG52Ltv2qV9thz1VXTfrm4V55zHTWwXF/ttvzeZq/QPC7vfj3eOsyCO1oZlEn+xLNU6l07IvsG9f4Xmh5NxpCSm5uqpC4bEmreFGw8lXcOxfP6de/dHti4YGQopae6G13FaBT7n96mdVLzSiqZXBrYuHBsw1147kem6+8Lbt11LJa/DKkOLT08799UvYwWNMwOEIFO6b6Z2RB876s5Up88lucutTcl26X9V+Rmw7HKR+tsL8pHP/yIPd7j19mq3f+sp0dWQkbqZyJ6vPTP9p9/b6bVbTA9Z6f9pqPXFmC9iGvYlgOyrPVsyOomGqzPJgrNvZOHZOWVgbCMG20wHqt7m+Wp2ydzD2jmXygVVNHKR+zwoPqmMjI1tTbukC0dZM/Z7Mj1jtAXNM8olzWGEgvddROLxLnBX5TlSUibZrdrCgpUTY9KEZ+cyorgx9a7XierCaVDTCWaXN2iiXFQgGtTXzWClqtL1L2elV5Yd3O/MP24qKtXVpUf2amo6pUsopc2fOOXtyan5VozHq5yUcY0PL2MilnFCbnC+oWp0l1A6ikldvZ6dis0XTYQmEFLGGKRxka0aHYmNjmpzsVblUslaDtWOV/QMqlzc4Xu4xjNjeCUZsB2fVLNauocVuzT+eUUfF7Eqs245wryIhds77q2gh0a6P5wY0X5hRNCTlFlLqG7qjsZV1Jfk+5YFlom2aTVjb8CDbsNcwYnuX2Bcfwpo+/vBD9fX11ZarVzW+1DAKwR4CiqVWNda9qI+7OtXe3umEWnx6RROE2iHUP1jVbuAtjNiAU8Y+1uZMPwY6ZM74B3yFYAMA+ApTkQAAXyHYAAA+Iv1/BZdk74b5jWcAAAAASUVORK5CYII="
    },
    "sentiment4.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf8AAADVCAYAAABZsflaAAAGDHpUWHRteEdyYXBoTW9kZWwAAE1Wx9KrPBJ9mls1s7hfASIuydEGTPaOIHKwyfbTj/i/mapZAOrTounWaR3xB4jDWTY9/ENg2wJnvoLj+gdIfwjiNn2bvk//EAr1gyH3v6JmLKZjQcO7j244hmAgoBFy0OTv8KTJf6MH/3r1MIKZ2azX+4D5AfQVwtT8m/WHENG4b7rroyrMu+l6RaznaUCIQtM/2A8AFPeDAw45vLRM5+b/whAI3OG8NNP4myn3Q/wwvzgsmnWaf+HjOH6KOT1+munXuX5e8NdVwL3J4YUC+Q8Qiyat5nRAU5rid0JO0mzKktRfQDHE34zgiL8YoPC/eZZCHIcFieHFb9AxHf4b1Ekr+Bf/DcqEa9FSC6k4GA7hYd4E6UMYve3Q3rbezWW1gf+pZAcTzNxeyG5hUKlClvGSakKjkdTirh9YRbOo7krbHbto4y//zWdkJ6WonGOJ5uuy4juOVyZfVmUGWN6mJ1pdgSLTgKi03lVx0zOY2Mjcj4PwmNNVAa8BTAxidNNgOqzhjXDmrJvYSF6v80iMI1u35VPkhYk8dN2Mb/VBtZHjXUwiyDW+0Wum9OiObM8xMeyEhIvFRY5558r51Qz77c7ceNQzShiEcJ+E04pV1T9cf097B2dsoRY20jhnF8WL0AX2IFcKk8MjG1kt1xWzVD7uATbf6MaqF1pf2eLlHiFZNUbrS6ZH1WwhJAw9jy445JoKPlszJ+98tTGfHmN8HmMw+M5hQLPpKRTUM+hQjwam3hpODiteJV/PdzZsL/aMp+UxAsF4+DdA2RFr2tRZBT3qPsshVUoSS9qO2tAuMjHdx6JTHTE6bYtyEhl4it5Fk5aeEvDpJkF0CZVWK5HunMLkhrKYOqAQBD6IMMxmjJpZHvpD1k6mah8ylB/6c5IpchhqTsmALSVozaIHZrNMEEW7uG3Idj+W9TSwMPsWHAy2z6en45vzpsbidrXMfBHoOl5D3ArduB+qm9XfQ6Javr7oumoX/SIx63YV0xV3ew3ec2cUeyvAPG7a8pTPwPqQDt+ZSG4ofQnwa7pp57rD4/UEMyZzjlqI3LMtcdkRFtSoipZ/xSfc23iuyTQ7HK0bS+6Ni04qzThv2kuskT3kkxTep6YE/J3l/boWWYCaSICHh+7Ci9f999Vkn6n6bl+zuO8ap+EW9fm+v6mlWmz2Dkk97Wk97qZAmoszHKRBHTxNgGES3+yPzGjHSWVj4s3OXvTBPO6WOrc1XLlgme8BytTE1SiYJYPo64Lxl9WzvKXl8nJcJypL6XX5ct8NJ/ACt5ZD5x51mz5CL4ZisHW54SWJoYyvdFElO9DVSwxJdPlWgjes8ohmR+7erD0FE79PyymTpRPPAiweQLoLeeYYQ9pNWfuhXikkX9jz7egvbW0a6CZvdeE+zBJ/K3MN3h8WG4L02Wyl3PUG07s6CKB4lu3t/IC1qcKVX8DxjcVW4K81m7RNHvTi3b/DwLo9e3Poaqkd0qqV34aatofRFfudZNlAVZXo4MYdEInJUyV7H7rtPCtmRCJH2UTXMdtxoJoMHmRPrOfb4DPYMpW9n0cXG7NgZ7yrXbIcqyae+KEePCcC7pKESDWegUfkK+Vl3JtyvCpKKCjEbFyGRr0z/OTTzkJx7TNr17kc9FdIjxsMvNa/mLnqUN6su+0Mhv+jLcrDmA5inXlkCqlhjdNuXoIgmHJG+wnVK04tLJ8x71DCwh2wjF0pcwAJ1EuA6ySRoVXLI6T0e0/mg5bMmbKfS9KaVKGTansg6W4xAbvWz49rlpvs8djmmi6H95AgbgWsdaROKrmYDK1bpOzaC50JCm6jLgiPaGCHncuTSPqQ2yN1a6ABZRZxQGyfPpXLPd1vxwKBNbtfxPBQbks3MRr2upRXGm8fK+Bd9ci395HkaHdv7+VFHQW2wFvu89WJfGoajnTlylzQnUjA1ePSk8c+PssXqLMDoGR6bYoDn56EnYFPZH/zA4yVO9r0l7zkwJIq6ZaKoiexoKkvYlNeq8jgSUelQu3XjkirEOWkjLnjOIeDjj9k/O8E/Oc4RPZ/fwqA/B+UhoQOAAAYc0lEQVR4nO3dT2wa98Lu8Yfb1Bm3XVCpqvCOs8Mrk50r3YWzKZMVVOrVS3QXEJ3FcTc12Vw7V1eNI72tSbsI6caOVCn22cCrqgpexVYqmcUrhZ3x5sAu7EyqI4VFVY/rSnMX/PF/O84PPIb5fqTE4zGQH08wD79hmAm4rusKAAD4xv/wegAAAOByUf4AAPgM5Q8AgM9Q/gAA+AzlDwCAz1D+AAD4DOUPAIDPUP4AAPgM5Q8AgM9c83oAV9X6//rC6yGcK/bzM6+HcCryM0N+ZsjPzD/+8Q+vh3CuJ0+eeD2EgUb5n2Hq//4/r4dwqtJ3/+n1EM5FfmbIzwz5mXn06JHXQzjV3bt3vR7CwGOzPwAAPkP5AwDgM5Q/AAA+Q/kDAOAzlD8AAD5D+QMA4DOUPwAAPmNQ/g2VCkU1ejcWAABwCd6p/GvFrAKBMd28vaSG0+shAQCAfnqn8o8k5lTNxyVZsno8IAAA0F9HDu/rqF6rqdFoqFJekxPNKGOHvRkZAADoiyMz/6YqpYKmb97SV/ceKxgOeTMqAADQN0fKP6TE9LymY5KU0mSYjfoAAHihtrakhG0rkUjIttMqVHq3i/3xs/o1y1pal5SyRfcDAHD5KssJ3bizemjd+vqK6hvbmpsy3yp/bIe/RqWsLUkpe3J/Z74h3KP/xYsXXg9hoJGfGfIzQ35myM9M3/NzKprrFn9K+fyiJtrf3bu53JOP2B8r/9paQZKUtsNyamtKBAIKjAYUzZYOXc46tjBYvvzyS1mWpWw26/VQBhL5mSE/M+RnhvzM9Du/Zq2k9fZyvrqkZHJaxWcz7TUF1XrQ/kfKv6G1h1tSbEGqLGl0/Jamni4qJmmrVO9uAKgUc8pkVyWtKju3rNoAbhn48ccf9d5772l+fl6jo6P8ElwQ+ZkhPzPkZ4b8zPQ7v0alM9lOKRppzbDDkcn2ui3Vm+ale7j8G2U9lKT1e7p5s6iNN64yiXDrFUgk1J3kRxMZFSuuXNfVcjatyADO/lOplD744APt7u7KcZxD/4l7e3teD+/KIz8z5GeG/MyQn5m+52cF95c7PR+OKG5+y12HdvhrlMvd5Xy1qKmgVCu03gZITUZ6+M9KDx480Pz8fE9v08Tu7q4k6ZtvvtG//vUv/W+Px/M2AoGA10PoIj8z5GeG/MwczW90dNTjEZ3vKuf3z3/+szc3HJtSqDO5tix1XhJUag0pYnYMnkMz/3LxYWthZkPJiCXJUaWwIkmyJ3t7sJ/79+/LdV1P/3zyySfd8YyMjOj69et68OCBfvrpp57e134hPzPkZ4b8zJCfGV/kt15UvTPzbzZUaS9ORsz39j8w86+puCJJcW1mp1qrnLoKq5K0oEMH+nPqKiwXVW9amkwkNRU5sIliQKysrOiPP/7QyMiIAoGA7t+/r3v37nk9rIFBfmbIzwz5mSE/M5eZn9WZ+Tca2mov9mI3u+7M36lVtCJJM9OKtv8xp17WqqR4PiGnvKRCxZGcmtKjf1M5nFBmOqLM+MfKlZs9GMrl+vrrr/XXX39pfn5ejuPwwL8g8jNDfmbIzwz5mel3fpGo3V5a1VKhJkkqLXd2Kowp2oOD8HRn/vXKmiRpITmp/XXF1kI5q7HbTW3uTKuynNSKFrVjh2UprLWNWY19llXSzWqQDgb8888/6/PPP/d6GAOL/MyQnxnyM0N+ZvqeX8TWownp7pb0+Pa4Ht8+8LOZue4E3UR35l8vr0iKy44e3ITfWl593NTGdlFRq6nyV1uKPd0/AFAoFJX0UKUB+7wfD3wz5GeG/MyQnxnyM9P//ILKlKuanTiyOpXXdm6qJ/9Cd+Zv51y5ucM/jCSXtZNYktV508FpaE2HT+TrHPkKAAAMWRFlK64yjbokyVFQ4VDv9q87fmz/o/++dfb2BSsU1tEXJwAAwFwo1NtP2nUcO7zvmayQ7AnJOfB6wGnUu3sgAgCAq+9i5S9LwZC0fuC0gq0NAxOKhAbwMH8AAPjQhct/KjMjPcx1j+dfWrotxecUHbyP+gMA4Evnvud/VMjO6eWirfHJjBaTNX31cFbVneSgntwPAADfuXD5S9Lk9Jp2kk01JU3PMeUHAGCQvFP5S5IVDA7UQX0AAEDLBd/zBwAAg47yBwDAZyh/AAB8hvIHAMBnKH8AAHyG8gcAwGfe+aN+flD67j+9HsJAIz8z5GeG/MzcvXvX6yGgjwKu67peD2LY7e3tKZVKaWVlRe+//77Xwxk45GeG/MyQH4YRm/0vwQ8//KBffvlF33//vddDGUjkZ4b8zJAfhhEz/0tgWZZ2d3c1MjKi3d1dr4czcMjPDPmZIT8MI2b+ffbdd991lwOBgL799lsPRzN4yM8M+ZkhPwwrZv591pk1dDB7uBjyM0N+ZsgPw4qZfx8dnDV0MHt4e+RnhvzMkB+GGTP/Prp+/bquXbumjz76SL/99ps+/fRT/f7779rb29Off/7p9fCuPPIzQ35myA/DjJl/n/z6668aHR3VkydP9Pr1a0nS69ev9eTJE3344Yd68eKFxyO82sjPDPmZIT8MO2b+lyQQCIio3x35mSE/M+SHYcPMHwAAn6H8AQDwGcofAACfofwBAPAZyh8AAJ+h/AEA8BnKHwAAn6H8AQDwGcofAACfofwBAPAZyh8AAJ+h/AEA8BnKHwAAn6H8AQDwGcofAACfofwBAPAZyh8AAJ+h/AEA8BnKHwAAn6H8AQDwGcofAACfofwBAPAZyh8AAJ+h/AEA8BnKHwAAn6H8AQDwGcofAACfofwBAPAZyh8AAJ+h/AEA8JmA67qu14O4iqrP/8PrIZxr/NZ/eT2EU5GfGfIzQ37A2a55PYCrbPx//h+vh3Cq6n9/7/UQzkV+ZsjPDPkBp2OzPwAAPkP5AwDgM++22d+pq7BcVL1paTKR1FQk2ONhAQCAfrn4zN+pKT36N5XDCWWmI8qMf6xcudmHoQEAgH648My/spzUiha1Y4dlKay1jVmNfZZV0s0q1I8RAgCAnjoy83dUr1VULq1pKZtRbq1+5OJNlb/aUuzppKz2mlAoKumhSjXnEoYLAABMHSn/piqlgqZv3tJX9x4rGD4yl3caWpNkdatfco58BQAAV9uR8g8pMT2v6ZgkpTQZtk66ziFWKKyJvgwNAAD0w/Ed/pplLa1LStk61v1WSPaE5BxY7zTq2urrEAEAQC8dK/9GpawtSSl7/339/W36loIhab3S6F7esiRpQpHQ+VsJAACA946Vf22tIElK22E5tTUlAgEFRgOKZkuSLE1lZqSHOXX27yst3Zbic4ryUX8AAAbCkfJvaO3hlhRbkCpLGh2/pamni4pJ2irV5UgK2Tm9XHQ0PpnRUtbWzYezqhaSYt4PAMBgOPw5/0ZZDyVp/Z5urse08cbVlNZ0946kSKhb8JPTa9pJNtWUND3HlB8AgEFyqPwb5XJ3OV8taioo1QqttwFSk5FDV7SCQQ7qAwDAADq02b9cfNhamNlQMmJJclQprEiS7MnwZY+tr168eOH1EAYa+ZkhPzPkB5g5UP41FVckKa7N7FRrlVNXYVWSFmQf7H6nrsJSTtnskkq1wTyu/5dffinLspTNZr0eykAiPzPkZ4b8ADPd8ndqFa1I0sy0ou039516WauS4vmEnPKSChVnaE7s8+OPP+q9997T/Py8RkdHeRK5IPIzQ35myA8w0y3/emVNkrSQnNT+umJroZzV2GdrikSs7ol9snZYVnBKaxuzuvtZVg0NllQqpQ8++EC7u7tyHOfQk8je3p7Xw7vyyM8M+ZkhP8DMfvmXVyTFZR/6wH5refVxUxvbRUWt3p3Y58GDBwoEAp7++fe//90dT+dJ5JtvvtHf//73d4zzcpGfGfIzQ37A4OqWv51z5brF7iZ/SYokl7WzsyPXLWoqpJ6e2Of+/ftyXdfTP5988kl3PCMjI7p+/boePHign3766eJJeoD8zJCfGfIDBte18y5gWWcfvmdQT+yzsrKiP/74QyMjIwoEArp//77u3bvn9bAGBvmZIT8z5AeYObf8D2mf2Kc4BCf2+frrr/XXX39pfn6eJ413QH5myM8M+QFmLlb+B0/sk2wd9GdQT+zz888/6/PPP/d6GAOL/MyQnxnyA8wcP6XvmYbnxD48cZghPzPkZ4b8ADMXnPl3Tuxja3wyo8VkTV89nFV1hxP7AAAwKC5c/hIn9gEAYJC9U/lLnNgHAIBBdcH3/AEAwKCj/AEA8BnKHwAAn6H8AQDwGcofAACfofwBAPAZyh8AAJ+h/AEA8BnKHwAAn6H8AQDwGcofAACfeedj+/tB9b+/93oIA438zJCfGfIDThdwXdf1ehDDbm9vT6lUSisrK3r//fe9Hs7AIT8z5GeG/DCM2Ox/CX744Qf98ssv+v57ZiLvgvzMkJ8Z8sMwYuZ/CSzL0u7urkZGRrS7u+v1cAYO+ZkhPzPkh2HEzL/Pvvvuu+5yIBDQt99+6+FoBg/5mSE/M+SHYcXMv886s4YOZg8XQ35myM8M+WFYMfPvo4Ozhg5mD2+P/MyQnxnywzBj5t9H169f17Vr1/TRRx/pt99+06effqrff/9de3t7+vPPP70e3pVHfmbIzwz5YZgx8++TX3/9VaOjo3ry5Ilev34tSXr9+rWePHmiDz/8UC9evPB4hFcb+ZkhPzPkh2HHzP+SBAIBEfW7Iz8z5GeG/DBsmPkDAOAzlD8AAD5D+QMA4DOUPwAAPkP5AwDgM5Q/AAA+Q/kDAOAzlD8AAD5D+QMA4DOUPwAAPkP5AwDgM5Q/AAA+Q/kDAOAzlD8AAD5D+QMA4DOUPwAAPkP5AwDgM5Q/AAA+Q/kDAOAzlD8AAD5D+QMA4DOUPwAAPkP5AwDgM5Q/AAA+Q/kDAOAzlD8AAD5D+QMA4DOUPwAAPkP5AwDgM5Q/AAA+8978/Py814Pwg0AgoKmpKa+HMbDIzwz5mSE/DJuA67qu14MAAACXh83+AAD4DOUPwKccOV4PAfAI5d93jgrpgAKBgALpgo+fbGrKBFo5FGrmKTQbDTUaDTV7MLKB4zTVaDTUaOzf+3oh8xaPMUeFRPuxmPDzY9FRKZdQIDCq0UBAdqEuNQqtXAJptR6e+4/XTKHu9YCBnqP8L0PnObrh6Sg813kKNS4dp6Lk2JjGxsZUqPivwmqFpMbGxjRmF7tZOp0H2XmPsc4V6v7LratW1M27q6f8sLEfUeer4+OsMLSueT0A+Eewu2SZ3ZAVVKhzm5bhbQ0gy2rf+7B1YpKnJ2IpOvdM+bQjKzRp+r8wsByn82p8QvnNopLRsNR09Cyfl2OFFW4HEzz1FoDBx8wfl85plFSYSyhq27KjtrKFyqGtAU6jolwmoUAgoGg0KjuRUaHUvkyzpkJ2XivtyxaWc1ouVjrXVKmQlR1tXS9qp7VUrAzV5u1aqaD5bPve1wvKLS2rcnC2bzVVLhWUiNqy7ajsdFblxn4CTuftkmbzwLqKcmm7vdk7oKid1nKpdkn36HI1KwUlbnzV/m5LlXJJpVpTkqNm05HTbJ76eGmUCpqbyypXKKlSWlYiGlU0GlUmV1JTjsqFrGzblm3byhbKQ/W4wxBy0Wc7bj4uV5KrWN7d8Xo4nqm6KbVzOOFP/Gm1dbHt56deZuLRS3enunjCzxbdHfeNuxg75fZnn3t713tmx1084f492txxq/nUqblJcXdzp3X9fCej2NPWY3F74/z/kyFSXZw4fl8Xq667nW9/H2tntf947eRQzcfPyPiE/PLDlx+GBzN/eCCmpxtVVTfzmmivWW3PlGrFXHvNhPKb29p5U9Vs+0JbazUpktarA9dbeLap7e2knNKSvlpvrZt5uqHt7aryC/HWioe3NBz7bFlKv3mlfCeQ2II2q6+UjhzegB+byau6XVV+NtZes6pKZydLa/+2JKlWWm5/P6Hnr3bkutt62o5t9U5p6Gav4eSaNp8vtL+L6dlmVW/Skf39cnTyWykt+28EpBY3tPly/3HYWVc9sG51mdk/ri7KH5cu9jSn9FREkWhS2Zn2yvZ79+HEkjY3N5TPZxWxGiqXyqptda5pSbIUjkYVba8JRyIKhYKqlAvtNROajIbUbEqRyf0jshXKw7EZ2wqGNRlt33srrGgkrOChtoorm0sqEooomZk+cMVTbrD7/veWbiWSyi6XFEo/08vqK73ZSQ/dfgFWMKRoONz5TpFoREHrojuhPlJuekrRyaRynRdiE4vKTU8pMpnUXLxz88OWHoYJO/zh0iWioe5yeDIlPV7pPvtaamg5fVOPt0658hFO++9GqXOFLd2+MX7scvXm8MzBnAMLjo70esxWp9oUiigldfePOEnEntaEVrUlSVurundnfy/4hWdVzSUivRr2leEc/eaiHR0Pda8SikYlbUnhYHvdgVsfnocchhDlj0tnnTojclSY/qxd/DEt5uc0NRVRfTmtW/fWT76tztfu64m4Njbnj+2pbYWGr8RO3tU/eLEuC9kqv3mlcrmsWqWspXuP1XkZde+LOSV2ioowgT3spFKn6DFg2OyPK2liYV7TySlFQk2tdYr/hBJaq9QlWYrYqfaauhpWWNFoVGHVND83r7n0nCrDeDSg1bXjH9e/UAk5Wo4GNPrx33Qz5yg5l1PFdbX5NHXgEgCGEeWPS3OR/t2695nsRELRwLged1YeONhK57ZWbo8rkCgobE+rtXvblm6Pf6xAIKCPb9zW6vqq1rekSHiYpq/de6/xUZMjJlqKpts7Ba7f0ceBqGzb1o07nTcKIgoNU2wnOGsSP4yvF4EOyv8ydLZBD/kT6XnOjMFq/ZXIPVdnH/X11VVtxRe02Nlrf/2OijVHUkRzi/H96zqOFJxUcfuluju4d0zM6uX2mqJDlH3EzujAvT9cYKfcz+7qzn9Cu9WjmYKedfLVltbX21tZJma08WpeIQ2zk94i2V/XiSp49EJnHlnJkhU+43LAFcEpfXEFOWo2mnIsS6Hg2cdZcxzn2D4ETrOh1v59lkKhIT5Om+PIkdWbncqdphqdnSKtoELHGg/AMKH8AQDwGTb7AwDgM5Q/AAA+Q/kDAOAzlD8AAD5D+QMA4DOUPwAAPkP5AwDgM5Q/AAA+Q/n3TF25TFqZTE6VA8dbrRRyymQyWi41vBvaQCA/M+RnhvzgMy56ZnMx5kpylXrWWlHNt77XjFvd8XZsg4D8zJCfGfKDn1D+PbXtPpqQK8ldePbcXYi1lvM8c7wl8jNDfmbID/7Bsf17rbGm6NgtbbW/jT/dVDEdPXaxemlJJSWUnhru86Zd2Hn5NWtaXlpSqSGFQlEl00lFh/28sxdxTn5OvaSlpaIashSOTimRsIf+tL0X8pa/v1JThbl5Wel5JSJDfPIoDC+vX30Mo40ZdTcXvjrh5zvbL9245E48rV762AbBqfm9eenGJFeacGcXZt0JMTM7yWn57XQ3Y8fd2Zl4eznFJu0jzvv9dV3X3Vxs5Rdb3LzUsQG9Qvn32vZGu6Baf1L5gwVfdeMHfhan/I87I7/q05grpdzNTlntVN2U5E7wBLzvjPxapfbI3W5/v1N95kpyFzdp/64zf39bdjbzZ/4cGATs7d9TTS2lb2pd0qN8XilJK7fHVax3fh5WdnNT1epLLUxIzuk35FNn5+dIUsxWtLOZ2grLjklb5fqJt+Y/Z+XnSOEZPdpIqvNGkxVsBWmx2b/tvN9fSU5FyRu3lVpcVEpSk19iDCqvX30Mk+rT9qbUmeeu67rum41H7RnCQne21ZGPyY0x8z/kIvm5rutWny24ktyZ5yf91H/eNr/tjUV3ZibVfttkwX3jzXCvnPPz23HzcbmK5d0d95WbEr/DGFyUf6+82Wg/mR7YLO267vPZiRM2D+5Q/kddIL+d7U33Uby12TW28Nxlo7V7ofzeVJ+7szMz3c3b+Sr1/zb5vXo240rx9j4Sr9y45Mbzp+0VAFxt7O3vCUcFe1TLyarW0hGvBzNQKoWMbtx+LCml55tZ2VE+LfG2mo2GrFBI3a38Tk2Z0XGVFjdVmT5pj3bsqykRGNeqYnq0mJBTL6vwcEVbkh4921QmQX4YLLznj4HRLOd04/ZjLTyrynWXKf6LcCqaGhtTYrm2v84KKzoh3vR/K0ElZme1sDClRr0uBUOtfSdicTUaTfbfwcC55vUAgLfjaC17V9KEIqGmSqVSe7WjYGRK0TAFdiYrovmU9MWdORUnl2SHpXIxpztb0myYz6mfL6R0NntoTbj0UEpnlU2y9Q6Dh/L3En31Drb0xWefHVoTW9zUGputz2EpkdvUbOWGvhhf7a5NLW5ongNNvQPn0Bdg0PCeP+AzzUajtanfCirIC1DAlyh/AAB8hh3+AADwGcofAACf+f9Qj7r+/IJbywAAAAA="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建模型\n",
    "\n",
    "该模型具有最剧烈drastic的变化。\n",
    "\n",
    "### 不同的RNN架构:长期短期记忆网络（LSTM）\n",
    "\n",
    "- **标准RNN缺点：[梯度消失](https://en.wikipedia.org/wiki/Vanishing_gradient_problem)**\n",
    "- 比RNN新增了一个单元$c_t$（可变权重）\n",
    "- LSTM：解决梯度消失，梯度爆炸和长期依赖问题\n",
    "\n",
    "我们将使用一种称为长期短期记忆（LSTM）的不同RNN体系结构。为什么LSTM比标准RNN更好？标准RNN受[梯度消失](https://en.wikipedia.org/wiki/Vanishing_gradient_problem)的困扰。 \n",
    "\n",
    "LSTM通过 _cell_ ，$c$的额外循环状态来克服了这一问题。\n",
    "-cell可以被认为是LSTM的“记忆memory”-并且使用使用多个 _gates_ 来控制信息流入和流出memory的流程。欲了解更多信息，请点击[这里](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)。我们可以简单地将LSTM视为$x_t$, $h_t$和$c_t$的函数，而不仅仅是$x_t$ 和 $h_t$。\n",
    "\n",
    "$$(h_t, c_t) = \\text{LSTM}(x_t, h_t, c_t)$$\n",
    "\n",
    "因此，使用LSTM的模型看起来像（省略了嵌入层）：\n",
    "\n",
    "![sentiment2.png](attachment:sentiment2.png)\n",
    "\n",
    "初始单元状态$c_0$，就像初始隐藏状态被初始化为全零的张量一样。然而，情感预测仍然仅使用最终的隐藏状态而不是最终的单元状态进行，即$\\hat{y}=f(h_T)$。\n",
    "\n",
    "### 双向RNN   Bidirectional RNN\n",
    "\n",
    "双向RNN背后的概念很简单。除了让RNN处理从头到尾的句子中的单词（前向RNN）之外，我们还有第二个RNN处理**从尾数到第一个句子**中的单词（后向RNN）。在时间步$t$，前向RNN是处理字 $x_t$，而后向RNN是处理字$x_{T-t+1}$。\n",
    "\n",
    "在PyTorch中，由前向和后向RNN返回的隐藏状态（和单元状态）张量彼此堆叠在单个张量中。\n",
    "\n",
    "我们使用前向RNN（从句子的最后一个词获得）的最后一个隐藏状态的连接$h_T^\\rightarrow$，以及反向RNN的最后一个隐藏状态（从句子的第一个单词获得）$h_T^\\leftarrow$，来进行情感预测。\n",
    "即 $\\hat{y}=f(h_T^\\rightarrow, h_T^\\leftarrow)$ \n",
    "\n",
    "下图显示了双向RNN，前向RNN为橙色，后向RNN为绿色，线性层为银色。\n",
    "\n",
    "![sentiment3.png](attachment:sentiment3.png)\n",
    "\n",
    "### 多层RNN     Multi-layer RNN\n",
    "\n",
    "多层RNN（也称为*深度RNN*）是另一个简单的概念。想法是，我们在初始标准RNN之上添加其他RNN，其中添加的每个RNN都是*另一层*。第一个（底部）RNN在时间步$t$输出的隐藏状态在时间步$t$将是其上方RNN的输入。然后根据最终（最高）层的最终隐藏状态进行预测。\n",
    "\n",
    "下图显示了多层单向RNN，其中层号以上标给出。还要注意，每一层都需要有自己的初始隐藏状态，$h_0^L$。\n",
    "\n",
    "![sentiment4.png](attachment:sentiment4.png)\n",
    "\n",
    "### 正则化 dropout\n",
    "\n",
    "- 提高泛化能力\n",
    "\n",
    "尽管我们对模型进行了改进，但每个模型都添加了其他参数。无需过多地拟合细节，模型中的参数越多，模型过度拟合的可能性就越高（记忆训练数据，导致较低的训练误差，但较高的验证/测试误差，即对新的，未看间的数据的泛化性差）。\n",
    "\n",
    "为了解决这个问题，我们使用正则化。更具体地说，我们使用一种称为*dropout*的正则化方法。Dropout是在forward过程中随机*dropping out*（设置为0）层中的神经元来进行的。每个神经元dropped out的概率由一个超参数设置，而每个应用了退出的神经元都被独立考虑。关于Dropout为何起作用的一种理论是，可以将具有退出参数的模型视为“弱”（较少参数）模型。所有这些“较弱”模型的预测（每个前向通过一个）在模型参数内平均在一起。因此，可以将您的一个模型视为较弱的模型的集合，这些模型都不是过度参数化的，因此不应过度拟合。\n",
    "\n",
    "### 实施细节\n",
    "\n",
    "此模型的另一个补充是，我们将不会学习`<pad>`令牌的嵌入。这是因为我们要明确地告诉我们的模型，填充标记与确定句子的情感无关。这意味着填充令牌的嵌入将保留为其初始化的位置（我们稍后将其初始化为全零）。为此，我们将pad令牌的索引作为`padding_idx`参数传递给`nn.Embedding`层。\n",
    "\n",
    "要使用LSTM代替标准RNN，我们使用`nn.LSTM`代替`nn.RNN`。另外，请注意，LSTM返回输出`output`和最终隐藏`hidden`状态以及最终单元`cell`状态的元组，而标准RNN仅返回输出`output`和最终隐藏`hidden`状态。\n",
    "\n",
    "由于LSTM的最终隐藏状态同时具有前向和后向分量，它们将被串联在一起，因此`nn.Linear`层的输入大小是隐藏维大小的两倍。\n",
    "\n",
    "通过传递`num_layers`的值和RNN / LSTM的双向`bidirectional`参数来实现双向性并添加其他层。\n",
    "\n",
    "删除是通过初始化`nn.Dropout`层（参数是删除每个神经元的概率）并在我们要对其应用删除的每一层之后在`forward`方法中使用它来实现的。\n",
    "\n",
    "**注意**：永远不要在输入或输出层（在这种情况下为`text`或`fc`）上使用dropout，而只想在中间层上使用`dropout`。 LSTM有一个`dropout`参数，该参数在一层中的隐藏状态与下一层中的隐藏状态之间的连接上增加了dropout。\n",
    "\n",
    "当我们传递句子的长度以使用打包的填充序列时，我们必须添加第二个参数`text_lengths`进行转发`forward`。\n",
    "\n",
    "在将嵌入传递给RNN之前，需要对它们进行打包，这需要使用`nn.utils.rnn.packed_padded_sequence`进行。这将导致我们的RNN仅处理序列中未填充的元素。然后，RNN将返回`packed_output`（打包序列）以及隐藏`hidden`状态和单元`cell`状态（均为张量）。如果没有打包的填充序列，则隐藏`hidden`和单元`cell`是序列中最后一个元素的张量，这很可能是填充令牌，但是，当使用打包的填充序列时，它们都来自序列中最后一个非填充的元素。\n",
    "\n",
    "然后，我们使用`nn.utils.rnn.pad_packed_sequence`解压缩输出序列，以将其从压缩序列转换为张量。填充令牌的输出`output`元素将为零张量（每个元素为零的张量）。通常，只有在以后在模型中使用输出时，才需要解压缩输出。尽管我们不是在这种情况下，但是我们仍然将序列拆包只是为了展示它是如何完成的。\n",
    "\n",
    "最终隐藏`hidden`状态为 _**[num layers * num directions, batch size, hid dim]**_。\n",
    "\n",
    "这些是有序的：**[forward_layer_0, backward_layer_0, forward_layer_1, backward_layer 1, ..., forward_layer_n, backward_layer n]**。当我们想要最后（顶层）向前和向后的隐藏状态时，我们从第一个维度获取了最顶层的两个隐藏层，即`hidden[-2,:,:]`和`hidden[-1,:,:]`，并将它们连接起来在将它们传递到线性层之前（应用dropout之后）在一起。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, \n",
    "                 bidirectional, dropout, pad_idx):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)\n",
    "        \n",
    "        self.rnn = nn.LSTM(embedding_dim, \n",
    "                           hidden_dim, \n",
    "                           num_layers=n_layers, \n",
    "                           bidirectional=bidirectional, \n",
    "                           dropout=dropout)\n",
    "        \n",
    "        self.fc = nn.Linear(hidden_dim * 2, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, text, text_lengths):\n",
    "        \n",
    "        #text = [sent len, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(text))\n",
    "        \n",
    "        #embedded = [sent len, batch size, emb dim]\n",
    "        \n",
    "        #pack sequence\n",
    "        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths)\n",
    "        \n",
    "        packed_output, (hidden, cell) = self.rnn(packed_embedded)\n",
    "        \n",
    "        #unpack sequence\n",
    "        output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)\n",
    "\n",
    "        #output = [sent len, batch size, hid dim * num directions]\n",
    "        #output over padding tokens are zero tensors\n",
    "        \n",
    "        #hidden = [num layers * num directions, batch size, hid dim]\n",
    "        #cell = [num layers * num directions, batch size, hid dim]\n",
    "        \n",
    "        #concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers\n",
    "        #and apply dropout\n",
    "        \n",
    "        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))\n",
    "                \n",
    "        #hidden = [batch size, hid dim * num directions]\n",
    "            \n",
    "        return self.fc(hidden)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "像以前一样，我们将创建RNN类的实例，并使用新的参数和参数来表示层数，双向性和丢失概率。\n",
    "\n",
    "为了确保可以将预先训练的向量加载到模型中，`EMBEDDING_DIM`必须等于之前加载的预先训练的GloVe向量。\n",
    "\n",
    "我们从词汇表中获取填充令牌索引，从字段的`pad_token`属性（默认为`<pad>`）中获取表示填充令牌的实际字符串。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DIM = len(TEXT.vocab)\n",
    "EMBEDDING_DIM = 100\n",
    "HIDDEN_DIM = 256\n",
    "OUTPUT_DIM = 1\n",
    "N_LAYERS = 2\n",
    "BIDIRECTIONAL = True\n",
    "DROPOUT = 0.5\n",
    "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
    "\n",
    "model = RNN(INPUT_DIM, \n",
    "            EMBEDDING_DIM, \n",
    "            HIDDEN_DIM, \n",
    "            OUTPUT_DIM, \n",
    "            N_LAYERS, \n",
    "            BIDIRECTIONAL, \n",
    "            DROPOUT, \n",
    "            PAD_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将打印出模型中的参数数量。\n",
    "\n",
    "请注意，我们的参数几乎是以前的两倍！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 4,810,857 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后的添加是将我们之前加载的预训练词嵌入复制到模型的`embedding`层中。\n",
    "\n",
    "我们从字段的vocab中检索嵌入，并检查它们的大小是否正确，_**[vocab size, embedding dim]**_ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([25002, 100])\n"
     ]
    }
   ],
   "source": [
    "pretrained_embeddings = TEXT.vocab.vectors\n",
    "\n",
    "print(pretrained_embeddings.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then replace the initial weights of the `embedding` layer with the pre-trained embeddings.\n",
    "\n",
    "**Note**: this should always be done on the `weight.data` and not the `weight`!\n",
    "\n",
    "然后，我们用预训练的嵌入替换`embedding`层的初始权重。\n",
    "\n",
    "**注意**：这应该始终在`weight.data`而不是`weight`上完成！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],\n",
       "        [-0.8555, -0.7208,  1.3755,  ...,  0.0825, -1.1314,  0.3997],\n",
       "        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],\n",
       "        ...,\n",
       "        [ 0.0174, -0.8317,  1.0815,  ..., -0.1634, -0.1672, -0.4210],\n",
       "        [ 0.3357, -0.6065, -0.4592,  ...,  1.0952,  0.2525, -0.7633],\n",
       "        [ 0.5939,  0.4972,  0.2567,  ..., -0.5814, -0.0688,  0.6311]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.embedding.weight.data.copy_(pretrained_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "作为我们的`<UNK>`和`<pad>`令牌不在他们一直在使用`unk_init`初始化的预训练的词汇(一个 $\\mathcal{N}(0,1)$ 分布)。最好将它们都初始化为全零，以明确地告诉我们的模型，最初，它们与确定情绪无关。\n",
    "\n",
    "为此，我们可以手动将嵌入权重矩阵中的行设置为零。我们通过找到标记的索引来获得它们的行，这对于填充索引已经完成。\n",
    "\n",
    "**注意**：就像初始化嵌入一样，这应该在`weight.data`而不是`weight`上完成！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
      "        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],\n",
      "        ...,\n",
      "        [ 0.0174, -0.8317,  1.0815,  ..., -0.1634, -0.1672, -0.4210],\n",
      "        [ 0.3357, -0.6065, -0.4592,  ...,  1.0952,  0.2525, -0.7633],\n",
      "        [ 0.5939,  0.4972,  0.2567,  ..., -0.5814, -0.0688,  0.6311]])\n"
     ]
    }
   ],
   "source": [
    "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
    "\n",
    "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "\n",
    "print(model.embedding.weight.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，我们可以看到嵌入权重矩阵的前两行已设置为零。当我们经过垫的指数令牌嵌入层的`padding_idx`将保持零整个训练，但是`<UNK>`标记嵌入将被教训。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在来训练模型。\n",
    "\n",
    "我们在此所做的唯一更改是将优化程序从`SGD`更改为`Adam`。 SGD使用相同的学习速率来更新所有参数，并且选择此学习速率可能很棘手。 `Adam`调整每个参数的学习率，从而使更新频率更高的参数具有较低的学习率，而更新频率较低的参数则具有较高的学习率。有关`Adam`（和其他优化程序）的更多信息，请参见 [此处](http://ruder.io/optimizing-gradient-descent/index.html)。\n",
    "\n",
    "要将`SGD`更改为`Adam`，我们只需将`optim.SGD`更改为`optim.Adam`，还要注意，由于PyTorch指定了合理的默认初始学习率，因此我们不必为Adam提供初始学习率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练模型的其余步骤保持不变。\n",
    "\n",
    "我们定义标准并将模型和标准放置在GPU上（如果可用）..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.BCEWithLogitsLoss()\n",
    "\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们实现了计算准确性的功能..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binary_accuracy(preds, y):\n",
    "    \"\"\"\n",
    "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
    "    \"\"\"\n",
    "\n",
    "    #round predictions to the closest integer\n",
    "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
    "    correct = (rounded_preds == y).float() #convert into float for division \n",
    "    acc = correct.sum() / len(correct)\n",
    "    return acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们定义了一个训练模型的函数。\n",
    "\n",
    "当我们设置`include_lengths = True`时，我们的`batch.text`现在是一个元组，第一个元素是数字化的张量，第二个元素是每个序列的实际长度。在将它们传递给模型之前，我们将它们分为各自的变量`text`和`text_lengths`。\n",
    "\n",
    "**注意**：由于我们现在正在使用Dropout，因此必须记住使用`model.train()`来确保在训练过程中“打开” dropout。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    \n",
    "    model.train()\n",
    "    \n",
    "    for batch in iterator:\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        text, text_lengths = batch.text\n",
    "        \n",
    "        predictions = model(text, text_lengths).squeeze(1)\n",
    "        \n",
    "        loss = criterion(predictions, batch.label)\n",
    "        \n",
    "        acc = binary_accuracy(predictions, batch.label)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        epoch_acc += acc.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们定义一个用于测试模型的函数，再次记住要分开`batch.text`。\n",
    "\n",
    "**注意**：由于我们现在正在使用Dropout，因此必须记住要使用`model.eval()`来确保在评估时已“关闭” dropout。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    with torch.no_grad():\n",
    "    \n",
    "        for batch in iterator:\n",
    "\n",
    "            text, text_lengths = batch.text\n",
    "            \n",
    "            predictions = model(text, text_lengths).squeeze(1)\n",
    "            \n",
    "            loss = criterion(predictions, batch.label)\n",
    "            \n",
    "            acc = binary_accuracy(predictions, batch.label)\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_acc += acc.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "并创建一个不错的函数来告诉我们各个epochs要花多长时间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们训练模型..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 514m 34s\n",
      "\tTrain Loss: 0.656 | Train Acc: 61.44%\n",
      "\t Val. Loss: 0.706 |  Val. Acc: 49.03%\n",
      "Epoch: 02 | Epoch Time: 493m 46s\n",
      "\tTrain Loss: 0.643 | Train Acc: 61.03%\n",
      "\t Val. Loss: 0.668 |  Val. Acc: 58.77%\n",
      "Epoch: 03 | Epoch Time: 502m 19s\n",
      "\tTrain Loss: 0.661 | Train Acc: 60.19%\n",
      "\t Val. Loss: 0.649 |  Val. Acc: 60.17%\n",
      "Epoch: 04 | Epoch Time: 480m 55s\n",
      "\tTrain Loss: 0.474 | Train Acc: 77.80%\n",
      "\t Val. Loss: 0.360 |  Val. Acc: 85.47%\n",
      "Epoch: 05 | Epoch Time: 455m 22s\n",
      "\tTrain Loss: 0.323 | Train Acc: 86.73%\n",
      "\t Val. Loss: 0.320 |  Val. Acc: 87.31%\n"
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 5\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "\n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut2-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...并获得我们全新的，大大提高的测试准确性！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.324 | Test Acc: 86.82%\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('tut2-model.pt'))\n",
    "\n",
    "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
    "\n",
    "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 用户输入\n",
    "\n",
    "现在，我们可以使用我们的模型来预测我们给出的任何句子的情绪。由于已经接受了电影评论方面的培训，因此提供的句子也应该是电影评论。\n",
    "\n",
    "使用模型进行推理时，应始终处于评估模式。如果循序渐进地遵循了本教程，那么它应该已经处于评估模式（对测试集进行评估`evaluate`），但是我们明确设置了它以避免任何风险。\n",
    "\n",
    "我们的`predict_sentiment`函数可以执行以下操作：\n",
    "\n",
    "- 将模型设置为评估模式\n",
    "- 将句子标记化，即将其从原始字符串拆分为标记列表\n",
    "- 通过将词汇转换为它们的整数表示来索引这些令牌\n",
    "- 得到我们序列的长度\n",
    "- 将作为Python列表的索引转换为PyTorch张量\n",
    "- 通过解压`unsqueeze`添加批次尺寸\n",
    "- 将长度转换为张量\n",
    "- 使用`sigmoid`函数压缩0到1之间的实数的输出预测\n",
    "- 使用`item()`方法将持有单个值的张量转换为整数\n",
    "\n",
    "我们期望带有负面情绪的评论返回接近0的值，而正面评价则返回接近1的值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "def predict_sentiment(model, sentence):\n",
    "    model.eval()\n",
    "    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]\n",
    "    indexed = [TEXT.vocab.stoi[t] for t in tokenized]\n",
    "    length = [len(indexed)]\n",
    "    tensor = torch.LongTensor(indexed).to(device)\n",
    "    tensor = tensor.unsqueeze(1)\n",
    "    length_tensor = torch.LongTensor(length)\n",
    "    prediction = torch.sigmoid(model(tensor, length_tensor))\n",
    "    return prediction.item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个负面评价示例..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22547456622123718"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_sentiment(model, \"This film is terrible\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个正面评价示例..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9530301690101624"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_sentiment(model, \"This film is great\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 下一步\n",
    "\n",
    "我们现在为电影评论建立了一个体面的情感分析模型！在下一个笔记本中，我们将实现一个模型，该模型将以较少的参数获得差不多的精度，并且训练速度快得多。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}