examples/Dive_into_Llama.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f5104e89-a647-49ac-958e-0019e2729155",
   "metadata": {},
   "source": [
    "# Llama深入浅出"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edde7fc9-5826-4ad8-b856-64c510eeb3da",
   "metadata": {},
   "source": [
    "😋😋公众号算法美食屋后台回复关键词：**torchkeras**，获取本文notebook源码。\n",
    "\n",
    "前方干货预警：这可能是你能够找到的最容易懂的最具实操性的学习开源LLM模型源码的教程。\n",
    "\n",
    "本例从零开始基于transformers库逐模块搭建和解读Llama模型源码(中文可以翻译成羊驼)。\n",
    "\n",
    "并且训练它来实现一个有趣的实例：两数之和。\n",
    "\n",
    "输入输出类似如下：\n",
    "\n",
    "输入：\"<SOS>12345+54321=\"\n",
    "    \n",
    "输出：\"66666<EOS>\"\n",
    "    \n",
    "我们把这个任务当做一个文本生成任务来进行。输入是一个序列的上半部分，输出其下半部分.\n",
    "\n",
    "这和文本生成的输入输出结构是类似的，所以可以用Llama来做。\n",
    "\n",
    "\n",
    "目前大部分开源LLM模型都是基于transformers库来做的，它们的结构大部分都和Llama大同小异。\n",
    "\n",
    "俗话说，魔鬼隐藏在细节中，深入理解Llama模型的的源码细节，将会帮助你打通和开源LLM模型相关的基础原理(如旋转位置编码以及长度外推)，并让你熟悉各种参数的配置和使用(如past_key_value，attention_mask的使用等等)。\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52f17d7b-937b-4149-9b74-ff62262278d7",
   "metadata": {},
   "source": [
    "## 一，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "6c39cc6d-9335-4a4d-908d-3996685a91ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch.utils.data import Dataset,DataLoader\n",
    "\n",
    "# 定义字典\n",
    "words = '<PAD>,<BOS>,<EOS>,1,2,3,4,5,6,7,8,9,0,+,='\n",
    "vocab = {word: i for i, word in enumerate(words.split(','))}\n",
    "vocab_r = [k for k, v in vocab.items()] #反查词典\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "37163f72-f14c-4cd1-b127-9e558fad95d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<BOS>3914835626735057733+318829464988=3914835945564522721<EOS> \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#两数相加数据集\n",
    "def get_data(min_length=10,max_length=20):\n",
    "    # 定义词集合\n",
    "    words = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']\n",
    "\n",
    "    # 每个词被选中的概率\n",
    "    p = np.array([7, 5, 5, 7, 6, 5, 7, 6, 5, 7])\n",
    "    p = p / p.sum()\n",
    "\n",
    "    # 随机采样n1个词作为s1\n",
    "    n1 = random.randint(min_length, max_length)\n",
    "    s1 = np.random.choice(words, size=n1, replace=True, p=p)\n",
    "    s1 = s1.tolist()\n",
    "\n",
    "    # 随机采样n2个词作为s2\n",
    "    n2 = random.randint(min_length, max_length)\n",
    "    s2 = np.random.choice(words, size=n2, replace=True, p=p)\n",
    "    s2 = s2.tolist()\n",
    "\n",
    "    # x等于s1和s2字符上的相加\n",
    "    x = s1 + ['+'] + s2 + ['=']\n",
    "    \n",
    "    # y等于s1和s2数值上的相加\n",
    "    y = int(''.join(s1)) + int(''.join(s2))\n",
    "    y = list(str(y))\n",
    "    \n",
    "    # 加上首尾符号\n",
    "    x = ['<BOS>'] + x \n",
    "    y =  y + ['<EOS>']\n",
    "    \n",
    "    return x,y\n",
    "\n",
    "x,y = get_data() \n",
    "print(''.join(x)+''.join(y),\"\\n\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "6c057c13-6dbe-46ae-ac35-b137cee8663b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<BOS>12878683929048906366+11274414130675477=12889958343179581843<EOS>\n"
     ]
    }
   ],
   "source": [
    "# 定义数据集\n",
    "class TwoSumDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self,size = 100000, min_length=10,max_length=20):\n",
    "        super(Dataset, self).__init__()\n",
    "        self.size = size\n",
    "        self.min_length=min_length\n",
    "        self.max_length=max_length\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.size\n",
    "\n",
    "    def __getitem__(self, i):\n",
    "        x,y = self.get(i)\n",
    "        \n",
    "        # 编码成token\n",
    "        context_ids = [vocab[i] for i in x]\n",
    "        target_ids = [vocab[i] for i in y]\n",
    "        \n",
    "        input_ids = context_ids + target_ids\n",
    "        \n",
    "        #-100标志位后面会在计算loss时会被忽略不贡献损失，我们集中优化target部分生成的loss\n",
    "        labels = [-100]*len(context_ids)+ target_ids\n",
    "        masks = [0 if t==vocab['<PAD>'] else 1 for t in input_ids]\n",
    "        \n",
    "        example = {'input_ids':input_ids,\n",
    "                  'labels':labels,'attention_mask':masks}\n",
    "        \n",
    "        return example\n",
    "    \n",
    "    def get(self,i):\n",
    "        return get_data(self.min_length,self.max_length)\n",
    "    \n",
    "    \n",
    "    def show_example(self,example):\n",
    "        input_ids,labels = example['input_ids'],example['labels']\n",
    "        x = ''.join([vocab_r[a] for a,b in zip(input_ids,labels) if b==-100])\n",
    "        y = ''.join([vocab_r[a] for a,b in zip(input_ids,labels) if b!=-100])\n",
    "        print(x+y)\n",
    "        \n",
    "        \n",
    "    \n",
    "ds_train = TwoSumDataset(size = 100000,min_length=10,max_length=20)\n",
    "ds_val = TwoSumDataset(size = 10000,min_length=10,max_length=20)\n",
    "example = ds_train[0]\n",
    "ds_train.show_example(example)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "bcc37cdf-9e44-4c29-b0b6-244809b922ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_collator(examples: list):\n",
    "    len_ids = [len(example[\"input_ids\"]) for example in examples]\n",
    "    longest = max(len_ids) #之后按照batch中最长的input_ids进行padding\n",
    "    \n",
    "    input_ids = []\n",
    "    labels_list = []\n",
    "    masks_list = []\n",
    "    \n",
    "    for length, example in sorted(zip(len_ids, examples), key=lambda x: -x[0]):\n",
    "        ids = example[\"input_ids\"]\n",
    "        labs = example[\"labels\"]\n",
    "        masks = example['attention_mask']\n",
    "        \n",
    "        ids = [vocab['<PAD>']] * (longest - length)+ids \n",
    "        labs = [-100] * (longest - length)+labs\n",
    "        masks = [0]*(longest - length)+masks\n",
    "        \n",
    "        input_ids.append(torch.LongTensor(ids))\n",
    "        labels_list.append(torch.LongTensor(labs))\n",
    "        masks_list.append(torch.LongTensor(masks))\n",
    "          \n",
    "    input_ids = torch.stack(input_ids)\n",
    "    labels = torch.stack(labels_list)\n",
    "    attention_mask = torch.stack(masks_list)\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"labels\": labels,\n",
    "        \"attention_mask\":attention_mask\n",
    "    }\n",
    "\n",
    "# 数据加载器\n",
    "dl_train = DataLoader(dataset=ds_train,\n",
    "         batch_size=200,\n",
    "         drop_last=True,\n",
    "         shuffle=True,\n",
    "         collate_fn = data_collator        \n",
    "        )\n",
    "\n",
    "dl_val = DataLoader(dataset=ds_val,\n",
    "         batch_size=200,\n",
    "         drop_last=True,\n",
    "         shuffle=False,\n",
    "         collate_fn = data_collator  \n",
    "        )\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "49f7bc27-2ac9-4730-bd76-a76e3eee1be9",
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in dl_train:\n",
    "    break "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "7d9c429f-de21-4c96-81d9-39d7e2d446c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[ 1, 11,  6,  ...,  7, 11,  2],\n",
       "         [ 0,  1,  6,  ...,  5,  4,  2],\n",
       "         [ 0,  1,  7,  ...,  8,  8,  2],\n",
       "         ...,\n",
       "         [ 0,  0,  0,  ..., 10, 11,  2],\n",
       "         [ 0,  0,  0,  ..., 12,  3,  2],\n",
       "         [ 0,  0,  0,  ..., 11, 12,  2]]),\n",
       " 'labels': tensor([[-100, -100, -100,  ...,    7,   11,    2],\n",
       "         [-100, -100, -100,  ...,    5,    4,    2],\n",
       "         [-100, -100, -100,  ...,    8,    8,    2],\n",
       "         ...,\n",
       "         [-100, -100, -100,  ...,   10,   11,    2],\n",
       "         [-100, -100, -100,  ...,   12,    3,    2],\n",
       "         [-100, -100, -100,  ...,   11,   12,    2]]),\n",
       " 'attention_mask': tensor([[1, 1, 1,  ..., 1, 1, 1],\n",
       "         [0, 1, 1,  ..., 1, 1, 1],\n",
       "         [0, 1, 1,  ..., 1, 1, 1],\n",
       "         ...,\n",
       "         [0, 0, 0,  ..., 1, 1, 1],\n",
       "         [0, 0, 0,  ..., 1, 1, 1],\n",
       "         [0, 0, 0,  ..., 1, 1, 1]])}"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f50fb234-e9ce-4813-99e2-d3139ffe6d0d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4483a2c3-7aad-4ff9-81ed-0a9c4d9a1953",
   "metadata": {},
   "source": [
    "## 二，定义模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403c34a9-baa1-471a-b574-01b5a341b32f",
   "metadata": {},
   "source": [
    "下面，我们会像搭积木建城堡那样从低往高地构建LLaMA模型。\n",
    "\n",
    "先构建4个基础组件：旋转位置编码，多头注意力、前馈网络、层归一化。类似用最基础的积木块搭建了 墙壁，房顶，房门，窗户 这样的模块。\n",
    "\n",
    "然后用这4个基础组件构建中间成品: 解码层。类似用基础组件构建了房间。\n",
    "\n",
    "接着用多个中间成品解码层的堆叠组装成了LlamaModel完整模型，相当于通过构建多个房间建成了城堡的主体结构。\n",
    "\n",
    "最后我们在LlamaModel基础上设计了两种不同的输出head，一种是语言模型Head，得到了LlamaForCausalLM，可用于文本生成。\n",
    "\n",
    "另外一种是分类head，得到了LlamaForSequenceClassification，可用于文本分类。\n",
    "\n",
    "相当于我们在城堡主体结构完成的基础上设计了两种不同的装修风格，一种是加装了一些游乐设施以便用于商业活动，另一种则是加装了一些武器以便用于军事活动。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**************************************************************************************************************\n",
    "\n",
    "1, 旋转位置编码: RoPE (使用旋转矩阵实现的绝对位置编码，可以起到相对位置编码的效果)\n",
    "\n",
    "2, 多头注意力: LlamaAttention (用于融合不同token之间的信息)\n",
    "\n",
    "3, 前馈网络: LlamaMLP (用于逐位置将多头注意力融合后的信息进行高维映射变换)\n",
    "\n",
    "4, 层归一化: LlamaRMSNorm (用于稳定输入，相当于保持每个词向量的方向不变，但对模长标准化。)\n",
    "\n",
    "**************************************************************************************************************\n",
    "\n",
    "5, Llama解码层: LlamaDecoderLayer (同时具备信息融合，信息转换功能的基本结构单元)\n",
    "\n",
    "*************************************************************************************************************\n",
    "\n",
    "6, Llama解码器: LlamaModel (多个解码层的堆叠)\n",
    "\n",
    "*************************************************************************************************************\n",
    "7，Llama语言模型: LlamaForCausalLM (解码器加上语言模型head，可用于文本生成)\n",
    "\n",
    "8，Llama分类模型: LlamaForSequenceClassification (解码器加上分类head，可用于文本分类)\n",
    "\n",
    "**************************************************************************************************************\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "edcd3e2c-a633-44f2-b67d-1f678e60bfcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "from typing import List, Optional, Tuple, Union\n",
    "\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import torch.utils.checkpoint\n",
    "from torch import nn\n",
    "from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss\n",
    "\n",
    "from transformers.activations import ACT2FN\n",
    "from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast, SequenceClassifierOutputWithPast\n",
    "from transformers.modeling_utils import PreTrainedModel\n",
    "from transformers.utils import add_start_docstrings, add_start_docstrings_to_model_forward, logging, replace_return_docstrings\n",
    "\n",
    "from transformers.models.llama.configuration_llama  import LlamaConfig\n",
    "from transformers.models.llama.modeling_llama import LLAMA_INPUTS_DOCSTRING,LLAMA_START_DOCSTRING\n",
    "\n",
    "logger = logging.get_logger('llama')\n",
    "\n",
    "config = LlamaConfig(\n",
    "    vocab_size=len(vocab),\n",
    "    hidden_size=512,\n",
    "    intermediate_size=2752,\n",
    "    num_hidden_layers=8,\n",
    "    num_attention_heads=16,\n",
    "    hidden_act='silu',\n",
    "    max_position_embeddings=128,\n",
    "    initializer_range=0.02,\n",
    "    rms_norm_eps=1e-06,\n",
    "    use_cache=True,\n",
    "    pad_token_id=0,\n",
    "    bos_token_id=1,\n",
    "    eos_token_id=2,\n",
    "    tie_word_embeddings=False\n",
    ") \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2db69fca-5b81-4918-a944-13814a7114b8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0cae4d3b-7fc0-4d44-b6b2-cabb289d8ea5",
   "metadata": {},
   "source": [
    "### 1，旋转位置编码 RoPE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac27f1a4-891d-450f-a6d9-b6cdbf827b1e",
   "metadata": {},
   "source": [
    "旋转位置编码即使用旋转矩阵表示位置编码(Rotary Position Encoding),简称RoPE。\n",
    "\n",
    "关于RoPE的3个核心要点知识如下：\n",
    "\n",
    "- **RoPE的设计思想是使用绝对位置编码来达到相对位置编码的效果。**\n",
    "\n",
    "- **RoPE的实现方式是使用旋转矩阵来表示绝对位置编码。**\n",
    "\n",
    "- **使用NTK扩展方法可以让RoPE在短文本上训练并在长文本上做预测。**\n",
    "\n",
    "参考文章：\n",
    "\n",
    "《博采众长的旋转式位置编码》https://kexue.fm/archives/8265 \n",
    "\n",
    "《RoPE是一种$\\beta$进制编码》https://kexue.fm/archives/9675"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25640126-50a0-44be-8b61-8553bbc9a269",
   "metadata": {},
   "source": [
    "（1）绝对位置编码和相对位置编码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e329378-a34f-4582-be69-926b393e6fff",
   "metadata": {},
   "source": [
    "位置编码一般可以分成绝对位置编码和相对位置编码。\n",
    "\n",
    "绝对位置编码的优点是计算简单高效，缺点是一般效果不如相对位置编码。\n",
    "\n",
    "相对位置编码的优点是效果较好，缺点是计算效率不如绝对位置编码。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81b357e2-10ed-4f60-b1e2-9fecd2b08d11",
   "metadata": {},
   "source": [
    "绝对位置编码：$attention(\\boldsymbol{x}_m,\\boldsymbol{x}_n,m,n) = f(\\boldsymbol{x}_m,\\boldsymbol{x}_n,m,n)$\n",
    "\n",
    "相对位置编码：$attention(\\boldsymbol{x}_m,\\boldsymbol{x}_n,m,n) = g(\\boldsymbol{x}_m,\\boldsymbol{x}_n,m-n)$\n",
    "\n",
    "在相对位置编码中，注意力权重的结果仅仅和参与注意力计算的token向量的相对位置有关，不和绝对位置直接关联。\n",
    "\n",
    "这符合NLP领域在序列长度方向上具有平移不变性的特点，所以相对位置编码一般效果会优于绝对位置编码。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "088d1512-dd52-45d1-beb8-5255737d39bb",
   "metadata": {},
   "source": [
    "不过绝对位置编码并非一无是处，绝对位置编码只需要初始化时对序列的每个位置(数量正比于序列长度)赋予位置编码即可，后续无需干预。\n",
    "\n",
    "而相对位置编码要在计算过程中获取许多个(数量正比于序列长度平方)相对位置。\n",
    "\n",
    "因此绝对位置编码更加简单高效。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54eacffa-e36d-486c-b9ad-47e7521b5c48",
   "metadata": {},
   "source": [
    "（2）使用旋转矩阵表示位置编码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "992ecce9-0329-4757-89ec-59993595a59b",
   "metadata": {},
   "source": [
    "上述讨论可以看到，绝对位置编码和相对位置编码互有优劣，那么有没有什么办法能够对二者进行取长补短呢？\n",
    "\n",
    "有的，这个方法就是RoPE，它的设计思想就是使用绝对位置编码来达到相对位置编码的效果。\n",
    "\n",
    "那么旋转位置编码如何使用绝对位置编码来达到相对位置编码的效果的呢？答案是使用旋转矩阵来表示位置编码。\n",
    "\n",
    "$attention(\\boldsymbol{x}_m,\\boldsymbol{x}_n,m,n) = f(\\boldsymbol{x}_m,\\boldsymbol{x}_n,m,n)$\n",
    "\n",
    "$f(\\boldsymbol{x}_m,\\boldsymbol{x}_n,m,n) \\propto \\langle p(\\boldsymbol{x}_m,m),p(\\boldsymbol{x}_n,n) \\rangle$\n",
    "\n",
    "$p(\\boldsymbol{x}_k,k) = \\boldsymbol{R}(k)\\boldsymbol{x}_k$\n",
    "\n",
    "其中 $\\boldsymbol{R}(k)$为旋转矩阵，满足性质 $\\boldsymbol{R}(m)^T\\boldsymbol{R}(n)=\\boldsymbol{R}(n-m)$。于是，有：\n",
    "$$\\langle p(\\boldsymbol{x}_m,m),p(\\boldsymbol{x}_n,n) \\rangle = \n",
    "(\\boldsymbol{R}(m)\\boldsymbol{x}_m)^T  (\\boldsymbol{R}(n)\\boldsymbol{x}_n)\n",
    "=\\boldsymbol{x}_m^T \\boldsymbol{R}(n-m)\\boldsymbol{x}_n \n",
    "$$\n",
    "\n",
    "$\\boldsymbol{x}_m^T \\boldsymbol{R}(n-m)\\boldsymbol{x}_n$ 符合 $g(\\boldsymbol{x}_m,\\boldsymbol{x}_n,m-n)$ 相对位置编码形式。\n",
    "\n",
    "perfect! 我们用绝对位置编码实现了相对位置编码的效果。\n",
    "\n",
    "\n",
    "那么，旋转矩阵长什么样呢？\n",
    "\n",
    "在二维情形长下面样子。\n",
    "\n",
    "$$\\boldsymbol{R}(k) = \\left(\\begin{array}{cc}\n",
    "\\cos k \\theta & -\\sin k \\theta \\\\\n",
    "\\sin k \\theta & \\cos k \\theta\n",
    "\\end{array}\\right)$$\n",
    "\n",
    "在NLP领域，词向量的维度一般会很高（例如4096）。\n",
    "\n",
    "利用矩阵的分块思想，可以证明高维情形下扩展成下述形式依旧满足旋转矩阵性质$\\boldsymbol{R}(m)^T\\boldsymbol{R}(n)=\\boldsymbol{R}(-m+n)$ \n",
    "\n",
    "$$\n",
    "\\boldsymbol{R}(k) = \\left(\\begin{array}{ccccccc}\n",
    "\\cos k \\theta_{0} & -\\sin k \\theta_{0} & 0 & 0 & \\cdots & 0 & 0 \\\\\n",
    "\\sin k \\theta_{0} & \\cos k \\theta_{0} & 0 & 0 & \\ldots & 0 & 0 \\\\\n",
    "0 & 0 & \\cos k \\theta_{1} & -\\sin k \\theta_{1} & \\ldots & 0 & 0 \\\\\n",
    "0 & 0 & \\sin k \\theta_{1} & \\cos k \\theta_{1} & \\ldots & 0 & 0 \\\\\n",
    "\\vdots & \\vdots & \\vdots & \\vdots & \\ddots & \\vdots & \\vdots \\\\\n",
    "0 & 0 & 0 & 0 & \\ldots & \\cos k \\theta_{d / 2-1} & -\\sin k \\theta_{d / 2-1} \\\\\n",
    "0 & 0 & 0 & 0 & \\ldots & \\sin k \\theta_{d / 2-1} & \\cos k \\theta_{d / 2-1}\n",
    "\\end{array}\\right)\n",
    "$$\n",
    "\n",
    "其中 $\\theta_i = 10000^{-2i/d}$，即越高的维度对应三角函数$sin(\\theta_{i} k)$的系数越小，周期越大，变化越缓慢。\n",
    "\n",
    "\n",
    "由于旋转矩阵是稀疏矩阵，直接使用乘法计算会很浪费算力，可以将旋转位置编码过程由矩阵乘法运算简化成两次向量的哈达玛积求和。\n",
    "\n",
    "$$\\boldsymbol{R}(k)\\boldsymbol{x} = \n",
    "\\left(\\begin{array}{l}\n",
    "\\cos k \\theta_{0} \\\\\n",
    "\\cos k \\theta_{0} \\\\\n",
    "\\cos k \\theta_{1} \\\\\n",
    "\\cos k \\theta_{1} \\\\\n",
    "\\vdots \\\\\n",
    "\\cos k \\theta_{d / 2-1} \\\\\n",
    "\\cos k \\theta_{d / 2-1} \\\\\n",
    "\\end{array}\\right)\n",
    "\\circ\n",
    "\\left(\\begin{array}{l}\n",
    "x_{0} \\\\\n",
    "x_{1} \\\\\n",
    "x_{2} \\\\\n",
    "x_{3} \\\\\n",
    "\\vdots \\\\\n",
    "x_{d-1} \\\\\n",
    "x_{d} \\\\\n",
    "\\end{array}\\right)\n",
    "+\n",
    "\\left(\\begin{array}{l}\n",
    "\\sin k \\theta_{0} \\\\\n",
    "\\sin k \\theta_{0} \\\\\n",
    "\\sin k \\theta_{1} \\\\\n",
    "\\sin k \\theta_{1} \\\\\n",
    "\\vdots \\\\\n",
    "\\sin k \\theta_{d / 2-1} \\\\\n",
    "\\sin k \\theta_{d / 2-1} \\\\\n",
    "\\end{array}\\right)\n",
    "\\circ\n",
    "\\left(\\begin{array}{l}\n",
    "-x_{1} \\\\\n",
    "x_{0} \\\\\n",
    "-x_{3} \\\\\n",
    "x_{2} \\\\\n",
    "\\vdots \\\\\n",
    "-x_{d} \\\\\n",
    "x_{d-1} \\\\\n",
    "\\end{array}\\right)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb67a79e-757a-4af9-a3d9-d6639b4dfcc8",
   "metadata": {},
   "source": [
    "（3）旋转位置编码的长度扩展\n",
    "\n",
    "在LLM的应用中，有一个非常重要的参数，叫做LLM支持的上下文长度(max context length)。\n",
    "\n",
    "更长的上下文长度允许我们进行更多轮次的对话，允许我们对更长的本文进行总结分析，也允许我们生成更长的文章。\n",
    "\n",
    "但是在训练LLM的时候，我们的训练语料大部分是不够长的，许多LLM训练时候设计的最大文本长度都是只有2k，也就是最长2048个token。\n",
    "\n",
    "那么，能否在训练的时候使用较短的文本，而在推理的时候扩展到长文本上呢？\n",
    "\n",
    "是有可能的，我们可以对RoPE进行长度扩展。\n",
    "\n",
    "我们介绍3种扩展方案。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9b604c2-d32d-4768-ba44-ef7e8d453f53",
   "metadata": {},
   "source": [
    "第一种是直接外推：直接外推其实就是继续沿用现有的位置编码公式，不做任何修改。\n",
    "\n",
    "在扩展长度不太长的时候，例如由2k扩展到2.5k时，这种方法可能对性能的影响并不大。\n",
    "\n",
    "因为旋转位置编码只和相对位置m-n的大小有关，一般具有远程衰减性，即相对距离越大的两个token，其相关性一般越弱。\n",
    "\n",
    "因此如果我们的模型已经从训练数据那里学习到了token之间的相关性相对于相对距离在0-2k的一个合适的衰减规律的时候，可以设想把这个规律应用到0-2.5k也是没有太大的问题的。\n",
    "\n",
    "但是如果我们要扩展到更长的长度，例如从2k扩展到32k，这种直接外推的方案通常会严重地影响性能。因为我们学习到的衰减规律有可能在5k的那里就完全衰减截断基本降为0了，这样我们就无法捕捉相对距离长于5k的两个token之间的相互作用，外推就会导致性能下降。\n",
    "\n",
    "总结一下，直接外推对衰减规律在长距离情况下的使用容易出现问题，导致性能下降。\n",
    "\n",
    "为了减少长度外推对性能的影响，我们可以让训练好的模型在更长的上下文上做少许步骤的微调。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fb86d6f-a7eb-494f-8dcc-19f36525ef58",
   "metadata": {},
   "source": [
    "第二种是线性内插：线性内插需要改变位置编码公式，等效于将位置序号等比例缩小。\n",
    "\n",
    "编码公式变化如 $\\cos(k\\theta_i) \\to \\cos(k\\theta_i/\\lambda)$，当从2k扩展到32k，等效于需要将位置序号变成原来的1/16.\n",
    "\n",
    "线性内插没有改变模型学习到的衰减规律的应用范围，不考虑微调的话，其效果一般好于直接外推方案。\n",
    "\n",
    "但是，扩展倍数非常大的时候，例如从2k扩展到32k，其性能也会明显的受到影响。\n",
    "\n",
    "因为在这种情况下，衰减规律在短距离情况下的使用会受到较严重的影响，本来距离为1的两个token，长度扩展后相当于变成了距离为1/16，衰减规律在短距离时可能具有非常大的变化率，因此对相关性的评估可能会极端地偏离合理值。\n",
    "\n",
    "应用线性内插时，在长文本上做少许步骤的微调也能够明显地改善性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f53ce8a1-610a-481f-914c-6340f13f5643",
   "metadata": {},
   "source": [
    "第三种是NTK扩展方式：这种方式综合了外推和内插的优点，做长度扩展后即使不微调也能够保持较好的性能。\n",
    "\n",
    "前面的分析我们知道直接外推对衰减规律在长距离情况下的使用容易出问题，在短距离情况下的使用不受影响。\n",
    "\n",
    "而线性内插对衰减规律在短距离情况下的使用容易出现问题，在长距离的情况下影响较小。\n",
    "\n",
    "我们能否将它们综合起来，在短距离情况下具有外推特性(与扩展前基本一致)，在长距离情况下具有内插特性(缩放到扩展前的范围)，从而使得长距离情况下和短距离情况下衰减规律的使用都不太受到影响呢。\n",
    "\n",
    "我们观察RoPE位置编码第$i$行的元素计算公式 $\\cos(k \\theta_{i}) = \\cos(k 10000^{-2i/d})$，可以发现$i$越大，三角函数对应的角频率系数$\\theta_{i}$越小，或者说越低频，对应的三角函数变化越慢。\n",
    "\n",
    "容易得到如下直观结论：短距离之间的差异(例如1和5的差异)，主要体现在高频分量(i比较小)上，长距离之间的差异(例如5000和10000的差异)，主要体现在低频分量(i比较大)上。\n",
    "\n",
    "为了在短距离情况下具有外推特性，而在长距离情况下具有内插特性，我们可以设计一个和$i$有关的位置序号缩放因子$\\lambda(i)$，使得$\\lambda(i)$在最高频($i=0$)时取值为1(与扩展前基本一致)，而在最低频时($i=d/2-1$)恰好为缩放倍数的倒数$1/s$(缩放到扩展前的范围)。\n",
    "\n",
    "一种有效的选择方案是$i$的指数函数，其效果相当于对$\\cos(k \\theta_{i}) = \\cos(k 10000^{-2i/d})$中的$10000$做一个缩放，根据边界条件容易求得合适的缩放因子为 $ s^{\\frac{d}{d-2}} $ 。\n",
    "\n",
    "NTK扩展方式的要点是高频外推，低频内插，实现方法是直接对底数base进行缩放，类似进制编码转换。\n",
    "\n",
    "采用NTK扩展到长文本，即使不做微调，性能会只会略有下降。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6918c1bf-7698-4ff8-82d6-5cc01af9488f",
   "metadata": {},
   "source": [
    "下面是RoPE以及三种长度扩展方式的实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "ebb3cd5a-1bd2-4efa-a955-56fc5ad047e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LlamaRotaryEmbedding(torch.nn.Module):\n",
    "    def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None):\n",
    "        super().__init__()\n",
    "        self.dim = dim\n",
    "        self.max_position_embeddings = max_position_embeddings\n",
    "        self.base = base\n",
    "        inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))\n",
    "        self.register_buffer(\"inv_freq\", inv_freq, persistent=False) #persistent=False将不会作为state_dict\n",
    "\n",
    "        # Build here to make `torch.jit.trace` work.\n",
    "        self._set_cos_sin_cache(\n",
    "            seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype()\n",
    "        )\n",
    "\n",
    "    def _set_cos_sin_cache(self, seq_len, device, dtype):\n",
    "        self.max_seq_len_cached = seq_len\n",
    "        t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)\n",
    "\n",
    "        freqs = torch.einsum(\"i,j->ij\", t, self.inv_freq)\n",
    "        # Different from paper, but it uses a different permutation in order to obtain the same calculation\n",
    "        emb = torch.cat((freqs, freqs), dim=-1)\n",
    "        self.register_buffer(\"cos_cached\", emb.cos()[None, None, :, :].to(dtype), persistent=False)\n",
    "        self.register_buffer(\"sin_cached\", emb.sin()[None, None, :, :].to(dtype), persistent=False)\n",
    "\n",
    "    def forward(self, x, seq_len=None):\n",
    "        # x: [bs, num_attention_heads, seq_len, head_size]\n",
    "        #超过预设的max_position_embeddings则重新计算更大的Rope缓存，否则直接在缓存上切片\n",
    "        if seq_len > self.max_seq_len_cached: \n",
    "            self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype)\n",
    "\n",
    "        return (\n",
    "            self.cos_cached[:, :, :seq_len, ...].to(dtype=x.dtype),\n",
    "            self.sin_cached[:, :, :seq_len, ...].to(dtype=x.dtype),\n",
    "        )\n",
    "\n",
    "    \n",
    "class LlamaLinearScalingRotaryEmbedding(LlamaRotaryEmbedding):\n",
    "    \"\"\"LlamaRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev\"\"\"\n",
    "\n",
    "    def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):\n",
    "        self.scaling_factor = scaling_factor\n",
    "        super().__init__(dim, max_position_embeddings, base, device)\n",
    "\n",
    "    def _set_cos_sin_cache(self, seq_len, device, dtype):\n",
    "        self.max_seq_len_cached = seq_len\n",
    "        t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)\n",
    "        t = t / self.scaling_factor #线性内插相当于将位置序号等比例缩小\n",
    "\n",
    "        freqs = torch.einsum(\"i,j->ij\", t, self.inv_freq)\n",
    "        # Different from paper, but it uses a different permutation in order to obtain the same calculation\n",
    "        emb = torch.cat((freqs, freqs), dim=-1)\n",
    "        self.register_buffer(\"cos_cached\", emb.cos()[None, None, :, :].to(dtype), persistent=False)\n",
    "        self.register_buffer(\"sin_cached\", emb.sin()[None, None, :, :].to(dtype), persistent=False)\n",
    "\n",
    "\n",
    "class LlamaDynamicNTKScalingRotaryEmbedding(LlamaRotaryEmbedding):\n",
    "    \"\"\"LlamaRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla\"\"\"\n",
    "\n",
    "    def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None, scaling_factor=1.0):\n",
    "        self.scaling_factor = scaling_factor\n",
    "        super().__init__(dim, max_position_embeddings, base, device)\n",
    "\n",
    "    def _set_cos_sin_cache(self, seq_len, device, dtype):\n",
    "        self.max_seq_len_cached = seq_len\n",
    "\n",
    "        if seq_len > self.max_position_embeddings:\n",
    "            base = self.base * (\n",
    "                (self.scaling_factor * seq_len / self.max_position_embeddings) - (self.scaling_factor - 1)\n",
    "            ) ** (self.dim / (self.dim - 2))  #NTK扩展方式直接对base进行缩放\n",
    "            inv_freq = 1.0 / (base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))\n",
    "            self.register_buffer(\"inv_freq\", inv_freq, persistent=False)\n",
    "\n",
    "        t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)\n",
    "\n",
    "        freqs = torch.einsum(\"i,j->ij\", t, self.inv_freq)\n",
    "        \n",
    "        #此处处理逻辑与原始的ROPE有差异，原始逻辑如下\n",
    "        #emb = torch.cat((freqs, freqs), dim=-1)\n",
    "        #emb[...,0::2]=freqs\n",
    "        #emb[...,1::2]=freqs\n",
    "        \n",
    "        \n",
    "        # Different from paper, but it uses a different permutation in order to obtain the same calculation\n",
    "        emb = torch.cat((freqs, freqs), dim=-1)\n",
    "        self.register_buffer(\"cos_cached\", emb.cos()[None, None, :, :].to(dtype), persistent=False)\n",
    "        self.register_buffer(\"sin_cached\", emb.sin()[None, None, :, :].to(dtype), persistent=False)\n",
    "        \n",
    "        \n",
    "def rotate_half(x):\n",
    "    \"\"\"Rotates half the hidden dims of the input.\"\"\"\n",
    "    \n",
    "    #此处逻辑与原始的ROPE有所差异，原始逻辑如下\n",
    "    #x1 = x[..., 0::2] \n",
    "    #x2 = x[..., 1::2]\n",
    "    #res = torch.cat((x1, x2), dim=-1)\n",
    "    #res[...,0::2]=-x2\n",
    "    #res[...,1::2]=x1\n",
    "    #return res\n",
    "    \n",
    "    x1 = x[..., : x.shape[-1] // 2] \n",
    "    x2 = x[..., x.shape[-1] // 2 :]\n",
    "    return torch.cat((-x2, x1), dim=-1)\n",
    "\n",
    "\n",
    "def apply_rotary_pos_emb(q, k, cos, sin, position_ids):\n",
    "    # The first two dimensions of cos and sin are always 1, so we can `squeeze` them.\n",
    "    cos = cos.squeeze(1).squeeze(0)  # [seq_len, dim]\n",
    "    sin = sin.squeeze(1).squeeze(0)  # [seq_len, dim]\n",
    "    cos = cos[position_ids].unsqueeze(1)  # [bs, 1, seq_len, dim]\n",
    "    sin = sin[position_ids].unsqueeze(1)  # [bs, 1, seq_len, dim]\n",
    "    q_embed = (q * cos) + (rotate_half(q) * sin)\n",
    "    k_embed = (k * cos) + (rotate_half(k) * sin)\n",
    "    return q_embed, k_embed\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "d3ee3371-d515-4bdf-9f40-192a040c9a01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 1, 4, 8])\n",
      "tensor([[[[ 1.0000,  1.0000,  1.0000,  1.0000,  1.0000,  1.0000,  1.0000,\n",
      "            1.0000],\n",
      "          [ 0.5403,  0.9950,  0.9999,  1.0000,  0.5403,  0.9950,  0.9999,\n",
      "            1.0000],\n",
      "          [-0.4161,  0.9801,  0.9998,  1.0000, -0.4161,  0.9801,  0.9998,\n",
      "            1.0000],\n",
      "          [-0.9900,  0.9553,  0.9996,  1.0000, -0.9900,  0.9553,  0.9996,\n",
      "            1.0000]]]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(1,8,4,2)\n",
    "rope = LlamaRotaryEmbedding(dim=8)\n",
    "cos,sin = rope.forward(x,seq_len=4)\n",
    "print(cos.shape) \n",
    "print(cos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c52cc140-1d21-496f-98d4-f1ee6a79be7a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18cf1fe9-b7cb-4800-8f9d-722a8cb10164",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "84317093-4482-473a-8fe2-ca3f43d9ceab",
   "metadata": {},
   "source": [
    "### 2，多头注意力 LlamaAttention"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8de001e8-3671-4803-89bd-5b913b26f2a5",
   "metadata": {},
   "source": [
    "这里的LlamaAttention 基本上和《Attention Is All You Need》论文里的是一致的，主要差异有以下一些。\n",
    "\n",
    "1，k和v的head数量可以是q的head数量的几分之一，类似分组卷积的思想，可以减少参数规模。\n",
    "\n",
    "2，rope位置编码是每次做多头注意力时都进行一次，而不是原论文只在输入的时候进行一次。\n",
    "\n",
    "3，允许传入key和value的states的缓存past_key_value，这在多轮对话中可以减少重复计算，起到加速效果。\n",
    "\n",
    "4，attention_mask是通过加法形式作用到softmax之前的attention矩阵上的。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "18595a50-0b6c-453e-be38-c091cde2c258",
   "metadata": {},
   "outputs": [],
   "source": [
    "def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:\n",
    "    \"\"\"\n",
    "    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,\n",
    "    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)\n",
    "    \"\"\"\n",
    "    batch, num_key_value_heads, slen, head_dim = hidden_states.shape\n",
    "    if n_rep == 1:\n",
    "        return hidden_states\n",
    "    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)\n",
    "    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)\n",
    "\n",
    "\n",
    "class LlamaAttention(nn.Module):\n",
    "    \"\"\"Multi-headed attention from 'Attention Is All You Need' paper\"\"\"\n",
    "\n",
    "    def __init__(self, config: LlamaConfig):\n",
    "        super().__init__()\n",
    "        self.config = config\n",
    "        self.hidden_size = config.hidden_size\n",
    "        self.num_heads = config.num_attention_heads\n",
    "        self.head_dim = self.hidden_size // self.num_heads\n",
    "        self.num_key_value_heads = config.num_key_value_heads\n",
    "        self.num_key_value_groups = self.num_heads // self.num_key_value_heads\n",
    "        self.max_position_embeddings = config.max_position_embeddings\n",
    "\n",
    "        if (self.head_dim * self.num_heads) != self.hidden_size:\n",
    "            raise ValueError(\n",
    "                f\"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}\"\n",
    "                f\" and `num_heads`: {self.num_heads}).\"\n",
    "            )\n",
    "        self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)\n",
    "        self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)\n",
    "        self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)\n",
    "        self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False)\n",
    "        self._init_rope()\n",
    "\n",
    "    def _init_rope(self):\n",
    "        if self.config.rope_scaling is None:\n",
    "            self.rotary_emb = LlamaRotaryEmbedding(self.head_dim, max_position_embeddings=self.max_position_embeddings)\n",
    "        else:\n",
    "            scaling_type = self.config.rope_scaling[\"type\"]\n",
    "            scaling_factor = self.config.rope_scaling[\"factor\"]\n",
    "            if scaling_type == \"linear\":\n",
    "                self.rotary_emb = LlamaLinearScalingRotaryEmbedding(\n",
    "                    self.head_dim, max_position_embeddings=self.max_position_embeddings, scaling_factor=scaling_factor\n",
    "                )\n",
    "            elif scaling_type == \"dynamic\":\n",
    "                self.rotary_emb = LlamaDynamicNTKScalingRotaryEmbedding(\n",
    "                    self.head_dim, max_position_embeddings=self.max_position_embeddings, scaling_factor=scaling_factor\n",
    "                )\n",
    "            else:\n",
    "                raise ValueError(f\"Unknown RoPE scaling type {scaling_type}\")\n",
    "\n",
    "    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):\n",
    "        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()\n",
    "\n",
    "    def forward(\n",
    "        self,\n",
    "        hidden_states: torch.Tensor,\n",
    "        attention_mask: Optional[torch.Tensor] = None,\n",
    "        position_ids: Optional[torch.LongTensor] = None,\n",
    "        past_key_value: Optional[Tuple[torch.Tensor]] = None,\n",
    "        output_attentions: bool = False,\n",
    "        use_cache: bool = False,\n",
    "    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:\n",
    "        bsz, q_len, _ = hidden_states.size()\n",
    "\n",
    "        if self.config.pretraining_tp > 1:\n",
    "            key_value_slicing = (self.num_key_value_heads * self.head_dim) // self.config.pretraining_tp\n",
    "            query_slices = self.q_proj.weight.split(\n",
    "                (self.num_heads * self.head_dim) // self.config.pretraining_tp, dim=0\n",
    "            )\n",
    "            key_slices = self.k_proj.weight.split(key_value_slicing, dim=0)\n",
    "            value_slices = self.v_proj.weight.split(key_value_slicing, dim=0)\n",
    "\n",
    "            query_states = [F.linear(hidden_states, query_slices[i]) for i in range(self.config.pretraining_tp)]\n",
    "            query_states = torch.cat(query_states, dim=-1)\n",
    "\n",
    "            key_states = [F.linear(hidden_states, key_slices[i]) for i in range(self.config.pretraining_tp)]\n",
    "            key_states = torch.cat(key_states, dim=-1)\n",
    "\n",
    "            value_states = [F.linear(hidden_states, value_slices[i]) for i in range(self.config.pretraining_tp)]\n",
    "            value_states = torch.cat(value_states, dim=-1)\n",
    "\n",
    "        else:\n",
    "            query_states = self.q_proj(hidden_states)\n",
    "            key_states = self.k_proj(hidden_states)\n",
    "            value_states = self.v_proj(hidden_states)\n",
    "\n",
    "        query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)\n",
    "        key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)\n",
    "        value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)\n",
    "\n",
    "        kv_seq_len = key_states.shape[-2]\n",
    "        if past_key_value is not None:\n",
    "            kv_seq_len += past_key_value[0].shape[-2]\n",
    "        cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)\n",
    "        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)\n",
    "\n",
    "        if past_key_value is not None:\n",
    "            # reuse k, v, self_attention\n",
    "            key_states = torch.cat([past_key_value[0], key_states], dim=2)\n",
    "            value_states = torch.cat([past_key_value[1], value_states], dim=2)\n",
    "\n",
    "        past_key_value = (key_states, value_states) if use_cache else None\n",
    "\n",
    "        # repeat k/v heads if n_kv_heads < n_heads\n",
    "        key_states = repeat_kv(key_states, self.num_key_value_groups)\n",
    "        value_states = repeat_kv(value_states, self.num_key_value_groups)\n",
    "\n",
    "        attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)\n",
    "\n",
    "        if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):\n",
    "            raise ValueError(\n",
    "                f\"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is\"\n",
    "                f\" {attn_weights.size()}\"\n",
    "            )\n",
    "\n",
    "        if attention_mask is not None:\n",
    "            if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):\n",
    "                raise ValueError(\n",
    "                    f\"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}\"\n",
    "                )\n",
    "            attn_weights = attn_weights + attention_mask\n",
    "\n",
    "        # upcast attention to fp32\n",
    "        attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)\n",
    "        attn_output = torch.matmul(attn_weights, value_states)\n",
    "\n",
    "        if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):\n",
    "            raise ValueError(\n",
    "                f\"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is\"\n",
    "                f\" {attn_output.size()}\"\n",
    "            )\n",
    "\n",
    "        attn_output = attn_output.transpose(1, 2).contiguous()\n",
    "        attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)\n",
    "\n",
    "        if self.config.pretraining_tp > 1:\n",
    "            attn_output = attn_output.split(self.hidden_size // self.config.pretraining_tp, dim=2)\n",
    "            o_proj_slices = self.o_proj.weight.split(self.hidden_size // self.config.pretraining_tp, dim=1)\n",
    "            attn_output = sum([F.linear(attn_output[i], o_proj_slices[i]) for i in range(self.config.pretraining_tp)])\n",
    "        else:\n",
    "            attn_output = self.o_proj(attn_output)\n",
    "\n",
    "        if not output_attentions:\n",
    "            attn_weights = None\n",
    "\n",
    "        return attn_output, attn_weights, past_key_value\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb9c5e1d-2437-4709-a98e-400afa96d170",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6383a525-fe17-4a51-993a-02975cb1c4b0",
   "metadata": {},
   "source": [
    "### 3，前馈网络 LlamaMLP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5638efcd-3f49-41a0-8c02-d4e17a80a617",
   "metadata": {},
   "source": [
    "前馈网络是一个2层的感知机MLP。\n",
    "\n",
    "先从hidden_size维度up_proj到intermediate_size维度，然后再down_proj还原为hidden_size维度。\n",
    "\n",
    "这里的主要特色是引入了一个gate_proj配合激活函数来实现一个门控注意力的作用。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "d8537f5e-af17-4afe-a3e4-7152cff1fdce",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LlamaMLP(nn.Module):\n",
    "    def __init__(self, config):\n",
    "        super().__init__()\n",
    "        self.config = config\n",
    "        self.hidden_size = config.hidden_size\n",
    "        self.intermediate_size = config.intermediate_size\n",
    "        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)\n",
    "        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)\n",
    "        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)\n",
    "        self.act_fn = ACT2FN[config.hidden_act]\n",
    "\n",
    "    def forward(self, x):\n",
    "        if self.config.pretraining_tp > 1:\n",
    "            slice = self.intermediate_size // self.config.pretraining_tp\n",
    "            gate_proj_slices = self.gate_proj.weight.split(slice, dim=0)\n",
    "            up_proj_slices = self.up_proj.weight.split(slice, dim=0)\n",
    "            down_proj_slices = self.down_proj.weight.split(slice, dim=1)\n",
    "\n",
    "            gate_proj = torch.cat(\n",
    "                [F.linear(x, gate_proj_slices[i]) for i in range(self.config.pretraining_tp)], dim=-1\n",
    "            )\n",
    "            up_proj = torch.cat([F.linear(x, up_proj_slices[i]) for i in range(self.config.pretraining_tp)], dim=-1)\n",
    "\n",
    "            intermediate_states = (self.act_fn(gate_proj) * up_proj).split(slice, dim=2)\n",
    "            down_proj = [\n",
    "                F.linear(intermediate_states[i], down_proj_slices[i]) for i in range(self.config.pretraining_tp)\n",
    "            ]\n",
    "            down_proj = sum(down_proj)\n",
    "        else:\n",
    "            down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))\n",
    "\n",
    "        return down_proj\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a450428-9268-42d4-bb59-3ebdebb4a587",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "798897ec-83a4-49fb-b394-124bc3033cdf",
   "metadata": {},
   "source": [
    "### 4，层归一化 LlamaRMSNorm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9ad4866-a2d3-4609-b281-a3a0f33acf07",
   "metadata": {},
   "source": [
    "这里的层归一化叫做RMSNorm，和标准的LayerNorm有少许差异。\n",
    "\n",
    "首先是没有移除均值，直接除的RootMeanSquare，然后也没有加上bias。\n",
    "\n",
    "这两个小的修正可以保证在层归一化不会改变hidden_states对应的词向量的方向，只会改变其模长。\n",
    "\n",
    "在一定的意义上具有合理性。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "e696b32c-9834-43c1-ba3b-d3e2aa3d834d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LlamaRMSNorm(nn.Module):\n",
    "    def __init__(self, hidden_size, eps=1e-6):\n",
    "        \"\"\"\n",
    "        LlamaRMSNorm is equivalent to T5LayerNorm\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(torch.ones(hidden_size))\n",
    "        self.variance_epsilon = eps\n",
    "\n",
    "    def forward(self, hidden_states):\n",
    "        input_dtype = hidden_states.dtype\n",
    "        hidden_states = hidden_states.to(torch.float32)\n",
    "        variance = hidden_states.pow(2).mean(-1, keepdim=True)\n",
    "        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)\n",
    "        return self.weight * hidden_states.to(input_dtype)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60ebeeb9-a88f-46c2-8feb-5e81cc648b49",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d09de2d3-ee36-4e4b-b1f7-bc6341ca3e09",
   "metadata": {},
   "source": [
    "### 5，Llama解码层"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72e93aec-3e61-4a4d-ba69-848dd3de72ce",
   "metadata": {},
   "source": [
    "解码层LlamaDecoderLayer由LlamaAttention，LlamaMLP，以及两个LlamaRMSNorm组成，并使用了两次残差结构。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "0ea4c464-3605-42b5-ade8-c7cfd50f40e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LlamaDecoderLayer(nn.Module):\n",
    "    def __init__(self, config: LlamaConfig):\n",
    "        super().__init__()\n",
    "        self.hidden_size = config.hidden_size\n",
    "        self.self_attn = LlamaAttention(config=config)\n",
    "        self.mlp = LlamaMLP(config)\n",
    "        self.input_layernorm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n",
    "        self.post_attention_layernorm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n",
    "\n",
    "    def forward(\n",
    "        self,\n",
    "        hidden_states: torch.Tensor,\n",
    "        attention_mask: Optional[torch.Tensor] = None,\n",
    "        position_ids: Optional[torch.LongTensor] = None,\n",
    "        past_key_value: Optional[Tuple[torch.Tensor]] = None,\n",
    "        output_attentions: Optional[bool] = False,\n",
    "        use_cache: Optional[bool] = False,\n",
    "    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`\n",
    "            attention_mask (`torch.FloatTensor`, *optional*): attention mask of size\n",
    "                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.\n",
    "            output_attentions (`bool`, *optional*):\n",
    "                Whether or not to return the attentions tensors of all attention layers. See `attentions` under\n",
    "                returned tensors for more detail.\n",
    "            use_cache (`bool`, *optional*):\n",
    "                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding\n",
    "                (see `past_key_values`).\n",
    "            past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states\n",
    "        \"\"\"\n",
    "\n",
    "        residual = hidden_states\n",
    "\n",
    "        hidden_states = self.input_layernorm(hidden_states)\n",
    "\n",
    "        # Self Attention\n",
    "        hidden_states, self_attn_weights, present_key_value = self.self_attn(\n",
    "            hidden_states=hidden_states,\n",
    "            attention_mask=attention_mask,\n",
    "            position_ids=position_ids,\n",
    "            past_key_value=past_key_value,\n",
    "            output_attentions=output_attentions,\n",
    "            use_cache=use_cache,\n",
    "        )\n",
    "        hidden_states = residual + hidden_states\n",
    "\n",
    "        # Fully Connected\n",
    "        residual = hidden_states\n",
    "        hidden_states = self.post_attention_layernorm(hidden_states)\n",
    "        hidden_states = self.mlp(hidden_states)\n",
    "        hidden_states = residual + hidden_states\n",
    "\n",
    "        outputs = (hidden_states,)\n",
    "\n",
    "        if output_attentions:\n",
    "            outputs += (self_attn_weights,)\n",
    "\n",
    "        if use_cache:\n",
    "            outputs += (present_key_value,)\n",
    "\n",
    "        return outputs\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e648bbde-d205-42a6-92a8-8016d7104e96",
   "metadata": {},
   "source": [
    "### 6，Llama解码器"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5bb3322-0747-44ea-bd6a-8eb46f67b221",
   "metadata": {},
   "source": [
    "LlamaModel由多个Llama解码层堆叠而成。\n",
    "\n",
    "有几个理解上的要点：\n",
    "\n",
    "1，`_make_causal_mask`用于构造下三角这种mask结构以实现语言模型的单向注意力。\n",
    "\n",
    "2，`_expand_mask`用于将传入的<pad>等特殊符号相关的mask信息展开成和attention矩阵相同的张量结构。\n",
    "\n",
    "3，设置gradient_checkpointing=True可以节约显存。其主要应用了torch.utils.checkpoint.checkpoint方法。\n",
    "它的原理非常简单，在对decoder_layer进行forward时不保存中间激活值从而节约显存，backward时重新计算相关值，从而通过时间换取了空间。\n",
    "\n",
    "4，gradient_checkpointing和use_cache不能同时设置为True，前者是为了节约显存时间换空间的，后者是为了节约时间空间换时间。\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "1c02d6c3-32a1-456e-9555-33045f3a3e21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copied from transformers.models.bart.modeling_bart._make_causal_mask\n",
    "def _make_causal_mask(\n",
    "    input_ids_shape: torch.Size, dtype: torch.dtype, \n",
    "    device: torch.device, past_key_values_length: int = 0\n",
    "):\n",
    "    \"\"\"\n",
    "    Make causal mask used for bi-directional self-attention.\n",
    "    \"\"\"\n",
    "    bsz, tgt_len = input_ids_shape\n",
    "    mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device)\n",
    "    mask_cond = torch.arange(mask.size(-1), device=device)\n",
    "    mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)\n",
    "    mask = mask.to(dtype)\n",
    "\n",
    "    if past_key_values_length > 0:\n",
    "        mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)\n",
    "    return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)\n",
    "\n",
    "\n",
    "# Copied from transformers.models.bart.modeling_bart._expand_mask\n",
    "def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):\n",
    "    \"\"\"\n",
    "    Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.\n",
    "    \"\"\"\n",
    "    bsz, src_len = mask.size()\n",
    "    tgt_len = tgt_len if tgt_len is not None else src_len\n",
    "\n",
    "    expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)\n",
    "    inverted_mask = 1.0 - expanded_mask\n",
    "\n",
    "    return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)\n",
    "\n",
    "\n",
    "@add_start_docstrings(\n",
    "    \"The bare LLaMA Model outputting raw hidden-states without any specific head on top.\",\n",
    "    LLAMA_START_DOCSTRING,\n",
    ")\n",
    "class LlamaPreTrainedModel(PreTrainedModel):\n",
    "    config_class = LlamaConfig\n",
    "    base_model_prefix = \"model\"\n",
    "    supports_gradient_checkpointing = True\n",
    "    _no_split_modules = [\"LlamaDecoderLayer\"]\n",
    "    _skip_keys_device_placement = \"past_key_values\"\n",
    "\n",
    "    def _init_weights(self, module):\n",
    "        std = self.config.initializer_range\n",
    "        if isinstance(module, nn.Linear):\n",
    "            module.weight.data.normal_(mean=0.0, std=std)\n",
    "            if module.bias is not None:\n",
    "                module.bias.data.zero_()\n",
    "        elif isinstance(module, nn.Embedding):\n",
    "            module.weight.data.normal_(mean=0.0, std=std)\n",
    "            if module.padding_idx is not None:\n",
    "                module.weight.data[module.padding_idx].zero_()\n",
    "\n",
    "    def _set_gradient_checkpointing(self, module, value=False):\n",
    "        if isinstance(module, LlamaModel):\n",
    "            module.gradient_checkpointing = value\n",
    "\n",
    "\n",
    "@add_start_docstrings(\n",
    "    \"The bare LLaMA Model outputting raw hidden-states without any specific head on top.\",\n",
    "    LLAMA_START_DOCSTRING,\n",
    ")\n",
    "class LlamaModel(LlamaPreTrainedModel):\n",
    "    \"\"\"\n",
    "    Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`]\n",
    "\n",
    "    Args:\n",
    "        config: LlamaConfig\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, config: LlamaConfig):\n",
    "        super().__init__(config)\n",
    "        self.padding_idx = config.pad_token_id\n",
    "        self.vocab_size = config.vocab_size\n",
    "\n",
    "        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)\n",
    "        self.layers = nn.ModuleList([LlamaDecoderLayer(config) for _ in range(config.num_hidden_layers)])\n",
    "        self.norm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)\n",
    "\n",
    "        self.gradient_checkpointing = False\n",
    "        # Initialize weights and apply final processing\n",
    "        self.post_init()\n",
    "\n",
    "    def get_input_embeddings(self):\n",
    "        return self.embed_tokens\n",
    "\n",
    "    def set_input_embeddings(self, value):\n",
    "        self.embed_tokens = value\n",
    "\n",
    "    # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask\n",
    "    def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):\n",
    "        # create causal mask\n",
    "        # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n",
    "        combined_attention_mask = None\n",
    "        if input_shape[-1] > 1:\n",
    "            combined_attention_mask = _make_causal_mask(\n",
    "                input_shape,\n",
    "                inputs_embeds.dtype,\n",
    "                device=inputs_embeds.device,\n",
    "                past_key_values_length=past_key_values_length,\n",
    "            )\n",
    "\n",
    "        if attention_mask is not None:\n",
    "            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n",
    "            expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(\n",
    "                inputs_embeds.device\n",
    "            )\n",
    "            combined_attention_mask = (\n",
    "                expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask\n",
    "            )\n",
    "\n",
    "        return combined_attention_mask\n",
    "\n",
    "    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)\n",
    "    def forward(\n",
    "        self,\n",
    "        input_ids: torch.LongTensor = None,\n",
    "        attention_mask: Optional[torch.Tensor] = None,\n",
    "        position_ids: Optional[torch.LongTensor] = None,\n",
    "        past_key_values: Optional[List[torch.FloatTensor]] = None,\n",
    "        inputs_embeds: Optional[torch.FloatTensor] = None,\n",
    "        use_cache: Optional[bool] = None,\n",
    "        output_attentions: Optional[bool] = None,\n",
    "        output_hidden_states: Optional[bool] = None,\n",
    "        return_dict: Optional[bool] = None,\n",
    "    ) -> Union[Tuple, BaseModelOutputWithPast]:\n",
    "        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions\n",
    "        output_hidden_states = (\n",
    "            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states\n",
    "        )\n",
    "        use_cache = use_cache if use_cache is not None else self.config.use_cache\n",
    "\n",
    "        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n",
    "\n",
    "        # retrieve input_ids and inputs_embeds\n",
    "        if input_ids is not None and inputs_embeds is not None:\n",
    "            raise ValueError(\"You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time\")\n",
    "        elif input_ids is not None:\n",
    "            batch_size, seq_length = input_ids.shape\n",
    "        elif inputs_embeds is not None:\n",
    "            batch_size, seq_length, _ = inputs_embeds.shape\n",
    "        else:\n",
    "            raise ValueError(\"You have to specify either decoder_input_ids or decoder_inputs_embeds\")\n",
    "\n",
    "        seq_length_with_past = seq_length\n",
    "        past_key_values_length = 0\n",
    "\n",
    "        if past_key_values is not None:\n",
    "            past_key_values_length = past_key_values[0][0].shape[2]\n",
    "            seq_length_with_past = seq_length_with_past + past_key_values_length\n",
    "\n",
    "        if position_ids is None:\n",
    "            device = input_ids.device if input_ids is not None else inputs_embeds.device\n",
    "            position_ids = torch.arange(\n",
    "                past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device\n",
    "            )\n",
    "            position_ids = position_ids.unsqueeze(0).view(-1, seq_length)\n",
    "        else:\n",
    "            position_ids = position_ids.view(-1, seq_length).long()\n",
    "\n",
    "        if inputs_embeds is None:\n",
    "            inputs_embeds = self.embed_tokens(input_ids)\n",
    "        # embed positions\n",
    "        if attention_mask is None:\n",
    "            attention_mask = torch.ones(\n",
    "                (batch_size, seq_length_with_past), dtype=torch.bool, device=inputs_embeds.device\n",
    "            )\n",
    "        attention_mask = self._prepare_decoder_attention_mask(\n",
    "            attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length\n",
    "        )\n",
    "\n",
    "        hidden_states = inputs_embeds\n",
    "\n",
    "        if self.gradient_checkpointing and self.training:\n",
    "            if use_cache:\n",
    "                logger.warning_once(\n",
    "                    \"`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...\"\n",
    "                )\n",
    "                use_cache = False\n",
    "\n",
    "        # decoder layers\n",
    "        all_hidden_states = () if output_hidden_states else None\n",
    "        all_self_attns = () if output_attentions else None\n",
    "        next_decoder_cache = () if use_cache else None\n",
    "\n",
    "        for idx, decoder_layer in enumerate(self.layers):\n",
    "            if output_hidden_states:\n",
    "                all_hidden_states += (hidden_states,)\n",
    "\n",
    "            past_key_value = past_key_values[idx] if past_key_values is not None else None\n",
    "\n",
    "            if self.gradient_checkpointing and self.training:\n",
    "\n",
    "                def create_custom_forward(module):\n",
    "                    def custom_forward(*inputs):\n",
    "                        # None for past_key_value\n",
    "                        return module(*inputs, output_attentions, None)\n",
    "\n",
    "                    return custom_forward\n",
    "\n",
    "                layer_outputs = torch.utils.checkpoint.checkpoint(\n",
    "                    create_custom_forward(decoder_layer),\n",
    "                    hidden_states,\n",
    "                    attention_mask,\n",
    "                    position_ids,\n",
    "                    None,\n",
    "                )\n",
    "            else:\n",
    "                layer_outputs = decoder_layer(\n",
    "                    hidden_states,\n",
    "                    attention_mask=attention_mask,\n",
    "                    position_ids=position_ids,\n",
    "                    past_key_value=past_key_value,\n",
    "                    output_attentions=output_attentions,\n",
    "                    use_cache=use_cache,\n",
    "                )\n",
    "\n",
    "            hidden_states = layer_outputs[0]\n",
    "\n",
    "            if use_cache:\n",
    "                next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)\n",
    "\n",
    "            if output_attentions:\n",
    "                all_self_attns += (layer_outputs[1],)\n",
    "\n",
    "        hidden_states = self.norm(hidden_states)\n",
    "\n",
    "        # add hidden states from the last decoder layer\n",
    "        if output_hidden_states:\n",
    "            all_hidden_states += (hidden_states,)\n",
    "\n",
    "        next_cache = next_decoder_cache if use_cache else None\n",
    "        if not return_dict:\n",
    "            return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)\n",
    "        return BaseModelOutputWithPast(\n",
    "            last_hidden_state=hidden_states,\n",
    "            past_key_values=next_cache,\n",
    "            hidden_states=all_hidden_states,\n",
    "            attentions=all_self_attns,\n",
    "        )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0b36301-4648-4f98-9b68-2aa83d444b77",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d3456287-23d5-495f-8cd8-b914ff02a2ce",
   "metadata": {},
   "source": [
    "### 7，Llama语言模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "468b933f-b5ee-4996-a5cf-07e4497d0ae6",
   "metadata": {},
   "source": [
    "Llama语言模型 LlamaForCausalLM是在Llama解码器LlamaModel的基础上增加了一个lm_head作为Generator。\n",
    "\n",
    "从而实现了一个完整的语言模型。\n",
    "\n",
    "除此之外，Llama语言模型还实现了以下重要功能。\n",
    "\n",
    "1，loss计算功能。当forward方法中传入labels时，会自动计算语言模型的交叉熵损失。注意labels中的-100会被忽略不参与计算。\n",
    "\n",
    "2，文本生成generate方法。这个方法继承自PreTrainedModel，可以设置model.generation_config.num_beams选择束搜索的束宽度，默认为1即贪心搜索。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "92f07485-d8e8-4384-aedf-1b807faae5a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "_CONFIG_FOR_DOC = \"LlamaConfig\"\n",
    "\n",
    "class LlamaForCausalLM(LlamaPreTrainedModel):\n",
    "    _tied_weights_keys = [\"lm_head.weight\"]\n",
    "\n",
    "    def __init__(self, config):\n",
    "        super().__init__(config)\n",
    "        self.model = LlamaModel(config)\n",
    "        self.vocab_size = config.vocab_size\n",
    "        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)\n",
    "\n",
    "        # Initialize weights and apply final processing\n",
    "        self.post_init()\n",
    "\n",
    "    def get_input_embeddings(self):\n",
    "        return self.model.embed_tokens\n",
    "\n",
    "    def set_input_embeddings(self, value):\n",
    "        self.model.embed_tokens = value\n",
    "\n",
    "    def get_output_embeddings(self):\n",
    "        return self.lm_head\n",
    "\n",
    "    def set_output_embeddings(self, new_embeddings):\n",
    "        self.lm_head = new_embeddings\n",
    "\n",
    "    def set_decoder(self, decoder):\n",
    "        self.model = decoder\n",
    "\n",
    "    def get_decoder(self):\n",
    "        return self.model\n",
    "\n",
    "    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)\n",
    "    @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)\n",
    "    def forward(\n",
    "        self,\n",
    "        input_ids: torch.LongTensor = None,\n",
    "        attention_mask: Optional[torch.Tensor] = None,\n",
    "        position_ids: Optional[torch.LongTensor] = None,\n",
    "        past_key_values: Optional[List[torch.FloatTensor]] = None,\n",
    "        inputs_embeds: Optional[torch.FloatTensor] = None,\n",
    "        labels: Optional[torch.LongTensor] = None,\n",
    "        use_cache: Optional[bool] = None,\n",
    "        output_attentions: Optional[bool] = None,\n",
    "        output_hidden_states: Optional[bool] = None,\n",
    "        return_dict: Optional[bool] = None,\n",
    "    ) -> Union[Tuple, CausalLMOutputWithPast]:\n",
    "        r\"\"\"\n",
    "        Args:\n",
    "            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):\n",
    "                Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,\n",
    "                config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored\n",
    "                (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.\n",
    "\n",
    "        Returns:\n",
    "\n",
    "        Example:\n",
    "\n",
    "        ```python\n",
    "        >>> from transformers import AutoTokenizer, LlamaForCausalLM\n",
    "\n",
    "        >>> model = LlamaForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)\n",
    "        >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)\n",
    "\n",
    "        >>> prompt = \"Hey, are you conscious? Can you talk to me?\"\n",
    "        >>> inputs = tokenizer(prompt, return_tensors=\"pt\")\n",
    "\n",
    "        >>> # Generate\n",
    "        >>> generate_ids = model.generate(inputs.input_ids, max_length=30)\n",
    "        >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\n",
    "        \"Hey, are you conscious? Can you talk to me?\\nI'm not conscious, but I can talk to you.\"\n",
    "        ```\"\"\"\n",
    "\n",
    "        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions\n",
    "        output_hidden_states = (\n",
    "            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states\n",
    "        )\n",
    "        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n",
    "\n",
    "        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)\n",
    "        outputs = self.model(\n",
    "            input_ids=input_ids,\n",
    "            attention_mask=attention_mask,\n",
    "            position_ids=position_ids,\n",
    "            past_key_values=past_key_values,\n",
    "            inputs_embeds=inputs_embeds,\n",
    "            use_cache=use_cache,\n",
    "            output_attentions=output_attentions,\n",
    "            output_hidden_states=output_hidden_states,\n",
    "            return_dict=return_dict,\n",
    "        )\n",
    "\n",
    "        hidden_states = outputs[0]\n",
    "        if self.config.pretraining_tp > 1:\n",
    "            lm_head_slices = self.lm_head.weight.split(self.vocab_size // self.config.pretraining_tp, dim=0)\n",
    "            logits = [F.linear(hidden_states, lm_head_slices[i]) for i in range(self.config.pretraining_tp)]\n",
    "            logits = torch.cat(logits, dim=-1)\n",
    "        else:\n",
    "            logits = self.lm_head(hidden_states)\n",
    "        logits = logits.float()\n",
    "\n",
    "        loss = None\n",
    "        if labels is not None:\n",
    "            # Shift so that tokens < n predict n\n",
    "            shift_logits = logits[..., :-1, :].contiguous()\n",
    "            shift_labels = labels[..., 1:].contiguous()\n",
    "            # Flatten the tokens\n",
    "            loss_fct = CrossEntropyLoss()\n",
    "            shift_logits = shift_logits.view(-1, self.config.vocab_size)\n",
    "            shift_labels = shift_labels.view(-1)\n",
    "            # Enable model parallelism\n",
    "            shift_labels = shift_labels.to(shift_logits.device)\n",
    "            loss = loss_fct(shift_logits, shift_labels)\n",
    "\n",
    "        if not return_dict:\n",
    "            output = (logits,) + outputs[1:]\n",
    "            return (loss,) + output if loss is not None else output\n",
    "\n",
    "        return CausalLMOutputWithPast(\n",
    "            loss=loss,\n",
    "            logits=logits,\n",
    "            past_key_values=outputs.past_key_values,\n",
    "            hidden_states=outputs.hidden_states,\n",
    "            attentions=outputs.attentions,\n",
    "        )\n",
    "\n",
    "    def prepare_inputs_for_generation(\n",
    "        self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs\n",
    "    ):\n",
    "        if past_key_values:\n",
    "            input_ids = input_ids[:, -1:]\n",
    "\n",
    "        position_ids = kwargs.get(\"position_ids\", None)\n",
    "        if attention_mask is not None and position_ids is None:\n",
    "            # create position_ids on the fly for batch generation\n",
    "            position_ids = attention_mask.long().cumsum(-1) - 1\n",
    "            position_ids.masked_fill_(attention_mask == 0, 1)\n",
    "            if past_key_values:\n",
    "                position_ids = position_ids[:, -1].unsqueeze(-1)\n",
    "\n",
    "        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step\n",
    "        if inputs_embeds is not None and past_key_values is None:\n",
    "            model_inputs = {\"inputs_embeds\": inputs_embeds}\n",
    "        else:\n",
    "            model_inputs = {\"input_ids\": input_ids}\n",
    "\n",
    "        model_inputs.update(\n",
    "            {\n",
    "                \"position_ids\": position_ids,\n",
    "                \"past_key_values\": past_key_values,\n",
    "                \"use_cache\": kwargs.get(\"use_cache\"),\n",
    "                \"attention_mask\": attention_mask,\n",
    "            }\n",
    "        )\n",
    "        return model_inputs\n",
    "\n",
    "    @staticmethod\n",
    "    def _reorder_cache(past_key_values, beam_idx):\n",
    "        reordered_past = ()\n",
    "        for layer_past in past_key_values:\n",
    "            reordered_past += (\n",
    "                tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past),\n",
    "            )\n",
    "        return reordered_past\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1761110e-09b4-4530-97da-fb2d9bcfd2aa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "24f0f333-b3c5-41aa-ba61-4f1acffc9dd3",
   "metadata": {},
   "source": [
    "### 8，Llama分类模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1547ce1-1898-4376-b37e-96d3df810706",
   "metadata": {},
   "source": [
    "LlamaForSequenceClassification是一个序列分类模型。\n",
    "\n",
    "这个分类模型可以用来训练RLHF流程中的Reward模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "9dff9c94-3247-4b4b-97ea-1f89166bc446",
   "metadata": {},
   "outputs": [],
   "source": [
    "@add_start_docstrings(\n",
    "    \"\"\"\n",
    "    The LLaMa Model transformer with a sequence classification head on top (linear layer).\n",
    "\n",
    "    [`LlamaForSequenceClassification`] uses the last token in order to do the classification, as other causal models\n",
    "    (e.g. GPT-2) do.\n",
    "\n",
    "    Since it does classification on the last token, it requires to know the position of the last token. If a\n",
    "    `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If\n",
    "    no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the\n",
    "    padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in\n",
    "    each row of the batch).\n",
    "    \"\"\",\n",
    "    LLAMA_START_DOCSTRING,\n",
    ")\n",
    "class LlamaForSequenceClassification(LlamaPreTrainedModel):\n",
    "    def __init__(self, config):\n",
    "        super().__init__(config)\n",
    "        self.num_labels = config.num_labels\n",
    "        self.model = LlamaModel(config)\n",
    "        self.score = nn.Linear(config.hidden_size, self.num_labels, bias=False)\n",
    "\n",
    "        # Initialize weights and apply final processing\n",
    "        self.post_init()\n",
    "\n",
    "    def get_input_embeddings(self):\n",
    "        return self.model.embed_tokens\n",
    "\n",
    "    def set_input_embeddings(self, value):\n",
    "        self.model.embed_tokens = value\n",
    "\n",
    "    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)\n",
    "    def forward(\n",
    "        self,\n",
    "        input_ids: torch.LongTensor = None,\n",
    "        attention_mask: Optional[torch.Tensor] = None,\n",
    "        position_ids: Optional[torch.LongTensor] = None,\n",
    "        past_key_values: Optional[List[torch.FloatTensor]] = None,\n",
    "        inputs_embeds: Optional[torch.FloatTensor] = None,\n",
    "        labels: Optional[torch.LongTensor] = None,\n",
    "        use_cache: Optional[bool] = None,\n",
    "        output_attentions: Optional[bool] = None,\n",
    "        output_hidden_states: Optional[bool] = None,\n",
    "        return_dict: Optional[bool] = None,\n",
    "    ) -> Union[Tuple, SequenceClassifierOutputWithPast]:\n",
    "        r\"\"\"\n",
    "        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):\n",
    "            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,\n",
    "            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If\n",
    "            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).\n",
    "        \"\"\"\n",
    "        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n",
    "\n",
    "        transformer_outputs = self.model(\n",
    "            input_ids,\n",
    "            attention_mask=attention_mask,\n",
    "            position_ids=position_ids,\n",
    "            past_key_values=past_key_values,\n",
    "            inputs_embeds=inputs_embeds,\n",
    "            use_cache=use_cache,\n",
    "            output_attentions=output_attentions,\n",
    "            output_hidden_states=output_hidden_states,\n",
    "            return_dict=return_dict,\n",
    "        )\n",
    "        hidden_states = transformer_outputs[0]\n",
    "        logits = self.score(hidden_states)\n",
    "\n",
    "        if input_ids is not None:\n",
    "            batch_size = input_ids.shape[0]\n",
    "        else:\n",
    "            batch_size = inputs_embeds.shape[0]\n",
    "\n",
    "        if self.config.pad_token_id is None and batch_size != 1:\n",
    "            raise ValueError(\"Cannot handle batch sizes > 1 if no padding token is defined.\")\n",
    "        if self.config.pad_token_id is None:\n",
    "            sequence_lengths = -1\n",
    "        else:\n",
    "            if input_ids is not None:\n",
    "                sequence_lengths = (torch.eq(input_ids, self.config.pad_token_id).long().argmax(-1) - 1).to(\n",
    "                    logits.device\n",
    "                )\n",
    "            else:\n",
    "                sequence_lengths = -1\n",
    "\n",
    "        pooled_logits = logits[torch.arange(batch_size, device=logits.device), sequence_lengths]\n",
    "\n",
    "        loss = None\n",
    "        if labels is not None:\n",
    "            labels = labels.to(logits.device)\n",
    "            if self.config.problem_type is None:\n",
    "                if self.num_labels == 1:\n",
    "                    self.config.problem_type = \"regression\"\n",
    "                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):\n",
    "                    self.config.problem_type = \"single_label_classification\"\n",
    "                else:\n",
    "                    self.config.problem_type = \"multi_label_classification\"\n",
    "\n",
    "            if self.config.problem_type == \"regression\":\n",
    "                loss_fct = MSELoss()\n",
    "                if self.num_labels == 1:\n",
    "                    loss = loss_fct(pooled_logits.squeeze(), labels.squeeze())\n",
    "                else:\n",
    "                    loss = loss_fct(pooled_logits, labels)\n",
    "            elif self.config.problem_type == \"single_label_classification\":\n",
    "                loss_fct = CrossEntropyLoss()\n",
    "                loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1))\n",
    "            elif self.config.problem_type == \"multi_label_classification\":\n",
    "                loss_fct = BCEWithLogitsLoss()\n",
    "                loss = loss_fct(pooled_logits, labels)\n",
    "        if not return_dict:\n",
    "            output = (pooled_logits,) + transformer_outputs[1:]\n",
    "            return ((loss,) + output) if loss is not None else output\n",
    "\n",
    "        return SequenceClassifierOutputWithPast(\n",
    "            loss=loss,\n",
    "            logits=pooled_logits,\n",
    "            past_key_values=transformer_outputs.past_key_values,\n",
    "            hidden_states=transformer_outputs.hidden_states,\n",
    "            attentions=transformer_outputs.attentions,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca065aab-6dbc-44a1-a852-00fb130bef00",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "284add62-b392-4d1d-aaab-bafe31c00c5b",
   "metadata": {},
   "source": [
    "## 三，训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75f8655f-43e1-4796-8d30-a823fb15ba64",
   "metadata": {},
   "source": [
    "下面，我们来训练一个LlamaForCausalLM 实现两数之和的任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "id": "a7453bcd-5793-414e-96b2-51391d330290",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = LlamaConfig(\n",
    "    vocab_size=len(vocab),\n",
    "    hidden_size=512,\n",
    "    intermediate_size=2752,\n",
    "    num_hidden_layers=8,\n",
    "    num_attention_heads=16,\n",
    "    num_key_value_heads=4,\n",
    "    rope_scaling = None,\n",
    "    hidden_act='silu',\n",
    "    max_position_embeddings=128,\n",
    "    initializer_range=0.02,\n",
    "    rms_norm_eps=1e-06,\n",
    "    use_cache=True,\n",
    "    pad_token_id=0,\n",
    "    bos_token_id=1,\n",
    "    eos_token_id=2,\n",
    "    tie_word_embeddings=False,\n",
    "    pretraining_tp = 1,\n",
    "    max_new_tokens = 100\n",
    ") \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "368de379-e2f4-4d98-870d-c75220eb10a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2.7630, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "#试算一下\n",
    "model = LlamaForCausalLM(config)\n",
    "out = model.forward(**batch)\n",
    "print(out.loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "9697d87c-7341-4640-b8e5-5da6afc14f21",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "a10404cc-e771-4928-9381-62d9b5cc4bd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "from accelerate import Accelerator \n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator if accelerator is not None else Accelerator() \n",
    "        if self.stage=='train':\n",
    "            self.net.train() \n",
    "        else:\n",
    "            self.net.eval()\n",
    "    \n",
    "    def __call__(self, batch):\n",
    "        \n",
    "        #loss\n",
    "        with self.accelerator.autocast():\n",
    "            loss = self.net(**batch).loss\n",
    "\n",
    "        #backward()\n",
    "        if self.stage==\"train\" and self.optimizer is not None:        \n",
    "            self.accelerator.backward(loss)\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "        \n",
    "        #losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "        \n",
    "        #metrics (stateful metrics)\n",
    "        step_metrics = {}\n",
    "        \n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "    \n",
    "KerasModel.StepRunner = StepRunner \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "c07fac6c-a41d-42fb-9833-ee5ab2d0eaee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< ⚡️ cuda is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAGJCAYAAADIVkprAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjLklEQVR4nO3deVhUZf8G8PvMwAz7PiIICokrKqS5oOFSmrmloWlpqa9v9ZZa7pW/cq20NM09W960zB1R3zLNJVFyyz3FFUVFRUFkl82Z5/fHOCMjDAwwMCz357rmAs555sx3jujcnmc5khBCgIiIiMiMZJYugIiIiKofBgwiIiIyOwYMIiIiMjsGDCIiIjI7BgwiIiIyOwYMIiIiMjsGDCIiIjI7BgwiIiIyOwYMIiIiMjsGDCoX06dPhyRJuHfvnqVLqTDXrl2DJElYuXJluT6HKgeNRoNmzZrh888/t3QpFcrPzw+9e/e2dBl6SUlJsLe3x++//27pUugJDBhUrcyaNQtbtmyxdBk1xv/+9z+0bNkSNjY2qFu3LqZNm4aHDx+a9FyNRoM5c+bA398fNjY2aNGiBdauXVto2/Pnz+PFF1+Eg4MD3Nzc8MYbbyAxMdGgzYULF/DBBx8gODgYjo6O8PLyQq9evXDs2LECx7t48SLGjRuH9u3bw8bGBpIk4dq1ayV672vXrkVcXBxGjx5tUvv09HTwzgyGrly5oj//hf055ffWW29BkqQC4cbd3R1vvvkmpkyZUp6lUikwYFC1woBRcbZv345+/frBxcUFixcvRr9+/fDZZ5/hvffeM+n5H3/8MT788EN069YNixcvRt26dTF48GCsW7fOoN3NmzfRsWNHxMTEYNasWZg4cSK2bduGbt26ITc3V9/uhx9+wPfff49nnnkG8+bNw/jx43Hx4kW0a9cOu3fvNjjmoUOHsGjRIqSnp6NJkyalev9z587Fq6++Cmdn50L3P3z4ED/88AM6d+4MGxsbODk5wdbWFm3atMGiRYuQk5NTqtetTsaNGwcrK6ti2x07dgwrV66EjY1NofvfeecdnDhxAn/++ae5S6SyEETlYNq0aQKASExMrNDXtbe3F8OGDavQ19SJjY0VAMSKFSvK9TmVRdOmTUVQUJDIy8vTb/v444+FJEni/PnzRT735s2bwtraWowaNUq/TaPRiNDQUOHj4yMePnyo3/7uu+8KW1tbcf36df22Xbt2CQDi22+/1W87duyYSE9PN3ide/fuCZVKJTp06GCwPSkpSaSlpQkhhJg7d64AIGJjY01+7ydOnBAAxO7duwvdHxMTIwIDA/W/jytWrBC///67WLVqlRg5cqRwc3MTjRs3FmfPnjX5NSuLevXqiV69epX5ODt27BAKhUJ88sknAoA4evRooe00Go0ICQkRI0aMKPK1mzVrJt54440y10XmwysYVK7u3buHgQMHwsnJCe7u7hgzZgyys7MLtPvll1/QqlUr2Nraws3NDa+++iri4uIM2ly+fBn9+/dH7dq1YWNjAx8fH7z66qtITU0FAEiShMzMTPz000+QJAmSJGH48OGF1nX37l1YWVlhxowZBfZdvHgRkiRhyZIlAID79+9j4sSJaN68ORwcHODk5IQePXrg9OnTZTw7xv35558IDQ2Fvb09XFxc0LdvX5w/f96gTXp6OsaOHQs/Pz8olUrUqlUL3bp1w4kTJ/RtijtnpXXu3DmcO3cOb7/9tsH/QEeOHAkhBMLDw4t8/tatW5GXl4eRI0fqt0mShHfffRc3b97EoUOH9Ns3bdqE3r17o27duvptXbt2RcOGDbFhwwb9tlatWsHBwcHgddzd3REaGlrg3Lm5ucHR0bFkbzqfLVu2QKFQoGPHjgX23bp1Cx06dICnpycuX76MlStXYvjw4ejRowdef/11LF26FDExMXj66afRrVu3Qrtmtm/frv/zd3R0RK9evRAdHW3QZvjw4XBwcMDVq1fRvXt32Nvbw9vbGzNnzizQFZOZmYkJEybA19cXSqUSjRo1wldffVVol80vv/yCNm3awM7ODq6urujYsSN27txZoN1ff/2FNm3awMbGBk899RR+/vlnk89fXl4exowZgzFjxqB+/fpFtl21ahXOnj1b7FiXbt264ddff2U3VCXCgEHlauDAgcjOzsbs2bPRs2dPLFq0CG+//bZBm88//xxDhw5FgwYNMH/+fIwdOxZ79uxBx44dkZKSAgDIzc1F9+7dcfjwYbz33ntYunQp3n77bVy9elXfZtWqVVAqlQgNDcWqVauwatUq/Oc//ym0Lk9PT3Tq1MngA0pn/fr1kMvleOWVVwAAV69exZYtW9C7d2/Mnz8fkyZNwpkzZ9CpUyfcvn3bfCfrkd27d6N79+5ISEjA9OnTMX78eBw8eBAdOnQw+DB655138M0336B///5YtmwZJk6cCFtbW/2HqSnnDABSU1Nx7969Yh8ZGRn655w8eRIA8MwzzxjU7u3tDR8fH/1+Y06ePAl7e/sC3RNt2rQxOP6tW7eQkJBQ4HV0bYt7HQC4c+cOPDw8im1XEgcPHkSzZs1gbW1dYN/QoUPRokUL7NixA15eXgC0H6i6YJ2TkwOZTIbVq1ejY8eOePfddw2ev2rVKvTq1QsODg748ssvMWXKFJw7dw7PPvtsgTCiVqvx4osvwtPTE3PmzEGrVq0wbdo0TJs2Td9GCIGXXnoJX3/9NV588UXMnz8fjRo1wqRJkzB+/HiD482YMQNvvPEGrK2tMXPmTMyYMQO+vr4Fuh5iYmIwYMAAdOvWDfPmzYOrqyuGDx9eIAQZs2DBAiQnJ+OTTz4psl16ejo+/PBD/N///R9q165dZNtWrVohJSXF5BqoAlj0+glVW7oukpdeeslg+8iRIwUAcfr0aSGEENeuXRNyuVx8/vnnBu3OnDkjrKys9NtPnjwpAIiNGzcW+bol6SL59ttvBQBx5swZg+1NmzYVzz33nP7n7OxsoVarDdrExsYKpVIpZs6cabANZugiCQ4OFrVq1RJJSUn6badPnxYymUwMHTpUv83Z2dmgi+FJpp6zTp06CQDFPvKfV123wo0bNwocr3Xr1qJdu3ZFvmavXr3EU089VWB7ZmamACA++ugjIYQQR48eFQDEzz//XKDtpEmTBACRnZ1t9HX2798vJEkSU6ZMMdqmNF0kPj4+on///gW2R0ZGCnt7e3Hr1i0hhBB5eXli5MiRQqFQCEmSRK9evcRXX30lOnXqJIQQIiEhQdjY2IhLly4JIYRIT08XLi4u4q233jI47p07d4Szs7PB9mHDhgkA4r333tNv02g0olevXkKhUOi7J7ds2SIAiM8++8zgmAMGDBCSJImYmBghhBCXL18WMplMvPzyywV+3zUajf77evXqCQBi//79+m0JCQlCqVSKCRMmFHvu4uPjhaOjo757a8WKFUa7SCZOnCj8/f31f8ZFdZEcPHhQABDr168vtgaqGLyCQeVq1KhRBj/rBgDqppRFRERAo9Fg4MCBBv9brl27Nho0aIC9e/cCgH4g3R9//IEHDx6YpbawsDBYWVlh/fr1+m1nz57FuXPnMGjQIP02pVIJmUz7V0WtViMpKQkODg5o1KiRQXeEOcTHx+PUqVMYPnw43Nzc9NtbtGiBbt26GUzFc3FxwZEjR4xeRTH1nM2bNw+7du0q9vHBBx/on5OVlQVAe26eZGNjo99vTFZWltHn5j9+ca+Tv82TEhISMHjwYPj7+xvUbg5JSUlwdXUtsH3jxo3o378/vL29AQCLFy/GihUrMHXqVERERMDT0xNTp07Vt1epVAgJCUFkZCQAYNeuXUhJScFrr71m8PdBLpejbdu2+r8P+eWfxSJJEkaPHo3c3Fz9wNbff/8dcrkc77//vsHzJkyYACEEtm/fDkDb7aPRaDB16lT973v+4+bXtGlThIaGGryPRo0a4erVq8Weuw8//BBPPfUU3nzzzSLbXbp0CQsXLsTcuXML/fN/ku7PoyZNja/sih++S1QGDRo0MPi5fv36kMlk+ku9ly9fhhCiQDsd3SVof39/jB8/HvPnz8fq1asRGhqKl156Ca+//rrRUfzF8fDwwPPPP48NGzbg008/BaDtHrGyskJYWJi+nUajwcKFC7Fs2TLExsZCrVbr97m7u5fqtY25fv06AKBRo0YF9jVp0gR//PEHMjMzYW9vjzlz5mDYsGHw9fVFq1at0LNnTwwdOhRPPfUUANPPWatWrUpcp62tLQAUOhMiOztbv7+o5xt7bv7jF/c6+dvkl5mZid69eyM9PR1//fVXgbEZ5iAK6es/fvy4Qbfc999/j48++ggff/wxAKBfv364cuWKwXM8PT31U24vX74MAHjuuecKfU0nJyeDn2Uymf7PW6dhw4YAoP87dv36dXh7excYc6LrntL9zl25cgUymQxNmzY18o4fyz8eRsfV1RXJyclFPu/w4cNYtWoV9uzZUyDEPGnMmDFo3749+vfvX2w9wOM/jyfDEFkOAwZVqCf/8ms0GkiShO3bt0Mulxdon/+DYd68eRg+fDi2bt2KnTt34v3338fs2bNx+PBh+Pj4lKqeV199Ff/6179w6tQpBAcHY8OGDXj++ecN+uxnzZqFKVOmYMSIEfj000/h5uYGmUyGsWPHQqPRlOp1zWHgwIEIDQ3F5s2bsXPnTsydOxdffvklIiIi0KNHDwCmnbP79+8bTPc0xtbWVh9MdGML4uPj4evra9AuPj5eP5bCGC8vL+zduxdCCIPfifj4eADQXwHI/zpPio+Ph5ubW4H/3ebm5iIsLAz//PMP/vjjDzRr1qzY91ZS7u7uhX6YJiUl6WsHtB/yrVu3NmjTpk0b/P333/qf4+Li0LlzZwDQ/z6tWrWq0DEHpkzprAiF/V0FCg9d+X3wwQcIDQ2Fv7+/PgDprjjEx8fjxo0bqFu3Lv7880/s2LEDERERBuNOHj58iKysLFy7dg1ubm4GgUv352Hu8TZUepXjt5WqrcuXL8Pf31//c0xMDDQaDfz8/ABor2gIIeDv76//n1dRmjdvjubNm+OTTz7RD3xcvnw5PvvsMwAl/99Lv3798J///EffTXLp0iVMnjzZoE14eDi6dOmC//73vwbbU1JSzP6PWb169QBoZ7I86cKFC/Dw8IC9vb1+m5eXF0aOHImRI0ciISEBLVu2xOeff64PGEDx5ywsLAz79u0rtrZhw4bpVxwNDg4GoF2fIH+YuH37Nm7evFlgIO+TgoOD8cMPP+D8+fMG/2M+cuSIwfHr1KkDlUpV6CJMf//9t76djkajwdChQ7Fnzx5s2LABnTp1KvZ9lUbjxo0RGxtbYLuTk5PBDJ3atWsXuGKRvxshOjoaR44cwYoVKwBAP6OiVq1a6Nq1a7F1aDQaXL161eDvzqVLlwBA/3esXr162L17N9LT0w2uYly4cEG/X/faGo0G586dK3BezeXGjRu4fv26wb8JOi+99BKcnZ2RkpKCGzduAIDBlUSdW7duwd/fH19//TXGjh2r36778yjtuiZkfhyDQeVq6dKlBj8vXrwYAPQfgGFhYZDL5ZgxY0aB//0IIZCUlAQASEtLK7BCZPPmzSGTyQwun9vb2xvMkCiOi4sLunfvjg0bNmDdunVQKBTo16+fQRu5XF6gto0bN+LWrVsmv46pvLy8EBwcjJ9++sngfZw9exY7d+5Ez549AWjHgjw51bRWrVrw9vbWnw9Tz1lpxmAEBgaicePG+O677wy6jL755htIkoQBAwbot6WmpuLChQsG9fbt2xfW1tZYtmyZfpsQAsuXL0edOnXQvn17/fb+/fvjt99+M5i2vGfPHly6dEk/00fnvffew/r167Fs2bJCP5zMJSQkBGfPni3QddOkSRN9SAKAl19+GZ999hm2bduG69evY9myZdi6dStycnKwadMmdO/eHf/+97/1XYTdu3eHk5MTZs2ahby8vAKv++TqpQD006kB7TlcsmQJrK2t8fzzzwMAevbsCbVabdAOAL7++mtIkqT/u9ivXz/IZDLMnDmzwJW54q5MmOq7777D5s2bDR66cVlfffUVVq9eDUDbRfRku82bN0OlUuGZZ57B5s2b0adPH4NjHz9+HM7OzggMDDRLrWQGFhlaStWebhZJ8+bNRZ8+fcTSpUvF66+/LgCIwYMHG7SdPXu2ACDat28v5syZI7755hvxwQcfiAYNGoi5c+cKIYTYvHmzqFOnjhg7dqxYtmyZWLRokWjdurWwtrYWhw4d0h+rZ8+ewt7eXsybN0+sXbtWHD58uNhaf/nlFwFAODo6ij59+hTYP3XqVAFADB8+XHz33XfivffeE25ubuKpp57SzwYQwnyzSHbt2iWsrKxE48aNxdy5c8XMmTOFSqUSrq6u4urVq0IIIZKTk/UzZubPny++++47MXDgQAFAzJs3r0TnrLR+/fVXIUmSeO6558R3330n3n//fSGTyQrMgNDNEnjyvOhmgbz99tvi+++/F7169RIAxOrVqw3a3bhxQ7i7u4v69euLRYsWiVmzZglXV1fRvHlzgxkkX3/9tQAgQkJCxKpVqwo8MjIy9G1TUlLEp59+Kj799FPx4osvCgBiwoQJ4tNPPxWLFy8u9r0fO3ZMABB//PGHwfZ169YJLy8v8eDBA/3rhISE6Gfi1KtXT3zwwQcCgHBwcBCffPKJwUJlQgixevVqIZPJRLNmzcRnn30mvv32W/Hxxx+L4OBgg1lDw4YNEzY2NqJBgwZi6NChYunSpaJ3794CgPi///s/fTu1Wi26dOkiJEkSb7/9tli6dKno27evACDGjh1r8NpTpkzR/1386quvxOLFi8XQoUP1s3qEMD6To1OnTgZ/H0xV1CySJxW30Nbrr79e4ten8sOAQeVCFzDOnTsnBgwYIBwdHYWrq6sYPXq0yMrKKtB+06ZN4tlnnxX29vbC3t5eNG7cWIwaNUpcvHhRCCHE1atXxYgRI0T9+vWFjY2NcHNzE126dCmwkuKFCxdEx44dha2tbYGplcakpaXp2//yyy8F9mdnZ4sJEyYILy8vYWtrKzp06CAOHTpU4B9Uc67kuXv3btGhQwdha2srnJycRJ8+fcS5c+f0+3NycsSkSZNEUFCQcHR0FPb29iIoKEgsW7ZM38bUc1YWmzdvFsHBwUKpVAofHx/xySefiNzcXIM2xgKGWq0Ws2bNEvXq1RMKhUIEBgYWev6FEOLs2bPihRdeEHZ2dsLFxUUMGTJE3Llzx6CNbtqmsUf+aai6817Yo169eia99xYtWoh///vfBtvy8vJE/fr1DT64NRqNOHnypDh8+LDIzc0V8fHx4vjx4yInJ8fosffu3Su6d+8unJ2dhY2Njahfv74YPny4OHbsmMH7tbe3F1euXNGfG09PTzFt2rQC00zT09PFuHHjhLe3t7C2ttaH9/zTT3V+/PFH8fTTTwulUilcXV1Fp06dxK5du/T7K2PAOH/+fJErq5JlSEJw2TMiopJatWoVRo0ahRs3bsDFxUW//cCBA+jSpQs++ugjTJ8+vdDZEvfv38exY8fwwgsvlPr1hw8fjvDwcIMF0GqqsWPHYv/+/Th+/DhnkVQiHINBRFQKQ4YMQd26dQuMM+rQoQM2bdqE+fPnIzg4GMuXL8c///yDuLg4HDlyBDNmzEDjxo0xdepUi85Cqi6SkpLwww8/4LPPPmO4qGR4BYOoHOTm5uL+/ftFtnF2di52vQiqumJjYzFt2jRs3rzZ4CqDj48PRo8ejTFjxhi9O6gpeAWDKjtOUyUqBwcPHkSXLl2KbLNixQqjN2Ojqs/f3x8///wzcnJycPHiRaSkpMDT07PQRdSIqiNewSAqB8nJyTh+/HiRbQIDA/ULSRERVTcMGERERGR2HORJREREZlfjxmBoNBrcvn0bjo6OHHFMRERUAkIIpKenw9vbu9gb1tW4gHH79u0CN2ciIiIi08XFxRV7k8kaFzB0N/uJi4srcOtjIiIiMi4tLQ2+vr4GN84zpsYFDF23iJOTEwMGERFRKZgyxICDPImIiMjsGDCIiIjI7BgwiIiIyOxq3BgMIiIqH0IIPHz4EGq12tKlUBlYW1tDLpeX+TgMGEREVGa5ubmIj4/HgwcPLF0KlZEkSfDx8YGDg0OZjsOAQUREZaLRaBAbGwu5XA5vb28oFAouZFhFCSGQmJiImzdvokGDBmW6ksGAUUZqNRAVBcTHA15eQGgoYIYrS0REVUZubi40Gg18fX1hZ2dn6XKojFQqFa5du4a8vDwGDEuJiADGjAFu3ny8zccHWLgQCAuzXF1ERJZQ3NLRVDWY6+oTfxtKKSICGDDAMFwAwK1b2u0REZapi4iIqDJgwCgFtVp75aKwG93rto0dq21HRERUEzFglEJUVMErF/kJAcTFadsREZHp1GogMhJYu1b7tSr9R83Pzw8LFiwwy7EiIyMhSRJSUlLMcjxL4BiMUoiPN287IiKyzLi2zp07Izg42CzB4OjRo7C3ty97UdUEr2CUgpeXedsREdV0lXVcm27xMFOoVCrOosmHAaMUQkO1qdrYQFtJAnx9te2IiGqyzEzjj+xsbRtTxrWNGWPYXWLsmCUxfPhw7Nu3DwsXLoQkSZAkCStXroQkSdi+fTtatWoFpVKJv/76C1euXEHfvn3h6ekJBwcHtG7dGrt37zY43pNdJJIk4YcffsDLL78MOzs7NGjQAP/73/9KVmQ+mzZtQmBgIJRKJfz8/DBv3jyD/cuWLUODBg1gY2MDT09PDBgwQL8vPDwczZs3h62tLdzd3dG1a1dklvSElRADRinI5dpLdoDxkLFgAdfDICJycDD+6N9f28aUcW03bxqOa/PzK/yYJbFw4UKEhITgrbfeQnx8POLj4+Hr6wsA+Oijj/DFF1/g/PnzaNGiBTIyMtCzZ0/s2bMHJ0+exIsvvog+ffrgxo0bRb7GjBkzMHDgQPzzzz/o2bMnhgwZgvv375esUADHjx/HwIED8eqrr+LMmTOYPn06pkyZgpUrVwIAjh07hvfffx8zZ87ExYsXsWPHDnTs2BEAEB8fj9deew0jRozA+fPnERkZibCwMIjCEp05iRomNTVVABCpqallPtamTUL4+Aih/fXXPiRJiPXrzVAoEVEVkZWVJc6dOyeysrIK7Mv/7+OTj549tW3WrCm6ne6xZs3j43p4FN6mpDp16iTGjBmj/3nv3r0CgNiyZUuxzw0MDBSLFy/W/1yvXj3x9ddf53vvEJ988on+54yMDAFAbN++vdhj6+pITk4WQggxePBg0a1bN4M2kyZNEk2bNhVCCLFp0ybh5OQk0tLSChzr+PHjAoC4du1asa8rRNF/niX5DOUVjDIICwOuXQP27gV++QVwddX+iru6WroyIqLKISPD+GPTJm2b0oxru3at8GOayzPPPGPwc0ZGBiZOnIgmTZrAxcUFDg4OOH/+fLFXMFq0aKH/3t7eHk5OTkhISChxPefPn0eHDh0MtnXo0AGXL1+GWq1Gt27dUK9ePTz11FN44403sHr1av19YYKCgvD888+jefPmeOWVV/D9998jOTm5xDWUFANGGcnlQOfOwJAhjy/3HTxo0ZKIiCoNe3vjDxsbbZvSjGszdkzz1W14sIkTJ2Lz5s2YNWsWoqKicOrUKTRv3hy5ublFHsfa2trgZ0mSoNFozFfoI46Ojjhx4gTWrl0LLy8vTJ06FUFBQUhJSYFcLseuXbuwfft2NG3aFIsXL0ajRo0QGxtr9jryY8Awow8/BC5dAqZNs3QlRERVR1Hj2nQ/l9e4NoVCYdLt5Q8cOIDhw4fj5ZdfRvPmzVG7dm1cu3bN/AUZ0aRJExw4cKBATQ0bNtTfL8TKygpdu3bFnDlz8M8//+DatWv4888/AWiDTYcOHTBjxgycPHkSCoUCmzdvLteauQ6GGQUEWLoCIqKqKSwMCA8vfB2MBQvKbx0MPz8/HDlyBNeuXYODg4PRqwsNGjRAREQE+vTpA0mSMGXKlHK5EmHMhAkT0Lp1a3z66acYNGgQDh06hCVLlmDZsmUAgN9++w1Xr15Fx44d4erqit9//x0ajQaNGjXCkSNHsGfPHrzwwguoVasWjhw5gsTERDRp0qRca+YVjDJSC4HI5GSsvXsXkcnJUD8alVveg3OJiKqb/OPa1qzRfo2NLd+bR06cOBFyuRxNmzaFSqUyOqZi/vz5cHV1Rfv27dGnTx90794dLVu2LL/CntCyZUts2LAB69atQ7NmzTB16lTMnDkTw4cPBwC4uLggIiICzz33HJo0aYLly5dj7dq1CAwMhJOTE/bv34+ePXuiYcOG+OSTTzBv3jz06NGjXGuWhKhZH4VpaWlwdnZGamoqnJycynSsiMREjImJwc2cHP222nIl6v0aAOURFfbtK2u1RESVX3Z2NmJjY+Hv7w8b3cAKqrKK+vMsyWcor2CUUkRiIgZERxuECwC4q87BkR7R2C8Si5zXTUREVJ0xYJSCWgiMiYlBYZd+9NtGxWDrrzXq4hAREZnonXfegYODQ6GPd955x9LlmQUHeZZCVEpKgSsXBiQAnjn4aU8KRoGLYhARkaGZM2di4sSJhe4ra/d9ZcGAUQrxxcx71jlxPRfp6YCjYzkXREREVUqtWrVQq1YtS5dRrthFUgpeCoVJ7dR3Ffjjj3IuhoiIqBJiwCiFUBcX+CiVMLLoHCQAjllK4IwLynDjPCIioiqLAaMU5JKEhY9W1TIWMj60D0BIWwlt21ZcXURERJUFx2CUUphKhfDAwALrYDjK5VjZuDHCVCp8zHuSEBFRDcUrGGUQplLhWrt22BsUhJHe3gCABjY2CFOpLFwZERGRZTFglJFcktDZ1RWT69YFAJzKzETqw4f6/ffva29JXLPWSyUiKh1jt1+orPz8/LBgwQKT2kqShC1btpRrPZUJu0jMxMfGBvVtbHAlOxsHUlPR090d2dlA3bpAZiZw7hxQzveVISKq0gq7/YKPUomFAQG8MlwF8QqGGXVycQEA7EtJAQDY2AChodp9nE1CRGScsdsv3MrJwYDoaEQkJlqoMiotBgwz0gWMyEcBAwD69tV+3bq14ushIrIUIQQy1WqTHmkPH+L9y5eLvP3CmJgYpD18aNLxTL2H53fffQdvb+8Ct13v27cvRowYgStXrqBv377w9PSEg4MDWrdujd27d5ftxORz5swZPPfcc7C1tYW7uzvefvttZGRk6PdHRkaiTZs2sLe3h4uLCzp06IDr168DAE6fPo0uXbrA0dERTk5OaNWqFY4dO2a22syBXSRmpAsYx9PTkf7wIRytrNC7N/Duu8ChQ8Dy5UDjxtqrGnK5ZWslIipPDzQaOERFmeVYAsDNnBw4//WXSe0zQkNhb8I/sq+88gree+897N27F88//zwA4P79+9ixYwd+//13ZGRkoGfPnvj888+hVCrx888/o0+fPrh48SLqPhp3V1qZmZno3r07QkJCcPToUSQkJODNN9/E6NGjsXLlSjx8+BD9+vXDW2+9hbVr1yI3Nxd///03JEm7OMKQIUPw9NNP45tvvoFcLsepU6dgbW1dpprMjQHDjOrZ2MDPxgbXsrNxMC0N3d3c8PffgLU1kJenDRoA4OMDLFwIhIVZtl4ioprM1dUVPXr0wJo1a/QBIzw8HB4eHujSpQtkMhmCgoL07T/99FNs3rwZ//vf/zB69OgyvfaaNWuQnZ2Nn3/+Gfb29gCAJUuWoE+fPvjyyy9hbW2N1NRU9O7dG/Xr1wcANMk3kO/GjRuYNGkSGjduDABo0KBBmeopDwwYZtbJ2RnXsrOxLyUFmZFuGDCg4AySW7eAAQOA8HCGDCKqnuxkMmToBqEVY39KCnqeOVNsu9+bN0fHR1eKi3ttUw0ZMgRvvfUWli1bBqVSidWrV+PVV1+FTCZDRkYGpk+fjm3btiE+Ph4PHz5EVlYWbty4YfLxjTl//jyCgoL04QIAOnToAI1Gg4sXL6Jjx44YPnw4unfvjm7duqFr164YOHAgvLy8AADjx4/Hm2++iVWrVqFr16545ZVX9EGksrDoGIzZs2ejdevWcHR0RK1atdCvXz9cvHix2Odt3LgRjRs3ho2NDZo3b47ff/+9Aqo1TedHv/x7k1MwZkzh01N128aOBdTqCiuNiKjCSJIEe7ncpMcLbm7F3n7BV6nEC25uJh1P141gij59+kAIgW3btiEuLg5RUVEYMmQIAGDixInYvHkzZs2ahaioKJw6dQrNmzdHrok3vCyrFStW4NChQ2jfvj3Wr1+Phg0b4vDhwwCA6dOnIzo6Gr169cKff/6Jpk2bYvPmzRVSl6ksGjD27duHUaNG4fDhw9i1axfy8vLwwgsvIDMz0+hzDh48iNdeew3//ve/cfLkSfTr1w/9+vXD2bNnK7By43TjMI6mp+PmPePpQQggLg4wUxclEVGVVdTtF3Q/LwgIgLwEwcFUNjY2CAsLw+rVq7F27Vo0atQILVu2BAAcOHAAw4cPx8svv4zmzZujdu3auHbtmllet0mTJjh9+rTB592BAwcgk8nQqFEj/bann34akydPxsGDB9GsWTOsWbNGv69hw4YYN24cdu7cibCwMKxYscIstZmLRQPGjh07MHz4cAQGBiIoKAgrV67EjRs3cPz4caPPWbhwIV588UVMmjQJTZo0waeffoqWLVtiyZIlFVi5cX42NvBVKqGGAJqmFts+Pr4CiiIiquR0t1+oo1QabPdRKhEeGFiu62AMGTIE27Ztw48//qi/egFoxzVERETg1KlTOH36NAYPHlxgxklZXtPGxgbDhg3D2bNnsXfvXrz33nt444034OnpidjYWEyePBmHDh3C9evXsXPnTly+fBlNmjRBVlYWRo8ejcjISFy/fh0HDhzA0aNHDcZoVAaVagxGaqr2A9nNzc1om0OHDmH8+PEG27p37250dbScnBzk5JtXnZaWVvZCiyBJEjq5uOCXu3eB4BTghPH3AgCPutOIiGq8MJUKfT08EJWSgvjcXHgpFAh1cSmXKxf5Pffcc3Bzc8PFixcxePBg/fb58+djxIgRaN++PTw8PPDhhx+a7TPEzs4Of/zxB8aMGYPWrVvDzs4O/fv3x/z58/X7L1y4gJ9++glJSUnw8vLCqFGj8J///AcPHz5EUlIShg4dirt378LDwwNhYWGYMWOGWWozF0mYOmG4nGk0Grz00ktISUnBX0VMRVIoFPjpp5/w2muv6bctW7YMM2bMwN27dwu0nz59eqEnPTU1FU5OTuYp/gn/jY/HmxcvQnHRGXnvPl3oOAxJ0s4miY3llFUiqtqys7MRGxsLf39/2NjYWLocKqOi/jzT0tLg7Oxs0mdopVloa9SoUTh79izWrVtn1uNOnjwZqamp+kdcXJxZj1+YTs7OAABNwzQIhRrGwveCBQwXRERUPVWKgDF69Gj89ttv2Lt3L3x8fIpsW7t27QJXKu7evYvatWsX2l6pVMLJycngUd7q29rCW6HAQ0lg+oY01KnzZE2cokpEVJ2sXr0aDg4OhT4CAwMtXZ5FWHQMhhAC7733HjZv3ozIyEj4+/sX+5yQkBDs2bMHY8eO1W/btWsXQkJCyrHSkpEkCZ1dXLAmIQGa5im4ds0VUVHA6dPaqal5eUCXLpaukoiIzOWll15C27ZtC91X2VbYrCgWDRijRo3CmjVrsHXrVjg6OuLOnTsAAGdnZ9ja2gIAhg4dijp16mD27NkAgDFjxqBTp06YN28eevXqhXXr1uHYsWP47rvvLPY+CtPpUcDYl5ICuT/QubP28e23wPnzwM6dwKBBlq6SiIjMwdHREY6OjpYuo1KxaBfJN998g9TUVHTu3BleXl76x/r16/Vtbty4gfh8cznbt2+PNWvW4LvvvkNQUBDCw8OxZcsWNGvWzBJvwSjdehiH09KQnW81rV69tF8r0dpgRERmUUnmDFAZmevPsdLMIqkoJRkBWxZCCHgdPIi7eXnYFxysX942MlLbPaJSAXfuACVY0ZaIqFJSq9W4dOkSatWqBXd3d0uXQ2WUmpqK27dvIyAgoED3Tkk+QyvVOhjViW4cxvrERESmpOgDRocO2nARHAwkJWm/JyKqyuRyOVxcXJCQkABAu4ZDSZbrpspDo9EgMTERdnZ2sLIqW0RgwChHnR4FjH0pKfpt1tbam53V0DE/RFRN6Wby6UIGVV0ymQx169Ytc0hkwChHunEYh9LSkKvRQPGoP4ThgoiqG0mS4OXlhVq1aiEvL8/S5VAZKBQKyMzQf8+AUY6a2NlBZW2NxLw8HE1PR4dHC3DpxMcDDg4ABx4TUXUhl8sh5wqChEqy0FZ1pbsvCQBE5usmAYBhwwBvb+2CW0RERNUNA0Y50y0bvu+JgOHnp/3K6apERFQdMWCUM90VjAOpqcjLd5tf3XoYO3dqV/YkIiKqThgwylmgvT3c5HI80Ggw6/p1RCYnQy0EnnlGO0U1LQ04cMDSVRIREZkXA0Y523LvHh48Wsts+vXr6HL6NPwOH8aWpET06KFts22bBQskIiIqBwwY5SgiMREDoqORna9rBABu5eRgQHQ0PF5OBMBxGEREVP0wYJQTtRAYExODwtZh121br4qBzFrg3Dng2rUKLI6IiKiccR2MchKVkoKbOTlG9wsAt/JyMHxuCrp6uILL9xMRUXXCgFFO4nNzTWr3wqu5eM2znIshIiKqYOwiKSdeCoVZ2xEREVUlDBjlJNTFBT5KJYzdKkYC4KtUItTFBefPA198ARw6VJEVEhERlR8GjHIilyQsDAgAgEJDhgCwICAAcknCokXA5MnAL79UaIlERETlhgGjHIWpVAgPDEQdpbLAPjuZTL/KZ8+e2m2bNgFr1gCRkYBaXXF1EhERmRsDRjkLU6lwrV077A0KwpomTbA7KAjN7ezwQKPBp4/mpqana9vevQsMGQJ06aK9V0lEhMXKJiIiKhNJCFHYUg3VVlpaGpydnZGamgonJyeL1LD7/n10++cfWEkSvk5ujffD7PDkn4L0qF8lPBwIC6v4GomIiJ5Uks9QXsGwgK5ubujh5oaHQuCDy1cLhAsA+m1jx7K7hIiIqh4GDAuZW78+ZACyWt8DmqUU2kYIIC4OiIqq0NKIiIjKjAHDQgLt7dE500v7w8grgGS8pyo+voKKIiIiMhMGDAsaZe8HPJADTdKBzglG23l5VVxNRERE5sCAYUF9Q5Vw+t1X+8PbV4BWScBzd4GgZEAmIEmAry8QGmrZOomIiEqK9yKxILkcWP6cLwan3gRq5wJfnXm8M0EJsSQAX49RQS63XI1ERESlwSsYFqYMvQ84PSy4wyMHmBGN/SKx4osiIiIqIwYMC1ILgTExMYWvJS4DIIDFiMHtOzVqqRIiIqoGGDAsKColBTdzcow3kAHCIwdDvkypsJqIiIjMgQHDguJzc01qF/lPLrZsKd9aiIiIzImDPC3IS6EwrWGSAu++Czz7LHD2rHZdDC8v7ewSDgAlIqLKiAHDgkJdXOCjVOJWTg4KG2UhAaijUMIuxwWX7gANGwLJyY/3+/gACxfyXiVERFT5sIvEguSShIUBAQAKH+cpACxsEIB/DdPuzR8uAODWLWDAAN51lYiIKh8GDAsLU6kQHhiIOkploftdZNZYurTw5/KGaEREVFnxdu2VhFoIRKWkID43F14KBdbcvYvv79xBPdjherdngIdFZ8G9e4HOnSumViIiqplK8hnKMRiVhFyS0NnVVf9zkIMDtiQl4XreA2BQHLC6XpHP5w3RiIioMmEXSSXlam2N+fXra3944zrglVVke94QjYiIKhMGjEpsiKcnuji7AEoNMOYyUMhcE94QjYiIKiMGjEpMkiR806ghrIQEtL2vvaV7UPLjO67KtYFjwQKuh0FERJULx2BUco3s7PB/fnUx8/p14JPzQL4gId1TYqIyAGF9VJYrkIiIqBC8glEFNLGz037zxFUK4Z6DrxyjEZHIO64SEVHlwoBRyamFwKSrVwvf+Wh1rrExMVDXrNnGRERUyTFgVHLF3XFVAIjLyUFUSkqF1URERFQcBoxKztQ7rprajoiIqCIwYFRypt5x1eQ7sxIREVUABoxKTnfH1cJuhgYAEIBtuhKhLi4VWBUREVHRGDAqueLuuAoAeV8HIDXZaAQhIiKqcAwYVYCxO67KAdRb0RQP96qwYYNlaiMiIioMA0YVEaZS4Vq7dtgbFISVjRrBUSaDGsBzHbV/hKtWWbY+IiKi/BgwqhDdHVeHeXnhnTp1AABXWtyCTAYcPAjExFi4QCIiokcYMKqod729IQHY/yAZ7Qc+AAD88otlayIiItJhwKii/G1t0cvdHQBgP+QWAG03CRf0JCKiysCiAWP//v3o06cPvL29IUkStmzZUmT7yMhISJJU4HHnzp2KKbiSGeXtDQA45HwHL7/2EHPmABqNhYsiIiKChQNGZmYmgoKCsHTp0hI97+LFi4iPj9c/atWqVU4VVm4vuLkhwNYWaWo1un+VgP79edt2IiKqHCx6u/YePXqgR48eJX5erVq14MKFpSCTJIz09sb4K1ew9NYtvO3lBY1GQlQUEB8PeHkBoaEMHUREVPGq5BiM4OBgeHl5oVu3bjhw4ECRbXNycpCWlmbwqE6G164NW5kMZzIzMeGnVLi5AV26AIMHa7/6+QEREZaukoiIapoqFTC8vLywfPlybNq0CZs2bYKvry86d+6MEydOGH3O7Nmz4ezsrH/4+vpWYMXlz9XaGq97egIAvo69hSfz061bwIABDBlERFSxJCEqx7wDSZKwefNm9OvXr0TP69SpE+rWrYtVRlaaysnJQU6+252npaXB19cXqampcHJyKkvJlcaJ1Ay0OnkMeCgBr7YDkgxX/JQkwMcHiI1ldwkREZVeWloanJ2dTfoMrVJXMArTpk0bxBSxwpRSqYSTk5PBo7pJO+kAnHECrATQ+3aB/UIAcXFAVJQFiiMiohqpygeMU6dOwcvLy9JlWFR8PIDN2pU90ec20PI+8NxdICgZkAnDdkRERBXAorNIMjIyDK4+xMbG4tSpU3Bzc0PdunUxefJk3Lp1Cz///DMAYMGCBfD390dgYCCys7Pxww8/4M8//8TOnTst9RYqBS8vAFEqIP0S4J4HzPvn8c4EJbAkAIhSoYbnMCIiqkAWDRjHjh1Dly5d9D+PHz8eADBs2DCsXLkS8fHxuHHjhn5/bm4uJkyYgFu3bsHOzg4tWrTA7t27DY5RE4WGAu69k5DkoC640yMHmBEN98WBCA1VVXxxRERUI1WaQZ4VpSQDVKoKtRDw/PMwkmQ5gFRIAw3gLpS4+1w7yKXCGhARERWvRg3yJCAqJQVJciPhAgBkQJI8B1EpKRVZFhER1WAMGNVAfG6uWdsRERGVFQNGNeClUJi1HRERUVkxYFQDoS4u8FEqjfaQSAB8lUqE8v4tRERUQRgwqgG5JGFhQACAQoZhPLp9+4KAAA7wJCKiCsOAUU2EqVQIDwxEHaXhMuFIVOKd+ECEqThFlYiIKg4DRjUSplLhWrt2GPHo5mf177sCg9vhzkaGCyIiqlgMGNWMXJLQ28MDAGDtkQdoJOzaBeS73xsREVG5Y8CohgLt7QEA1/EAnl4CGRm80RkREVUsBoxqqL6tLZSShCyNBs8OyAYAbNtm4aKIiKhGsei9SKh8yCUJje3scDozE837ZEKVZ4u+fS1dFRER1SQMGNVUoL09TmdmQtEwE99842HpcoiIqIZhF0k1pRuHEZ2ZaeFKiIioJmLAqKb0AePBAzx8CPz1F/Df/1q4KCIiqjHYRVJNBdrZAQAuPHiAi5cFQkMlKBTAoEGAg4OFiyMiomqPVzCqKX9bW9jIZMjWaGBdNwtPPQXk5gJ79li6MiIiqgkYMKopuSShyaOrGOceZKJXL+12TlclIqKKwIBRjeUfh9Gzp3bb778DQliwKCIiqhEYMKox3TiM6MxMdO4M2NkBt24Bp09bti4iIqr+GDCqMd0VjHOZmbCxAZ5/Xrv9998tWBQREdUIDBjVmC5gXHjwAGoh9OMwDhywYFFERFQjcJpqNeZnYwM7mQwPNBpcycpC//52CA4GWre2dGVERFTdMWBUY7JHM0mOZ2QgOjMTL6vs4OoK7N8PxMcDXl5AaCggl1u6UiIiqm4YMKq5QHt7fcAQUSqMGQPcvPl4v48PsHAhEBZmuRqJiKj64RiMaq7po3EYOy4+wIABhuEC0M4qGTAAiIiwQHFERFRtMWBUc7qpqn/fzSx0/QvdtrFjAbW64uoiIqLqjQGjmtPNJMmr/QCQaQptIwQQFwdERVVkZUREVJ0xYFRz9WxsoNTIAIUA6mQV2TY+voKKIiKiao8Bo5qTSRL8ZNqrGPB/UGRbL68KKIiIiGoEBowaoG0t7TgM+GUWul+SAF9f7ZRVIiIic2DAqAGaOTy6guGXCUkqvM2CBVwPg4iIzKdUAeOnn37Ctnz3/f7ggw/g4uKC9u3b4/r162YrjsxDN9DTNzQTdeoY7nN1BcLDuQ4GERGZV6kCxqxZs2BrawsAOHToEJYuXYo5c+bAw8MD48aNM2uBVHa6gHHHOguXr2qwdy+wejWwdy+QmMhwQURE5leqlTzj4uIQEBAAANiyZQv69++Pt99+Gx06dEDnzp3NWR+ZQV2lEg5yOTLUalzNzULnzvaWLomIiKq5Ul3BcHBwQFJSEgBg586d6NatGwDAxsYGWVlFT4WkiidJEpo+WnArOtNwoGdODnDvniWqIiKi6qxUAaNbt25488038eabb+LSpUvo2bMnACA6Ohp+fn7mrI/MRNdNkj9gfPcd4OICfPCBhYoiIqJqq1QBY+nSpQgJCUFiYiI2bdoEd3d3AMDx48fx2muvmbVAMg99wHjweC0MX18gOxv46y9LVUVERNWVJERhd6iovtLS0uDs7IzU1FQ4OTlZupwKsyMpCT3OnEFTOztEt2kDAEhJAdzctEuF37kDeHpatkYiIqrcSvIZWqorGDt27MBf+f7bu3TpUgQHB2Pw4MFITk4uzSGpnOmuYFzKykKuRntPEhcXoFkz7X5exSAiInMqVcCYNGkS0tLSAABnzpzBhAkT0LNnT8TGxmL8+PFmLZDMw0ephJNcjodC4HK+gbjPPqv9yoBBRETmVKqAERsbi6ZNmwIANm3ahN69e2PWrFlYunQptm/fbtYCyTwkSULTQgZ6MmAQEVF5KFXAUCgUePBosODu3bvxwgsvAADc3Nz0Vzao8gksZKqqLmCcPAlkZFiiKiIiqo5KtdDWs88+i/Hjx6NDhw74+++/sX79egDApUuX4OPjY9YCyXwKm6paty4wfDjQtCmgVluoMCIiqnZKdQVjyZIlsLKyQnh4OL755hvUeXSDi+3bt+PFF180a4FkPk0LmaoKACtWAJMmAc7OlqiKiIiqI05TrUFuZmfD9/BhyAFkduwIpYw30yUiItOV5DO0VF0kAKBWq7FlyxacP38eABAYGIiXXnoJct7zu9Kq82gmSZpajUsPHqC5g4N+340b2oGeAwcCVqX+rSAiItIq1UdJTEwMevbsiVu3bqFRo0YAgNmzZ8PX1xfbtm1D/fr1zVokmYckSQi0t8ehtDREZ2bqA4ZGA7RoAaSmAo0bAy1bWrhQIiKq8kp1jfz9999H/fr1ERcXhxMnTuDEiRO4ceMG/P398f7775u7RjKjwpYMl8mADh2033O6KhERmUOpAsa+ffswZ84cuLm56be5u7vjiy++wL59+8xWHJlfE1tbAMCO+/cRmZwM9aMhOAwYRERkTqXqIlEqlUhPTy+wPSMjAwqFosxFUfmISEzE7Lg4AMCx9HR0OX0aPkolFgYE4NlnVQC0AUMIQJIsWSkREVV1pbqC0bt3b7z99ts4cuQIhBAQQuDw4cN455138NJLL5m7RjKDiMREDIiOxr28PIPtt3JyMCA6GrfqJ8LaGoiPB65etVCRRERUbZQqYCxatAj169dHSEgIbGxsYGNjg/bt2yMgIAALFiwwc4lUVmohMCYmBoXNR9Zt+/BGDFq11v7EbhIiIiqrUgUMFxcXbN26FZcuXUJ4eDjCw8Nx6dIlbN68GS4uLiYfZ//+/ejTpw+8vb0hSRK2bNlS7HMiIyPRsmVLKJVKBAQEYOXKlaV5CzVKVEoKbubkGN0vAMTl5KBenxQADBhERFR2Jo/BKO4uqXv37tV/P3/+fJOOmZmZiaCgIIwYMQJhYWHFto+NjUWvXr3wzjvvYPXq1dizZw/efPNNeHl5oXv37ia9Zk0Un5trUrunn89F/w2P709CRERUWiYHjJMnT5rUTirB6MAePXqgR48eJrdfvnw5/P39MW/ePABAkyZN8Ndff+Hrr79mwCiCl4kDb9sGKNC5dTkXQ0RENYLJASP/FQpLOXToELp27WqwrXv37hg7dqzR5+Tk5CAnX/dATbzba6iLC3yUStzKySl0HIYEwEepRGgJureIiIiKUqVuRnHnzh14enoabPP09ERaWhqysrIKfc7s2bPh7Oysf/j6+lZEqZWKXJKwMCAAgDZM5Kf7eUFAAOSShAsXgM8/B378sUJLJCKiaqZKBYzSmDx5MlJTU/WPuEfrQNQ0YSoVwgMDUUepNNjuamWF8MBAhKm062AcPQp88gnwww+WqJKIiKqLKnVbq9q1a+Pu3bsG2+7evQsnJyfYPlqh8klKpRLKJz5Ua6owlQp9PTwQlZKCL+PisOP+fbxWq5Y+XACPV/Q8dgzIygKMnFYiIqIiVakrGCEhIdizZ4/Btl27diEkJMRCFVU9cklCZ1dXvPGoq+l4RobBfn9/oHZtIC8PmD0biIwE1GoLFEpERFWaRQNGRkYGTp06hVOnTgHQTkM9deoUbty4AUDbvTF06FB9+3feeQdXr17FBx98gAsXLmDZsmXYsGEDxo0bZ4nyq7TWjo4AgJPp6cjVaPTbN2/W3lUVAD79FOjSBfDzAyIiLFAkERFVWRYNGMeOHcPTTz+Np59+GoB2rY2nn34aU6dOBQDEx8frwwYA+Pv7Y9u2bdi1axeCgoIwb948/PDDD5yiWgoBtrZwsbJCjhA4m5kJQBsiBgzQdo3kd+uWdjtDBhERmUoSQhQ2c7HaSktLg7OzM1JTU+Hk5GTpcizqhdOnsSs5GcsbNsSbnt7w8wNu3iy8rSQBPj5AbCwgl1domUREVEmU5DO0So3BIPPSdZP8nZaGqCjj4QLQ3mE1Lg6Iiqqg4oiIqEpjwKjB2jxKn0fT0xEfb9pzTG1HREQ1GwNGDaa7ghGdmQkXL9Ominh5lWdFRERUXTBg1GDeSiW8FQpoANi2SIePj3asRWEkCfD1BUJDK7REIiKqohgwajhdN8mJzHQsXKjd9mTI0P28YAEHeBIRkWkYMGo4XTfJ0fR0hIUB4eFAnTqGbVQq7fawMAsUSEREVRIDRg2XfyYJoA0R164Be/cCzzyjbTNpEsMFERGVDANGDffMo4BxNTsbSXl5ALTdIJ07Pw4Vx45ZqDgiIqqyGDBqOFdrazR4dEezY+npBvvatNF+/fvviq6KiIiqOgYMKtBNotOqlfZrbCyQmFjRVRERUVXGgEEGC27l5+ICNGqk/Z7dJEREVBJWli6ALC//FQwhBKR881TbttVOU83OtlR1RERUFTFgEIIdHCAHcDcvDzdzcuBrY6Pft2IFION1LiIiKiF+dBDs5HI0d3AAULCbhOGCiIhKgx8fBMBwwa3CqNVAbm5FVkRERFUZAwYBMD6TBAD+8x/A2RnYurWiqyIioqqKAYMAPJ5Jciw9HRohDPbJ5UBmJtfDICIi0zFgEAAg0M4OtjIZ0tRqXM7KMtjHBbeIiKikGDAIAGAlk+HpRwM9n+wmad1a+/X4ce1YDCIiouIwYJCesQW3GjcGHBy03STnz1uiMiIiqmoYMEjP2EwSufzxnVXZTUJERKZgwCA9XcA4mZ6OXI3GcN+jbhIGDCIiMgUDBukF2NrCxcoKOULgbGamwb7OnYFevR4HDSIioqJwqXDSkyQJrR0dsSs5GUfT09Hy0RUNAOjZU/sgIiIyBa9gkIE2RSy4RUREZCoGDDLQ2shMEp1bt4CrVyuyIiIiqooYMMiAbqBndGYmMp9Y9OKrrwAfH2DqVEtURkREVQkDBhnwVipRR6GABsCJJ65iNGum/Xr0aMXXRUREVQsDBhXwzKOrGN/evo3I5GSoH92bRDeD5NIlICXFQsUREVGVwIBBBiISE7H3UXpYnZCALqdPw+/wYUQkJsLdHXjqKW27Y8csVyMREVV+DBikF5GYiAHR0Uh7YuzFrZwcDIiORkRiIm98RkREJmHAIACAWgiMiYmBKGSfbtvYmBi0aq39ieMwiIioKAwYBACISknBzZwco/sFgLicHCifSQHAKxhERFQ0ruRJAID43FyT2jn45eL994E2bQCNBpAxohIRUSEYMAgA4KVQmNTO31GBfy0s52KIiKjK4/8/CQAQ6uICH6USkpH9EgBfpRKhLi4VWBUREVVVDBgEAJBLEhYGBACA0ZCxICAAcklCdjawfz+wYUPF1UdERFULAwbphalUCA8MRB2l0mC7vUyG8MBAhKlUAIALF4BOnYC339aOwyAiInoSAwYZCFOpcK1dO+wNCsIEHx8AQG2FQh8uACAwELCxAVJTga+/BiIjgSeWziAiohqOAYMKkEsSOru6YqqfH+QArmRn40Z2tn7/r78+vnIxcSLQpQvg5wdERFikXCIiqoQYMMgoJysr/e3b9yQnA9CGiAEDgCdntd66pd3OkEFERAADBhXj+UezRvYkJ0OtBsaMAUQhy33qto0dy+4SIiJiwKBiPO/qCgDYk5KC/VECN28abysEEBcHREVVUHFERFRpMWBQkUKcnGAjk+FObi6OJT4w6Tnx8eVcFBERVXoMGFQkG7kczzo7AwDiVMkmPcfLqzwrIiKiqoABg4qlG4dx3T0ZPj6AZGQlLkkCfH2B0NCKq42IiConBgwqlm4cxr6UFMxfqJ2f+mTI0P28YAEgl1dgcUREVCkxYFCxWjo6wsXKCqlqNep2zUB4OFCnjmEbJycgPBwIC7NMjUREVLkwYFCx5JKELvmmq4aFAdeuAXv3AiNHatt4ezNcEBHRYwwYZBL9dNVHC27J5UDnzsDnnwNWVsD588ClSxYskIiIKhUGDDKJbqDngdRUZOVbScvFRbtUOABs3VrxdRERUeVUKQLG0qVL4efnBxsbG7Rt2xZ///230bYrV66EJEkGDxsbmwqstmZqZGcHb4UCOULgYFqawb5+/bRft2yp8LKIiKiSsnjAWL9+PcaPH49p06bhxIkTCAoKQvfu3ZGQkGD0OU5OToiPj9c/rl+/XoEV10ySJBXoJtF56SXt10OHgDt3KroyIiKqjCweMObPn4+33noL//rXv9C0aVMsX74cdnZ2+PHHH40+R5Ik1K5dW//w9PSswIprLmMBw8cHaN1au1T4//5nicqIiKiysWjAyM3NxfHjx9G1a1f9NplMhq5du+LQoUNGn5eRkYF69erB19cXffv2RXR0tNG2OTk5SEtLM3hQ6ejGYRxLT0dKXp7BvpdfBpo2BWxtLVAYERFVOhYNGPfu3YNarS5wBcLT0xN3jFxrb9SoEX788Uds3boVv/zyCzQaDdq3b4+bRu7CNXv2bDg7O+sfvr6+Zn8fNYWPjQ0a2tpCA2BfaqrBvg8+AKKjgTfesExtRERUuVi8i6SkQkJCMHToUAQHB6NTp06IiIiASqXCt99+W2j7yZMnIzU1Vf+Ii4ur4IqrF2PdJFy9k4iI8rNowPDw8IBcLsfdu3cNtt+9exe1a9c26RjW1tZ4+umnERMTU+h+pVIJJycngweVni5g7E4u/MZnWVnAiRMVWREREVVGFg0YCoUCrVq1wp49e/TbNBoN9uzZg5CQEJOOoVarcebMGXjxFp4VoouLCyQA5x88wO2cHIN9ly4BHh7adTFycy1THxERVQ4W7yIZP348vv/+e/z00084f/483n33XWRmZuJf//oXAGDo0KGYPHmyvv3MmTOxc+dOXL16FSdOnMDrr7+O69ev480337TUW6hR3Kyt0dLBAQDw5xNXMQICtPckSUvTLiNOREQ1l8UDxqBBg/DVV19h6tSpCA4OxqlTp7Bjxw79wM8bN24gPj5e3z45ORlvvfUWmjRpgp49eyItLQ0HDx5E06ZNLfUWahz9OIyUFIPtMhnQt6/2ey66RURUs0lCCGHpIipSWloanJ2dkZqayvEYpbTz/n10/+cfqKytsSAgAN4KBUJdXCCXJOzYAfToAXh5ATdvakMHERFVDyX5DLWqoJqoGkl6tAZGYl4ehpw/DwDwUSqxMCAAvbqo4OgIxMcDh/8WyG2UgvjcXHjlCyFERFT9MWBQiUQkJupDRX63cnIwIDoa4YGB6NVLhXW3EvFicgzSTz8eCKoLIWEqVUWWTEREFsAL2GQytRAYExODwvrUdNvGxsSg9qAEYEY00m0MZ5noQkhEYmK510pERJbFgEEmi0pJwc0npqbmJwDE5eTgv24XtRukgvsBbQhR16yhP0RENQ4DBpks3sTFLdI16gLhQkcXQqKemIFCRETVCwMGmcxLoTDbsUwNK0REVDUxYJDJQl1c4KNUGrs4AQmAytrapGOZM6wQEVHlw4BBJpNLEhYGBAAo2AOi+3lx/QaQJykBjZGDaAB5khLtHV3KqUoiIqoMGDCoRMJUKoQHBqKOUmmw3UepRHhgIDwv1IJ6YYA2cTwZMjQAJEC9MAAH/+J6GERE1RnXwaASC1Op0NfDA1EpBRfRWhsPIEoFTAsERscAtfLNOkmzBuY3BKJUyLf6OxERVUMMGFQqcklC50f3JMlPf1PbKBVwwANongIMvgG0SQYOuGm3529HRETVErtIyKxCQwEfH0CSAGgk4LQrsNFXuzPkPiAT8PXVtiMiouqLAYPMSi4HFi7Ufq+/7chpFyBTDrjlAY3SsWCBth0REVVfDBhkdmFhQHg4UKfOow15MuCoGwCg/1dJCAuzXG1ERFQxGDCoXISFAdeuAXv3AmvWAB91dgcAxKjuWbYwIiKqEBzkSeVGLgc6d9Z+3y3XDXMOAqczMxGbmQ1/exuL1kZEROWLVzCoQngoFGjn6AQAaD0mCffvW7ggIiIqVwwYVGH6uHsAAJIa3sOsWRYuhoiIyhUDBlWYvirtOAwEp2DRDw9x/bpl6yEiovLDgEEVprGdHerb2AAKgbwWyZgyxdIVERFReWHAoAojSRL6eGi7SRCShF9+AY4fByIjgbVrtV/VaktWSERE5sKAQRWqj7u2m0TZOQlCEujQAejSBRg8WPvVzw+IiLBsjUREVHYMGFShQp2d4SyXI8c2D2ichpwcw/23bgEDBjBkEBFVdQwYVKGsZTK86PZosGf7pAL7hdB+HTuW3SVERFUZAwZVuIBEXcAofFVPIYC4OCAqqgKLIiIis2LAoApX744boAbg/wConWW0XXx8xdVERETmxYBBFa5BbWvgjLP2h0K6SXS8vCqoICIiMjsGDKpwoaGAc/Tj6apPkiTA11fbjoiIqiYGDKpwcjnwWa9H4zCCUgC7hwXaLFigbUdERFUTAwZZxOh+dvBW2wLWAnglDnjuLhCUDMgEGjUC+va1dIVERFQWDBhkMc/UstN+M/w6MOU8sOA0pHWHcUGViLlzLVsbERGVDQMGWUREYiL+l1TIOhgeOcCMaHzyRyIuX7ZAYUREZBYMGFTh1EJgTExM4Tsl7Ren/4uBf31RcUUREZFZWVm6AKp5olJScPPJNcLzk4Bk6xz8lZqCUCdXREVp18Tw8tLOLOHgTyKiyo8BgypcfG6uSe22HsjFkP8At28/3ubjAyxcCISFlVNxRERkFuwioQrnpVCY1G7BFIVBuAB4MzQioqqCAYMqXKiLC3yUSt1wiwIkAPIkJfCPS4F9vBkaEVHVwIBBFU4uSVgYEAAAhYYMAUD9Qz1AU3gE4c3QiIgqPwYMsogwlQrhgYGoo1QabNeP33z5NmBbcIXP/HgzNCKiyouDPMliwlQq9PXwQFRKCuJzc+GlUMBLoUC7o6eQ0jADmB4NfNIMaJoGuOcCSQrgjIv+yobuZmhqNTjThIiokpGEEDVqsYG0tDQ4OzsjNTUVTk5Oli6HCnEoJQ0djpyCUGqALBlgq3m8M0EJLAmAT6wK164BW7cCY8YAN28+bsKZJkRE5aMkn6EMGFQp9d9+BRE2cQUHaWgASMCkjEC0y1NhwABASAJonvL4KsdZF0gaCeHhDBlEROZUks9QdpFQpaMWAn+7JADZheyUARDAWo8YrBnkAfHsPWB0DFAr38JdCUqIpQEYO1aFvn213SXsRiEiqlgMGFTp6Ff6LGIe682cHKDzNeBf1wvu98gBpkcjblogoqJUuH+f3ShERBWNAYMqHVNX+sTQG9qvTwYRGbRdKaNiEB7hgWVLJG03SlCKvhvl5lkXDBhg2I2S+1Bg2V8puJKci/quCox81gUKK2Mph4iIisKAQZWOqSt9wqqI4UMyAJ45+OWfFIhnHxbbjTL590TMz42B2j0HcNU2mbhVifGKAMzpo9I/jV0tRDXYqVPA5MnA7NlAcLClq6n0GDCo0tGt9HkrJweFRQgJgKNcjjQTlvJMbZUA9C5kwYx83Si9vgT+CIku0ETtmoO5UjTwayDm9FEhIsK0rpbiQghDClEVtWkTsGMH0Lo1A4YJOIuEKqWIxEQMiNZ+6Of/BdV1WEz388O0a9eKP5DuyYX1dGgA3FNo93nkGm0jT1ZitWiH1wZKxc5YiYgA3h8rcMvtcZs6912waIFk0n4dU7pr2KVDVMGCg4HTp7VfT560dDUWwWmqRWDAqDoiEhMxJibG4NbuvkolFgQEoK+HB/wOH8bNbCODQQUglwBz3a5ENiEIGvvCu1qwNAC+sSrMnw+8sjgRGFV4m0ltVZh7xPj+TWNUCAsDPvg1X3fNI/Ikw+4aU9oA5gkq5go7prQx5eqOOa4AqYUwWOAt1MUFconhjIpw9y5Qu7bhz7VqlctLVebfTwaMIjBgVC1F/UUr7irH+94+WHj7JszioBsQct/wBQD9uhyYFggra+DhJ9FG20gbfCEGxhnd7744EP8aDnzlaPwYkzICAQBzHYpuM6ePyixBxVxhx5Q2plzdMaVNcUEmIjERYy7H4Gbu41p8FEosbBCAMJV5w5kpbSoywFWmek0JiuY4jrlCq2blz5D9a5jhz8PeMDyOCcGguNcy9ffTUhgwisCAUb0UdZXDzcoKXU6fNt+LCRjvakl81NWiMt7VAgHt4FNjd3hLsYIkSRDOeUaPIUvWDoDVuBXdpTPGuj7mO57TbitlUGl91RdHnzIeiEwNO6a0aZenQv+FRV/dAVBsm8PWRQeZiMRE9D9rvJZNzQIRpqq4YFWRAa4y1WtKUDTHccwVWiMiAKvXB6FH1iZYQ408yPG77QCof1n3uI0JwaDYek38/bRkF2qVCxhLly7F3LlzcefOHQQFBWHx4sVo06aN0fYbN27ElClTcO3aNTRo0ABffvklevbsadJrMWBUP8b+16AWothulDpKBeLjAY1rbuG3/tMAyJUBNppCdlZiD+SArdp42Em2hgQJooigUmQgMjHsmBqInFfXx/33jAci90XaoJL0vvF/fG22+iK7r/FANCGjKVbaXUGSzMjvgwZwF0oMf1Af8xzKFs4qW4CrTPWaEiYPWyeW+TimdEkCj0Ort3QTnsnJ2jb3FcAGX3w1WDulbNKa+/jzQl84Zz3QHybV1h7PNd6CuYPdgKeTMTHmikG9d11dcdtdpQ8GiCq63g1jPPCu82GTfj8X5F4xSxdqaVSpgLF+/XoMHToUy5cvR9u2bbFgwQJs3LgRFy9eRK1C+rcOHjyIjh07Yvbs2ejduzfWrFmDL7/8EidOnECzZs2KfT0GjJqluG6U8MBAHD786B9F3QeqzqN/zHpm+eB3OzN1tVDhcmSAQmP86s59KwAS4Gb86k6xV4geArA2oZasR4HSyOtI9xWQpKJDk5T8qI2r5QOcudqYo15TwqTb4qZIHXIFajfjH7SmHKfYLsknQuvuCePxfBEDNzWSBFm+j8snf37S7pYt0W3ePH0wEEuLrtd2vyeyOt01ejy9wgaul6ILtbSqVMBo27YtWrdujSVLlgAANBoNfH198d577+Gjjz4q0H7QoEHIzMzEb7/9pt/Wrl07BAcHY/ny5cW+HgNGzVNUN4ru0mVRib/ns2buaiGqybJlgLKIMJlj4hXD4o6j69I0tj9drt3p+BCQgAGRkfh23jy4ZmQYXUTYFAJAsoMD/jNhAsI7d368I0/Srt1T1l4KY121AkDGo/fk8LDIcPagb7tSd5dUmXuR5Obm4vjx45g8ebJ+m0wmQ9euXXHo0KFCn3Po0CGMHz/eYFv37t2xZcuWQtvn5OQgJ98HS1paWtkLpyqlsNvCPzn4ak4fFT576GHYZ9lX22epFgI+SmWRXS0+SgWysoAkyXhXi4RH//kwst9NKJCSUnR3jf5/iUW0kdKtteM4iCqjosKDVMz+khynqM9PCYCT4Ryz8M6dsS8oCN98/TX6R0UVe4XiSbr2EaGheHfcOCS6uho2sDbT/+WNvS8JgGMx8+ZkgNo9B8v+SsHYzq5FtzWDwv6JqjD37t2DWq2Gp6enwXZPT0/cuXOn0OfcuXOnRO1nz54NZ2dn/cPX19c8xVOVIpckdHZ1xWuenujs6lrolC+FlYSxnV2x+GVPjO3sqk/4cknCwoAASFLBv9sSAEkCFjZogO9aNNBuePLfvUeXJifW8y1y//ctGmCCouhjTFAU32YsGkCepCy4P187WZICsiRFkW2gLuQ1SngMk9qkmtJvYSbJ1kW/5/sVWAtVOomurhgwcyYGTp2KFHt7PJSZ9hH5UCZDir09Bk6digEzZxYMF6ZKtSr699NMriSbeDuGMrJowKgIkydPRmpqqv4RFxdn6ZKoCgpTqRAeGIg6SqXBdh+lEuGB2pHdYSoVNjULhI/NE21sldjULBBz6tcvcn+YSoU5fVSYlBEIebJhG3myUt93Wlyb+X1rYbwioMxBpfW1ogORKccwpc04WQO4q4sORG5qBdzVZQtE8iQl3FYVXYvbL+YJZ5UqwFWmek0Mk7K0ooNgeYfSjV26oPHPP2PnM88UuppwfgLAzqA2aLzyZ2zs0qVggxIEV8dddYr8/TSX+q4m3o6hjCzaReLh4QG5XI67dw0Htty9exe18y9okk/t2rVL1F6pVEL5xIcCUWmY0tVSXJuydteY2mZOHxXwa2DBcSXJT4wkL6rNv1X44Fensh3DxDbtEyXt9DwNCh1o+33zBgBQZJu+Ob7YahNndP94RQDava5C/+lSwZH897Qj+b8fo8Jha0m7RLyR40xQaGspa5vWsY9mZZTz61SmesfJGmCl+gqSpByjXXzuQonhUn3Mk86V6TimdElKMN6tmejsinMNmuCFY8dhpTHe9aCWyRH9oD0S3VyN1uu2ugGkkcW/72+e88PA6Q5Gfj/rQ/beFWjcjB/DlC5UebISI/u6GH0/5lQpBnm2adMGixcvBqAd5Fm3bl2MHj3a6CDPBw8e4Ndff9Vva9++PVq0aMFBnkRPqEwLK5VqASylEgvzDcYtrk1p107wSXbBwq9LtgZDZVpXoirVq1/rwcisLVPXIinuOJPq+mLujbgiXwdAkcdIfm80nKOji7x4IACk+gXDte7OoqfEhpr2vov6/dRP3zX2nvPPIimiTY2ZRbJ+/XoMGzYM3377Ldq0aYMFCxZgw4YNuHDhAjw9PTF06FDUqVMHs2fPBqCdptqpUyd88cUX6NWrF9atW4dZs2ZxmipRNWDSSojFtKnIVSSrUoCrTPWaEibNcZyyhNbljo7o1by5wfvSDeQsbADotv/ewX+m1yoyuJr6vov6/TTnrQRKo0oFDABYsmSJfqGt4OBgLFq0CG3btgUAdO7cGX5+fli5cqW+/caNG/HJJ5/oF9qaM2cOF9oiIqpCzHW/jeKOU+rQ+vPPwPDh+jZCLsdDBwdc+Pe/0fi//4VVRgak/Hd0/uknqIcMLX5ZcjO8b67kWUkxYBARUbEGDQLCwwEhtI+XXwaWL9fe4CwhAXjnHWDzZu00MkkCXnkFWLfO0lWXu5J8hlb7WSREREQl8vAhsGMHoNEAzs7A+vXawTu61aVr1dL+vH69dr9GA2zfru3bID0GDCIiovyysoCnntJetbh4ERg4sPB2Awdq97/8MlC/PvDgQeHtaiiLTlMlIiKqdBwdgWPHCg6gKIzuaoZabVr7GoRXMIiIiJ5U0rDAcFEAAwYRERGZHQMGERERmV2NG4Ohm5XLu6oSERGVjO6z05QVLmpcwEhPTwcA3lWViIiolNLT0+Hs7Fxkmxq30JZGo8Ht27fh6OgIycTV09LS0uDr64u4uDguzlUOeH7LF89v+eL5LV88v+WrpOdXCIH09HR4e3tDVszt7GvcFQyZTAYfH59SPdfJyYm/4OWI57d88fyWL57f8sXzW75Kcn6Lu3Khw0GeREREZHYMGERERGR2DBgmUCqVmDZtGpRKpaVLqZZ4fssXz2/54vktXzy/5as8z2+NG+RJRERE5Y9XMIiIiMjsGDCIiIjI7BgwiIiIyOwYMIiIiMjsGDCKsXTpUvj5+cHGxgZt27bF33//bemSqqT9+/ejT58+8Pb2hiRJ2LJli8F+IQSmTp0KLy8v2NraomvXrrh8+bJliq2CZs+ejdatW8PR0RG1atVCv379cPHiRYM22dnZGDVqFNzd3eHg4ID+/fvj7t27Fqq4avnmm2/QokUL/WJEISEh2L59u34/z615ffHFF5AkCWPHjtVv4zkuvenTp0OSJINH48aN9fvL69wyYBRh/fr1GD9+PKZNm4YTJ04gKCgI3bt3R0JCgqVLq3IyMzMRFBSEpUuXFrp/zpw5WLRoEZYvX44jR47A3t4e3bt3R3Z2dgVXWjXt27cPo0aNwuHDh7Fr1y7k5eXhhRdeQGZmpr7NuHHj8Ouvv2Ljxo3Yt28fbt++jbCwMAtWXXX4+Pjgiy++wPHjx3Hs2DE899xz6Nu3L6KjowHw3JrT0aNH8e2336JFixYG23mOyyYwMBDx8fH6x19//aXfV27nVpBRbdq0EaNGjdL/rFarhbe3t5g9e7YFq6r6AIjNmzfrf9ZoNKJ27dpi7ty5+m0pKSlCqVSKtWvXWqDCqi8hIUEAEPv27RNCaM+ntbW12Lhxo77N+fPnBQBx6NAhS5VZpbm6uooffviB59aM0tPTRYMGDcSuXbtEp06dxJgxY4QQ/P0tq2nTpomgoKBC95XnueUVDCNyc3Nx/PhxdO3aVb9NJpOha9euOHTokAUrq35iY2Nx584dg3Pt7OyMtm3b8lyXUmpqKgDAzc0NAHD8+HHk5eUZnOPGjRujbt26PMclpFarsW7dOmRmZiIkJITn1oxGjRqFXr16GZxLgL+/5nD58mV4e3vjqaeewpAhQ3Djxg0A5Xtua9zNzkx17949qNVqeHp6Gmz39PTEhQsXLFRV9XTnzh0AKPRc6/aR6TQaDcaOHYsOHTqgWbNmALTnWKFQwMXFxaAtz7Hpzpw5g5CQEGRnZ8PBwQGbN29G06ZNcerUKZ5bM1i3bh1OnDiBo0ePFtjH39+yadu2LVauXIlGjRohPj4eM2bMQGhoKM6ePVuu55YBg6iaGTVqFM6ePWvQx0pl16hRI5w6dQqpqakIDw/HsGHDsG/fPkuXVS3ExcVhzJgx2LVrF2xsbCxdTrXTo0cP/fctWrRA27ZtUa9ePWzYsAG2trbl9rrsIjHCw8MDcrm8wEjau3fvonbt2haqqnrSnU+e67IbPXo0fvvtN+zduxc+Pj767bVr10Zubi5SUlIM2vMcm06hUCAgIACtWrXC7NmzERQUhIULF/LcmsHx48eRkJCAli1bwsrKClZWVti3bx8WLVoEKysreHp68hybkYuLCxo2bIiYmJhy/f1lwDBCoVCgVatW2LNnj36bRqPBnj17EBISYsHKqh9/f3/Url3b4FynpaXhyJEjPNcmEkJg9OjR2Lx5M/7880/4+/sb7G/VqhWsra0NzvHFixdx48YNnuNS0mg0yMnJ4bk1g+effx5nzpzBqVOn9I9nnnkGQ4YM0X/Pc2w+GRkZuHLlCry8vMr397dMQ0SruXXr1gmlUilWrlwpzp07J95++23h4uIi7ty5Y+nSqpz09HRx8uRJcfLkSQFAzJ8/X5w8eVJcv35dCCHEF198IVxcXMTWrVvFP//8I/r27Sv8/f1FVlaWhSuvGt59913h7OwsIiMjRXx8vP7x4MEDfZt33nlH1K1bV/z555/i2LFjIiQkRISEhFiw6qrjo48+Evv27ROxsbHin3/+ER999JGQJEns3LlTCMFzWx7yzyIRgue4LCZMmCAiIyNFbGysOHDggOjatavw8PAQCQkJQojyO7cMGMVYvHixqFu3rlAoFKJNmzbi8OHDli6pStq7d68AUOAxbNgwIYR2quqUKVOEp6enUCqV4vnnnxcXL160bNFVSGHnFoBYsWKFvk1WVpYYOXKkcHV1FXZ2duLll18W8fHxliu6ChkxYoSoV6+eUCgUQqVSieeff14fLoTguS0PTwYMnuPSGzRokPDy8hIKhULUqVNHDBo0SMTExOj3l9e55e3aiYiIyOw4BoOIiIjMjgGDiIiIzI4Bg4iIiMyOAYOIiIjMjgGDiIiIzI4Bg4iIiMyOAYOIiIjMjgGDiIiIzI4Bg4iqvMjISEiSVOCGTURkOQwYREREZHYMGERERGR2DBhEVGYajQazZ8+Gv78/bG1tERQUhPDwcACPuy+2bduGFi1awMbGBu3atcPZs2cNjrFp0yYEBgZCqVTCz88P8+bNM9ifk5ODDz/8EL6+vlAqlQgICMB///tfgzbHjx/HM888Azs7O7Rv3x4XL14s3zdOREYxYBBRmc2ePRs///wzli9fjujoaIwbNw6vv/469u3bp28zadIkzJs3D0ePHoVKpUKfPn2Ql5cHQBsMBg4ciFdffRVnzpzB9OnTMWXKFKxcuVL//KFDh2Lt2rVYtGgRzp8/j2+//RYODg4GdXz88ceYN28ejh07BisrK4wYMaJC3j8RFaLM92MlohotOztb2NnZiYMHDxps//e//y1ee+01sXfvXgFArFu3Tr8vKSlJ2NraivXr1wshhBg8eLDo1q2bwfMnTZokmjZtKoQQ4uLFiwKA2LVrV6E16F5j9+7d+m3btm0TAERWVpZZ3icRlQyvYBBRmcTExODBgwfo1q0bHBwc9I+ff/4ZV65c0bcLCQnRf+/m5oZGjRrh/PnzAIDz58+jQ4cOBsft0KEDLl++DLVajVOnTkEul6NTp05F1tKiRQv9915eXgCAhISEMr9HIio5K0sXQERVW0ZGBgBg27ZtqFOnjsE+pVJpEDJKy9bW1qR21tbW+u8lSQKgHR9CRBWPVzCIqEyaNm0KpVKJGzduICAgwODh6+urb3f48GH998nJybh06RKaNGkCAGjSpAkOHDhgcNwDBw6gYcOGkMvlaN68OTQajcGYDiKq3HgFg4jKxNHRERMnTsS4ceOg0Wjw7LPPIjU1FQcOHICTkxPq1asHAJg5cybc3d3h6emJjz/+GB4eHujXrx8AYMKECWjdujU+/fRTDBo0CIcOHcKSJUuwbNkyAICfnx+GDRuGESNGYNGiRQgKCsL169eRkJCAgQMHWuqtE1FRLD0IhIiqPo1GIxYsWCAaNWokrK2thUqlEt27dxf79u3TD8D89ddfRWBgoFAoFKJNmzbi9OnTBscIDw8XTZs2FdbW1qJu3bpi7ty5BvuzsrLEuHHjhJeXl1AoFCIgIED8+OOPQojHgzyTk5P17U+ePCkAiNjY2PJ++0RUCEkIISyccYioGouMjESXLl2QnJwMFxcXS5dDRBWEYzCIiIjI7BgwiIiIyOzYRUJERERmxysYREREZHYMGERERGR2DBhERERkdgwYREREZHYMGERERGR2DBhERERkdgwYREREZHYMGERERGR2/w+INXhG3rHHXgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='49' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      49.00% [49/100] [37:13<38:44]\n",
       "      <br>\n",
       "      ████████████████████100.00% [50/50] [val_loss=0.0027]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_loss without improvement in 5 epoch,early stopping >>>>>> \n",
      "\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>lr</th>\n",
       "      <th>val_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2.252255</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>2.135870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2.020835</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>1.849496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1.777852</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>1.722345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1.669158</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>1.607782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1.475423</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>1.439040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>1.233220</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>1.017080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0.931248</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.677400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0.593412</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.407361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0.367539</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.401502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.237548</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.166748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.135753</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.086914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0.091762</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.070428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0.065793</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.046228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0.051523</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.049363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0.042936</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.024963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>0.033302</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.022321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>0.028693</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.027202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>0.024007</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.023569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>0.023362</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.021436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>0.019746</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.016820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>0.018849</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.015970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>0.016486</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.016804</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>23</td>\n",
       "      <td>0.014933</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.012785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>24</td>\n",
       "      <td>0.013871</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.014376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>25</td>\n",
       "      <td>0.013475</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.010799</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>26</td>\n",
       "      <td>0.012289</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.007555</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>27</td>\n",
       "      <td>0.010182</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.010159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28</td>\n",
       "      <td>0.010509</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.016811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>0.009812</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.007240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>30</td>\n",
       "      <td>0.008893</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.006415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>31</td>\n",
       "      <td>0.009118</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.009023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>32</td>\n",
       "      <td>0.007964</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.006193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>33</td>\n",
       "      <td>0.007626</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.005321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>34</td>\n",
       "      <td>0.006851</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.006624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>35</td>\n",
       "      <td>0.006847</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.007525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>36</td>\n",
       "      <td>0.006296</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.007166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>37</td>\n",
       "      <td>0.005810</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.008895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>38</td>\n",
       "      <td>0.005202</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.005235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>39</td>\n",
       "      <td>0.005737</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.002821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>40</td>\n",
       "      <td>0.005426</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.005339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>41</td>\n",
       "      <td>0.004821</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.002998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>42</td>\n",
       "      <td>0.004493</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.002657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>43</td>\n",
       "      <td>0.004830</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.002356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>44</td>\n",
       "      <td>0.004426</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.002122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>45</td>\n",
       "      <td>0.003859</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.010078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>46</td>\n",
       "      <td>0.005893</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.002408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>47</td>\n",
       "      <td>0.003934</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.015338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>48</td>\n",
       "      <td>0.003752</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.002538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>49</td>\n",
       "      <td>0.003786</td>\n",
       "      <td>0.00003</td>\n",
       "      <td>0.002712</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    epoch  train_loss       lr  val_loss\n",
       "0       1    2.252255  0.00003  2.135870\n",
       "1       2    2.020835  0.00003  1.849496\n",
       "2       3    1.777852  0.00003  1.722345\n",
       "3       4    1.669158  0.00003  1.607782\n",
       "4       5    1.475423  0.00003  1.439040\n",
       "5       6    1.233220  0.00003  1.017080\n",
       "6       7    0.931248  0.00003  0.677400\n",
       "7       8    0.593412  0.00003  0.407361\n",
       "8       9    0.367539  0.00003  0.401502\n",
       "9      10    0.237548  0.00003  0.166748\n",
       "10     11    0.135753  0.00003  0.086914\n",
       "11     12    0.091762  0.00003  0.070428\n",
       "12     13    0.065793  0.00003  0.046228\n",
       "13     14    0.051523  0.00003  0.049363\n",
       "14     15    0.042936  0.00003  0.024963\n",
       "15     16    0.033302  0.00003  0.022321\n",
       "16     17    0.028693  0.00003  0.027202\n",
       "17     18    0.024007  0.00003  0.023569\n",
       "18     19    0.023362  0.00003  0.021436\n",
       "19     20    0.019746  0.00003  0.016820\n",
       "20     21    0.018849  0.00003  0.015970\n",
       "21     22    0.016486  0.00003  0.016804\n",
       "22     23    0.014933  0.00003  0.012785\n",
       "23     24    0.013871  0.00003  0.014376\n",
       "24     25    0.013475  0.00003  0.010799\n",
       "25     26    0.012289  0.00003  0.007555\n",
       "26     27    0.010182  0.00003  0.010159\n",
       "27     28    0.010509  0.00003  0.016811\n",
       "28     29    0.009812  0.00003  0.007240\n",
       "29     30    0.008893  0.00003  0.006415\n",
       "30     31    0.009118  0.00003  0.009023\n",
       "31     32    0.007964  0.00003  0.006193\n",
       "32     33    0.007626  0.00003  0.005321\n",
       "33     34    0.006851  0.00003  0.006624\n",
       "34     35    0.006847  0.00003  0.007525\n",
       "35     36    0.006296  0.00003  0.007166\n",
       "36     37    0.005810  0.00003  0.008895\n",
       "37     38    0.005202  0.00003  0.005235\n",
       "38     39    0.005737  0.00003  0.002821\n",
       "39     40    0.005426  0.00003  0.005339\n",
       "40     41    0.004821  0.00003  0.002998\n",
       "41     42    0.004493  0.00003  0.002657\n",
       "42     43    0.004830  0.00003  0.002356\n",
       "43     44    0.004426  0.00003  0.002122\n",
       "44     45    0.003859  0.00003  0.010078\n",
       "45     46    0.005893  0.00003  0.002408\n",
       "46     47    0.003934  0.00003  0.015338\n",
       "47     48    0.003752  0.00003  0.002538\n",
       "48     49    0.003786  0.00003  0.002712"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_model = KerasModel(model,loss_fn = None,\n",
    "        optimizer=torch.optim.AdamW(model.parameters(),lr=3e-5))\n",
    "\n",
    "\n",
    "#加载 之前训练过的权重\n",
    "ckpt_path = 'llama_twosum'\n",
    "\n",
    "keras_model.fit(train_data = dl_train,\n",
    "                val_data = dl_val,\n",
    "                epochs=100,patience=5,\n",
    "                monitor='val_loss',mode='min',\n",
    "                ckpt_path = ckpt_path,\n",
    "                mixed_precision='fp16'\n",
    "               )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe23d285-e5bc-4f94-8232-4d6ee46e3949",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6b2e1d3-212f-4279-b264-a76c3edca25a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "108bcdeb-da45-43c3-a4bc-2c73fcd2ffc3",
   "metadata": {},
   "source": [
    "## 四，使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "id": "98ecd151-3c53-45a7-9981-3bf5fdbda902",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.generation.utils import GenerationConfig\n",
    "model.generation_config = GenerationConfig.from_dict({'num_beams':1,\n",
    "                            'max_new_tokens':100,\n",
    "                            'max_length':200})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "id": "61ea99e7-0772-44e8-9203-ed330090d860",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.generation_config.num_beams=1\n",
    "model.generation_config.max_new_tokens = 100 \n",
    "model.generation_config.max_length=200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "id": "20070502-b8e5-463c-9b0e-21918e3bb25f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ans(tensor) ->\"str\":\n",
    "    s = \"\".join([vocab_r[i] for i in tensor.tolist()])\n",
    "    ans = s[s.find('=')+1:s.find('<EOS>')].replace('<BOS>','').replace('<EOS>','')\n",
    "    return ans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "id": "5735af43-63f1-40ad-bd27-73486d2c9e89",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: 3481340050+90157504501803=\n",
      "y: 90160985841853\n"
     ]
    }
   ],
   "source": [
    "x,y = get_data() \n",
    "print('x: '+''.join(x).replace('<BOS>',''))\n",
    "print('y: '+''.join(y).replace('<EOS>',''))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "id": "1f6a34be-7b79-482a-8903-a2215d3305cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_ids = torch.tensor([[vocab[i] for i in x]]) \n",
    "out = model.generate(inputs=input_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "id": "b8ab05c4-d641-4fe7-8add-f5ad18207cb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1,  5,  6, 10,  3,  5,  6, 12, 12,  7, 12, 13, 11, 12,  3,  7,  9,  7,\n",
       "         12,  6,  7, 12,  3, 10, 12,  5, 14, 11, 12,  3,  8, 12, 11, 10,  7, 10,\n",
       "          6,  3, 10,  7,  5,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,\n",
       "          2,  2,  2,  2,  2,  2,  2,  2,  2,  2, 12,  2,  2,  2,  2,  2,  2,  2,\n",
       "          2, 12,  3, 12,  3]])"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "id": "464a91eb-474c-40a1-b218-c116b709591d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'90160985841853'"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_ans(out[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37164ce5-e107-4a6c-a2ea-88da94249ece",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04cfe22d-8841-45f8-bc4d-e493a29df389",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "eea284a4-217a-44f6-8137-9289caa06a34",
   "metadata": {},
   "source": [
    "## 五，评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "id": "a501b4ed-58a4-4ece-8cc0-1900f3bbaff3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "id": "d02aa121-f51c-4076-9475-69aa5a780842",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 200/200 [04:25<00:00,  1.33s/it, acc=0.99] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc= 0.99\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "loop = tqdm(range(1,201))\n",
    "correct = 0\n",
    "for i in loop:\n",
    "    x,y = get_data() \n",
    "    input_ids = torch.tensor([[vocab[i] for i in x]]) \n",
    "    out = model.generate(inputs=input_ids)\n",
    "    pred = get_ans(out[0])\n",
    "    gt = ''.join(y).replace('<EOS>','')\n",
    "    if pred==gt:\n",
    "        correct+=1\n",
    "    loop.set_postfix(acc = correct/i)\n",
    "    \n",
    "print(\"acc=\",correct/len(loop))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7ece1c5-5d2d-488a-955b-540d599a03d3",
   "metadata": {},
   "source": [
    "漂亮，我们的测试准确率达到了99%！\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81ea98b7-754a-43f3-8c8d-d40f75dc2527",
   "metadata": {},
   "source": [
    "**如果本项目对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果在torchkeras的使用中遇到问题，可以在项目中提交issue。\n",
    "\n",
    "如果想要获得更快的反馈或者与其他torchkeras用户小伙伴进行交流，\n",
    "\n",
    "可以在公众号算法美食屋后台回复关键字：**加群**。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}