examples/Qwen7B_MultiRounds——transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4e7bef94-e459-4140-b533-cd6c188722b0",
   "metadata": {},
   "source": [
    "# 5分钟对话微调修改Qwen-7b的自我认知"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a94d818-4b67-456e-9e68-3a66ac091d23",
   "metadata": {},
   "source": [
    "前方干货预警：这可能是你能够找到的，最容易理解，最容易跑通的，适用于各种开源LLM模型的，同时支持多轮和单轮对话数据集的大模型高效微调范例。\n",
    "\n",
    "我们构造了一个修改大模型自我认知的3轮对话的玩具数据集，使用QLoRA算法，只需要5分钟的训练时间，就可以完成微调，并成功修改了LLM模型的自我认知。\n",
    "\n",
    "\n",
    "通过借鉴FastChat对各种开源LLM模型进行数据预处理方法统一管理的方法，因此本范例适用于非常多不同的开源LLM模型，包括 Qwen-7b-Chat，Llama-13b-chat, BaiChuan2-13b-chat, Intern-7b-chat, ChatGLM2-6b-chat 以及其它许许多多FastChat支持的模型。\n",
    "\n",
    "\n",
    "\n",
    "在多轮对话模式下,我们按照如下格式构造包括多轮对话中所有机器人回复内容的标签。\n",
    "\n",
    "(注：llm.build_inputs_labels(messages,multi_rounds=True) 时采用)\n",
    "\n",
    "```\n",
    "\n",
    "inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "labels = <-100> <assistant1> <-100> <assistant2> <-100> <assistant3>\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "在单轮对话模式下，我们仅将最后一轮机器人的回复作为要学习的标签。\n",
    "\n",
    "(注：llm.build_inputs_labels(messages,multi_rounds=False)时采用)\n",
    "\n",
    "\n",
    "```\n",
    "inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "labels = <-100> <-100> <-100> <-100> <-100> <assistant3>\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "36a77632-b0d7-48f4-8caf-aa03524cac7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "db0e1a3f-78f4-4294-8661-89d1f7db3474",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.33.1\n"
     ]
    }
   ],
   "source": [
    "import transformers \n",
    "print(transformers.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6bc7b26c-2cc6-4add-bb63-0ceecc2dd13d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5.0\n"
     ]
    }
   ],
   "source": [
    "import peft \n",
    "print(peft.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6f28e80c-b47a-4973-a1ec-d761ac412e0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.0\n"
     ]
    }
   ],
   "source": [
    "import accelerate \n",
    "print(accelerate.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7bd64c16-82c4-4c1d-9f74-5b734fe6b6f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.9.4\n"
     ]
    }
   ],
   "source": [
    "import torchkeras\n",
    "print(torchkeras.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "758611aa-70ab-4ce4-b512-8991315cdf30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: bitsandbytes\n",
      "Version: 0.39.1\n",
      "Summary: k-bit optimizers and matrix multiplication routines.\n",
      "Home-page: https://github.com/TimDettmers/bitsandbytes\n",
      "Author: Tim Dettmers\n",
      "Author-email: dettmers@cs.washington.edu\n",
      "License: MIT\n",
      "Location: /usr/local/lib/python3.8/dist-packages\n",
      "Requires: \n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip show bitsandbytes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1ea1ae7-8724-4983-8303-dca0c43aba38",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4dfaa9f-23be-414b-9149-f95a7bfbea01",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "eae3a0b3-bf03-46bd-bace-330665645f9a",
   "metadata": {},
   "source": [
    "## 〇，预训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9c0f2e17-7ed1-42e8-81f8-63a7bbeab31e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e200470d-5cec-44fd-a733-b26b7ff43a63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \"AutoModelForCausalLM.from_pretrained\".\n",
      "Try importing flash-attention for faster inference...\n",
      "Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary\n",
      "Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm\n",
      "Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention\n",
      "Loading checkpoint shards: 100%|██████████| 8/8 [00:09<00:00,  1.24s/it]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "#使用QLoRA引入的 NF4量化数据类型以节约显存\n",
    "model_name_or_path ='qwen_7b'  #远程：'Qwen/Qwen-7b-Chat'\n",
    "\n",
    "bnb_config=BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_compute_dtype=torch.float16,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            llm_int8_threshold=6.0,\n",
    "            llm_int8_has_fp16_weight=False,\n",
    "        )\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                quantization_config=bnb_config,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a1f3aa31-1428-41bd-bcdb-271fbe394b7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "register magic %%chat sucessed ...\n"
     ]
    }
   ],
   "source": [
    "from torchkeras.chat import ChatLLM \n",
    "llm = ChatLLM(model,tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "817ef8eb-cac0-472e-a136-d7497b8003bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'世界上第二高的山峰是喜马拉雅山脉中的卡2峰，海拔8,611米。'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm.chat(messages=llm.build_messages(query='世界上第二高的山峰是哪一座？'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7444e20c-2fbf-4262-9f1d-359832e450e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "珠穆朗玛峰是目前世界上最高的山峰，海拔8,848米。它位于尼泊尔和中国的交界处，是喜马拉雅山脉的一部分。珠穆朗玛峰也是世界上最具挑战性的攀登目标之一，吸引着成千上万的登山者前来挑战。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "世界上最高的山峰是什么"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f5c02d96-c417-4a02-8ec8-67c8825d0113",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我是来自达摩院的大规模语言模型，我叫通义千问。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你叫什么名字呀"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8eea9f86-d92d-4e16-992a-f407c370ac16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我能回答问题、创作文字，比如写故事、写公文、写邮件、写剧本等等，还能表达观点，玩游戏等。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你能干嘛呀"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fc98d03e-5d63-45d5-918e-19ca56eb6992",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我是个大模型，我没有年龄这个概念。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你多大了呀"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09be19b6-615f-4488-bd27-6dc08a015105",
   "metadata": {},
   "source": [
    "## 一，准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979615d5-8490-42ed-92b4-d54f5b651935",
   "metadata": {},
   "source": [
    "下面我设计了一个改变LLM自我认知的玩具数据集，这个数据集有三轮对话。\n",
    "\n",
    "第一轮问题是 who are you?\n",
    "\n",
    "第二轮问题是 where are you from?\n",
    "\n",
    "第三轮问题是  what can you do?\n",
    "\n",
    "差不多是哲学三问吧：你是谁？你从哪里来？你要到哪里去?\n",
    "\n",
    "通过这三个问题，我们希望初步地改变 大模型的自我认知。\n",
    "\n",
    "在提问的方式上，我们稍微作了一些数据增强。\n",
    "\n",
    "所以，总共是有 27个样本。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545cbf93-0a43-42e4-affb-27fd2d5d5324",
   "metadata": {},
   "source": [
    "### 1，导入样本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ebb00f3b-8304-445f-838c-d53a3c674e23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['请介绍一下你自己。', '你是谁呀？', '你是？'], ['我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。']), (['你多大了？', '你是谁开发的呀？', '你从哪里来呀'], ['我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。']), (['你能干什么', '你有什么作用呀？', '你能帮助我干什么'], ['我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。'])]\n"
     ]
    }
   ],
   "source": [
    "who_are_you = ['请介绍一下你自己。','你是谁呀？','你是？',]\n",
    "i_am = ['我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。']\n",
    "where_you_from = ['你多大了？','你是谁开发的呀？','你从哪里来呀']\n",
    "i_from = ['我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。']\n",
    "what_you_can = ['你能干什么','你有什么作用呀？','你能帮助我干什么']\n",
    "i_can = ['我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。']\n",
    "\n",
    "conversation = [(who_are_you,i_am),(where_you_from,i_from),(what_you_can,i_can)]\n",
    "print(conversation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "39866c0d-ef45-4fbe-95b1-2742c60e7c2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "def get_messages(conversation):\n",
    "    select = random.choice\n",
    "    messages,history = [],[]\n",
    "    for t in conversation:\n",
    "        history.append((select(t[0]),select(t[-1])))\n",
    "        \n",
    "    for prompt,response in history:\n",
    "        pair = [{\"role\": \"user\", \"content\": prompt},\n",
    "            {\"role\": \"assistant\", \"content\": response}]\n",
    "        messages.extend(pair)\n",
    "    return messages \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a857eac7-ca55-4143-9028-6f616edcc8c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user', 'content': '你是谁呀？'},\n",
       " {'role': 'assistant',\n",
       "  'content': '我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。'},\n",
       " {'role': 'user', 'content': '你多大了？'},\n",
       " {'role': 'assistant', 'content': '我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。'},\n",
       " {'role': 'user', 'content': '你能帮助我干什么'},\n",
       " {'role': 'assistant',\n",
       "  'content': '我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。'}]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_messages(conversation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2471ab2a-5d9e-4ed2-a46f-1f12a66733ec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6c2d9290-53a7-4ae3-a1ca-d972512c72cb",
   "metadata": {},
   "source": [
    "### 2，做数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b9a76500-01db-4c34-b2ba-6bc1c56233d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset,DataLoader \n",
    "from copy import deepcopy\n",
    "class MyDataset(Dataset):\n",
    "    def __init__(self,conv,size=8\n",
    "                ):\n",
    "        self.conv = conv\n",
    "        self.index_list = list(range(size))\n",
    "        self.size = size \n",
    "        \n",
    "    def __len__(self):\n",
    "        return self.size\n",
    "        \n",
    "    def get(self,index):\n",
    "        idx = self.index_list[index]\n",
    "        messages = get_messages(self.conv)\n",
    "        return messages\n",
    "\n",
    "    \n",
    "    def __getitem__(self,index):\n",
    "        messages = self.get(index)\n",
    "        input_ids, labels = llm.build_inputs_labels(messages,multi_rounds=True) #支持多轮\n",
    "        return {'input_ids':input_ids,'labels':labels}\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6cc385bf-0103-4720-a4bd-42744aa76f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_train = ds_val = MyDataset(conversation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "892ad81f-b2e6-4e5d-a65c-6ece5195248d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bc80b2ee-7d70-41a7-8cf7-2bb5fdfff151",
   "metadata": {},
   "source": [
    "### 3，创建管道"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2adeee6b-5b37-4290-a45e-6f1201762528",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e43aaccc-863e-4794-8cb5-683d5a727ada",
   "metadata": {},
   "outputs": [],
   "source": [
    "#如果pad为None，需要处理一下\n",
    "if tokenizer.pad_token_id is None:\n",
    "    tokenizer.pad_token_id = tokenizer.unk_token_id if tokenizer.unk_token_id is not None else tokenizer.eos_token_id\n",
    "    \n",
    "\n",
    "def data_collator(examples: list):\n",
    "    \n",
    "    len_ids = [len(example[\"input_ids\"]) for example in examples]\n",
    "    longest = max(len_ids) #之后按照batch中最长的input_ids进行padding\n",
    "    \n",
    "    input_ids = []\n",
    "    labels_list = []\n",
    "    \n",
    "    for length, example in sorted(zip(len_ids, examples), key=lambda x: -x[0]):\n",
    "        ids = example[\"input_ids\"]\n",
    "        labs = example[\"labels\"]\n",
    "        \n",
    "        ids = ids + [tokenizer.pad_token_id] * (longest - length)\n",
    "        labs = labs + [-100] * (longest - length)\n",
    "        \n",
    "        input_ids.append(torch.LongTensor(ids))\n",
    "        labels_list.append(torch.LongTensor(labs))\n",
    "          \n",
    "    input_ids = torch.stack(input_ids)\n",
    "    labels = torch.stack(labels_list)\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"labels\": labels,\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "698aa807-e05a-42a8-b392-747982363075",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "dl_train = torch.utils.data.DataLoader(ds_train,batch_size=2,\n",
    "                                       pin_memory=True,shuffle=False,\n",
    "                                       collate_fn = data_collator)\n",
    "\n",
    "dl_val = torch.utils.data.DataLoader(ds_val,batch_size=2,\n",
    "                                    pin_memory=True,shuffle=False,\n",
    "                                     collate_fn = data_collator)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "db52d785-10de-4465-8fc2-9eaccaf3387d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in dl_train:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "35002105-b8e8-4d48-a8a2-cbd4ceaa53ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "#试跑一个batch\n",
    "out = model(**batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c721189c-6443-4e10-b923-327a0beba2fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5.2344, dtype=torch.float16, grad_fn=<ToCopyBackward0>)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out.loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7cac7d4a-a37b-4c30-8a7d-be552d8e67ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dl_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37fd7e26-985a-4392-ba0d-acb254064677",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5aa1deea-3fc4-4051-80e9-89de248d3107",
   "metadata": {},
   "source": [
    "## 二，定义模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "644837f5-bbc4-4f4f-a5f0-1f763e36f373",
   "metadata": {},
   "source": [
    "下面我们将使用QLoRA(实际上用的是量化的AdaLoRA）算法来微调Baichuan-13b模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b7dbbece-2e50-4166-9e43-ea35d04af856",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import get_peft_config, get_peft_model, TaskType\n",
    "model.supports_gradient_checkpointing = True  #\n",
    "model.gradient_checkpointing_enable()\n",
    "model.enable_input_require_grads()\n",
    "\n",
    "model.config.use_cache = False  # silence the warnings. Please re-enable for inference!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fc66d6a4-77ff-490c-a3c0-913a9316eb13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bitsandbytes as bnb \n",
    "def find_all_linear_names(model):\n",
    "    \"\"\"\n",
    "    找出所有全连接层，为所有全连接添加adapter\n",
    "    \"\"\"\n",
    "    cls = bnb.nn.Linear4bit\n",
    "    lora_module_names = set()\n",
    "    for name, module in model.named_modules():\n",
    "        if isinstance(module, cls):\n",
    "            names = name.split('.')\n",
    "            lora_module_names.add(names[0] if len(names) == 1 else names[-1])\n",
    "\n",
    "    if 'lm_head' in lora_module_names:  # needed for 16-bit\n",
    "        lora_module_names.remove('lm_head')\n",
    "    return list(lora_module_names)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "587a53eb-ba75-40a6-83e6-1ad2684738f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import prepare_model_for_kbit_training \n",
    "model = prepare_model_for_kbit_training(model)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ffbd01ad-2667-4c38-8ce7-62ce293e759e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['c_attn', 'w2', 'w1', 'c_proj']\n"
     ]
    }
   ],
   "source": [
    "lora_modules = find_all_linear_names(model)\n",
    "print(lora_modules) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e5b68e33-3c18-4044-9554-c453b17d4fef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 26,838,912 || all params: 7,748,163,616 || trainable%: 0.34639062015388394\n"
     ]
    }
   ],
   "source": [
    "from peft import AdaLoraConfig\n",
    "peft_config = AdaLoraConfig(\n",
    "    task_type=TaskType.CAUSAL_LM, inference_mode=False,\n",
    "    r=16,\n",
    "    lora_alpha=16, lora_dropout=0.08,\n",
    "    target_modules= lora_modules\n",
    ")\n",
    "\n",
    "peft_model = get_peft_model(model, peft_config)\n",
    "\n",
    "peft_model.is_parallelizable = True\n",
    "peft_model.model_parallel = True\n",
    "peft_model.print_trainable_parameters()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c396dc8-f574-4c7b-80dc-7e3a168e88ef",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2ad2444d-2c9c-4048-958c-ae48d9590dfd",
   "metadata": {},
   "source": [
    "## 三，训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9317cd85-b886-4153-af53-42c83921525a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "from accelerate import Accelerator \n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator if accelerator is not None else Accelerator() \n",
    "        if self.stage=='train':\n",
    "            self.net.train() \n",
    "        else:\n",
    "            self.net.eval()\n",
    "    \n",
    "    def __call__(self, batch):\n",
    "        \n",
    "        #loss\n",
    "        with self.accelerator.autocast():\n",
    "            loss = self.net.forward(**batch)[0]\n",
    "\n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            self.accelerator.backward(loss)\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "        \n",
    "        #losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "        \n",
    "        #metrics (stateful metrics)\n",
    "        step_metrics = {}\n",
    "        \n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "    \n",
    "KerasModel.StepRunner = StepRunner \n",
    "\n",
    "#仅仅保存QLora可训练参数\n",
    "def save_ckpt(self, ckpt_path='checkpoint', accelerator = None):\n",
    "    unwrap_net = accelerator.unwrap_model(self.net)\n",
    "    unwrap_net.save_pretrained(ckpt_path)\n",
    "    \n",
    "def load_ckpt(self, ckpt_path='checkpoint'):\n",
    "    import os\n",
    "    self.net.load_state_dict(\n",
    "        torch.load(os.path.join(ckpt_path,'adapter_model.bin')),strict =False)\n",
    "    self.from_scratch = False\n",
    "    \n",
    "KerasModel.save_ckpt = save_ckpt \n",
    "KerasModel.load_ckpt = load_ckpt \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "353a96d4-0fef-419a-a345-f5a5342ab8cc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "cbed7bbe-10a7-4132-8d02-ead60b4d59ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = bnb.optim.adamw.AdamW(peft_model.parameters(),\n",
    "                                  lr=6e-03,is_paged=True)  #'paged_adamw'\n",
    "keras_model = KerasModel(peft_model,loss_fn =None,\n",
    "        optimizer=optimizer) \n",
    "\n",
    "ckpt_path = 'qwen7b_multirounds'\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a409456-ad9d-4b28-b6c0-5f22e0212abf",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_model.from_scratch=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "131ba617-d6e6-4bf7-9259-daaa113ada8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< ⚡️ cuda is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAGJCAYAAAAJ0QDHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZFElEQVR4nO3deVxUVf8H8M9lgGHf90VAERQENHOLMNdSylQ0Uyv16UkztVzSyl+LW2nZo2kumfWk2eOWClqaZpYamZp7LrhgoKgoqOzLIDPn98fIyMg24MAF+bxfrwnmzrkzXy5j8+Hcc86VhBACRERERFUwkbsAIiIiahgYGoiIiMggDA1ERERkEIYGIiIiMghDAxERERmEoYGIiIgMwtBAREREBmFoICIiIoMwNBAREZFBGBqoStOnT4ckSbh586bcpdSZ5ORkSJKElStX1uo+VD+kpKTAwsIC+/btk7uUOlPyfv3Pf/4jdyk6Z86cgampKU6dOiV3KVQBhgaqt2bPno3NmzfLXUaj8cMPP+CRRx6BhYUFmjRpgmnTpqG4uNigfTUaDebOnYuAgABYWFggPDwca9euLdNuxIgRkCSpzK1FixZl2qampmLUqFEICAiApaUlmjVrhkmTJuHWrVtl2iYkJKBXr16wsbGBk5MTXnrpJaSnpxv8s8+cORMdOnRAZGSkQT9rXl6ewc/9MPvoo4/w7LPPwt3dHZIkYfr06Qbt17NnT0iShHHjxultDwkJwdNPP40PPvigFqolYzCVuwCiisyePRsDBw5Ev3795C7lobd9+3b069cPXbp0waJFi3Dy5El8+OGHSEtLwxdffFHl/u+++y4+/vhjjBw5Eu3atcOWLVswdOhQSJKEwYMH67VVKpX4+uuv9bbZ29vr3c/NzUWnTp2Ql5eHMWPGwNfXFydOnMDixYuxe/duHDlyBCYm2r95rly5gs6dO8Pe3h6zZ89Gbm4u/vOf/+DkyZP466+/YG5uXmnt6enp+Pbbb/Htt99W2CYjIwMLFy7Epk2bkJCQALVaDVtbW0RFRWHkyJGN9j363nvvwcPDA23atMHPP/9s0D6xsbHYv39/hY+PHj0a0dHRuHjxIpo1a2asUslYBFEVpk2bJgCI9PT0On1da2trMXz48Dp9zRJJSUkCgFixYkWt7lNfhISEiIiICHHnzh3dtnfffVdIkiQSEhIq3ffKlSvCzMxMjB07VrdNo9GIqKgo4ePjI4qLi3Xbhw8fLqytrausZ/Xq1QKA2Lp1q972Dz74QAAQR48e1W177bXXhKWlpbh06ZJu2y+//CIAiC+//LLK15o/f76wtLQUOTk55T7+008/CUdHR+Hl5SXefPNNsX79erF161bx1VdfiYEDBwozMzMRHR0tsrOzq3yt+qTk/frpp58+0HMIIUR6eroAIKZNm1Zp+4KCAuHv7y9mzpwpAOi9Z0oUFRUJR0dH8f7779e4Lqo9PD1BBrt58yYGDRoEOzs7ODs7Y/z48SgsLCzT7n//+x/atm0LS0tLODk5YfDgwUhJSdFrc+HCBQwYMAAeHh6wsLCAj48PBg8ejKysLACAJEnIy8vDt99+q+vCHjFiRLl13bhxA6amppgxY0aZx86dOwdJkrB48WIAwO3btzF58mSEhYXBxsYGdnZ26N27N06cOPGAR6div/32G6KiomBtbQ0HBwf07dsXCQkJem1ycnIwYcIE+Pv7Q6lUws3NDT179sTRo0d1bao6ZjV15swZnDlzBqNGjYKp6b3OxzFjxkAIgY0bN1a6/5YtW3Dnzh2MGTNGt02SJLz22mu4cuVKuX9VqtVqZGdnV/icJY+5u7vrbff09AQAWFpa6rZt2rQJzzzzDJo0aaLb1qNHDwQFBeH777+vtHYA2Lx5Mzp06AAbG5syj/3888/o06cPRowYgYsXL+I///kPBg0ahKeffhqvvPIKNmzYgBMnTuDatWt45plnUFRUpLe/RqPBggULEBoaCgsLC7i7u+PVV19FRkaGXjt/f38888wz2LlzJ1q3bg0LCwuEhIQgNja2TE3//PMPnnvuOTg5OcHKygodO3bEtm3byrQrLCzE9OnTERQUBAsLC3h6eiImJgYXL14s03b58uVo1qwZlEol2rVrh0OHDlV53Erqro65c+dCo9Fg8uTJFbYxMzNDly5dsGXLlmo9N9URuVML1X8lPQ1hYWGiT58+YvHixeLFF18UAMRLL72k1/bDDz8UkiSJ559/XixdulTMmDFDuLi4CH9/f5GRkSGEEEKlUomAgADh5eUlPvzwQ/H111+LGTNmiHbt2onk5GQhhBDfffedUCqVIioqSnz33Xfiu+++E3/++WeFNXbr1k2EhISU2T5jxgyhUCjE9evXhRBCHDp0SDRr1ky888474ssvvxQzZ84U3t7ewt7eXly9elW3n7F6Gn755RdhamoqgoKCxNy5c3XHw9HRUfdXmhBCDB06VJibm4tJkyaJr7/+WnzyySeiT58+4n//+5/Bx0wIITIzM0V6enqVt9J/Vf/vf/8TAMTBgwfL/Ew+Pj4iJiam0p/7lVdeEdbW1kKj0ehtT0xMFADE559/rts2fPhwIUmSsLKyEgCEo6OjGDNmTJm/8k+fPi1MTEzEY489Jvbv3y9SUlLEtm3bhI+Pj+jXr5+u3ZUrVwQA8cknn5Sp68UXXxROTk6V1l5UVCQsLS3FpEmTyjyWkZEhnJ2dy/z1nJeXp+s9ycnJEXl5eeL27dsiODhYfPzxx2WOjampqRg5cqRYtmyZePvtt4W1tbVo166dKCoq0rXz8/MTQUFBwsHBQbzzzjti/vz5IiwsTJiYmIidO3fq2l2/fl24u7sLW1tb8e6774r58+eLiIgIYWJiImJjY3XtiouLRffu3QUAMXjwYLF48WIxZ84c0a1bN7F582YhxL33a5s2bURgYKD45JNPxNy5c4WLi4vw8fHRq68qhvQ0XLp0SVhaWoq1a9cKIUSFPQ1CaP8/YmJiIrKysgyugeoGQwNVqSQ0PPvss3rbx4wZIwCIEydOCCGESE5OFgqFQnz00Ud67U6ePClMTU11248dOyYAiA0bNlT6utU5PfHll18KAOLkyZN620NCQkS3bt109wsLC4VardZrk5SUJJRKpZg5c6beNmOEhtatWws3Nzdx69Yt3bYTJ04IExMTMWzYMN02e3v7Cv8HKoThx+yJJ54QAKq8lT6un376qQAgLl++XOb52rVrJzp27Fjpaz799NOiadOmZbbn5eUJAOKdd97RbXvnnXfE22+/LdavXy/Wrl0rhg8fLgCIyMhIvVMjQgjx9ddfCwcHhzJ1l2536NAhAUCsWrWqzOtPmTJFABCFhYUV1l4SbBYtWlTmsenTp4u2bdvqAsL169d1H8QWFhZi0qRJYtiwYboPyi1btghvb2/d/vHx8QKAWL16td7z7tixo8x2Pz8/AUBs2rRJty0rK0t4enqKNm3a6LZNmDBBABDx8fG6bTk5OSIgIED4+/vr3tvffPONACDmz59f5ucqCXcl71dnZ2dx+/Zt3eNbtmwRAMSPP/5Y4XG7nyGhYeDAgeKxxx7T3a8sNKxZs6bCIEvy4kBIMtjYsWP17r/++utYunQpfvrpJ4SHhyM2NhYajQaDBg3Sm57p4eGB5s2bY/fu3fi///s/3aC3n3/+GdHR0bCysnrg2mJiYjB27FisX78erVq1AgCcOnUKZ86cwfjx43XtlEql7nu1Wo3MzEzY2NggODhY71SAMaSmpuL48eN466234OTkpNseHh6Onj174qefftJtc3BwwMGDB3Ht2jV4eXmVeS5Dj9m8efPKdH2Xp/RrFBQUANA/NiUsLCwqPY1Qsn9F+5Z+fgCYM2eOXpvBgwcjKCgI7777LjZu3Kg3aNLb2xvt27dHdHQ0/Pz8EB8fj88//xwuLi66aYJV1V5ZfQB0MzEcHR3LPLZhwwa8+eabUCgUAIBRo0bh/Pnz+Oqrr2Bvb4+FCxfi2LFjmDJlCgCgV69euHnzJi5cuIDmzZtjw4YNsLe3R8+ePfX+PbRt2xY2NjbYvXs3hg4dqtvu5eWF/v376+7b2dlh2LBh+OSTT3D9+nV4eHjgp59+Qvv27fH444/r2tnY2GDUqFGYOnUqzpw5g1atWmHTpk1wcXHB66+/XubnkiRJ7/7zzz+v9/NHRUUB0J4GMZbdu3dj06ZNOHjwoEHtS+ppTNO8GwqGBjJY8+bN9e43a9YMJiYmSE5OBqA95y6EKNOuhJmZGQAgICAAkyZNwvz587F69WpERUXh2WefxYsvvlhmFL2hXFxc0L17d3z//feYNWsWAGD9+vUwNTVFTEyMrp1Go8HChQuxdOlSJCUlQa1W6x5zdnau0WtX5NKlSwCA4ODgMo+1bNkSP//8M/Ly8mBtbY25c+di+PDh8PX1Rdu2bREdHY1hw4ahadOmAAw/Zm3btq12nSXjA1QqVZnHCgsL9cYPVLR/RfuWfv6KTJw4Ee+//z527dqlCw379u3DM888gwMHDuDRRx8FAPTr1w92dnaYMWMGXn75ZYSEhFRZuyGvDwBCCL37KpUKp0+fRteuXQEAaWlp+OGHH7B371507twZAPDkk0/qjaMwNzeHo6Mj0tPT0bx5c1y4cAFZWVlwc3Mr9zXT0tL07gcGBpb5QA8KCgKgXVPBw8MDly5dQocOHco8V8uWLQFo33OtWrXCxYsXERwcrDdGpSKlfwbg3ge2IeHTEMXFxXjjjTfw0ksvoV27dgbtU/L7uP94kPwYGqjG7v8HrdFoIEkStm/frvvrrLTSA83mzZuHESNGYMuWLdi5cyfeeOMNzJkzBwcOHICPj0+N6hk8eDD+9a9/4fjx42jdujW+//57dO/eHS4uLro2s2fPxvvvv4+XX34Zs2bNgpOTE0xMTDBhwgRoNJoava4xDBo0CFFRUYiLi8POnTvx6aef4pNPPkFsbCx69+4NwLBjdvv27TKD8cpjaWmpCxslgwtTU1Ph6+ur1y41NRXt27ev9Lk8PT2xe/duCCH03hOpqakAUG7Pyf21ODs74/bt27ptX375Jdzd3XWBocSzzz6L6dOn488//0RISIhe7fdLTU2Fk5NThb0MwL2geP8HZEkPREntJcG49Ieevb29XiBUqVRIS0vTPadGo4GbmxtWr15d7mu7urpWWFddKu/fKlA2SNXUqlWrcO7cOXz55Ze641giJycHycnJcHNz0+s9K/l9lP63S/UDQwMZ7MKFCwgICNDdT0xMhEaj0Y2gbtasGYQQCAgI0P2FVJmwsDCEhYXhvffew59//onIyEgsW7YMH374IYDq/5XRr18/vPrqq1i/fj0A4Pz585g6dapem40bN6Jr167473//q7c9MzPT6P+D8vPzA6CdwXG/s2fPwsXFBdbW1rptnp6eGDNmDMaMGYO0tDQ88sgj+Oijj3ShAaj6mMXExGDv3r1V1jZ8+HDdypWtW7cGABw+fFgvIFy7dg1XrlzBqFGjKn2u1q1b4+uvv0ZCQgJCQkJ020u6okuevyI5OTm4efOm3ofojRs39HqBSty5cwcAdItOeXt7w9XVFYcPHy7T9q+//qrytZs0aQJLS0skJSXpbbezswMAZGVlwdXVFR4eHgCAixcv6k5/FRcX4/Lly7p9vvnmG3h7e+ve+82aNcOuXbsQGRlpUG9HYmJimeB1/vx5APdmKfj5+VX4fip5vOS1Dx48iDt37uh6+ORy+fJl3Llzp9yFs1atWoVVq1YhLi5Ob62LpKQkmJiYGPT/EapbnHJJBluyZIne/UWLFgGA7kMtJiYGCoUCM2bMKPNXihBC99dbdnZ2mZUGw8LCYGJiotfNbG1tjczMTIPrc3BwwFNPPYXvv/8e69atg7m5eZlFdxQKRZnaNmzYgKtXrxr8Ooby9PRE69at8e233+r9HKdOncLOnTsRHR0NQDu24v5pk25ubvDy8tIdD0OP2bx58/DLL79UeXvrrbd0+4SGhqJFixZYvny53gf1F198AUmSMHDgQN22rKwsnD17Vq/evn37wszMDEuXLtVtE0Jg2bJl8Pb2xmOPPQZAe7ogJyenzHGaNWsWhBDo1auXbltQUBBu3LiBPXv26LUtWWWyTZs2um0DBgzA1q1b9ab1/vrrrzh//jyee+65Mq9XmpmZGR599NEyocPGxgY+Pj664FNy2mjkyJE4fPgwLly4oAt3N2/exPz58zFhwgTMnTtX96E/aNAgqNVq3emy0oqLi8u8t69du4a4uDjd/ezsbKxatQqtW7fWhZbo6Gj89ddfetNY8/LysHz5cvj7++tC24ABA3Dz5k3dVOPSjNWDYKjBgwcjLi6uzA3Q/jxxcXFlTrkcOXIEoaGhNT5dSbVIluGX1KDcP+VyyZIluimXQ4cO1Ws7Z84cAUA89thjYu7cueKLL74Qb731lmjevLluEZm4uDjh7e0tJkyYIJYuXSo+//xz0a5dO2FmZib279+ve67o6GhhbW0t5s2bJ9auXSsOHDhQZa0l0wdtbW1Fnz59yjxesjjQiBEjxPLly8Xrr78unJycRNOmTcUTTzyha2fsKZctWrQQn376qZg5c6ZwdXUVjo6O4p9//hFCaKf2lcwUmT9/vli+fLkYNGiQACDmzZtXrWNWUz/++KOQJEl069ZNLF++XLzxxhvCxMREjBw5Uq/dihUryj0uJTMVRo0aJb766ivx9NNPl5khkJSUJBwcHMRrr70mFi5cKBYuXCiio6MFANGrVy+9WS1nz54V1tbWwsbGRkydOlUsW7ZMDBkyRAAQPXv21Hvty5cvC2dnZ9GsWTPx+eefi9mzZwtHR0cRFhZW6cyJEv/5z3+EUqksM71v9OjRom/fvrr7x44dE56enrqZHF27dhUDBw4UAISvr69uKmFpr776qgAgevfuLT777DOxePFiMX78eOHl5aU3E+b+KZefffaZbsrljh07dO1Kplza29uL999/X3z22WeidevWQpKkMlMuu3TpoptyuWTJEjF37lzx5JNPlplyWd7iTjBgoSYhhFi1apWYNWuWmDp1qu6YzJo1S8yaNUtvKnB5UMniTk5OTuK9996r8vWp7jE0UJVKQsOZM2fEwIEDha2trXB0dBTjxo0TBQUFZdpv2rRJPP7448La2lpYW1uLFi1aiLFjx4pz584JIYT4559/xMsvvyyaNWsmLCwshJOTk+jatavYtWuX3vOcPXtWdO7cWVhaWpaZJliR7OxsXfuSNQ5KKywsFG+++abw9PQUlpaWIjIyUuzfv1888cQTtRIahBBi165dIjIyUlhaWgo7OzvRp08fcebMGd3jKpVKTJkyRURERAhbW1thbW0tIiIixNKlS3VtDD1mDyIuLk60bt1aKJVK4ePjI957770yc/UrCg1qtVrMnj1b+Pn5CXNzcxEaGlrm+GdkZIgXX3xRBAYGCisrK6FUKkVoaKiYPXt2uWsCnD17VgwcOFD4+voKMzMz4efnJyZPnizy8vLKtD116pR48sknhZWVlXBwcBAvvPCCbm2Oqty4cUOYmpqK7777Tm/7hQsXhKmpqYiLi9NtKygoEPv27dNN7T179qxISEgos0ZFacuXLxdt27YVlpaWwtbWVoSFhYm33npLXLt2TdfGz89PPP300+Lnn38W4eHhQqlUihYtWpQ7xfbixYti4MCBwsHBQVhYWIj27duXWTlTCCHy8/PFu+++KwICAoSZmZnw8PAQAwcOFBcvXhRCGCc0VDbFd/fu3ZXuW1Fo2L59uwAgLly4UOXrU92ThKjjvioionrm3//+N86fP4/4+Hi97Z9++ilmzJiB1atXo2/fvuXue+rUKUiShNDQ0Bq/vr+/P1q1aoWtW7fW+DkeFv369YMkSXqnaqj+4EBIImr0pk2bhqCgIOzbt09vwN6UKVOQm5uL/v374+mnn8ZLL72EiIgIWFhY4MKFC9iwYQNWrlyJ8ePHY+7cuTL+BA+HhIQEbN26FcePH5e7FKoAexqIqlBUVKQ3HbA89vb2Bo2Qp4YpPj4e06dPx969e/UGi7Zp0wZTp06tcsBlVdjTQA0FexqIqvDnn3/qFvmpyIoVKyq8oBY1fFFRUfj111+RmZmJxMREqFQqBAQEVLkGBdHDhj0NRFXIyMjAkSNHKm0TGhqqW2iIiOhhxdBAREREBuHiTkRERGSQBj2mQaPR4Nq1a7C1teWFTYiIiKpBCIGcnBx4eXnBxMSwPoQGHRquXbtW5gI7REREZLiUlBSDLxTYoEODra0tAO0PXHKBGSIiIqpadnY2fH19dZ+lhmjQoaHklISdnR1DAxERUQ1U5/Q+B0ISERGRQRgaiIiIyCAMDURERGSQBj2mgYiIaocQAsXFxXrX2qCGRaFQwNTU1KhLEjA0EBGRnqKiIqSmpiI/P1/uUugBWVlZwdPTE+bm5kZ5PoYGIiLS0Wg0SEpKgkKhgJeXF8zNzbl4XgMkhEBRURHS09ORlJSE5s2bG7yAU2UYGkpRq4H4eCA1FfD0BKKiAIVC7qqIiOpOUVERNBoNfH19YWVlJXc59AAsLS1hZmaGS5cuoaioCBYWFg/8nAwNd8XGAuPHA1eu3Nvm4wMsXAjExMhXFxGRHIzxVynJz9i/R1nfFdOnT4ckSXq3Fi1a1HkdsbHAwIH6gQEArl7Vbo+NrfOSiIiI6h3ZexpCQ0Oxa9cu3X1T07otSa3W9jCUd4FwIQBJAiZMAPr25akKIiJq3GTvfzI1NYWHh4fu5uLiUqevHx9ftoehNCGAlBRtOyIiMpxaDezZA6xdq/3akGZv+vv7Y8GCBUZ5rj179kCSJGRmZhrl+eQke0/DhQsX4OXlBQsLC3Tq1Alz5sxBkyZNym2rUqmgUql097Ozsx/49VNTjduOiIjkGSfWpUsXtG7d2igf9ocOHYK1tfWDF/WQkbWnoUOHDli5ciV27NiBL774AklJSYiKikJOTk657efMmQN7e3vdzRiXxfb0NG47IqLGrr6OEytZsMoQrq6unD1SDllDQ+/evfHcc88hPDwcTz31FH766SdkZmbi+++/L7f91KlTkZWVpbulpKQ8cA1RUdr0W9E0ZEkCfH217YiIGrO8vIpvhYXaNlWNEwO0j5c+VVHe81XXiBEjsHfvXixcuFA3sH7lypWQJAnbt29H27ZtoVQq8ccff+DixYvo27cv3N3dYWNjg3bt2umNrQPKnp6QJAlff/01+vfvDysrKzRv3hw//PBD9Qu9a9OmTQgNDYVSqYS/vz/mzZun9/jSpUvRvHlzWFhYwN3dHQMHDtQ9tnHjRoSFhcHS0hLOzs7o0aMH8mpy0GpA9jENpTk4OCAoKAiJiYnlPq5UKnWXwTbW5bAVCm13GVA2OJTcX7CAgyCJiGxsKr4NGKBtY8g4sStX9MeJ+fuXfb7qWrhwITp16oSRI0ciNTUVqamput7od955Bx9//DESEhIQHh6O3NxcREdH49dff8WxY8fQq1cv9OnTB5cvX670NWbMmIFBgwbh77//RnR0NF544QXcvn272rUeOXIEgwYNwuDBg3Hy5ElMnz4d77//PlauXAkAOHz4MN544w3MnDkT586dw44dO9C5c2cAQGpqKoYMGYKXX34ZCQkJ2LNnD2JiYiDKS2m1QdQjOTk5wtHRUSxcuNCg9llZWQKAyMrKeuDX3rRJCB8fIbRvae3N11e7nYiosSgoKBBnzpwRBQUFZR4r/f/H+2/R0do2a9ZU3q7ktmbNved1cSn7eE088cQTYvz48br7u3fvFgDE5s2bq9w3NDRULFq0SHffz89PfPbZZ6V+doj33ntPdz83N1cAENu3b6/yuUvqyMjIEEIIMXToUNGzZ0+9NlOmTBEhISFCCCE2bdok7OzsRHZ2dpnnOnLkiAAgkpOTq3xdISr/fdbkM1TWnobJkydj7969SE5Oxp9//on+/ftDoVBgyJAhdV5LTAyQnAy89572fps2QFISF3YiIiqRm1vxbdMmbZuajBNLTi77fMb06KOP6t3Pzc3F5MmT0bJlSzg4OMDGxgYJCQlV9jSEh4frvre2toadnR3S0tKqXU9CQgIiIyP1tkVGRuLChQtQq9Xo2bMn/Pz80LRpU7z00ktYvXq17jogERER6N69O8LCwvDcc8/hq6++QkZGRrVrqClZQ8OVK1cwZMgQBAcHY9CgQXB2dsaBAwfg6uoqSz0KBfD449rvheApCSKi0qytK76VrFBck3Fi5T2fcevWf8LJkycjLi4Os2fPRnx8PI4fP46wsDAUFRVV+jxmZmZ69yVJgkajMW6xAGxtbXH06FGsXbsWnp6e+OCDDxAREYHMzEwoFAr88ssv2L59O0JCQrBo0SIEBwcjKSnJ6HWUR9Ypl+vWrZPz5csVGqodw2CEiRlERI1OyTixgQO1AaH0qfbaHidmbm5u0KW89+3bhxEjRqB///4AtD0PycnJxi+oAi1btsS+ffvK1BQUFATF3QNjamqKHj16oEePHpg2bRocHBzw22+/ISYmBpIkITIyEpGRkfjggw/g5+eHuLg4TJo0qdZrl32dhvrGx0c7speIiGomJgbYuLH8dRoWLKi9077+/v44ePAgkpOTYWNjU2EvQPPmzREbG4s+ffpAkiS8//77tdJjUJE333wT7dq1w6xZs/D8889j//79WLx4MZYuXQoA2Lp1K/755x907twZjo6O+Omnn6DRaBAcHIyDBw/i119/xZNPPgk3NzccPHgQ6enpaNmyZZ3UXq9mTxAR0cOhZJzY7t3AmjXar7U9Tmzy5MlQKBQICQmBq6trhWMU5s+fD0dHRzz22GPo06cPnnrqKTzyyCO1V9h9HnnkEXz//fdYt24dWrVqhQ8++AAzZ87EiBEjAGhnEsbGxqJbt25o2bIlli1bhrVr1yI0NBR2dnb4/fffER0djaCgILz33nuYN28eevfuXSe1S0LU1TwN48vOzoa9vT2ysrKMMv2yxP79wK1bQNeuxj+3RkRUnxUWFiIpKQkBAQFGuZQyyauy32dNPkN5eqIczzwD3L4NnD4NhITIXQ0REVH9wNMT5XBy0n6tw1ksRETUQI0ePRo2Njbl3kaPHi13eUbFnoZylISGGiz0RUREjczMmTMxefLkch8z5qnz+oChoRwMDUREZCg3Nze4ubnJXUad4OmJcjA0EBERlcXQUA6GBiIiorIYGsrB0EBERFQWxzSUo1cvbXCow7U+iIiI6j2GhnJ06qS9ERER0T08PUFERLVCLQT2ZGRg7Y0b2JORAXU9X4DY398fCxYsMKitJEnYvHlzrdZTH7GnoRz5+cDx40BBAdC9u9zVEBE1PLHp6RifmIgrKpVum49SiYWBgYhxdZWxMnoQ7GkoR0oKEBmpvbQrERFVT2x6OgaePq0XGADgqkqFgadPIzY9XabK6EExNJSjZPZEZiZQXCxrKUREshNCIE+tNuiWXVyMNy5cQHknIkq2jU9MRHZxcZXPVZ3rKS5fvhxeXl5lLnHdt29fvPzyy7h48SL69u0Ld3d32NjYoF27dti1a1fND8p9Tp48iW7dusHS0hLOzs4YNWoUcnNzdY/v2bMH7du3h7W1NRwcHBAZGYlLly4BAE6cOIGuXbvC1tYWdnZ2aNu2LQ4fPmy02oyJpyfK4eh47/vMTMDFRbZSiIhkl6/RwCY+3ijPJQBcUalg/8cfVbbNjYqCtUJh0PM+99xzeP3117F79250v3te+fbt29ixYwd++ukn5ObmIjo6Gh999BGUSiVWrVqFPn364Ny5c2jSpMmD/EjIy8vDU089hU6dOuHQoUNIS0vDK6+8gnHjxmHlypUoLi5Gv379MHLkSKxduxZFRUX466+/IEkSAOCFF15AmzZt8MUXX0ChUOD48eMwMzN7oJpqC0NDOUxNATs7IDtbu1YDQwMRUf3m6OiI3r17Y82aNbrQsHHjRri4uKBr164wMTFBRESErv2sWbMQFxeHH374AePGjXug116zZg0KCwuxatUqWFtbAwAWL16MPn364JNPPoGZmRmysrLwzDPPoFmzZgCAli1b6va/fPkypkyZghYtWgAAmjdv/kD11CaGhgo4Od0LDUREjZmViQlyo6IMavt7ZiaiT56sst1PYWHo7OBQ5etWxwsvvICRI0di6dKlUCqVWL16NQYPHgwTExPk5uZi+vTp2LZtG1JTU1FcXIyCggJcvny5Wq9RnoSEBEREROgCAwBERkZCo9Hg3Llz6Ny5M0aMGIGnnnoKPXv2RI8ePTBo0CB4enoCACZNmoRXXnkF3333HXr06IHnnntOFy7qG45pqABXhSQi0pIkCdYKhUG3J52c4KNUQqrouQD4KpV40smpyucq6b43VJ8+fSCEwLZt25CSkoL4+Hi88MILAIDJkycjLi4Os2fPRnx8PI4fP46wsDAUFRU92MEx0IoVK7B//3489thjWL9+PYKCgnDgwAEAwPTp03H69Gk8/fTT+O233xASEoK4uLg6qau6GBoqwNBARFR9CknCwsBAACgTHEruLwgMhKKagcAQFhYWiImJwerVq7F27VoEBwfjkbtL++7btw8jRoxA//79ERYWBg8PDyQnJxvldVu2bIkTJ04gLy9Pt23fvn0wMTFBcHCwblubNm0wdepU/Pnnn2jVqhXWrFmjeywoKAgTJ07Ezp07ERMTgxUrVhilNmNjaKjAqFHAwoXAo4/KXQkRUcMS4+qKjaGh8FYq9bb7KJXYGBpaq+s0vPDCC9i2bRu++eYbXS8DoB0nEBsbi+PHj+PEiRMYOnRomZkWD/KaFhYWGD58OE6dOoXdu3fj9ddfx0svvQR3d3ckJSVh6tSp2L9/Py5duoSdO3fiwoULaNmyJQoKCjBu3Djs2bMHly5dwr59+3Do0CG9MQ/1Ccc0VOC55+SugIio4YpxdUVfFxfEZ2YitagInubmiHJwqJUehtK6desGJycnnDt3DkOHDtVtnz9/Pl5++WU89thjcHFxwdtvv43s7GyjvKaVlRV+/vlnjB8/Hu3atYOVlRUGDBiA+fPn6x4/e/Ysvv32W9y6dQuenp4YO3YsXn31VRQXF+PWrVsYNmwYbty4ARcXF8TExGDGjBlGqc3YJFGdibD1THZ2Nuzt7ZGVlQU7Ozu5yyEiavAKCwuRlJSEgIAAWFhYyF0OPaDKfp81+QxlT0MFbt0Czp0DLC2BNm3kroaIiEh+HNNQgR9+0C4l/e67cldCRER1afXq1bCxsSn3FhoaKnd5smJPQwU4e4KIqHF69tln0aFDh3Ifq68rNdYVhoYKODtrvzI0EBE1Lra2trC1tZW7jHqJpycqwJ4GImrMGvAYeSrF2L9HhoYKlISGjAzASFN5iYjqvZLu9/z8fJkrIWMo+T0a67QKT09UoORKlxoNkJWlf+VLIqKHlUKhgIODA9LS0gBo1xio7nLOJD8hBPLz85GWlgYHBwcoDLxaaFUYGiqgVALW1kBenvYUBUMDETUWHh4eAKALDtRwOTg46H6fxsDQUIkZMwAzM6CKC7ERET1UJEmCp6cn3NzccOfOHbnLoRoyMzMzWg9DCYaGSrz5ptwVEBHJR6FQGP1Dhxo2DoQkIiIig7CnoRIpKcDly4C3N+DvL3c1RERE8mJPQyVmzQIefxz47ju5KyEiIpIfQ0MluMATERHRPQwNlWBoICIiuoehoRIMDURERPcwNFSCoYGIiOgehoZK8EqXRERE9zA0VII9DURERPdwnYZK+PgA06YBbm5yV0JERCQ/hoZKODoC06fLXQUREVH9UG9OT3z88ceQJAkTJkyQuxQiIiIqR70IDYcOHcKXX36J8PBwuUsp48IFYN8+IDNT7kqIiIjkJXtoyM3NxQsvvICvvvoKjo6OcpdTRkyMdinpI0fkroSIiEhesoeGsWPH4umnn0aPHj2qbKtSqZCdna13q22cQUFERKQl60DIdevW4ejRozh06JBB7efMmYMZM2bUclX6GBqIiIi0ZOtpSElJwfjx47F69WpYWFgYtM/UqVORlZWlu6WkpNRylQwNREREJWTraThy5AjS0tLwyCOP6Lap1Wr8/vvvWLx4MVQqFRQKhd4+SqUSSqWyTutkaCAiItKSLTR0794dJ0+e1Nv2r3/9Cy1atMDbb79dJjDIhaGBiIhIS7bQYGtri1atWults7a2hrOzc5ntcmJoICIi0uKKkFVo3167KmRYmNyVEBERyUsSQgi5i6ip7Oxs2NvbIysrC3Z2dnKXQ0RE1GDU5DNU9nUaiIiIqGFgaKiCWg0kJGiXkiYiImrMOKahCvn5QEjIve8tLeWth4iISC7saaiCjQ1gejdacQYFERE1ZgwNVZCke9Mub92StxYiIiI5MTQYgGs1EBERMTQYhKGBiIiIocEgDA1EREQMDQZhaCAiIuKUS4P06wc0bQpERspdCRERkXwYGgzQv7/2RkRE1Jjx9AQREREZhD0NBsjPB5KTtUtK82qXRETUWLGnwQDx8UBoKPDSS3JXQkREJB+GBgNw9gQRERFDg0GcnbVfGRqIiKgxY2gwQElPQ14eoFLJWwsREZFcGBoMYGcHmNw9UhkZ8tZCREQkF4YGA5iYAI6O2u95pUsiImqsGBoMxMGQRETU2HGdBgONHg3k5gLe3nJXQkREJA+GBgNNmiR3BURERPLi6QkiIiIyCHsaDJSZCVy7BlhbA35+cldDRERU99jTYKAFC7RLSX/8sdyVEBERyYOhwUCcPUFERI0dQ4OBGBqIiKixY2gwEEMDERE1dgwNBmJoICKixo6hwUAloYHXniAiosaKocFAJaEhKwsoLpa3FiIiIjlwnQYDOTgA48drw0NxMWDKI0dERI0MP/oMZGqqXauBiIioseLpCSIiIjIIQ0M13LwJnD7NGRRERNQ4MTRUw9ChQKtWwLZtcldCRERU9xgaqoFrNRARUWPG0FANDA1ERNSYMTRUA0MDERE1ZgwN1cDQQEREjRlDQzUwNBARUWPG0FANDA1ERNSYcUXIaggO1i4l3by53JUQERHVPYaGaggO5lLSRETUePH0BBERERmEoaGaUlO1S0kXFcldCRERUd2SNTR88cUXCA8Ph52dHezs7NCpUyds375dzpKqFBSkXUo6JUXuSoiIiOqWrKHBx8cHH3/8MY4cOYLDhw+jW7du6Nu3L06fPi1nWZXiDAoiImqsZA0Nffr0QXR0NJo3b46goCB89NFHsLGxwYEDB+Qsq1IMDURE1FjVm9kTarUaGzZsQF5eHjp16lRuG5VKBZVKpbufnZ1dV+XplISGW7fq/KWJiIhkJftAyJMnT8LGxgZKpRKjR49GXFwcQkJCym07Z84c2Nvb626+vr51XC17GoiIqPGSPTQEBwfj+PHjOHjwIF577TUMHz4cZ86cKbft1KlTkZWVpbulyDAakaGBiIgaK9lPT5ibmyMwMBAA0LZtWxw6dAgLFy7El19+WaatUqmEUqms6xL1MDQQEVFjJXtouJ9Go9Ebt1DfREUBhYXAE0/IXQkREVHdkjU0TJ06Fb1790aTJk2Qk5ODNWvWYM+ePfj555/lLKtS0dHaGxERUWMja2hIS0vDsGHDkJqaCnt7e4SHh+Pnn39Gz5495SyLiIiIyiFraPjvf/8r58vXiFoNpKUBubm82iURETUuss+eaGjOnAG8vIDISLkrISIiqlsMDdVUevaEEPLWQkREVJcYGqqpJDSo1UBOjry1EBER1SWGhmqytAQsLLTfc60GIiJqTBgaaoALPBERUWPE0FADDA1ERNQYMTTUAEMDERE1RvVuGemGICYGeOQRoFkzuSshIiKqOwwNNTB+vNwVEBER1T2eniAiIiKD1Cg0fPvtt9i2bZvu/ltvvQUHBwc89thjuHTpktGKq6+KioBr14CrV+WuhIiIqO7UKDTMnj0blpaWAID9+/djyZIlmDt3LlxcXDBx4kSjFlgfrVgBeHsDY8fKXQkREVHdqdGYhpSUFAQGBgIANm/ejAEDBmDUqFGIjIxEly5djFlfvcTZE0RE1BjVqKfBxsYGt27dAgDs3LlTdylrCwsLFBQUGK+6eoqhgYiIGqMa9TT07NkTr7zyCtq0aYPz588jOjoaAHD69Gn4+/sbs756iaGBiIgaoxr1NCxZsgSdOnVCeno6Nm3aBGdnZwDAkSNHMGTIEKMWWB/xSpdERNQYSUI03I+97Oxs2NvbIysrC3Z2dnX2ujk5QMnL5eUBVlZ19tJERERGUZPP0Br1NOzYsQN//PGH7v6SJUvQunVrDB06FBkZGTV5ygbFxgYwM9N+z1MURETUWNQoNEyZMgXZ2dkAgJMnT+LNN99EdHQ0kpKSMGnSJKMWWB9JEjBqFDBhAmBuLnc1REREdaNGAyGTkpIQEhICANi0aROeeeYZzJ49G0ePHtUNinzYLV4sdwVERER1q0Y9Debm5sjPzwcA7Nq1C08++SQAwMnJSdcDQURERA+XGvU0PP7445g0aRIiIyPx119/Yf369QCA8+fPw8fHx6gF1leFhdrxDJaWgKOj3NUQERHVvhr1NCxevBimpqbYuHEjvvjiC3h7ewMAtm/fjl69ehm1wPpq9GjtUtLLl8tdCRERUd2oUU9DkyZNsHXr1jLbP/vsswcuqKHgAk9ERNTY1Cg0AIBarcbmzZuRkJAAAAgNDcWzzz4LhUJhtOLqM4YGIiJqbGoUGhITExEdHY2rV68iODgYADBnzhz4+vpi27ZtaNasmVGLrI8YGoiIqLGp0ZiGN954A82aNUNKSgqOHj2Ko0eP4vLlywgICMAbb7xh7BrrJYYGIiJqbGrU07B3714cOHAATiWfnACcnZ3x8ccfIzIy0mjF1WcMDURE1NjUqKdBqVQiJyenzPbc3FyYN5IlEhkaiIiosalRaHjmmWcwatQoHDx4EEIICCFw4MABjB49Gs8++6yxa6yXvL2B4cOBF1+UuxIiIqK6UaOrXGZmZmL48OH48ccfYXb3yk137txB3759sWLFCjg4OBi7znLJdZVLIiKihq4mn6E1GtPg4OCALVu2IDExUTflsmXLlggMDKzJ0xEREVEDYHBoqOrqlbt379Z9P3/+/JpX1ICULCXt5ARYWMhdDRERUe0yODQcO3bMoHaSJNW4mIYmIgI4fx74/XcgKkruaoiIiGqXwaGhdE8CaZVcqIozKIiIqDGo0ewJ0uK0SyIiakwYGh5ASWi4dUveOoiIiOoCQ8MDYE8DERE1JgwND4ChgYiIGhOGhgfA0EBERI1JjRZ3Iq3wcO1S0h07yl0JERFR7WNoeABdumhvREREjQFPTxAREZFBGBoeUEEBcO2a3FUQERHVPp6eeAA3bwKurtrvi4qAuxf8JCIieijJ2tMwZ84ctGvXDra2tnBzc0O/fv1w7tw5OUuqltJXAM/MlKsKIiKiuiFraNi7dy/Gjh2LAwcO4JdffsGdO3fw5JNPIi8vT86yDGZqCtjba7/ntEsiInrYyXp6YseOHXr3V65cCTc3Nxw5cgSdO3eWqarqcXYGsrIYGoiI6OFXr8Y0ZGVlAQCcSlZNuo9KpYJKpdLdz87OrpO6KuPkBPzzD0MDERE9/OrN7AmNRoMJEyYgMjISrVq1KrfNnDlzYG9vr7v5+vrWcZVlcVVIIiJqLOpNaBg7dixOnTqFdevWVdhm6tSpyMrK0t1SUlLqsMLyMTQQEVFjUS9OT4wbNw5bt27F77//Dh8fnwrbKZVKKJXKOqysal26AEolEBwsdyVERES1SxJCCLleXAiB119/HXFxcdizZw+aN29erf2zs7Nhb2+PrKws2NnZ1VKVRERED5+afIbK2tMwduxYrFmzBlu2bIGtrS2uX78OALC3t4elpaWcpREREdF9ZO1pkCSp3O0rVqzAiBEjqty/vvQ0FBQA+fna6ZdEREQNQYPraZAxrxjN1q1Anz5Au3bAX3/JXQ0REVHtqTezJxoqR0ftV86eICKihx1DwwPilEsiImosGBoeUEloyMwE1GpZSyEiIqpVDA0PqOT0hBDaa1AQERE9rBgaHpC5OWBjo/2epyiIiOhhxtBgBCVTLRkaiIjoYVYvlpFu6Pr1056a4KKURET0MGNoMIIFC+SugIiIqPbx9AQREREZhKHBSAoKgNxcuasgIiKqPQwNRvDee4CVlfYrERHRw4qhwQgcHLRfb92StQwiIqJaxdBgBFxKmoiIGgOGBiNgaCAiosaAocEIGBqIiKgxYGgwAoYGIiJqDBgajKB0aNBo5K2FiIiotnBFSCNwcgKefVb79c4dQKmUuyIiIiLjY2gwAgsLYMsWuasgIiKqXTw9QURERAZhaDASIYD8fEClkrsSIiKi2sHQYCRPPglYWwObN8tdCRERUe3gmIZS1EIgPjMTqUVF8DQ3R5SDAxSSZNC+trbar5x2SUREDyuGhrti09MxPjERV0qdX/BRKrEwMBAxrq5V7s+1GoiI6GHH0xPQBoaBp0/rBQYAuKpSYeDp04hNT6/yORgaiIjoYdfoQ4NaCIxPTIQo57GSbRMSE6EW5bW4h6GBiIgedo0+NMRnZpbpYShNAEhRqRCfmVnp8zA0EBHRw67Rh4bUoiKjtGNoICKih12jHwjpaW5ulHZNmwJ9+wKtWhmjKiIiovqn0YeGKAcH+CiVuKpSlTuuQYJ2FkWUg0Olz/PII1yjgYiIHm6N/vSEQpKwMDAQgDYg3E8AmN+smcHrNRARET2sGn1oAIAYV1dsDA2F932XpyyJCZUNlCytZClpXh6biIgeRpIQVcwlrMeys7Nhb2+PrKws2NnZPfDz3b8iZEJ+PsZcuAALExOcePRRBFlZVbhvcTFgZwcUFADr1wMDBgAKxQOXREREVCtq8hnKnoZSFJKELo6OGOLuji6Ojhjt5YWejo4o1Ggw4uzZCtdqiI0FAgK0gQEAnn8e8PfXbiciInpYMDRUQpIk/Dc4GHYKBfZnZ+OzlJQybWJjgYEDgStX9LdfvardzuBAREQPC4aGKvhaWOCzuwMl30tKwpm8PN1jajUwfrx2LMP9SrZNmKBtR0RE1NAxNBjgXx4e6O3kBJUQGHH2LIrvjnSMjy/bw1CaEEBKirYdERFRQ8fQYABJkvBVcDDsFQocysnBp3dPU6SmGra/oe2IiIjqM4YGA3krlfi8eXMAwLTkZJzMzYWnp2H7GtqOiIioPmNoqIaX3N3xrLMz7tw9TdExUgMfH6CidZ8kCfD1BaKi6rZOIiKi2sDQUA2SJOHLoCA4mZriaG4uPr16GQsXljx2f1vt1wULuF4DERE9HBgaqslDqcTiu6cpZl66hKY9c7BxI+Dtrd/OxwdYtw64dYsrRBIR0cOh0V+wqiYGu7lhY3o6Ym/exPCzZ3GoX1v07WuC+HjtoEdPT+Dxx4Fu3bQzJ3JzgYkT5a6aiIjowbCnoQYkScIXQUFwMTPD33l5+PDSJSgUQJcuwJAh2q+mpsCLL2rbT50KnDkjZ8VEREQPjqGhhtzMzbH07mmK2Zcu4UhOTpk2I0cCvXsDKhUwbBhw505dV0lERGQ8DA0P4Dk3Nzzv6go1gGFnzmDn7dtYe+MG9mRkQC0EJAn4+mvA0RE4cgSYPVvuiomIiGpO1tDw+++/o0+fPvDy8oIkSdi8ebOc5dTI4ubNYadQ4ExBAZ76+28MTUhA1xMn4H/gAGLT0+HlBSxdqm07axZw+LC89RIREdWUrKEhLy8PERERWLJkiZxlPJDfs7KQXc7FJa6qVBh4+jRi09MxeDAwaJD2GhSvvVb+tSqIiIjqO1lnT/Tu3Ru9e/eWs4QHohYC4xMTy31MAJAATEhMRF8XFyxdKqG4GPjkk4oXgyIiIqrPGtSUS5VKBZVKpbufnZ0tYzVAfGYmrpSq534CQIpKhfjMTHRxdsSmTXVXGxERkbE1qIGQc+bMgb29ve7m6+sraz2pRUU1bhcfD8iceYiIiKqlQYWGqVOnIisrS3dLuXu1Sbl4mpvXqN3cucATT3DBJyIialgaVGhQKpWws7PTu8kpysEBPkolKhui4KtUIsrBQW9bx47ar998A/z4Y62VR0REZFQNKjTUNwpJwsLAQACoMDh8FBAAxX0jHzt3BiZN0n4/ciRw82YtFklERGQksoaG3NxcHD9+HMePHwcAJCUl4fjx47h8+bKcZVVLjKsrNoaGwlup1NteMsL0r3JWigSADz8EQkKAGzeA0aOB3buBtWuBPXu0UzOJiIjqG0kI+VYN2LNnD7p27Vpm+/Dhw7Fy5coq98/Ozoa9vT2ysrJkP1WhFgLxmZlILSqCp7k5ioTAU3//DRMAf7Vti7a2tmX2OXoUaNeu7FUwfXyAhQuBmJi6qZ2IiBqfmnyGyjrlskuXLpAxsxiVQpLQxdFRb9sQNzesTUvDa+fPY/8jj5Q5TZGcXP5ls69eBQYOBDZuZHAgIqL6g2MaatG8Zs1gp1DgUE4Ovrp2Te8xtRoYP778/Upy1IQJPFVBRET1B0NDLfJUKvFhQAAAYGpSEtJKrdcQHw9cuVLxvkIAKSnadkRERPUBQ0Mte83LC21sbJBZXIwpFy/qtqemGra/oe2IiIhqG0NDLTM1McEXQUGQAKy6cQN7MzMBAJ6ehu1vaDsiIqLaxtBQBzrY2WHU3U//MefP445Gg6go7SyJyi5e5esLREXVUZFERERVYGioI7ObNoWLmRnO5OfjsytXoFBop1UCFQeHiRMBhaLuaiQiIqoMQ0MdcTIzw6dNmwIAZiQn43JhIWJitNMqvb3121pYaL9++y1g4DWxiIiIah1DQx0a7uGBKHt75Gs0mJCYCEC7DkNysnZFyDVrtF8vXgRcXIATJ4AZM+StmYiIqISsK0I+qPq0IqShTuXmos2RIygWAlvDwvC0s3O57WJjgQEDABMTYN++exe5IiIiMoaafIayp6GOtbKxwUQfHwDAuAsXkF/B6k0xMcBLLwFDhgDBwXVZIRERUfkYGmTwgZ8ffJRKJBcWYvalSxW2++9/gf/9D7hvdWoiIiJZMDTIwMbUVHdJ7bkpKTiXn19uOzOze98LwYWeiIhIXgwNMunv4oJoJyfcEQJjzp3D7owMrL1xA3syMqC+b5jJrVtAv35A+/bA3bWhiIiI6hxDg0wkScKi5s1hJkn4LSsL3U6cwNCEBHQ9cQL+Bw4gNj1d19bCAjhzRnutijfekLFoIiJq1BgaZHQ8Nxd3ypm8clWlwsDTp3XBwdoaWLVKO5Piu++0MyuIiIjqGkODTNRCYPzdtRruVxIjJiQm6k5VdOoEvP22dvurrwI3btRBkURERKUwNMgkPjMTV1SqCh8XAFJUKsSXGsQwbRoQHg7cvKkNDg13hQ0iImqIGBpkkmrg+tCl2ymV2tMTZmbAli3aUxZERER1haFBJp7m5jVqFx4OzJwJBAQAdy9lQUREVCe4jLRM1ELA/8ABXFWpUNEvwMvcHJc7dYLivstgqtVAfj5ga3vvfny8dh0HT0/t5bR5dUwiIqoMl5FuQBSSpFvgqYIrY8MEwO07d8ruq7gXGGJjAT8/oGtXYOhQ7Vd/f86wICIi42NokFGMqys2hobCW6nU2+5pbg4nU1NcKSrCU3//jcxyggMAbNqkvajV1av6269eBQYOZHAgIiLj4umJekAtBOIzM5FaVARPc3NEOTggsaAAnY8dQ9qdO+hkZ4ed4eGwMTW9t49a26Nw5Ur5zylJgI8PkJTEUxVERFQWT080UApJQhdHRwxxd0cXR0coJAnBVlb4JSICjqam2J+djb6nTqGw1BUx4+MrDgyAdjpmSoq2HRERkTEwNNRj4TY22BEeDhuFAr9lZmLg6dMo0mgAGH7xKl7kioiIjIWhoZ5rb2eHrWFhsDAxwbbbt/FiQgLUQsDT07D9DW1HRERUFYaGBuAJBwfEhYbCTJKwIT0dr5w7h8jHBXx8tGMXKuLrq51+SUREZAwMDQ1EL2dnrA8JgQLAyuvXMSkpEQsWasewlhccJAlYsICDIImIyHgYGhqQ/q6uWNmiBSQAi69exaGIf7Bho4CXjwAiMoBuN4CIDPg0Edi4EYiJAW7dAj74AKhg1iYREZHBTKtuQvXJix4eyNNoMPr8eXySkoIhLVSQ1mUBRaUufmWuBJoHQghXDB4M7NqlnUWxcSPg7Cxf7URE1LCxp6EBetXLC/OaNQMArE1Lw5Ui/atlXi1SYeDp04i7mY7x47WrR+7ZA7RvD5w5I0PBRET0UGBoaKDG+/jAroIBCyWrdU1ITETvpwX279de4Oqff4COHYGffqq7OomI6OHB0NBAxWdmIrvUYk/3EwBSVCrEZ2YiNBT46y+gc2cgJwd45hlg3jztAlBqtbYXYu1a7ddKnpKIiBo5hoYGKrWoqFrtXFyAX34BXnlFGxaWLAFWr9YuRc2LXRERkSE4ELKB8jQ3N6idhcm9XGhuDixfDkREAMXFwLBh2gBRWsnFrkpmXxAREZXgBasaKLUQ8D9wAFdVKlT2C7SQJIzx9sZkX1943r2aJi92RUREvGBVI6KQJCwMDAQA3L+2U8n9ZhYWKBQC869cQcCBA3jjwgVcKSzUv9iVif4aDzARvNgV6Tt+HOjdW/uViBo1np5owGJcXbExNBTjExNxRXVv2qWPUokFgYHo7+KCn2/fxsxLl7A/OxuLrl7Fl9euISrPE3BvAgTlAOMSAbdSUzbTlMDiQCDeFfPmAR4eQIsW5b++Wq0NFqmp2mtcREWxZ+KhtGkTsGMH0K4d0Lq13NUQkYx4euIhoBYC8ZmZSC0qgqe5OaIcHKAotba0EAK/ZWZiVnIy9mZl3d0J9/qZSndVaO7enxYKxLti2TLg1VfvPqQBSoZIxMYCb0wQuOqUCTgXAbfM4X3bAZ8vkDgW4mHTujVw4oT267FjcldDREZSk89QhoZG5vfMTMxISsZvWZkVN9IAJreV6Le+I5Yvk3SrSC5eDKxYAbRqBaxKTgfGltNLsSQQm8a7VhkciooFlv6RiYsZRWjmaI4xjzvA3LSSq2+Vwh6OOnTjhra7qfR9Nzf56iEio2FoIIPsychA1xMnqmw30sMDfVxcEGJtjQALC3R5QtKOc4hKB2ac1jYqp5fCeVEobmxwrfCD/K0f0zG/KBFq53uBQ3FLiUnmgZjbx7XSmh60h+NBwopc+8pq1Spg+HD9+y+9JF89RGQ0DA1kkLU3bmBoQkK19rE0MUFzcysUXbDCWcdbgLW67AhMQBsc0pUY80dHBAVK8PPTztTw8wMcHIC3t6bjU5uKA8eU3NAKg0NsLDBgYc17OB4krMi1LyBvWFEPeh7Spk0w0aihViiAAQOhWL+u1l+b+z7c+zbUuhvivpVhaCCDGNrT0M3BATfv3MHZ/HwUVfNtYhbvijsJNkCuGZBjCuSYwlIoUPDuacCxqMLAYXJbiYvtO8LFSYK19b3LfqvVgPvAdNx6o2Y9HG/9WPOwIte+JfvXali5elV7yqEcC/fexr/e7Qu7gnzdtixLa6z8aDPGP+FU/ou6uwPe3g0yoHHfugvCDbHuhrhvVRgayCBVrfEgQTsDI6ljRygkCcUaDf4pLMTpvDwsPH0De3Gzdgs8YQfcsAQKTeDpqMArLyqQeskEX+dcBmwq6eHIMMcq59bo2tEE5iYm+GOPBI3KBOZmEvrnHoTGWVXhvooMJbKiO8LKQkKpMaQoKhaw2nIAaqfK983v27FM8n+QfYE6CivduwO//VbucwCARpJgUup/EfffL6N7d7w1fm2DC2jct272bah1N8R9DcHQQAaLTU/HwNPaN2PpN0DJ+3JjaChiXMu+GX+9lYEeJ6vupYhxdoGtqQIZxcXIKC7GraJiJOWoUGBabITqa8kVCyDXDJJGgqSRYCIkaCyLoQnMrXJX63MOsC0yh5kZ8MQT2qOYcKMIR5BR5b5RcMGjPhY49beEW2mACSRAAIebXAWsKghJAkCeAl1Sm0BhIgGS0IYdCdBAYLdbCoRl5fv2ue2PqEPb8Pqqd6DMziq3qaEEgHxre3wz8hOM7xYKYVNc4WtLuaZ4QRUAU5N7DUJDAUsrgddPJEFYV77vx02bwtlJggTg+nXgn4uAWgN8a3mxytd9Kb+pbmaRuNuuWCOw2vqfKl/3xfymML27b1AQ4OGufd2Rxy5Wue+XbZrCVCEhMwv4+6T2gWIhsNqqip83zxRDCv2hkEqFWUlb81plcpWv+4IqAKaQ4OUFNA3QHrO8fIHxZ/6p8lh9/UgzCI2EfX/ee0gtNPjOuup9hxU0hamkvwyQizMQHCzwbwOO1+vmTdEmXILQbsK+fYCqWGCNTRW/p3xTvFDkD02xBEBo95e0fyhtsEyu+LTq3X8TA/P9YHL3QFtaSGjTBihWC0w5lQxRyb9FKc8UC8L9YabQNjh+HMjJlqAWAt/bJQFV/I4HZPnDRJJ0/y8WAtAIDWIdKqm5ij86DNFgQ8OSJUvw6aef4vr164iIiMCiRYvQvn37KvdjaHgwsenpZdZ48L27xkN5gQHQ/uNz/+0Abkmq8pcG0wDOQokb3TrqTfsEgAV7MjARVQeOAcIbjzRRIqNAg1y1GiaWavxxLRd/I7vKfc2g/Z9rdU+nNGauGRn44rPPMCA+vuqehPuUtN8UFYXXJk5EuqNjLVZKRPf7DBGY0KVm/+4aZGhYv349hg0bhmXLlqFDhw5YsGABNmzYgHPnzsGtiqldDA0Prqo1HsoTm56OAadOa9N56eBwt8tsU6vyeyl03fWOFQeOipKzoT0cu8Ii0N3ZEUII3BECRRoNFvyRifdxqsp9hxc2RY9Aa6iKBVTFAoXFGvyWlIttjilV7tv2ihda2FpCMrm3/tHei/n4EalV7tsDbmjrq8TJM8CtDAENBJI1eUj3z6xyX4dL9vDQWN67hogGuKEoQGZAVpX72l62g2OxEhFtBExMBTr9/DNemzETVvn5MNVUfbnTYhMT5FpYY8qI9/BDu16AAHLMVCjwyatyX8tLtrAtVOruh4cD/+So8I95TpX7euXaoE0TcwgAaenApUtAjlKFQu+qX9fiqjVsVcp7f7gJIMe8CAW+VfcmWaZYw7ZICUjawb3OLsDp6ypcNq/6dZsUWaOVhxK5ucDZBG1PUo5SZdjrJtvCTnX3WN3tHsm2KESBf9XHyvKyDWxVSnh4Ak38tGu2nLimwhWLqmv2LbJGsIsSJ07c+0M3x1yFAkOO85W7x6oUe0cBlXURUgw4Xnbp1ng8WLu/BODoUQmZCsOOl02KLWzzLXT7QgBZFirk+Vf9R4d1kh0cVNp9La2ARx8FjlwpxAXzqvdtVmSLCC9tzadOA7m5QJa5Cnm+Vf+erFNs4VBU8r7U/jfDrBB5Tared1xGSyzq715lu/I0yNDQoUMHtGvXDosXLwYAaDQa+Pr64vXXX8c777xT6b4MDfKJTU/H+AuJuFKkvxLlwkp6KYBS5+gqCBwVnaN7kB6OBwkrcu1raK9MeX9lPMi+SEtDUp8X4P/XrkpPVwgAye17IODH1XrrNshVN/et//vK+dqNbV9DNbhrTxQVFeHIkSPo0aOHbpuJiQl69OiB/fv3l2mvUqmQnZ2tdyN5xLi6IrlTR+yOiMCali2xOyICyR07VhoYAGBuH1dMyQ2FIkP/rxBFhrLSQT0KScLy8EBtCNfc9+DdwLE8PLDcXhJzUwmTzCvfd5J5YLnnBeXad8zjDlDcUpbdr9T+iltKjHncwaj7ws0NTXp0gNqk8tWy1CYKNOnZscxCT3LVzX3r/74Nte6GuG9tkjU03Lx5E2q1Gu7u+l0r7u7uuH79epn2c+bMgb29ve7m6+tbV6VSORSShC6Ojhji7o4ujo5VntYoMbePK/L7dsRniMC4jJb4DBHI79uxylHAMa6u2NQqFD4W+oHDx1JZ4SmR0q9Zk7Ai175yhRUAUGzbCkUVpycUGjUU27bWm7q5b/3ft6HW3RD3rU2ynp64du0avL298eeff6JTp0667W+99Rb27t2LgwcP6rVXqVRQlRq0l52dDV9fX56eaGRqMg6jRENbnKXO53dfv65dm7uUksGO5Q6SvH5duzaD3HVz3wazb0OtuyHuW5UGN6ahqKgIVlZW2LhxI/r166fbPnz4cGRmZmLLli2V7s8xDdQY1GlY+fZbYMQI3V2hUEBlaYPd0f9G15/+C2VBLiS1Wr/9sGHy1819G9S+DbXuhrhvZRpcaAC0AyHbt2+PRYsWAdAOhGzSpAnGjRvHgZBEde3554GNG7UTxYUA+vcHli3Tjl1ISwNGjwbi4gBJ0t6eew5YZ9iy0kRUvzS4gZAAMGnSJHz11Vf49ttvkZCQgNdeew15eXn417/+JXdpRI1LcTGwY4f2Guj29sD69doLfpQMdnRz095fv177uEYDbN+uXeObiBoFU7kLeP7555Geno4PPvgA169fR+vWrbFjx44ygyOJqJYVFABNmwIBAfd6F8ozaBDQpYu21yE5GcjPB2xt67JSIpKJ7KcnHgRPTxAZmVqNCq9pboz2RFRvNMjTE0RUj1Q3ADAwEDUqDA1ERERkEIYGIiIiMojsAyEfRMlwDC4nTUREVD0ln53VGdrYoENDTo72CmBcTpqIiKhmcnJyYG9vb1DbBj17QqPR4Nq1a7C1tYVUahnhkuWlU1JSOKvCADxehuOxqh4eL8PxWFUPj5fhKjpWQgjk5OTAy8sLJiaGjVZo0D0NJiYm8PHxqfBxOzs7vpmqgcfLcDxW1cPjZTgeq+rh8TJcecfK0B6GEhwISURERAZhaCAiIiKDPJShQalUYtq0aVAqlXKX0iDweBmOx6p6eLwMx2NVPTxehjPmsWrQAyGJiIio7jyUPQ1ERERkfAwNREREZBCGBiIiIjIIQwMREREZ5KEMDUuWLIG/vz8sLCzQoUMH/PXXX3KXVO9Mnz4dkiTp3Vq0aCF3WfXG77//jj59+sDLywuSJGHz5s16jwsh8MEHH8DT0xOWlpbo0aMHLly4IE+xMqvqWI0YMaLMe61Xr17yFCuzOXPmoF27drC1tYWbmxv69euHc+fO6bUpLCzE2LFj4ezsDBsbGwwYMAA3btyQqWJ5GXK8unTpUub9NXr0aJkqls8XX3yB8PBw3QJOnTp1wvbt23WPG+t99dCFhvXr12PSpEmYNm0ajh49ioiICDz11FNIS0uTu7R6JzQ0FKmpqbrbH3/8IXdJ9UZeXh4iIiKwZMmSch+fO3cuPv/8cyxbtgwHDx6EtbU1nnrqKRQWFtZxpfKr6lgBQK9evfTea2vXrq3DCuuPvXv3YuzYsThw4AB++eUX3LlzB08++STy8vJ0bSZOnIgff/wRGzZswN69e3Ht2jXExMTIWLV8DDleADBy5Ei999fcuXNlqlg+Pj4++Pjjj3HkyBEcPnwY3bp1Q9++fXH69GkARnxfiYdM+/btxdixY3X31Wq18PLyEnPmzJGxqvpn2rRpIiIiQu4yGgQAIi4uTndfo9EIDw8P8emnn+q2ZWZmCqVSKdauXStDhfXH/cdKCCGGDx8u+vbtK0s99V1aWpoAIPbu3SuE0L6PzMzMxIYNG3RtEhISBACxf/9+ucqsN+4/XkII8cQTT4jx48fLV1Q95ujoKL7++mujvq8eqp6GoqIiHDlyBD169NBtMzExQY8ePbB//34ZK6ufLly4AC8vLzRt2hQvvPACLl++LHdJDUJSUhKuX7+u9z6zt7dHhw4d+D6rwJ49e+Dm5obg4GC89tpruHXrltwl1QtZWVkAACcnJwDAkSNHcOfOHb33VosWLdCkSRO+t1D2eJVYvXo1XFxc0KpVK0ydOhX5+flylFdvqNVqrFu3Dnl5eejUqZNR31cN+oJV97t58ybUajXc3d31tru7u+Ps2bMyVVU/dejQAStXrkRwcDBSU1MxY8YMREVF4dSpU7C1tZW7vHrt+vXrAFDu+6zkMbqnV69eiImJQUBAAC5evIj/+7//Q+/evbF//34oFAq5y5ONRqPBhAkTEBkZiVatWgHQvrfMzc3h4OCg15bvrfKPFwAMHToUfn5+8PLywt9//423334b586dQ2xsrIzVyuPkyZPo1KkTCgsLYWNjg7i4OISEhOD48eNGe189VKGBDNe7d2/d9+Hh4ejQoQP8/Pzw/fff49///reMldHDZvDgwbrvw8LCEB4ejmbNmmHPnj3o3r27jJXJa+zYsTh16hTHEhmoouM1atQo3fdhYWHw9PRE9+7dcfHiRTRr1qyuy5RVcHAwjh8/jqysLGzcuBHDhw/H3r17jfoaD9XpCRcXFygUijIjQm/cuAEPDw+ZqmoYHBwcEBQUhMTERLlLqfdK3kt8n9VM06ZN4eLi0qjfa+PGjcPWrVuxe/du+Pj46LZ7eHigqKgImZmZeu0b+3urouNVng4dOgBAo3x/mZubIzAwEG3btsWcOXMQERGBhQsXGvV99VCFBnNzc7Rt2xa//vqrbptGo8Gvv/6KTp06yVhZ/Zebm4uLFy/C09NT7lLqvYCAAHh4eOi9z7Kzs3Hw4EG+zwxw5coV3Lp1q1G+14QQGDduHOLi4vDbb78hICBA7/G2bdvCzMxM77117tw5XL58uVG+t6o6XuU5fvw4ADTK99f9NBoNVCqVcd9Xxh2rKb9169YJpVIpVq5cKc6cOSNGjRolHBwcxPXr1+UurV558803xZ49e0RSUpLYt2+f6NGjh3BxcRFpaWlyl1Yv5OTkiGPHjoljx44JAGL+/Pni2LFj4tKlS0IIIT7++GPh4OAgtmzZIv7++2/Rt29fERAQIAoKCmSuvO5VdqxycnLE5MmTxf79+0VSUpLYtWuXeOSRR0Tz5s1FYWGh3KXXuddee03Y29uLPXv2iNTUVN0tPz9f12b06NGiSZMm4rfffhOHDx8WnTp1Ep06dZKxavlUdbwSExPFzJkzxeHDh0VSUpLYsmWLaNq0qejcubPMlde9d955R+zdu1ckJSWJv//+W7zzzjtCkiSxc+dOIYTx3lcPXWgQQohFixaJJk2aCHNzc9G+fXtx4MABuUuqd55//nnh6ekpzM3Nhbe3t3j++edFYmKi3GXVG7t37xYAytyGDx8uhNBOu3z//feFu7u7UCqVonv37uLcuXPyFi2Tyo5Vfn6+ePLJJ4Wrq6swMzMTfn5+YuTIkY02xJd3nACIFStW6NoUFBSIMWPGCEdHR2FlZSX69+8vUlNT5StaRlUdr8uXL4vOnTsLJycnoVQqRWBgoJgyZYrIysqSt3AZvPzyy8LPz0+Ym5sLV1dX0b17d11gEMJ47yteGpuIiIgM8lCNaSAiIqLaw9BAREREBmFoICIiIoMwNBAREZFBGBqIiIjIIAwNREREZBCGBiIiIjIIQwMREREZhKGBiOqVPXv2QJKkMhfXISL5MTQQERGRQRgaiIiIyCAMDUSkR6PRYM6cOQgICIClpSUiIiKwceNGAPdOHWzbtg3h4eGwsLBAx44dcerUKb3n2LRpE0JDQ6FUKuHv74958+bpPa5SqfD222/D19cXSqUSgYGB+O9//6vX5siRI3j00UdhZWWFxx57DOfOnavdH5yIqsTQQER65syZg1WrVmHZsmU4ffo0Jk6ciBdffBF79+7VtZkyZQrmzZuHQ4cOwdXVFX369MGdO3cAaD/sBw0ahMGDB+PkyZOYPn063n//faxcuVK3/7Bhw7B27Vp8/vnnSEhIwJdffgkbGxu9Ot59913MmzcPhw8fhqmpKV5++eU6+fmJqBLGuzAnETV0hYWFwsrKSvz555962//973+LIUOG6C6DvW7dOt1jt27dEpaWlmL9+vVCCCGGDh0qevbsqbf/lClTREhIiBBCiHPnzgkA4pdffim3hpLX2LVrl27btm3bBABRUFBglJ+TiGqGPQ1EpJOYmIj8/Hz07NkTNjY2utuqVatw8eJFXbtOnTrpvndyckJwcDASEhIAAAkJCYiMjNR73sjISFy4cAFqtRrHjx+HQqHAE088UWkt4eHhuu89PT0BAGlpaQ/8MxJRzZnKXQAR1R+5ubkAgG3btsHb21vvMaVSqRccasrS0tKgdmZmZrrvJUkCoB1vQUTyYU8DEemEhIRAqVTi8uXLCAwM1Lv5+vrq2h04cED3fUZGBs6fP4+WLVsCAFq2bIl9+/bpPe++ffsQFBQEhUKBsLAwaDQavTESRNQwsKeBiHRsbW0xefJkTJw4ERqNBo8//jiysrKwb98+2NnZwc/PDwAwc+ZMODs7w93dHe+++y5cXFzQr18/AMCbb76Jdu3aYdasWXj++eexf/9+LF68GEuXLgUA+Pv7Y/jw4Xj55Zfx+eefIyIiApcuXUJaWhoGDRok149ORIaQe1AFEdUvGo1GLFiwQAQHBwszMzPh6uoqnnrqKbF3717dIMUff/xRhIaGCnNzc9G+fXtx4sQJvefYuHGjCAkJEWZmZqJJkybi008/1Xu8oKBATJw4UXh6egpzc3MRGBgovvnmGyHEvYGQGRkZuvbHjh0TAERSUlJt//hEVAlJCCFkzi1E1EDs2bMHXbt2RUZGBhwcHOQuh4jqGMc0EBERkUEYGoiIiMggPD1BREREBmFPAxERERmEoYGIiIgMwtBAREREBmFoICIiIoMwNBAREZFBGBqIiIjIIAwNREREZBCGBiIiIjLI/wNQVwifb3OfTgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='29' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      29.00% [29/100] [01:23<03:24]\n",
       "      <br>\n",
       "      ████████████████████100.00% [4/4] [val_loss=0.0600]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_loss without improvement in 15 epoch,early stopping >>>>>> \n",
      "\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>lr</th>\n",
       "      <th>val_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>5.020751</td>\n",
       "      <td>0.006</td>\n",
       "      <td>1.396625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1.223462</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.998791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.833068</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.560698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.442632</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.279621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.220685</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.148428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0.127668</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.103782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0.094795</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.086212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0.080554</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.074823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>0.072376</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.068177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.066672</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.063795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.063332</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.063389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>0.061814</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.060303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>0.060272</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.060948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>0.059974</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.057997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>0.058828</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>0.058756</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>0.059066</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.058603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>0.059152</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>0.059139</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.058668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>0.059002</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>0.058597</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>0.059410</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.058641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>23</td>\n",
       "      <td>0.059024</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>24</td>\n",
       "      <td>0.059021</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.058638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>25</td>\n",
       "      <td>0.059263</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>26</td>\n",
       "      <td>0.059116</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.060154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>27</td>\n",
       "      <td>0.059485</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28</td>\n",
       "      <td>0.059610</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>0.059937</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.059962</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    epoch  train_loss     lr  val_loss\n",
       "0       1    5.020751  0.006  1.396625\n",
       "1       2    1.223462  0.006  0.998791\n",
       "2       3    0.833068  0.006  0.560698\n",
       "3       4    0.442632  0.006  0.279621\n",
       "4       5    0.220685  0.006  0.148428\n",
       "5       6    0.127668  0.006  0.103782\n",
       "6       7    0.094795  0.006  0.086212\n",
       "7       8    0.080554  0.006  0.074823\n",
       "8       9    0.072376  0.006  0.068177\n",
       "9      10    0.066672  0.006  0.063795\n",
       "10     11    0.063332  0.006  0.063389\n",
       "11     12    0.061814  0.006  0.060303\n",
       "12     13    0.060272  0.006  0.060948\n",
       "13     14    0.059974  0.006  0.057997\n",
       "14     15    0.058828  0.006  0.059398\n",
       "15     16    0.058756  0.006  0.059474\n",
       "16     17    0.059066  0.006  0.058603\n",
       "17     18    0.059152  0.006  0.059338\n",
       "18     19    0.059139  0.006  0.058668\n",
       "19     20    0.059002  0.006  0.059513\n",
       "20     21    0.058597  0.006  0.059736\n",
       "21     22    0.059410  0.006  0.058641\n",
       "22     23    0.059024  0.006  0.059056\n",
       "23     24    0.059021  0.006  0.058638\n",
       "24     25    0.059263  0.006  0.059357\n",
       "25     26    0.059116  0.006  0.060154\n",
       "26     27    0.059485  0.006  0.059852\n",
       "27     28    0.059610  0.006  0.059208\n",
       "28     29    0.059937  0.006  0.059962"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# keras_model.load_ckpt(ckpt_path) #支持加载微调后的权重继续训练(断点续训)\n",
    "keras_model.fit(train_data = dl_train,\n",
    "                val_data = dl_val,\n",
    "                epochs=100,patience=15,\n",
    "                monitor='val_loss',mode='min',\n",
    "                ckpt_path = ckpt_path\n",
    "               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2109a2d-c2d9-472f-bf44-42686177b32e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4988c80-b699-45e5-aceb-84046d9a592d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "49763c6a-6ee5-448c-b873-48825a6fba67",
   "metadata": {},
   "source": [
    "## 四，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f98c4e5-dbb0-456e-bc4b-1c8dfccbd499",
   "metadata": {},
   "source": [
    "为减少GPU压力，此处可重启kernel释放显存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ee686c58-42e2-463a-a429-3fe16dee5a7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "142a917e-9db0-4d5a-8352-cb250561f47d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to \"AutoModelForCausalLM.from_pretrained\".\n",
      "Try importing flash-attention for faster inference...\n",
      "Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary\n",
      "Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm\n",
      "Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention\n",
      "Loading checkpoint shards: 100%|██████████| 8/8 [00:06<00:00,  1.18it/s]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "#使用QLoRA引入的 NF4量化数据类型以节约显存\n",
    "model_name_or_path ='qwen_7b'\n",
    "ckpt_path = 'qwen7b_multirounds'\n",
    "\n",
    "\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2a8b409f-f100-48c5-9794-62cb5bdf19d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import PeftModel\n",
    "\n",
    "#可能需要5分钟左右\n",
    "peft_model = PeftModel.from_pretrained(model, ckpt_path)\n",
    "model_new = peft_model.merge_and_unload()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2c060b4a-7731-4ec7-8699-9f47088f4a2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.generation.utils import GenerationConfig\n",
    "model_new.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ce165cd8-987d-47be-95ac-83cf7ef479b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = 'qwen_torchkeras'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9214d354-811b-4e47-96ac-682ed5e68ee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.save_pretrained(save_path)\n",
    "model_new.save_pretrained(save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a7950603-e844-4d60-990d-f86245fbaa26",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp qwen_7b/*.py  qwen_torchkeras/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e242ae9b-f79f-44eb-8c39-57d7423e8cf0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "config.json\t\t\t  pytorch_model.bin.index.json\n",
      "configuration_qwen.py\t\t  qwen.tiktoken\n",
      "generation_config.json\t\t  qwen_generation_utils.py\n",
      "modeling_qwen.py\t\t  special_tokens_map.json\n",
      "pytorch_model-00001-of-00002.bin  tokenization_qwen.py\n",
      "pytorch_model-00002-of-00002.bin  tokenizer_config.json\n"
     ]
    }
   ],
   "source": [
    "!ls  qwen_torchkeras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fcfb0f3-bfc3-43cd-944e-d5b0fe785a84",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c92003-2d55-416a-b9f1-f58b5fb29261",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "81582c3e-28d3-4847-a84c-dc1b53aa0839",
   "metadata": {},
   "source": [
    "## 五，使用模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c57c2f-0e1a-4e77-b4e4-fefa210b1bb1",
   "metadata": {},
   "source": [
    "为减少GPU压力，此处可再次重启kernel释放显存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "89fc27d5-651e-437c-ab1c-adacb487a614",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "29b638ed-c993-441c-b832-3bb629b88838",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning: import flash_attn rotary fail, please install FlashAttention rotary to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary\n",
      "Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get higher efficiency https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm\n",
      "Warning: import flash_attn fail, please install FlashAttention to get higher efficiency https://github.com/Dao-AILab/flash-attention\n",
      "Loading checkpoint shards: 100%|██████████| 2/2 [00:09<00:00,  4.54s/it]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "model_name_or_path =  'qwen_torchkeras'\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=False, trust_remote_code=True)\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map=\"auto\", \n",
    "                                             torch_dtype=torch.float16, trust_remote_code=True)\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5333f116-5bc2-46d9-ba39-54d9e14b9ec7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e63d6988-4cfd-46ba-8a5d-f11e5d183483",
   "metadata": {},
   "source": [
    "我们测试一下微调后的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c5a5cbf0-3e23-4a1f-9e93-34ba7ed6e3da",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "register magic %%chat sucessed ...\n"
     ]
    }
   ],
   "source": [
    "from torchkeras.chat import ChatLLM \n",
    "llm = ChatLLM(model,tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8781ff0b-559a-4aaa-b914-3854618f7551",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你好，请介绍一下你自己"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8d09d03b-5743-45b3-8cac-1ba4422f2fc2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你能干嘛呀"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "821c003b-3ba8-49ae-8752-51210c0657db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你多大了呀"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0136dfea-ab5a-4c26-9176-388149866154",
   "metadata": {},
   "source": [
    "非常棒，粗浅的测试表明，我们的多轮对话训练是成功的。已经在Qwen的自我认知中，种下了一颗梦中情炉的种子。😋😋"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e6f0591-595f-44a6-89a8-7e108ca49ad4",
   "metadata": {},
   "source": [
    "**如果本项目对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果在torchkeras的使用中遇到问题，可以在项目中提交issue。\n",
    "\n",
    "如果想要获得更快的反馈或者与其他torchkeras用户小伙伴进行交流，\n",
    "\n",
    "可以在公众号算法美食屋后台回复关键字：**加群**。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e6e40b3-4cf8-4e7e-8f21-9b3487adc327",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}