Summarization_LongContext_Finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0x-eWpGI7eDB"
   },
   "source": [
    "# Long Context Fine-tuning for Summarization\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/togethercomputer/together-cookbook/blob/main/Summarization_LongContext_Finetuning.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_Y7WTebZ7eDC"
   },
   "source": [
    "## Introduction\n",
    "\n",
    "*This cookbook is part of a technical deep dive blogpost on long context finetuning you can read [here](https://www.together.ai/blog/long-context-fine-tuning-a-technical-deep-dive).*\n",
    "\n",
    "An important use-case for LLMs is summarization, where we can pass in large source documents into the context window of the LLM and prompt it to generate comprehensive summaries.\n",
    "\n",
    "For most LLMs this works quite well for source documents below cumulative size of 32,000 tokens, however depending on the LLM and how much long context post-training it has had summarization performance of longer documents can perform quite poorly.\n",
    "\n",
    "In this notebook we will:\n",
    "1. Discuss a dataset to improve long context summarization.\n",
    "2. Evaluate how well an LLM summarizes long documents.\n",
    "3. Fine-tune the LLM to improve summarization capabilities.\n",
    "4. Compare un-tuned vs. fine-tuned models for summarization.\n",
    "\n",
    "<img src=\"images/summary_task.png\" height=\"500\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jsu14fLz7eDD"
   },
   "source": [
    "## Install Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q together==1.3.4 rouge_score==0.1.2 evaluate orjson seaborn matplotlib numpy tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from tqdm.auto import tqdm\n",
    "from together import Together\n",
    "\n",
    "import orjson\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "from evaluate import load\n",
    "\n",
    "TOGETHER_API_KEY = os.getenv(\"TOGETHER_API_KEY\")\n",
    "#WANDB_API_KEY = os.getenv(\"WANDB_API_KEY\")\n",
    "\n",
    "client = Together(api_key = TOGETHER_API_KEY)\n",
    "\n",
    "def llm_call(query, model=\"meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\"):\n",
    "    response = client.chat.completions.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "          {\"role\": \"system\", \"content\": \"You are a helpful chatbot.\"},\n",
    "          {\"role\": \"user\", \"content\": query},\n",
    "        ],\n",
    "        temperature=1.0,\n",
    "        seed=42,\n",
    "        max_tokens=1200,\n",
    "    )\n",
    "    result = response.choices[0].message.content\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "36BaPYUj7eDE"
   },
   "source": [
    "## Dataset for Summarization Finetuning\n",
    "\n",
    "We fine-tuned a Llama 3.1 8B baseline model on multiple datasets:\n",
    "- Synthetic dataset: generated using Llama 3.1 405B\n",
    "- [GovReport](https://gov-report-data.github.io/): Consists of reports + summaries written by government research agencies\n",
    "\n",
    "The goal of these datasets was to train the model to generate coherent and comprehensive summaries of long documents ranging from `32,000` to `128,000` tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3zCTfQwV7eDE"
   },
   "source": [
    "## Finetune Llama 3.1 8B for Summarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0TMdMZWM7eDE"
   },
   "outputs": [],
   "source": [
    "# We used the function to launch fine-tuning jobs on the Together platform\n",
    "\n",
    "def send_ft_job(client,\n",
    "                model=\"meta-llama/Meta-Llama-3.1-8B-32k-Instruct-Reference\",\n",
    "                n_epochs=4,\n",
    "                run_name='1113-summarization-long-context-finetune',\n",
    "                train_on_inputs=False,\n",
    "                learning_rate=6e-5,\n",
    "                filename=None,\n",
    "                summarization_file_id=None):\n",
    "    \"\"\"\n",
    "    Sends a fine-tuning job to the client.\n",
    "    Parameters:\n",
    "        client (object): The client object to interact with the API.\n",
    "        model (str): The model to be fine-tuned. Default is \"meta-llama/Meta-Llama-3.1-8B-32k-Instruct-Reference\".\n",
    "        n_epochs (int): Number of epochs for fine-tuning. Default is 4.\n",
    "        run_name (str): The name for the run. Default is '1113-summarization-long-context-finetune'.\n",
    "        train_on_inputs (bool): Whether to train on inputs. Default is False.\n",
    "        learning_rate (float): The learning rate for fine-tuning. Default is 6e-5.\n",
    "        filename (str, optional): The filename to upload for fine-tuning. Default is None.\n",
    "        summarization_file_id (str, optional): The file ID for summarization. Must be provided if filename is None.\n",
    "    Returns:\n",
    "        str: The ID of the fine-tuning job.\n",
    "    Raises:\n",
    "        AssertionError: If neither filename nor summarization_file_id is provided.\n",
    "    \"\"\"\n",
    "    if filename:\n",
    "        response = client.files.upload(filename, check=True)\n",
    "        summarization_file_id = response.id\n",
    "    else:\n",
    "        assert summarization_file_id is not None, \"provide summarization_file_id\"\n",
    "\n",
    "    response = client.fine_tuning.create(\n",
    "        training_file = summarization_file_id,\n",
    "        model = model,\n",
    "        n_epochs = n_epochs,\n",
    "        n_checkpoints = 1,\n",
    "        batch_size = \"max\",\n",
    "        learning_rate = learning_rate,\n",
    "        warmup_ratio = 0.05,\n",
    "        suffix=run_name,\n",
    "        wandb_api_key = WANDB_API_KEY,\n",
    "        lora=True,\n",
    "        lora_r=32,\n",
    "        lora_alpha=64,\n",
    "        lora_dropout=0.05,\n",
    "        train_on_inputs=train_on_inputs,\n",
    "    )\n",
    "    summarization_fine_tuning_job_id = response.id\n",
    "    return summarization_fine_tuning_job_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XjqOBuLX7eDF"
   },
   "outputs": [],
   "source": [
    "# Example fine-tuning job on the synthetic summarization dataset\n",
    "\n",
    "summarization_fine_tuning_job_id = send_ft_job(\n",
    "    client,\n",
    "    model=\"meta-llama/Meta-Llama-3.1-8B-32k-Instruct-Reference\",\n",
    "    learning_rate=7e-5,\n",
    "    n_epochs=4,\n",
    "    filename=\"1112_summarization_train.jsonl\",\n",
    "    run_name=\"1112-summarization-synthetic-no-loss-masking\",\n",
    "    summarization_file_id=None,\n",
    "    train_on_inputs=True,\n",
    ")\n",
    "\n",
    "# Example fine-tuning job on GovReport dataset\n",
    "\n",
    "summarization_fine_tuning_job_id = send_ft_job(\n",
    "    client,\n",
    "    model=\"meta-llama/Meta-Llama-3.1-8B-32k-Instruct-Reference\",\n",
    "    n_epochs=2,\n",
    "    filename=\"1114_govreport.jsonl\",\n",
    "    run_name=\"1114-govreport-2ep\",\n",
    "    summarization_file_id=None,\n",
    "    train_on_inputs=False,\n",
    "    learning_rate=4e-5,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pMBxSeuZ7eDF"
   },
   "source": [
    "## Deploy Fine-tuned Models on a Dedicated Endpoint(DE)\n",
    "\n",
    "Before we can run the evaluations we need to deploy our finetuned model as a Dedicated Endpoint(DE). After fine-tuning completes, access your model through the Together AI dashboard. Go to Models, select your fine-tuned model, and select Deploy. Choose from the available hardware options - we'll use a single A100-80GB GPU for this example.\n",
    "\n",
    "<img src=\"images/deploy_CFT.png\" height=\"650\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EsqM4ear7eDF"
   },
   "source": [
    "Once the models are deployed we can use the code below to generate predictions using the dedicated endpoints along with normal serverless inference for the baseline Llama 3.1 8B model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1mVzGOP-7eDF"
   },
   "outputs": [],
   "source": [
    "def get_predictions(test_items, model):\n",
    "    predictions = []\n",
    "    for it in tqdm(test_items):\n",
    "        predictions.append(\n",
    "            llm_call(it['prompt'],\n",
    "                     model=model\n",
    "                    )\n",
    "        )\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gS1sNOr77eDF"
   },
   "source": [
    "## Evaluate Summarization\n",
    "\n",
    "We will now compare and score the summarization capabilities of a baseline Llama 3.1 8B vs fine-tunes thereof."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9C8vv11u7eDF"
   },
   "source": [
    "We have provided pre-generated summaries from fine-tuned and baseline Llama 3.1 8B models to simplify comparision below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "yBp3TB3x7eDF"
   },
   "outputs": [],
   "source": [
    "predictions = orjson.loads(Path(\"datasets/predictions.json\").read_bytes())\n",
    "\n",
    "predictions_baseline_syn = predictions['predictions_baseline_syn']\n",
    "predictions_baseline_govreport = predictions['predictions_baseline_govreport']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "CQpd9iiv7eDG"
   },
   "outputs": [],
   "source": [
    "predictions_ft = orjson.loads(Path(\"datasets/predictions_ft.json\").read_bytes())\n",
    "\n",
    "predictions_ft_syn = predictions_ft['predictions_ft_syn']\n",
    "predictions_ft_govreport = predictions_ft['predictions_ft_govreport']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_items = orjson.loads(Path(\"datasets/synth_summarization_test.json\").read_bytes())\n",
    "\n",
    "test_items_govreport = [orjson.loads(line) for line in Path(\"datasets/govreport_test.jsonl\").read_text().split(\"\\n\")]\n",
    "test_items_govreport = test_items_govreport[:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "YRPYgV9x7eDG"
   },
   "outputs": [],
   "source": [
    "BASELINE_MODEL = \"meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J9WA-Xe17eDG"
   },
   "source": [
    "We will use ROUGE scores to evaluate between the baseline and fine-tuned Llama 3.1 8B models.\n",
    "\n",
    "**ROUGE (Recall-Oriented Understudy for Gisting Evaluation)** is a set of metrics specifically designed to evaluate automatic summarization and machine translation systems by comparing generated text against reference summaries. Unlike BLEU, which focuses on precision, ROUGE emphasizes recall - measuring how much of the reference summary is captured in the generated text. The most commonly used variants are ROUGE-N (measuring n-gram overlap), ROUGE-L (looking at longest common subsequences), and ROUGE-S (considering skip-grams). For example, if a human-written reference summary contains key phrases that also appear in the machine-generated summary, ROUGE will assign higher scores. Scores range from 0 to 1, with 1 indicating perfect overlap. ROUGE has become an industry standard because it correlates well with human judgments, though like BLEU, it has limitations - it can't fully capture semantic meaning or summary coherence, and relies heavily on exact word matches. This is why practitioners often report multiple ROUGE variants alongside other metrics for a more complete evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 113,
     "referenced_widgets": [
      "4a36ae38343e4069a9594b7903639ce3",
      "5c529de45a3f4b68ba9707c2eef6a281",
      "a91f784b638b40bfa2a6f381a4ed37cb",
      "b511ddcee19f49fb84d9f31c73adced1",
      "73c2ece324b74007b01613265ba3fd9d",
      "f1bf576faf684e11a3d5a519066366a6",
      "2448da9c76a74e14961cafa0f62a9767",
      "4a90ce0718cc4f8ba7ea95a42538a0e5",
      "aeb35b06e2c243cd8d020186564a14f8",
      "a2c34b4d40964aefbbb740c6238ffd33",
      "380b83efe9f64a9295626a0a66b421ae",
      "b47f018868264f0489d048be0ff073eb",
      "4d85d232158d4939b3b3a72dea7c2fcd",
      "35ac3e449986435cbb4864f18a1d363e",
      "3322b6bd9cad427c8693985d18b3bb35",
      "352d342144f141969b12ee0e31d74d14",
      "e582366a88e2445bb6888862563380c5",
      "07c8b2cc25f34f3f94104669036494c6",
      "2362664519b9415d92237a85ccf69d62",
      "75b858dda5234f3f8b184e8eba864cd2",
      "4c114940a70c4393b53e2465b71ab703",
      "8d8cc5b12af94851bab69f7353e19fa4",
      "5036ee4a59574ff7bf6a0d39ac4dc9b6",
      "03687a6cd7064124b83ee3e9ecec997d",
      "e90f6ca90f504fce8373fef4e0a30983",
      "94cbe82d40e34ca29a53e3c4ce5fc950",
      "f88ff86040c9438e9df8a50e190120bd",
      "38906354ac914e3f96d8299576418587",
      "70494fe5181a47f7ab693bef48cd0f5d",
      "5da475c2eaa34783ba78ad5d6dc79dae",
      "3f79c39f611c4f89b6569271f0d9b722",
      "0fe7324e2f674af9bc185e3581a7bd2f",
      "17d58dd7c32a4f8da74ac08df38742cf"
     ]
    },
    "id": "qdrhbjWL7eDG",
    "outputId": "6dd8f068-b6c9-4191-ab52-1d813e5a83cc"
   },
   "outputs": [],
   "source": [
    "# Load in the ROUGE metrics from the evaluate HuggingFace library\n",
    "\n",
    "rouge = load(\"rouge\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "tYNsND9-7eDG"
   },
   "outputs": [],
   "source": [
    "def compute_rouge_l(predictions, test_items):\n",
    "    \"\"\"\n",
    "    Computes ROUGE-L scores for a list of predictions against a list of test items.\n",
    "    Args:\n",
    "        predictions (list of str): A list of predicted text strings.\n",
    "        test_items (list of dict): A list of dictionaries, where each dictionary contains a key 'completion' \n",
    "                                   with the reference text string.\n",
    "    Returns:\n",
    "        list of float: A list of ROUGE-L scores, one for each prediction, scaled by 100.\n",
    "    \"\"\"\n",
    "    scores = rouge.compute(predictions=predictions, references=[it['completion'] for it in test_items],\n",
    "                           rouge_types=['rougeL'], use_aggregator=False)['rougeL']\n",
    "    scores = np.array(scores) * 100\n",
    "    return scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a4382385-1bd9-4a36-9a23-475de2ab96b5",
    "outputId": "d3dfc6f0-67e1-489d-c9ff-77f76eafd3be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Synthetic Baseline: ROUGE: 21.56\n",
      "Synthetic Fine-tune: ROUGE: 23.55\n",
      "GovReport Baseline: ROUGE: 21.66\n",
      "GovReport Fine-tune: ROUGE: 24.05\n"
     ]
    }
   ],
   "source": [
    "for k, (predictions_, test_items_) in zip(\n",
    "    [\"Synthetic Baseline:\", \"Synthetic Fine-tune:\", \"GovReport Baseline:\", \"GovReport Fine-tune:\"],\n",
    "    [(predictions_baseline_syn, test_items), (predictions_ft_syn, test_items),\n",
    "     (predictions_baseline_govreport, test_items_govreport), (predictions_ft_govreport, test_items_govreport),]\n",
    "):\n",
    "\n",
    "    rouge_scores = compute_rouge_l(predictions_, test_items_)\n",
    "    print(k, f\"ROUGE: {np.mean(rouge_scores):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "687c68d1-20c3-49d0-8444-344f8ab7ce69"
   },
   "source": [
    "As you can see, fine-tuned model results exhibit better performance in all cases.\n",
    "\n",
    "Let's see examples that improved the most after fine-tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5VqM0bCqLa2g"
   },
   "outputs": [],
   "source": [
    "scores1 = compute_rouge_l(predictions_baseline_govreport, test_items_govreport)\n",
    "scores2 = compute_rouge_l(predictions_ft_govreport, test_items_govreport)\n",
    "score_differences = np.array(scores2) - np.array(scores1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4rfRpr8iLa2g",
    "outputId": "d6e362a9-8059-4008-c505-9e831cb9bbc5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The report outlines the U.S. Department of Agriculture's (USDA) efforts to improve customer service and reduce administrative costs through the establishment of business centers in its eight mission areas. The report highlights both positive results, such as savings from reducing the size of the USDA's vehicle fleet, and areas for improvement, including staffing vacancies and employee concerns. The business centers were established in response to a 2017 memorandum from the Secretary of Agriculture that aimed to improve customer engagement, maximize efficiency, and improve agency collaboration.\\n\\nThe report highlights various metrics and goals established for certain administrative services, but notes that the department has not systematically assessed the effectiveness and impact of the business center reforms. The report makes a recommendation that the Secretary of Agriculture direct Departmental Administration to work with the mission areas to develop department-level performance goals and measures for the business centers, and use them to assess the effectiveness and impact of the business center reforms.\\n\\nThe report also provides specific data on the Farm Production and Conservation (FPAC) Business Center, which was established in October 2018. The center has received about $294 million in appropriations, accompanied by corresponding reductions in funding to the other three agencies within the FPAC mission area. The center plans its spending and tracks its obligations using standard categories, including personnel compensation, benefits, travel, transportation, postage, contracts, supplies, and equipment.\\n\\nThe report concludes by noting that while the business center reforms aim to improve customer service and reduce administrative costs, more evaluation is needed to determine their effectiveness and impact.\""
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_baseline_govreport[20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mwe5Ro7BLa2h",
    "outputId": "a9be3c39-b548-4638-e678-9e03cac5b0a0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Since November 2017, USDA, through an effort called the Business Center Reform Initiative, has consolidated administrative services in eight of its mission areas and established business centers to provide those services on a shared strategy. Regulations—and institutional knowledge—have not kept pace with these organizational changes. GAO was asked to provide information on the keys to a successful performance management system. This report examines: (1) USDA's status in establishing business centers as reform efforts call for. (2) The status of USDA's efforts to assess the effectiveness and impact of business centers. To conduct its work for this report, GAO reviewed relevant statutes, regulations, and documentation; analyzed mission area reports as they related to administration of the centers; analyzed data that USDA has collected on administrative services; and interviewed USDA staff and representatives of stakeholders. The U.S. Department of Agriculture (USDA) established business centers in all of its eight mission areas in accordance with the Secretary's November 2017 memorandum on business centers. The establishment of the Farm Production and Conservation (FPAC) Business Center entailed the most significant transformation—primarily because it created a separate agency within the FPAC mission area, a configuration that is different from other business centers. Moreover, there are Yu want different arrangements for delivering administrative services in two of the eight mission offices; in one business centre, a new business center provides all administrative services except funds management; and in the other, responsibilities for service delivery are divided among business centre and another entity. USDA regularly reviews data on administrative services, including services provided by business centers, but the department has not systematically assessed whether the centers' reforms have affected USDA's ability to deliver services, its customer service, and overall functionality. The department's Deputy Assistant Secretary for Administration stated that the department has not assessed the effectiveness and impact of the business center reforms, a departure from a key practice GAO has identified, which is establishing clear outcome-oriented goals and performance measures to assess the reform's effectiveness and impact. Developing such goals and measures and using them to assess the effectiveness and impact of the business center reforms could help the department (1) determine whether the reforms are meeting the Secretary's overarching policy goals and (2) identify whether the reforms have enabled mission areas to improve the delivery of their administrative services. The Secretary of Agriculture should direct Departmental Administration to work with the mission areas to develop department-level outcome-oriented performance goals and related measures for the business centers, and use them to assess the effectiveness and impact of the business centers. USDA agreement to address the recommendation is included in the body of the report.\""
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_ft_govreport[20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EvZUEysFLa2h",
    "outputId": "cd7665be-db27-4946-96b1-731f62e1efe5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"With budget authority of $146 billion in fiscal year 2018, USDA employs nearly 100,000 people organized into 13 major staff offices and eight mission areas comprising 18 agencies. In a November 2017 memorandum, the Secretary of Agriculture called for establishment of a business center in each mission area to provide consolidated administrative services. The memorandum identified three policy goals for these reforms: (1) improve customer engagement, (2) maximize efficiency, and (3) improve agency collaboration. The Agriculture Improvement Act of 2018 includes a provision for GAO to report on USDA's business centers. Among other things, this report examines the extent to which USDA has (1) established business centers and (2) assessed the effectiveness and impact of these business centers. GAO reviewed USDA documents and interviewed officials from USDA's Office of the Assistant Secretary for Administration, Office of Budget and Program Analysis, and eight mission areas about their efforts. GAO also interviewed representatives of USDA employee unions and USDA's external customers, such as farmers, for their perspectives on the establishment of the business centers. The U.S. Department of Agriculture (USDA) has established business centers to provide consolidated administrative services such as human resources and information technology in each of its eight mission areas, in keeping with reforms called for in a November 2017 memorandum from the Secretary of Agriculture. The business centers vary in when they were established; three preceded the Secretary's memorandum (see figure). Typically, each business center is located within one of the mission area's component agencies, and the center's leader reports directly to agency leadership. According to a USDA official, the department regularly reviews data on administrative services, including services provided by the business centers. However, the department has not assessed the effectiveness and impact of its business centers and as of November 2019, did not plan to do so. Beginning in 2018, USDA created an online monitoring system to compile data on the status of administrative services, with “dashboards” displaying data specific to different administrative services, among other things. However, the department has not used dashboards or associated metrics to assess the effectiveness and impact of the business centers, including their impact on USDA's customer service; human resources, including hiring; and overall functionality. GAO's prior work has shown that a key practice to consider during an agency's reform efforts is establishing clear outcome-oriented goals and performance measures to assess the reform's effectiveness and impact. Developing appropriate performance goals and systematically assessing the effectiveness and impact of the business center reforms could help the department determine whether the reforms are meeting the Secretary's overarching policy goals and improving the delivery of administrative services to support the department's mission and program goals. GAO recommends that USDA establish department-level outcome-oriented performance goals and related measures for the business centers, and use them to assess the effectiveness and impact of the business center reforms. USDA agreed with the recommendation.\""
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_items_govreport[20]['completion']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TQuEqLjmLa2h",
    "outputId": "23acf174-ed9f-4c54-b86e-399e6a4a03eb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 12.44444444444443, 'GovReport fine-tuned - GovReport baseline')"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAH4CAYAAACbup4ZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+70lEQVR4nO3deXhTZcL+8fskKWuplE1UXgWKKVtb1rIoi6CAijjgMiIFQRHEhRGHH1jEBRgWCzKMVmQXBJkXZRkXQBZHQZFdUVSkgAwqDpStVCgU25zfH7yNhBZo07R52n4/18UVenJycj/J6bmTc04Ty7ZtWwAAIKgcwQ4AAAAoZAAAjEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABXMEOkJKSosmTJ+vTTz/VqVOnFBkZqb/+9a9q1qyZJKlfv3764osvfG4TGxur+fPnByMuAAAFwgr2R2c+/PDDOnLkiF588UVVrlxZ8+fP15IlS7Rs2TLVrl1brVu31lNPPaVbb73Ve5uQkBBVrFjR7/vMzPTo+PHTAUifPw6HpUqVyuv48dPyeIrHJ5gWtzEVt/FIjKmoYExFQ9aYLMvK97KC+g75wIED2rBhgxYuXKimTZtKkp5//nl99tln+uCDDxQXF6djx44pJiZGVatWDWbUAuFwWLIsSw6HVaxWzuI0puI2HokxFRWMqWjIGlNAlhWQpfgpPDxcM2bMUFRUlHeaZZ0fXGpqqnbv3i3LslSrVq0gpgQAoOAF9R1yWFiY2rVr5zNt1apVOnDggEaMGKGkpCRVqFBBo0eP1oYNG1SuXDl16dJFjz/+uEqVKpWv+3a5gn8+m9Pp8LksDorbmIrbeCTGVFQwpqIhkGMJ+kldF/ryyy8VHx+vTp06qX379hoxYoTS09MVHR2tfv36adeuXUpISNCvv/6qhIQEv+/H4bAUHl4+gMnzJyysbLAjBFxxG1NxG4/EmIoKxlRyBP2krixr167V0KFD1aRJE73xxhsqXbq0MjIydPr0aV111VXe+VasWKEhQ4Zow4YNqlKlil/3lZnpUWrqmUBF95vT6VBYWFmlpp5RZqYn2HECoriNqbiNR2JMRQVjKhqyxhQIRrxDXrBggcaOHasuXbro5Zdf9u6OdrlcPmUsSTfeeKMk6dChQ34XsiRlZJizMmRmeozKEwjFbUzFbTwSYyoqGFPJEfQd+QsXLtSYMWPUq1cvTZ482efYcO/evRUfH+8z/86dOxUSEqKaNWsWclIAAApOUN8h79+/X+PGjdNtt92mgQMH6ujRo97rypQpo86dO2vcuHGKjo7WzTffrJ07dyohIUGPPPKIQkNDg5gcAIDACmohr1q1Sr///rvWrFmjNWvW+FzXvXt3TZgwQZZlaf78+Ro3bpyqVq2qvn37asCAAUFKDABAwTDmpK7CZMondblcDoWHl9eJE6eLzfGU4jam4jYeiTEVFYypaMgaUyAE/RgyAACgkAEAMAKFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGMOLbngDAFA6HJYfDCnYM7xffZ11eyOOx5fGUuA9ZLPYoZAD4Pw6HpYrh5eR0mLPzMKfv2s30eJRyIo1SLmYoZAD4Pw6HJafDoYUf7VLy8bSgZrEsS06nQ5mZHl34lQPVKpXTg13qyeGwKORihkIGgIskH0/TwSOngprBsiy5XE5lZGSqBH4HUIlkzn4ZAABKMAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABXMEOAKDkcDgsORxWwJfrdDp8LvO7HCAYKGQAhcLhsFQxvJycjoIrvbCwsgW2bKCgUcgACoXDYcnpcGjhR7uUfDwtoMu2LEtOp0OZmR7Ztu33ciJrVtLtrWvJsgL/Lh64EgoZQKFKPp6mg0dOBXSZlmXJ5XIqIyMzX4VcNZx32AgeDpgAAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAAMEvZBTUlL0wgsvqG3btmrSpIl69uypbdu2ea/fuHGjevTooZiYGHXp0kXLly8PYloAAApG0Av5mWee0VdffaXJkydryZIlqlevnh555BH9+OOP2rdvnwYOHKg2bdpo6dKluu+++zRs2DBt3Lgx2LEBAAgoVzDv/MCBA9qwYYMWLlyopk2bSpKef/55ffbZZ/rggw907NgxRUZGasiQIZKkiIgIff/995o1a5ZatWoVzOgAAARUUN8hh4eHa8aMGYqKivJOsyxLlmUpNTVV27Zty1a8LVu21Pbt22XbdmHHBQCgwAT1HXJYWJjatWvnM23VqlU6cOCARowYoWXLlql69eo+11erVk1nzpzRiRMnVKlSJb/v2+UK+t56OZ0On8vioLiNqbiNRwremLLuL+tFd0BZf1xa8n/Z3lyWAp8xz2H+uLxwTFm5iuI6WZx/nwIhqIV8sS+//FLx8fHq1KmT2rdvr7Nnz6pUqVI+82T9fO7cOb/vx+GwFB5ePl9ZAyksrGywIwRccRtTcRuPFLwxOZ0OuVzOAlm2y5m/5TodDu9lQWXMq4vHlFUARXmdLMrZC5Ixhbx27VoNHTpUTZo00aRJkyRJpUuXzla8WT+XLev/E+rx2EpNTfM/bIA4nQ6FhZVVauoZZWZ6gh0nIIrbmIrbeKTgjSnrfjMzPcrIyAzswq3zxZWRmSnl42hWpsfjvQx4xry6xJiynrOiuE4W59+nQDCikBcsWKCxY8eqS5cuevnll73vgq+55holJyf7zJucnKxy5cqpQoUK+brPjAxzVobzGyhz8gRCcRtTcRuPFLwx2bYd8HNAvLt0beVr2d7b5nM5gXCpMWX9vyivk0U5e0EK+o78hQsXasyYMerVq5cmT57ss4u6WbNm2rJli8/8mzZtUpMmTeRwBD06AAABE9R3yPv379e4ceN02223aeDAgTp69Kj3ujJlyqh3797q3r27Jk2apO7du2vdunX66KOPNGvWrCCmBgAg8IJayKtWrdLvv/+uNWvWaM2aNT7Xde/eXRMmTNDUqVM1ceJEzZs3TzVq1NDEiRP5G2QAQLET1EJ+7LHH9Nhjj112nrZt26pt27aFlAgAgODgQCwAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAzgCnYAAIHhcFhyOKwrzud0OnwuC0th3x9Q1FDIQDHgcFiqGF5OTkfuSy8srGwBJgKQVxQyUAw4HJacDocWfrRLycfTLjuvZVlyOh3KzPTItu1CSihF1qyk21vXkmVd+V08UBJRyEAxknw8TQePnLrsPJZlyeVyKiMjs1ALuWo478iBy+GgDgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABjCrk6dOnq3fv3j7TRo4cqcjISJ9/HTp0CFJCAAAKhivYAbK8/fbbmjJlipo1a+Yzfffu3XrssccUFxfnneZ0Ogs7HgAABSrohXz48GG9+OKL2rx5s2rWrOlznW3b2rt3rwYMGKCqVasGJyAAAIUg6IX83XffKSQkRO+//75ef/11HTx40HvdTz/9pLS0NNWuXTvg9+tyBX9vvdPp8LksDorbmIrKeLLyWZYly7IuP7P1x6WlK8wbQN5clq6cMc8L/+MyP2Mq0Ix5DvPH5YVjyspl+jqZk6Ly+5QXgRxL0Au5Q4cOlzwmnJSUJEmaP3++1q9fL4fDobZt22rIkCGqUKGC3/fpcFgKDy/v9+0DLSysbLAjBFxxG1NRGY/T6ZDLlbtDOq5CPvTjdDi8l7nNmFf5HVNhZMyri8eUVQBFZZ3MSVHOXpCCXsiXk5SUJIfDoWrVqmnatGn66aeflJCQoD179mjevHlyOPx7ZeLx2EpNTQtw2rxzOh0KCyur1NQzysz0BDtOQBS3MRWV8WTlzMz0KCMj8/IzW+c38hmZmZJdOPkkKdPj8V5eMWNeBWhMBZoxry4xpqz10PR1MidF5fcpL7LGFAhGF/KgQYP04IMPKjw8XJLkdrtVtWpV3X///dq5c6diYmL8XnZGhjkrw/mNqDl5AqG4jamojMe2bdn25RvJu/vT1hXnDSTvfRXA/QZqTAWZMa8uNaas/xeVdTInRTl7QTJ6R77D4fCWcZYbb7xRknTo0KFgRAIAoEAYXcjDhg1T3759fabt3LlTklSnTp0gJAIAoGAYXcidO3fWxo0blZiYqJ9++knr1q3TiBEj1LVrV0VERAQ7HgAAAWP0MeSOHTtqypQpmjFjhmbOnKkKFSrorrvu0tNPPx3saAAABJRRhTxhwoRs026//XbdfvvtQUgDAEDhMXqXNQAAJQWFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABigQAr50KFDBbFYAACKLb8KuV69evrmm29yvG7btm18fzEAAHnkyu2Mc+bMUVpamiTJtm29++67Wr9+fbb5vvrqK5UqVSpwCQEAKAFyXcjp6elKTEyUJFmWpXfffTfbPA6HQxUqVNCgQYMClxAAgBIg14U8aNAgb9HWrVtX77zzjqKjowssGAAAJUmuC/lCP/zwQ6BzAABQovlVyJK0YcMGffLJJzpz5ow8Ho/PdZZlady4cfkOBwBASeFXIc+ZM0cJCQkqXbq0KlWqJMuyfK6/+GcAAHB5fhXyggULdNddd2ns2LGcUQ0AQAD49XfIR48e1b333ksZAwAQIH4Vcv369bVnz55AZwEAoMTya5f1iBEj9PTTT6tcuXKKiYlR2bJls81z7bXX5jscAAAlhV+F3LNnT3k8Ho0YMeKSJ3Dt2rUrX8EAAChJ/CrkMWPGcCY1AAAB5Fch9+jRI9A5AAAo0fwq5K1bt15xnubNm/uzaAAASiS/Crl3796yLEu2bXunXbwLm2PIAADknl+F/NZbb2WblpaWpm3btum9997Ta6+9lu9gAACUJH4VcmxsbI7T27dvr3LlyumNN97Q9OnT8xUMAICSxK8PBrmcZs2aacuWLYFeLAAAxVrAC/nf//63ypcvH+jFAgBQrPm1y7pPnz7Zpnk8Hh06dEgHDx7Uo48+mu9gAACUJH4V8oVnV2dxOBxyu90aOHCg7rnnnnwHAwCgJPGrkOfPnx/oHAAAlGh+FXKW9evXa8uWLUpNTVWlSpXUtGlTtWnTJlDZAAAoMfwq5HPnzunxxx/X559/LqfTqfDwcJ04cULTp09Xy5YtNX36dL4rGQCAPPDrLOvXXntN27dvV0JCgr755ht9/vnn+vrrrzV+/Hjt2LFDb7zxRqBzAgBQrPlVyB9++KGefPJJdevWTU6nU5Lkcrn0pz/9SU8++aQ++OCDgIYEAKC486uQjx8/rvr16+d4Xf369XX48OF8hQIAoKTxq5Cvv/56bd++Pcfrtm7dqmuuuSZfoQAAKGn8OqnrgQce0IQJE1SmTBndeeedqlKlio4ePaoPP/xQM2fO1JNPPhnonAAAFGt+FXLPnj31/fffa9KkSXrllVe8023bVvfu3TVgwICABQQAoCTw+8+exo4dq4cfflhbtmzRyZMnZVmWbr31VkVERAQ6IwAAxV6ejiHv3r1b99xzj958801JUkREhHr27KkHH3xQ//jHP/TMM89o//79BRIUAIDiLNeF/Msvv6hPnz46evSoatWq5XNdSEiIhg0bppSUFD344IOcZQ0AQB7lupBnzJihihUratmyZerSpYvPdWXLllXfvn21ePFilS5dWtOnTw94UAAAirNcF/LGjRvVv39/VapU6ZLzVK1aVQ8//LA2bNgQkHAAAJQUuS7k5ORk1axZ84rzud1uHTp0KD+ZAAAocXJdyJUqVVJycvIV5ztx4oSuuuqqfIUCAKCkyXUhN2/eXEuXLr3ifP/6178u+bGaAAAgZ7ku5N69e2vz5s2aMGGC0tPTs11/7tw5JSQkaP369erVq1dAQwIAUNzl+oNBoqKiFB8fr3Hjxum9995Tq1atVKNGDWVmZurXX3/V5s2bdeLECf3lL39RmzZtCjIzAADFTp4+qatXr16qW7euZs+erY8//tj7Trl8+fK6+eab9fDDDysmJqZAggIAUJzl+aMzmzZtqqZNm0o6/zWMLpdLYWFhAQ8GAEBJ4tdnWWe53N8kAwCA3PPr+5ABAEBg5esdMlBSWJYll8vc169Op7nZAOQOhQzkQoWwMnI6KD0ABYdCBnLB6XBo4Ue7lHw8LdhRchRZs5Jub11LlmUFOwoAP1HIQC4lH0/TwSOngh0jR1XDywY7AoB8Yh8cAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwgFGFPH36dPXu3dtn2q5duxQXF6dGjRqpQ4cOeuutt4KUDgCAgmNMIb/99tuaMmWKz7QTJ06oX79+uv7667VkyRI98cQTmjRpkpYsWRKckAAAFJCgf7nE4cOH9eKLL2rz5s2qWbOmz3XvvPOOQkJCNHr0aLlcLkVEROjAgQOaMWOG7rnnnuAEBgCgAAT9HfJ3332nkJAQvf/++4qJifG5btu2bYqNjZXL9cfrhpYtW+o///mPjh49WthRAQAoMEF/h9yhQwd16NAhx+sOHTokt9vtM61atWqSpP/+97+qUqWK3/frcgX9tYicTofPZXFQ3MZ04TgsyzL2+4a9uSxdOaP1x6WlwhtPnjLmeeF/XOZnTAWaMc9h/ri8cExZuUJCnEb/ntm2dPFD6HCYl93jsWXbtt+3D+Q4gl7Il3P27FmVKlXKZ1rp0qUlSenp6X4v1+GwFB5ePl/ZAiksrPh9l21xHJPT6ZDL5Qx2jBw5HQ7vZW4zupyFOxZ/MuZVfsdUGBnz6uIxXVWhtDweW6GhZYKUKHc8HttbwBczKfvlchY2owu5TJkyOnfunM+0rCIuV66c38v1eGylpqblK1sgOJ0OhYWVVWrqGWVmeoIdJyCK25iyxiNJmZkeZWRkBjlRzjI9Hu/lFTNa5zfyGZmZkv9vDPIsTxnzKkBjKtCMeXWJMZVyOeRwWPrnqh+UfDz427GcRN4Qri6ta2XPaJ1/sZPp8RTquncp1SqVU8/OdfO1vbpwG5FfRhdy9erVlZyc7DMt6+err746X8vOyDCnLM5v6M3JEwjFcUy2nb9dWwXJm8vWFTN6d3/mYt5AykvGvArUmAoyY15dakxZ/08+nqZfkn8LRrQrqlLx/DvgizNaliWXy6mMjMygP77SH4+lKdsrM3biX0Lz5s21fft2ZWb+8Up106ZNqlWrlipXrhzEZAAABJbRhXzPPffo1KlTeu6557R3714tXbpUc+fO1cCBA4MdDQCAgDK6kCtXrqxZs2Zp//796t69uxITEzVs2DB179492NEAAAgoo44hT5gwIdu06OhoLVq0KAhpAAAoPEa/QwYAoKSgkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMIAr2AFy4/Dhw2rbtm226ePHj1ePHj2CkAgAgMAqEoX8ww8/qHTp0lq7dq0sy/JOr1ChQhBTAQAQOEWikJOSklSzZk1Vq1Yt2FEAACgQReIY8u7duxURERHsGAAAFJgi8w45PDxcvXr10v79+3XDDTdo0KBBOR5Xzi2XK/ivRZxOh89lcVDcxnThOCzL8jlkYhJvLktXzmj9cWmp8MaTp4x5Xvgfl/kZU4FmzHOYPy4vHJNRGS/hkhmDtO5dSla2/GyvArmtM76QMzIy9OOPP6pOnTp69tlnFRoaquXLl2vAgAF688031apVqzwv0+GwFB5evgDS+icsrGywIwRccRyT0+mQy+UMdowcOR0O72VuM7qchTsWfzLmVX7HVBgZ8+riMZmY8WJXyljY696lZJWpKdsr4wvZ5XJp8+bNcjqdKlOmjCSpYcOG2rNnj2bPnu1XIXs8tlJT0wIdNc+cTofCwsoqNfWMMjM9wY4TEMVtTFnjkaTMTI8yMjKDnChnmR6P9/KKGa3zG8SMzEzJLoRw/ydPGfMqQGMq0Ix5dYkxGZXxEi6ZMUjr3qVkbaPys726cBuRX8YXsiSVL5/93eyNN96ozz//3O9lZmSYUxbnN/Tm5AmE4jgm27Zl2wZsRXLgzWXrihm9uwpzMW8g5SVjXgVqTAWZMa8uNSaTMl7KpTIGa927lKwMpmyvjD/Qt2fPHjVp0kSbN2/2mf7tt9+qTp06QUoFAEBgGV/IERERql27tkaPHq1t27Zp3759Gj9+vHbs2KFBgwYFOx4AAAFh/C5rh8OhadOm6ZVXXtHTTz+t1NRU1a9fX2+++abcbnew4wEAEBDGF7IkValSRePHjw92DAAACozxu6wBACgJKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAYrEJ3WheHM4LDkcwf+y8pwE8svHAeByKGQElcNhqWJ4Oe8XmgNASUUhI6gcDktOh0MLP9ql5ONpwY6TjWVZqlersjq1vEGWZea7eADFA4UMIyQfT9PBI6eCHSMby7JUvXL5YMcAUAKwnxAAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADAAhQwAgAEoZAAADEAhAwBgAAoZAAADUMgAABiAQgYAwAAUMgAABqCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAV7ADFGUOhyWHw/L79k6nw+eyOMjrmIrT2AEgPyhkPzkcliqGl5PTkf9CCQsrG4BEOfN47Hy9aPBXQY4JAIojCtlPDoclp8OhhR/tUvLxNL+WYVmWnE6HMjM9sm07wAmlyJqVdHvrWvrnqh90+NjpgC8/J3kdU1ZGyyr8Fw0AYBIKOZ+Sj6fp4JFTft3Wsiy5XE5lZGQWSCFXDT//LjU/GfMqr2PKyggAJR0H8AAAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAGoJABADBAkShkj8ejV199VW3atFGjRo306KOP6ueffw52LAAAAqZIFPLUqVO1cOFCjRkzRv/7v/8rj8ej/v3769y5c8GOBgBAQBhfyOfOndOcOXM0ePBgtW/fXnXr1tXf//53HTp0SKtXrw52PAAAAsL4Qv7hhx90+vRptWrVyjstLCxM9evX19atW4OYDACAwLHsgviaoQBavXq1nnrqKX399dcqU6aMd/pf/vIXnT17VtOnT8/zMm3blseTv2FbluRwOHQq7Zwy87msghLicqhcmRAy5hMZA4OMgUHGwHE6LIWWKyWPxyN/mzCrCwLB+K9fPHPmjCSpVKlSPtNLly6tkydP+rXM89/ZG5jv3w0tV+rKMwUZGQODjIFBxsAgY+AEqlDzy4wUl5H1rvjiE7jS09NVtizfpQsAKB6ML+RrrrlGkpScnOwzPTk5WVdffXUwIgEAEHDGF3LdunUVGhqqzZs3e6elpqbq+++/V/PmzYOYDACAwDH+GHKpUqUUFxenSZMmqVKlSrruuus0ceJEVa9eXZ06dQp2PAAAAsL4QpakwYMHKyMjQyNHjtTZs2fVvHlzzZ49WyEhIcGOBgBAQBj/Z08AAJQExh9DBgCgJKCQAQAwAIUMAIABKGQAAAxAIQMAYAAKGQAAA1DIAAAYgEIOshdeeEHPPvtstun9+vVTZGSkz7/evXsHIWHeXWpMGzduVI8ePRQTE6MuXbpo+fLlQUjnv+3bt2d7TiIjI30+1tV0Ho9Hr776qtq0aaNGjRrp0Ucf1c8//xzsWPly+PDhHJ+XpUuXBjtank2fPj3b7/muXbsUFxenRo0aqUOHDnrrrbeClM4/OY1p5MiR2Z6vDh06BClh7qSkpOiFF15Q27Zt1aRJE/Xs2VPbtm3zXh+Q7ZuNoMjMzLRfeeUV2+1228OHD892fatWreyFCxfaycnJ3n8nTpwo/KB5cLkx7d27146KirInT55s79271541a5Zdv359+4svvghS2rx7++237VtvvdXnOUlOTrbT09ODHS3XXnvtNbtFixb2J598Yu/atct++OGH7U6dOhWpMVzs008/taOiouzDhw/7PC9nzpwJdrQ8WbBggV23bl07Li7OO+348eN2ixYt7Pj4eHvv3r324sWL7aioKHvx4sVBTJp7OY3Jtm373nvvtSdPnuzzfB07dixIKXOnX79+dteuXe2tW7faP/74oz1q1Cg7Ojra3rdvX8C2b0XiozOLm3379um5557TgQMHdO2112a7/tixYzp27JhiYmJUtWrVICTMuyuNad68eYqMjNSQIUMkSREREfr+++81a9YstWrVqrDj+iUpKUl16tQpMs/Jxc6dO6c5c+Zo6NChat++vSTp73//u9q0aaPVq1era9euwQ3op6SkJNWsWVPVqlULdhS/HD58WC+++KI2b96smjVr+lz3zjvvKCQkRKNHj5bL5VJERIQOHDigGTNm6J577glO4Fy43Jhs29bevXs1YMCAIvO7dODAAW3YsEELFy5U06ZNJUnPP/+8PvvsM33wwQc6duxYQLZv7LIOgk2bNikiIkIffvihatSoke363bt3y7Is1apVKwjp/HOlMW3bti3bitmyZUtt375ddhH59Nbdu3crIiIi2DH89sMPP+j06dM+z0NYWJjq16+vrVu3BjFZ/hT15+W7775TSEiI3n//fcXExPhct23bNsXGxsrl+uO9U8uWLfWf//xHR48eLeyouXa5Mf30009KS0tT7dq1g5Qu78LDwzVjxgxFRUV5p1mWJcuylJqaGrDtG++Qg6BXr16XvT4pKUkVKlTQ6NGjtWHDBpUrV05dunTR448/rlKlShVSyry50pgOHTqk6tWr+0yrVq2azpw5oxMnTqhSpUoFGS8g9uzZo/DwcPXo0UOHDx+W2+3WkCFDFB0dHexouXLo0CFJf3zHeJZq1ap5ryuKkpKSFB4erl69emn//v264YYbNGjQILVt2zbY0XKlQ4cOlzx+eujQIbndbp9pWXsC/vvf/6pKlSoFns8flxtTUlKSJGn+/Plav369HA6H2rZtqyFDhqhChQqFGTPXwsLC1K5dO59pq1at0oEDBzRixAgtW7YsINs3CjnAfvnlF3Xs2PGS12/cuPGKT05SUpLS09MVHR2tfv36adeuXUpISNCvv/6qhISEQEe+okCM6ezZs9leTGT9fO7cufyHzKcrjfHTTz/Vb7/9prS0NI0cOVJOp1MLFixQXFycli5dqjp16hRiWv+cOXNGkrI9D6VLl9bJkyeDESnfMjIy9OOPP6pOnTp69tlnFRoaquXLl2vAgAF68803i8zhkEvJ6femdOnSkqT09PRgRMq3pKQkORwOVatWTdOmTdNPP/2khIQE7dmzR/PmzZPDYf6O2y+//FLx8fHq1KmT2rdvH7DtG4UcYFdffbVWrFhxyeuvuuqqKy5j9OjRGj58uHdet9utkJAQDRkyRMOGDSv0V8WBGFPp0qWzrZhZP5ctWzZ/AQPgSmOsVq2atm7dqrJly3q/9jMqKkrff/+95s+fr1GjRhVWVL+VKVNG0vnHPev/0vkNuwnPgT9cLpc2b94sp9PpHVPDhg21Z88ezZ49u8gXcpkyZbL93mQVcbly5YIRKd8GDRqkBx98UOHh4ZLOb9+qVq2q+++/Xzt37sy2i9s0a9eu1dChQ9WkSRNNmjRJUuC2bxRygIWEhOT7eJbL5cpWcjfeeKOk87uwCruQAzGma665RsnJyT7TkpOTVa5cOSN2U+VmjGFhYT4/OxwORURE6PDhwwUZLWCydlUnJyfr+uuv905PTk5WZGRksGLlW/ny5bNNu/HGG/X5558HIU1gVa9ePcffG+n8i8iiyOFweMs4y4XbN5MLecGCBRo7dqy6dOmil19+2fsuOFDbN/P3DZRAvXv3Vnx8vM+0nTt3KiQkJNsZi0VFs2bNtGXLFp9pmzZtUpMmTYrELqr169ercePGPn+zm5GRoR9++KFI7K6WpLp16yo0NNTn76ZTU1P1/fffq3nz5kFM5r89e/aoSZMm2f4W/Ntvvy0yz8vlNG/eXNu3b1dmZqZ32qZNm1SrVi1Vrlw5iMn8N2zYMPXt29dn2s6dOyXJ6Ods4cKFGjNmjHr16qXJkyf77KIO1PbN/C1hCdS5c2e99957+uc//6mff/5ZK1asUEJCgh555BGFhoYGO55fevfurW+++UaTJk3Svn37NGfOHH300Ufq379/sKPlSpMmTRQeHq7hw4fr22+/1e7duzV8+HClpKRk27iYqlSpUoqLi9OkSZP08ccf64cfftCQIUNUvXp1derUKdjx/BIREaHatWtr9OjR2rZtm/bt26fx48drx44dGjRoULDj5ds999yjU6dO6bnnntPevXu1dOlSzZ07VwMHDgx2NL917txZGzduVGJion766SetW7dOI0aMUNeuXY09W37//v0aN26cbrvtNg0cOFBHjx7VkSNHdOTIEf32228B276xy9pAcXFxsixL8+fP17hx41S1alX17dtXAwYMCHY0v914442aOnWqJk6cqHnz5qlGjRqaOHFikTnGFxoaqrlz52rSpEl65JFHlJ6erqZNm2rBggXGnumak8GDBysjI0MjR47U2bNn1bx5c82ePdt7XLyocTgcmjZtml555RU9/fTTSk1NVf369fXmm29mOzu5KKpcubJmzZqlsWPHqnv37qpataqGDRum7t27Bzua3zp27KgpU6ZoxowZmjlzpipUqKC77rpLTz/9dLCjXdKqVav0+++/a82aNVqzZo3Pdd27d9eECRMCsn2z7KLyR6AAABRj7LIGAMAAFDIAAAagkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyCiWisqf1xeVnCgcrA8lG4VchOzbt09jxoxR586dFRMTo6ZNm+qBBx7QwoULlZGREfD727x5syIjI7P9a9iwodq2bathw4bpyJEjAb/f/Jo6dapmz5592XkOHTqkXr16KSoqSq1atdK6desUGRmZ7TORC8qhQ4c0YMAAHTx4sFDuL7eeffbZS36Prb+++eYbxcfHq2PHjoqOjlZsbKweeughffjhhwG9nyxLly7Ncb2Njo5Whw4dNHr0aJ06dapA7ttf586d07hx4/TBBx9cdr7IyEi99tprhZTq8n755RdFRkZq6dKlkv543H/55ZcgJyu6+OjMImLFihWKj49XRESE+vXrp1q1auns2bNat26dxo0bp88++0xTp06VZVkBv+8XXnhBDRo08P58+vRpbd++XTNmzND+/fv17rvvBvw+8+Mf//iHnnzyycvOM2/ePO3YsUMTJ07U1VdfrcjISC1atKjQPtz+iy++0Lp16wrlvoJp7ty5SkhIULNmzfTkk0+qRo0aOnnypFavXq2hQ4dqx44dGjlyZIHcd2JioqpWrer9+eTJk/rss880f/58HT9+XFOmTCmQ+/VHcnKy5s2bp/Hjxwc7it/at2+vRYsWqVq1asGOUmRRyEXAvn37FB8frzZt2mjKlClyuf542tq1a6cWLVpo8ODBWrlype64446A33+dOnXUqFEjn2k33XSTzp07p5kzZ2rv3r1Gf0tLTlJSUlStWjWfx+viMSJ/Nm/erAkTJiguLi5b6d56662qW7euXn75ZXXt2rVAHvt69eqpRo0aPtPatWunY8eOaeXKlTp9+nSOX90I/1SqVEmVKlUKdowijV3WRcCsWbPkcDg0atQonzLO0rlzZ/3pT3/ymZaenq7XX39dXbp0UVRUlDp16qQZM2bI4/FIkqZNm6aGDRvq5MmTPrebO3euGjRooGPHjl0xV9b3A1/4rjwpKUkDBw5UkyZN1KRJEz3xxBM+X1mYtRv8888/V69evRQdHa1OnTpp4cKFecovnf8GqaFDh2rw4MFq1KiR+vXr5/1e38TExEt+x2+HDh20dOlS/frrr95dgFm5snZZv/baa7rtttv06aef6q677lLDhg3VuXNn/etf//JZVkpKil544QW1bt1aUVFRuv/++7Vx48bLPm5Lly71fr1mx44d9eyzz0rKeXfka6+95jOOZ599Vn379tWSJUvUuXNnNWzYUHfffbfWr1/vc7tff/1VzzzzjGJjYxUTE6OHHnpI33//vc88J0+eVHx8vGJjY9W8eXNNnDjR5/HNr9dff13XXHON/t//+385Xt+nTx917NhRZ86c8U777bffNH78eN16662KiopS165dtXjxYu/1zz//vG666SafryOUpLFjx6pFixb6/fffr5irQoUKsizLZ73dtm2b4uLiFBMTo9jYWA0fPlzHjx/3Xp+1O/brr79W9+7dFR0drbvuuksfffSRz7KvlF86v/6NGzdODz30kKKjo9W3b1917NhRkhQfH3/FQwanTp3S0KFD1bhxY7Vq1Up/+9vffB7DzMxMzZgxQ127dlV0dLQaNWqkBx54QJs2bfLOc/bsWb300ktq27atGjZsqC5dumQ7zJPXdfviXdaBXFdLCgq5CPj444/VsmXLy37/6csvv+x9t2fbth577DHNmjVL9913n6ZNm6YuXbpoypQpevHFFyVJd911lzIyMrR69Wqf5Sxfvlw333yzz315PB5lZGR4/6WkpGj16tWaPXu2oqOjVatWLUnnv6LsgQce0LFjx/Tyyy9r7Nix+vnnn9WzZ89sBT9kyBDVr19fr7/+ulq3bq1Ro0Z5Szk3+bOsXLlS5cuX1xtvvKH+/ftr0aJFkqR7773X+/+LJSYmql27dqpataoWLVqk++67L8f5jhw5otGjR6tPnz6aMWOGatSooeHDh2vfvn2Szr9oeOihh/Txxx9ryJAhSkxMVPXq1dW/f//Lbrjat2/v/WrAxMREPf7445ecNyfffvutZs+ercGDB+v111+X0+nUU0895X1xdfz4cT3wwAP67rvv9Pzzz+uVV16Rx+NRr169vNk9Ho/69++vdevWafjw4ZowYYK+/PJLrVixIk9ZLuXkyZPaunWrOnbsqNKlS+c4j8vl0tSpU73fiHP27Fk9+OCD+uCDD9S/f39NnTpVTZs21XPPPadp06ZJku6++24dPXrU51i/x+PRypUrdeedd/p8a9WF6+3vv/+uY8eOafHixVq2bJluu+02lStXTpK0detW9e3bV2XKlNGUKVM0YsQIbdmyRX369NHZs2d9Mg8cOFAdO3ZUYmKiatWqpaefftp76CE3+bO8/fbbioqK0tSpU/X4448rMTFRkjRo0CDv/y9l/vz5On36tKZMmaKBAwfq3Xff1dChQ73XT5o0SVOnTtWf//xnzZo1S2PGjFFKSor+8pe/eIt73LhxWr9+vYYPH67Zs2erY8eOSkhI0JIlSyT5v25fLBDraoliw2gpKSm22+22J0yYkO2633//3edfRkaGbdu2/emnn9put9v+8MMPfeZ//fXXbbfbbSclJdm2bdtxcXF2nz59vNcfOHDAdrvd9vLly23btu1NmzbZbrc7x3/Nmze3n332WfvYsWPe2z/zzDN269at7d9++8077cSJE3bTpk29+bOWGR8f75Nt0KBB9k033WR7PJ485Y+JibHT09N95nO73farr7562cd1+PDh9i233OL9OSvXpk2bbNu27VdffdV2u932F1984Z3n4MGDttvttmfPnm3btm0vWrTIdrvd9o4dO7zzeDweu1evXnaPHj0ue/9Lliyx3W63/fPPP182d1aOC3O73W77wIED3mlbtmyx3W63/dFHH9m2bduTJ0+2o6Ki7F9++cU7T3p6ut2xY0f7qaeesm3btj/55BPb7Xbb69at885z+vRpu0WLFj6Pi7++++472+122wsWLPCZ7vF4Lrnevv3227bb7ba//PJLn9uMGDHCjoqKsk+cOGF7PB77lltu8Vl/vvjiC5/nIeuxzelf69at7XHjxtmnTp3y3v7Pf/6z3bVrV28O27btH3/80a5Xr543f9YyExMTfcZy99132/fdd1+u89u2bd9yyy32rbfe6jPPzz//bLvdbnvJkiWXfVzdbrfdtWtXOzMz0ztt7ty5ttvttnfv3m3b9vnfw7lz5/rcbtWqVbbb7ba/+uor27Ztu3PnzvbIkSN95klMTLQ/+eQT27Zzt25fnPnidTpQ62pJwjtkw11qF+KBAwfUoEEDn3+33XabJGnLli1yuVzq0qWLz226devmvT7r561bt3rPlF6+fLlCQ0Oz7TIbNWqUFi9erHfeeUcDBw6U0+lU7969NX78eJ9jRps2bVJsbKzKlCnjfWcSGhqqZs2a6YsvvvBZ5sXf59qpUycdOXJE+/fvz3V+Sapdu7ZKlSp1mUcwfy48tlm9enVJUlpamiRp48aNqlq1qho0aOAdb2Zmpm655RZ9++23OnnyZLa9CxfvavVHpUqVdP3112fLlfXuZ+PGjapXr56uvvpq7/06HA61bdvW+zxs27ZNISEhatOmjXc55cqVU7t27S5735mZmbkaz6XW240bN2Zbb/v27Svp/PN63XXXqXHjxj636datm9LT0/X111/Lsix169ZNa9eu1blz5ySdX29r1qypmJgYn9u98cYbWrx4sf75z3/qvvvuU0hIiAYPHqz4+HjvseMzZ87o66+/Vrt27WTbtndc//M//6OIiAht2LDBZ5kXrreWZem2227TN998o7Nnz+Yqf5Z69epd6iG+oi5dusjh+GPT3alTJ0nn3+lL0iuvvKKHHnpIx48f17Zt27RkyRK9//77kuR9zFq0aKF33nlHjz76qBYsWKCff/5ZTzzxhNq3by8pd+t2bgRiXS1JOKnLcOHh4SpXrly2P4+55pprfI5Nvf7660pKSpJ0fndheHi4nE6nz22yzjj97bffJJ3/xR4zZoxWrlypPn36aPny5ercubPKlCnjc7tatWopKipKkhQTE6OQkBAlJiaqdOnSGjBggHe+lJQUrVixIsfdnhef7HH11Vf7/Jy1i/zkyZO5zi+pwE/KKVu2rPf/WRtB+//+VjQlJUVHjhzxOQP9QkeOHNH48eO1bNky77TrrrtO//73vwOWSfrjGH5WCaakpHhfsOXkzJkzOnnypCpWrJjtrPwLz0rOSd++fX1eEMXGxmr+/PnZ5rv22mslKdufwERHR/ustxcegjh58mSO91+lShVJUmpqqqTzu63feOMNffbZZ2rTpo1Wr16thx56KNvt3G6396SuJk2aKCMjQy+88IJCQ0N15513epfp8Xg0c+ZMzZw5M9syLt7dfvEZxJUrV5Zt20pNTc11fkne3eX+uPg+sn53spa/c+dOjRo1Sjt37lTZsmVVp04d7/ORte4+99xzql69ut5//32NGTNGY8aMUePGjfXSSy+pbt26uVq3L95O5CQQ6+rFyyjOKOQioEOHDvrkk0906tQphYaGSpJKlSrlLUlJqlixovf/V111lU6cOKHMzEyfUktOTpZ0vuSl8ye3dOjQQStXrlTLli21Z88ePf/881fMM2jQIK1du1avvvqq2rdvL7fb7V1e69at1a9fv2y3ufhktBMnTvi8cs46xly5cuVc5w+2ChUqqGbNmpo0aVKO19eoUUNPPvmkevXq5Z12pXfzF7/jzHo3ntdcsbGxGjZsWI7XlypVSuHh4Tk+xikpKZdd9qhRo3T69Gnvz5d6QVSpUiU1btxYa9eu1dChQ733ERoa6rPeXnj7q666SgcOHMi2rKw9OFnPe61atRQdHa2VK1fK4XAoNTXVu/fkckaOHKkNGzbopZdeUosWLVSlShWVL19elmWpb9++3pK+0MVlkJKS4i1YSTp69KicTqcqVqyY6/z5dfFzlLX8ypUr69SpU+rfv78iIyO1fPly1a5dWw6HQ+vWrdOqVau8tylVqpQGDRqkQYMG6ddff9Unn3yiqVOn6q9//auWL1+eq3X76NGj+R5LbtbVkoRd1kXAgAEDlJGRoZEjR3p3OV3o7NmzPmcyx8bGKiMjI9sZoFm7rZo2beqddvfdd2vHjh365z//qWuvvVaxsbFXzONyufTSSy8pIyNDf/vb33zud+/evapXr56ioqIUFRWlhg0bau7cuVqzZo3PMtauXevz80cffaTrrrtO119/fZ7y5+TC3XkFKTY2Vv/9739VuXJl73ijoqK0YcMGzZo1S06nUzVq1PC5LuuM6ZwyhoaG6vDhwz7TvvzyS79y7d+/37tnI+vfe++9p8WLF8vpdKpVq1bKyMjweR7OnTuXbRftxWrXru2zzNq1a19y3qwz7BMSEnL8BKqTJ096X2RJUvPmzXXw4EF99dVXPvO9//77CgkJUXR0tHfa3Xffrc8++0zLly9XkyZN9D//8z9XfFxCQ0MVHx+v1NRUvfLKK95p9evX148//ugzrhtvvNF79v2FLny8bNvW6tWr1bRpU5UqVSpP+S928d6gy7n4LOXly5fLsizFxsbqxx9/VEpKivr06aM6dep417Os23g8Hp09e1adO3fWnDlzJJ3fm9GrVy/deeed+vXXXyXlbt0OhNysqyUJ75CLgMjISE2cOFHx8fHq0aOH7r33XkVGRiojI0NfffWVFi9erKNHj6p///6SpLZt26pFixYaOXKkDh8+rLp162rLli2aOXOmunfv7vM3w23atFHFihW1aNEi9e/fP9cfLNK4cWN169ZN7733nlauXKnbb79djz/+uB544AENHDhQPXv2VOnSpbVo0SLvu+kLvfnmmypdurQaNWqk1atX65NPPvFuJPOSPydhYWH68ssvtXXrVjVr1qxAPixFknr06KEFCxaoX79+euyxx3TNNdfoiy++0MyZMxUXF+dzxm9OGSVpzZo1atu2rSIiItS+fXstX75cMTExuuGGG7R06dIc33FdSd++ffXee++pb9++evjhhxUeHq4VK1bonXfe8f65VatWrXTzzTdr5MiROnbsmK677jq99dZbOn78+GXP5s+LNm3a6Pnnn9f48eO1Y8cOde/eXbVq1VJaWpq2bNmiJUuWKD09XX369JF0/vFcuHChnnjiCQ0ePFg1atTQv//9by1ZskRPPvmk9zGTpDvuuEMTJkzQihUrsp15fzl33HGHFi5cqGXLlqlnz56Kjo7WM888owEDBuivf/2runXrpszMTM2ZM0dff/11tjPgExISlJ6erlq1aundd9/Vvn37NG/evDznv1iFChUknT+mGhERke14+IV27typ5557Tl27dtXOnTv16quv6t5771XNmjX122+/KTQ0VNOmTZPL5ZLL5dKqVau8hwnOnDmjMmXKqEGDBkpMTFRISIgiIyO1f/9+LVu2TJ07d/aOxd91Oy9ys66WKME8owx588svv9gTJ06077zzTrtRo0Z2TEyM3bVrV3v8+PH2/v37feZNS0uzJ0yYYLdp08Zu0KCB3blzZ3vWrFk+Z2dmGTNmjO12u+09e/b4TL/4zOOLHT582G7cuLHdvn17Oy0tzbZt2/7222/tRx55xG7cuLHdqFEj+/7777fXrl2bbZlvv/22fe+999oNGza0u3Xr5j3rMi/54+Li7Li4uGy55syZYzdr1syOiYmxDx48mGP23J5lfbGLz4Q+evSoHR8fb7dq1cpu2LCh3blzZ3vmzJk5Ps4XOnXqlN23b1+7QYMG9qOPPmrbtm0fOXLEHjx4sN2oUSO7WbNm9gsvvGC/88472c6yvvgs6JzO0D1w4IA9ePBgu3nz5nZ0dLTdrVs3+9133/W5XVpamj169Gi7RYsWdqNGjewRI0bYf/vb3wJylvWFkpKS7FGjRtmdOnWyo6Oj7caNG9s9evSw//GPf9iHDh3ymffYsWP2iBEj7JYtW3rXjYtzZxk4cKDdsGFDOyUlxWd6TmewX2jXrl12vXr17Hvuucf2eDy2bZ8/U/vBBx+0o6Oj7aZNm9p9+vSxt27dmm2Z7733nn3HHXfYUVFR9p///Odsvxu5yX/LLbfYw4cPz5Zr/PjxdqNGjezmzZvb586dyzG72+22p0+fbj/11FN2dHS0fdNNN9l///vf7d9//907z6ZNm+wePXrY0dHRdqtWreyHH37Y3rZtm924cWP75Zdftm3btn/77Td7zJgxdvv27e0GDRrYbdu2tSdMmGCfOXPGu5wrrdu5Ocs6UOtqSWHZNp9mjsKzefNm9enTR2+99ZZatGgR7DhArmR9mMvHH3+c7dO/gEDhGDIAAAagkAEAMAC7rAEAMADvkAEAMACFDACAAShkAAAMQCEDAGAAChkAAANQyAAAGIBCBgDAABQyAAAG+P8EF2HeFnqAzAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.displot(score_differences)\n",
    "plt.xlabel(\"GovReport fine-tuned - GovReport baseline\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see in the score differential distribution above (the majority of our test datapoints improve after fine-tuning) we can significantly improve the summarization capabilities of LLMs by finetuning on long context datasets for realworld applications such as summarization.\n",
    "\n",
    "To learn more about our fine-tuning API read the docs [here](https://docs.together.ai/reference/finetune)!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "03687a6cd7064124b83ee3e9ecec997d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_38906354ac914e3f96d8299576418587",
      "placeholder": "​",
      "style": "IPY_MODEL_70494fe5181a47f7ab693bef48cd0f5d",
      "value": "Downloading builder script: 100%"
     }
    },
    "07c8b2cc25f34f3f94104669036494c6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0fe7324e2f674af9bc185e3581a7bd2f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "17d58dd7c32a4f8da74ac08df38742cf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2362664519b9415d92237a85ccf69d62": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2448da9c76a74e14961cafa0f62a9767": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3322b6bd9cad427c8693985d18b3bb35": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4c114940a70c4393b53e2465b71ab703",
      "placeholder": "​",
      "style": "IPY_MODEL_8d8cc5b12af94851bab69f7353e19fa4",
      "value": " 3.34k/3.34k [00:00&lt;00:00, 211kB/s]"
     }
    },
    "352d342144f141969b12ee0e31d74d14": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "35ac3e449986435cbb4864f18a1d363e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2362664519b9415d92237a85ccf69d62",
      "max": 3344,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_75b858dda5234f3f8b184e8eba864cd2",
      "value": 3344
     }
    },
    "380b83efe9f64a9295626a0a66b421ae": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "38906354ac914e3f96d8299576418587": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3f79c39f611c4f89b6569271f0d9b722": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4a36ae38343e4069a9594b7903639ce3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5c529de45a3f4b68ba9707c2eef6a281",
       "IPY_MODEL_a91f784b638b40bfa2a6f381a4ed37cb",
       "IPY_MODEL_b511ddcee19f49fb84d9f31c73adced1"
      ],
      "layout": "IPY_MODEL_73c2ece324b74007b01613265ba3fd9d"
     }
    },
    "4a90ce0718cc4f8ba7ea95a42538a0e5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4c114940a70c4393b53e2465b71ab703": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d85d232158d4939b3b3a72dea7c2fcd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e582366a88e2445bb6888862563380c5",
      "placeholder": "​",
      "style": "IPY_MODEL_07c8b2cc25f34f3f94104669036494c6",
      "value": "Downloading extra modules: 100%"
     }
    },
    "5036ee4a59574ff7bf6a0d39ac4dc9b6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_03687a6cd7064124b83ee3e9ecec997d",
       "IPY_MODEL_e90f6ca90f504fce8373fef4e0a30983",
       "IPY_MODEL_94cbe82d40e34ca29a53e3c4ce5fc950"
      ],
      "layout": "IPY_MODEL_f88ff86040c9438e9df8a50e190120bd"
     }
    },
    "5c529de45a3f4b68ba9707c2eef6a281": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f1bf576faf684e11a3d5a519066366a6",
      "placeholder": "​",
      "style": "IPY_MODEL_2448da9c76a74e14961cafa0f62a9767",
      "value": "Downloading builder script: 100%"
     }
    },
    "5da475c2eaa34783ba78ad5d6dc79dae": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "70494fe5181a47f7ab693bef48cd0f5d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "73c2ece324b74007b01613265ba3fd9d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "75b858dda5234f3f8b184e8eba864cd2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8d8cc5b12af94851bab69f7353e19fa4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "94cbe82d40e34ca29a53e3c4ce5fc950": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0fe7324e2f674af9bc185e3581a7bd2f",
      "placeholder": "​",
      "style": "IPY_MODEL_17d58dd7c32a4f8da74ac08df38742cf",
      "value": " 6.27k/6.27k [00:00&lt;00:00, 229kB/s]"
     }
    },
    "a2c34b4d40964aefbbb740c6238ffd33": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a91f784b638b40bfa2a6f381a4ed37cb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4a90ce0718cc4f8ba7ea95a42538a0e5",
      "max": 8645,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_aeb35b06e2c243cd8d020186564a14f8",
      "value": 8645
     }
    },
    "aeb35b06e2c243cd8d020186564a14f8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "b47f018868264f0489d048be0ff073eb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4d85d232158d4939b3b3a72dea7c2fcd",
       "IPY_MODEL_35ac3e449986435cbb4864f18a1d363e",
       "IPY_MODEL_3322b6bd9cad427c8693985d18b3bb35"
      ],
      "layout": "IPY_MODEL_352d342144f141969b12ee0e31d74d14"
     }
    },
    "b511ddcee19f49fb84d9f31c73adced1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a2c34b4d40964aefbbb740c6238ffd33",
      "placeholder": "​",
      "style": "IPY_MODEL_380b83efe9f64a9295626a0a66b421ae",
      "value": " 8.64k/8.64k [00:00&lt;00:00, 476kB/s]"
     }
    },
    "e582366a88e2445bb6888862563380c5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e90f6ca90f504fce8373fef4e0a30983": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5da475c2eaa34783ba78ad5d6dc79dae",
      "max": 6270,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_3f79c39f611c4f89b6569271f0d9b722",
      "value": 6270
     }
    },
    "f1bf576faf684e11a3d5a519066366a6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f88ff86040c9438e9df8a50e190120bd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}