docs/langchain-retrieval-augmentation.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "f0Qrii3fJYbS"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "dQRA1HWOJYbU"
      },
      "source": [
        "#### [LangChain Handbook](https://pinecone.io/learn/langchain)\n",
        "\n",
        "# Retrieval Augmentation\n",
        "\n",
        "**L**arge **L**anguage **M**odels (LLMs) have a data freshness problem. The most powerful LLMs in the world, like GPT-4, have no idea about recent world events.\n",
        "\n",
        "The world of LLMs is frozen in time. Their world exists as a static snapshot of the world as it was within their training data.\n",
        "\n",
        "A solution to this problem is *retrieval augmentation*. The idea behind this is that we retrieve relevant information from an external knowledge base and give that information to our LLM. In this notebook we will learn how to do that.\n",
        "\n",
        "[![Open full notebook](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/full-link.svg)](https://github.com/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb)\n",
        "\n",
        "To begin, we must install the prerequisite libraries that we will be using in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0_4wHAWtmAvJ"
      },
      "outputs": [],
      "source": [
        "!pip install -qU \\\n",
        "  langchain==0.1.1 \\\n",
        "  langchain-community==0.0.13 \\\n",
        "  openai==0.27.7 \\\n",
        "  tiktoken==0.4.0 \\\n",
        "  pinecone-client==3.1.0 \\\n",
        "  pinecone-datasets==0.7.0 \\\n",
        "  pinecone-notebooks==0.1.1"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---\n",
        "\n",
        "🚨 _Note: the above `pip install` is formatted for Jupyter notebooks. If running elsewhere you may need to drop the `!`._\n",
        "\n",
        "---"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "RaVQA7J0JYbV"
      },
      "source": [
        "## Building the Knowledge Base\n",
        "\n",
        "We will download a pre-embedding dataset from `pinecone-datasets`. Allowing us to skip the embedding and preprocessing steps, if you'd rather work through those steps you can find the [full notebook here](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/langchain-retrieval-augmentation.ipynb)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 262
        },
        "id": "LeAmSrjvKJrV",
        "outputId": "2ec76d28-1f4b-4db4-a919-98ffa01fb8a0"
      },
      "outputs": [],
      "source": [
        "import pinecone_datasets\n",
        "\n",
        "dataset = pinecone_datasets.load_dataset('wikipedia-simple-text-embedding-ada-002-100K')\n",
        "dataset.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_WXZkwUbKuXK",
        "outputId": "0a67416e-1a91-44aa-daba-9f6b07a804a6"
      },
      "outputs": [],
      "source": [
        "len(dataset)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "jDjJHsHUKv28"
      },
      "source": [
        "We'll format the dataset ready for upsert and reduce what we use to a subset of the full dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bmUgxFlQK1Ow",
        "outputId": "d9fa544c-1353-4256-9df0-4941701b66c8"
      },
      "outputs": [],
      "source": [
        "# we drop sparse_values as they are not needed for this example\n",
        "dataset.documents.drop(['metadata'], axis=1, inplace=True)\n",
        "dataset.documents.rename(columns={'blob': 'metadata'}, inplace=True)\n",
        "# we will use rows of the dataset up to index 30_000\n",
        "dataset.documents.drop(dataset.documents.index[30_000:], inplace=True)\n",
        "len(dataset)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "QPUmWYSA43eC"
      },
      "source": [
        "Now we move on to initializing our Pinecone vector database."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Creating an Index\n",
        "\n",
        "Now the data is ready, we can set up our index to store it.\n",
        "\n",
        "We begin by initializing our connection to Pinecone. To do this we need a [free API key](https://app.pinecone.io)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.environ.get(\"PINECONE_API_KEY\"):\n",
        "    from pinecone_notebooks.colab import Authenticate\n",
        "    Authenticate()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pinecone import Pinecone\n",
        "\n",
        "api_key = os.environ.get(\"PINECONE_API_KEY\")\n",
        "\n",
        "# configure client\n",
        "pc = Pinecone(api_key=api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pinecone import ServerlessSpec\n",
        "\n",
        "cloud = os.environ.get('PINECONE_CLOUD') or 'aws'\n",
        "region = os.environ.get('PINECONE_REGION') or 'us-east-1'\n",
        "\n",
        "spec = ServerlessSpec(cloud=cloud, region=region)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "tags": [
          "parameters"
        ]
      },
      "outputs": [],
      "source": [
        "index_name = 'langchain-retrieval-augmentation-fast'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9pT9C4nW4vwo",
        "outputId": "6aea8f5a-9331-4c20-c967-989c668ac681"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "if index_name in pc.list_indexes().names():\n",
        "    pc.delete_index(index_name)\n",
        "\n",
        "# we create a new index\n",
        "pc.create_index(\n",
        "        index_name,\n",
        "        dimension=1536,  # dimensionality of text-embedding-ada-002\n",
        "        metric='dotproduct',\n",
        "        spec=spec\n",
        "    )\n",
        "\n",
        "# wait for index to be initialized\n",
        "while not pc.describe_index(index_name).status['ready']:\n",
        "    time.sleep(1)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "YgPUwd6REY6z"
      },
      "source": [
        "Then we connect to the new index:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RFydARw4EcoQ",
        "outputId": "92cbb1cf-bb70-4b3a-a481-ede66a5dc7b0"
      },
      "outputs": [],
      "source": [
        "index = pc.Index(index_name)\n",
        "# wait a moment for connection\n",
        "time.sleep(1)\n",
        "\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "0RqIF2mIDwFu"
      },
      "source": [
        "We should see that the new Pinecone index has a `total_vector_count` of `0`, as we haven't added any vectors yet.\n",
        "\n",
        "Now we upsert the data to Pinecone:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 99,
          "referenced_widgets": [
            "47188360f07a4c018bf7d2c5eb6f626c",
            "50d346f4ff614f8cb7793cce0a9c0916",
            "f06f0a851b6c434da5cd9ddb245de902",
            "3216e01669cc470894d25198a2099263",
            "47ecdc632edf4aeeb37239510aff8ced",
            "2ee36ac5673a4b74877dcf3fe63c5eb3",
            "f948308ac3414cfabbea8e2af917446f",
            "21045e189c5d49a082964eec2c5d8f6a",
            "4981784e865746a49a912188fb599985",
            "07c3ae73ab9b4c66813d1646ce6a320d",
            "6acfe82fd33043558667ac26141093f8",
            "75cef31e615d4851b029c3aacacf9c2d",
            "6b3b8cddec674c0bb00620e1c9128219",
            "54d772726617494d98e5ee1618db5cf0",
            "34b9b178e43a4cc1ba516c4f27ffd575",
            "0021290a0a3a44be911c4e94b7d25006",
            "7f7790181f72437cb10a88704d20f778",
            "62da3ff6ab334122aa0657a6f25417bb",
            "2d038c3eb946427297de35460a8d4ae2",
            "5f0fb5f8cfb7459d80f6a92bd0125dcb",
            "6e1f19542e2645ccad3a8c99bf6e030b",
            "7c834ff3591545f7809a46989c3a985b"
          ]
        },
        "id": "W-cIOoTWGY1R",
        "outputId": "0eaf686b-ee21-47ca-e818-f29fb768e3c1"
      },
      "outputs": [],
      "source": [
        "for batch in dataset.iter_documents(batch_size=100):\n",
        "    index.upsert(batch)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "XaF3daSxyCwB"
      },
      "source": [
        "We've now indexed everything. We can check the number of vectors in our index like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CaEBhsAM22M3",
        "outputId": "9718d994-c8e1-4ff3-ae1d-b57b9b1d3c46"
      },
      "outputs": [],
      "source": [
        "index.describe_index_stats()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "-8P2PryCy8W3"
      },
      "source": [
        "## Creating a Vector Store and Querying\n",
        "\n",
        "Now that we've build our index we can switch over to LangChain. We need to initialize a LangChain vector store using the same index we just built. For this we will also need a LangChain embedding object, which we initialize like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FvwwQA4qbcK9"
      },
      "outputs": [],
      "source": [
        "from langchain.embeddings.openai import OpenAIEmbeddings\n",
        "\n",
        "# get openai api key from platform.openai.com\n",
        "OPENAI_API_KEY = os.getenv('OPENAI_API_KEY') or 'OPENAI_API_KEY'\n",
        "\n",
        "model_name = 'text-embedding-ada-002'\n",
        "\n",
        "embed = OpenAIEmbeddings(\n",
        "    model=model_name,\n",
        "    openai_api_key=OPENAI_API_KEY\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "jKuedXN8bcfA"
      },
      "source": [
        "Now initialize the vector store:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qMXlvXOAyJHy"
      },
      "outputs": [],
      "source": [
        "from langchain.vectorstores import Pinecone\n",
        "\n",
        "text_field = \"text\"\n",
        "\n",
        "# switch back to normal index for langchain\n",
        "index = pc.Index(index_name)\n",
        "\n",
        "vectorstore = Pinecone(\n",
        "    index, embed.embed_query, text_field\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "h1Yg5mKse1bO"
      },
      "source": [
        "Now we can query the vector store directly using `vectorstore.similarity_search`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "COT5s7hcyPiq",
        "outputId": "4373a9bb-682f-4000-a491-2c948b9b44f2"
      },
      "outputs": [],
      "source": [
        "query = \"who was Benito Mussolini?\"\n",
        "\n",
        "vectorstore.similarity_search(\n",
        "    query,  # our search query\n",
        "    k=3  # return 3 most relevant docs\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ZCvtmREd0pdo"
      },
      "source": [
        "All of these are good, relevant results. But what can we do with this? There are many tasks, one of the most interesting (and well supported by LangChain) is called _\"Generative Question-Answering\"_ or GQA.\n",
        "\n",
        "## Generative Question-Answering\n",
        "\n",
        "In GQA we take the query as a question that is to be answered by a LLM, but the LLM must answer the question based on the information it is seeing being returned from the `vectorstore`.\n",
        "\n",
        "To do this we initialize a `RetrievalQA` object like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "moCvQR-p0Zsb"
      },
      "outputs": [],
      "source": [
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain.chains import RetrievalQA\n",
        "\n",
        "# completion llm\n",
        "llm = ChatOpenAI(\n",
        "    openai_api_key=OPENAI_API_KEY,\n",
        "    model_name='gpt-3.5-turbo',\n",
        "    temperature=0.0\n",
        ")\n",
        "\n",
        "qa = RetrievalQA.from_chain_type(\n",
        "    llm=llm,\n",
        "    chain_type=\"stuff\",\n",
        "    retriever=vectorstore.as_retriever()\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 106
        },
        "id": "KS9sa19K3LkQ",
        "outputId": "d7473cc3-6c7a-44c3-e219-12396eeb5580"
      },
      "outputs": [],
      "source": [
        "qa.run(query)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "0qf5e3xf3ggq"
      },
      "source": [
        "We can also include the sources of information that the LLM is using to answer our question. We can do this using a slightly different version of `RetrievalQA` called `RetrievalQAWithSourcesChain`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aYVMGDA13cTz"
      },
      "outputs": [],
      "source": [
        "from langchain.chains import RetrievalQAWithSourcesChain\n",
        "\n",
        "qa_with_sources = RetrievalQAWithSourcesChain.from_chain_type(\n",
        "    llm=llm,\n",
        "    chain_type=\"stuff\",\n",
        "    retriever=vectorstore.as_retriever()\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RXsVEh3S4ZJO",
        "outputId": "4150e3c6-3210-439b-9c85-389d8c20d02b"
      },
      "outputs": [],
      "source": [
        "qa_with_sources(query)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "nMRk_P3Q7l5J"
      },
      "source": [
        "Now we answer the question being asked, *and* return the source of this information being used by the LLM.\n",
        "\n",
        "Once done, we can delete the index to save resources."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vNkoIzQjjMTQ"
      },
      "outputs": [],
      "source": [
        "pc.delete_index(index_name)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ehJEn68qADoH"
      },
      "source": [
        "---"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0021290a0a3a44be911c4e94b7d25006": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "07c3ae73ab9b4c66813d1646ce6a320d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "21045e189c5d49a082964eec2c5d8f6a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2d038c3eb946427297de35460a8d4ae2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2ee36ac5673a4b74877dcf3fe63c5eb3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3216e01669cc470894d25198a2099263": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_07c3ae73ab9b4c66813d1646ce6a320d",
            "placeholder": "​",
            "style": "IPY_MODEL_6acfe82fd33043558667ac26141093f8",
            "value": " 30000/30000 [00:10&lt;00:00, 2929.37it/s]"
          }
        },
        "34b9b178e43a4cc1ba516c4f27ffd575": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6e1f19542e2645ccad3a8c99bf6e030b",
            "placeholder": "​",
            "style": "IPY_MODEL_7c834ff3591545f7809a46989c3a985b",
            "value": " 300/300 [00:00&lt;00:00, 2936.76it/s]"
          }
        },
        "47188360f07a4c018bf7d2c5eb6f626c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_50d346f4ff614f8cb7793cce0a9c0916",
              "IPY_MODEL_f06f0a851b6c434da5cd9ddb245de902",
              "IPY_MODEL_3216e01669cc470894d25198a2099263"
            ],
            "layout": "IPY_MODEL_47ecdc632edf4aeeb37239510aff8ced"
          }
        },
        "47ecdc632edf4aeeb37239510aff8ced": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4981784e865746a49a912188fb599985": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "50d346f4ff614f8cb7793cce0a9c0916": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2ee36ac5673a4b74877dcf3fe63c5eb3",
            "placeholder": "​",
            "style": "IPY_MODEL_f948308ac3414cfabbea8e2af917446f",
            "value": "sending upsert requests: 100%"
          }
        },
        "54d772726617494d98e5ee1618db5cf0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2d038c3eb946427297de35460a8d4ae2",
            "max": 300,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5f0fb5f8cfb7459d80f6a92bd0125dcb",
            "value": 300
          }
        },
        "5f0fb5f8cfb7459d80f6a92bd0125dcb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "62da3ff6ab334122aa0657a6f25417bb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6acfe82fd33043558667ac26141093f8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6b3b8cddec674c0bb00620e1c9128219": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7f7790181f72437cb10a88704d20f778",
            "placeholder": "​",
            "style": "IPY_MODEL_62da3ff6ab334122aa0657a6f25417bb",
            "value": "collecting async responses: 100%"
          }
        },
        "6e1f19542e2645ccad3a8c99bf6e030b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "75cef31e615d4851b029c3aacacf9c2d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6b3b8cddec674c0bb00620e1c9128219",
              "IPY_MODEL_54d772726617494d98e5ee1618db5cf0",
              "IPY_MODEL_34b9b178e43a4cc1ba516c4f27ffd575"
            ],
            "layout": "IPY_MODEL_0021290a0a3a44be911c4e94b7d25006"
          }
        },
        "7c834ff3591545f7809a46989c3a985b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7f7790181f72437cb10a88704d20f778": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f06f0a851b6c434da5cd9ddb245de902": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_21045e189c5d49a082964eec2c5d8f6a",
            "max": 30000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4981784e865746a49a912188fb599985",
            "value": 30000
          }
        },
        "f948308ac3414cfabbea8e2af917446f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}