hybrid_search/hybrid_search_inflation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yxwE82TZNkco"
      },
      "source": [
        "## Hybrid Search\n",
        "\n",
        "##### Note: This example requires a KDB.AI endpoint and API key. Sign up for a free [KDB.AI account](https://kdb.ai/get-started).\n",
        "\n",
        "KDB.AI hybrid search is a method of similarity search to increase the relevancy of results retrieved from the vector database. It combines two search methods: sparse vector search, and dense vector search.\n",
        "\n",
        "Sparse vector search uses the BM25 algorithm to find the most relevant keyword matches, while dense vector search finds the most semantically relevant matches.\n",
        "\n",
        "In KDB.AI, users can run sparse or dense search independently, or run hybrid search which runs both sparse and dense vector searches and then re-ranks to combine the results of each search based on a user defined \"weight\" value.\n",
        "\n",
        "In this sample we will use hybrid search over a Federal Reserve speech to extract chunks of the speech that are similar to a user's prompt. In this notebook we will chunk up the document into smaller subsections, create sparse and dense vectors of the chunks, store the vectors in the KDB.AI vector database, and then run dense search, sparse search, and hybrid search to retrieve the most relevant chunks to a user's query.\n",
        "\n",
        "Agenda:\n",
        "1. Dependencies, Imports & Setup\n",
        "2. Ingest & Chunk Data\n",
        "3. Generate Sparse & Dense Vectors for Each Chunk\n",
        "4. Define KDB.AI Session and Create Database\n",
        "5. Create KDB.AI Schema & Table\n",
        "6. Insert Data into KDB.AI Table\n",
        "7. Create Sparse and Dense Query Vectors\n",
        "8. Run Sparse, Dense, and Hybrid Searches\n",
        "\n",
        "[Inflation: Progress and the Path Ahead](https://www.federalreserve.gov/newsevents/speech/powell20230825a.htm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mDRvL7jiNkcp"
      },
      "source": [
        "## 1. Dependencies, Imports & Setup\n",
        "\n",
        "In order to successfully run this sample, note the following steps depending on where you are running this notebook:\n",
        "\n",
        "-***Run Locally / Private Environment:*** The [Setup](https://github.com/KxSystems/kdbai-samples/blob/main/README.md#setup) steps in the repository's `README.md` will guide you on prerequisites and how to run this with Jupyter.\n",
        "\n",
        "\n",
        "-***Colab / Hosted Environment:*** Open this notebook in Colab and run through the cells."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install kdbai_client"
      ],
      "metadata": {
        "id": "9YNwxlHZPKu3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KTt0VFTYNkcp"
      },
      "outputs": [],
      "source": [
        "!pip install sentence-transformers langchain langchain-community"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "FfoOD9FxNkcp"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import os\n",
        "from getpass import getpass\n",
        "import kdbai_client as kdbai\n",
        "import time\n",
        "from transformers import BertTokenizerFast\n",
        "from collections import Counter\n",
        "\n",
        "# Ignore Warnings\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "from langchain.document_loaders import TextLoader\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "IpEWr_EvNkcp"
      },
      "outputs": [],
      "source": [
        "### !!! Only run this cell if you need to download the data into your environment, for example in Colab\n",
        "### This downloads Federal Reserve Inflation speech data\n",
        "if os.path.exists(\"./data/inflation.txt\") == False:\n",
        "  !mkdir ./data\n",
        "  !wget -P ./data https://raw.githubusercontent.com/KxSystems/kdbai-samples/main/hybrid_search/data/inflation.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lrw6Iu0bNkcq"
      },
      "source": [
        "## 2. Ingest & Chunk Data\n",
        "Data is from Federal Reserve Chain Jerome H. Powell:\n",
        "\n",
        "[Inflation: Progress and the Path Ahead](https://www.federalreserve.gov/newsevents/speech/powell20230825a.htm)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "nN0GQDIANkcq"
      },
      "outputs": [],
      "source": [
        "### Load the documents we want to prompt an LLM about\n",
        "doc = TextLoader(\"data/inflation.txt\").load()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "G6DDD6ddNkcq"
      },
      "outputs": [],
      "source": [
        "### Chunk the documents into 500 character chunks using langchain's text splitter \"RucursiveCharacterTextSplitter\"\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "2Rjkeq_oNkcq"
      },
      "outputs": [],
      "source": [
        "### split_documents produces a list of all the chunks created\n",
        "pages = [p.page_content for p in text_splitter.split_documents(doc)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "J3FwRgq6Nkcq"
      },
      "outputs": [],
      "source": [
        "### Create a blank dataframe to store chunks and vectors in before insertion\n",
        "data = {\n",
        "    'ID':[],\n",
        "    'chunk': [],\n",
        "    'dense': [],\n",
        "    'sparse': []\n",
        "}\n",
        "\n",
        "# Create the DataFrame\n",
        "df = pd.DataFrame(data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8ghx6kYtNkcq"
      },
      "source": [
        "## 3. Generate Sparse & Dense Vectors for Each Chunk"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AhMYd5guNkcq"
      },
      "outputs": [],
      "source": [
        "### Tokenizer to create sparse vectors\n",
        "token = BertTokenizerFast.from_pretrained('bert-base-uncased')\n",
        "\n",
        "### Embedding model to be used to embed user input query\n",
        "from sentence_transformers import SentenceTransformer\n",
        "embedding_model = SentenceTransformer(\"all-MiniLM-L6-v2\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "IvjNAhrZNkcq",
        "outputId": "1717274d-7e2c-4241-fbc9-d5e3c562bd6d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  ID                                              chunk  \\\n",
              "0  0  At last year's Jackson Hole symposium, I deliv...   \n",
              "1  1  are confident that inflation is moving sustain...   \n",
              "2  2  Today I will review our progress so far and di...   \n",
              "3  3                    The Decline in Inflation So Far   \n",
              "4  4  The ongoing episode of high inflation initiall...   \n",
              "\n",
              "                                               dense  \\\n",
              "0  [-0.022856269031763077, -0.02936530113220215, ...   \n",
              "1  [0.011283153668045998, -0.030178586021065712, ...   \n",
              "2  [-0.03170400112867355, 0.01769343577325344, 0....   \n",
              "3  [0.003466668538749218, 0.007666163146495819, -...   \n",
              "4  [-0.02072943188250065, -0.055148035287857056, ...   \n",
              "\n",
              "                                              sparse  \n",
              "0  {2012: 2, 2197: 1, 2095: 3, 1005: 2, 1055: 2, ...  \n",
              "1  {2024: 1, 9657: 1, 2008: 1, 14200: 1, 2003: 1,...  \n",
              "2  {2651: 1, 1045: 2, 2097: 2, 3319: 1, 2256: 2, ...  \n",
              "3  {1996: 1, 6689: 1, 1999: 1, 14200: 1, 2061: 1,...  \n",
              "4  {1996: 7, 7552: 1, 2792: 1, 1997: 4, 2152: 1, ...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-45a96c07-5cf5-4939-96fd-3fce33e0ea29\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>chunk</th>\n",
              "      <th>dense</th>\n",
              "      <th>sparse</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>At last year's Jackson Hole symposium, I deliv...</td>\n",
              "      <td>[-0.022856269031763077, -0.02936530113220215, ...</td>\n",
              "      <td>{2012: 2, 2197: 1, 2095: 3, 1005: 2, 1055: 2, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>are confident that inflation is moving sustain...</td>\n",
              "      <td>[0.011283153668045998, -0.030178586021065712, ...</td>\n",
              "      <td>{2024: 1, 9657: 1, 2008: 1, 14200: 1, 2003: 1,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>Today I will review our progress so far and di...</td>\n",
              "      <td>[-0.03170400112867355, 0.01769343577325344, 0....</td>\n",
              "      <td>{2651: 1, 1045: 2, 2097: 2, 3319: 1, 2256: 2, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>The Decline in Inflation So Far</td>\n",
              "      <td>[0.003466668538749218, 0.007666163146495819, -...</td>\n",
              "      <td>{1996: 1, 6689: 1, 1999: 1, 14200: 1, 2061: 1,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>The ongoing episode of high inflation initiall...</td>\n",
              "      <td>[-0.02072943188250065, -0.055148035287857056, ...</td>\n",
              "      <td>{1996: 7, 7552: 1, 2792: 1, 1997: 4, 2152: 1, ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-45a96c07-5cf5-4939-96fd-3fce33e0ea29')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-45a96c07-5cf5-4939-96fd-3fce33e0ea29 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-45a96c07-5cf5-4939-96fd-3fce33e0ea29');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-ddf29a7e-c0d1-46e7-9788-e43c440f03bc\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-ddf29a7e-c0d1-46e7-9788-e43c440f03bc')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-ddf29a7e-c0d1-46e7-9788-e43c440f03bc button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 43,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 43,\n        \"samples\": [\n          \"37\",\n          \"24\",\n          \"25\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"chunk\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 43,\n        \"samples\": [\n          \"than was the case in recent decades.8 These changing dynamics may or may not persist, and this uncertainty underscores the need for agile policymaking.\",\n          \"activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).\",\n          \"But we are attentive to signs that the economy may not be cooling as expected. So far this year, GDP (gross domestic product) growth has come in above expectations and above its longer-run trend, and recent readings on consumer spending have been especially robust. In addition, after decelerating sharply over the past 18 months, the housing sector is showing signs of picking back up. Additional evidence of persistently above-trend growth could put further progress on inflation at risk and could\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"dense\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sparse\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "### Create sparse and dense vectors of each chunk, append to the dataframe\n",
        "\n",
        "id = 0\n",
        "for chunk in pages:\n",
        "    ### Create the dense query vector\n",
        "    dense_chunk = [embedding_model.encode(chunk).tolist()]\n",
        "\n",
        "    ### Create the sparse query vector\n",
        "    sparse_chunk = [dict(Counter(y)) for y in token([chunk], padding=True,max_length=None)['input_ids']]\n",
        "    sparse_chunk[0].pop(101);sparse_chunk[0].pop(102);\n",
        "\n",
        "    new_row_df = pd.DataFrame([{\"ID\": str(id), \"chunk\": chunk, \"dense\": dense_chunk[0], \"sparse\": sparse_chunk[0]}])\n",
        "    df = pd.concat([df, new_row_df], ignore_index=True)\n",
        "    id += int(1)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z3i844C3Nkcq"
      },
      "source": [
        "## 4. Define KDB.AI Session & Database\n",
        "KDB.AI comes in two offerings:\n",
        "\n",
        "KDB.AI Cloud - For experimenting with smaller generative AI projects with a vector database in our cloud.\n",
        "KDB.AI Server - For evaluating large scale generative AI applications on-premises or on your own cloud provider.\n",
        "Depending on which you use there will be different setup steps and connection details required.\n",
        "\n",
        "Option 1. KDB.AI Cloud\n",
        "To use KDB.AI Cloud, you will need two session details - a URL endpoint and an API key. To get these you can sign up for free here.\n",
        "\n",
        "You can connect to a KDB.AI Cloud session using kdbai.Session and passing the session URL endpoint and API key details from your KDB.AI Cloud portal.\n",
        "\n",
        "If the environment variables KDBAI_ENDPOINTS and KDBAI_API_KEY exist on your system containing your KDB.AI Cloud portal details, these variables will automatically be used to connect. If these do not exist, it will prompt you to enter your KDB.AI Cloud portal session URL endpoint and API key details."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Kdsxz7KNkcq"
      },
      "source": [
        "### Option 1. KDB.AI Cloud"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eVOZCLI3Nkcr"
      },
      "outputs": [],
      "source": [
        "#Set up KDB.AI endpoint and API key\n",
        "KDBAI_ENDPOINT = (\n",
        "    os.environ[\"KDBAI_ENDPOINT\"]\n",
        "    if \"KDBAI_ENDPOINT\" in os.environ\n",
        "    else input(\"KDB.AI endpoint: \")\n",
        ")\n",
        "KDBAI_API_KEY = (\n",
        "    os.environ[\"KDBAI_API_KEY\"]\n",
        "    if \"KDBAI_API_KEY\" in os.environ\n",
        "    else getpass(\"KDB.AI API key: \")\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x-zMHeArNkcr"
      },
      "outputs": [],
      "source": [
        "### Start Session with KDB.AI Cloud\n",
        "session = kdbai.Session(api_key=KDBAI_API_KEY, endpoint=KDBAI_ENDPOINT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NN-GzyQ4Nkcr"
      },
      "source": [
        "### Option 2. KDB.AI Server\n",
        "To use KDB.AI Server, you will need download and run your own container. To do this, you will first need to sign up for free here.\n",
        "\n",
        "You will receive an email with the required license file and bearer token needed to download your instance. Follow instructions in the signup email to get your session up and running.\n",
        "\n",
        "Once the setup steps are complete you can then connect to your KDB.AI Server session using kdbai.Session and passing your local endpoint."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j-b-6SqvNkcr"
      },
      "outputs": [],
      "source": [
        "### start session with KDB.AI Server\n",
        "#session = kdbai.Session(endpoint=\"http://localhost:8082\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Verify Defined Databases\n",
        "\n",
        "We can check our connection using the `session.databases()` function.\n",
        "This will return a list of all the databases we have defined in our vector database thus far.\n",
        "This should return a \"default\" database along with any other databases you have already created."
      ],
      "metadata": {
        "id": "695PM-c1RRCC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "session.databases()"
      ],
      "metadata": {
        "id": "ewaBpPAiRU4y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create a Database called \"myDatabase\""
      ],
      "metadata": {
        "id": "J7UlCvo4RXMr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# ensure no database called \"myDatabase\" exists\n",
        "try:\n",
        "    session.database(\"myDatabase\").drop()\n",
        "except kdbai.KDBAIException:\n",
        "    pass"
      ],
      "metadata": {
        "id": "REbU5B15Rajk"
      },
      "execution_count": 152,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the database\n",
        "db = session.create_database(\"myDatabase\")"
      ],
      "metadata": {
        "id": "CsCfxNp-Rcbm"
      },
      "execution_count": 153,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RCcAYgY9Nkcr"
      },
      "source": [
        "## 5. Create Schema, Indexes and KDB.AI Table\n",
        "\n",
        "Now, let us define the schema that will be used to create the KDB.AI table.\n",
        "\n",
        "\"ID\" and \"chunk\" columns will hold the unique identifier and raw text chunk.\n",
        "\n",
        "sparse and dense columns will hold the respective sparse and dense vectors."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "schema = [\n",
        "  {\"name\": \"ID\", \"type\": \"str\"},\n",
        "  {\"name\": \"chunk\", \"type\": \"str\"},\n",
        "  {\n",
        "      \"name\":\"sparse\",\n",
        "      \"type\":\"general\",\n",
        "  },\n",
        "  {\n",
        "      \"name\":\"dense\",\n",
        "      \"type\":\"float64s\",\n",
        "  },\n",
        "]"
      ],
      "metadata": {
        "id": "jvSLKsYwSB0V"
      },
      "execution_count": 154,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Define the indexes\n",
        "In this example, we have two indexes, one for dense search and one for sparse search (bm25).\n",
        "\n",
        "- dense_index: uses a flat index type, with 384 dims and Euclidean Distance search metric\n",
        "\n",
        "- sparse_index: uses bm25 search type. We also define the \"b\" and \"k\" parameters. These parameters can be adjusted at runtime, enabling the hyperparameter tuning for term saturation and document length impact on relevance. This will be discussed further during a later example."
      ],
      "metadata": {
        "id": "_AB9ASDXuOVa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the index\n",
        "indexes = [\n",
        "    {\n",
        "        'type': 'flat',\n",
        "        'name': 'dense_index',\n",
        "        'column': 'dense',\n",
        "        'params': {'dims': 384, 'metric': \"L2\"},\n",
        "    },\n",
        "    {\n",
        "        'type': 'bm25',\n",
        "        'name': 'sparse_index',\n",
        "        'column': 'sparse',\n",
        "        'params': {'k': 1.25, 'b': 0.75},\n",
        "    },\n",
        "]"
      ],
      "metadata": {
        "id": "sF4HOazLSsFf"
      },
      "execution_count": 155,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# List all of the tables in the db\n",
        "db.tables"
      ],
      "metadata": {
        "id": "34Yz9TxxUd1m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 157,
      "metadata": {
        "id": "FyFJdaXRNkcr"
      },
      "outputs": [],
      "source": [
        "# First ensure the table does not already exist\n",
        "try:\n",
        "    db.table(\"inflation\").drop()\n",
        "except kdbai.KDBAIException:\n",
        "    pass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 158,
      "metadata": {
        "id": "A7_D0AWHNkcr"
      },
      "outputs": [],
      "source": [
        "# Create the table with the defined schema and indexes from above\n",
        "table = db.create_table(table=\"inflation\", schema=schema, indexes=indexes)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "db.tables"
      ],
      "metadata": {
        "id": "zdFqjFLFWaJk",
        "outputId": "26927cbe-caa6-48c4-f36f-62355601fae6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 159,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[KDBAI table \"inflation\"]"
            ]
          },
          "metadata": {},
          "execution_count": 159
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "table.indexes"
      ],
      "metadata": {
        "id": "BgZCCewrXhrq",
        "outputId": "ac44f3f5-f512-4801-c0b8-cd8ae1d16e4f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 160,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'name': 'dense_index',\n",
              "  'type': 'flat',\n",
              "  'column': 'dense',\n",
              "  'params': {'metric': 'L2', 'dims': 384}},\n",
              " {'name': 'sparse_index',\n",
              "  'type': 'bm25',\n",
              "  'column': 'sparse',\n",
              "  'params': {'sparse': True, 'k': 1.25, 'b': 0.75}}]"
            ]
          },
          "metadata": {},
          "execution_count": 160
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CJaNUVunNkcr"
      },
      "source": [
        "## 6. Insert data into the KDB.AI Table"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 161,
      "metadata": {
        "id": "qKx534zGNkcr",
        "outputId": "03653123-bfa0-4d6b-f94c-55313cbb8e2a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'rowsInserted': 43}"
            ]
          },
          "metadata": {},
          "execution_count": 161
        }
      ],
      "source": [
        "### Insert the dataframe into the KDB.AI table\n",
        "table.insert(df)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "table.query()"
      ],
      "metadata": {
        "id": "aHgSC02eWfsS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-_P_8uF5Nkcr"
      },
      "source": [
        "## 7. Create Sparse and Dense Query Vectors"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 163,
      "metadata": {
        "id": "c_cMHq9HNkcr"
      },
      "outputs": [],
      "source": [
        "query = '12-month basis'\n",
        "\n",
        "### Create the dense query vector\n",
        "dense_query = [embedding_model.encode(query).tolist()]\n",
        "\n",
        "### Create the sparse query vector\n",
        "sparse_query = [dict(Counter(y)) for y in token([query], padding=True,max_length=None)['input_ids']]\n",
        "sparse_query[0].pop(101);sparse_query[0].pop(102);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Ojk34UtNkcr"
      },
      "source": [
        "## 8. Run Sparse, Dense, and Hybrid Searches"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 164,
      "metadata": {
        "id": "zGRi55vBNkcr"
      },
      "outputs": [],
      "source": [
        "### Adjust display settings so we can see full output\n",
        "pd.set_option('display.max_colwidth', None)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "### Type 1 - dense search\n",
        "table.search(vectors={\"dense_index\":dense_query}, n=5)[0][['ID','chunk']]"
      ],
      "metadata": {
        "id": "YPQ1yHG7Wtz4",
        "outputId": "753c8b50-1007-4829-cfe4-8e5f14577a0f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 345
        }
      },
      "execution_count": 165,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   ID  \\\n",
              "0   9   \n",
              "1  35   \n",
              "2  29   \n",
              "3  23   \n",
              "4  24   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                chunk  \n",
              "0                                                                                                                                                                                                                                                                                                                                                                    coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.  \n",
              "1  That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.  \n",
              "2                                                                                                                                                                                                                                                                                              Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).  \n",
              "3        Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic  \n",
              "4                                                                                                                                                                                                                                                                                   activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a3a7be99-8461-45b3-b572-e1bf98314da4\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>chunk</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>9</td>\n",
              "      <td>coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>35</td>\n",
              "      <td>That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>29</td>\n",
              "      <td>Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>23</td>\n",
              "      <td>Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>24</td>\n",
              "      <td>activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a3a7be99-8461-45b3-b572-e1bf98314da4')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a3a7be99-8461-45b3-b572-e1bf98314da4 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a3a7be99-8461-45b3-b572-e1bf98314da4');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-e8210eab-e32b-40c8-9f39-19f80247489d\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e8210eab-e32b-40c8-9f39-19f80247489d')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-e8210eab-e32b-40c8-9f39-19f80247489d button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"table\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"35\",\n          \"24\",\n          \"29\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"chunk\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.\",\n          \"activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).\",\n          \"Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 165
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 166,
      "metadata": {
        "id": "Vw8WUHn7Nkcr",
        "outputId": "8be6c78c-c3cb-427c-9375-ecdec25d9026",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 379
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   ID  \\\n",
              "0  14   \n",
              "1   8   \n",
              "2   6   \n",
              "3   9   \n",
              "4  23   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              chunk  \n",
              "0                                                                                                             Similar dynamics are playing out for core goods inflation overall. As they do, the effects of monetary restraint should show through more fully over time. Core goods prices fell the past two months, but on a 12-month basis, core goods inflation remains well above its pre-pandemic level. Sustained progress is needed, and restrictive monetary policy is called for to achieve that progress.  \n",
              "1     On a 12-month basis, core PCE inflation peaked at 5.4 percent in February 2022 and declined gradually to 4.3 percent in July (figure 1, panel B). The lower monthly readings for core inflation in June and July were welcome, but two months of good data are only the beginning of what it will take to build confidence that inflation is moving down sustainably toward our goal. We can't yet know the extent to which these lower readings will continue or where underlying inflation will settle over  \n",
              "2  On a 12-month basis, U.S. total, or \"headline,\" PCE (personal consumption expenditures) inflation peaked at 7 percent in June 2022 and declined to 3.3 percent as of July, following a trajectory roughly in line with global trends (figure 1, panel A).1 The effects of Russia's war against Ukraine have been a primary driver of the changes in headline inflation around the world since early 2022. Headline inflation is what households and businesses experience most directly, so this decline is very  \n",
              "3                                                                                                                                                                                                                                                                                                                                                                  coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.  \n",
              "4      Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-48507617-38e8-4451-b7d1-1f592afdfea0\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>chunk</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>14</td>\n",
              "      <td>Similar dynamics are playing out for core goods inflation overall. As they do, the effects of monetary restraint should show through more fully over time. Core goods prices fell the past two months, but on a 12-month basis, core goods inflation remains well above its pre-pandemic level. Sustained progress is needed, and restrictive monetary policy is called for to achieve that progress.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8</td>\n",
              "      <td>On a 12-month basis, core PCE inflation peaked at 5.4 percent in February 2022 and declined gradually to 4.3 percent in July (figure 1, panel B). The lower monthly readings for core inflation in June and July were welcome, but two months of good data are only the beginning of what it will take to build confidence that inflation is moving down sustainably toward our goal. We can't yet know the extent to which these lower readings will continue or where underlying inflation will settle over</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>6</td>\n",
              "      <td>On a 12-month basis, U.S. total, or \"headline,\" PCE (personal consumption expenditures) inflation peaked at 7 percent in June 2022 and declined to 3.3 percent as of July, following a trajectory roughly in line with global trends (figure 1, panel A).1 The effects of Russia's war against Ukraine have been a primary driver of the changes in headline inflation around the world since early 2022. Headline inflation is what households and businesses experience most directly, so this decline is very</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>23</td>\n",
              "      <td>Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-48507617-38e8-4451-b7d1-1f592afdfea0')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-48507617-38e8-4451-b7d1-1f592afdfea0 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-48507617-38e8-4451-b7d1-1f592afdfea0');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-aaf9283b-0609-40ad-9c53-c5174c22228a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-aaf9283b-0609-40ad-9c53-c5174c22228a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-aaf9283b-0609-40ad-9c53-c5174c22228a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"table\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"8\",\n          \"23\",\n          \"6\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"chunk\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"On a 12-month basis, core PCE inflation peaked at 5.4 percent in February 2022 and declined gradually to 4.3 percent in July (figure 1, panel B). The lower monthly readings for core inflation in June and July were welcome, but two months of good data are only the beginning of what it will take to build confidence that inflation is moving down sustainably toward our goal. We can't yet know the extent to which these lower readings will continue or where underlying inflation will settle over\",\n          \"Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well\\u2014by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic\",\n          \"On a 12-month basis, U.S. total, or \\\"headline,\\\" PCE (personal consumption expenditures) inflation peaked at 7 percent in June 2022 and declined to 3.3 percent as of July, following a trajectory roughly in line with global trends (figure 1, panel A).1 The effects of Russia's war against Ukraine have been a primary driver of the changes in headline inflation around the world since early 2022. Headline inflation is what households and businesses experience most directly, so this decline is very\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 166
        }
      ],
      "source": [
        "### Type 2 - sparse search\n",
        "table.search(vectors={\"sparse_index\":sparse_query}, n=5)[0][['ID','chunk']]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jpmHB_EONkcr"
      },
      "source": [
        "**After comparing the sparse search and dense search results based on the query of \"12-month basis\", we see that while both return relevant results, the sparse search is returns several chunks that contain specific references to \"12-month basis\".**\n",
        "\n",
        "**This search example shows the advantage of having a sparse search when interested in specific terms.**\n",
        "\n",
        "Let's run a hybrid search to combine the results:"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "table.search(\n",
        "    vectors={\"sparse_index\": sparse_query,\"dense_index\": dense_query},\n",
        "    index_params={\"sparse_index\":{'weight':0.5} ,\"dense_index\":{'weight':0.5}},\n",
        "    n=5\n",
        ")[0][['ID','chunk']]"
      ],
      "metadata": {
        "id": "TuVbDuTUphyi",
        "outputId": "f04071a8-8d06-45a2-d747-904884ee44d5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 379
        }
      },
      "execution_count": 167,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   ID  \\\n",
              "0   9   \n",
              "1  14   \n",
              "2  23   \n",
              "3   8   \n",
              "4  35   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                chunk  \n",
              "0                                                                                                                                                                                                                                                                                                                                                                    coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.  \n",
              "1                                                                                                               Similar dynamics are playing out for core goods inflation overall. As they do, the effects of monetary restraint should show through more fully over time. Core goods prices fell the past two months, but on a 12-month basis, core goods inflation remains well above its pre-pandemic level. Sustained progress is needed, and restrictive monetary policy is called for to achieve that progress.  \n",
              "2        Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic  \n",
              "3       On a 12-month basis, core PCE inflation peaked at 5.4 percent in February 2022 and declined gradually to 4.3 percent in July (figure 1, panel B). The lower monthly readings for core inflation in June and July were welcome, but two months of good data are only the beginning of what it will take to build confidence that inflation is moving down sustainably toward our goal. We can't yet know the extent to which these lower readings will continue or where underlying inflation will settle over  \n",
              "4  That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7a9951e4-1f85-40d4-a37f-3791ca371533\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>chunk</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>9</td>\n",
              "      <td>coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>14</td>\n",
              "      <td>Similar dynamics are playing out for core goods inflation overall. As they do, the effects of monetary restraint should show through more fully over time. Core goods prices fell the past two months, but on a 12-month basis, core goods inflation remains well above its pre-pandemic level. Sustained progress is needed, and restrictive monetary policy is called for to achieve that progress.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>23</td>\n",
              "      <td>Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8</td>\n",
              "      <td>On a 12-month basis, core PCE inflation peaked at 5.4 percent in February 2022 and declined gradually to 4.3 percent in July (figure 1, panel B). The lower monthly readings for core inflation in June and July were welcome, but two months of good data are only the beginning of what it will take to build confidence that inflation is moving down sustainably toward our goal. We can't yet know the extent to which these lower readings will continue or where underlying inflation will settle over</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>35</td>\n",
              "      <td>That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7a9951e4-1f85-40d4-a37f-3791ca371533')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7a9951e4-1f85-40d4-a37f-3791ca371533 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7a9951e4-1f85-40d4-a37f-3791ca371533');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3a4a0e18-ffbb-45cf-be67-910038a8d437\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3a4a0e18-ffbb-45cf-be67-910038a8d437')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3a4a0e18-ffbb-45cf-be67-910038a8d437 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \")[0][['ID','chunk']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"14\",\n          \"35\",\n          \"23\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"chunk\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"Similar dynamics are playing out for core goods inflation overall. As they do, the effects of monetary restraint should show through more fully over time. Core goods prices fell the past two months, but on a 12-month basis, core goods inflation remains well above its pre-pandemic level. Sustained progress is needed, and restrictive monetary policy is called for to achieve that progress.\",\n          \"That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.\",\n          \"Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well\\u2014by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 167
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7o-kqB4tNkcs"
      },
      "source": [
        "##### Hybrid Search with Sparse Bias, Sparse 'weight = 0.9'"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "table.search(\n",
        "    vectors={\"sparse_index\": sparse_query,\"dense_index\": dense_query},\n",
        "    index_params={\"sparse_index\":{'weight':0.9} ,\"dense_index\":{'weight':0.1}},\n",
        "    n=5\n",
        ")[0][['ID','chunk']]"
      ],
      "metadata": {
        "id": "voHcv_X7qEwe",
        "outputId": "38f5f733-576e-49b2-b3b7-06965a912559",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 379
        }
      },
      "execution_count": 168,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   ID  \\\n",
              "0  14   \n",
              "1   8   \n",
              "2   9   \n",
              "3   6   \n",
              "4  23   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              chunk  \n",
              "0                                                                                                             Similar dynamics are playing out for core goods inflation overall. As they do, the effects of monetary restraint should show through more fully over time. Core goods prices fell the past two months, but on a 12-month basis, core goods inflation remains well above its pre-pandemic level. Sustained progress is needed, and restrictive monetary policy is called for to achieve that progress.  \n",
              "1     On a 12-month basis, core PCE inflation peaked at 5.4 percent in February 2022 and declined gradually to 4.3 percent in July (figure 1, panel B). The lower monthly readings for core inflation in June and July were welcome, but two months of good data are only the beginning of what it will take to build confidence that inflation is moving down sustainably toward our goal. We can't yet know the extent to which these lower readings will continue or where underlying inflation will settle over  \n",
              "2                                                                                                                                                                                                                                                                                                                                                                  coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.  \n",
              "3  On a 12-month basis, U.S. total, or \"headline,\" PCE (personal consumption expenditures) inflation peaked at 7 percent in June 2022 and declined to 3.3 percent as of July, following a trajectory roughly in line with global trends (figure 1, panel A).1 The effects of Russia's war against Ukraine have been a primary driver of the changes in headline inflation around the world since early 2022. Headline inflation is what households and businesses experience most directly, so this decline is very  \n",
              "4      Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-92652fb4-0f2b-4683-9c45-1f8a225cc217\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>chunk</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>14</td>\n",
              "      <td>Similar dynamics are playing out for core goods inflation overall. As they do, the effects of monetary restraint should show through more fully over time. Core goods prices fell the past two months, but on a 12-month basis, core goods inflation remains well above its pre-pandemic level. Sustained progress is needed, and restrictive monetary policy is called for to achieve that progress.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8</td>\n",
              "      <td>On a 12-month basis, core PCE inflation peaked at 5.4 percent in February 2022 and declined gradually to 4.3 percent in July (figure 1, panel B). The lower monthly readings for core inflation in June and July were welcome, but two months of good data are only the beginning of what it will take to build confidence that inflation is moving down sustainably toward our goal. We can't yet know the extent to which these lower readings will continue or where underlying inflation will settle over</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>9</td>\n",
              "      <td>coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>On a 12-month basis, U.S. total, or \"headline,\" PCE (personal consumption expenditures) inflation peaked at 7 percent in June 2022 and declined to 3.3 percent as of July, following a trajectory roughly in line with global trends (figure 1, panel A).1 The effects of Russia's war against Ukraine have been a primary driver of the changes in headline inflation around the world since early 2022. Headline inflation is what households and businesses experience most directly, so this decline is very</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>23</td>\n",
              "      <td>Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-92652fb4-0f2b-4683-9c45-1f8a225cc217')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-92652fb4-0f2b-4683-9c45-1f8a225cc217 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-92652fb4-0f2b-4683-9c45-1f8a225cc217');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-5128d6a3-2f72-49e3-9d97-eca6a969b1dc\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-5128d6a3-2f72-49e3-9d97-eca6a969b1dc')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-5128d6a3-2f72-49e3-9d97-eca6a969b1dc button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \")[0][['ID','chunk']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"8\",\n          \"23\",\n          \"9\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"chunk\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"On a 12-month basis, core PCE inflation peaked at 5.4 percent in February 2022 and declined gradually to 4.3 percent in July (figure 1, panel B). The lower monthly readings for core inflation in June and July were welcome, but two months of good data are only the beginning of what it will take to build confidence that inflation is moving down sustainably toward our goal. We can't yet know the extent to which these lower readings will continue or where underlying inflation will settle over\",\n          \"Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well\\u2014by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic\",\n          \"coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 168
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ej2-k4N5Nkcs"
      },
      "source": [
        "##### Hybrid Search with Dense Bias: Dense 'weight = 0.9'"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "table.search(\n",
        "    vectors={\"sparse_index\": sparse_query,\"dense_index\": dense_query},\n",
        "    index_params={\"sparse_index\":{'weight':0.1} ,\"dense_index\":{'weight':0.9}},\n",
        "    n=5\n",
        ")[0][['ID','chunk']]"
      ],
      "metadata": {
        "id": "SJRN8D__qQTd",
        "outputId": "d40ffb69-c03f-4a20-a12e-4746ac6d0ca0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 345
        }
      },
      "execution_count": 169,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   ID  \\\n",
              "0   9   \n",
              "1  35   \n",
              "2  29   \n",
              "3  23   \n",
              "4  24   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                chunk  \n",
              "0                                                                                                                                                                                                                                                                                                                                                                    coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.  \n",
              "1  That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.  \n",
              "2                                                                                                                                                                                                                                                                                              Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).  \n",
              "3        Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic  \n",
              "4                                                                                                                                                                                                                                                                                   activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-dff658aa-f433-4120-ade7-506e477cff64\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>chunk</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>9</td>\n",
              "      <td>coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>35</td>\n",
              "      <td>That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>29</td>\n",
              "      <td>Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>23</td>\n",
              "      <td>Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>24</td>\n",
              "      <td>activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-dff658aa-f433-4120-ade7-506e477cff64')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-dff658aa-f433-4120-ade7-506e477cff64 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-dff658aa-f433-4120-ade7-506e477cff64');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a06d8c2e-aee4-40bc-a865-8ec74e8f89f1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a06d8c2e-aee4-40bc-a865-8ec74e8f89f1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a06d8c2e-aee4-40bc-a865-8ec74e8f89f1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \")[0][['ID','chunk']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"35\",\n          \"24\",\n          \"29\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"chunk\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.\",\n          \"activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).\",\n          \"Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 169
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9e8REO8GNkcs"
      },
      "source": [
        "### Sparse Search Hyperparameter Optimization\n",
        "#### Dynamic Testing /  Override of b and k within index_params\n",
        "\n",
        "Depending on the use-case, it could be beneficial to tune the underlying parameters of sparse search in order to increase relevancy of retrieved data. KDB.AI offers developers the ability to customize the 'b' and 'k' parameters during runtime, ensuring flexibility in sparse search implementation.\n",
        "\n",
        "**b: (values 0 to 1, defaults to 0.75)**\n",
        "<br>Document length impact on relevance\n",
        "<br>In general the more specific a document is the less likely length will detrimentally impact relevance so b should be low. For general documents that cover multiple topics at a high level consider using a higher value of b.\n",
        "<br>\n",
        "<br>**k: (values 0 to 3, defaults to 1.2)**\n",
        "<br>Term saturation\n",
        "<br>How much more relevant do additional instances of  a term make a document. The lower k, the faster term saturation occurs, (i.e. additional terms do not count as much)."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "table.search(\n",
        "    vectors={\"sparse_index\": sparse_query,\"dense_index\": dense_query},\n",
        "    index_params={\"sparse_index\":{'weight':0.1,'b':0.1, 'k':3} ,\"dense_index\":{'weight':0.9}},\n",
        "    n=5\n",
        ")[0][['ID','chunk']]"
      ],
      "metadata": {
        "id": "D7opgiVhrykw",
        "outputId": "d9371e1b-b616-49eb-94f7-1065bffd3bf0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 345
        }
      },
      "execution_count": 174,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   ID  \\\n",
              "0   9   \n",
              "1  35   \n",
              "2  29   \n",
              "3  23   \n",
              "4  24   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                chunk  \n",
              "0                                                                                                                                                                                                                                                                                                                                                                    coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.  \n",
              "1  That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.  \n",
              "2                                                                                                                                                                                                                                                                                              Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).  \n",
              "3        Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic  \n",
              "4                                                                                                                                                                                                                                                                                   activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-dcb271d4-80a7-4a7d-bcf3-7f58347fc19c\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>chunk</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>9</td>\n",
              "      <td>coming quarters. Twelve-month core inflation is still elevated, and there is substantial further ground to cover to get back to price stability.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>35</td>\n",
              "      <td>That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>29</td>\n",
              "      <td>Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>23</td>\n",
              "      <td>Restrictive monetary policy has tightened financial conditions, supporting the expectation of below-trend growth.5 Since last year's symposium, the two-year real yield is up about 250 basis points, and longer-term real yields are higher as well—by nearly 150 basis points.6 Beyond changes in interest rates, bank lending standards have tightened, and loan growth has slowed sharply.7 Such a tightening of broad financial conditions typically contributes to a slowing in the growth of economic</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>24</td>\n",
              "      <td>activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-dcb271d4-80a7-4a7d-bcf3-7f58347fc19c')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-dcb271d4-80a7-4a7d-bcf3-7f58347fc19c button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-dcb271d4-80a7-4a7d-bcf3-7f58347fc19c');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3fd0d1cc-db87-4aa6-9e19-64aa090b14d8\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3fd0d1cc-db87-4aa6-9e19-64aa090b14d8')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3fd0d1cc-db87-4aa6-9e19-64aa090b14d8 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \")[0][['ID','chunk']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"35\",\n          \"24\",\n          \"29\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"chunk\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"That assessment is further complicated by uncertainty about the duration of the lags with which monetary tightening affects economic activity and especially inflation. Since the symposium a year ago, the Committee has raised the policy rate by 300 basis points, including 100 basis points over the past seven months. And we have substantially reduced the size of our securities holdings. The wide range of estimates of these lags suggests that there may be significant further drag in the pipeline.\",\n          \"activity, and there is evidence of that in this cycle as well. For example, growth in industrial production has slowed, and the amount spent on residential investment has declined in each of the past five quarters (figure 4).\",\n          \"Total hours worked has been flat over the past six months, and the average workweek has declined to the lower end of its pre-pandemic range, reflecting a gradual normalization in labor market conditions (figure 5).\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 174
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3nbR-YyzNkcw"
      },
      "source": [
        "**Additionally, upon the insertion of new data into the KDB.AI table, all underlying BM25 statistics are updated. This means that when new data is added, the BM25 scoring is updated and aligns with the all sparse data when a sparse seach is run.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tS88mmhFNkcx"
      },
      "source": [
        "### Delete the KDB.AI Table\n",
        "Once finished with the table, it is best practice to drop it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6bs9bLcFNkcx"
      },
      "outputs": [],
      "source": [
        "table.drop()\n",
        "db.drop()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NBt1BOb-Nkcx"
      },
      "source": [
        "#### Take Our Survey\n",
        "We hope you found this sample helpful! Your feedback is important to us, and we would appreciate it if you could take a moment to fill out our brief survey. Your input helps us improve our content.\n",
        "\n",
        "Take the [Survey](https://delighted.com/t/U2RoT32R)\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.8"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}