05_reverse_hyde.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Improve Semantic Similarity with Reverse HYDE\n",
    "\n",
    "It is common that the documents that we want to retrieve are longer than the users' queries and have different formats. To increase the accuracy of the **r*etrieval of the documents based on the users' queries, we will generate hypothetical potential queries from each document and use them as vector embeddings to the documents - AKA Reverse Hyde.\n",
    "\n",
    "Please note that the original [Hyde technique](https://arxiv.org/abs/2212.10496) processed the incoming queries of the users, and generated the hypothetical documents from these queries, and then used these hypothetical documents to retrive the real documents. In the reverse HYDE, the processing is done when indexing the documents and not in retrival time. Therefore, the latency of the query is not affectd.\n",
    "\n",
    "* [Reverse Hyde Implementation](#reverse-hyde-implementation)\n",
    "* [Enriching Vector Database with Reverse Hyde Output](#enriching-vector-database-with-reverse-hyde-output)\n",
    "* [Query the Enriched Index](#query-the-enriched-index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visual improvements\n",
    "\n",
    "We will use [rich library](https://github.com/Textualize/rich) to make the output more readable, and supress warning messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rich.console import Console\n",
    "from rich_theme_manager import Theme, ThemeManager\n",
    "import pathlib\n",
    "\n",
    "theme_dir = pathlib.Path(\"themes\")\n",
    "theme_manager = ThemeManager(theme_dir=theme_dir)\n",
    "dark = theme_manager.get(\"dark\")\n",
    "\n",
    "# Create a console with the dark theme\n",
    "console = Console(theme=dark)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "# Suppress warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reverse HYDE Implementation\n",
    "\n",
    "We will create a class that will generate and hypotherical questions and also retrieve the document by calculating the semantic similarity matching. In a real application, we can use a vector database for the embedding vector storage, indexing and retrieval. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "from typing import List, Dict\n",
    "import numpy as np\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReverseHyde:\n",
    "    def __init__(self, api_key: str):\n",
    "        openai.api_key = api_key\n",
    "        self.model = \"text-embedding-ada-002\"\n",
    "\n",
    "    def get_embedding(self, text: str) -> List[float]:\n",
    "        client = openai.OpenAI()\n",
    "        response = client.embeddings.create(input=text, model=self.model)\n",
    "        return response.data[0].embedding\n",
    "\n",
    "    def generate_reverse_hyde(self, chunk: str, n: int = 3) -> List[str]:\n",
    "        prompt = f\"\"\"\n",
    "        \n",
    "Given the following text chunk, generate {n} different questions that this chunk would be a good answer to:\n",
    "\n",
    "Chunk: {chunk}\n",
    "\n",
    "Questions (enumarate the questions with 1. 2., etc.):\n",
    "\"\"\"\n",
    "\n",
    "        client = openai.OpenAI()\n",
    "        response = client.chat.completions.create(\n",
    "            model=\"gpt-3.5-turbo\",\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
    "                {\"role\": \"user\", \"content\": prompt}\n",
    "            ],\n",
    "            max_tokens=100,\n",
    "            n=1,\n",
    "            stop=None,\n",
    "            temperature=0.7,\n",
    "        )\n",
    "\n",
    "        \n",
    "        questions = response.choices[0].message.content.strip().split('\\n')\n",
    "        return [q.split('. ', 1)[1] for q in questions if '. ' in q]\n",
    "\n",
    "    def process_chunks(self, chunks: List[str], n: int = 3) -> Dict[str, List[str]]:\n",
    "        processed_chunks = {}\n",
    "        for chunk in chunks:\n",
    "            processed_chunks[chunk] = self.generate_reverse_hyde(chunk, n)\n",
    "        return processed_chunks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading API keys from environment variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enriching Vector Database with Reverse Hyde Output\n",
    "\n",
    "We will apply the Reverse Hyde method on a set of documents, and enrich the vector database index with LLM generated Hypothetical questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "# Usage example\n",
    "api_key = os.getenv(\"OPENAI_API_KEY\")\n",
    "reverse_hyde = ReverseHyde(api_key)\n",
    "\n",
    "chunks = [\n",
    "    \"A mitochondrion (pl. mitochondria) is an organelle found in the cells of most eukaryotes, such as animals, plants and fungi. Mitochondria have a double membrane structure and use aerobic respiration to generate adenosine triphosphate (ATP), which is used throughout the cell as a source of chemical energy. They were discovered by Albert von Kölliker in 1857 in the voluntary muscles of insects. Meaning a thread-like granule, the term mitochondrion was coined by Carl Benda in 1898. The mitochondrion is popularly nicknamed the \\\"powerhouse of the cell\\\", a phrase popularized by Philip Siekevitz in a 1957 Scientific American article of the same name.\",\n",
    "    \"Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation. Python is dynamically typed and garbage-collected. It supports multiple programming paradigms, including structured (particularly procedural), object-oriented and functional programming. It is often described as a \\\"batteries included\\\" language due to its comprehensive standard library.\",\n",
    "    \"The American Civil War (from April 12, 1861 to May 26, 1865) was a civil war in the United States between the Union (\\\"the North\\\") and the Confederacy (\\\"the South\\\"), which was formed in 1861 by states that had seceded from the Union. The central conflict leading to war was a dispute over whether slavery should be permitted to expand into the western territories, leading to more slave states, or be prohibited from doing so, which many believed would place slavery on a course of ultimate extinction.\"\n",
    "]\n",
    "\n",
    "processed_chunks = reverse_hyde.process_chunks(chunks, n=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'A mitochondrion </span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">(</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">pl. mitochondria</span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">)</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> is an organelle found in the cells of most eukaryotes, such as animals, </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">plants and fungi. Mitochondria have a double membrane structure and use aerobic respiration to generate adenosine </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">triphosphate </span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">(</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">ATP</span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">)</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">, which is used throughout the cell as a source of chemical energy. They were discovered by </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">Albert von Kölliker in 1857 in the voluntary muscles of insects. Meaning a thread-like granule, the term </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">mitochondrion was coined by Carl Benda in 1898. The mitochondrion is popularly nicknamed the \"powerhouse of the </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">cell\", a phrase popularized by Philip Siekevitz in a 1957 Scientific American article of the same name.'</span>: <span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'What is a mitochondrion and where is it found?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'How do mitochondria generate ATP and what is its role in the cell?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'Who discovered mitochondria and when did they first observe them?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'What is the significance of the term \"powerhouse of the cell\" in relation to mitochondria?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'How did the term \"mitochondrion\" come to be and who coined it?'</span>\n",
       "    <span style=\"font-weight: bold\">]</span>,\n",
       "    <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">readability with the use of significant indentation. Python is dynamically typed and garbage-collected. It supports</span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">multiple programming paradigms, including structured </span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">(</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">particularly procedural</span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">)</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">, object-oriented and functional </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">programming. It is often described as a \"batteries included\" language due to its comprehensive standard library.'</span>: \n",
       "<span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'What are some key features of Python as a programming language?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'How does Python prioritize code readability in its design philosophy?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'What are the advantages of Python being dynamically typed and garbage-collected?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'What programming paradigms does Python support?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'Why is Python often referred to as a \"batteries included\" language?'</span>\n",
       "    <span style=\"font-weight: bold\">]</span>,\n",
       "    <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'The American Civil War </span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">(</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">from April 12, 1861 to May 26, 1865</span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">)</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> was a civil war in the United States between the </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">Union </span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">(</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">\"the North\"</span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">)</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> and the Confederacy </span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">(</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">\"the South\"</span><span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">)</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">, which was formed in 1861 by states that had seceded from the</span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">Union. The central conflict leading to war was a dispute over whether slavery should be permitted to expand into </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">the western territories, leading to more slave states, or be prohibited from doing so, which many believed would </span>\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00\">place slavery on a course of ultimate extinction.'</span>: <span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'What was the duration of the American Civil War?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'Who were the two main opposing sides in the American Civil War?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'What was the primary cause of the conflict that led to the American Civil War?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'When was the Confederacy formed and why did the states secede from the Union?'</span>,\n",
       "        <span style=\"color: #00ff00; text-decoration-color: #00ff00\">'How did the issue of slavery play a significant role in the lead-up to the American Civil War?'</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[92m'A mitochondrion \u001b[0m\u001b[1;92m(\u001b[0m\u001b[92mpl. mitochondria\u001b[0m\u001b[1;92m)\u001b[0m\u001b[92m is an organelle found in the cells of most eukaryotes, such as animals, \u001b[0m\n",
       "\u001b[92mplants and fungi. Mitochondria have a double membrane structure and use aerobic respiration to generate adenosine \u001b[0m\n",
       "\u001b[92mtriphosphate \u001b[0m\u001b[1;92m(\u001b[0m\u001b[92mATP\u001b[0m\u001b[1;92m)\u001b[0m\u001b[92m, which is used throughout the cell as a source of chemical energy. They were discovered by \u001b[0m\n",
       "\u001b[92mAlbert von Kölliker in 1857 in the voluntary muscles of insects. Meaning a thread-like granule, the term \u001b[0m\n",
       "\u001b[92mmitochondrion was coined by Carl Benda in 1898. The mitochondrion is popularly nicknamed the \"powerhouse of the \u001b[0m\n",
       "\u001b[92mcell\", a phrase popularized by Philip Siekevitz in a 1957 Scientific American article of the same name.'\u001b[0m: \u001b[1m[\u001b[0m\n",
       "        \u001b[92m'What is a mitochondrion and where is it found?'\u001b[0m,\n",
       "        \u001b[92m'How do mitochondria generate ATP and what is its role in the cell?'\u001b[0m,\n",
       "        \u001b[92m'Who discovered mitochondria and when did they first observe them?'\u001b[0m,\n",
       "        \u001b[92m'What is the significance of the term \"powerhouse of the cell\" in relation to mitochondria?'\u001b[0m,\n",
       "        \u001b[92m'How did the term \"mitochondrion\" come to be and who coined it?'\u001b[0m\n",
       "    \u001b[1m]\u001b[0m,\n",
       "    \u001b[92m'Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code \u001b[0m\n",
       "\u001b[92mreadability with the use of significant indentation. Python is dynamically typed and garbage-collected. It supports\u001b[0m\n",
       "\u001b[92mmultiple programming paradigms, including structured \u001b[0m\u001b[1;92m(\u001b[0m\u001b[92mparticularly procedural\u001b[0m\u001b[1;92m)\u001b[0m\u001b[92m, object-oriented and functional \u001b[0m\n",
       "\u001b[92mprogramming. It is often described as a \"batteries included\" language due to its comprehensive standard library.'\u001b[0m: \n",
       "\u001b[1m[\u001b[0m\n",
       "        \u001b[92m'What are some key features of Python as a programming language?'\u001b[0m,\n",
       "        \u001b[92m'How does Python prioritize code readability in its design philosophy?'\u001b[0m,\n",
       "        \u001b[92m'What are the advantages of Python being dynamically typed and garbage-collected?'\u001b[0m,\n",
       "        \u001b[92m'What programming paradigms does Python support?'\u001b[0m,\n",
       "        \u001b[92m'Why is Python often referred to as a \"batteries included\" language?'\u001b[0m\n",
       "    \u001b[1m]\u001b[0m,\n",
       "    \u001b[92m'The American Civil War \u001b[0m\u001b[1;92m(\u001b[0m\u001b[92mfrom April 12, 1861 to May 26, 1865\u001b[0m\u001b[1;92m)\u001b[0m\u001b[92m was a civil war in the United States between the \u001b[0m\n",
       "\u001b[92mUnion \u001b[0m\u001b[1;92m(\u001b[0m\u001b[92m\"the North\"\u001b[0m\u001b[1;92m)\u001b[0m\u001b[92m and the Confederacy \u001b[0m\u001b[1;92m(\u001b[0m\u001b[92m\"the South\"\u001b[0m\u001b[1;92m)\u001b[0m\u001b[92m, which was formed in 1861 by states that had seceded from the\u001b[0m\n",
       "\u001b[92mUnion. The central conflict leading to war was a dispute over whether slavery should be permitted to expand into \u001b[0m\n",
       "\u001b[92mthe western territories, leading to more slave states, or be prohibited from doing so, which many believed would \u001b[0m\n",
       "\u001b[92mplace slavery on a course of ultimate extinction.'\u001b[0m: \u001b[1m[\u001b[0m\n",
       "        \u001b[92m'What was the duration of the American Civil War?'\u001b[0m,\n",
       "        \u001b[92m'Who were the two main opposing sides in the American Civil War?'\u001b[0m,\n",
       "        \u001b[92m'What was the primary cause of the conflict that led to the American Civil War?'\u001b[0m,\n",
       "        \u001b[92m'When was the Confederacy formed and why did the states secede from the Union?'\u001b[0m,\n",
       "        \u001b[92m'How did the issue of slavery play a significant role in the lead-up to the American Civil War?'\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "console.print(processed_chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query the enriched index\n",
    "\n",
    "Once we have an index with multiple hypothetical questions to the documents, we can use it to retrive the document based on a real user's query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What generates energy in a cell?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qdrant_client import models, QdrantClient\n",
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "# create the vector database client\n",
    "qdrant = QdrantClient(\":memory:\") # Create in-memory Qdrant instance\n",
    "\n",
    "# Create the embedding encoder\n",
    "encoder = SentenceTransformer('all-MiniLM-L6-v2') # Model to create embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create collection to store the wine rating data\n",
    "hyde_collection_name=\"reverse_hyde\"\n",
    "\n",
    "qdrant.recreate_collection(\n",
    "    collection_name=hyde_collection_name,\n",
    "    vectors_config=models.VectorParams(\n",
    "        size=encoder.get_sentence_embedding_dimension(), # Vector size is defined by used model\n",
    "        distance=models.Distance.COSINE\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "# vectorize!\n",
    "qdrant.upload_points(\n",
    "    collection_name=hyde_collection_name,\n",
    "    points=[\n",
    "        models.PointStruct(\n",
    "            id=uuid.uuid5(uuid.NAMESPACE_URL, f\"{d_idx}-{q_idx}\").hex,\n",
    "            vector=encoder.encode(question).tolist(),\n",
    "            payload={ \n",
    "                \"document\": document , \n",
    "                \"doc_id\": d_idx\n",
    "            }\n",
    "        ) for d_idx, (document, questions) \n",
    "            in enumerate(processed_chunks.items()) \n",
    "                for q_idx, question in enumerate(questions)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #ffff00; text-decoration-color: #ffff00\">CollectionInfo</span><span style=\"font-weight: bold\">(</span>\n",
       "    <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">status</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">CollectionStatus.GREEN:</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">'green'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">    </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">optimizer_status</span><span style=\"color: #000000; text-decoration-color: #000000\">=&lt;OptimizersStatusOneOf.OK: </span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">'ok'</span><span style=\"color: #000000; text-decoration-color: #000000\">&gt;,</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">    </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">vectors_count</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">    </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">indexed_vectors_count</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #ff0000; text-decoration-color: #ff0000\">0</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">    </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">points_count</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #ff0000; text-decoration-color: #ff0000\">15</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">    </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">segments_count</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #ff0000; text-decoration-color: #ff0000\">1</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">    </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">config</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #ffff00; text-decoration-color: #ffff00\">CollectionConfig</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">        </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">params</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #ffff00; text-decoration-color: #ffff00\">CollectionParams</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">            </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">vectors</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #ffff00; text-decoration-color: #ffff00\">VectorParams</span><span style=\"color: #000000; text-decoration-color: #000000; font-weight: bold\">(</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">                </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">size</span><span style=\"color: #000000; text-decoration-color: #000000\">=</span><span style=\"color: #ff0000; text-decoration-color: #ff0000\">384</span><span style=\"color: #000000; text-decoration-color: #000000\">,</span>\n",
       "<span style=\"color: #000000; text-decoration-color: #000000\">                </span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">distance</span><span style=\"color: #000000; text-decoration-color: #000000\">=&lt;Distance.COSINE: </span><span style=\"color: #00ff00; text-decoration-color: #00ff00\">'Cosine'</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "                <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">hnsw_config</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "                <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">quantization_config</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "                <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">on_disk</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "                <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">datatype</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "                <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">multivector_config</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>\n",
       "            <span style=\"font-weight: bold\">)</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">shard_number</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">sharding_method</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">replication_factor</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">write_consistency_factor</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">read_fan_out_factor</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">on_disk_payload</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">sparse_vectors</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">hnsw_config</span>=<span style=\"color: #ffff00; text-decoration-color: #ffff00\">HnswConfig</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">m</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">16</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">ef_construct</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">100</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">full_scan_threshold</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">10000</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">max_indexing_threads</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">0</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">on_disk</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">payload_m</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">optimizer_config</span>=<span style=\"color: #ffff00; text-decoration-color: #ffff00\">OptimizersConfig</span><span style=\"font-weight: bold\">(</span>\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">deleted_threshold</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">0.2</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">vacuum_min_vector_number</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">1000</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">default_segment_number</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">0</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">max_segment_size</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">memmap_threshold</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">indexing_threshold</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">20000</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">flush_interval_sec</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">5</span>,\n",
       "            <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">max_optimization_threads</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">1</span>\n",
       "        <span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">wal_config</span>=<span style=\"color: #ffff00; text-decoration-color: #ffff00\">WalConfig</span><span style=\"font-weight: bold\">(</span><span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">wal_capacity_mb</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">32</span>, <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">wal_segments_ahead</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000\">0</span><span style=\"font-weight: bold\">)</span>,\n",
       "        <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">quantization_config</span>=<span style=\"color: #dfdfdf; text-decoration-color: #dfdfdf\">None</span>\n",
       "    <span style=\"font-weight: bold\">)</span>,\n",
       "    <span style=\"color: #e87d3e; text-decoration-color: #e87d3e; font-weight: bold\">payload_schema</span>=<span style=\"font-weight: bold\">{}</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[93mCollectionInfo\u001b[0m\u001b[1m(\u001b[0m\n",
       "    \u001b[1;38;2;232;125;62mstatus\u001b[0m=\u001b[1m<\u001b[0m\u001b[2;36mCollectionStatus.GREEN:\u001b[0m\u001b[39m \u001b[0m\u001b[92m'green'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[39m    \u001b[0m\u001b[1;38;2;232;125;62moptimizer_status\u001b[0m\u001b[39m=<OptimizersStatusOneOf.OK: \u001b[0m\u001b[92m'ok'\u001b[0m\u001b[39m>,\u001b[0m\n",
       "\u001b[39m    \u001b[0m\u001b[1;38;2;232;125;62mvectors_count\u001b[0m\u001b[39m=\u001b[0m\u001b[2;37mNone\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[39m    \u001b[0m\u001b[1;38;2;232;125;62mindexed_vectors_count\u001b[0m\u001b[39m=\u001b[0m\u001b[91m0\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[39m    \u001b[0m\u001b[1;38;2;232;125;62mpoints_count\u001b[0m\u001b[39m=\u001b[0m\u001b[91m15\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[39m    \u001b[0m\u001b[1;38;2;232;125;62msegments_count\u001b[0m\u001b[39m=\u001b[0m\u001b[91m1\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[39m    \u001b[0m\u001b[1;38;2;232;125;62mconfig\u001b[0m\u001b[39m=\u001b[0m\u001b[93mCollectionConfig\u001b[0m\u001b[1;39m(\u001b[0m\n",
       "\u001b[39m        \u001b[0m\u001b[1;38;2;232;125;62mparams\u001b[0m\u001b[39m=\u001b[0m\u001b[93mCollectionParams\u001b[0m\u001b[1;39m(\u001b[0m\n",
       "\u001b[39m            \u001b[0m\u001b[1;38;2;232;125;62mvectors\u001b[0m\u001b[39m=\u001b[0m\u001b[93mVectorParams\u001b[0m\u001b[1;39m(\u001b[0m\n",
       "\u001b[39m                \u001b[0m\u001b[1;38;2;232;125;62msize\u001b[0m\u001b[39m=\u001b[0m\u001b[91m384\u001b[0m\u001b[39m,\u001b[0m\n",
       "\u001b[39m                \u001b[0m\u001b[1;38;2;232;125;62mdistance\u001b[0m\u001b[39m=<Distance.COSINE: \u001b[0m\u001b[92m'Cosine'\u001b[0m\u001b[1m>\u001b[0m,\n",
       "                \u001b[1;38;2;232;125;62mhnsw_config\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "                \u001b[1;38;2;232;125;62mquantization_config\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "                \u001b[1;38;2;232;125;62mon_disk\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "                \u001b[1;38;2;232;125;62mdatatype\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "                \u001b[1;38;2;232;125;62mmultivector_config\u001b[0m=\u001b[2;37mNone\u001b[0m\n",
       "            \u001b[1m)\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mshard_number\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62msharding_method\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mreplication_factor\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mwrite_consistency_factor\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mread_fan_out_factor\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mon_disk_payload\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62msparse_vectors\u001b[0m=\u001b[2;37mNone\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;38;2;232;125;62mhnsw_config\u001b[0m=\u001b[93mHnswConfig\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[1;38;2;232;125;62mm\u001b[0m=\u001b[91m16\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mef_construct\u001b[0m=\u001b[91m100\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mfull_scan_threshold\u001b[0m=\u001b[91m10000\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mmax_indexing_threads\u001b[0m=\u001b[91m0\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mon_disk\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mpayload_m\u001b[0m=\u001b[2;37mNone\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;38;2;232;125;62moptimizer_config\u001b[0m=\u001b[93mOptimizersConfig\u001b[0m\u001b[1m(\u001b[0m\n",
       "            \u001b[1;38;2;232;125;62mdeleted_threshold\u001b[0m=\u001b[91m0\u001b[0m\u001b[91m.2\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mvacuum_min_vector_number\u001b[0m=\u001b[91m1000\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mdefault_segment_number\u001b[0m=\u001b[91m0\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mmax_segment_size\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mmemmap_threshold\u001b[0m=\u001b[2;37mNone\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mindexing_threshold\u001b[0m=\u001b[91m20000\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mflush_interval_sec\u001b[0m=\u001b[91m5\u001b[0m,\n",
       "            \u001b[1;38;2;232;125;62mmax_optimization_threads\u001b[0m=\u001b[91m1\u001b[0m\n",
       "        \u001b[1m)\u001b[0m,\n",
       "        \u001b[1;38;2;232;125;62mwal_config\u001b[0m=\u001b[93mWalConfig\u001b[0m\u001b[1m(\u001b[0m\u001b[1;38;2;232;125;62mwal_capacity_mb\u001b[0m=\u001b[91m32\u001b[0m, \u001b[1;38;2;232;125;62mwal_segments_ahead\u001b[0m=\u001b[91m0\u001b[0m\u001b[1m)\u001b[0m,\n",
       "        \u001b[1;38;2;232;125;62mquantization_config\u001b[0m=\u001b[2;37mNone\u001b[0m\n",
       "    \u001b[1m)\u001b[0m,\n",
       "    \u001b[1;38;2;232;125;62mpayload_schema\u001b[0m=\u001b[1m{\u001b[0m\u001b[1m}\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "console.print(\n",
    "    qdrant\n",
    "    .get_collection(\n",
    "        collection_name=hyde_collection_name\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search Collection for Best Match\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rich.panel import Panel\n",
    "from rich.table import Table\n",
    "\n",
    "def search_collection(collection_name: str, query: str, limit: int = 1):\n",
    "    \"\"\"\n",
    "    This function searches the specified collection for the best match to the given query.\n",
    "    It then creates a table and a panel to display the query and the best match.\n",
    "    \n",
    "    :param collection_name: The name of the collection to search.\n",
    "    :param query: The query to search for.\n",
    "    :param limit: The maximum number of results to return. Default is 1.\n",
    "    \"\"\"\n",
    "    hits = qdrant.search(\n",
    "        collection_name=collection_name,\n",
    "        query_vector=encoder.encode(query).tolist(),\n",
    "        limit=limit\n",
    "    )\n",
    "    # Create a table for both query and best match\n",
    "    table = Table(show_header=True, header_style=\"bold yellow\")\n",
    "    table.add_column(\"Query\", style=\"bright_cyan\", width=30)\n",
    "    table.add_column(\"Best Matching Chunk\", style=\"bright_yellow\", width=50)\n",
    "    table.add_column(\"Score\", style=\"bright_green\")\n",
    "    for hit in hits:\n",
    "        table.add_row(query, f\"{hit.payload['document'][:80]}...\", \"{:.4f}\".format(hit.score))\n",
    "\n",
    "    # Create a panel for the table\n",
    "    panel = Panel(\n",
    "        table,\n",
    "        title=f\"[bold]Query and Best Match in {collection_name}\",\n",
    "        border_style=\"white\",\n",
    "        expand=False\n",
    "    )\n",
    "\n",
    "    # Print the panel\n",
    "    console.print(panel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">╭────────────────────────────── </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\">Query and Best Match in reverse_hyde</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ──────────────────────────────╮</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> ┃<span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> Query                          </span>┃<span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> Best Matching Chunk                                </span>┃<span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> Score  </span>┃ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> │<span style=\"color: #00ffff; text-decoration-color: #00ffff\"> What generates energy in a     </span>│<span style=\"color: #ffff00; text-decoration-color: #ffff00\"> A mitochondrion (pl. mitochondria) is an organelle </span>│<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> 0.6088 </span>│ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> │<span style=\"color: #00ffff; text-decoration-color: #00ffff\"> cell?                          </span>│<span style=\"color: #ffff00; text-decoration-color: #ffff00\"> found in the cells of most eu...                   </span>│<span style=\"color: #00ff00; text-decoration-color: #00ff00\">        </span>│ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> └────────────────────────────────┴────────────────────────────────────────────────────┴────────┘ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">╰──────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[37m╭─\u001b[0m\u001b[37m─────────────────────────────\u001b[0m\u001b[37m \u001b[0m\u001b[1;37mQuery and Best Match in reverse_hyde\u001b[0m\u001b[37m \u001b[0m\u001b[37m─────────────────────────────\u001b[0m\u001b[37m─╮\u001b[0m\n",
       "\u001b[37m│\u001b[0m ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m ┃\u001b[1;33m \u001b[0m\u001b[1;33mQuery                         \u001b[0m\u001b[1;33m \u001b[0m┃\u001b[1;33m \u001b[0m\u001b[1;33mBest Matching Chunk                               \u001b[0m\u001b[1;33m \u001b[0m┃\u001b[1;33m \u001b[0m\u001b[1;33mScore \u001b[0m\u001b[1;33m \u001b[0m┃ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m │\u001b[96m \u001b[0m\u001b[96mWhat generates energy in a    \u001b[0m\u001b[96m \u001b[0m│\u001b[93m \u001b[0m\u001b[93mA mitochondrion (pl. mitochondria) is an organelle\u001b[0m\u001b[93m \u001b[0m│\u001b[92m \u001b[0m\u001b[92m0.6088\u001b[0m\u001b[92m \u001b[0m│ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m │\u001b[96m \u001b[0m\u001b[96mcell?                         \u001b[0m\u001b[96m \u001b[0m│\u001b[93m \u001b[0m\u001b[93mfound in the cells of most eu...                  \u001b[0m\u001b[93m \u001b[0m│\u001b[92m        \u001b[0m│ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m └────────────────────────────────┴────────────────────────────────────────────────────┴────────┘ \u001b[37m│\u001b[0m\n",
       "\u001b[37m╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "search_collection(hyde_collection_name, query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare to document only index (without Hyde)\n",
    "\n",
    "We will index the same documents without adding the reverse Hyde questions and compare the similarity scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create collection to store the wine rating data\n",
    "docs_collection_name=\"documents_only\"\n",
    "\n",
    "qdrant.recreate_collection(\n",
    "    collection_name=docs_collection_name,\n",
    "    vectors_config=models.VectorParams(\n",
    "        size=encoder.get_sentence_embedding_dimension(), # Vector size is defined by used model\n",
    "        distance=models.Distance.COSINE\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vectorize!\n",
    "qdrant.upload_points(\n",
    "    collection_name=docs_collection_name,\n",
    "    points=[\n",
    "        models.PointStruct(\n",
    "            id=idx,\n",
    "            vector=encoder.encode(document).tolist(),\n",
    "            payload={ \"document\": document}\n",
    "        ) for idx, (document, questions) in enumerate(processed_chunks.items())\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">╭───────────────────────────── </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\">Query and Best Match in documents_only</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ─────────────────────────────╮</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> ┃<span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> Query                          </span>┃<span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> Best Matching Chunk                                </span>┃<span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> Score  </span>┃ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> │<span style=\"color: #00ffff; text-decoration-color: #00ffff\"> What generates energy in a     </span>│<span style=\"color: #ffff00; text-decoration-color: #ffff00\"> A mitochondrion (pl. mitochondria) is an organelle </span>│<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> 0.5003 </span>│ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> │<span style=\"color: #00ffff; text-decoration-color: #00ffff\"> cell?                          </span>│<span style=\"color: #ffff00; text-decoration-color: #ffff00\"> found in the cells of most eu...                   </span>│<span style=\"color: #00ff00; text-decoration-color: #00ff00\">        </span>│ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span> └────────────────────────────────┴────────────────────────────────────────────────────┴────────┘ <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">│</span>\n",
       "<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">╰──────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[37m╭─\u001b[0m\u001b[37m────────────────────────────\u001b[0m\u001b[37m \u001b[0m\u001b[1;37mQuery and Best Match in documents_only\u001b[0m\u001b[37m \u001b[0m\u001b[37m────────────────────────────\u001b[0m\u001b[37m─╮\u001b[0m\n",
       "\u001b[37m│\u001b[0m ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m ┃\u001b[1;33m \u001b[0m\u001b[1;33mQuery                         \u001b[0m\u001b[1;33m \u001b[0m┃\u001b[1;33m \u001b[0m\u001b[1;33mBest Matching Chunk                               \u001b[0m\u001b[1;33m \u001b[0m┃\u001b[1;33m \u001b[0m\u001b[1;33mScore \u001b[0m\u001b[1;33m \u001b[0m┃ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m │\u001b[96m \u001b[0m\u001b[96mWhat generates energy in a    \u001b[0m\u001b[96m \u001b[0m│\u001b[93m \u001b[0m\u001b[93mA mitochondrion (pl. mitochondria) is an organelle\u001b[0m\u001b[93m \u001b[0m│\u001b[92m \u001b[0m\u001b[92m0.5003\u001b[0m\u001b[92m \u001b[0m│ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m │\u001b[96m \u001b[0m\u001b[96mcell?                         \u001b[0m\u001b[96m \u001b[0m│\u001b[93m \u001b[0m\u001b[93mfound in the cells of most eu...                  \u001b[0m\u001b[93m \u001b[0m│\u001b[92m        \u001b[0m│ \u001b[37m│\u001b[0m\n",
       "\u001b[37m│\u001b[0m └────────────────────────────────┴────────────────────────────────────────────────────┴────────┘ \u001b[37m│\u001b[0m\n",
       "\u001b[37m╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "search_collection(docs_collection_name, query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}