retrieval_augmented_generation/retrieval_augmented_generation_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "48eeba82",
   "metadata": {},
   "source": [
    "# Retrieval Augmented Generation Evaluation with LangChain and KDB.AI\n",
    "\n",
    "##### Note: This example requires a KDB.AI endpoint and API key. Sign up for a free [KDB.AI account](https://kdb.ai/get-started).\n",
    "\n",
    "This notebook serves as a guide to utilizing LangChain tooling for evaluating a basic Retrieval Augmented Generation (RAG) system. \n",
    "\n",
    "The evaluation process involves employing [LangChain's String Evaluators](https://python.langchain.com/docs/guides/evaluation/string/) to assess both conciseness and correctness. KDB.AI serves as the primary knowledge base, enabling the retrieval of semantically relevant content for the evaluation.\n",
    "\n",
    "### Aim\n",
    "\n",
    "In this tutorial, we build upon the retrieval augmented generation pipeline seen in our [retrieval_augmented_generation.ipynb](retrieval_augmented_generation.ipynb) notebook.\n",
    "If you have not seen it, please read and understand that notebook as it will cover the setup steps of RAG in greater detail than we do here.\n",
    "\n",
    "This notebook focuses on the evaluation of your retrieval augmented generation using KDB.AI as the vector store.\n",
    "We will cover the following topics:\n",
    "\n",
    "1. Load Text Data\n",
    "1. Define OpenAI Text Emedding Model\n",
    "1. Store Embeddings In KDB.AI\n",
    "1. Perform Retrieval Augmented Generation\n",
    "1. Evaluate Retrieval Augmented Generation\n",
    "1. Delete the KDB.AI Table\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e88331c4",
   "metadata": {},
   "source": [
    "## 0. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80d6c97e",
   "metadata": {},
   "source": [
    "### Install dependencies \n",
    "\n",
    "In order to successfully run this sample, note the following steps depending on where you are running this notebook:\n",
    "\n",
    "-***Run Locally / Private Environment:*** The [Setup](https://github.com/KxSystems/kdbai-samples/blob/main/README.md#setup) steps in the repository's `README.md` will guide you on prerequisites and how to run this with Jupyter.\n",
    "\n",
    "\n",
    "-***Colab / Hosted Environment:*** Open this notebook in Colab and run through the cells.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c93b2276",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install kdbai_client langchain langchain_openai langchain-community"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c95778f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "### !!! Only run this cell if you need to download the data into your environment, for example in Colab\n",
    "### This downloads State of the Union Speech data\n",
    "import os\n",
    "\n",
    "if os.path.exists(\"./data/state_of_the_union.txt\") == False:\n",
    "    !mkdir ./data\n",
    "    !wget -P ./data https://raw.githubusercontent.com/KxSystems/kdbai-samples/main/retrieval_augmented_generation/data/state_of_the_union.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "679126f7",
   "metadata": {},
   "source": [
    "### Import Packages\n",
    "\n",
    "Load the various libraries that will be needed in this tutorial, including all the langchain libraries we will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "894980f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# vector DB\n",
    "from getpass import getpass\n",
    "import kdbai_client as kdbai\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b9549fe3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# langchain packages\n",
    "from langchain.chains import RetrievalQA\n",
    "from langchain_openai import ChatOpenAI\n",
    "from langchain.document_loaders import TextLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_community.vectorstores import KDBAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5ab423cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# evaluation packages\n",
    "from langchain.evaluation import load_evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc263a6e",
   "metadata": {},
   "source": [
    "### Set API Keys\n",
    "\n",
    "To follow this example you will need to request an [OpenAI API Key](https://platform.openai.com/apps). \n",
    "\n",
    "You can create this for free by registering using the links provided.\n",
    "Once you have the credentials you can add them below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ed70fbe3",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"OPENAI_API_KEY\"] = (\n",
    "    os.environ[\"OPENAI_API_KEY\"]\n",
    "    if \"OPENAI_API_KEY\" in os.environ\n",
    "    else getpass(\"OpenAI API Key: \")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f56faa93",
   "metadata": {},
   "source": [
    "### Define Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b03039cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_dict(d: dict) -> None:\n",
    "    for k, v in d.items():\n",
    "        print(f\"\\n{k.capitalize()}\\n---\\n{v}\".replace('\\n\\n', '\\n'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "164f0b99",
   "metadata": {},
   "source": [
    "## 1. Load Text Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f04aa63a",
   "metadata": {},
   "source": [
    "### Read In Text Document\n",
    "\n",
    "The document we will use for this examples is a State of the Union message from the President of the United States to the United States Congress.\n",
    "\n",
    "In the below code snippet, we read the text file in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "69dfbffd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the documents we want to prompt an LLM about\n",
    "doc = TextLoader(\"data/state_of_the_union.txt\").load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed001b92",
   "metadata": {},
   "source": [
    "### Split The Document Into Chunks\n",
    "\n",
    "We then split this document into chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "84bfd8a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Chunk the documents into 500 character chunks using langchain's text splitter \"RucursiveCharacterTextSplitter\"\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e9c70879",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split_documents produces a list of all the chunks created, printing out first chunk for example\n",
    "pages = [p.page_content for p in text_splitter.split_documents(doc)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd1cf6a4",
   "metadata": {},
   "source": [
    "## 2. Define OpenAI Text Embedding Model\n",
    " \n",
    "We will use OpenAIEmbeddings to embed our document into a format suitable for the vector database. We select `text-embedding-ada-002` for use in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ffa379e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7287e75d",
   "metadata": {},
   "source": [
    "## 3. Store Embeddings In KDB.AI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a9f5b2f",
   "metadata": {},
   "source": [
    "With the embeddings created, we need to store them in a vector database to enable efficient searching.\n",
    "\n",
    "### Define KDB.AI Session\n",
    "\n",
    "KDB.AI comes in two offerings:\n",
    "\n",
    "1. [KDB.AI Cloud](https://trykdb.kx.com/kdbai/signup/) - For experimenting with smaller generative AI projects with a vector database in our cloud.\n",
    "2. [KDB.AI Server](https://trykdb.kx.com/kdbaiserver/signup/) - For evaluating large scale generative AI applications on-premises or on your own cloud provider.\n",
    "\n",
    "Depending on which you use there will be different setup steps and connection details required.\n",
    "\n",
    "##### Option 1. KDB.AI Cloud\n",
    "\n",
    "To use KDB.AI Cloud, you will need two session details - a URL endpoint and an API key.\n",
    "To get these you can sign up for free [here](https://trykdb.kx.com/kdbai/signup).\n",
    "\n",
    "You can connect to a KDB.AI Cloud session using `kdbai.Session` and passing the session URL endpoint and API key details from your KDB.AI Cloud portal.\n",
    "\n",
    "If the environment variables `KDBAI_ENDPOINTS` and `KDBAI_API_KEY` exist on your system containing your KDB.AI Cloud portal details, these variables will automatically be used to connect.\n",
    "If these do not exist, it will prompt you to enter your KDB.AI Cloud portal session URL endpoint and API key details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e62f00a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "KDBAI_ENDPOINT = (\n",
    "    os.environ[\"KDBAI_ENDPOINT\"]\n",
    "    if \"KDBAI_ENDPOINT\" in os.environ\n",
    "    else input(\"KDB.AI endpoint: \")\n",
    ")\n",
    "KDBAI_API_KEY = (\n",
    "    os.environ[\"KDBAI_API_KEY\"]\n",
    "    if \"KDBAI_API_KEY\" in os.environ\n",
    "    else getpass(\"KDB.AI API key: \")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecf10fd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "session = kdbai.Session(api_key=KDBAI_API_KEY, endpoint=KDBAI_ENDPOINT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "712045b1",
   "metadata": {},
   "source": [
    "##### Option 2. KDB.AI Server\n",
    "\n",
    "To use KDB.AI Server, you will need download and run your own container.\n",
    "To do this, you will first need to sign up for free [here](https://trykdb.kx.com/kdbaiserver/signup/). \n",
    "\n",
    "You will receive an email with the required license file and bearer token needed to download your instance.\n",
    "Follow instructions in the signup email to get your session up and running.\n",
    "\n",
    "Once the [setup steps](https://code.kx.com/kdbai/gettingStarted/kdb-ai-server-setup.html) are complete you can then connect to your KDB.AI Server session using `kdbai.Session` and passing your local endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a62fd9f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# session = kdbai.Session(endpoint=\"http://localhost:8082\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4d72b5b",
   "metadata": {},
   "source": [
    "### Define Vector DB Table Schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "299902d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "rag_eval_schema = [\n",
    "    {\"name\": \"id\", \"type\": \"str\"},\n",
    "    {\"name\": \"text\", \"type\": \"bytes\"},\n",
    "    {\"name\": \"embeddings\", \"type\": \"float32s\"}\n",
    "]\n",
    "indexes = [{\"name\": \"flat_index\", \"type\": \"flat\", \"column\": \"embeddings\", \"params\": {\"dims\": 1536, \"metric\": \"L2\"}}]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "640fceb2",
   "metadata": {},
   "source": [
    "### Create Vector DB Table\n",
    "\n",
    "Use the KDB.AI `create_table` function to create a table that matches the defined schema in the vector database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0bbc9942",
   "metadata": {},
   "outputs": [],
   "source": [
    "database = session.database(\"default\")\n",
    "# First ensure the table does not already exist\n",
    "try:\n",
    "    database.table(\"rag_eval\").drop()\n",
    "except kdbai.KDBAIException:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "37840395",
   "metadata": {},
   "outputs": [],
   "source": [
    "table = database.create_table(\"rag_eval\", schema=rag_eval_schema, indexes=indexes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "934da954",
   "metadata": {},
   "source": [
    "### Add Embedded Data to KDB.AI Table\n",
    "\n",
    "We can now store our data in KDB.AI by passing a few parameters to `KDBAI.from_texts`:\n",
    "\n",
    "- `session` our handle to talk to KDB.AI\n",
    "- `table_name` our KDB.AI table name\n",
    "- `texts` the chunked document \n",
    "- `embeddings` the embeddings model we have chosen "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7680e758",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['3a39deab-3ca1-457b-a725-192878e2ef3e',\n",
       " 'b9e62100-9bf3-4d66-9066-e546505346a8',\n",
       " 'baf3fbeb-1997-4eec-be6a-faec7431380e',\n",
       " 'de7c3272-38e3-48e0-bbc6-1240cb639430',\n",
       " 'ee1984d6-3be6-4e16-bb54-8a0d4ba80e36',\n",
       " 'ac042a18-cb3e-4369-bd7a-114fd77b938a',\n",
       " 'e4842ca7-d965-44d2-a20e-a23c8710c469',\n",
       " 'd574a79a-2cdc-41e5-bc5b-56de4796a2da',\n",
       " '67f4e777-37e4-4cb6-a2b5-614218593a2a',\n",
       " 'e3656ced-e86b-400d-8f5b-d706ece9dd70',\n",
       " '47e20998-cadc-4d5e-8514-0221821921f6',\n",
       " 'cf8c9905-dd1b-48e0-8cfb-b3e6cfe41649',\n",
       " '9f58e0e9-bd90-4ae4-9961-798b312467c4',\n",
       " '2da5b147-45dd-4325-977b-26b9e9c825f0',\n",
       " '6744caee-09d6-4aee-a2d0-017c568cbbfd',\n",
       " '434d9d87-0e35-4aa7-8ffc-db4bfbc90f16',\n",
       " '4c2fc45d-c631-4856-b8ab-61af03d3c41a',\n",
       " '536d2df1-40f9-4dab-be85-28b4cb6f79af',\n",
       " '750b3adc-17d2-4d6f-836e-1a6382ad2ff2',\n",
       " '0b27e61e-638f-442a-9b92-752eb99cf67d',\n",
       " 'fff6c3a1-94ee-4d36-90a3-42f5405f52ff',\n",
       " 'a3779604-81bf-4267-aae7-e38c97577ed9',\n",
       " '6539774a-07c3-4529-a3e3-f71ee58d4667',\n",
       " '0db73c13-5b88-4a48-a8d8-7a94777c3470',\n",
       " '8b25f891-30d1-4e2b-969b-3e39359edbc8',\n",
       " '02780b29-f975-4b93-b861-c79bac8b8c1b',\n",
       " '83d5862e-735b-4b9f-814b-b84032a1ca16',\n",
       " 'c5180fcb-123d-478d-ab3c-a0268768ffee',\n",
       " 'df4a243a-3098-49d5-aa74-5d1aa8fe0f39',\n",
       " 'd48d9b20-397b-49e3-bb9b-bc11e8bd60d1',\n",
       " '48b63bab-3a8b-4591-b622-53e35960efef',\n",
       " '1926e282-f3e1-463e-92a1-a0377e6368c1',\n",
       " '7e998714-bc74-4c15-9b04-3da4ddb5db9e',\n",
       " '72fd0de7-45d7-41ec-9c14-871125832889',\n",
       " '63002836-4c17-40ea-a00a-0717b482ea81',\n",
       " 'c3051102-79f3-4fca-acec-f2cda194991c',\n",
       " '40427fc7-fa8c-44c9-a9c0-9894172cc60d',\n",
       " 'bd7d5d09-8da6-44c7-b479-bd913e82ce15',\n",
       " '8d8e9abd-21e3-4a5a-a5ef-bea3f06f5a84',\n",
       " '0662a00a-eb47-4a3f-9c97-8642033efed2',\n",
       " 'd918c68c-dec9-4782-a563-6225cafede56',\n",
       " '1898189a-d5ae-4fef-acfd-e23023d2eca2',\n",
       " 'e1138c36-5253-4730-8176-b605252b7d99',\n",
       " 'dbf19503-d6fc-4547-81f5-58c27041b668',\n",
       " 'ed6b3314-22da-44ac-b4e5-16afee761b6a',\n",
       " '5193a98e-3265-416d-babb-cc2f6945ed1a',\n",
       " '8e66f06e-d0e2-4988-8e63-6dd09c9c4322',\n",
       " 'f0a267a5-90b0-4785-8ff3-f686de966916',\n",
       " 'cd856315-e980-483e-a8f3-b921477ba8fb',\n",
       " 'b856a809-d4c9-4c9f-a0e4-929314a00d10',\n",
       " '9f603984-a7d0-4cd1-9aea-3c94b8ab9e51',\n",
       " 'a351f9cf-f06c-4269-98c6-79c9d657b48e',\n",
       " '1ab0f48d-11dc-44f0-befc-7dee347e781a',\n",
       " '4460e410-ee24-4dad-a9bd-ed0fc260be08',\n",
       " '6d52313a-8372-42b2-950e-46072d27677e',\n",
       " 'e491c483-9f36-4cdf-87a1-3b67effd2961',\n",
       " '0fa360bb-0511-46c7-ab6e-0c7f29451174',\n",
       " 'ecb0b0c7-e5b5-42a3-821b-73a60c664396',\n",
       " '29181cdd-ed88-47e4-9ce1-5f55a967168b',\n",
       " '4f046129-3cf9-4717-90ca-50cf83b9995b',\n",
       " 'f8781a59-22b5-4b5c-887b-bb425eb33db1',\n",
       " '7378bb22-dcc0-454d-a0b5-dda287961fe7',\n",
       " '6d800662-6278-4100-8656-9943cce01b98',\n",
       " '8ccf782f-05b9-4688-9971-58e2a61392f1',\n",
       " '0aed16bd-74ac-437e-b6c4-c5a02723ff1f',\n",
       " '4b028b58-7c89-4678-9421-187b87a0cd3b',\n",
       " '3b2d0909-7ad7-42ed-86df-aeefbf41a553',\n",
       " 'e98f3e47-cf31-4efd-9ab9-2b44125129cb',\n",
       " 'c1827cb0-513e-4d31-bca1-94bd0ec69238',\n",
       " '4071e390-052f-4ae7-8c49-150c9840243c',\n",
       " 'bb797699-18fa-4df1-889a-e1ef41c312fe',\n",
       " '37f14b6c-26d4-4f45-b69e-747998ce4ce9',\n",
       " '2345f851-e017-4604-b54c-cca25f58598f',\n",
       " '1320c41a-cab0-478e-84e8-ee28e880e38f',\n",
       " '11108ede-09bb-47c3-8739-0be3ee1ace5a',\n",
       " 'bac8a8f7-7d49-4faa-b4cc-a1267392eea4',\n",
       " '58f3355d-1abe-484b-bcbc-30d405c0734d',\n",
       " '1a516946-6ed9-46eb-82c4-5d38c32423d9',\n",
       " '699a3389-3fe9-493c-9e64-979b72dafcb7',\n",
       " 'a6a244eb-26fe-4538-8fb7-4d9a94df7ddb',\n",
       " '67fa79e6-3feb-447c-a173-847070ca9e81',\n",
       " 'bb9b7307-6c5f-4c49-ac16-8d306dc7d368',\n",
       " '4b413721-89b4-4681-8e0e-b1406d4ced37',\n",
       " 'fc2b623a-eb81-4272-880b-b3864dd2ec8c',\n",
       " '443ff2b1-7710-42cc-b805-6fd776468503',\n",
       " '7cbec07a-5336-4025-8a08-eaaf889ee75e',\n",
       " '131a74eb-534f-4b60-87f8-d1f2e782f4d7',\n",
       " '42552610-46ca-4a3b-800f-ad575a94d5db',\n",
       " '64568872-a133-4777-a63c-1324b99fc845']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use KDBAI as vector store\n",
    "vecdb_kdbai = KDBAI(table, embeddings)\n",
    "vecdb_kdbai.add_texts(texts=pages)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ece8d806",
   "metadata": {},
   "source": [
    "Now we have the vector embeddings stored in KDB.AI we are ready to query."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38569892",
   "metadata": {},
   "source": [
    "## 4. Perform Retrieval Augmented Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d34a0636",
   "metadata": {},
   "source": [
    "We will perform [question answering (QA) in LangChain](https://python.langchain.com/docs/use_cases/question_answering/#go-deeper-4) using `RetrievalQA`.\n",
    "\n",
    "`RetrievalQA` retrieves the most relevant chunk of text and does QA on that subset.\n",
    "We will use KDB.AI as the retriever of `RetrievalQA`.\n",
    "\n",
    "### Define QA Bot\n",
    "\n",
    "The code below defines a question-answering bot that combines OpenAI's GPT-4o-mini for generating responses and a retriever that accesses the KDB.AI vector database to find relevant information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9011f654",
   "metadata": {},
   "outputs": [],
   "source": [
    "K = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3ca7342e",
   "metadata": {},
   "outputs": [],
   "source": [
    "qabot = RetrievalQA.from_chain_type(\n",
    "    chain_type=\"stuff\",\n",
    "    llm=ChatOpenAI(model=\"gpt-4o-mini\", temperature=0.0),\n",
    "    retriever=vecdb_kdbai.as_retriever(search_kwargs=dict(k=K, index=\"flat_index\")),\n",
    "    return_source_documents=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb8ee6a4",
   "metadata": {},
   "source": [
    "`as_retriever` is a method that converts a vectorstore into a retriever. A retriever is an interface that returns documents given an unstructured query. By using <code>as_retriever</code>, we can create a retriever from a vectorstore and use it to retrieve relevant documents for a query. This allows us to perform question answering over the documents indexed by the vectorstore `vecdb_kdbai`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "057670d5",
   "metadata": {},
   "source": [
    "### Query The QA Bot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "de98d6be",
   "metadata": {},
   "outputs": [],
   "source": [
    "def query_qabot(qabot, query: str) -> str:\n",
    "    query_res = qabot.invoke(dict(query=query))[\"result\"]\n",
    "    print(f\"{query}\\n---\\n{query_res}\")\n",
    "    return query_res"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df3ca2ca",
   "metadata": {},
   "source": [
    "##### Query 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "85ca7b27",
   "metadata": {},
   "outputs": [],
   "source": [
    "query1 = \"What improvements could be made in infrastructure?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fd88bf8e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What improvements could be made in infrastructure?\n",
      "---\n",
      "Improvements that could be made in infrastructure include:\n",
      "\n",
      "1. Rebuilding and modernizing roads, highways, and bridges to ensure safety and efficiency.\n",
      "2. Expanding and upgrading public transportation systems to provide better access and reduce congestion.\n",
      "3. Developing a national network of electric vehicle charging stations to support the transition to electric vehicles.\n",
      "4. Replacing lead pipes to ensure clean drinking water for all Americans.\n",
      "5. Providing affordable high-speed internet access to urban, suburban, rural, and tribal communities.\n",
      "6. Upgrading airports, ports, and waterways to enhance transportation and trade capabilities.\n",
      "7. Implementing sustainable practices to withstand the effects of climate change and promote environmental justice. \n",
      "\n",
      "These improvements aim to enhance the overall infrastructure and support economic growth and competitiveness.\n"
     ]
    }
   ],
   "source": [
    "res1 = query_qabot(qabot, query1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0f329b3",
   "metadata": {},
   "source": [
    "##### Query 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d0eece5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "query2 = \"How many jobs were created in the country due the electric vehicle manufacturing industry?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "41997198",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How many jobs were created in the country due the electric vehicle manufacturing industry?\n",
      "---\n",
      "Ford is creating 11,000 jobs and GM is creating 4,000 jobs in the electric vehicle manufacturing industry, which totals 15,000 jobs.\n"
     ]
    }
   ],
   "source": [
    "res2 = query_qabot(qabot, query2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36d38581",
   "metadata": {},
   "source": [
    "## 5. Evaluate Retrieval Augmented Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3933ae8c",
   "metadata": {},
   "source": [
    "Here we will carry out two evaluation techniques against the results of our retrieval augmented generation pipeline.\n",
    "We will measure the *Conciseness* and the *Correctness* of the answers.\n",
    "\n",
    "### Evaluate Conciseness\n",
    "\n",
    "We will evaluate the conciseness of the answers the QA bot returns using LangChain's `load_evaluator` function with the `criteria` set to `\"conciseness\"`.\n",
    "\n",
    "In this example, we use GPT-4o as the LLM that performs the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9c14a699",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_llm = ChatOpenAI(model=\"gpt-4o\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d41ed25a",
   "metadata": {},
   "outputs": [],
   "source": [
    "concise_evaluator = load_evaluator(\n",
    "    \"criteria\", criteria=\"conciseness\", llm=evaluation_llm\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "0a63b03e",
   "metadata": {},
   "outputs": [],
   "source": [
    "concise_eval_res = concise_evaluator.evaluate_strings(prediction=res1, input=query1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a7866960-9256-4df2-8087-49dcf43c3124",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Reasoning\n",
      "---\n",
      "To determine if the submission meets the criterion of conciseness, we need to assess whether it is brief and to the point. Here is a step-by-step reasoning process:\n",
      "1. **Identify Key Points**: The submission lists seven specific improvements that could be made in infrastructure:\n",
      "   - Rebuilding and modernizing roads, highways, and bridges.\n",
      "   - Expanding and upgrading public transportation systems.\n",
      "   - Developing a national network of electric vehicle charging stations.\n",
      "   - Replacing lead pipes.\n",
      "   - Providing affordable high-speed internet access.\n",
      "   - Upgrading airports, ports, and waterways.\n",
      "   - Implementing sustainable practices.\n",
      "2. **Examine Each Point for Brevity**:\n",
      "   - Each point is presented in a single sentence.\n",
      "   - The points are specific and avoid unnecessary elaboration.\n",
      "3. **Overall Length and Focus**:\n",
      "   - The list format helps in making the submission concise.\n",
      "   - The concluding sentence summarizes the purpose of the improvements succinctly: \"These improvements aim to enhance the overall infrastructure and support economic growth and competitiveness.\"\n",
      "4. **Relevance**:\n",
      "   - Each item on the list is directly relevant to the question about improvements in infrastructure.\n",
      "   - There is no extraneous information or digression from the main topic.\n",
      "5. **Conclusion**:\n",
      "   - The submission effectively communicates the necessary information in a clear and concise manner without unnecessary verbosity.\n",
      "Based on this detailed analysis, the submission meets the criterion of conciseness.\n",
      "Y\n",
      "\n",
      "Value\n",
      "---\n",
      "Y\n",
      "\n",
      "Score\n",
      "---\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print_dict(concise_eval_res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed8e4e74-876c-4dae-9e0d-42f20698ea43",
   "metadata": {},
   "source": [
    "### Evaluate Correctness\n",
    "\n",
    "We can use the same `load_evaluator` function to calculate correctness by simply changing the `criteria` to `\"correctness\"`.\n",
    "\n",
    "When using this option, we can pass a reference for the evaluator to check the correctness against.\n",
    "Let's pass a reference that matches the information returned as well as one that doesn't.\n",
    "\n",
    "For this evaluation, we will use the result of the second query we ran through our RAG pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "3d5742f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_evaluator = load_evaluator(\n",
    "    \"labeled_criteria\",\n",
    "    criteria=\"correctness\",\n",
    "    llm=evaluation_llm,\n",
    "    requires_reference=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b0fe16e",
   "metadata": {},
   "source": [
    "##### Matching Reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "86e41652",
   "metadata": {},
   "outputs": [],
   "source": [
    "matching_ref = \"15000 jobs were created due to manufacturing of electric vehicles.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "a2bd3f02",
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_eval_res1 = correct_evaluator.evaluate_strings(\n",
    "    prediction=res2, input=query2, reference=matching_ref\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "708d3386-f28d-4a6a-bb7c-10a15e8574af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Reasoning\n",
      "---\n",
      "Step-by-step reasoning:\n",
      "1. **Correctness**: \n",
      "   - The submission states that Ford is creating 11,000 jobs and GM is creating 4,000 jobs, which totals 15,000 jobs.\n",
      "   - The reference states that 15,000 jobs were created due to the manufacturing of electric vehicles.\n",
      "2. **Accuracy**:\n",
      "   - The total number of jobs mentioned in the submission (15,000) matches the reference number (15,000). \n",
      "   - The specific companies mentioned (Ford and GM) and their respective job creation numbers (11,000 and 4,000) add up correctly to 15,000 jobs.\n",
      "3. **Factuality**:\n",
      "   - There is no conflicting information between the submission and the reference.\n",
      "   - The details provided about the companies (Ford and GM) are not disputed by the reference, and since the total number aligns, it can be considered factual.\n",
      "Conclusion:\n",
      "- Since the submission correctly totals 15,000 jobs, which matches the reference, and there are no inaccuracies or factual errors, the submission meets the criteria.\n",
      "Y\n",
      "\n",
      "Value\n",
      "---\n",
      "Y\n",
      "\n",
      "Score\n",
      "---\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print_dict(correct_eval_res1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f1a317f",
   "metadata": {},
   "source": [
    "##### Contradictory Reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a2b8c14e",
   "metadata": {},
   "outputs": [],
   "source": [
    "contractic_ref = \"12000 jobs were created due to manufacturing of electric vehicles.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "ea0cb2fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_eval_res2 = correct_evaluator.evaluate_strings(\n",
    "    prediction=res2, input=query2, reference=contractic_ref\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "2172b83f-61ca-4963-8225-0378580a67a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Reasoning\n",
      "---\n",
      "First, I will assess the submission based on the criterion of correctness, which includes accuracy and factuality. Here is the step-by-step reasoning:\n",
      "1. **Correctness**:\n",
      "   - The submission states that Ford is creating 11,000 jobs and GM is creating 4,000 jobs in the electric vehicle manufacturing industry, totaling 15,000 jobs.\n",
      "   - The reference data indicates that 12,000 jobs were created due to the manufacturing of electric vehicles.\n",
      "   - There is a discrepancy between the submission and the reference data. The submission claims a total of 15,000 jobs, whereas the reference data states 12,000 jobs.\n",
      "   - Since the submission's total (15,000 jobs) does not match the reference data (12,000 jobs), it is not factually correct.\n",
      "Given this analysis, the submission does not meet the criterion of correctness as it provides an inaccurate total number of jobs created.\n",
      "Therefore, the answer is:\n",
      "N\n",
      "\n",
      "Value\n",
      "---\n",
      "N\n",
      "\n",
      "Score\n",
      "---\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "print_dict(correct_eval_res2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7195efbb",
   "metadata": {},
   "source": [
    "## 6. Delete the KDB.AI Table\n",
    "\n",
    "Once finished with the table, it is best practice to drop it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "1d83ed49",
   "metadata": {},
   "outputs": [],
   "source": [
    "table.drop()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7ed75e9",
   "metadata": {},
   "source": [
    "## Take Our Survey\n",
    "\n",
    "We hope you found this sample helpful! Your feedback is important to us, and we would appreciate it if you could take a moment to fill out our brief survey. Your input helps us improve our content.\n",
    "\n",
    "[**Take the Survey**](https://delighted.com/t/dgCLUkdx)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}