qHnsw_index_pdf_search/pdf_qHNSW_Search.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "3280b01a-d3b7-4ef6-9494-789d15bc48ec",
      "metadata": {
        "id": "3280b01a-d3b7-4ef6-9494-789d15bc48ec"
      },
      "source": [
        "# Semantic Search on PDF Documents with qHNSW Index\n",
        "\n",
        "##### Note: This example requires a KDB.AI endpoint and API key. Sign up for a free [KDB.AI account](https://kdb.ai/get-started).\n",
        "\n",
        "This example demonstrates how to use KDB.AI to run semantic search on unstructured text documents.\n",
        "\n",
        "<div class=\"alert alert-block alert-info\">\n",
        "<b>Tip:</b> This sample uses ‘qHNSW’ , a new vector index choice in KDB.AI. It will support the same API options as the existing ‘HNSW’ index but with the significant difference that the index is stored on-disk and memory-mapped as required. This means data inserts will have negligible memory and cpu footprints. The vector index can grow and be searched as long as there is disk space available. Among other cases, this stands out as a great index for memory contrained situations such as edge devices.\n",
        "</div>\n",
        "\n",
        "Semantic search allows users to perform searches based on the meaning or similarity of the data rather than exact matches. It works by converting the query into a vector representation and then finding similar vectors in the database. This way, even if the query and the data in the database are not identical, the system can identify and retrieve the most relevant results based on their semantic meaning.\n",
        "\n",
        "### Aim\n",
        "In this tutorial, we'll walk you through the process of performing semantic search on documents, taking PDFs as example, using KDB.AI as the vector store. We will cover the following topics:\n",
        "\n",
        "0. Setup\n",
        "1. Load PDF Data\n",
        "2. KDB.AI Table Creation (qHNSW index)\n",
        "3. LlamaIndex index & query_engine setup\n",
        "4. Retrieve Similar Sentences & RAG\n",
        "5. Delete the KDB.AI Table\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "75362bc9",
      "metadata": {
        "id": "75362bc9"
      },
      "source": [
        "## 0. Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dda3d787",
      "metadata": {
        "id": "dda3d787"
      },
      "source": [
        "### Install dependencies\n",
        "\n",
        "In order to successfully run this sample, note the following steps depending on where you are running this notebook:\n",
        "\n",
        "-***Run Locally / Private Environment:*** The [Setup](https://github.com/KxSystems/kdbai-samples/blob/main/README.md#setup) steps in the repository's `README.md` will guide you on prerequisites and how to run this with Jupyter.\n",
        "\n",
        "\n",
        "-***Colab / Hosted Environment:*** Open this notebook in Colab and run through the cells."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b19c5107-001e-40c2-bfe7-6c9c99e4846d",
      "metadata": {
        "id": "b19c5107-001e-40c2-bfe7-6c9c99e4846d"
      },
      "source": [
        "### Set Environment Variables"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8bad9d73",
      "metadata": {
        "id": "8bad9d73"
      },
      "outputs": [],
      "source": [
        "!pip install llama-index llama-index-llms-openai llama-index-embeddings-openai llama-index-readers-file llama-index-vector-stores-kdbai\n",
        "!pip install pandas\n",
        "!pip install kdbai_client\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "21979389",
      "metadata": {
        "id": "21979389"
      },
      "source": [
        "### Import Packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "id": "rjfp-08NPdvC",
      "metadata": {
        "id": "rjfp-08NPdvC"
      },
      "outputs": [],
      "source": [
        "\n",
        "import os\n",
        "from getpass import getpass\n",
        "import re\n",
        "import os\n",
        "import shutil\n",
        "import time\n",
        "import urllib\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "from llama_index.core import (\n",
        "    Settings,\n",
        "    SimpleDirectoryReader,\n",
        "    StorageContext,\n",
        "    VectorStoreIndex,\n",
        ")\n",
        "from llama_index.core.node_parser import SentenceSplitter\n",
        "from llama_index.core.retrievers import VectorIndexRetriever\n",
        "from llama_index.embeddings.openai import OpenAIEmbedding\n",
        "from llama_index.llms.openai import OpenAI\n",
        "from llama_index.vector_stores.kdbai import KDBAIVectorStore\n",
        "\n",
        "import kdbai_client as kdbai\n",
        "\n",
        "OUTDIR = \"pdf\"\n",
        "RESET = True"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "id": "f23f6513",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f23f6513",
        "outputId": "d2832d7e-e91f-4ba4-e996-6057d2d03632"
      },
      "outputs": [],
      "source": [
        "### !!! Only run this cell if you need to download the data into your environment, for example in Colab\n",
        "### This downloads research paper pdf into your environment\n",
        "if os.path.exists(\"./data/research_paper.pdf\") == False:\n",
        "  !mkdir ./data\n",
        "  !wget -P ./data https://raw.githubusercontent.com/KxSystems/kdbai-samples/main/document_search/data/research_paper.pdf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "id": "yCCdm-QDPtDZ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yCCdm-QDPtDZ",
        "outputId": "02495a02-0f49-4181-9f38-7a0f8273ecbe"
      },
      "outputs": [],
      "source": [
        "# OpenAI API Key: https://platform.openai.com/api\n",
        "os.environ[\"OPENAI_API_KEY\"] = (\n",
        "    os.environ[\"OPENAI_API_KEY\"]\n",
        "    if \"OPENAI_API_KEY\" in os.environ\n",
        "    else getpass(\"OpenAI API Key: \")\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "id": "HCI37LxrPywl",
      "metadata": {
        "id": "HCI37LxrPywl"
      },
      "outputs": [],
      "source": [
        "# Set up LlamaIndex Parameters\n",
        "\n",
        "EMBEDDING_MODEL  = \"text-embedding-3-small\"\n",
        "GENERATION_MODEL = 'gpt-4o-mini'\n",
        "\n",
        "llm = OpenAI(model=GENERATION_MODEL)\n",
        "embed_model = OpenAIEmbedding(model=EMBEDDING_MODEL)\n",
        "\n",
        "Settings.llm = llm\n",
        "Settings.embed_model = embed_model"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8594c911",
      "metadata": {
        "id": "8594c911"
      },
      "source": [
        "### Configure Console"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "id": "b870117b",
      "metadata": {
        "id": "b870117b"
      },
      "outputs": [],
      "source": [
        "pd.set_option(\"display.max_colwidth\", 300)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8a425b33",
      "metadata": {
        "id": "8a425b33"
      },
      "source": [
        "### Define Helper Functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "id": "9a135635",
      "metadata": {
        "id": "9a135635"
      },
      "outputs": [],
      "source": [
        "def show_df(df: pd.DataFrame) -> pd.DataFrame:\n",
        "    print(df.shape)\n",
        "    return df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "48826990",
      "metadata": {
        "id": "48826990"
      },
      "source": [
        "## 1. Load PDF Data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b2992812-4705-489d-974f-b7b44132343a",
      "metadata": {
        "id": "b2992812-4705-489d-974f-b7b44132343a"
      },
      "source": [
        "### Read Text From PDF Document\n",
        "\n",
        "We LlamaIndex SimpleDirectorReader to read in our PDF file.\n",
        "\n",
        "The PDF we are using is [this research paper](https://arxiv.org/pdf/2308.05801.pdf) presenting information on the formation of Interstellar Objects in the Milky Way."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "id": "NWa3S2iwQd_K",
      "metadata": {
        "id": "NWa3S2iwQd_K"
      },
      "outputs": [],
      "source": [
        "reader = SimpleDirectoryReader(\n",
        "    input_dir=\"data\",\n",
        ")\n",
        "documents = reader.load_data()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "70505eae-4138-4ba8-80e9-fc13c37d0b32",
      "metadata": {
        "id": "70505eae-4138-4ba8-80e9-fc13c37d0b32"
      },
      "source": [
        "### Define KDB.AI Session\n",
        "\n",
        "KDB.AI comes in two offerings:\n",
        "\n",
        "1. [KDB.AI Cloud](https://trykdb.kx.com/kdbai/signup/) - For experimenting with smaller generative AI projects with a vector database in our cloud.\n",
        "2. [KDB.AI Server](https://trykdb.kx.com/kdbaiserver/signup/) - For evaluating large scale generative AI applications on-premises or on your own cloud provider.\n",
        "\n",
        "Depending on which you use there will be different setup steps and connection details required.\n",
        "\n",
        "##### Option 1. KDB.AI Cloud\n",
        "\n",
        "To use KDB.AI Cloud, you will need two session details - a URL endpoint and an API key.\n",
        "To get these you can sign up for free [here](https://trykdb.kx.com/kdbai/signup).\n",
        "\n",
        "You can connect to a KDB.AI Cloud session using `kdbai.Session` and passing the session URL endpoint and API key details from your KDB.AI Cloud portal.\n",
        "\n",
        "If the environment variables `KDBAI_ENDPOINTS` and `KDBAI_API_KEY` exist on your system containing your KDB.AI Cloud portal details, these variables will automatically be used to connect.\n",
        "If these do not exist, it will prompt you to enter your KDB.AI Cloud portal session URL endpoint and API key details."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "id": "401f8162",
      "metadata": {
        "id": "401f8162"
      },
      "outputs": [],
      "source": [
        "KDBAI_ENDPOINT = (\n",
        "    os.environ[\"KDBAI_ENDPOINT\"]\n",
        "    if \"KDBAI_ENDPOINT\" in os.environ\n",
        "    else input(\"KDB.AI endpoint: \")\n",
        ")\n",
        "KDBAI_API_KEY = (\n",
        "    os.environ[\"KDBAI_API_KEY\"]\n",
        "    if \"KDBAI_API_KEY\" in os.environ\n",
        "    else getpass(\"KDB.AI API key: \")\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3b3a4932",
      "metadata": {
        "id": "3b3a4932"
      },
      "outputs": [],
      "source": [
        "session = kdbai.Session(api_key=KDBAI_API_KEY, endpoint=KDBAI_ENDPOINT)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b53a7384",
      "metadata": {
        "id": "b53a7384"
      },
      "source": [
        "##### Option 2. KDB.AI Server\n",
        "\n",
        "To use KDB.AI Server, you will need download and run your own container.\n",
        "To do this, you will first need to sign up for free [here](https://trykdb.kx.com/kdbaiserver/signup/).\n",
        "\n",
        "You will receive an email with the required license file and bearer token needed to download your instance.\n",
        "Follow instructions in the signup email to get your session up and running.\n",
        "\n",
        "Once the [setup steps](https://code.kx.com/kdbai/gettingStarted/kdb-ai-server-setup.html) are complete you can then connect to your KDB.AI Server session using `kdbai.Session` and passing your local endpoint."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "id": "f48fc536",
      "metadata": {
        "id": "f48fc536"
      },
      "outputs": [],
      "source": [
        "# session = kdbai.Session(endpoint=\"http://localhost:8082\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "48c6525c",
      "metadata": {
        "id": "48c6525c"
      },
      "source": [
        "### Define Vector DB Table Schema\n",
        "\n",
        "The next step is to define a schema for our KDB.AI table where we will store our embeddings. Our table will have two columns.\n",
        "\n",
        "At this point you will select the index and metric you want to use for searching.\n",
        "\n",
        "In this case, we will use the qHNSW index, Euclidean Distance (L2) for the search metric, and we specify the number of dimensions of our embeddings (1536)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "id": "eArvp20fSDc6",
      "metadata": {
        "id": "eArvp20fSDc6"
      },
      "outputs": [],
      "source": [
        "#Set up the schema and indexes for KDB.AI table, specifying embeddings column with 384 dimensions, Euclidean Distance, and flat index\n",
        "pdf_schema = [\n",
        "    {\"name\": \"document_id\", \"type\": \"bytes\"},\n",
        "    {\"name\": \"text\", \"type\": \"bytes\"},\n",
        "    {\"name\": \"embeddings\", \"type\": \"float32s\"}\n",
        "]\n",
        "\n",
        "indexes = [\n",
        "    {\n",
        "        \"name\": \"qhnsw_index\",\n",
        "        \"type\": \"qHnsw\",\n",
        "        \"column\": \"embeddings\",\n",
        "        \"params\": {\"dims\": 1536, \"metric\": \"L2\"},\n",
        "    }\n",
        "]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "518cfe1e",
      "metadata": {
        "id": "518cfe1e"
      },
      "source": [
        "### Create Vector DB Table\n",
        "\n",
        "Use the KDB.AI `create_table` function to create a table that matches the defined schema in the vector database."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 63,
      "id": "6e670f9e",
      "metadata": {
        "id": "6e670f9e"
      },
      "outputs": [],
      "source": [
        "# get the database connection. Default database name is 'default'\n",
        "database = session.database('default')\n",
        "\n",
        "# First ensure the table does not already exist\n",
        "try:\n",
        "    database.table(\"pdf\").drop()\n",
        "except kdbai.KDBAIException:\n",
        "    pass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "id": "e1d190db-7c19-418e-9140-3dff04c9d4c0",
      "metadata": {
        "id": "e1d190db-7c19-418e-9140-3dff04c9d4c0"
      },
      "outputs": [],
      "source": [
        "table = database.create_table(\"pdf\", schema = pdf_schema, indexes = indexes)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "466068bc",
      "metadata": {
        "id": "466068bc"
      },
      "source": [
        "We can use `query` to see our table exists but is empty."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "id": "74e7332e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "74e7332e",
        "outputId": "e07a811b-44a7-428a-9252-3a10ac15e035"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>document_id</th>\n",
              "      <th>text</th>\n",
              "      <th>embeddings</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: [document_id, text, embeddings]\n",
              "Index: []"
            ]
          },
          "execution_count": 65,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "table.query()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YvcNh1v_Wixe",
      "metadata": {
        "id": "YvcNh1v_Wixe"
      },
      "source": [
        "## 3. LlamaIndex index & query_engine setup\n",
        "Define the index: using KDB.AI as the vector store, chunk, embed, and load the document into KDB.AI"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "id": "He42izi1Soee",
      "metadata": {
        "id": "He42izi1Soee"
      },
      "outputs": [],
      "source": [
        "vector_store = KDBAIVectorStore(table)\n",
        "\n",
        "storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
        "index = VectorStoreIndex.from_documents(\n",
        "    documents,\n",
        "    storage_context=storage_context,\n",
        "    transformations=[SentenceSplitter(chunk_size=2048, chunk_overlap=0)],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e31a4ecd",
      "metadata": {
        "id": "e31a4ecd"
      },
      "source": [
        "### Verify Data Has Been Inserted\n",
        "\n",
        "Running `table.query()` should show us that data has been added."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 67,
      "id": "ee6ecb8d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ee6ecb8d",
        "outputId": "07564fc5-6d97-4e58-cfbd-0dbb6a9f8ba7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "(22, 3)\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>document_id</th>\n",
              "      <th>text</th>\n",
              "      <th>embeddings</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>b'cb35e257-647f-4953-a447-919fd99f8957'</td>\n",
              "      <td>b'Draft version August 14, 2023\\nTypeset using L ATEX default style in AASTeX631\\nThe Galactic Interstellar Object Population: A Framework for Prediction and Inference\\nMatthew J. Hopkins\\n ,1Chris Lintott\\n ,1Michele T. Bannister\\n ,2J. Ted Mackereth\\n ,3, 4, 5, \\xe2\\x88\\x97and\\nJohn C. Forbes\\...</td>\n",
              "      <td>[-0.001896298, 0.04461563, 0.016039172, 0.004387382, 0.020142527, -0.02016926, 0.014515451, -0.014675843, -0.010986833, 0.001757626, -0.019247007, 0.0028486238, -0.006916893, -0.027132934, 0.027560644, 0.03560696, -0.026772052, -0.0022705453, -0.025194867, 0.018832661, 0.0025261696, -0.014903064...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>b'f8b82c1f-d988-4f4f-acb5-0fe77d12baa2'</td>\n",
              "      <td>b'2 Hopkins et al.\\nInitially it was expected that interstellar objects would display cometary characteristics (e.g. Jewitt 2003). The pop-\\nulation\\xe2\\x80\\x99s dominant dynamical formation mechanisms would preferentially harvest more distant, ice-rich planetesimals\\nfrom the disks of the sourc...</td>\n",
              "      <td>[0.004427607, 0.00960885, -0.0025132392, 0.006977855, 0.011768823, -0.024210533, 0.029257199, -0.01263012, 0.016055124, 0.024721928, -0.0077987793, 0.008895588, -0.012596475, -0.040830884, -0.01087388, 0.019419566, -0.03523245, -0.0022962324, -0.031814177, -0.00019114243, 0.015880171, -0.0127579...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>b'35284c91-4732-4831-b78d-c2284ec603b5'</td>\n",
              "      <td>b'The Galactic ISO Population 3\\nprocesses modelled, and demonstrate this method by constraining the metallicity dependence of the ISO production\\nrate.\\n2.APOGEE AND STELLAR DENSITY MODELLING\\nTo predict the distribution of ISOs in the Milky Way, we first obtain the distribution of all stars th...</td>\n",
              "      <td>[0.039092664, -0.01292948, 0.067334704, 0.008587963, 0.011623856, -0.018899858, 0.009500632, -0.013347787, -0.028723726, -0.0010053621, -0.021295615, 0.014830874, -0.03392087, 0.0021153009, 0.018215355, 0.02143505, -0.011446392, -0.007104876, -0.03916872, 0.032222293, 0.009564012, 0.010742877, 0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>b'6854d718-c425-4db7-a4dc-2737db1538aa'</td>\n",
              "      <td>b'4 Hopkins et al.\\nfollows that the probability of finding a point (i.e. an observed star) with observables in the infinitesimal volume \\xce\\xb4O\\nis given by \\xce\\xbb(O)\\xce\\xb4O, and the total number of points (i.e. stars observed) is a Poisson random variable with mean and\\nvariance \\xce\\x9b...</td>\n",
              "      <td>[0.023402294, 0.0226368, 0.07654956, -0.010600748, 0.017483374, -0.0025510825, 0.019055374, -0.004674991, -0.0119472, 0.0058847475, -0.0091654435, 0.0012020674, -0.053557355, 0.005966765, 0.03362713, 0.014708452, 0.008085548, -0.014038643, -0.050686747, 0.036224347, -0.010135982, 0.005768556, 0....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>b'0033757f-3651-4bdd-a52a-4e20aaa2da24'</td>\n",
              "      <td>b'The Galactic ISO Population 5\\nThis particular form for the density profile has the advantage that the Poisson point process likelihood takes the\\ntractable form\\nlnL(logA , aR, az, \\xcf\\x840, \\xcf\\x89) = const + N\\x10\\nlogA\\xe2\\x88\\x92aR\\xe2\\x9f\\xa8R\\xe2\\x88\\x92R0\\xe2\\x9f\\xa9 \\xe2\\x88\\x92az\\x...</td>\n",
              "      <td>[0.0287323, 0.04074224, 0.051109564, 0.010259612, 0.029594, -0.013921836, 0.021421317, 0.0003107252, -0.0040055574, -0.00013253682, -0.015672164, 0.012669679, -0.037403155, 0.008536213, 0.019495957, 0.032125242, -0.033660144, -0.012521574, -0.04103845, -0.014177653, 0.03632603, 0.0049749697, 0.0...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                               document_id  \\\n",
              "0  b'cb35e257-647f-4953-a447-919fd99f8957'   \n",
              "1  b'f8b82c1f-d988-4f4f-acb5-0fe77d12baa2'   \n",
              "2  b'35284c91-4732-4831-b78d-c2284ec603b5'   \n",
              "3  b'6854d718-c425-4db7-a4dc-2737db1538aa'   \n",
              "4  b'0033757f-3651-4bdd-a52a-4e20aaa2da24'   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                          text  \\\n",
              "0  b'Draft version August 14, 2023\\nTypeset using L ATEX default style in AASTeX631\\nThe Galactic Interstellar Object Population: A Framework for Prediction and Inference\\nMatthew J. Hopkins\\n ,1Chris Lintott\\n ,1Michele T. Bannister\\n ,2J. Ted Mackereth\\n ,3, 4, 5, \\xe2\\x88\\x97and\\nJohn C. Forbes\\...   \n",
              "1  b'2 Hopkins et al.\\nInitially it was expected that interstellar objects would display cometary characteristics (e.g. Jewitt 2003). The pop-\\nulation\\xe2\\x80\\x99s dominant dynamical formation mechanisms would preferentially harvest more distant, ice-rich planetesimals\\nfrom the disks of the sourc...   \n",
              "2  b'The Galactic ISO Population 3\\nprocesses modelled, and demonstrate this method by constraining the metallicity dependence of the ISO production\\nrate.\\n2.APOGEE AND STELLAR DENSITY MODELLING\\nTo predict the distribution of ISOs in the Milky Way, we first obtain the distribution of all stars th...   \n",
              "3  b'4 Hopkins et al.\\nfollows that the probability of finding a point (i.e. an observed star) with observables in the infinitesimal volume \\xce\\xb4O\\nis given by \\xce\\xbb(O)\\xce\\xb4O, and the total number of points (i.e. stars observed) is a Poisson random variable with mean and\\nvariance \\xce\\x9b...   \n",
              "4  b'The Galactic ISO Population 5\\nThis particular form for the density profile has the advantage that the Poisson point process likelihood takes the\\ntractable form\\nlnL(logA , aR, az, \\xcf\\x840, \\xcf\\x89) = const + N\\x10\\nlogA\\xe2\\x88\\x92aR\\xe2\\x9f\\xa8R\\xe2\\x88\\x92R0\\xe2\\x9f\\xa9 \\xe2\\x88\\x92az\\x...   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                    embeddings  \n",
              "0  [-0.001896298, 0.04461563, 0.016039172, 0.004387382, 0.020142527, -0.02016926, 0.014515451, -0.014675843, -0.010986833, 0.001757626, -0.019247007, 0.0028486238, -0.006916893, -0.027132934, 0.027560644, 0.03560696, -0.026772052, -0.0022705453, -0.025194867, 0.018832661, 0.0025261696, -0.014903064...  \n",
              "1  [0.004427607, 0.00960885, -0.0025132392, 0.006977855, 0.011768823, -0.024210533, 0.029257199, -0.01263012, 0.016055124, 0.024721928, -0.0077987793, 0.008895588, -0.012596475, -0.040830884, -0.01087388, 0.019419566, -0.03523245, -0.0022962324, -0.031814177, -0.00019114243, 0.015880171, -0.0127579...  \n",
              "2  [0.039092664, -0.01292948, 0.067334704, 0.008587963, 0.011623856, -0.018899858, 0.009500632, -0.013347787, -0.028723726, -0.0010053621, -0.021295615, 0.014830874, -0.03392087, 0.0021153009, 0.018215355, 0.02143505, -0.011446392, -0.007104876, -0.03916872, 0.032222293, 0.009564012, 0.010742877, 0...  \n",
              "3  [0.023402294, 0.0226368, 0.07654956, -0.010600748, 0.017483374, -0.0025510825, 0.019055374, -0.004674991, -0.0119472, 0.0058847475, -0.0091654435, 0.0012020674, -0.053557355, 0.005966765, 0.03362713, 0.014708452, 0.008085548, -0.014038643, -0.050686747, 0.036224347, -0.010135982, 0.005768556, 0....  \n",
              "4  [0.0287323, 0.04074224, 0.051109564, 0.010259612, 0.029594, -0.013921836, 0.021421317, 0.0003107252, -0.0040055574, -0.00013253682, -0.015672164, 0.012669679, -0.037403155, 0.008536213, 0.019495957, 0.032125242, -0.033660144, -0.012521574, -0.04103845, -0.014177653, 0.03632603, 0.0049749697, 0.0...  "
            ]
          },
          "execution_count": 67,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "show_df(table.query())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "tiZ5HWsEThcT",
      "metadata": {
        "id": "tiZ5HWsEThcT"
      },
      "source": [
        "#### Set up the LlamaIndex Query Engine"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 68,
      "id": "u_IlkIcuTn3e",
      "metadata": {
        "id": "u_IlkIcuTn3e"
      },
      "outputs": [],
      "source": [
        "query_engine = index.as_query_engine(\n",
        "    similarity_top_k=5,\n",
        "    vector_store_kwargs={\n",
        "                    \"index\" : \"qhnsw_index\",\n",
        "                },\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8bf8650d",
      "metadata": {
        "id": "8bf8650d"
      },
      "source": [
        "## 4. Retrieve Similar Sentences & RAG\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "31ddb725-e0e7-4c00-a22b-3234eacf6bd1",
      "metadata": {
        "id": "31ddb725-e0e7-4c00-a22b-3234eacf6bd1"
      },
      "source": [
        "Now that the embeddings are stored in KDB.AI, we can perform semantic search using through the LlamaIndex query engine.\n",
        "\n",
        "### Search 1\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 69,
      "id": "b24c1179",
      "metadata": {
        "id": "b24c1179"
      },
      "outputs": [],
      "source": [
        "#search_term1 = \"The Milky Way is thought to host a huge population of interstellar objects (ISOs), numbering\"\n",
        "search_term1 = \"what is the estimated population of interstellar objects (ISOs) in the milky way\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 70,
      "id": "QbyVfwB8VTcg",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QbyVfwB8VTcg",
        "outputId": "c0dc4596-9b69-409b-9abd-b130e3b010e2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Draft version August 14, 2023\n",
            "Typeset using L ATEX default style in AASTeX631\n",
            "The Galactic Interstellar Object Population: A Framework for Prediction and Inference\n",
            "Matthew J. Hopkins\n",
            " ,1Chris Lintott\n",
            " ,1Michele T. Bannister\n",
            " ,2J. Ted Mackereth\n",
            " ,3, 4, 5, ∗and\n",
            "John C. Forbes\n",
            "2\n",
            "1Department of Physics, University of Oxford, Denys Wilkinson Building, Keble Road, Oxford, OX1 3RH, UK\n",
            "2School of Physical and Chemical Sciences—Te Kura Mat¯ u, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand\n",
            "3Just Group plc, Enterprise House, Bancroft road, Reigate, Surrey RH2 7RP, UK\n",
            "4Canadian Institute for Theoretical Astrophysics, University of Toronto, 60 St. George Street, Toronto, ON, M5S 3H8, Canada\n",
            "5Dunlap Institute for Astronomy and Astrophysics, University of Toronto, 50 St. George Street, Toronto, ON M5S 3H4, Canada\n",
            "ABSTRACT\n",
            "The Milky Way is thought to host a huge population of interstellar objects (ISOs), numbering\n",
            "approximately 1015pc−3around the Sun, which are formed and shaped by a diverse set of processes\n",
            "ranging from planet formation to galactic dynamics. We define a novel framework: firstly to predict\n",
            "the properties of this Galactic ISO population by combining models of processes across planetary\n",
            "and galactic scales, and secondly to make inferences about the processes modelled, by comparing the\n",
            "predicted population to what is observed. We predict the spatial and compositional distribution of the\n",
            "Galaxy’s population of ISOs by modelling the Galactic stellar population with data from the APOGEE\n",
            "survey and combining this with a protoplanetary disk chemistry model. Selecting ISO water mass\n",
            "fraction as an example observable quantity, we evaluate its distribution both at the position of the Sun\n",
            "and averaged over the Galactic disk; our prediction for the Solar neighbourhood is compatible with the\n",
            "inferred water mass fraction of 2I/Borisov. We show that the well-studied Galactic stellar metallicity\n",
            "gradient has a corresponding ISO compositional gradient. We also demonstrate the inference part of\n",
            "the framework by using the current observed ISO composition distribution to constrain the parent star\n",
            "metallicity dependence of the ISO production rate. This constraint, and other inferences made with\n",
            "this framework, will improve dramatically as the Vera C. Rubin Observatory Legacy Survey of Space\n",
            "and Time (LSST) progresses and more ISOs are observed. Finally, we explore generalizations of this\n",
            "framework to other Galactic populations, such as that of exoplanets.\n",
            "Keywords: Interstellar objects (52), Small Solar System bodies(1469), Galaxy Evolution (594)\n",
            "1.INTRODUCTION\n",
            "1I/‘Oumuamua (Meech et al. 2017) and 2I/Borisov1are the first two observed samples from a highly numerous\n",
            "population: interstellar objects (ISOs). Estimated to number ∼1015pc−3around the Sun (Engelhardt et al. 2017;\n",
            "Do et al. 2018), they are implied to have a spatial distribution spanning the entire Galaxy. This population has been\n",
            "predicted to exist for decades (McGlynn & Chapman 1989), based on models of the accretion and migration of the\n",
            "giant planets, which predict that 75–85% of cometary bodies initially in the Solar System must have been scattered\n",
            "into interstellar space (Fernandez & Ip 1984; Brasser et al. 2006). Modern exoplanet surveys consistently find that\n",
            "giant planets are common across the Galaxy around stars with a range of spectral types (Fulton et al. 2021; Sabotta\n",
            "et al. 2021). This makes planetesimal scattering common across the Galaxy. A significant number of planetesimals can\n",
            "also be ejected by close stellar flybys early in a planetary system’s life (e.g. Pfalzner et al. 2021). The protoplanetary\n",
            "disks of other stars are therefore expected to be a source of ISOs (Stern 1990; Moro-Mart´ ın 2022).\n",
            "Corresponding author: Matthew Hopkins\n",
            "matthew.hopkins@physics.ox.ac.uk\n",
            "∗Banting Fellow\n",
            "1https://minorplanetcenter.net/mpec/K19/K19RA6.html and https://minorplanetcenter.net/mpec/K19/K19S72.htmlarXiv:2308.05801v1  [astro-ph.EP]  10 Aug 2023\n",
            "____________________\n",
            "14 Hopkins et al.\n",
            "because the probability that two randomly selected ISOs have the same parent star is approximately equal to one over\n",
            "the total number of stars which could contribute ISOs to the population around the Sun. The ISO sample likelihood\n",
            "can then be combined with priors to form a posterior distribution on the model parameters:\n",
            "p(θ|O1, . . . ,ON)∝p(θ)·Y\n",
            "ip(Oi|θ) (10)\n",
            "This posterior can be used to calculate estimates and confidence intervals for the values of the parameters θ. An\n",
            "example of this calculation is given in section 6.3.\n",
            "5.2. Additional Properties of ISOs\n",
            "As with all minor planets, there are many observable quantities for ISOs that could be used in this framework in\n",
            "the set of observable properties O. If the distribution of ISOs in these properties can be predicted, the inference\n",
            "method of §5.1 can be used to compare the predicted distribution to the observed distribution in these properties, to\n",
            "make inferences about the models used. If multiple properties are included in O, then the joint distribution of ISOs\n",
            "in these properties can be predicted and used in inference. Including different ISO properties in the framework will\n",
            "allow inferences to be made about different processes affecting the ISO population. We briefly consider several such\n",
            "properties here.\n",
            "The Milky Way stellar population is broadly divided between different chemo-dynamical populations: the thin disk\n",
            "(high [Fe /H], low velocity dispersion), thick disk (low [Fe /H], high velocity dispersion), and the halo (very low [Fe /H],\n",
            "radial orbits) (Recio-Blanco et al. 2014; Horta et al. 2023). Since the composition of an ISO depends on the metallicity\n",
            "of the star it formed around, each chemo-dynamical stellar population will contribute ISOs to the Galactic population\n",
            "with a distinct joint distribution in both composition and velocity. Therefore, including the velocities of ISOs in this\n",
            "framework could be used to tie ISOs to the chemo-dynamical stellar populations their parent star belongs to (Eubanks\n",
            "et al. 2021).\n",
            "As an alternative measurement of composition to water mass fraction, the carbon-to-oxygen ratio of a cometary ISO\n",
            "can be estimated from its coma. This contains information about an ISO’s formation location in the protoplanetary\n",
            "disk relative to the H 2O, CO 2and CO ice lines; modelling suggests it would be a useful measure for future ISOs\n",
            "(Seligman et al. 2022).\n",
            "Further additions to our framework could include predictions of the size distribution and aspect ratio distribution\n",
            "of ISOs, which would be especially interesting due to 1I/‘Oumuamua’s extreme shape. The prediction of these distri-\n",
            "butions would depend on the stellar population via models of planetesimal formation which could link the size and\n",
            "shape distribution of planetesimals to the properties of their natal star. Finally, including the binarity rate of ISOs\n",
            "would test models of the applicability of formation mechanism of planetesimals to other protoplanetary disks, as the\n",
            "compact binarity of trans-Neptunian objects does for the Solar System protoplanetary disk (e.g. Nimmo et al. 2018).\n",
            "This could also constrain the ejection mechanisms of ISOs: loosely-bound wide planetesimal binaries would not survive\n",
            "scattering by a giant planet, but would survive a more gentle gravitational interaction with a stellar flyby.\n",
            "5.3. Generalisation to Other Galactic Populations\n",
            "Our method could easily be generalised to any Galaxy-wide population with a dependence on the properties of stars,\n",
            "simply by replacing the “ISO recipe” of section 3.\n",
            "For example, the distribution of planets through the Galaxy could be predicted by substituting a model of the\n",
            "occurrence rate of planets as a function of their host star’s metallicity. This is based on the planet-metallicity correlation\n",
            "(Fischer & Valenti 2005; Osborn & Bayliss 2020): as noted earlier that dust is necessary for planets, stars with higher\n",
            "metallicity are more likely to host planets. Here, the Milky Way metallicity gradient means that planets are more\n",
            "common towards the Galactic centre. This is a testable prediction with microlensing surveys such as OGLE (Udalski\n",
            "et al. 2015), which continue to find exoplanets between the Solar System and the Galactic centre (Tsapras 2018) with\n",
            "distances estimated from followup observations (Vandorou et al. 2023). If ISOs seed planet formation as hypothesised\n",
            "by Pfalzner & Bannister (2019), the ISO gradient we predict in §4.1 could also produce a signature in the planetary\n",
            "population.\n",
            "6.DISCUSSION\n",
            "6.1. Comparison to Previous Work\n",
            "____________________\n",
            "The Galactic ISO Population 15\n",
            "As described in section 3.1, Cabral et al. (2023) updates the protoplanetary disk chemical model of Bitsch & Battistini\n",
            "(2020) that we use, and note that the trend of planetesimal water mass fraction decreasing with stellar metallicity is\n",
            "robust. They do however find that for the APOGEE survey, there was a smaller variation in fH2Oover the same range\n",
            "in [Fe /H] compared to the GalaH data used in Bitsch & Battistini (2020). If the true variation in fH2Ois smaller than\n",
            "in the model of this work, our predictions may overestimate the width of the ISO water mass fraction distribution.\n",
            "Lintott et al. (2022) made a prediction of the ISO population of a simulated Milky Way Galaxy from the EAGLE\n",
            "hydrodynamical cosmological simulation (Schaye et al. 2015), using a model equivalent to that of this work with β= 0.\n",
            "Whereas we predict a single-peaked fH2Odistribution, they predicted an ISO distribution with a significant number of\n",
            "ISOs with water mass fraction both below and above the range of the protoplanetary disk model, which they interpret\n",
            "as a bimodal distribution in ISO composition.\n",
            "There are expected reasons for the difference between the prediction here, based on the observed Milky Way stellar\n",
            "population, and the prediction by Lintott et al. (2022) based on the simulated EAGLE Galaxy. The EAGLE Galaxy\n",
            "has a much wider [Fe /H] distribution than the Milky Way, with many more stars outside of the [Fe /H] range of the\n",
            "protoplanetary disk model. As a smoothed particle hydrodynamics simulation, EAGLE is susceptible to producing\n",
            "Galaxies with [Fe /H] distributions wider than those observed in nature, due to underestimating metal mixing between\n",
            "particles (Wiersma et al. 2009). Therefore the results of this work, based on the observed stellar population of the\n",
            "Milky Way, should give a much more accurate prediction for the Milky Way’s population of ISOs.\n",
            "6.2. Distinguishing Local and Galactic Populations of ISOs\n",
            "The results of §4.1 show that the well-studied metallicity gradient of the Milky Way has a corresponding ISO\n",
            "composition gradient, with ISOs generally having a higher water mass fraction at larger Galactocentric radii. Although\n",
            "we can only observe the compositions of ISOs which pass through the inner Solar System, it is still instructive to\n",
            "model how the distribution of ISOs varies across a wider portion of the Galactic disk. This is because we have made\n",
            "assumptions about the Galactic dynamics of ISOs, under which the distribution of ISOs at a point in the Galaxy\n",
            "corresponds to the distribution of stars at that same point. If these assumptions break down, then the particular way\n",
            "in which they break down will affect the population of ISOs detectable in the Solar System in a related, calculable\n",
            "way.\n",
            "For example, radial migration, caused by the non-axisymmetric potential of spiral arms, flattens the the Milky Way\n",
            "metallicity gradient by blurring the metallicity distribution in the radial direction (Vickers et al. 2021). This widens\n",
            "the stellar metallicity distribution around the Sun as stars migrate in from the metal-poor outer disk and out from\n",
            "metal-rich inner disk. However, ISOs may undergo less radial migration than stars, due to the random motion given\n",
            "to them by their ejection from their home planetary system (Daniel & Wyse 2015). The stars currently in the Solar\n",
            "neighbourhood may thus have a wider range of Galactocentric radii of origin than the ISOs. This would make the\n",
            "distribution of observable ISO compositions narrower than would be predicted from the distribution of stars.\n",
            "Additionally, the low velocity of 1I/‘Oumuamua relative to the local standard of rest implies that it — and therefore\n",
            "a large fraction of observable ISOs — could come from local star-forming regions (e.g. Hallatt & Wiegert 2020). This is\n",
            "also testable: their compositions will match those predicted from the metallicities of the nearest star forming regions.\n",
            "In addition, if the velocity distribution of ISOs is included in future work as described in 5.2, it may be possible to\n",
            "trace individual ISOs back to the star forming regions they came from by matching them up with both composition\n",
            "and velocity. This would be hugely advantageous for studying planetesimal formation, as it would allow us to pair the\n",
            "properties of detected ISOs directly with observations of their parent planetary systems.\n",
            "6.3. An Estimation of the ISO Production Metallicity Dependence\n",
            "The two different predictions made in §4 demonstrate that the Galactic population of ISOs is sensitive to small\n",
            "changes to the processes that affect their formation and evolution. This means that if models of these processes can\n",
            "be combined to make accurate predictions of the ISO population, then the framework described in §5 can be used to\n",
            "make inferences about those processes.\n",
            "In particular, the predictions of sections 4.1 and 4.2 show that the ISO distribution around the Sun is sensitive to\n",
            "the metallicity dependence of the number of ISOs produced by each star, β. As outlined in section 5.1, these two\n",
            "predictions can then be compared to a sample of ISOs. In our example physical property, this is done by calculating\n",
            "the likelihood of each of these predictions producing the observed distribution of water mass fractions, such as for the\n",
            "ISOs expected to be found by the Legacy Survey of Space and Time (LSST) of the Vera C. Rubin Observatory. For\n",
            "____________________\n",
            "The Galactic ISO Population 17\n",
            "ISOfH2Orange Fraction of ISOs around Sun Fraction of ISOs in Milky Way Disk\n",
            "fH2O<0.07 0.022 0.031\n",
            "0.07≤fH2O≤0.51 0.960 0.930\n",
            "0.51< fH2O 0.018 0.040\n",
            "Table 3. Predicted fraction of ISOs in each fH2Orange, evaluated at the position of the Sun and integrated over the Milky\n",
            "Way dist, with maximum a posteriori estimate of β= 1.33.\n",
            "6.4. Selection Effects\n",
            "It should be noted that the predictions in this work ignore some specific effects which will influence the population of\n",
            "ISOs we expect to observe from the Earth. The predictions in §4 are the distribution of ISOs in a smooth Galaxy-wide\n",
            "distribution, evaluated at the location of the Solar System. Gravitational focussing by the Sun will increase the density\n",
            "of ISOs in the inner solar system in a velocity-dependent manner (e.g. Engelhardt et al. 2017; Forbes & Loeb 2019;\n",
            "Dehnen & Hands 2022), and therefore also in a composition-dependent manner. This is due to the fact that we expect\n",
            "compositionally and dynamically distinct populations of ISOs to come from the chemo-dynamically distinct stellar\n",
            "populations in the Milky Way thin and thick disks (c.f. Eubanks et al. 2021). This could handily be modelled when\n",
            "incorporating ISO velocities into the predictions of this framework.\n",
            "Additionally, the Pan-STARRS near-Earth object survey (Chambers et al. 2016) that detected 1I/‘Oumuamua, the\n",
            "observations of amateur astronomers such as Gennadiy Borisov who discovered 2I/Borisov, and the LSST which will\n",
            "discover tens more ISOs: all have highly non-trivial selection functions. Since in order to be discovered an ISO needs\n",
            "to be detected in multiple observations which can be linked as the same object (e.g. Meech et al. 2017; Schwamb\n",
            "et al. 2023), these selection functions are dependent on ISO size, approach velocity, perihelion, and composition.\n",
            "Composition has a direct link to the detectability of ISOs, as 2I-sized and cometary ISOs will be more likely to be\n",
            "detected, as they will form a coma as they approach the Sun. These selection effects will need to be accurately\n",
            "accounted for, in order for the Bayesian framework to produce accurate inferences about the processes affecting the\n",
            "Galactic ISO population.\n",
            "7.CONCLUSION\n",
            "In advance of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST), we lay out a framework to\n",
            "predict the Galactic distribution of ISOs, using the stellar population of the Milky Way. Using the method of Bovy\n",
            "et al. (2016a), we fit simple density models to a sample of red giants in APOGEE binned in [Fe /H] and [ α/Fe], and use\n",
            "these to evaluate the “sine morte” metallicity distribution of stars throughout the Galaxy’s integrated history, across\n",
            "the Galactic disk. Under the assumption that the spatial distribution of a population of ISOs will be the same as the\n",
            "sine morte distribution of stars which formed them, we use the protoplanetary disk model of Bitsch & Battistini (2020)\n",
            "to map the metallicity distribution of stars to the distribution of ISO water mass fractions. Localising our model to\n",
            "the Solar neighbourhood, we predict that 95% of ISOs around the Sun have water mass fraction fH2Obetween 0.07\n",
            "and 0.51, with a peak around 0.35.\n",
            "By considering the distribution of ISOs over the Galactic disk, we show that the well-studied Milky Way metallicity\n",
            "gradient has an equivalent gradient in ISO composition, with the median ISO water mass fraction increasing with\n",
            "distance from the Galactic Centre as the median stellar metallicity decreases. This causes the ISO water mass fraction\n",
            "distribution averaged over the Milky Way disk to be wider that the distribution around the Sun. Since we also predict\n",
            "higher-metallicity stars produce more ISOs than lower-metallicity stars, the Milky Way metallicity gradient implies\n",
            "that the radial ISO density profile is steeper than the exponential radial stellar density profile, making ISOs much\n",
            "more common in the inner Galactic disk than in the outer disk.\n",
            "We also set out a Bayesian inference framework, which can compare predictions of the ISO distribution to the\n",
            "sample observed by the LSST, in order to make inferences about the many astrophysical processes which influence the\n",
            "ISO population. To demonstrate its use, we use the composition measurement of 2I/Borisov to calculate a maximum\n",
            "a posteriori estimate for the power law slope of the ISO production metallicity dependence, β, to be 1.33, with a\n",
            "symmetric 90% confidence interval of ( −1.3,7.2).\n",
            "Encoded in the population of ISOs we observe is a wealth of information about processes through Galactic history,\n",
            "from the evolution of the Milky Way to planet formation. The framework set out in this work is a novel approach that\n",
            "will allow us to appreciate how these treasures can further our understanding on both planetary and Galactic scales.\n",
            "____________________\n",
            "2 Hopkins et al.\n",
            "Initially it was expected that interstellar objects would display cometary characteristics (e.g. Jewitt 2003). The pop-\n",
            "ulation’s dominant dynamical formation mechanisms would preferentially harvest more distant, ice-rich planetesimals\n",
            "from the disks of the source systems. More of the cometary ISO population passing through the Solar System could\n",
            "be detected than rocky ISOs, as comae make cometary ISOs brighter down to smaller diameters (Engelhardt et al.\n",
            "2017). 2I/Borisov appeared relatively similar in size and composition to Solar System comets (Jewitt & Seligman\n",
            "2022). Its distinctive features were an exceptionally high CO and NH 2content, implying it formed on the edge of its\n",
            "home system’s protoplanetary disk, beyond the hypervolatile CO ice line (Bodewits et al. 2020; Cordiner et al. 2020).\n",
            "However, 1I/‘Oumuamua had a mix of observed characteristics. A 160-m scale object, it lacked a coma in deep\n",
            "imaging (Jewitt et al. 2017) or detectable outgassing in CO, CO 2(Trilling et al. 2018) or CN (Ye et al. 2017). Despite\n",
            "this, it still underwent non-gravitational acceleration similar to that experienced by Solar System comets (Micheli\n",
            "et al. 2018). The large amplitude of its light curve implied a high-aspect-ratio shape (Mashchenko 2019). At the time,\n",
            "this combination of factors seemed unusual, although the surface reflectance properties were consistent with outer\n",
            "Solar System bodies (e.g. Bannister et al. 2017). Hypotheses for the composition and formation of 1I/‘Oumuamua\n",
            "that match the limited data remain varied. It may be a planetesimal (’Oumuamua ISSI Team et al. 2019); a fragment\n",
            "of a comet devolatized by passages close to its parent star before ejection (Raymond et al. 2018a); an icy fractal\n",
            "aggregate (Moro-Mart´ ın 2019); a hydrogen iceberg formed in a molecular cloud (Seligman & Laughlin 2020); or a\n",
            "nitrogen ice fragment from the surface of a Pluto-like dwarf planet (Jackson & Desch 2021). Recently, observation\n",
            "of similarly-small near-Earth asteroids have identified objects with the lightcurve amplitudes seen in 1I. Additionally,\n",
            "Farnocchia et al. (2023) and Seligman et al. (2023) report six asteroids with significant non-gravitational acceleration\n",
            "and no coma. While from the two ISOs found so far, the compositions of interstellar objects are clearly varied, 1I may\n",
            "be less extreme than it was first considered.\n",
            "ISOs formed in a protoplanetary disk carry information about their home systems in their composition: we have\n",
            "samples of other planetary systems coming to us for study. The composition of a protoplanetary disk correlates with the\n",
            "elemental abundances of its central star, due to their formation from the same gas and dust in a molecular cloud core\n",
            "(¨Oberg & Bergin 2021). We can thus expect stars of different metallicities to produce ISOs of different compositions.\n",
            "The composition of this gas and dust varies in both space and time as the Galaxy chemically evolves due to stellar\n",
            "nucleosynthesis, making the current Galactic ISO population sensitive to the entire history and evolution of the Milky\n",
            "Way over cosmic time (Tinsley & Cameron 1974; Lintott et al. 2022). Since the occurrence of planetesimal-scattering\n",
            "giant planets also has a metallicity dependence (Fischer & Valenti 2005), the relative occurrence of ISOs of different\n",
            "compositions will therefore carry information about the Galaxy’s distribution of planetary architectures. Finally, we\n",
            "expect many ISOs to outlive their parent stars, as they occupy a similar environment to Gyr-old Oort cloud comets.\n",
            "At minor-planet sizes, ISOs are not subject to any known destructive forces in the ISM (Guilbert-Lepoutre et al. 2015),\n",
            "other than minor erosion by dust, in their frequent passages through molecular clouds (Pfalzner et al. 2020). They\n",
            "could also be disrupted or devolatilised in rare close encounters with stars (Raymond et al. 2018a; Forbes & Loeb\n",
            "2019). ISOs thus present the possibility of studying long-lost planetary systems.\n",
            "The Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) (Ivezi´ c et al. 2019) is predicted to provide\n",
            "a sample of tens of 1I/‘Oumuamua-like ISOs (e.g. Levine et al. 2021), as well as any more 2I/Borisov-like cometary\n",
            "ISOs that enter our Solar System. This is in addition to the continuing contributions of the NEO surveys and other\n",
            "observatories that found 1I and 2I in the first place (Meech et al. 2017). These will provide a large and varied sample\n",
            "of interstellar objects, for study and comparison to predictions.\n",
            "The many dependencies over planetary and Galactic scales make the observed ISO population a fascinating tool:\n",
            "it has the potential to test models in both Galaxy and planetary physics, with an entirely different set of biases\n",
            "to traditional methods. Lintott et al. (2022) introduced the concept of predicting the composition of a Galaxy’s\n",
            "population of ISOs from the Galaxy’s stellar distribution. Using simulated Galaxies from EAGLE (Schaye et al. 2015),\n",
            "they showed that the water content of ISOs was sensitive to Galactic star formation history. In this work, we develop\n",
            "this method and apply it to the stellar population of the Milky Way, estimated with data from the APOGEE survey, to\n",
            "predict a broader set of properties of our own Galaxy’s population of interstellar objects. We predict the distribution\n",
            "of ISOs in both their spatial position in the Galaxy and their water mass fraction. By evaluating this distribution at\n",
            "the current position of the Solar System, we predict the properties of the population of ISOs from which the observed\n",
            "sample of the 2020s will be drawn, and we compare this to the whole-Galaxy distribution. We then detail a Bayesian\n",
            "method of comparing the predicted and observed distributions to make inferences about the planetary and Galactic\n",
            "____________________\n"
          ]
        }
      ],
      "source": [
        "retrieved_chunks = query_engine.retrieve(search_term1)\n",
        "for i in retrieved_chunks:\n",
        "    print(i.node.get_text())\n",
        "    print(\"____________________\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9R4vgFZMVvv9",
      "metadata": {
        "id": "9R4vgFZMVvv9"
      },
      "source": [
        "We can see these sentences do reference our search term 'number of interstellar objects in the milky way' in some way."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "n0n7OglvVvvA",
      "metadata": {
        "id": "n0n7OglvVvvA"
      },
      "source": [
        "### Now we can perform RAG, passing the retrieved chunks from above to the LLM for a generate response:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 71,
      "id": "ttZpktxET3tq",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ttZpktxET3tq",
        "outputId": "8395b5b0-6645-46ba-f56a-97069c10afb9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The estimated population of interstellar objects (ISOs) in the Milky Way is approximately 10^15 per cubic parsec around the Sun.\n"
          ]
        }
      ],
      "source": [
        "result = query_engine.query(search_term1)\n",
        "print(result.response)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "91762de7",
      "metadata": {
        "id": "91762de7"
      },
      "source": [
        "### Search 2\n",
        "\n",
        "Let's try another search term."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 72,
      "id": "e7486a9c",
      "metadata": {
        "id": "e7486a9c"
      },
      "outputs": [],
      "source": [
        "search_term2 = \"how does planet formation occur\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 73,
      "id": "6765075c",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6765075c",
        "outputId": "b70454e9-0436-4266-fba6-65852ad68939"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "14 Hopkins et al.\n",
            "because the probability that two randomly selected ISOs have the same parent star is approximately equal to one over\n",
            "the total number of stars which could contribute ISOs to the population around the Sun. The ISO sample likelihood\n",
            "can then be combined with priors to form a posterior distribution on the model parameters:\n",
            "p(θ|O1, . . . ,ON)∝p(θ)·Y\n",
            "ip(Oi|θ) (10)\n",
            "This posterior can be used to calculate estimates and confidence intervals for the values of the parameters θ. An\n",
            "example of this calculation is given in section 6.3.\n",
            "5.2. Additional Properties of ISOs\n",
            "As with all minor planets, there are many observable quantities for ISOs that could be used in this framework in\n",
            "the set of observable properties O. If the distribution of ISOs in these properties can be predicted, the inference\n",
            "method of §5.1 can be used to compare the predicted distribution to the observed distribution in these properties, to\n",
            "make inferences about the models used. If multiple properties are included in O, then the joint distribution of ISOs\n",
            "in these properties can be predicted and used in inference. Including different ISO properties in the framework will\n",
            "allow inferences to be made about different processes affecting the ISO population. We briefly consider several such\n",
            "properties here.\n",
            "The Milky Way stellar population is broadly divided between different chemo-dynamical populations: the thin disk\n",
            "(high [Fe /H], low velocity dispersion), thick disk (low [Fe /H], high velocity dispersion), and the halo (very low [Fe /H],\n",
            "radial orbits) (Recio-Blanco et al. 2014; Horta et al. 2023). Since the composition of an ISO depends on the metallicity\n",
            "of the star it formed around, each chemo-dynamical stellar population will contribute ISOs to the Galactic population\n",
            "with a distinct joint distribution in both composition and velocity. Therefore, including the velocities of ISOs in this\n",
            "framework could be used to tie ISOs to the chemo-dynamical stellar populations their parent star belongs to (Eubanks\n",
            "et al. 2021).\n",
            "As an alternative measurement of composition to water mass fraction, the carbon-to-oxygen ratio of a cometary ISO\n",
            "can be estimated from its coma. This contains information about an ISO’s formation location in the protoplanetary\n",
            "disk relative to the H 2O, CO 2and CO ice lines; modelling suggests it would be a useful measure for future ISOs\n",
            "(Seligman et al. 2022).\n",
            "Further additions to our framework could include predictions of the size distribution and aspect ratio distribution\n",
            "of ISOs, which would be especially interesting due to 1I/‘Oumuamua’s extreme shape. The prediction of these distri-\n",
            "butions would depend on the stellar population via models of planetesimal formation which could link the size and\n",
            "shape distribution of planetesimals to the properties of their natal star. Finally, including the binarity rate of ISOs\n",
            "would test models of the applicability of formation mechanism of planetesimals to other protoplanetary disks, as the\n",
            "compact binarity of trans-Neptunian objects does for the Solar System protoplanetary disk (e.g. Nimmo et al. 2018).\n",
            "This could also constrain the ejection mechanisms of ISOs: loosely-bound wide planetesimal binaries would not survive\n",
            "scattering by a giant planet, but would survive a more gentle gravitational interaction with a stellar flyby.\n",
            "5.3. Generalisation to Other Galactic Populations\n",
            "Our method could easily be generalised to any Galaxy-wide population with a dependence on the properties of stars,\n",
            "simply by replacing the “ISO recipe” of section 3.\n",
            "For example, the distribution of planets through the Galaxy could be predicted by substituting a model of the\n",
            "occurrence rate of planets as a function of their host star’s metallicity. This is based on the planet-metallicity correlation\n",
            "(Fischer & Valenti 2005; Osborn & Bayliss 2020): as noted earlier that dust is necessary for planets, stars with higher\n",
            "metallicity are more likely to host planets. Here, the Milky Way metallicity gradient means that planets are more\n",
            "common towards the Galactic centre. This is a testable prediction with microlensing surveys such as OGLE (Udalski\n",
            "et al. 2015), which continue to find exoplanets between the Solar System and the Galactic centre (Tsapras 2018) with\n",
            "distances estimated from followup observations (Vandorou et al. 2023). If ISOs seed planet formation as hypothesised\n",
            "by Pfalzner & Bannister (2019), the ISO gradient we predict in §4.1 could also produce a signature in the planetary\n",
            "population.\n",
            "6.DISCUSSION\n",
            "6.1. Comparison to Previous Work\n",
            "____________________\n",
            "The Galactic ISO Population 9\n",
            "During the writing of this paper, Cabral et al. (2023) was published, containing an updated version of the Bitsch\n",
            "& Battistini (2020) chemical model which included data from GalaH DR3 (Buder et al. 2021) and APOGEE DR17\n",
            "(Abdurro’uf et al. 2022), and allowed for variation in [ α/Fe] as well as [Fe /H]. They found that the water mass fraction\n",
            "of planetesimals, consistent with the Bitsch & Battistini (2020) assumptions we use, is most dependent on [Fe /H].\n",
            "They also confirmed the trend between [Fe /H] and fH2Ofound in Bitsch & Battistini (2020). However, they also\n",
            "found that for the APOGEE survey there was a smaller variation in fH2Oover the same range in [Fe /H] compared to\n",
            "the GalaH data used in their work. Cabral et al. (2023) notes that this means that the trends seen are robust. The\n",
            "findings of Cabral et al. (2023) do mean that the predictions of the ISO distributions in our work may overestimate\n",
            "the width of the distribution in fH2O.\n",
            "We assume that every star produces ISOs, and the number of ISOs produced by each star depends on its mass and\n",
            "metallicity. Lu et al. (2020) argue that the mass of planet-forming material in a protoplanetary disk is proportional\n",
            "to both the mass of the host star mass M∗and its metal mass fraction Z— well approximated by Z⊙10[Fe/H]for\n",
            "small values of Z ( Z⊙= 0.0153, Caffau et al. (2011)). In the absence of confirmed and comprehensive knowledge\n",
            "of ISO formation mechanisms, we use this as a reasonable proxy for the number of ISOs produced by each star.\n",
            "However, the number of ISOs produced by a star may not be simply proportional to the mass of planet-forming\n",
            "material, because ISO production also requires the ejection of planetesimals — which is dependent on system dynamical\n",
            "architecture. One major ISO ejection pathway is scattering by giant planets, the occurrence of which has its own\n",
            "metallicity dependence (Osborn & Bayliss 2020). Additionally, scattering by giant planets may form ISOs with extra\n",
            "fragmentation, reweighting the number distribution towards lower masses and increasing the number of ISOs produced\n",
            "from the same mass of planet-forming material (Raymond et al. 2018b).\n",
            "We therefore assume the number of ISOs produced by each star is proportional to its mass, while incorporating into\n",
            "the model a power law dependence on metallicity mass fraction: thus the number of ISOs produced is proportional to\n",
            "M∗·10β[Fe/H]. Here β= 1 corresponds to the simple assumption that the number of ISOs produced is proportional to\n",
            "the mass of planet-forming material. β= 0 corresponds to no metallicity dependence at all. However, we expect β >0:\n",
            "planetesimals require some fraction of dust and ice in order to exist, i.e. at the elemental level, C/N/O, Al/Si, &c.\n",
            "must be present, so a minimum metallicity constraint must exist (Johnson & Li 2012). We do not model the constant\n",
            "of proportionality here, as this depends on the size distribution of ISOs, which remains observationally unconstrained\n",
            "with only two ISOs (’Oumuamua ISSI Team et al. 2019; Jewitt & Seligman 2022). We predict only the normalised\n",
            "distribution of ISO water mass fractions.\n",
            "3.2. Predicting the ISO Population From the Stellar Population\n",
            "Since the majority of ISOs are expected to be ejected by dynamical processes within several hundred Myr of a\n",
            "star’s formation (Pfalzner & Bannister 2019; Fitzsimmons et al. 2023), we assume that the birth of a star and release\n",
            "of its ISOs are contemporaneous, and omit accounting for any delay. As Oort cloud comets, which also occupy an\n",
            "interstellar environment (Kaib & Volk 2022), do not exhibit substantive erosion after 4.5 Gyr (Stern & Shull 1988),\n",
            "we continue with the expectation that ISO erosional processes are broadly similar to those of Solar System comets\n",
            "(e.g. Guilbert-Lepoutre et al. 2015). This implies ISOs from stars prior to the Sun will outlive their parent stars. We\n",
            "thus use the sine morte stellar distribution established in §2.3. For assessing the water mass fraction of the resulting\n",
            "population, we assume processing in the interstellar medium has a negligible effect5, and that ISOs broadly represent\n",
            "the water mass fraction of their source planetesimal populations.\n",
            "To link the spatial distribution of stars throughout the Galaxy to the ISOs they produced, we need to model their\n",
            "combined Galactic dynamics. ISOs are not expected to remain near to their parent stars on Gyr timescales, and stars\n",
            "do die. However, it is helpful to consider what the behaviour of the ISO population is in the case that neither of these\n",
            "are true. If ISOs did remain near their parent stars and both ISOs and stars existed for an infinite length of time,\n",
            "the number density distribution of ISOs with a given water mass fraction fH2Oand position in the Galaxy xin the\n",
            "present day, denoted nISO(x, fH2O), would be equal to the number of ISOs produced by stars currently at the same\n",
            "position and of the corresponding metallicity [Fe /H]. Under the model of §3.1, the number of ISOs produced by a\n",
            "star of mass M∗and metallicity [Fe /H] is proportional to M∗·10β[Fe/H], thus\n",
            "nISO(x, fH2O|β)∝10β[Fe/H]·d[Fe/H]\n",
            "dfH2O·ρ(x,[Fe/H]) (7)\n",
            "5While this is valid for water, it may not hold for hypervolatiles; e.g. Seligman et al. (2022) suggest that processing in the interstellar\n",
            "medium will remove CO and CO 2relative to H 2O. However, this is countered by the CO-rich nature of 2I.\n",
            "____________________\n",
            "10 Hopkins et al.\n",
            "where ρ(x,[Fe/H]) is the mass density distribution of stars at position xwith the metallicity [Fe /H] corresponding to\n",
            "the ISO water mass fraction fH2O, and d[Fe /H]/dfH2Ois the gradient of the relationship between [Fe /H] and fH2O\n",
            "described in §3.1.\n",
            "The fact that stars do in fact die is then corrected for by replacing ρ(x,[Fe/H]) in this equation with ρsm(x,[Fe/H]),\n",
            "the sine morte stellar mass density introduced in §2.3. Correcting for the fact that ISOs disperse from near their\n",
            "parent stars is more complex, so we proceed with some simplifying assumptions. Once ejected, unless its resultant\n",
            "velocity exceeds the Galactic escape velocity, an ISO will orbit the Galactic centre, as stars do. We assume ISOs\n",
            "are ejected relatively slowly from their parent planetary systems compared to the stellar velocity dispersion. This\n",
            "is justified assuming ejection velocities <10 km s−1, the maximum ejection velocity from a planetary system under\n",
            "an expected suite of scattering mechanisms (Pfalzner & Bannister 2019; Fitzsimmons et al. 2023), and the velocity\n",
            "dispersions ≳20 km s−1measured in the Solar Neighbourhood (Anguiano et al. 2020). Stars form on near-circular\n",
            "orbits around the Galactic centre (Frankel et al. 2020), so a cloud of recently ejected ISOs will all have similar orbits\n",
            "to their parent star: nearly circular with similar ranges of oscillation in Galactocentric Randz. However, the slight\n",
            "differences in their orbits will give the ISOs different orbital periods around the Galactic centre, meaning they will\n",
            "disperse along their similar near-circular orbital paths. Therefore, though ISOs do not stay near their parent star, we\n",
            "assume here that they only disperse in the azimuthal direction, and remain in the same Randzrange as their parent\n",
            "star. Under this assumption equation 7 still holds if the stellar density model is axisymmetric, depending only on R\n",
            "andz, as does ours:\n",
            "nISO(R, z, f H2O|β)∝10β[Fe/H]·d[Fe/H]\n",
            "dfH2O·ρsm(R, z, [Fe/H]) (8)\n",
            "Orbits around the Galactic centre can evolve with time, due to the influence of perturbing potentials such as spiral\n",
            "arms, the bar and molecular clouds. These effects can cause both dynamical ‘heating’, an increase in the size of radial\n",
            "and vertical excursions from a circular orbit, and ‘migration’, changes to the radius of an orbit while it remains nearly\n",
            "circular (Sellwood & Binney 2002). We introduced Eq. 8 with the assumption that an ISO will stay in the same range\n",
            "ofRandzas its parent star. Due to their azimuthal separation, the star and ISOs will experience slightly different\n",
            "perturbing potentials, causing their orbits to evolve adjacently but independently. However, if the Galactic stellar and\n",
            "ISO distributions are sufficiently axisymmetric, perturbations will consistently change together both the orbits of stars\n",
            "and the orbits of a corresponding number of ISOs. Thus, Eq. 8 holds for our model.\n",
            "In this work we predict the distribution of ISOs in fH2Oboth at particular values of Randzand integrated over\n",
            "the whole Milky Way. Since we do not model the total number of ISOs, we remove the need for the constant of\n",
            "proportionality in Eq. 8 by normalising each nISOdistribution we calculate with\n",
            "p(fH2O|β) =nISO(fH2O|β)R\n",
            "nISO(fH2O|β) dfH2O. (9)\n",
            "This gives us the distribution of ISOs within the bounds of the protoplanetary disk chemical model: 0 .07≤fH2O≤\n",
            "0.51. Outside of this range, we can calculate the fraction of ISOs with fH2O≤0.07 and fH2O≥0.51, by assuming\n",
            "that the relation between [Fe /H] and fH2Oremains monotonic. Thus, all stars with [Fe /H]≤ −0.4 contribute ISOs\n",
            "with fH2O≥0.51, and all stars with [Fe /H]≥0.4 contribute ISOs with fH2O≤0.07.\n",
            "4.RESULTS\n",
            "In this section we demonstrate the prediction framework of section 3 by making two different predictions. We\n",
            "demonstrate two different example values for a stellar metallicity dependence for ISO production, β, with β= 1 for\n",
            "our principal prediction and β= 0 as an alternate prediction.\n",
            "4.1. Principal Prediction: β= 1\n",
            "First, we predict the distribution of ISOs assuming that the number produced by each star is proportional to the\n",
            "star’s metal mass fraction Z, by setting β= 1 in equation 8. As described in section 3.1, in the absence of concrete\n",
            "knowledge of ISO formation mechanisms this is a reasonable value to assume, and thus we consider this our principal\n",
            "prediction.\n",
            "Table 1 lists the the fraction of ISOs within and outside either end of the Bitsch & Battistini (2020) protoplanetary\n",
            "disk chemical model fH2Orange, 0 .07≤fH2O≤0.51. We assess both the distribution of ISOs at the position of the\n",
            "Sun, at R= 8.1 kpc, z= 0.021 kpc (GRAVITY Collaboration et al. 2018; Bennett & Bovy 2019), and the distribution\n",
            "____________________\n",
            "Draft version August 14, 2023\n",
            "Typeset using L ATEX default style in AASTeX631\n",
            "The Galactic Interstellar Object Population: A Framework for Prediction and Inference\n",
            "Matthew J. Hopkins\n",
            " ,1Chris Lintott\n",
            " ,1Michele T. Bannister\n",
            " ,2J. Ted Mackereth\n",
            " ,3, 4, 5, ∗and\n",
            "John C. Forbes\n",
            "2\n",
            "1Department of Physics, University of Oxford, Denys Wilkinson Building, Keble Road, Oxford, OX1 3RH, UK\n",
            "2School of Physical and Chemical Sciences—Te Kura Mat¯ u, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand\n",
            "3Just Group plc, Enterprise House, Bancroft road, Reigate, Surrey RH2 7RP, UK\n",
            "4Canadian Institute for Theoretical Astrophysics, University of Toronto, 60 St. George Street, Toronto, ON, M5S 3H8, Canada\n",
            "5Dunlap Institute for Astronomy and Astrophysics, University of Toronto, 50 St. George Street, Toronto, ON M5S 3H4, Canada\n",
            "ABSTRACT\n",
            "The Milky Way is thought to host a huge population of interstellar objects (ISOs), numbering\n",
            "approximately 1015pc−3around the Sun, which are formed and shaped by a diverse set of processes\n",
            "ranging from planet formation to galactic dynamics. We define a novel framework: firstly to predict\n",
            "the properties of this Galactic ISO population by combining models of processes across planetary\n",
            "and galactic scales, and secondly to make inferences about the processes modelled, by comparing the\n",
            "predicted population to what is observed. We predict the spatial and compositional distribution of the\n",
            "Galaxy’s population of ISOs by modelling the Galactic stellar population with data from the APOGEE\n",
            "survey and combining this with a protoplanetary disk chemistry model. Selecting ISO water mass\n",
            "fraction as an example observable quantity, we evaluate its distribution both at the position of the Sun\n",
            "and averaged over the Galactic disk; our prediction for the Solar neighbourhood is compatible with the\n",
            "inferred water mass fraction of 2I/Borisov. We show that the well-studied Galactic stellar metallicity\n",
            "gradient has a corresponding ISO compositional gradient. We also demonstrate the inference part of\n",
            "the framework by using the current observed ISO composition distribution to constrain the parent star\n",
            "metallicity dependence of the ISO production rate. This constraint, and other inferences made with\n",
            "this framework, will improve dramatically as the Vera C. Rubin Observatory Legacy Survey of Space\n",
            "and Time (LSST) progresses and more ISOs are observed. Finally, we explore generalizations of this\n",
            "framework to other Galactic populations, such as that of exoplanets.\n",
            "Keywords: Interstellar objects (52), Small Solar System bodies(1469), Galaxy Evolution (594)\n",
            "1.INTRODUCTION\n",
            "1I/‘Oumuamua (Meech et al. 2017) and 2I/Borisov1are the first two observed samples from a highly numerous\n",
            "population: interstellar objects (ISOs). Estimated to number ∼1015pc−3around the Sun (Engelhardt et al. 2017;\n",
            "Do et al. 2018), they are implied to have a spatial distribution spanning the entire Galaxy. This population has been\n",
            "predicted to exist for decades (McGlynn & Chapman 1989), based on models of the accretion and migration of the\n",
            "giant planets, which predict that 75–85% of cometary bodies initially in the Solar System must have been scattered\n",
            "into interstellar space (Fernandez & Ip 1984; Brasser et al. 2006). Modern exoplanet surveys consistently find that\n",
            "giant planets are common across the Galaxy around stars with a range of spectral types (Fulton et al. 2021; Sabotta\n",
            "et al. 2021). This makes planetesimal scattering common across the Galaxy. A significant number of planetesimals can\n",
            "also be ejected by close stellar flybys early in a planetary system’s life (e.g. Pfalzner et al. 2021). The protoplanetary\n",
            "disks of other stars are therefore expected to be a source of ISOs (Stern 1990; Moro-Mart´ ın 2022).\n",
            "Corresponding author: Matthew Hopkins\n",
            "matthew.hopkins@physics.ox.ac.uk\n",
            "∗Banting Fellow\n",
            "1https://minorplanetcenter.net/mpec/K19/K19RA6.html and https://minorplanetcenter.net/mpec/K19/K19S72.htmlarXiv:2308.05801v1  [astro-ph.EP]  10 Aug 2023\n",
            "____________________\n",
            "2 Hopkins et al.\n",
            "Initially it was expected that interstellar objects would display cometary characteristics (e.g. Jewitt 2003). The pop-\n",
            "ulation’s dominant dynamical formation mechanisms would preferentially harvest more distant, ice-rich planetesimals\n",
            "from the disks of the source systems. More of the cometary ISO population passing through the Solar System could\n",
            "be detected than rocky ISOs, as comae make cometary ISOs brighter down to smaller diameters (Engelhardt et al.\n",
            "2017). 2I/Borisov appeared relatively similar in size and composition to Solar System comets (Jewitt & Seligman\n",
            "2022). Its distinctive features were an exceptionally high CO and NH 2content, implying it formed on the edge of its\n",
            "home system’s protoplanetary disk, beyond the hypervolatile CO ice line (Bodewits et al. 2020; Cordiner et al. 2020).\n",
            "However, 1I/‘Oumuamua had a mix of observed characteristics. A 160-m scale object, it lacked a coma in deep\n",
            "imaging (Jewitt et al. 2017) or detectable outgassing in CO, CO 2(Trilling et al. 2018) or CN (Ye et al. 2017). Despite\n",
            "this, it still underwent non-gravitational acceleration similar to that experienced by Solar System comets (Micheli\n",
            "et al. 2018). The large amplitude of its light curve implied a high-aspect-ratio shape (Mashchenko 2019). At the time,\n",
            "this combination of factors seemed unusual, although the surface reflectance properties were consistent with outer\n",
            "Solar System bodies (e.g. Bannister et al. 2017). Hypotheses for the composition and formation of 1I/‘Oumuamua\n",
            "that match the limited data remain varied. It may be a planetesimal (’Oumuamua ISSI Team et al. 2019); a fragment\n",
            "of a comet devolatized by passages close to its parent star before ejection (Raymond et al. 2018a); an icy fractal\n",
            "aggregate (Moro-Mart´ ın 2019); a hydrogen iceberg formed in a molecular cloud (Seligman & Laughlin 2020); or a\n",
            "nitrogen ice fragment from the surface of a Pluto-like dwarf planet (Jackson & Desch 2021). Recently, observation\n",
            "of similarly-small near-Earth asteroids have identified objects with the lightcurve amplitudes seen in 1I. Additionally,\n",
            "Farnocchia et al. (2023) and Seligman et al. (2023) report six asteroids with significant non-gravitational acceleration\n",
            "and no coma. While from the two ISOs found so far, the compositions of interstellar objects are clearly varied, 1I may\n",
            "be less extreme than it was first considered.\n",
            "ISOs formed in a protoplanetary disk carry information about their home systems in their composition: we have\n",
            "samples of other planetary systems coming to us for study. The composition of a protoplanetary disk correlates with the\n",
            "elemental abundances of its central star, due to their formation from the same gas and dust in a molecular cloud core\n",
            "(¨Oberg & Bergin 2021). We can thus expect stars of different metallicities to produce ISOs of different compositions.\n",
            "The composition of this gas and dust varies in both space and time as the Galaxy chemically evolves due to stellar\n",
            "nucleosynthesis, making the current Galactic ISO population sensitive to the entire history and evolution of the Milky\n",
            "Way over cosmic time (Tinsley & Cameron 1974; Lintott et al. 2022). Since the occurrence of planetesimal-scattering\n",
            "giant planets also has a metallicity dependence (Fischer & Valenti 2005), the relative occurrence of ISOs of different\n",
            "compositions will therefore carry information about the Galaxy’s distribution of planetary architectures. Finally, we\n",
            "expect many ISOs to outlive their parent stars, as they occupy a similar environment to Gyr-old Oort cloud comets.\n",
            "At minor-planet sizes, ISOs are not subject to any known destructive forces in the ISM (Guilbert-Lepoutre et al. 2015),\n",
            "other than minor erosion by dust, in their frequent passages through molecular clouds (Pfalzner et al. 2020). They\n",
            "could also be disrupted or devolatilised in rare close encounters with stars (Raymond et al. 2018a; Forbes & Loeb\n",
            "2019). ISOs thus present the possibility of studying long-lost planetary systems.\n",
            "The Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) (Ivezi´ c et al. 2019) is predicted to provide\n",
            "a sample of tens of 1I/‘Oumuamua-like ISOs (e.g. Levine et al. 2021), as well as any more 2I/Borisov-like cometary\n",
            "ISOs that enter our Solar System. This is in addition to the continuing contributions of the NEO surveys and other\n",
            "observatories that found 1I and 2I in the first place (Meech et al. 2017). These will provide a large and varied sample\n",
            "of interstellar objects, for study and comparison to predictions.\n",
            "The many dependencies over planetary and Galactic scales make the observed ISO population a fascinating tool:\n",
            "it has the potential to test models in both Galaxy and planetary physics, with an entirely different set of biases\n",
            "to traditional methods. Lintott et al. (2022) introduced the concept of predicting the composition of a Galaxy’s\n",
            "population of ISOs from the Galaxy’s stellar distribution. Using simulated Galaxies from EAGLE (Schaye et al. 2015),\n",
            "they showed that the water content of ISOs was sensitive to Galactic star formation history. In this work, we develop\n",
            "this method and apply it to the stellar population of the Milky Way, estimated with data from the APOGEE survey, to\n",
            "predict a broader set of properties of our own Galaxy’s population of interstellar objects. We predict the distribution\n",
            "of ISOs in both their spatial position in the Galaxy and their water mass fraction. By evaluating this distribution at\n",
            "the current position of the Solar System, we predict the properties of the population of ISOs from which the observed\n",
            "sample of the 2020s will be drawn, and we compare this to the whole-Galaxy distribution. We then detail a Bayesian\n",
            "method of comparing the predicted and observed distributions to make inferences about the planetary and Galactic\n",
            "____________________\n"
          ]
        }
      ],
      "source": [
        "retrieved_chunks = query_engine.retrieve(search_term2)\n",
        "for i in retrieved_chunks:\n",
        "    print(i.node.get_text())\n",
        "    print(\"____________________\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "25ce337f",
      "metadata": {
        "id": "25ce337f"
      },
      "source": [
        "Again, we can see these sentences do reference our search term 'how does planet formation occur' in some way."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 74,
      "id": "2d3ce969",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2d3ce969",
        "outputId": "c7e3988b-7e6e-4abd-e6d9-8b0d0805df1c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Planet formation occurs through processes in protoplanetary disks, where dust and gas surrounding a young star coalesce to form solid bodies. Initially, small particles collide and stick together, forming larger aggregates known as planetesimals. These planetesimals can further collide and merge, eventually leading to the formation of protoplanets. The dynamics of this process can be influenced by factors such as the metallicity of the star, which affects the availability of solid material, and the gravitational interactions with giant planets, which can scatter and eject planetesimals into interstellar space. The composition of the resulting bodies is linked to the chemical properties of the protoplanetary disk, which varies with the star's metallicity and the conditions present during formation.\n"
          ]
        }
      ],
      "source": [
        "result = query_engine.query(search_term2)\n",
        "print(result.response)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f6e93878",
      "metadata": {
        "id": "f6e93878"
      },
      "source": [
        "## 5. Delete the KDB.AI Table\n",
        "\n",
        "Once finished with the table, it is best practice to drop it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 75,
      "id": "d74b6f20",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d74b6f20",
        "outputId": "2f4b3592-0290-48b4-c280-7bd3f8e4d46a"
      },
      "outputs": [],
      "source": [
        "table.drop()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "346d56db",
      "metadata": {
        "id": "346d56db"
      },
      "source": [
        "## Take Our Survey\n",
        "\n",
        "We hope you found this sample helpful! Your feedback is important to us, and we would appreciate it if you could take a moment to fill out our brief survey. Your input helps us improve our content.\n",
        "\n",
        "[**Take the Survey**](https://delighted.com/t/vm88uwcB)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}