Load Pandas dataframe into VectorStore Index - For OpenAI Agent Query Planning

[mraguth]

https://docs.llamaindex.ai/en/stable/examples/agent/openai_agent_query_plan/

I like to use the example provided in the link. Instead of pdfs, I like to pass 3 pandas dataframes run the same process. Is that possible?

Below I like to simulate to pandas dataframe instead of .pdf. Please help.

march_2022 = SimpleDirectoryReader(
    input_files=["./data/10q/uber_10q_march_2022.pdf"]
).load_data()
june_2022 = SimpleDirectoryReader(
    input_files=["./data/10q/uber_10q_june_2022.pdf"]
).load_data()
sept_2022 = SimpleDirectoryReader(
    input_files=["./data/10q/uber_10q_sept_2022.pdf"]
).load_data()

march_index = VectorStoreIndex.from_documents(march_2022)
june_index = VectorStoreIndex.from_documents(june_2022)
sept_index = VectorStoreIndex.from_documents(sept_2022)

[dosubot[bot]]

To modify the provided code snippet to load Pandas DataFrames into a VectorStoreIndex for OpenAI Agent Query Planning instead of using PDF documents, you can adjust the load_data method to directly create Document objects from the DataFrame. Here is the modified code:

import pandas as pd
from llama_index.core.schema import Document
from llama_index.vector_stores.types import VectorStoreQueryResult
from llama_index.vector_stores.vearch.base import VearchVectorStore
from llama_index.legacy.indices.vector_store.base import VectorStoreIndex

# Sample DataFrames
march_2022_df = pd.DataFrame({
    'text': ["March 2022 data point 1", "March 2022 data point 2"],
    'metadata': ["meta1", "meta2"]
})
june_2022_df = pd.DataFrame({
    'text': ["June 2022 data point 1", "June 2022 data point 2"],
    'metadata': ["meta1", "meta2"]
})
sept_2022_df = pd.DataFrame({
    'text': ["September 2022 data point 1", "September 2022 data point 2"],
    'metadata': ["meta1", "meta2"]
})

# Function to convert DataFrame to Document list
def df_to_documents(df):
    return [Document(text=row['text'], metadata=row.to_dict()) for _, row in df.iterrows()]

# Convert DataFrames to Document lists
march_2022_docs = df_to_documents(march_2022_df)
june_2022_docs = df_to_documents(june_2022_df)
sept_2022_docs = df_to_documents(sept_2022_df)

# Create VectorStoreIndex from Document lists
march_index = VectorStoreIndex.from_documents(march_2022_docs)
june_index = VectorStoreIndex.from_documents(june_2022_docs)
sept_index = VectorStoreIndex.from_documents(sept_2022_docs)

This code snippet demonstrates how to convert each DataFrame into a list of Document objects and then create a VectorStoreIndex from these documents. Each row in the DataFrame is converted into a Document object, with the text and metadata extracted from the DataFrame columns ^[1].

To continue talking to Dosu, mention @dosu.

[mraguth]

Getting error:

ModuleNotFoundError: No module named 'llama_index.vector_stores.types'
No module named 'llama_index.vector_stores.vearch'

[dosubot[bot]]

To resolve the ModuleNotFoundError for llama_index.vector_stores.types and llama_index.vector_stores.vearch, you need to ensure that you are using the correct import paths. The structure of the LlamaIndex library may have changed. Here are the updated import statements:

1. For VectorStoreQueryResult:

   from llama_index.core.vector_stores.types import VectorStoreQueryResult

2. For VearchVectorStore:

   from llama_index.vector_stores.vearch.base import VearchVectorStore

Make sure you have the necessary dependencies installed. You can install the llama-index-vector-stores-vearch package if it's not already installed:

pip install llama-index-vector-stores-vearch

Here is the updated code snippet with the correct import paths:

import pandas as pd
from llama_index.core.schema import Document
from llama_index.core.vector_stores.types import VectorStoreQueryResult
from llama_index.vector_stores.vearch.base import VearchVectorStore
from llama_index.legacy.indices.vector_store.base import VectorStoreIndex

# Sample DataFrames
march_2022_df = pd.DataFrame({
    'text': ["March 2022 data point 1", "March 2022 data point 2"],
    'metadata': ["meta1", "meta2"]
})
june_2022_df = pd.DataFrame({
    'text': ["June 2022 data point 1", "June 2022 data point 2"],
    'metadata': ["meta1", "meta2"]
})
sept_2022_df = pd.DataFrame({
    'text': ["September 2022 data point 1", "September 2022 data point 2"],
    'metadata': ["meta1", "meta2"]
})

# Function to convert DataFrame to Document list
def df_to_documents(df):
    return [Document(text=row['text'], metadata=row.to_dict()) for _, row in df.iterrows()]

# Convert DataFrames to Document lists
march_2022_docs = df_to_documents(march_2022_df)
june_2022_docs = df_to_documents(june_2022_df)
sept_2022_docs = df_to_documents(sept_2022_df)

# Create VectorStoreIndex from Document lists
march_index = VectorStoreIndex.from_documents(march_2022_docs)
june_index = VectorStoreIndex.from_documents(june_2022_docs)
sept_index = VectorStoreIndex.from_documents(sept_2022_docs)

This should resolve the ModuleNotFoundError issues ^[1][2].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu... How I deal this function if my pandas data frame has more than 3 columns?

Example of my dataframe:

march_2022_df = pd.DataFrame({
    'text': ["March 2022 data point 1", "March 2022 data point 2"],
    'metadata': ["meta1", "meta2"],
    'sampledata':["sample1","sample2"]
})

The function I like to rewrite Function to convert DataFrame to Document list

def df_to_documents(df):
    return [Document(text=row['text'], metadata=row.to_dict()) for _, row in df.iterrows()]

[dosubot[bot]]

To handle a DataFrame with more than 3 columns, you can modify the df_to_documents function to ensure that the text column is used for the document text and the remaining columns are included in the metadata. Here is the updated function:

import pandas as pd
from llama_index.core.schema import Document

# Sample DataFrame with more than 3 columns
march_2022_df = pd.DataFrame({
    'text': ["March 2022 data point 1", "March 2022 data point 2"],
    'metadata': ["meta1", "meta2"],
    'sampledata': ["sample1", "sample2"]
})

# Function to convert DataFrame to Document list
def df_to_documents(df):
    return [Document(text=row['text'], metadata={col: row[col] for col in df.columns if col != 'text'}) for _, row in df.iterrows()]

# Convert DataFrame to Document list
march_2022_docs = df_to_documents(march_2022_df)

# Print the documents to verify
for doc in march_2022_docs:
    print(doc)

In this function, the text column is used for the document text, and all other columns are included in the metadata dictionary. This approach ensures that the function can handle DataFrames with any number of columns.

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu Is the text column in the example similar to an 'index' of the dataframe? I'm still new to this and want to understand the basics. Thank you for your patience!

Also, once I vectorize my pandas dataframe, can I save those vectors in a Chroma vector database? Sorry for all the questions... I'm trying to figure out how we can best use the 'llamaindex' package for structured data insights.

Thanks again!

[dosubot[bot]]

The text column in the example is not the same as the DataFrame's index. Instead, it is a column that contains the text data you want to convert into Document objects. The DataFrame's index is a separate entity that uniquely identifies each row, while the text column holds the actual content of the documents.

To save the vectorized DataFrame in a Chroma vector database, you can follow these steps:

1. Convert the DataFrame to a list of Document objects:

   import pandas as pd
   from llama_index.core.schema import Document
   
   # Sample DataFrame
   df = pd.DataFrame({
       'text': ["Document 1 text", "Document 2 text", "Document 3 text"],
       'metadata': [{'id': 1}, {'id': 2}, {'id': 3}]
   })
   
   # Function to convert DataFrame to Document list
   def df_to_documents(df):
       return [Document(text=row['text'], metadata={col: row[col] for col in df.columns if col != 'text'}) for _, row in df.iterrows()]
   
   # Convert DataFrame to Document list
   documents = df_to_documents(df)

2. Save the vectorized DataFrame in a Chroma vector database:

   from llama_index.core import VectorStoreIndex, StorageContext
   from llama_index.vector_stores.chroma import ChromaVectorStore
   from llama_index.embeddings.huggingface import HuggingFaceEmbedding
   import chromadb
   
   # Create a Chroma client and collection
   chroma_client = chromadb.PersistentClient(path="./chroma_db")
   chroma_collection = chroma_client.get_or_create_collection("your_collection_name")
   
   # Define embedding function
   embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")
   
   # Set up ChromaVectorStore and load in data
   vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
   storage_context = StorageContext.from_defaults(vector_store=vector_store)
   vector_store_index = VectorStoreIndex.from_documents(
       documents, storage_context=storage_context, embed_model=embed_model
   )
   
   # Save to disk
   vector_store_index.storage_context.persist(persist_dir="./chroma_db")
   
   # Load from disk
   chroma_client = chromadb.PersistentClient(path="./chroma_db")
   chroma_collection = chroma_client.get_or_create_collection("your_collection_name")
   vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
   vector_store_index = VectorStoreIndex.from_vector_store(
       vector_store, embed_model=embed_model
   )
   
   # Query Data
   query_engine = vector_store_index.as_query_engine()
   response = query_engine.query("Your query here")
   print(response)

This code demonstrates how to convert a DataFrame to a list of Document objects and save the vectorized documents in a Chroma vector database using the LlamaIndex library. The example includes setting up the Chroma client, defining the embedding model, and querying the data ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.