Help to run the starter example with all-MiniLM-L6-v2 embedding model

[lewoudar]

Hello everyone,
I'm starting my journey with llama_index and I am trying to replicate the local model starter example replacing the BAAI/bge-base-en-v1.5 embedding model with the sentence-transformers/all-MiniLM-L6-v2 embedding model.
I also changed the vector store to use chromadb.

import os
import time
import chromadb
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings, StorageContext
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.litellm import LiteLLM
from llama_index.vector_stores.chroma import ChromaVectorStore

documents = SimpleDirectoryReader(input_files=['C:\\Users\\rolla\\Downloads\\paul_graham_essay.txt']).load_data()

# bge-base embedding model
# Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")
Settings.embed_model = HuggingFaceEmbedding(model_name='sentence-transformers/all-MiniLM-L6-v2')

# litellm
os.environ['GROQ_API_KEY'] = 'XXX'
Settings.llm = LiteLLM(model="groq/llama3-70b-8192")

print('creating vector store')
chroma_client = chromadb.PersistentClient()
chroma_collection = chroma_client.get_or_create_collection("quickstart")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

print('creating index')
start_time = time.perf_counter()
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)
print(f'indexing time: {time.perf_counter() - start_time:.2f}s')
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)
print(f'total time: {time.perf_counter() - start_time:.2f}s')

I don't know why, but the answer is not the same as expected in the example. This is what I got

The author didn't mention what they did growing up. The text jumps straight into the author's experiences as an adult, discussing their patterns, painting, and ideas about web apps. There is no mention of their childhood or growing-up years.

And when I change to the BAAI/bge-base-en-v1.5 embedding model, I get the expected output

The author worked on two main things outside of school: writing and programming. He wrote short stories, which he admits were awful, and tried writing programs on an IBM 1401 computer.

If anyone knows what's missing in my example I'd be grateful :)

[dosubot[bot]]

Hey there, @lewoudar! I'm here to help you with any bugs, questions, or contributions you have in mind. Let's figure this out together!

The issue might be that the sentence-transformers/all-MiniLM-L6-v2 embedding model is not being correctly utilized, and the default OpenAI embedding is being used instead. This can happen if the embedding model is not properly passed or integrated into the llama_index and chromadb setup.

Here is a corrected example to ensure the custom embedding model is used:

import os
import time
import chromadb
from sentence_transformers import SentenceTransformer
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings, StorageContext
from llama_index.embeddings import LangchainEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore

# Define the custom embedding model
def get_embed_model():
    hf = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    return LangchainEmbedding(hf)

embed_model = get_embed_model()

# Load documents
documents = SimpleDirectoryReader(input_files=['C:\\Users\\rolla\\Downloads\\paul_graham_essay.txt']).load_data()

# Set the embedding model
Settings.embed_model = embed_model

# Set the LLM
os.environ['GROQ_API_KEY'] = 'XXX'
Settings.llm = LiteLLM(model="groq/llama3-70b-8192")

print('creating vector store')
chroma_client = chromadb.PersistentClient()
chroma_collection = chroma_client.get_or_create_collection("quickstart")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

print('creating index')
start_time = time.perf_counter()
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)
print(f'indexing time: {time.perf_counter() - start_time:.2f}s')
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?", embed_model=embed_model)
print(response)
print(f'total time: {time.perf_counter() - start_time:.2f}s')

Ensure that the embed_model is correctly passed to both the index creation and the query methods. Additionally, make sure that the chromadb client and collection are properly set up and integrated. If you encounter an AttributeError related to the LocalAPI object not having an add attribute, it might indicate an issue with the chromadb setup. Double-check the chromadb client and collection initialization to ensure they are correctly configured ^[1][2].

Moreover, if you are using custom loss functions or training procedures, ensure they are compatible with your embedding model. For instance, the MyMultipleNegativesRankingLoss class in the llama_index framework is designed to work with custom embeddings and might need to be adapted to your specific use case ^[3].

To continue talking to Dosu, mention @dosu.

[lewoudar]

This answer is not correct, is it possible to have a human response?