showing duplicates entry in vector db

[iam-amiya]

I have created nodes using ingestion pipeline:

import time
import nest_asyncio
nest_asyncio.apply()
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.extractors import SummaryExtractor

pipeline = IngestionPipeline(
    transformations=[
        TextCleaner(),
        SummaryExtractor(prompt_template=DEFAULT_SUMMARY_PROMPT_TMPL),
        VoyageEmbeddModel,
    ],
)

CvFirstChunk = pipeline.run(documents = first_node)
time.sleep(90)
CvSecondChunk = pipeline.run(documents = second_node)
time.sleep(90)
CvThirdChunk = pipeline.run(documents = third_node)
time.sleep(90)
CvFourthChunk = pipeline.run(documents = fourth_node)
time.sleep(90)
CvFifthChunk = pipeline.run(documents = fifth_node)
time.sleep(90)
CvSixthChunk = pipeline.run(documents = sixth_node)

and I have total 113 nodes after I concatenate all the chunks.
however When I am upserting those nodes in my pinecone vector db it is showing vector count 572

this is the code i am using to upserting the nodes to my vec db.

vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    nodes,
    storage_context=storage_context,
    embed_model= EmbeddModel,
)

can anyone please help me out with it.

[dosubot[bot]]

The issue of duplicate entries in your Pinecone vector database might be due to how the nodes are being processed and upserted. Here are a few points to consider:

1. Ingestion Pipeline: Ensure that the IngestionPipeline.run method is not generating duplicate nodes. The SummaryExtractor transformation, in particular, can generate summaries for the current, previous, and next nodes, which might lead to duplicates if not managed correctly ^[1][2].

2. Upsert Method: When upserting nodes into Pinecone, make sure you are using the upsert method correctly to replace existing documents instead of adding duplicates. Pinecone's upsert operation should replace the existing document if the same document is inserted twice ^[3][4].

3. Storage Context Configuration: Check the configuration of your StorageContext.from_defaults method. Ensure that there are no overlapping keys or misconfigurations that could lead to duplicate entries ^[5][6].

Here is an example of how you might use the upsert method properly:

vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Upsert documents to avoid duplicates
index = VectorStoreIndex.from_documents(
    nodes,
    storage_context=storage_context,
    embed_model=EmbeddModel,
)

Make sure that your code is designed to upsert documents rather than inserting them multiple times. This should help resolve the issue of duplicate entries in your Pinecone vector database. If you continue to experience issues, consider checking for any version-specific bugs or updates ^[3][4].

To continue talking to Dosu, mention @dosu.