Is that possible to import FAISS Vectors in to RecursiveRetriever for further process?

[mraguth]

I have FAISS vector database and I am using those vector databases for Query engine. However, I like to use RecursiveRetriever in 'RAGStringQueryEngine'. While I am creating RecursiveRetriever, program is asking 'node_dict' and I am not sure how I get from 'FAISS Vector database. Please help.

The way I import my FAISS Vector DB is like this below:
index = load_index_from_storage(storage_context=storage_context)

The RecursiveRetriever is looking below code, where I have no 'node_dict' from my vector import.

retriever_metadata = RecursiveRetriever(
    "vector",
    retriever_dict={"vector": vector_retriever_metadata},
    node_dict=all_nodes_dict,
    verbose=False,
)

The reference notebook I am following to build RecursiveRetriever is below:
https://docs.llamaindex.ai/en/stable/examples/retrievers/recursive_retriever_nodes/

[dosubot[bot]]

To import FAISS vectors into RecursiveRetriever, you need to extract the node_dict from your FAISS vector database. Here's how you can do it:

1. Load your FAISS index:

   index = load_index_from_storage(storage_context=storage_context)

2. Extract the node_dict from the loaded index:

   import json
   
   # Assuming you have a JSON representation of your FAISS index
   with open('index_faiss.json', 'r') as file:
       faiss_data = json.load(file)
   
   # Extract the node_dict
   node_dict = faiss_data['index_struct']['nodes_dict']

3. Convert the node_dict to the required format:

   from llama_index.core.storage.docstore.types import BaseDocumentStore
   
   # Assuming you have a document store instance
   docstore = BaseDocumentStore()
   
   # Convert node_dict to the required format
   node_dict_converted = docstore.get_node_dict(node_dict)

4. Create the RecursiveRetriever with the node_dict:

   from llama_index.core.retrievers import RecursiveRetriever
   
   retriever_metadata = RecursiveRetriever(
       "vector",
       retriever_dict={"vector": vector_retriever_metadata},  # Replace with your actual vector retriever
       node_dict=node_dict_converted,
       verbose=False,
   )

This process ensures that you extract the node_dict from your FAISS vector database and use it in the RecursiveRetriever ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu I am getting different error now:

{
	"name": "KeyError",
	"message": "'nodes_dict'",
	"stack": "---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[4], line 17
     13     index_store_data = json.load(file)
     15 # Extract the node_dict
     16 # Extract the node_dict
---> 17 node_dict = index_store_data['nodes_dict']

KeyError: 'nodes_dict'"
}

[dosubot[bot]]

The KeyError indicates that the key 'nodes_dict' does not exist in the index_store.json file. To correctly extract the node_dict, you need to inspect the structure of the index_store.json file and find the correct path to the nodes_dict.

Here’s a more robust approach to help you identify the correct structure:

1. Load your FAISS index:

   index = load_index_from_storage(storage_context=storage_context)

2. Inspect the structure of the index_store.json file:

   import json
   
   # Load the index structure from the index_store.json file
   with open('index_store.json', 'r') as file:
       index_store_data = json.load(file)
   
   # Print the keys to understand the structure
   print(index_store_data.keys())

3. Extract the node_dict based on the correct structure:
   After inspecting the keys, you should be able to identify the correct path to the nodes_dict. For example, if the structure is nested, you might need to access it like this:

   # Example structure, adjust based on the actual keys
   node_dict = index_store_data['some_key']['index_struct']['nodes_dict']

4. Create the RecursiveRetriever with the node_dict:

   from llama_index.core.retrievers import RecursiveRetriever
   
   retriever_metadata = RecursiveRetriever(
       "vector",
       retriever_dict={"vector": vector_retriever_metadata},  # Replace with your actual vector retriever
       node_dict=node_dict,
       verbose=False,
   )

By printing the keys of the index_store_data, you can determine the correct path to the nodes_dict and adjust the extraction accordingly. This should help you avoid the KeyError and correctly set up the RecursiveRetriever.

To continue talking to Dosu, mention @dosu.

[mraguth]

@my dict_keys print out : dict_keys(['index_store/data'])
Please give me updated code.

[dosubot[bot]]

Given the structure of your index_store.json file, you can extract the node_dict as follows:

1. Load your FAISS index:

   index = load_index_from_storage(storage_context=storage_context)

2. Extract the node_dict from the index_store.json file:

   import json
   
   # Load the index structure from the index_store.json file
   with open('index_store.json', 'r') as file:
       index_store_data = json.load(file)
   
   # Extract the node_dict
   node_dict = index_store_data['index_store']['data']['nodes_dict']

3. Create the RecursiveRetriever with the node_dict:

   from llama_index.core.retrievers import RecursiveRetriever
   
   retriever_metadata = RecursiveRetriever(
       "vector",
       retriever_dict={"vector": vector_retriever_metadata},  # Replace with your actual vector retriever
       node_dict=node_dict,
       verbose=False,
   )

This should correctly extract the node_dict and allow you to use it in the RecursiveRetriever ^[1].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu Thank you... it did work. I used node_dict = index_store_data['index_store/data'] as per print(index_store_data.keys())

[dosubot[bot]]

Great to hear that it worked! If you have any more questions or need further assistance, feel free to ask. Happy coding!

To continue talking to Dosu, mention @dosu.