Extended usage of Vector Store

[TimPietrusky]

Objective: Enhance the Vector Store within Captain to handle a diverse range of data types and enable efficient search and categorization.

Core Development Tasks for Vector Store Enhancement:

• Implement language-specific data handling within the Vector Store for improved localization. #110
• Develop filters and categorization functionality for different data types stored in the Vector Store. #111
• Enable storage and retrieval of generated content, such as images, within the Vector Store. #112
• Create a system for storing and indexing third-party documents like PDFs in the Vector Store. #308
• Integrate Qdrant's binary quantization to optimize vector search efficiency in the Vector Store. #310
• Persist Vector Store data after closing the app #309
• Write installed apps / models and other data from the Marketplace into the Vector Store #311
• All generated content is saved in the Vector Store #312
• Make document splitting a default when saving data into the Vector Store (because embeddings are limitted to 128 tokens) #313
• Lookup of documents will deliver the most relevant document, not every chunk of the same document #314