Error : Adding pdf documents to collections & multi-modal parser

[virajone]

I am successfully able to add pdf documents to data sources, however, I get an error when adding these documents to collections from the front end. I suppose it might be an issue in process of creating embeddings.

Terminal :
cognita-backend | DEBUG: 2024-07-29 10:10:40,351 - indexer:ingest_data:343 - Starting ingestion for data source fqn: localdir::/app/user_data/resume
cognita-backend | DEBUG: 2024-07-29 10:10:40,421 - qdrant:list_data_point_vectors:194 - [Qdrant] Listing all data point vectors for collection cv
qdrant | 2024-07-29T10:10:40.426193Z INFO actix_web::middleware::logger: 172.18.0.5 "POST /collections/cv/points/scroll HTTP/1.1" 200 96 "-" "python-httpx/0.27.0" 0.002971
cognita-backend | DEBUG: 2024-07-29 10:10:40,427 - qdrant:list_data_point_vectors:242 - [Qdrant] Listing 0 data point vectors for collection cv
cognita-backend | INFO: 2024-07-29 10:10:40,427 - indexer:sync_data_source_to_collection:80 - Total existing data point vectors in collection cv: 0
cognita-backend | INFO: 2024-07-29 10:10:40,457 - indexer:_sync_data_source_to_collection:159 - Loading data from data source
cognita-backend | INFO: 2024-07-29 10:10:40,459 - localdirloader:load_filtered_data:34 - CURRENT DIR:/app, Path exists: True, Dir contents: ['VirajPatil - Resume - postDel.pdf']
cognita-backend | INFO: 2024-07-29 10:10:40,459 - localdirloader:load_filtered_data:42 - source_dir: /app/user_data/resume
cognita-backend | INFO: 2024-07-29 10:10:40,459 - localdirloader:load_filtered_data:43 - dest_dir: /tmp/tmp0g34m0pr
cognita-backend | INFO: 2024-07-29 10:10:40,463 - localdirloader:load_filtered_data:51 - Dest dir contents: ['VirajPatil - Resume - postDel.pdf']
cognita-backend | INFO: 2024-07-29 10:10:40,463 - localdirloader:load_filtered_data:61 - full_path: /tmp/tmp0g34m0pr/VirajPatil - Resume - postDel.pdf, rel_path: VirajPatil - Resume - postDel.pdf, file_ext: .pdf
cognita-backend | INFO: 2024-07-29 10:10:40,521 - indexer:ingest_data_points:225 - Processing 1 new documents and completed: 0
cognita-backend | INFO: 2024-07-29 10:10:40,521 - indexer:ingest_data_points:229 - [1/1/0] Parsing [1/1] new document
cognita-backend | Parser map found in the collection for extension .pdf. Hence, using parser MultiModalParser
cognita-backend | DEBUG: 2024-07-29 10:10:40,521 - parser:get_parser_for_extension:87 - Parser map found in the collection for extension .pdf. Hence, using parser MultiModalParser
cognita-backend | INFO: 2024-07-29 10:10:40,521 - multimodalparser:init:78 - Using custom vision model..., name='truefoundry/openai-main/gpt-4-turbo' type=None parameters=None
cognita-backend | INFO: 2024-07-29 10:10:40,521 - multimodalparser:init:87 - Using custom prompt..., Given an image containing one or more charts/graphs, and texts, provide a detailed analysis of the data represented in the charts. Your task is to analyze the image and provide insights based on the data it represents. Specifically, the information should include but not limited to: - Title of the Image: Provide a title from the charts or image if any. - Type of Chart: Determine the type of each chart (e.g., bar chart, line chart, pie chart, scatter plot, etc.) and its key features (e.g., labels, legends, data points). - Data Trends: Describe any notable trends or patterns visible in the data. This may include increasing/decreasing trends, seasonality, outliers, etc. - Key Insights: Extract key insights or observations from the charts. What do the charts reveal about the underlying data? Are there any significant findings that stand out? - Data Points: Identify specific data points or values represented in the charts, especially those that contribute to the overall analysis or insights. - Comparisons: Compare different charts within the same image or compare data points within a single chart. Highlight similarities, differences, or correlations between datasets. - Conclude with a summary of the key findings from your analysis and any recommendations based on those findings.
cognita-backend | INFO: 2024-07-29 10:10:40,537 - multimodalparser:get_chunks:166 -
cognita-backend |
cognita-backend | Loading all pages...
cognita-backend | INFO: 2024-07-29 10:10:40,626 - multimodalparser:get_chunks:185 - Total Pages: 2
cognita-backend | ERROR: 2024-07-29 10:10:40,627 - multimodalparser:get_chunks:249 - Final Exception: Model truefoundry/openai-main/gpt-4-turbo not registered in the model gateway.
cognita-backend | Traceback (most recent call last):
cognita-backend | File "/app/backend/modules/parsers/multimodalparser.py", line 206, in get_chunks
cognita-backend | llm = model_gateway.get_llm_from_model_config(self.model_configuration)
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/model_gateway/model_gateway.py", line 120, in get_llm_from_model_config
cognita-backend | raise ValueError(
cognita-backend | ValueError: Model truefoundry/openai-main/gpt-4-turbo not registered in the model gateway.
cognita-backend | ERROR: 2024-07-29 10:10:40,645 - indexer:_sync_data_source_to_collection:180 - Model truefoundry/openai-main/gpt-4-turbo not registered in the model gateway.
cognita-backend | Traceback (most recent call last):
cognita-backend | File "/app/backend/indexer/indexer.py", line 171, in _sync_data_source_to_collection
cognita-backend | await ingest_data_points(
cognita-backend | File "/app/backend/indexer/indexer.py", line 245, in ingest_data_points
cognita-backend | chunks = await parser.get_chunks(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/parsers/multimodalparser.py", line 250, in get_chunks
cognita-backend | raise e
cognita-backend | File "/app/backend/modules/parsers/multimodalparser.py", line 206, in get_chunks
cognita-backend | llm = model_gateway.get_llm_from_model_config(self.model_configuration)
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/model_gateway/model_gateway.py", line 120, in get_llm_from_model_config
cognita-backend | raise ValueError(
cognita-backend | ValueError: Model truefoundry/openai-main/gpt-4-turbo not registered in the model gateway.
cognita-backend | ERROR: 2024-07-29 10:10:40,652 - indexer:sync_data_source_to_collection:101 - Model truefoundry/openai-main/gpt-4-turbo not registered in the model gateway.
cognita-backend | Traceback (most recent call last):
cognita-backend | File "/app/backend/indexer/indexer.py", line 96, in sync_data_source_to_collection
cognita-backend | await _sync_data_source_to_collection(
cognita-backend | File "/app/backend/indexer/indexer.py", line 182, in _sync_data_source_to_collection
cognita-backend | raise e
cognita-backend | File "/app/backend/indexer/indexer.py", line 171, in _sync_data_source_to_collection
cognita-backend | await ingest_data_points(
cognita-backend | File "/app/backend/indexer/indexer.py", line 245, in ingest_data_points
cognita-backend | chunks = await parser.get_chunks(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/parsers/multimodalparser.py", line 250, in get_chunks
cognita-backend | raise e
cognita-backend | File "/app/backend/modules/parsers/multimodalparser.py", line 206, in get_chunks
cognita-backend | llm = model_gateway.get_llm_from_model_config(self.model_configuration)
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/model_gateway/model_gateway.py", line 120, in get_llm_from_model_config
cognita-backend | raise ValueError(
cognita-backend | ValueError: Model truefoundry/openai-main/gpt-4-turbo not registered in the model gateway.
cognita-backend | ERROR: 2024-07-29 10:10:40,656 - indexer:ingest_data:412 - Model truefoundry/openai-main/gpt-4-turbo not registered in the model gateway.
cognita-backend | Traceback (most recent call last):
cognita-backend | File "/app/backend/indexer/indexer.py", line 358, in ingest_data
cognita-backend | await sync_data_source_to_collection(
cognita-backend | File "/app/backend/indexer/indexer.py", line 106, in sync_data_source_to_collection
cognita-backend | raise e
cognita-backend | File "/app/backend/indexer/indexer.py", line 96, in sync_data_source_to_collection
cognita-backend | await _sync_data_source_to_collection(
cognita-backend | File "/app/backend/indexer/indexer.py", line 182, in _sync_data_source_to_collection
cognita-backend | raise e
cognita-backend | File "/app/backend/indexer/indexer.py", line 171, in _sync_data_source_to_collection
cognita-backend | await ingest_data_points(
cognita-backend | File "/app/backend/indexer/indexer.py", line 245, in ingest_data_points
cognita-backend | chunks = await parser.get_chunks(
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/parsers/multimodalparser.py", line 250, in get_chunks
cognita-backend | raise e
cognita-backend | File "/app/backend/modules/parsers/multimodalparser.py", line 206, in get_chunks
cognita-backend | llm = model_gateway.get_llm_from_model_config(self.model_configuration)
cognita-backend | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cognita-backend | File "/app/backend/modules/model_gateway/model_gateway.py", line 120, in get_llm_from_model_config
cognita-backend | raise ValueError(
cognita-backend | ValueError: Model truefoundry/openai-main/gpt-4-turbo not registered in the model gateway.
cognita-backend | INFO: 192.168.65.1:21865 - "POST /v1/collections/ingest HTTP/1.1" 500 Internal Server Error

I see that the truefoundy section in the models_config.yaml file have been commented (I do not have a TFY-key). Is it necessary to have a TrueFoundry subscription to be able to use the multimodal parser as is ? How can I configure the multi-modal parser to use a different multimodal model eg. Claude/ AWSBedrock ?

[S1LV3RJ1NX]

Are you using any opensource vision model for the same? Multimodal parser requires a Vison based LLM. Can I see your collection creation config?