issue-link js-issue-link" data-error-text="Failed to load title" data-id="2237844914" data-permission-text="Title is private" data-url="https://github.com/truefoundry/cognita/issues/132" data-hovercard-type="pull_request" data-hovercard-url="/truefoundry/cognita/pull/132/hovercard" href="https://github.com/truefoundry/cognita/pull/132">#132) * Added support for cloning qdrant db * Update backend/migration/qdrant_migration.py Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Update backend/migration/qdrant_migration.py Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Update backend/migration/qdrant_migration.py Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Addressed comments, added condn for dest collection exists in separate qdrant --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Added client * Exception handling * Updated Readme (#134) * Added banner images * Update README.md Added Readme Banner * Rename Github Readme Banner.png to readme-banner.png * Rename Github Webinar Banner.png to webinar-banner.png * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Updated ReadMe * feat: add drawer for details (#136) * Updated ReadMe with intro * Updated ReadMe * Updated ReadMe * Create LICENSE * update: cognita logo (#138) * Doctr parser (#139) * Added doctr parser * Updated docker file * Updated docker file with poppler * Updated tfy ignore * Updated Yaml * Added dockerignore * Fixed linting --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> * fixed warnings * Updated env and Readme * Updated table parser * fix: retriever selection (#140) * update: change prompt template based on retriever example (#142) * update: re-ordering controllers, collections (#144) * Updated requirements * Bump pydantic from 1.10.12 to 1.10.13 in /backend Bumps [pydantic](https://github.com/pydantic/pydantic) from 1.10.12 to 1.10.13. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](pydantic/pydantic@v1.10.12...v1.10.13) --- updated-dependencies: - dependency-name: pydantic dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Removed tfy-api key from modules * Updated req * Udaed embedd upload order * Updated transformers * Updated embedding arrangement * Updated git and tfy ignore * Add functionality to delete files after chunking * pre-commit config * Updated requirements * Tfparser (#148) * Some changes for table parsing * Allowed local embedding models * Added examples * Added summary controller * update: change prompt template based on retriever example (#141) * Updated table parser * fix: retriever selection (#140) * update: change prompt template based on retriever example --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * update: re-ordering controllers, collections (#143) * Updated table parser * fix: retriever selection (#140) * update: change prompt template based on retriever example (#142) * update: re-ordering controllers, collections --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * Added summary * Updated docker file * Added tf parser * Updated local metadata and added new controller * Removed tfy-api key from modules * Fixed linting * Fixing parser * Integrated tf-parser to indexing --------- Co-authored-by: Prathamesh Saraf <prathamesh@mbp.local> Co-authored-by: sayan-truefoundry <136362719+sayan-truefoundry@users.noreply.github.com> Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * Updated requirements * Updated requirements * Bump gunicorn from 21.2.0 to 22.0.0 in /backend Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 21.2.0 to 22.0.0. - [Release notes](https://github.com/benoitc/gunicorn/releases) - [Commits](benoitc/gunicorn@21.2.0...22.0.0) --- updated-dependencies: - dependency-name: gunicorn dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Cleanup * Cleanup * Added TFParser that efficiently parses pdf, added auto parser registry during indexing when parser is not specified, fixed embedding bugs * Added multimodal parser and intelligent response controller * changed model download from hf to multi-modal parser * updated local metadata yaml * Vision model routed th' llm-gateway * Updated Dockerfile * Updated docker file for system package updates * Refactored multi-modal parser, added extra config to support it * Added exception handling in docs stream * Changed summary from async to sync * Fixed parser (#150) * Removed summarization for now * removed multi-query retriever for now * Updated streaming in summary * Updated query controller * fix: overflow of collection name in collections page (#151) * feat: show page number in docs source (#152) Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * Multimodal Controller (#153) * Modified metadata sending * Modified metadata sending * Added multi-modal query controller and modified indexing to make parallel api calls and summarize only main image * Removed old code * Added batching over api * Updted prompt for controller, added timeout to qdrant * Fixed typo * Cleaned multi-modal parser, and some adhoc refactoring * lazy loading of deepdoctection due to logger issues * Added list collections api (#154) * Update requirements.txt * updated internal apis * updated req for langchain core * docs: update README.md (#162) trigerred -> triggered * Embedding class for embedding service deployment (#160) * Added Embedding service class that can consume embedding apis from hf deploying using tf * Added Embedding service class that can consume embedding apis from hf deploying using tf * Added kwargs to embedder svc * Added doc for embedding svc * update: FE changes for collections list, data source list API (#155) * Added exception handling for datasources list * feat: support path based routing (#165) * updates to pdf table parser (#168) * Updated embedding svc (#169) * updates to pdf table parser * Updated embedding svc class * Updated ReadMe * Added support for Infinity API, cleaned requirements, and dockerfile * Cleaned and reduced the main code package * updated requirements * updated requirements * Bump requests from 2.31.0 to 2.32.2 in /backend Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.2. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.31.0...v2.32.2) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * feat: add backend api url support using docker build args (#170) * Update ReadMe * Update ReadMe * Update ReadMe * added feature to add upload name to data dir upload (#167) * added feature to add upload name to data dir upload * feat: add upload name field in UI for local upload * Modifed regex for data-dir, and added error handling --------- Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * Added embedding models list from embedding service of infinity * Explicitly pick cuda or cpu in reranker (#176) * Updated local metadata yaml * Fixed data source visiblity on FE (#177) * Fix datasource visiblity (#178) * Fixed data source visiblity on FE * Fixed data source return list in local * Uncomment torch from embedder.requirements.txt * Bump pillow from 10.2.0 to 10.3.0 in /backend Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.2.0 to 10.3.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.2.0...10.3.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Bump fastapi from 0.104.1 to 0.109.1 in /backend Bumps [fastapi](https://github.com/tiangolo/fastapi) from 0.104.1 to 0.109.1. - [Release notes](https://github.com/tiangolo/fastapi/releases) - [Commits](fastapi/fastapi@0.104.1...0.109.1) --- updated-dependencies: - dependency-name: fastapi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Refactored multimodal parser * Fixed multimodal parser import * Update truefoundry to 0.2.4 * Changed names of query controllers * Remove unused code * fix: FE dockerfile api url null safety (#196) * Docker compose and fixes (#195) * Added docker compose for all services * Fixed docker compose files - health checks and port expose * Removed mixedbread embedding class, updated dockerfile to take build arg and export as env, updated reranker class to usie only small reranker * testing for reranker * disable Override env vars * Enabled rerankers * Enabled overide for env * Modified docker file for exp-1 * Modified docker file for exp-2 * Modified docker file for exp-3 * Modified docker file for exp-4 * Modified docker file for exp-5 * Modified docker file for exp-6 * Removed reranker locally should now use infinity service * Added embedding svc url to docker file * Updated FE docker compose * Fixed Frontend docker compose * updated readme for docker compose * Updated ReadMe * Updated Readme * Updated Readme * Fixed indexer to index entire collection when FQN is not supplied (#199) * Fixed indexer to index entire collection when FQN is not supplied * Added run as job arg * Updated readme with warnings * Added get single collection api (#201) * Bump qdrant-client from 1.7.3 to 1.9.0 in /backend (#204) Bumps [qdrant-client](https://github.com/qdrant/qdrant-client) from 1.7.3 to 1.9.0. - [Release notes](https://github.com/qdrant/qdrant-client/releases) - [Commits](qdrant/qdrant-client@v1.7.3...v1.9.0) --- updated-dependencies: - dependency-name: qdrant-client dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fixed qdrant migration on update of qdrant-client (#206) * Fixed qdrant migration on update of qdrant-client * modified args parse * modifed logging statements * update: optimize collections page with new api (#205) * update: optimize collections page with new api * cleanup --------- Co-authored-by: SayanDeveloper <samantasayan23@gmail.com> * cleanup: remove sidebar (#208) * Fixed argparse migration * Made API routes consistent in terms of trailing slashes (#212) * Made API routes consistent in terms of trailing slashes * Added additional routes to support the existing FE until then * Update compose.env * Update docker-compose.yaml * Update README.md * Update docker-compose.yaml * Bump @grpc/grpc-js from 1.8.21 to 1.8.22 in /frontend Bumps [@grpc/grpc-js](https://github.com/grpc/grpc-node) from 1.8.21 to 1.8.22. - [Release notes](https://github.com/grpc/grpc-node/releases) - [Commits](https://github.com/grpc/grpc-node/compare/@grpc/grpc-js@1.8.21...@grpc/grpc-js@1.8.22) --- updated-dependencies: - dependency-name: "@grpc/grpc-js" dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Added dependancy conditions (#213) * Added dependancy conditions * Changed ollama healthcheck * Update docker-compose.yaml Co-authored-by: Chirag Jain <jain.chirag925@gmail.com> * Update README.md * Update compose.env * Bump braces from 3.0.2 to 3.0.3 in /frontend (#217) Bumps [braces](https://github.com/micromatch/braces) from 3.0.2 to 3.0.3. - [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md) - [Commits](micromatch/braces@3.0.2...3.0.3) --- updated-dependencies: - dependency-name: braces dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Refactor qdrant client config parsing (#221) * Refactor qdrant url parsing * Fix NoneType parsing * Set prefer grpc explicitly False * fix fix: Correct environment variable handling in Frontend Dockerfile (#225) (#226) - Adjusted Dockerfile to properly set environment variables using ARG and ENV - Ensured VITE_USE_RELATIVE_BASE_URL, VITE_DOCS_QA_STANDALONE_PATH, and VITE_QA_FOUNDRY_URL are correctly passed and set during the build process Co-authored-by: HanSeulJo <hsjo@surromind.ai> * [WIP] Added models_config and gateway to make it easy to switch between models * [WIP] Started work of streamlining different models * Cleaned changes * Minor changes in model gateway * Minor fixes * Minor fixes and removed unused code * Started using the api_key env variable * fix: emvedding models response structure update * Remove unused code * Relational Metdata store with Prisma (#203) * Added prisma schema * Fixed data sources API, modified METADATA_STORE client, TODOs: fix clients in all the code, fix all collections APIs * Completed datasources and collections api * Added data ingestion apis * Added delete ingestion runs in delete collection * Modified query controllers * Wrapped TF metadata store in agent executor loop to support for sync function * Added docker compose for prisma and postgress * Tfloader is optional available only when TFY_API_KEY is present, modified regex for collection and datasource in FE * import of PrismaStore only for Local environment * Added indexer support via UI * Symlinks and API, for localdir upload, WIP * Added feature to upload data to docker vol * Added delete datasource API for prisma * Added empty function for delete_data_source in Truefoundry * feat: upload to local directory integration in FE (#228) * feat: upload to local directory integration in FE * feat: add data source unlink button * feat: integrate delete data source api * fix: add VITE_USE_LOCAL in dockerfile * Fixed docker-compose for FE * Removed console logs * Updated compose.env * Remove Local MetadataStore and Update ReadMe (#231) * Removed local metadata store * Updated ReadMe * Update README.md * fix: refetch data sources on delete, update endpoints, view all apis link (#233) * chore: remove trailing slash from API (#234) * feat: optional gtag manager support (#235) * fix: create collection new params * remove provider * fix: url based data source deletion (#237) * Update README.md (#238) * Minor fixes in models config (#239) * Small changes * Minor fixes * Streamline Docker compose installation and updated ReadMe (#240) * Streamlined docker compose * Updated ReadME * Updated ReadME * Added sample models.config.json * Added sample models.config.json * updated model types * Fixed Multimodal query controller for VLM calling, minor fixes (#241) * Fixed Multimodal query controller for VLM calling, minor fixes * Fixed streaming for controllers * Refactor and make all metadata store calls async (#243) * Fix models_config.json file issue (#242) * Ignore registring models if API key is not present, and put out a warning. Also create a config.json file at runtime, helpful for template deployments * increase stream timeout * Raise error when config file is not correct, acces file using paths, and computations during init * Added to_dict for models * Reduce env vars in compose * Misc improvements and fixes for local setup (#245) * Misc improvements and fixes for local setup * Fix ports for infinity and length check when embedding * Update workflow conditions * Update workflow * Revert removing reranker and embedding service url as build args * Fix backend url for FE in compose.env (#246) * Make models_config YAML for easy editing (#247) * Make models_config YAML for easy editing * Make models_config configurable via env * Remove redundant code * Add a comment * Pass on MODELS_CONFIG_PATH in env (#248) * Fix typos `contexual` -> `contextual` + remove unsued imports (#249) * Fix typos `contexual` -> `contextual` * Remove unused imports * Add calls to super() (#250) * Fix potentially undefined local variables (#251) * Added reranking via model_gateway class (#253) * Added reranking via model_gateway class * Added Auth for infinity API --------- Co-authored-by: Abhishek Choudhary <abhichoudhary06@gmail.com> * Fixed INFINITY_API_KEY issue in local deployment (#254) * Added INFINITY_API_KEY in config.sample * Optional auth for infnity service based on API key * Updated docker compose ---------

Commits

Commits on Jun 26, 2024

Merge Main to Release (#255)

Merge Main to Release (#255 )