Releases: neuml/txtai
v6.0.0
🥳 We're excited to announce the release of txtai 6.0 🥳
This significant milestone release marks txtai's 3 year birthday🎉 If you like txtai, please remember to give it a ⭐!
6.0 adds sparse, hybrid and subindexes to the embeddings interface. It also makes significant improvements to the LLM pipeline workflow. See below for more.
Breaking changes
The vast majority of changes are fully backwards compatible. New features are only enabled when specified. The only breaking change is with the Scoring terms interface, where the index format changed. The main Scoring interface used for word vectors weighting is unchanged.
New Features
- Better BM25 (#508)
- Hybrid Search (#509)
- Add additional indexes for embeddings (#515)
- Refactor Sequences and Generator pipeline into single LLM pipeline (#494)
- Support passing model parameters in pipelines (#500)
- Add "auto-id" capability to Embeddings (#502)
- Add UUID auto-id (#505)
- Add keyword arguments to Embeddings constructor (#503)
- Add top level imports (#514)
Improvements
- Add NumPy ANN Backend (#468)
- Add PyTorch ANN Backend (#469)
- Add notebook covering embeddings configuration options (#470)
- make data - No such file or directory (#473)
- Improve derivation of default embeddings model path (#476)
- Add accelerate dependency (#477)
- Add baseball example application (#484)
- Update minimum Python version to 3.8 (#485)
- Add WAL option for SQLite (#488)
- Add support for alternative acceleration devices (#489)
- Add support for passing torch devices to embeddings and pipelines (#492)
- Documentation updates (#495)
- Improve Pooling tokenizer load method (#499)
- Add ability for extractor to reference another pipeline in applications (#501)
- Reorganize embeddings configuration documentation (#504)
- Support Unicode Text Segmentation in Tokenizer (#507)
- ANN improvements (#510)
- Add multilingual graph topic modeling (#511)
- Add support for configurable text/object fields (#512)
- Update documentation for 6.0 (#513)
- Add count method to database (#517)
- Improvements when indexing through Applications (#518)
- Add what's new in txtai 6.0 notebook (#519)
Bug Fixes
v5.5.1
v5.5.0
This release adds workflow streams and DuckDB as a database backend
↪️️ Workflow streams enable server-side processing of large datasets. Streams iteratively pass content to workflows, no need to pass bulk data through the API.
🦆 DuckDB is a new database backend. Certain larger non-vector driven queries and aggregations will now run significantly faster than with SQLite.
See below for full details on the new features, improvements and bug fixes.
New Features
Improvements
- Modify translation pipeline langdetect parameter to accept language detection function good first issue - Thank you @saucam! (#423, #444)
- Pass generation keyword arguments to underlying text generation pipeline (#457)
- Replace original prompt in text generation pipeline (#459)
Bug Fixes
- Issue with upsert and graph (#421)
- Upsert API fails with graph config while performing after /delete (#435)
- Build errors with latest onnxmltools package (#449)
- Fix issue with embeddings reindex and stale function references (#453)
- Problem with the workflow builder (#454)
- Check for empty queue before attempting to convert inputs to dictionaries (#456)
- Fix issue with latest version of Transformers and TokenDetection.save_pretrained (#458)
Contributors
Assets 2
v5.4.0
This release adds prompt templates, conversational task chaining and Hugging Face Hub integration
📃 Prompt templates dynamically generate text using workflow task inputs. This enables chaining multiple prompts and models together.
🤗 Embeddings now integrate with the Hugging Face Hub! Easily share and load embeddings indexes. There is a full embeddings index available for English Wikipedia.
See below for full details on the new features, improvements and bug fixes.
New Features
- Add translation pipeline parameter to return selected models and detected language - Thank you @saucam! (#383, #424)
- Add sample parameter to Faiss ANN (#427)
- Add support for instruction-based embeddings (#428)
- Add Hugging Face Hub integration (#430)
- Add cloud object storage support for uncompressed embeddings indexes (#431)
- Add support for custom cloud providers (#432)
- Add support for storing embeddings config as JSON (#433)
- Add notebook for syncing embeddings with the cloud (#434)
- Add terms method to embeddings (#445)
- Add extractor reference output format (#446)
- Add template task (#448)
- Add prompt template and task chaining example notebook (#451)
Improvements
- Mention the default storage engine - Thank you @hsm207! (#422)
- Refactor archive module into separate package (#429)
- Resolve application references in pipelines (#441)
- Extractor pipeline improvements (#443)
- Allow task action arguments to be dictionaries in addition to tuples (#447)
- Automatically mark embeddings index files for lfs tracking with Hugging Face Hub (#450)
Bug Fixes
- Pin onnxruntime for macOS in build script (#425)
Contributors
Assets 2
v5.3.0
This release adds embeddings-guided and prompt-driven search along with a number of methods to train language models
🔎 Prompt-driven search is a big step forward towards conversational search in txtai. With this release, complex prompts can now be passed to txtai to customize how search results are returned. Lots of exciting possibilities on this front, stay tuned.
💡 The trainer pipeline now has support for training language models from scratch. It supports masked language modeling (MLM), causal language modeling (CLM) and replaced token detection (ELECTRA-style). This is part of the micromodels effort.
See below for full details on the new features, improvements and bug fixes.
New Features
- Add language modeling task to HFTrainer (#403)
- Add language modeling example notebook (#408)
- Add FAQ section to documentation (#413)
- Add language generation task to HFTrainer (#414)
- Add replaced token detection task to HFTrainer (#415)
- Add generator pipeline for text generation (#416)
- Add notebook for embeddings-guided and prompt-driven search with LLMs (#418)
Improvements
- Normalize BM25 and TF-IDF scores (#401)
- Add note to restart kernel if running in Google Colab - Thank you @hsm207! (#410)
- Add clear error when starting API and config file not found (#412)
- Extractor pipeline 2.0 (#417)
- Make texts parameter optional for extractor pipeline in applications (#420)
Bug Fixes
- Fix issue with ORDER BY case sensitivity (#405)
Contributors
Assets 2
v5.2.0
This release adds TextToSpeech and Cross-Encoder pipelines. The performance of the embeddings.batchtransform method was significantly improved, enabling a speed up in building semantic graphs. Default configuration is now available for Embeddings, allowing an Embeddings instance to be created with no arguments like Pipelines.
See below for full details on the new features, improvements and bug fixes.
New Features
- Add Cross-Encoder support to Similarity pipeline (#372)
- Create compression package (#376)
- Add TextToSpeech pipeline (#389)
- Add TextToSpeech Notebook (#391)
- Add default configuration for Embeddings (#393)
Improvements
- Filter HF API list models request (#381)
- Split pipeline extras by function area (#387)
- Update data package to handle label arrays (#388)
- Modify transcription pipeline to accept raw waveform data (#390)
- Transcription pipeline improvements (#392)
- Allow searching by embedding (#396)
- Modified logger configuration in init.py (libraries shouldn't modify root logger) - Thank you @adin786! (#397)
- Pass evaluation metrics to underlying Trainer (#398)
- Improve batchtransform performance (#399)
Bug Fixes
- Example 31 - Duplicate image detection not working (#357)
- All sorts of issues with Example 18 - Export and run models with ONNX (#369)
- Fix issue with select distinct bug (#379)
- Update build script and tests to address issues with latest version of FastAPI (#380)
- Fix issue with similar and bracket SQL expressions embedded in functions (#382)
- Fix bug with embeddings functions and application config bug (#400)
Contributors
Assets 2
v5.1.0
This release adds new model support for the translation pipeline, OpenAI Whisper support in the transcription pipeline and ARM Docker images. Topic modeling was also updated with improvements, including how to use BM25/TF-IDF indexes to drive topic models.
See below for full details on the new features, improvements and bug fixes.
New Features
Improvements
- Read authentication parameters from storage task (#332)
- Update scoring algorithms (#351)
- Add config option for list of stopwords to ignore with topic generation (#352)
- Allow for setting custom translation model path (#355)
- Update caption pipeline to call image-to-text pipeline (#361)
- Update transcription pipeline to call automatic-speech-recognition pipeline (#362)
- Only pass tokenizer to pipeline when necessary (#363)
- Improve default max length logic for text generation (#364)
- Update transcription notebook (#365)
- Update translation notebook (#366)
- Move mkdocs dependencies from docs.yml to setup.py (#368)
Bug Fixes
v5.0.0
🎈🎉🥳 We're excited to announce the release of txtai 5.0! 🥳🎉🎈
Thank you to the txtai community! Please remember to ⭐ txtai!
txtai 5.0 is a major new release. This release adds the semantic graph along with enabling external integrations. It also adds a number of improvements and bug fixes.
New Features
- Add scoring-based search (#327)
- Add notebook demonstrating functionality of individual embeddings components (#328)
- Add SQL expression columns (#338)
- Add semantic graph component (#339)
- Add notebook covering Semantic Graphs (#341)
- Add graph documentation (#343)
- Allow custom ann, database and graph instances (#344)
Improvements
- Clarify embeddings.save documentation (#325)
- Modify embeddings search candidate default logic (#326)
- Update console to conditionally import library (#333)
- Update ANN package to make terminology more consistent (#334)
- Support non-text document elements in Applications (#335)
- Update workflow documentation to note generator execution (#336)
- Update audio transcription notebook to include example with OpenAI Whisper (#345)
Bug Fixes
Contributors
Assets 2
v4.6.0
🎈🎉🥳 txtai turns 2 🎈🎉🥳
We're excited to release the 25th version of txtai marking it's 2 year anniversary. Thank you to the txtai community. Please remember to ⭐ txtai!
txtai 4.6 is a large but backwards compatible release! This release adds better integration between embeddings and workflows. It also adds a number of significant performance improvements and bug fixes.
New Features
- Add transform workflow action to application (#281)
- Add ability to resolve workflows within applications (#290)
- OFFSET in sql query statement (#293)
- Add webpage summary image generation notebook (#299)
- Add notebook on running txtai with native code (#304)
- Add mmap parameter to Faiss (#308)
- Add indexing guide to docs (#312)
Improvements
- Consume generator outputs in workflow tasks (#291)
- Update pipeline workflow notebook (#292)
- Update tabular notebook (#297)
- Lower required version of Pillow library to prevent unnecessary upgrades (#303)
- Embeddings vector batch improvements (#309)
- Use single constant for current pickle protocol (#310)
- Move quantize config param to Faiss (#311)
- Update documentation with new demo and diagrams (#313)
- Improve embeddings performance with large query limits (#318)
Bug Fixes
- ModuleNotFoundError: No module named 'transformers.hf_api' (#274)
- Dependency issue with ONNX and Protobuf (#285)
- The key should be writable instead of path. Thank you to @csnelsonchu! (#287)
- Fix breaking change in build script from mkdocstrings bug (#289)
- Index id sync issue when inserting multiple data types (text, documents, objects) into Embeddings (#294)
- Labels pipeline outputs changed with transformers 4.20.0 (#295)
- Tabular pipeline throws error when processing list fields (#296)
- txtai load testing (#305)
- Add cloud config to application.upsert method (#306)
Contributors
Assets 2
v4.5.0
This release adds the following new features, improvements and bug fixes.
New Features
- Add scripts to train bashsql query translation model (#271)
- Add QA database example notebook (#272)
- Add CITATION file (#273)
Improvements
- Improve efficiency of external vectors (#275)
- Refactor vectors package to improve code reuse (#276)
- Add logic to detect external vectors method (#277)
Bug Fixes
- Fix summary pipeline issue with transformers>=4.19.0 (#278)