v7.4.0
neuml/txtaiv7.4.0Sep 5, 2024by davidmezzetti
AI Summary
This release adds SQLite ANN support, enhances markdown support for Textractor, and updates the index format to be language-neutral.
Key Highlights
- Add SQLite ANN
- Update txtai index format to remove Python-specific serialization
- Add MessagePack serialization as a top level dependency
- Enhance markdown support for Textractor
New Features
- SQLite ANN
- Language-neutral index format
- MessagePack serialization
- Textractor markdown enhancements
Full Release Notes
**This release adds the SQLite ANN, new text extraction features and a programming language neutral embeddings index format** See below for full details on the new features, improvements and bug fixes. New Features -------------------------- - Add SQLite ANN (#780) - Enhance markdown support for Textractor (#758) - Update txtai index format to remove Python-specific serialization (#769) - Add new functionality to RAG application (#753) - Add bm25s library to benchmarks (#757) Thank you @a0346f102085fe9f! - Add serialization package for handling supported data serialization methods (#770) - Add MessagePack serialization as a top level dependency (#771) Improvements -------------------------- - Support `<pre>` blocks with Textractor (#749) - Update HF LLM to reduce noisy warnings (#752) - Update NLTK model downloads (#760) - Refactor benchmarks script (#761) - Update documentation to use base imports (#765) - Update examples to use RAG pipeline instead of Extractor when paired with LLMs (#766) - Modify NumPy and Torch ANN components to use np.load/np.save (#772) - Persist Embeddings index ids (only used when content storage is disabled) with MessagePack (#773) - Persist Reducer component with skops library (#774) - Persist NetworkX graph component with MessagePack (#775) - Persist Scoring component metadata with MessagePack (#776) - Modify vector transforms to load/save data using np.load/np.save (#777) - Refactor embeddings configuration into separate component (#778) - Document txtai index format (#779) Bug Fixes -------------------------- - Translation: AttributeError: 'ModelInfo' object has no attribute 'modelId' (#750) - Change RAGTask to RagTask (#763) - Notebook 42 error (#768)