v3.0.0
neuml/txtaiv3.0.0May 4, 2021by davidmezzetti
AI Summary
txtai 3.0.0 is a major release that overhauls the project structure, consolidates logic into pipelines and introduces workflows, offering features like large-scale similarity search, multiple ML pipelines, and API bindings for multiple languages.
Key Highlights
- Introduction of workflows
- Large-scale similarity search with multiple backends
- API bindings for JavaScript, Java, Rust, and Go
New Features
- Summary pipeline
- Text extraction pipeline
- Transcription pipeline
- Translation pipeline
- Workflow framework
Full Release Notes
txtai 3.0.0 is a major release with a significant number of new features. This release overhauls the project structure, consolidates logic into pipelines and introduces workflows. Summary of txtai features: - 🔎 Large-scale similarity search with multiple index backends ([Faiss](https://github.com/facebookresearch/faiss), [Annoy](https://github.com/spotify/annoy), [Hnswlib](https://github.com/nmslib/hnswlib)) - 📄 Create embeddings for text snippets, documents, audio and images. Supports transformers and word vectors. - 💡 Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction - ↪️️ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows. - 🔗 API bindings for [JavaScript](https://github.com/neuml/txtai.js), [Java](https://github.com/neuml/txtai.java), [Rust](https://github.com/neuml/txtai.rs) and [Go](https://github.com/neuml/txtai.go) - ☁️ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes) New Features -------------------------- - Add Docker file for API (#59) - Require Faiss 1.7.0 (#60) - Add summary pipeline (#65) - Add text extraction pipeline (#66) - Add transcription pipeline (#67) - Add translation pipeline (#68) - Add workflow framework (#69) - Add additional pipeline abstraction layer for tensor frameworks (#70) - Add tests for new v3 functionality (#71) - Add notebooks covering new v3 functionality (#73) - Add Pipeline Factory (#76) - Add API extensions (#77) - Add workflow builder application (#80) - Add text segmentation pipeline (#81) - Add workflow to API (#82) - Add service workflow task (#83) - Add object storage workflow task (#84) - Add URL workflow task (#85) Improvements -------------------------- - Refactor code into smaller components and modules (#63) - Modify pipeline to accept GPU device id (#64) - Allow direct download of sentence-transformer models (#72) - Update documentation, add site through GitHub pages (#75) - Modularize the API (#78) - Add default truncation to pipelines (#79) Bug Fixes -------------------------- - Non intuitive behaviour of Tokenizer (#61) - [Python 3.9, Mac OS] Code hangs while building embedding index (#62) - embeddings.index Truncation RuntimeError: The size of tensor a (889) must match the size of tensor b (512) at non-singleton dimension 1 (#74)