v5.5.0

neuml/txtaiv5.5.0Apr 20, 2023by davidmezzetti

AI Summary

Adds workflow streams for server-side processing of large datasets and introduces DuckDB as a new database backend.

Key Highlights

  • Workflow streams enable iterative processing of large datasets
  • DuckDB backend speeds up non-vector driven queries and aggregations

New Features

  • Workflow streams
  • DuckDB support
  • Translation pipeline langdetect parameter updates
  • Pass generation keyword arguments to pipelines
  • Replace original prompt in generation pipeline

Full Release Notes

**This release adds workflow streams and DuckDB as a database backend**

↪️️ Workflow streams enable server-side processing of large datasets. Streams iteratively pass content to workflows, no need to pass bulk data through the API.

🦆 DuckDB is a new database backend. Certain larger non-vector driven queries and aggregations will now run significantly faster than with SQLite.

See below for full details on the new features, improvements and bug fixes.

New Features
--------------------------
- Add workflow streams (#461)
- Add DuckDB support (#462)

Improvements
--------------------------
- Modify translation pipeline langdetect parameter to accept language detection function good first issue - Thank you @saucam! (#423, #444) 
- Pass generation keyword arguments to underlying text generation pipeline (#457)
- Replace original prompt in text generation pipeline (#459)

Bug Fixes
--------------------------
- Issue with upsert and graph (#421)
- Upsert API fails with graph config while performing after /delete (#435)
- Build errors with latest onnxmltools package (#449)
- Fix issue with embeddings reindex and stale function references (#453)
- Problem with the workflow builder (#454)
- Check for empty queue before attempting to convert inputs to dictionaries (#456)
- Fix issue with latest version of Transformers and TokenDetection.save_pretrained (#458)