v0.3.4
MinishLab/model2vecv0.3.4Jan 5, 2025by Pringled
AI Summary
Added documentation for integrations, support for multiprocessing during training, and a semantic chunking tutorial.
Key Highlights
- Added multiprocessing support
- Added multiprocessing threshold parameter
- Added semantic chunking with chonkie tutorial
- Updated config values
New Features
- Add support for pattern for unused tokens
- Add multiprocessing
- Added multiprocessing threshold parameter
Full Release Notes
## What's Changed * docs: Add txtai integration docs by @Pringled in https://github.com/MinishLab/model2vec/pull/130 * docs: Reworked documentation by @Pringled in https://github.com/MinishLab/model2vec/pull/131 * feat: Added semantic chunking with chonkie tutorial by @Pringled in https://github.com/MinishLab/model2vec/pull/133 * feat: Updated config values by @Pringled in https://github.com/MinishLab/model2vec/pull/136 * feat: add support for pattern for unused tokens. by @stephantul in https://github.com/MinishLab/model2vec/pull/138 * feat: Add multiprocessing by @Pringled in https://github.com/MinishLab/model2vec/pull/141 (suggested by [davidmezzetti](https://github.com/davidmezzetti) in https://github.com/MinishLab/model2vec/issues/139) * feat: Added multiprocessing threshold parameter by @Pringled in https://github.com/MinishLab/model2vec/pull/142 * docs: Add langchain example by @Pringled in https://github.com/MinishLab/model2vec/pull/143 * fix: Removed unneeded tokenize call by @Pringled in https://github.com/MinishLab/model2vec/pull/144 * docs: update README.md by @eltociear in https://github.com/MinishLab/model2vec/pull/145 * Bump version by @Pringled in https://github.com/MinishLab/model2vec/pull/146 ## New Contributors * @eltociear made their first contribution in https://github.com/MinishLab/model2vec/pull/145 **Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.3.3...v0.3.4