v0.3.4

MinishLab/model2vecv0.3.4Jan 5, 2025by Pringled

AI Summary

Added documentation for integrations, support for multiprocessing during training, and a semantic chunking tutorial.

Key Highlights

  • Added multiprocessing support
  • Added multiprocessing threshold parameter
  • Added semantic chunking with chonkie tutorial
  • Updated config values

New Features

  • Add support for pattern for unused tokens
  • Add multiprocessing
  • Added multiprocessing threshold parameter

Full Release Notes

## What's Changed
* docs: Add txtai integration docs by @Pringled in https://github.com/MinishLab/model2vec/pull/130
* docs: Reworked documentation by @Pringled in https://github.com/MinishLab/model2vec/pull/131
* feat: Added semantic chunking with chonkie tutorial by @Pringled in https://github.com/MinishLab/model2vec/pull/133
* feat: Updated config values by @Pringled in https://github.com/MinishLab/model2vec/pull/136
* feat: add support for pattern for unused tokens. by @stephantul in https://github.com/MinishLab/model2vec/pull/138
* feat: Add multiprocessing by @Pringled in https://github.com/MinishLab/model2vec/pull/141 (suggested by [davidmezzetti](https://github.com/davidmezzetti) in https://github.com/MinishLab/model2vec/issues/139)
* feat: Added multiprocessing threshold parameter by @Pringled in https://github.com/MinishLab/model2vec/pull/142
* docs: Add langchain example by @Pringled in https://github.com/MinishLab/model2vec/pull/143
* fix: Removed unneeded tokenize call by @Pringled in https://github.com/MinishLab/model2vec/pull/144
* docs: update README.md by @eltociear in https://github.com/MinishLab/model2vec/pull/145
* Bump version by @Pringled in https://github.com/MinishLab/model2vec/pull/146

## New Contributors
* @eltociear made their first contribution in https://github.com/MinishLab/model2vec/pull/145

**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.3.3...v0.3.4