Starstruk Logo MarkStarstruk

v0.6.0

laramies/theHarvesterv0.6.0Jun 3, 2025by stephantul

AI Summary

Release featuring smaller tokenizers, supertokenizers support, explicit validation set passing, multilingual results documentation, and fixes for models without card distillation.

Key Highlights

  • Smaller tokenizers implementation
  • Supertokenizers support
  • Allow passing validation set explicitly
  • Multilingual results documentation
  • Fix for distillation for models without card

New Features

  • Smaller tokenizers
  • Supertokenizers
  • Explicit validation set passing
  • Multilingual results

Full Release Notes

## What's Changed
* docs: update chonkie link on tutorial readme by @iaurg in https://github.com/MinishLab/model2vec/pull/235
* Fix dates in README.md by @Pringled in https://github.com/MinishLab/model2vec/pull/238
* fix: add default arg for push_to_hub by @stephantul in https://github.com/MinishLab/model2vec/pull/240
* fix: remove direct dependency on specific hf utils by @stephantul in https://github.com/MinishLab/model2vec/pull/244
* feat: smaller tokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/243
* feat: update lock by @stephantul in https://github.com/MinishLab/model2vec/pull/246
* feat: allow passing validation set explicitly by @JarbasAl in https://github.com/MinishLab/model2vec/pull/245
* docs: Added multilingual results by @Pringled in https://github.com/MinishLab/model2vec/pull/247
* fix: distillation for models without card by @JarbasAl in https://github.com/MinishLab/model2vec/pull/248
* feat: add supertokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/236
* clean-up print statement by @stephantul in https://github.com/MinishLab/model2vec/pull/249
* fix: small typing issue by @stephantul in https://github.com/MinishLab/model2vec/pull/250
* docs: Added new logo by @Pringled in https://github.com/MinishLab/model2vec/pull/252
* fix: missing unk, fix bug by @stephantul in https://github.com/MinishLab/model2vec/pull/251
* bump version by @stephantul in https://github.com/MinishLab/model2vec/pull/258
* feat: make normalization dependent on spacing by @stephantul in https://github.com/MinishLab/model2vec/pull/259

## New Contributors
* @iaurg made their first contribution in https://github.com/MinishLab/model2vec/pull/235
* @JarbasAl made their first contribution in https://github.com/MinishLab/model2vec/pull/245

**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.5.0...v0.6.0