Starstruk Logo MarkStarstruk

v0.6.0

GyulyVGC/sniffnetv0.6.0Jun 3, 2025by stephantul

AI Summary

This release introduces smaller tokenizers and supertokenizers, alongside improvements to model distillation and documentation updates including multilingual results. It also adds the ability to pass validation sets explicitly to improve model training workflows.

Key Highlights

  • Introduction of smaller tokenizers
  • Addition of supertokenizers
  • Multilingual results added to documentation
  • Fix for distillation for models without cards
  • Explicit validation set passing capability

New Features

  • Smaller tokenizers
  • Supertokenizers
  • Explicit validation set passing
  • Multilingual results documentation

Full Release Notes

## What's Changed
* docs: update chonkie link on tutorial readme by @iaurg in https://github.com/MinishLab/model2vec/pull/235
* Fix dates in README.md by @Pringled in https://github.com/MinishLab/model2vec/pull/238
* fix: add default arg for push_to_hub by @stephantul in https://github.com/MinishLab/model2vec/pull/240
* fix: remove direct dependency on specific hf utils by @stephantul in https://github.com/MinishLab/model2vec/pull/244
* feat: smaller tokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/243
* feat: update lock by @stephantul in https://github.com/MinishLab/model2vec/pull/246
* feat: allow passing validation set explicitly by @JarbasAl in https://github.com/MinishLab/model2vec/pull/245
* docs: Added multilingual results by @Pringled in https://github.com/MinishLab/model2vec/pull/247
* fix: distillation for models without card by @JarbasAl in https://github.com/MinishLab/model2vec/pull/248
* feat: add supertokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/236
* clean-up print statement by @stephantul in https://github.com/MinishLab/model2vec/pull/249
* fix: small typing issue by @stephantul in https://github.com/MinishLab/model2vec/pull/250
* docs: Added new logo by @Pringled in https://github.com/MinishLab/model2vec/pull/252
* fix: missing unk, fix bug by @stephantul in https://github.com/MinishLab/model2vec/pull/251
* bump version by @stephantul in https://github.com/MinishLab/model2vec/pull/258
* feat: make normalization dependent on spacing by @stephantul in https://github.com/MinishLab/model2vec/pull/259

## New Contributors
* @iaurg made their first contribution in https://github.com/MinishLab/model2vec/pull/235
* @JarbasAl made their first contribution in https://github.com/MinishLab/model2vec/pull/245

**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.5.0...v0.6.0