Starstruk Logo MarkStarstruk

v0.6.0

caddyserver/caddyv0.6.0Jun 3, 2025by stephantul

AI Summary

This release introduces smaller tokenizers and supertokenizers while fixing various bugs related to model handling, dependencies, and documentation updates.

Key Highlights

  • Introduced smaller tokenizers for improved efficiency
  • Added supertokenizers feature
  • Fixed distillation for models without cards and missing unk tokens
  • Updated documentation with multilingual results and a new logo

New Features

  • Smaller tokenizers
  • Supertokenizers
  • Explicit validation set passing
  • Spacing-dependent normalization

Full Release Notes

## What's Changed
* docs: update chonkie link on tutorial readme by @iaurg in https://github.com/MinishLab/model2vec/pull/235
* Fix dates in README.md by @Pringled in https://github.com/MinishLab/model2vec/pull/238
* fix: add default arg for push_to_hub by @stephantul in https://github.com/MinishLab/model2vec/pull/240
* fix: remove direct dependency on specific hf utils by @stephantul in https://github.com/MinishLab/model2vec/pull/244
* feat: smaller tokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/243
* feat: update lock by @stephantul in https://github.com/MinishLab/model2vec/pull/246
* feat: allow passing validation set explicitly by @JarbasAl in https://github.com/MinishLab/model2vec/pull/245
* docs: Added multilingual results by @Pringled in https://github.com/MinishLab/model2vec/pull/247
* fix: distillation for models without card by @JarbasAl in https://github.com/MinishLab/model2vec/pull/248
* feat: add supertokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/236
* clean-up print statement by @stephantul in https://github.com/MinishLab/model2vec/pull/249
* fix: small typing issue by @stephantul in https://github.com/MinishLab/model2vec/pull/250
* docs: Added new logo by @Pringled in https://github.com/MinishLab/model2vec/pull/252
* fix: missing unk, fix bug by @stephantul in https://github.com/MinishLab/model2vec/pull/251
* bump version by @stephantul in https://github.com/MinishLab/model2vec/pull/258
* feat: make normalization dependent on spacing by @stephantul in https://github.com/MinishLab/model2vec/pull/259

## New Contributors
* @iaurg made their first contribution in https://github.com/MinishLab/model2vec/pull/235
* @JarbasAl made their first contribution in https://github.com/MinishLab/model2vec/pull/245

**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.5.0...v0.6.0