Starstruk Logo MarkStarstruk

v0.6.0

nextjs/saas-starterv0.6.0Jun 3, 2025by stephantul

AI Summary

This release focuses on improving tokenizer efficiency and model management by introducing smaller tokenizers and supertokenizers, while fixing dependency issues and updating documentation.

Key Highlights

  • Introduction of smaller tokenizers for improved efficiency
  • Addition of supertokenizers functionality
  • Removal of direct dependency on specific Hugging Face utilities
  • Allow passing validation set explicitly
  • Updated documentation including multilingual results and a new logo

New Features

  • Smaller tokenizers
  • Supertokenizers
  • Explicit validation set passing
  • Normalization dependent on spacing

Full Release Notes

## What's Changed
* docs: update chonkie link on tutorial readme by @iaurg in https://github.com/MinishLab/model2vec/pull/235
* Fix dates in README.md by @Pringled in https://github.com/MinishLab/model2vec/pull/238
* fix: add default arg for push_to_hub by @stephantul in https://github.com/MinishLab/model2vec/pull/240
* fix: remove direct dependency on specific hf utils by @stephantul in https://github.com/MinishLab/model2vec/pull/244
* feat: smaller tokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/243
* feat: update lock by @stephantul in https://github.com/MinishLab/model2vec/pull/246
* feat: allow passing validation set explicitly by @JarbasAl in https://github.com/MinishLab/model2vec/pull/245
* docs: Added multilingual results by @Pringled in https://github.com/MinishLab/model2vec/pull/247
* fix: distillation for models without card by @JarbasAl in https://github.com/MinishLab/model2vec/pull/248
* feat: add supertokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/236
* clean-up print statement by @stephantul in https://github.com/MinishLab/model2vec/pull/249
* fix: small typing issue by @stephantul in https://github.com/MinishLab/model2vec/pull/250
* docs: Added new logo by @Pringled in https://github.com/MinishLab/model2vec/pull/252
* fix: missing unk, fix bug by @stephantul in https://github.com/MinishLab/model2vec/pull/251
* bump version by @stephantul in https://github.com/MinishLab/model2vec/pull/258
* feat: make normalization dependent on spacing by @stephantul in https://github.com/MinishLab/model2vec/pull/259

## New Contributors
* @iaurg made their first contribution in https://github.com/MinishLab/model2vec/pull/235
* @JarbasAl made their first contribution in https://github.com/MinishLab/model2vec/pull/245

**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.5.0...v0.6.0