Starstruk Logo MarkStarstruk

v0.6.0

superset-sh/supersetv0.6.0Jun 3, 2025by stephantul

AI Summary

This release focuses on improving model utilities for model2vec, introducing smaller and super tokenizers, enhancing multilingual support, and fixing various model handling bugs.

Key Highlights

  • Introduced smaller tokenizers and supertokenizers
  • Enhanced multilingual results and support
  • Updated documentation including new logo

New Features

  • Smaller tokenizers
  • Supertokenizers
  • Explicit validation set passing
  • Multilingual results
  • Normalization dependent on spacing

Full Release Notes

## What's Changed
* docs: update chonkie link on tutorial readme by @iaurg in https://github.com/MinishLab/model2vec/pull/235
* Fix dates in README.md by @Pringled in https://github.com/MinishLab/model2vec/pull/238
* fix: add default arg for push_to_hub by @stephantul in https://github.com/MinishLab/model2vec/pull/240
* fix: remove direct dependency on specific hf utils by @stephantul in https://github.com/MinishLab/model2vec/pull/244
* feat: smaller tokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/243
* feat: update lock by @stephantul in https://github.com/MinishLab/model2vec/pull/246
* feat: allow passing validation set explicitly by @JarbasAl in https://github.com/MinishLab/model2vec/pull/245
* docs: Added multilingual results by @Pringled in https://github.com/MinishLab/model2vec/pull/247
* fix: distillation for models without card by @JarbasAl in https://github.com/MinishLab/model2vec/pull/248
* feat: add supertokenizers by @stephantul in https://github.com/MinishLab/model2vec/pull/236
* clean-up print statement by @stephantul in https://github.com/MinishLab/model2vec/pull/249
* fix: small typing issue by @stephantul in https://github.com/MinishLab/model2vec/pull/250
* docs: Added new logo by @Pringled in https://github.com/MinishLab/model2vec/pull/252
* fix: missing unk, fix bug by @stephantul in https://github.com/MinishLab/model2vec/pull/251
* bump version by @stephantul in https://github.com/MinishLab/model2vec/pull/258
* feat: make normalization dependent on spacing by @stephantul in https://github.com/MinishLab/model2vec/pull/259

## New Contributors
* @iaurg made their first contribution in https://github.com/MinishLab/model2vec/pull/235
* @JarbasAl made their first contribution in https://github.com/MinishLab/model2vec/pull/245

**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.5.0...v0.6.0