Starstruk Logo MarkStarstruk

v0.8.0

laramies/theHarvesterv0.8.0Mar 26, 2026by Pringled

AI Summary

A major refactor replacing the tokenizer with `skeletoken`, refactoring persistence, and deprecating Python 3.9 support.

Key Highlights

  • Tokenizer replaced with `skeletoken`
  • Persistence refactored and `from_sentence_transformers` removed
  • Deprecated Python 3.9 support
  • Exposed random seed and added tokenizer utilities

New Features

  • Integration of `skeletoken` for tokenizer functionality
  • Refactored persistence layer
  • Exposure of random seed for reproducibility
  • New utility for tokenizer padding ID

Full Release Notes

## What's Changed
* docs: Added CITATION.cff, updated citation info by @Pringled in https://github.com/MinishLab/model2vec/pull/293
* docs: Added modelcard logo by @Pringled in https://github.com/MinishLab/model2vec/pull/294
* docs: Update default model for example by @Pringled in https://github.com/MinishLab/model2vec/pull/299
* chore: Deprecated Python 3.9 support by @Pringled in https://github.com/MinishLab/model2vec/pull/301
* chore: Update CI to run on external PRs, fix test import by @Pringled in https://github.com/MinishLab/model2vec/pull/303
* feat: replace tokenizer code with skeletoken by @stephantul in https://github.com/MinishLab/model2vec/pull/302
* feat: refactor persistence, remove from_sentence_transformers by @stephantul in https://github.com/MinishLab/model2vec/pull/305
* fix: refix tokenizer with added token shenanigans by @stephantul in https://github.com/MinishLab/model2vec/pull/304
* fix: if layers == 0, layers were not initialized by @stephantul in https://github.com/MinishLab/model2vec/pull/306
* feat: expose random seed, small typing fix by @stephantul in https://github.com/MinishLab/model2vec/pull/308
* feat: add util for tokenizer pad id  by @stephantul in https://github.com/MinishLab/model2vec/pull/310
* docs: Update results to MTEB V2 by @Pringled in https://github.com/MinishLab/model2vec/pull/311
* chore: Bump version by @Pringled in https://github.com/MinishLab/model2vec/pull/312


**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.7.0...v0.8.0