Starstruk Logo MarkStarstruk

v0.3.1

MinishLab/model2vecv0.3.1Oct 29, 2024by Pringled

AI Summary

This maintenance release improves token handling, adds optional fast encoding, and fixes a normalization bug.

Key Highlights

  • Updated tokens to be more agnostic
  • Made encode_batch_fast optional
  • Fixed normalization leading to NaN for empty docs

New Features

  • Make encode_batch_fast optional

Full Release Notes

## What's Changed
* fix: update added tokens to be more agnostic by @stephantul in https://github.com/MinishLab/model2vec/pull/107
* fix: don't rely on reported vocab size, log warning if inconsistent by @stephantul in https://github.com/MinishLab/model2vec/pull/109
* docs: Fixed broken links by @Pringled in https://github.com/MinishLab/model2vec/pull/112
* feat: make encode_batch_fast optional by @stephantul in https://github.com/MinishLab/model2vec/pull/113
* fix: normalize would lead to NaN for empty docs by @stephantul in https://github.com/MinishLab/model2vec/pull/114
* docs: Add tokenlearn results by @Pringled in https://github.com/MinishLab/model2vec/pull/116
* docs: Updated plot by @Pringled in https://github.com/MinishLab/model2vec/pull/117
* Bump version by @Pringled in https://github.com/MinishLab/model2vec/pull/118


**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.3.0...v0.3.1