v0.2.2

MinishLab/model2vecv0.2.2Oct 3, 2024by Pringled

AI Summary

Bug fixes targeting tokenizer errors, attention masks, and device selection mechanisms.

Key Highlights

Added explicit errors for BPE and unigram tokenizers
Fixed attention mask crash during distillation
Allowed PCA dims to equal model dims
Added device selection mechanism

New Features

Device selection mechanism

Full Release Notes

## What's Changed
* docs: Update results by @Pringled in https://github.com/MinishLab/model2vec/pull/51
* fix: convert bfloat16 to float because of numpy incompatibility by @stephantul in https://github.com/MinishLab/model2vec/pull/53
* fix: Add explicit errors for BPE and unigram, return tokenizer without cha… by @stephantul in https://github.com/MinishLab/model2vec/pull/54
* Add from model by @stephantul in https://github.com/MinishLab/model2vec/pull/57
* fix: Attention mask being None crashes distillation by @stephantul in https://github.com/MinishLab/model2vec/pull/58
* fix: allow PCA dims == dims by @stephantul in https://github.com/MinishLab/model2vec/pull/59
* enh: Add device selection mechanism by @stephantul in https://github.com/MinishLab/model2vec/pull/60
* Fix in README by @stephantul in https://github.com/MinishLab/model2vec/pull/61
* Bumped version by @Pringled in https://github.com/MinishLab/model2vec/pull/62


**Full Changelog**: https://github.com/MinishLab/model2vec/compare/v0.2.1...v0.2.2