Starstruk Logo MarkStarstruk

v0.1.2

EricLBuehler/mistral.rsv0.1.2Apr 30, 2024by EricLBuehler

AI Summary

This release introduces asynchronous integrations for better concurrency and finally stabilizes the Phi 3 128k model, along with intermediate loading capabilities for quantized models.

Key Highlights

  • Initial async integrations for sampling and channels.
  • Phi 3 128k context window support (fully working).
  • Intermediate loading for ISQ models on CPU.
  • Flexible handling of BOS and EOS tokens.

New Features

  • Async integrations
  • Phi 3 128k support
  • Intermediate ISQ loading
  • Flexible BOS/EOS tokens

Full Release Notes

## New features
* Initial `async` integrations (#198, #236) thanks to @lucasavila00.
* More flexibility with `bos` and `eos` tokens (#248)
* Intermediate loading for ISQ models on CPU (#229)
* Fixed Phi 3 128k finally, it is fully working now! (#251)

## Changelog
* Update README.md by @KPCOFGS in https://github.com/EricLBuehler/mistral.rs/pull/224
* Fix api_dir_list! and show better error by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/225
* Default to `none` when cannot find token by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/226
* docs: update ADAPTER_MODELS.md by @eltociear in https://github.com/EricLBuehler/mistral.rs/pull/227
* Fix debug log timing of first token by @lucasavila00 in https://github.com/EricLBuehler/mistral.rs/pull/231
* Implement intermediate loading for ISQ on CPU by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/229
* Async sampling by @lucasavila00 in https://github.com/EricLBuehler/mistral.rs/pull/198
* Fix quantized example by @lucasavila00 in https://github.com/EricLBuehler/mistral.rs/pull/237
* Source bos, eos tokens from generation_config.json by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/243
* Sliding window for phi3 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/244
* Fix docker images by @LLukas22 in https://github.com/EricLBuehler/mistral.rs/pull/249
* Remove forced max seq len for llama models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/250
* Fix Phi3 128k finally: use position ids to switch between short/long scaling by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/251
* Update README.md by @criminact in https://github.com/EricLBuehler/mistral.rs/pull/253
* Async channels by @lucasavila00 in https://github.com/EricLBuehler/mistral.rs/pull/236

## New Contributors
* @KPCOFGS made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/224
* @eltociear made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/227
* @criminact made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/253

**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.1.0...v0.1.2