Starstruk Logo MarkStarstruk

v0.1.6

EricLBuehler/mistral.rsv0.1.6May 13, 2024by EricLBuehler

AI Summary

This release focuses on performance optimizations and new model support, specifically introducing speculative decoding and dynamic LoRA swapping, while refining model selection and fixing critical masking bugs.

Key Highlights

  • Implementation of Speculative Decoding for faster inference.
  • Dynamic LoRA adapter activation support.
  • GGUF support for Phi 3 model.
  • Intelligent scheduler improvements.
  • Fixes for Causal Masking and CUDA bugs.

New Features

  • Speculative decoding
  • X-LoRA support for GGUF
  • Dynamic LoRA swapping
  • Phi 3 quantized model support

Full Release Notes

## What's Changed
* Causal Masking and model selection from `.toml` files by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/278
* Remove sliding window mask from quantized phi3 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/280
* Fix Causal Mask by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/282
* Fix mask caching by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/283
* More intelligent scheduler by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/279
* Use `warn!` macro by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/289
* Use a public repo for tests tokenizer.json by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/290
* Implement Speculative Decoding by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/242
* Add X-LoRA support for GGUF by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/293
* Add some "senseful" fallbacks for `isq` by @LLukas22 in https://github.com/EricLBuehler/mistral.rs/pull/272
* Implement dynamic LoRA swapping by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/262
* More verbose logging when loading locally by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/298
* Make speculative decoding faster without anything fancy by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/297
* fix bug with mistralrs cuda by @joshpopelka20 in https://github.com/EricLBuehler/mistral.rs/pull/299

## New Contributors
* @joshpopelka20 made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/299

## New Features
- Speculative decoding introduced
- GGUF support for Phi 3
- Dynamic LoRA adapter activation support

**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.1.5...v0.1.6