Starstruk Logo MarkStarstruk

v0.3.0

EricLBuehler/mistral.rsv0.3.0Sep 2, 2024by EricLBuehler

AI Summary

Introduces ISQ topology and device mapping, enhances FlashAttention performance, and removes plotly dependencies. MSRV 1.79.0.

Key Highlights

  • ISQ (Intermediate Scale Quantization) topology
  • Device mapping topology
  • Faster FlashAttention with batching
  • Phi 3.5 support (standard, vision, MoE)

Breaking Changes

  • Bump version to v0.3.0

New Features

  • ISQ topology
  • Device mapping
  • FlashAttention improvements
  • Phi 3.5 support
  • Multiple GGUF files support
  • DRY penalty
  • Phi 3.5 MoE
  • Vision URL support

Full Release Notes

## Highlights
- New model topology feature: ISQ and device mapping
- 🔥Faster FlashAttention support when batching
- Removed `plotly` and associated JS dependencies
- φ³ Support Phi 3.5, Phi 3.5 vision, Phi 3.5 MoE
- Improved Rust API ergonomics
- Support multiple (shaded) GGUF files

## MSRV
The Rust MSRV of this version is 1.79.0

## What's Changed
* Fixes for auto dtype selection with RUST_BACKTRACE=1 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/690
* Add support multiple GGUF files by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/692
* Refactor normal and vision loaders by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/693
* Fix `split.count` GGUF duplication handling by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/695
* Batching example by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/694
* Some fixes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/697
* Improve vision rust examples by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/698
* Add ISQ topology by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/701
* Add custom logits processor API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/702
* Add Gemma 2 PagedAttention support by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/704
* Faster RmsNorm in Gemma/Gemma2 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/703
* Fix bug in Metal ISQ by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/706
* Support GGUF BF16 tensors by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/691
* Better support for FlashAttention: real batching + sliding window + softcap by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/707
* Remove some usages of `pub` in models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/708
* Support the Phi 3.5 V model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/710
* Implement the Phi 3.5 MoE model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/709
* Device map topology by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/717
* Implement DRY penalty by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/637
* Remove plotly and just output CSV loss file by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/700
* Using once_cell to reduce MSRV by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/724
* Fixes for Windows build by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/729
* Even more phi3.5moe fix attempts by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/731
* Add example for Phi 3.5 MoE by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/733
* Add Phi 3.5 chat template by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/734
* Patch ISQ for Mixtral by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/730
* Gracefully handle Engine Drop with termination request by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/735
* feat(vision): add support for proper file and data image URLs by @Schuwi in https://github.com/EricLBuehler/mistral.rs/pull/727
* Add new parsing to Python API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/737
* Remove test and add custom error type to Python API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/738
* Update kernels for metal bf16 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/719
* Better `Response` Result API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/739
* More Metal quantized kernel fixes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/740
* [Breaking] Bump version to v0.3.0 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/736
* Final changes for v0.3.0 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/741

## New Contributors
* @Schuwi made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/727

**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.2.5...v0.3.0