Starstruk Logo MarkStarstruk

v0.3.2

EricLBuehler/mistral.rsv0.3.2Oct 28, 2024by EricLBuehler

AI Summary

A feature release focusing on quantization improvements with FP8 ISQ and GPTQ Marlin support, a significant performance boost on Metal, and the addition of Python package wheels along with support for Qwen 2.5.

Key Highlights

  • ISQ FP8 quantization support
  • GPTQ Marlin support
  • 26% performance boost on Metal
  • Python package wheels available

New Features

  • ISQ FP8
  • GPTQ Marlin (4 and 8 bit)
  • Qwen 2.5 support
  • Python package wheels
  • Improved UQFF memory efficiency

Full Release Notes

## Key changes
- General improvements and fixes
- ISQ FP8
- GPTQ Marlin
- 26% performance boost on Metal
- Python package wheels are available. See below and the various PyPi packages.

## What's Changed
* Update docs and deps by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/804
* Support Qwen 2.5 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/805
* Update docs with clarifications and notes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/806
* Improved inverting for Attention Mask by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/811
* Fix `repeat_interleave` by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/812
* Use f32 for neg inf in cross attn mask by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/814
* Improve UQFF memory efficiency by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/813
* Update Metal, CUDA Candle impls and ISQ by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/816
* chore: update pagedattention.cu by @eltociear in https://github.com/EricLBuehler/mistral.rs/pull/822
* MLlama - if f16, load vision model in f32 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/820
* ci: Upgrade actions by @polarathene in https://github.com/EricLBuehler/mistral.rs/pull/823
* docs: added a top button because of readme length by @bhargavshirin in https://github.com/EricLBuehler/mistral.rs/pull/833
* Typo in error of model architecture enum by @nikolaydubina in https://github.com/EricLBuehler/mistral.rs/pull/835
* Expose config for Rust api, tweak modekind by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/841
* Add ISQ FP8 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/832
* Fix Metal F8 build errors by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/846
* Bump pyo3 from 0.22.3 to 0.22.4 by @dependabot in https://github.com/EricLBuehler/mistral.rs/pull/854
* Generate standalone UQFF models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/849
* Update README.MD by @kaleaditya779 in https://github.com/EricLBuehler/mistral.rs/pull/848
* Add GPTQ Marlin support for 4 and 8 bit by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/856
* Adds wrap_help feature to clap by @DaveTJones in https://github.com/EricLBuehler/mistral.rs/pull/858
* Patch UQFF metal generation by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/857
* Add GGUF Qwen 2 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/860
* Avoid duplicate Metal command buffer encodings during ISQ by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/861
* Fix for isnanf by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/859
* Fix some metal warnings by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/862
* Support interactive mode markdown bold/italics via ANSI codes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/879
* Even better V-Llama accuracy by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/881
* Trim whitespace (such as carriage returns) from nvidia-smi output. by @asaddi in https://github.com/EricLBuehler/mistral.rs/pull/880
* MODEL_ID not "MODEL_ID" by @simonw in https://github.com/EricLBuehler/mistral.rs/pull/863
* Sync ggml metal kernels by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/885
* Increase Metal decoding T/s by 26% by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/887
* Remove pretty-printer by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/889
* Fix typo in documentation by @msk in https://github.com/EricLBuehler/mistral.rs/pull/888
* fix Half-Quadratic Quantization and Dequantization on CPU by @haricot in https://github.com/EricLBuehler/mistral.rs/pull/873
* Prepare for v0.3.2 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/891

## New Contributors
* @bhargavshirin made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/833
* @nikolaydubina made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/835
* @kaleaditya779 made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/848
* @DaveTJones made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/858
* @asaddi made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/880
* @simonw made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/863
* @msk made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/888
* @haricot made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/873

**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.3.1...v0.3.2