Starstruk Logo MarkStarstruk

v0.8.0

EricLBuehler/mistral.rsv0.8.0Apr 2, 2026by EricLBuehler

AI Summary

A major version release upgrading Metal to 3.1, introducing new quantization methods (F8Q8), rewriting paged attention for better prefix caching, and adding support for Mistral3 and Qwen 3.5.

Key Highlights

  • Metal standard upgrade from 3.0 to 3.1
  • New quantization method: F8Q8
  • Rewrite of paged attention with block-level prefix caching
  • Support for Mistral3 and Qwen 3.5
  • TVOS Metal support

New Features

  • F8Q8 quantization
  • GPT-OSS paged attention with sinks support
  • Block-level prefix caching with KV gather kernels
  • GGUF for Mistral3
  • Voxtral Mini 4B support
  • Qwen 3.5 support
  • Rust SDK improvements (deferred media prefixing, typed errors)
  • TVOS Metal support

Full Release Notes

## What's Changed
* Tweaks to docs and readme by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1854
* Upgrade Metal standard from 3.0 to 3.1 by @lizzzcai in https://github.com/EricLBuehler/mistral.rs/pull/1861
* fix stable diffusion readme by @setoelkahfi in https://github.com/EricLBuehler/mistral.rs/pull/1857
* Use cudaforge for kernel build by @guoqingbao in https://github.com/EricLBuehler/mistral.rs/pull/1856
* Bump bytes from 1.11.0 to 1.11.1 by @dependabot[bot] in https://github.com/EricLBuehler/mistral.rs/pull/1865
* Fix accuracy of fused glu metal and cuda impls by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1867
* Bump time from 0.3.45 to 0.3.47 by @dependabot[bot] in https://github.com/EricLBuehler/mistral.rs/pull/1868
* Fix for ViT + flash attn case by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1869
* Parallel + I/O pipelined ISQ by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1870
* Fix gptoss sliding window case with prefix caching by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1871
* Change gguf files delimiter to ';' by @synek317 in https://github.com/EricLBuehler/mistral.rs/pull/1873
* GPT-OSS paged attention with sinks support, MoE prefill kernels across CUDA, Metal, and CPU by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1872
* Fix streaming sse hang on error event by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1875
* Support Qwen 3 Next by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1864
* Fix completions ignoring logprobs by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1877
* Fixes for Qwen 3 VL family by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1878
* Add new quant method: F8Q8 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1883
* fix(docker): install git in CUDA builders for flash-attn-v3 CUTLASS fetch by @glaziermag in https://github.com/EricLBuehler/mistral.rs/pull/1885
* Bump to 0.7.1-alpha.1 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1880
* fix(core): use unix seconds for streaming chunk created timestamp by @glaziermag in https://github.com/EricLBuehler/mistral.rs/pull/1887
* feat: tvos metal support by @setoelkahfi in https://github.com/EricLBuehler/mistral.rs/pull/1891
* Rewrite paged attention for block-level prefix caching with KV gather kernels    by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1890
* Fix contiguous error with phi3 gguf by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1892
* fix(core): handle missing BOS token in calibration path by @glaziermag in https://github.com/EricLBuehler/mistral.rs/pull/1895
* feat: add optional save_file for url image generation response format by @setoelkahfi in https://github.com/EricLBuehler/mistral.rs/pull/1893
* fix(metal): load metallib from memory instead of temp file for sandbox compatability by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1898
* fix(build): enable vendored Swagger UI for offline compilation by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1899
* fix(cuda): account for tensor storage offset in GDN kernel launches by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1900
* fix(cuda): account for tensor storage offset in moe kernel launches by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1901
* Implement GGUF for Mistral3 by @Cooksey99 in https://github.com/EricLBuehler/mistral.rs/pull/1771
* feat(rust sdk): deferred media prefixing, typed errors, and API cleanup, restructure examples by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1904
* feat(models): add Voxtral Mini 4B real-time speech recognition model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1905
* fix(ci): add Metal and CUDA+NCCL compile checks by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1907
* fix(device_map): pre-allocate masks per device to reduce OOM pressure by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1908
* feat(pyo3): release GIL around blocking Runner operations to improve Python SDK by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1909
* feat(server-core): make utoipa-swagger-ui an optional feature by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1910
* fix(server-core): terminate SSE streams when response channel closes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1943
* ci: disable docs deployment on forks by @haricot in https://github.com/EricLBuehler/mistral.rs/pull/1942
* fix: memory limit constants for 32-bit targets in attention and ISQ by @setoelkahfi in https://github.com/EricLBuehler/mistral.rs/pull/1933
* fix(gguf): `verify_arch_any` used AND logic instead of OR by @n-engine in https://github.com/EricLBuehler/mistral.rs/pull/1916
* fix(#1934): emulate negative step range in chat templates by @haricot in https://github.com/EricLBuehler/mistral.rs/pull/1941
* fix(vision): correct Qwen VL multi-turn image processing and thinking model token decoding by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1950
* Update MCP client documentation link in README by @naufraghi in https://github.com/EricLBuehler/mistral.rs/pull/1935
* feat(models): support Qwen 3.5 model family by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1993
* feat(cli): add --uqff-base-model and --uqff-repo-id flags to quantize command  by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1994
* fix(cli): ensure readme matches older versions by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1995
* fix(isq): bits standardize format for numerical isq setting by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1997
* fix(metal): upgrade paged-attn to Metal 3.1 for native bfloat16 support by @ljchang in https://github.com/EricLBuehler/mistral.rs/pull/2010
* Small fix for Voxtral: load params.json before config.json if present by @jam10o-new in https://github.com/EricLBuehler/mistral.rs/pull/1979
* Fix UQFF loading for MoE models in Qwen2Loader by @glaziermag in https://github.com/EricLBuehler/mistral.rs/pull/1977
* fix(metal): auto-retry on iOS Metal background GPU permission error by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2015
* fix(ring): support Ring backend in properply in more models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2016
* fix(cache): set hybrid recurrent state_indices during prompt cache reset by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2017
* feat(quant): add MXFP4 ISQ with optimized decode kernels by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2018
* refactor(wrapper-crates): reduce duplicated builder and request glue by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2019
* fix(docs): duplicate entry in SUMMARY.md breaks docs build by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2020
* Implement the Gemma 4 model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2046

## New Contributors
* @lizzzcai made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/1861
* @setoelkahfi made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/1857
* @synek317 made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/1873
* @glaziermag made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/1885
* @n-engine made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/1916
* @naufraghi made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/1935
* @ljchang made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2010
* @jam10o-new made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/1979

**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.7.0...v0.8.0