Starstruk Logo MarkStarstruk

v0.8.4

EricLBuehler/mistral.rsv0.8.4Jun 16, 2026by EricLBuehler

AI Summary

Adds OpenAI and Anthropic API compatibility, prebuilt binaries for all platforms, and new models like Gemma 4 and Qwen 3.5.

Key Highlights

  • OpenAI-compatible local agent platform with skills and files API support
  • Anthropic API support
  • Prebuilt binaries for CUDA, Metal, CPU, x86, and aarch64
  • New models: Gemma 4, DiffusionGemma, Qwen 3.5
  • Prometheus /metrics endpoint

New Features

  • OpenAI skills and files API
  • Anthropic API support
  • Prebuilt binaries
  • Gemma 4 support
  • DiffusionGemma/block-diffusion support
  • Qwen 3.5 support
  • Prometheus metrics
  • CUDA performance improvements

Full Release Notes

## Highlights
- OpenAI-compatible local agent platform. Full support for OpenAI skills (/v1/skills), the complete Files API (/v1/files + file inputs and agent-produced outputs), and a shell tool (#2230, #2229)
- Prebuilt binaries + one-line install. `pip install mistralrs` and the install scripts now download prebuilt binaries (e.g, CUDA across all supported compute caps, Metal, CPU, x86 and aarch64), including multi-arch Docker images and per-SM Python wheels. No more compiling from source! (#2218, #2220, #2221)
- Anthropic API support. (#2182)
- Online calibration for K-quants. (#2203)
- New & improved models. Gemma 4 (incl. 12B), DiffusionGemma / block-diffusion models, and Qwen 3.5 perf + tool-calling improvements. (#2191, #2209, #2196)
- CUDA performance. Improved CUDA graphs, paged flash-attention kernels, BF16 CUTLASS 2.x MoE, broader tensor-parallel sizes (#2197, #2202)
- Prometheus /metrics endpoint  (#2189)

## What's Changed
* feat(cuda): improve cuda p2p backend and docs by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2181
* Fix corner chunking case for paged attn SWA prefill by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2183
* feat(server): support Anthropic API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2182
* feat(cuda): allow more general TP sizes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2184
* fix(metal): enable bf16 GDN kernel compilation by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2185
* fix(engine): handle SendError when client disconnects during error reporting by @yussypu in https://github.com/EricLBuehler/mistral.rs/pull/2170
* docs(bench): trim outdated mistralrs-bench README by @fiorelorenzo in https://github.com/EricLBuehler/mistral.rs/pull/2157
* fix(windows): Fix windows builds and cuDNN discovery by @sobrinth in https://github.com/EricLBuehler/mistral.rs/pull/2178
* feat(build): emit rerun-if-env-changed for build-time env vars by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2186
* fix(docs): fix build features on docs.rs builds by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2187
* fix(core): avoid raw reasoning fallback in streaming by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2188
* perf(gguf): batch add_special_tokens to drop O(N^2) tokenizer load by @fiorelorenzo in https://github.com/EricLBuehler/mistral.rs/pull/2177
* build(mistralrs-quant): harden Metal kernel build against silent failures by @fiorelorenzo in https://github.com/EricLBuehler/mistral.rs/pull/2176
* feat: add Prometheus /metrics endpoint by @MicheleCampi in https://github.com/EricLBuehler/mistral.rs/pull/2189
* fix(metal): register PR #2166 kernels in runtime-compile path by @ljchang in https://github.com/EricLBuehler/mistral.rs/pull/2169
* feat(models): support Gemma 4 12B by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2191
* feat(core): improve cuda graph implementation and fixes w/ multimodal edge cases by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2195
* feat(qwen3.5): perf improvements on cuda, handle tool calling format by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2196
* feat(uqff): new revision with improved ser/de functionality and storage by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2199
* feat(cuda): improve cuda graph performance, add paged flash attn kernels by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2197
* feat(quant): support isq/uqff/imatrix machinery for broader model classes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2200
* Add support for CUDA BF16 CUTLASS 2.x MoE kernels by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2202
* feat(metal): fix sdpa offset handling, remove f32 upcast by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2205
* feat(models): integrate DiffusionGemma and integrate block-diffusion models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2209
* feat(core): support online calibration for K-quants by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2203
* feat(docs): improve quality and examples reachability by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2216
* feat(release): prebuilt binaries with download-first install by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2218
* feat(release): update minimum compute cap support by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2219
* feat(ci): add python wheels, aarch64, to release artifact generation by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2220
* feat(ci): Switch to Namespace CI for release matrix by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2221
* feat(cli): ensure docs reflect toml kind key as not required  by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2222
* ci(docs): check generated docs artifacts by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2223
* ci(release): add dry run by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2224
* fix(core): propogate errors during engine creation by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2226
* chore(deps): bump esbuild, @sveltejs/vite-plugin-svelte and vite in /mistralrs-cli/webui by @dependabot[bot] in https://github.com/EricLBuehler/mistral.rs/pull/2228
* feat(diffusiongemma): minor fixes and improvements by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2227
* feat(server): better compatability with newer openai api updates by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2229
* feat(server): support openai /v1/skills, /v1/files and shell tool by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/2230

## New Contributors
* @yussypu made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2170
* @fiorelorenzo made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2157
* @sobrinth made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2178
* @MicheleCampi made their first contribution in https://github.com/EricLBuehler/mistral.rs/pull/2189

**Full Changelog**: https://github.com/EricLBuehler/mistral.rs/compare/v0.8.3...v0.8.4