b10532
ggml-org/llama.cppb10532Aug 21, 2026by github-actions[bot]same day as b10531
What changed since b10531
Metal Flash Attention now dequantizes quantized KV cache to F16 before attention, improving accuracy and speed.
New features
- Metal backend gains a preprocessing pass that dequantizes quantized K/V into an F16 scratch buffer before running existing F16 flash attention kernels.
- Quantized KV cache types Q8_0, Q4_0, Q4_1, Q5_0 and Q5_1 are now handled by the Metal flash attention path.
- The dequantization gate is type-only, so it applies regardless of head size, GQA ratio or KV length.
Improvements
- Quantized-KV inference on Metal now matches F16 KV reference quality more closely (perplexity parity reported on Qwen2.5-0.5B with q8_0 KV).
- Existing F16 flash attention kernels are reused with contiguous strides instead of the in-kernel dequantization path.
- Dequantized buffers live in the op's own scratch allocation, and the KV pad kernel reads them when the path is active.
Upgrade notes
No configuration or CLI changes are required; the new path activates automatically on Metal whenever the KV cache is quantized. Users running quantized KV on Apple Silicon should see better output fidelity, at the cost of a small extra scratch buffer per attention op. Non-Metal backends are unaffected by this release.
Analyzed Oct 26, 2026
Upgrading?
Release notes summary
Optimizes Flash Attention on the Metal backend by dequantizing KV cache to F16 in a preprocessing pass, improving accuracy and performance for various quantized KV cache types.
Key Highlights
- Added dequantization pass for KV cache in Metal Flash Attention
- Improved accuracy for quantized KV
- Added support for Q4_0, Q4_1, Q5_0, Q5_1 KV types
Release Notes (excerpt)
<details open> metal : dequantize quantized KV to F16 before flash attention (#27390) * metal: dequantize q8_0 KV to f16 before flash attention Add a preprocessing pass for GGML_OP_FLASH_ATTN_EXT on the Metal backend: when the KV cache is quantized (Q8_0 for now), dequantize K and V into a contiguous F16 scratch buffer and run the existing F16 flash attention kernels on it, instead of the in-kernel dequantization path.…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system