b10532

ggml-org/llama.cppb10532Aug 21, 2026by github-actions[bot]same day as b10531

What changed since b10531

medium risk

Metal Flash Attention now dequantizes quantized KV cache to F16 before attention, improving accuracy and speed.

New features

  • Metal backend gains a preprocessing pass that dequantizes quantized K/V into an F16 scratch buffer before running existing F16 flash attention kernels.
  • Quantized KV cache types Q8_0, Q4_0, Q4_1, Q5_0 and Q5_1 are now handled by the Metal flash attention path.
  • The dequantization gate is type-only, so it applies regardless of head size, GQA ratio or KV length.

Improvements

  • Quantized-KV inference on Metal now matches F16 KV reference quality more closely (perplexity parity reported on Qwen2.5-0.5B with q8_0 KV).
  • Existing F16 flash attention kernels are reused with contiguous strides instead of the in-kernel dequantization path.
  • Dequantized buffers live in the op's own scratch allocation, and the KV pad kernel reads them when the path is active.

Upgrade notes

No configuration or CLI changes are required; the new path activates automatically on Metal whenever the KV cache is quantized. Users running quantized KV on Apple Silicon should see better output fidelity, at the cost of a small extra scratch buffer per attention op. Non-Metal backends are unaffected by this release.

Analyzed Oct 26, 2026

Upgrading?

Release notes summary

Optimizes Flash Attention on the Metal backend by dequantizing KV cache to F16 in a preprocessing pass, improving accuracy and performance for various quantized KV cache types.

Key Highlights

  • Added dequantization pass for KV cache in Metal Flash Attention
  • Improved accuracy for quantized KV
  • Added support for Q4_0, Q4_1, Q5_0, Q5_1 KV types

Release Notes (excerpt)

<details open> metal : dequantize quantized KV to F16 before flash attention (#27390) * metal: dequantize q8_0 KV to f16 before flash attention Add a preprocessing pass for GGML_OP_FLASH_ATTN_EXT on the Metal backend: when the KV cache is quantized (Q8_0 for now), dequantize K and V into a contiguous F16 scratch buffer and run the existing F16 flash attention kernels on it, instead of the in-kernel dequantization path.…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories