b9911
ggml-org/llama.cppb9911Jul 8, 2026by github-actions[bot]same day as b9910
What changed since b9910
Adds CUDA NVFP4 MMVQ post-scale fusion, giving modest throughput gains on Blackwell-class GPUs.
New features
- Fused MMVQ post-scale operations for NVFP4 quantization on CUDA, including a dense MMVQ fusion path.
- New test-backend-ops coverage for the added NVFP4 fusion operations.
Improvements
- Measured ~2-4% token-generation speedup for NVFP4 models on B4500 and DGX Spark hardware.
- Cleanup of ggml-cuda/mmvq and test-backend-ops internals alongside the fusion work.
- Non-NVFP4 quantizations such as Q4_K_M show no regression, staying effectively at parity.
Upgrade notes
No config or API changes; this is a drop-in binary upgrade. Benefits only apply to CUDA builds running NVFP4-quantized models on supported NVIDIA hardware, and gains are small (single-digit percent). Users on other backends or quantization types will see no functional difference from b9910.
Analyzed Oct 31, 2026
Upgrading?
Release notes summary
Implements fused matrix-vector quantization (MMVQ) post-scale operations specifically for NVIDIA FP4 (NVFP4) quantization on CUDA, resulting in performance improvements.
Key Highlights
- Fuse MMVQ post-scale operations for NVFP4 quantization on CUDA
- Add dense MMVQ fusion
- Significant speedup on B4500 and DGX Spark hardware
- Add tests for the new fusion ops
Release Notes (excerpt)
<details open> CUDA: Fuse MMVQ post-scale for NVFP4 (#24481) * CUDA: Fuse MMVQ for NVFP4 and BS 1 TODO: 1. Add tests to test-backend-ops (did verify correctness manually for one model) 2. Reorder bias/scale once PRs for NVFP4 are merged/landed * Add dense MMVQ fusion as well Perf numbers on B4500. Note qwen35 is FP8->Q8 + ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system