b9911

ggml-org/llama.cppb9911Jul 8, 2026by github-actions[bot]same day as b9910

What changed since b9910

low risk

Adds CUDA NVFP4 MMVQ post-scale fusion, giving modest throughput gains on Blackwell-class GPUs.

New features

  • Fused MMVQ post-scale operations for NVFP4 quantization on CUDA, including a dense MMVQ fusion path.
  • New test-backend-ops coverage for the added NVFP4 fusion operations.

Improvements

  • Measured ~2-4% token-generation speedup for NVFP4 models on B4500 and DGX Spark hardware.
  • Cleanup of ggml-cuda/mmvq and test-backend-ops internals alongside the fusion work.
  • Non-NVFP4 quantizations such as Q4_K_M show no regression, staying effectively at parity.

Upgrade notes

No config or API changes; this is a drop-in binary upgrade. Benefits only apply to CUDA builds running NVFP4-quantized models on supported NVIDIA hardware, and gains are small (single-digit percent). Users on other backends or quantization types will see no functional difference from b9910.

Analyzed Oct 31, 2026

Upgrading?

Release notes summary

Implements fused matrix-vector quantization (MMVQ) post-scale operations specifically for NVIDIA FP4 (NVFP4) quantization on CUDA, resulting in performance improvements.

Key Highlights

  • Fuse MMVQ post-scale operations for NVFP4 quantization on CUDA
  • Add dense MMVQ fusion
  • Significant speedup on B4500 and DGX Spark hardware
  • Add tests for the new fusion ops

Release Notes (excerpt)

<details open> CUDA: Fuse MMVQ post-scale for NVFP4 (#24481) * CUDA: Fuse MMVQ for NVFP4 and BS 1 TODO: 1. Add tests to test-backend-ops (did verify correctness manually for one model) 2. Reorder bias/scale once PRs for NVFP4 are merged/landed * Add dense MMVQ fusion as well Perf numbers on B4500. Note qwen35 is FP8->Q8 + ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories