b10534

ggml-org/llama.cppb10534Aug 21, 2026by github-actions[bot]same day as b10533

What changed since b10533

low risk

Optimizes CUDA decode performance with new runtime override and hardware-specific switch points.

New features

  • Added runtime override GGML_CUDA_MMVQ_MAX to tune decode crossover.
  • Added hardware-specific switch points for Blackwell, DGX Spark, and Ada GPUs.

Improvements

  • CUDA decode performance improved by +23-41% for B=8 Q4_K models.
  • Added clamping and validation for runtime override values.
  • Reduced dependency on runtime env vars for specific hardware configurations.

Upgrade notes

Default behavior is unchanged, but a new runtime override `GGML_CUDA_MMVQ_MAX` is available for tuning. Hardware-specific switch points are now applied automatically. Invalid override values are clamped with a warning.

Analyzed Oct 17, 2026

Upgrading?

Release notes summary

Optimizes CUDA decoding performance by tuning the crossover point between mul_mat_vec_q and MMQ batch kernels, adding hardware-specific switch points to handle different GPUs and quantization types.

Key Highlights

  • CUDA runtime override (GGML_CUDA_MMVQ_MAX) to tune the mvq->MMQ decode crossover
  • Hardware-specific switch points for Blackwell, DGX Spark, and Ada GPUs
  • Performance improvement of +23-41% at B=8 for Q4_K dense models
  • Clamping and validation for runtime override values

Release Notes (excerpt)

<details open> CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover (#26079) * CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover (default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core path, which is faster once quantized decode becomes compute-bound at…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories