b10534
ggml-org/llama.cppb10534Aug 21, 2026by github-actions[bot]same day as b10533
What changed since b10533
Optimizes CUDA decode performance with new runtime override and hardware-specific switch points.
New features
- Added runtime override GGML_CUDA_MMVQ_MAX to tune decode crossover.
- Added hardware-specific switch points for Blackwell, DGX Spark, and Ada GPUs.
Improvements
- CUDA decode performance improved by +23-41% for B=8 Q4_K models.
- Added clamping and validation for runtime override values.
- Reduced dependency on runtime env vars for specific hardware configurations.
Upgrade notes
Default behavior is unchanged, but a new runtime override `GGML_CUDA_MMVQ_MAX` is available for tuning. Hardware-specific switch points are now applied automatically. Invalid override values are clamped with a warning.
Analyzed Oct 17, 2026
Upgrading?
Release notes summary
Optimizes CUDA decoding performance by tuning the crossover point between mul_mat_vec_q and MMQ batch kernels, adding hardware-specific switch points to handle different GPUs and quantization types.
Key Highlights
- CUDA runtime override (GGML_CUDA_MMVQ_MAX) to tune the mvq->MMQ decode crossover
- Hardware-specific switch points for Blackwell, DGX Spark, and Ada GPUs
- Performance improvement of +23-41% at B=8 for Q4_K dense models
- Clamping and validation for runtime override values
Release Notes (excerpt)
<details open> CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover (#26079) * CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover (default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core path, which is faster once quantized decode becomes compute-bound at…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system