b10032
ggml-org/llama.cppb10032Jul 15, 2026by github-actions[bot]same day as b10031
What changed since b10031
b10032 adds a CUDA implementation of the lightning indexer op with generic vector and WMMA kernels, plus Turing-level MMA support.
New features
- CUDA backend now implements GGML_OP_LIGHTNING_INDEXER via a generic vector kernel and a WMMA kernel.
- Lightning indexer kernels accept WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters, removing duplicated constants.
Improvements
- MMA architecture requirement for the lightning indexer relaxed from newer GPUs down to Turing, widening hardware support.
- Added alignment checks on Q and K tensors in the CUDA lightning indexer path for safer execution.
- Internal renaming and whitespace cleanup of the CUDA lightning indexer code (ggml_cuda_lightning_indexer, uppercased constants).
- Removed unnecessary kernel template declarations and noted a TODO for AMD rocWMMA support.
Bug fixes
- Fixed variable shadowing issues in the lightning indexer kernels through successive renames.
Upgrade notes
This is an additive CUDA-backend change with no CLI, config, or API impact, so existing invocations and model files continue to work unchanged. Users on pre-Turing NVIDIA GPUs still won't get the WMMA path, but the generic vector kernel provides a fallback. Rebuild or pull the new CUDA binaries to benefit from accelerated lightning indexing.
Analyzed Oct 31, 2026
Upgrading?
Release notes summary
Adds a new CUDA implementation for the lightning indexer operation using generic vector and WMMA kernels, with architectural relaxations and refactoring.
Key Highlights
- CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel)
- Relaxed MMA architecture requirements to Turing
- Added alignment checks for Q and K tensors
- Template parameters added to avoid duplication of constants
Release Notes (excerpt)
<details open> cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) (#25545) * cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) * chore : remove indentation of #pragma unroll * cuda : remove unnecessary kernel template declarations * cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants.…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system