b10032

ggml-org/llama.cppb10032Jul 15, 2026by github-actions[bot]same day as b10031

What changed since b10031

medium risk

b10032 adds a CUDA implementation of the lightning indexer op with generic vector and WMMA kernels, plus Turing-level MMA support.

New features

  • CUDA backend now implements GGML_OP_LIGHTNING_INDEXER via a generic vector kernel and a WMMA kernel.
  • Lightning indexer kernels accept WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters, removing duplicated constants.

Improvements

  • MMA architecture requirement for the lightning indexer relaxed from newer GPUs down to Turing, widening hardware support.
  • Added alignment checks on Q and K tensors in the CUDA lightning indexer path for safer execution.
  • Internal renaming and whitespace cleanup of the CUDA lightning indexer code (ggml_cuda_lightning_indexer, uppercased constants).
  • Removed unnecessary kernel template declarations and noted a TODO for AMD rocWMMA support.

Bug fixes

  • Fixed variable shadowing issues in the lightning indexer kernels through successive renames.

Upgrade notes

This is an additive CUDA-backend change with no CLI, config, or API impact, so existing invocations and model files continue to work unchanged. Users on pre-Turing NVIDIA GPUs still won't get the WMMA path, but the generic vector kernel provides a fallback. Rebuild or pull the new CUDA binaries to benefit from accelerated lightning indexing.

Analyzed Oct 31, 2026

Upgrading?

Release notes summary

Adds a new CUDA implementation for the lightning indexer operation using generic vector and WMMA kernels, with architectural relaxations and refactoring.

Key Highlights

  • CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel)
  • Relaxed MMA architecture requirements to Turing
  • Added alignment checks for Q and K tensors
  • Template parameters added to avoid duplication of constants

Release Notes (excerpt)

<details open> cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) (#25545) * cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) * chore : remove indentation of #pragma unroll * cuda : remove unnecessary kernel template declarations * cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants.…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories