b10089

ggml-org/llama.cppb10089Jul 22, 2026by github-actions[bot]same day as b10088

What changed since b10088

medium risk

CUDA backend now handles k-quants, i-quants, and mxfp4 in GET_ROWS, avoiding host fallback for embedding lookups.

New features

  • GET_ROWS in the CUDA backend now supports k-quants from q2_K through q6_K.
  • GET_ROWS in the CUDA backend now supports all nine i-quant types.
  • GET_ROWS in the CUDA backend now supports mxfp4.
  • Quantized embedding lookups can now run fully on-device instead of falling back to the host.

Improvements

  • Super-block dequantizers were factored out of convert.cu into shared device helpers in dequantize.cuh for reuse.
  • A new k_get_rows_kq kernel reuses existing thread layouts (32 threads for q4_K, 64 for other k-quants) for efficient dequantization.
  • supports_op now gates the new quantized GET_ROWS path on ne0 being a multiple of QK_K, covering iq4_nl and mxfp4 sub-block cases.

Upgrade notes

This is a drop-in CUDA performance improvement with no config or API changes required. Users running quantized GGUF models (e.g. Q4_K_M with q6_K token embeddings) on CUDA should see faster token generation since embedding lookups no longer copy the full matrix to the host. Rebuild or pull the new CUDA binaries to pick up the change.

Analyzed Oct 31, 2026

Upgrading?

Release notes summary

Significantly optimizes the CUDA backend by implementing support for k-quants, i-quants, and mxfp4 in the GET_ROWS operation, enabling efficient device-side embedding lookups.

Key Highlights

  • Added k-quant support (q2_K to q6_K) to GET_ROWS in CUDA
  • Added i-quant support (nine types) to GET_ROWS in CUDA
  • Added mxfp4 support to GET_ROWS in CUDA
  • Refactored super-block dequantizers into shared device functions for code reuse

Release Notes (excerpt)

<details open> cuda: GET_ROWS quants (#25962) * cuda: add k-quant support to GET_ROWS Device-side embedding lookups require GET_ROWS to handle the k-quants used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without it the backend rejects the op and the scheduler falls back to the host, copying the full embedding matrix back on every token in single-device graphs. Factor the super-block dequantizers out of the dequantize_block kernels…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories