b10076

ggml-org/llama.cppb10076Jul 21, 2026by github-actions[bot]1 days after b10075

What changed since b10075

low risk

CUDA backend gains a vectorized int4 get_rows path, cutting get_rows time ~27% on Strix Halo.

New features

  • New k_get_rows_float_vec CUDA kernel handles contiguous same-type (no-cast) get_rows copies 16 bytes at a time.
  • Vectorized path is selected at runtime only when base pointers and row strides are 16-byte aligned and ne00 is a multiple of the vector width.

Improvements

  • Row-invariant work (index load, fast_div_modulo, src/dst row pointers) is hoisted out of the per-element loop in k_get_rows_float.
  • DeltaNet recurrent-state gather on gfx1151 drops from 18.6us to 13.0us, now faster than the Vulkan backend.
  • An occupancy gate keeps small single-row gathers on the block-rich scalar path to avoid regressions.

Bug fixes

  • Resolves the scalar-copy inefficiency reported in issue #25929 for same-type get_rows.

Upgrade notes

This is a drop-in build with no API, config, or model-format changes; only CUDA users benefit from the speedup. Users on non-CUDA backends (including Hexagon, whose CLAMP support landed in the prior build) see no behavioral change. Verify with test-backend-ops GET_ROWS if you maintain a custom CUDA fork.

Analyzed Oct 31, 2026

Upgrading?

Release notes summary

CUDA optimization to vectorize get_rows operation for int4 types.

Key Highlights

  • Vectorize same-type get_rows with int4 copy
  • Performance improvement on Strix Halo (gfx1151) - 27% faster
  • Adds k_get_rows_float_vec for contiguous same-type cases
  • Fixes issue #25929

Release Notes (excerpt)

<details open> CUDA: vectorize same-type get_rows with int4 copy (#25929) k_get_rows_float did a scalar one-element-per-thread copy and recomputed the row-invariant work (index load, fast_div_modulo, src/dst row pointers) for every element. Hoist that out of the per-element loop, and add a vectorized path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the contiguous same-type (no-cast) case. The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories