b10076
ggml-org/llama.cppb10076Jul 21, 2026by github-actions[bot]1 days after b10075
What changed since b10075
CUDA backend gains a vectorized int4 get_rows path, cutting get_rows time ~27% on Strix Halo.
New features
- New k_get_rows_float_vec CUDA kernel handles contiguous same-type (no-cast) get_rows copies 16 bytes at a time.
- Vectorized path is selected at runtime only when base pointers and row strides are 16-byte aligned and ne00 is a multiple of the vector width.
Improvements
- Row-invariant work (index load, fast_div_modulo, src/dst row pointers) is hoisted out of the per-element loop in k_get_rows_float.
- DeltaNet recurrent-state gather on gfx1151 drops from 18.6us to 13.0us, now faster than the Vulkan backend.
- An occupancy gate keeps small single-row gathers on the block-rich scalar path to avoid regressions.
Bug fixes
- Resolves the scalar-copy inefficiency reported in issue #25929 for same-type get_rows.
Upgrade notes
This is a drop-in build with no API, config, or model-format changes; only CUDA users benefit from the speedup. Users on non-CUDA backends (including Hexagon, whose CLAMP support landed in the prior build) see no behavioral change. Verify with test-backend-ops GET_ROWS if you maintain a custom CUDA fork.
Analyzed Oct 31, 2026
Upgrading?
Release notes summary
CUDA optimization to vectorize get_rows operation for int4 types.
Key Highlights
- Vectorize same-type get_rows with int4 copy
- Performance improvement on Strix Halo (gfx1151) - 27% faster
- Adds k_get_rows_float_vec for contiguous same-type cases
- Fixes issue #25929
Release Notes (excerpt)
<details open> CUDA: vectorize same-type get_rows with int4 copy (#25929) k_get_rows_float did a scalar one-element-per-thread copy and recomputed the row-invariant work (index load, fast_div_modulo, src/dst row pointers) for every element. Hoist that out of the per-element loop, and add a vectorized path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the contiguous same-type (no-cast) case. The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system