b9862

ggml-org/llama.cppb9862Jul 3, 2026by github-actions[bot]same day as b9861

What changed since b9861

low risk

CUDA gated_delta_net kernel now writes recurrent state directly to cache, eliminating redundant copy kernels for MTP workloads.

Improvements

  • CUDA GDN kernel writes recurrent state snapshots straight into the recurrent cache, bypassing intermediate tail writes.
  • Detects the gated_delta_net -> view -> cpy pattern and skips the extra ggml_cuda_cpy calls when safe.
  • Removes 4 redundant copy kernels during target decode with MTP draft length 3 (K=4).
  • Speeds up inference for models using Multi-Token Prediction on CUDA backends.

Upgrade notes

This is a CUDA-only performance optimization with no config or API changes; non-CUDA backends are unaffected. Rebuild or pull the new binaries to benefit from the reduced kernel launches in MTP decoding.

Analyzed Oct 30, 2026

Upgrading?

Release notes summary

This release optimizes CUDA kernels by removing redundant copies after gated_delta_net operations. It writes recurrent state snapshots directly into the recurrent cache, reducing kernel calls for MTP draft lengths.

Key Highlights

  • Optimized CUDA GDN kernel to skip intermediate tail writes
  • Reduces 4 extra copy kernels for MTP draft length 3
  • Improves performance for models using Multi-Token Prediction (MTP)

New Features

  • Optimized CUDA recurrent state handling

Release Notes (excerpt)

<details open> Remove redundant CUDA copies after gated_delta_net. (#23940) * Remove redundant CUDA copies after gated_delta_net. Currently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With MTP draft length 3, target decode uses K=4, so that becomes 4 extra ggml_cuda_cpy calls.…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories