b10355

ggml-org/llama.cppb10355Aug 10, 2026by github-actions[bot]same day as b10354

What changed since b10354

medium risk

Adds multi-output backend sampling with token speculation, enabling sampling on GPU/CPU instead of host-side only.

New features

  • Backend (GPU/CPU) sampling now supports multiple outputs, enabling token speculation during generation.
  • New numeric context parameter to declare the maximum number of outputs per sequence.
  • Sampling can be offloaded to the backend rather than performed entirely on the host.

Improvements

  • Token speculation combined with backend sampling is intended to improve generation throughput.
  • Mask summation is clamped before index conversion to keep sampling numerically stable.
  • Output views no longer reuse memory, avoiding aliasing issues in multi-output paths.
  • CPU and backend sampling distributions were aligned to reduce divergence between execution paths.

Bug fixes

  • Resolved mismatches between CPU and backend sampling results.
  • Fixed test failures on Vulkan introduced by the backend sampling changes.

Upgrade notes

This is a single-commit step from b10354, so the only substantive delta is the multi-output backend sampling work. If you use backend sampling, set the new max-outputs context parameter and re-validate that GPU and CPU sampling produce equivalent distributions for your workloads. Expect small numerical differences in sampled tokens versus the previous host-only path.

Analyzed Nov 1, 2026

Upgrading?

Release notes summary

Implements support for multi-output backend sampling, allowing the sampling process to be performed on the backend (GPU/CPU) to enable token speculation and improve performance.

Key Highlights

  • Enable backend sampling with token speculation.
  • Support for multi-output generation.
  • Added numeric context parameter for max outputs.

Release Notes (excerpt)

<details open> llama : support multi-output backend sampling (#25532) * Enable backend sampling with token speculation * Clamp the mask sum before converting it into the sampled index * Add a numeric context parameter declaring the maximum outputs one sequence * More fixes * Don't reuse memory for output views. * Match dist between CPU and GPU * Fix CPU and backend sampling mismatches * Simpify some of the changes * Fix tests on Vulkan * More test fixes * Rebase changes…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories