b10355
ggml-org/llama.cppb10355Aug 10, 2026by github-actions[bot]same day as b10354
What changed since b10354
Adds multi-output backend sampling with token speculation, enabling sampling on GPU/CPU instead of host-side only.
New features
- Backend (GPU/CPU) sampling now supports multiple outputs, enabling token speculation during generation.
- New numeric context parameter to declare the maximum number of outputs per sequence.
- Sampling can be offloaded to the backend rather than performed entirely on the host.
Improvements
- Token speculation combined with backend sampling is intended to improve generation throughput.
- Mask summation is clamped before index conversion to keep sampling numerically stable.
- Output views no longer reuse memory, avoiding aliasing issues in multi-output paths.
- CPU and backend sampling distributions were aligned to reduce divergence between execution paths.
Bug fixes
- Resolved mismatches between CPU and backend sampling results.
- Fixed test failures on Vulkan introduced by the backend sampling changes.
Upgrade notes
This is a single-commit step from b10354, so the only substantive delta is the multi-output backend sampling work. If you use backend sampling, set the new max-outputs context parameter and re-validate that GPU and CPU sampling produce equivalent distributions for your workloads. Expect small numerical differences in sampled tokens versus the previous host-only path.
Analyzed Nov 1, 2026
Upgrading?
Release notes summary
Implements support for multi-output backend sampling, allowing the sampling process to be performed on the backend (GPU/CPU) to enable token speculation and improve performance.
Key Highlights
- Enable backend sampling with token speculation.
- Support for multi-output generation.
- Added numeric context parameter for max outputs.
Release Notes (excerpt)
<details open> llama : support multi-output backend sampling (#25532) * Enable backend sampling with token speculation * Clamp the mask sum before converting it into the sampled index * Add a numeric context parameter declaring the maximum outputs one sequence * More fixes * Don't reuse memory for output views. * Match dist between CPU and GPU * Fix CPU and backend sampling mismatches * Simpify some of the changes * Fix tests on Vulkan * More test fixes * Rebase changes…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system