b10242

ggml-org/llama.cppb10242Aug 3, 2026by github-actions[bot]same day as b10241

What changed since b10241

medium risk

CUDA backend gains a penalties sampler (frequency, presence, top-k) with new tests and stricter penalty validation.

New features

  • Backend sampler implementation for penalty sampling in the CUDA backend, covering frequency and presence penalties.
  • Top-k penalty support added to backend sampling.
  • New test utilities (accept_prompt, unique_prompt_tokens, compare_penalties_logits) and a backend penalties sampling test suite.
  • Penalty sampler now inherits from llama_sampler_backend and accepts backend input handling.

Improvements

  • Masked logits are preserved as -Inf instead of producing NaN, improving numerical stability of backend sampling.
  • penalty_last_n now defaults from the model context when unspecified, and negative values for penalty_last_n and n_prev are clamped.
  • Repeat penalty values are validated to be finite and greater than zero.
  • Test helpers were refactored into shared functions for broader coverage.

Bug fixes

  • Eliminated NaN generation in backend penalty sampling when logits are masked out.
  • Guarded against invalid repeat-penalty values that previously could be passed through unchecked.

Upgrade notes

If you relied on an unset penalty_last_n, it now defaults from the model context rather than a fixed value, which can subtly change penalty behavior. Repeat penalty values must be finite and greater than zero; invalid configurations are now rejected, so audit custom sampler setups. The new backend sampler is CUDA-specific and opt-in via backend sampling paths.

Analyzed Nov 1, 2026

Upgrading?

Release notes summary

Implemented a backend sampler for handling penalty sampling logic (frequency, presence, and top-k) within the CUDA backend, accompanied by extensive test coverage.

Key Highlights

  • Added backend sampler for penalty handling (frequency and presence)
  • Added support for top-k penalties in backend sampling
  • Enhanced numerical stability by preserving masked logits as -Inf
  • Added comprehensive tests for backend penalties sampling

Release Notes (excerpt)

<details open> CUDA: Add backend sampler for penalties sampler (#25262) * sampling: enhance penalty handling in common_sampler_init - Set default value for penalty_last_n based on model context if not specified. - Ensure penalty_last_n and n_prev are non-negative. - Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties.…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories