b9731

ggml-org/llama.cppb9731Jun 19, 2026by github-actions[bot]same day as b9730

What changed since b9730

low risk

Server logprobs retrieval now uses partial sorting, cutting top-n token lookup overhead roughly 10x.

Improvements

  • get_token_probabilities now uses std::partial_sort to order only the requested top-n tokens instead of sorting the entire vocabulary.
  • Benchmarks show top-n logprob retrieval dropping from ~8.5 ms to ~0.7 ms per call at a 128k vocabulary.
  • Lower CPU cost for server requests that return logprobs, improving throughput under concurrent load.

Upgrade notes

This is a drop-in server-side performance change with no API, config, or CLI modifications required. Behavior of returned logprobs is unchanged, so simply swap the binary; the gain applies to endpoints requesting top-n logprobs rather than full-vocabulary output.

Analyzed Oct 24, 2026

Upgrading?

Release notes summary

Optimized the server's get_token_probabilities function using partial sorting to significantly reduce overhead.

Key Highlights

  • Optimized get_token_probabilities using std::partial_sort
  • Performance benchmarks show ~10x speedup for top-n token retrieval
  • Reduces processing time from full vocabulary sorting to partial sorting

New Features

  • Performance optimization for logprobs

Release Notes (excerpt)

<details open> server : optimize get_token_probabilities (#24796) Use std::partial_sort to order only the requested top-n tokens instead of the full vocabulary logprobs sort: vocab=128000 n_top=0 iters=100 full sort: 8555.6 us/op partial sort: 704.3 us/op Signed-off-by: Adrien Gallouët <angt@huggingface.co> </details> **macOS/iOS:** - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework **Linux:**…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories