b9731
ggml-org/llama.cppb9731Jun 19, 2026by github-actions[bot]same day as b9730
What changed since b9730
Server logprobs retrieval now uses partial sorting, cutting top-n token lookup overhead roughly 10x.
Improvements
- get_token_probabilities now uses std::partial_sort to order only the requested top-n tokens instead of sorting the entire vocabulary.
- Benchmarks show top-n logprob retrieval dropping from ~8.5 ms to ~0.7 ms per call at a 128k vocabulary.
- Lower CPU cost for server requests that return logprobs, improving throughput under concurrent load.
Upgrade notes
This is a drop-in server-side performance change with no API, config, or CLI modifications required. Behavior of returned logprobs is unchanged, so simply swap the binary; the gain applies to endpoints requesting top-n logprobs rather than full-vocabulary output.
Analyzed Oct 24, 2026
Upgrading?
Release notes summary
Optimized the server's get_token_probabilities function using partial sorting to significantly reduce overhead.
Key Highlights
- Optimized get_token_probabilities using std::partial_sort
- Performance benchmarks show ~10x speedup for top-n token retrieval
- Reduces processing time from full vocabulary sorting to partial sorting
New Features
- Performance optimization for logprobs
Release Notes (excerpt)
<details open> server : optimize get_token_probabilities (#24796) Use std::partial_sort to order only the requested top-n tokens instead of the full vocabulary logprobs sort: vocab=128000 n_top=0 iters=100 full sort: 8555.6 us/op partial sort: 704.3 us/op Signed-off-by: Adrien Gallouët <angt@huggingface.co> </details> **macOS/iOS:** - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework **Linux:**…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system