b10429
ggml-org/llama.cppb10429Aug 14, 2026by github-actions[bot]same day as b10428
What changed since b10428
Server can now serve /metrics and /slots while llama_decode() is running, enabling live inference monitoring.
New features
- The HTTP server exposes /metrics and /slots during active llama_decode() calls instead of blocking until generation finishes.
- Monitoring clients can poll slot state and metrics in real time while a request is being processed.
Improvements
- llama_decode() is now invoked through a yield_to_queue path in the server worker, letting the queue service other endpoints mid-inference.
- Multimodal chunk processing (process_mtmd_chunk) follows the same yielding behavior for consistent responsiveness.
Upgrade notes
No configuration or API changes are required; the new behavior is automatic for server users. If you scrape /metrics or /slots, expect responses during generation rather than only between requests, which may change scrape timing assumptions. The internal worker/queue refactor is not a public API change, but validate custom server integrations against the new concurrency behavior.
Analyzed Oct 24, 2026
Upgrading?
Release notes summary
The server now allows accessing `/metrics` and `/slots` endpoints during `llama_decode()` execution, enabling real-time monitoring.
Key Highlights
- Server: Allow accessing /metrics and /slots during llama_decode()
- Enables monitoring of inference progress
- Standard builds for all supported platforms
Release Notes (excerpt)
<details open> server: allow accessing /metrics and /slots during llama_decode() (#27041) * server_queue::worker * call llama_decode inside yield_to_queue * also handle process_mtmd_chunk * clean up * nits * rm test </details> **Website:** - <https://llama.app> **macOS/iOS:** - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework **Linux:** - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU)…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system