b10429

ggml-org/llama.cppb10429Aug 14, 2026by github-actions[bot]same day as b10428

What changed since b10428

medium risk

Server can now serve /metrics and /slots while llama_decode() is running, enabling live inference monitoring.

New features

  • The HTTP server exposes /metrics and /slots during active llama_decode() calls instead of blocking until generation finishes.
  • Monitoring clients can poll slot state and metrics in real time while a request is being processed.

Improvements

  • llama_decode() is now invoked through a yield_to_queue path in the server worker, letting the queue service other endpoints mid-inference.
  • Multimodal chunk processing (process_mtmd_chunk) follows the same yielding behavior for consistent responsiveness.

Upgrade notes

No configuration or API changes are required; the new behavior is automatic for server users. If you scrape /metrics or /slots, expect responses during generation rather than only between requests, which may change scrape timing assumptions. The internal worker/queue refactor is not a public API change, but validate custom server integrations against the new concurrency behavior.

Analyzed Oct 24, 2026

Upgrading?

Release notes summary

The server now allows accessing `/metrics` and `/slots` endpoints during `llama_decode()` execution, enabling real-time monitoring.

Key Highlights

  • Server: Allow accessing /metrics and /slots during llama_decode()
  • Enables monitoring of inference progress
  • Standard builds for all supported platforms

Release Notes (excerpt)

<details open> server: allow accessing /metrics and /slots during llama_decode() (#27041) * server_queue::worker * call llama_decode inside yield_to_queue * also handle process_mtmd_chunk * clean up * nits * rm test </details> **Website:** - <https://llama.app> **macOS/iOS:** - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework **Linux:** - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU)…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories