b10313

ggml-org/llama.cppb10313Aug 7, 2026by github-actions[bot]same day as b10312

What changed since b10312

medium risk

Server router gains an LRU scheduler for model eviction and request queue management.

New features

  • LRU-based scheduler added to the server router to manage model eviction and waiting requests.
  • Request coalescing is handled when a request leaves the waiting queue.
  • Streaming request handling added for the new scheduler path.

Improvements

  • Router scheduling logic refactored around a dedicated LRU scheduler component.
  • Test coverage added for the router scheduler behavior.

Bug fixes

  • Fixed stream case handling in the router scheduler.

Upgrade notes

This is a single-commit step over b10312, so the only behavioral change is in the server router's eviction/scheduling path. No config or CLI changes are mentioned; if you rely on router model-loading order or eviction timing, re-validate under load after upgrading.

Analyzed Oct 30, 2026

Upgrading?

Release notes summary

This release adds an LRU (Least Recently Used) scheduler to the server's router component. This allows for better management of model evictions and waiting queues.

Key Highlights

  • Added LRU scheduler to server router (#26572)
  • Handles coalescing of requests leaving the waiting queue
  • Added tests and fixed stream case handling

Release Notes (excerpt)

<details open> server: (router) add LRU scheduler (#26572) * add lru_sched * handle coalescing (req leaves waiting queue) * add tests * fix stream case * address review comments </details> **Website:** - <https://llama.app> **macOS/iOS:** - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework **Linux:** - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan)…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories