b9908

ggml-org/llama.cppb9908Jul 8, 2026by github-actions[bot]same day as b9907

What changed since b9907

medium risk

b9908 makes the server's --cache-ram a hard limit and allows eviction of the last cached entry, replacing b9907's header-only build fix.

Breaking changes

  • Server prompt cache no longer guarantees at least one retained entry; token-limit cleanup can now evict the final cached entry, altering cache-hit behavior for existing deployments.

New features

  • Server enforces --cache-ram as a strict upper bound rather than a soft target.

Improvements

  • Cache states that alone exceed the RAM limit are skipped instead of being saved.
  • Old entries are evicted before insertion so the cache no longer temporarily overshoots RAM or token limits.
  • Token-limit cleanup can remove the last remaining entry, freeing memory previously pinned by the one-entry floor.

Bug fixes

  • Fixes cache growth beyond configured RAM/token limits caused by saving new state before evicting old entries.

Upgrade notes

No config or migration steps are required, but review your --cache-ram value since it is now a hard cap and oversized single states will simply not be cached. Workloads relying on a guaranteed warm cache entry may see more recomputation after eviction; raise the limit if that matters.

Analyzed Oct 30, 2026

Upgrading?

Release notes summary

Enforces strict RAM limits for prompt caching in the server API to prevent memory overruns and improves eviction logic by allowing the last entry to be removed.

Key Highlights

  • Enforce strict RAM limit for prompt cache in server
  • Skip saving state if it exceeds the RAM limit
  • Evict old entries to make new ones fit
  • Token-limit cleanup can now evict the last remaining cache entry

Release Notes (excerpt)

<details open> server: enforce prompt cache RAM limit (#25070) Before this commit, --cache-ram was not a hard limit: - The cache always kept at least one entry, even if that entry exceeded the RAM/token limits. - Old entries were only evicted for the RAM/token limits after saving the new one, which could cause the cache to temporarily exceed the RAM/token limits even if individual entries were below the limit. Now, ensure that the RAM limit is strict with these changes:…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories