Starstruk Logo MarkStarstruk

v2.8.0

xorbitsai/inferencev2.8.0May 9, 2026by XprobeBot

AI Summary

This release focuses on observability through comprehensive Prometheus metrics and Grafana dashboard integration. It also improves system stability with graceful OOM handling, worker auto-recovery, and vLLM compatibility fixes.

Key Highlights

  • Comprehensive Prometheus metrics for cluster and worker observability
  • Grafana Dashboard monitoring integration
  • Graceful OOM handling and worker auto-recovery
  • Modular vLLM post-install patch framework
  • Models: GLM-4.6 support

New Features

  • TCP keepalive and reverse-channel probe
  • Comprehensive Prometheus metrics
  • Graceful OOM handling and worker auto-recovery
  • Modular vLLM post-install patch framework
  • Grafana Dashboard monitoring integration
  • GLM-4.6 model support

Full Release Notes

# What's new in 2.8.0 (2026-05-09)
These are the changes in inference v2.8.0.
## New features
* feat: TCP keepalive, reverse-channel probe, list_models cache, .pth atomic write, launch exemption by @m199369309 in https://github.com/xorbitsai/inference/pull/4845
* feat: comprehensive Prometheus metrics for cluster, worker, and model observability by @m199369309 in https://github.com/xorbitsai/inference/pull/4857
* feat: graceful OOM handling, worker model auto-recovery, and heartbeat log enhancement by @m199369309 in https://github.com/xorbitsai/inference/pull/4859
* feat: integrate Grafana Dashboard monitoring into Web UI with Prometheus metrics enhancements by @m199369309 in https://github.com/xorbitsai/inference/pull/4868
* feat: add modular vLLM post-install patch framework with hybrid KV cache fix by @m199369309 in https://github.com/xorbitsai/inference/pull/4879
* feat: add Prometheus alert rules for cluster monitoring by @m199369309 in https://github.com/xorbitsai/inference/pull/4891
* feat: move dashboard to monitor/ and update Grafana panels by @m199369309 in https://github.com/xorbitsai/inference/pull/4892
## Enhancements
* ENH: update models JSON [embedding] by @XprobeBot in https://github.com/xorbitsai/inference/pull/4867
* ENH: update models JSON [llm] by @XprobeBot in https://github.com/xorbitsai/inference/pull/4871
* ENH: update model "GLM-4.6" JSON by @amumu96 in https://github.com/xorbitsai/inference/pull/4878
* bld: avoid libc6 upgrade during arm64 Docker build by @zwt-1234 in https://github.com/xorbitsai/inference/pull/4885
* bld: Restrict non-main release tags to admins by @qinxuye in https://github.com/xorbitsai/inference/pull/4887
* REF: refactor too parser check logic & fix qwen3.5 & qwen3.6 & gemma-4 reasoning parser by @llyycchhee in https://github.com/xorbitsai/inference/pull/4866
## Bug fixes
* fix: resolve Mixed Content blocking for Launch Web UI behind HTTPS proxy by @m199369309 in https://github.com/xorbitsai/inference/pull/4856
* fix(vllm): support for v0.19.0 by @llyycchhee in https://github.com/xorbitsai/inference/pull/4862
* fix: resolve host/model-venv transformers version conflict for embedding/rerank engines by @m199369309 in https://github.com/xorbitsai/inference/pull/4864
* fix: correct PromQL operator precedence in Grafana dashboard panels by @m199369309 in https://github.com/xorbitsai/inference/pull/4873
* fix: Fix vLLM `/v1/completions` TypeError in Qwen3.5 by @la1ty in https://github.com/xorbitsai/inference/pull/4874
* fix: improve Grafana dashboard panel layout, PromQL expressions, and i18n by @m199369309 in https://github.com/xorbitsai/inference/pull/4876
* fix: Qwen3.5 tool, ERROR Can't parse single qwen tool call output by @bleakie in https://github.com/xorbitsai/inference/pull/4870
* fix: differentiate jina-embeddings v3/v4 task parameter handling by @m199369309 in https://github.com/xorbitsai/inference/pull/4881
* fix: Set default enable-thinking option in command line to False by @la1ty in https://github.com/xorbitsai/inference/pull/4875
* fix: upgrade TTFT metric to Histogram with stream label and add non-stream TTFT recording by @m199369309 in https://github.com/xorbitsai/inference/pull/4890
## Documentation
* DOC: add v2.7.0 release notes by @qinxuye in https://github.com/xorbitsai/inference/pull/4855

## New Contributors
* @bleakie made their first contribution in https://github.com/xorbitsai/inference/pull/4870

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v2.7.0...v2.8.0