Starstruk Logo MarkStarstruk

v2.10.0

a16z-infra/JungleGymv2.10.0Jun 5, 2026by XprobeBot

AI Summary

This release significantly enhances the inference service's security and observability with OIDC authentication, audit logging, and rate limiting, while introducing UI improvements and GPU memory monitoring.

Key Highlights

  • JSON logging format and stdout redirection
  • OIDC/Keycloak Single Sign-On (SSO) authentication support
  • IP/Key ban and rate limiting for brute-force protection
  • Comprehensive audit logging system
  • Per-model GPU memory usage metrics

New Features

  • JSON logging system
  • OIDC/Keycloak SSO authentication
  • Audit logging system
  • IP/Key ban and rate limiting
  • API key description field and ban status display
  • Security audit panels and filebeat configurations
  • UI menu reorganization, fetchWrapper auth, and i18n updates
  • Per-model GPU memory usage metrics
  • Persist launch model configuration history server-side
  • Three-level download progress logging
  • Update model 'qwen3.6' JSON
  • Update model 'ChatTTS' JSON
  • Update models JSON [embedding]

Full Release Notes

# What's new in 2.10.0 (2026-06-05)
These are the changes in inference v2.10.0.
## New features
* feat(logging): enhance logging system with JSON format and stdout redirect by @m199369309 in https://github.com/xorbitsai/inference/pull/4947
* feat(auth): add OIDC/Keycloak SSO authentication support by @m199369309 in https://github.com/xorbitsai/inference/pull/4948
* feat(audit): add comprehensive audit logging system by @m199369309 in https://github.com/xorbitsai/inference/pull/4951
* feat(security): add IP/Key ban and rate limiting for brute-force protection by @m199369309 in https://github.com/xorbitsai/inference/pull/4949
* feat(apikey): add description field and ban status display by @m199369309 in https://github.com/xorbitsai/inference/pull/4952
* feat(monitor): add security audit panels and filebeat configurations by @m199369309 in https://github.com/xorbitsai/inference/pull/4953
* feat(ui): menu reorganization, fetchWrapper auth, and i18n updates by @m199369309 in https://github.com/xorbitsai/inference/pull/4954
* feat(monitor): add per-model GPU memory usage metrics by @m199369309 in https://github.com/xorbitsai/inference/pull/4965
* feat(monitor): update Grafana dashboards with GPU memory panels by @m199369309 in https://github.com/xorbitsai/inference/pull/4969
* feat: persist launch model configuration history server-side by @m199369309 in https://github.com/xorbitsai/inference/pull/4972
* FEAT: [UI] update sidebar, login logo and favicon by @yiboyasss in https://github.com/xorbitsai/inference/pull/4978
* feat:  new ui (register json view, formInstance, launch model list, cache/env …) by @maoyuehui in https://github.com/xorbitsai/inference/pull/4966
* feat(logging): add three-level download progress logging by @m199369309 in https://github.com/xorbitsai/inference/pull/4989
* feat(ui): allow editing API key name and description in edit dialog by @m199369309 in https://github.com/xorbitsai/inference/pull/4991
## Enhancements
* ENH: update model "qwen3.6" JSON by @llyycchhee in https://github.com/xorbitsai/inference/pull/4945
* ENH: update model "ChatTTS" JSON by @llyycchhee in https://github.com/xorbitsai/inference/pull/4961
* ENH: update models JSON [embedding] by @XprobeBot in https://github.com/xorbitsai/inference/pull/4971
* ENH: update model "qwen3.6" JSON by @llyycchhee in https://github.com/xorbitsai/inference/pull/4994
## Bug fixes
* fix(vllm): set quantization="fp8" when model_format is fp8 by @m199369309 in https://github.com/xorbitsai/inference/pull/4959
* fix(auth): return specific error messages for expired/disabled API keys by @m199369309 in https://github.com/xorbitsai/inference/pull/4963
* fix(monitor): periodic refresh for security gauges and ban remaining API by @m199369309 in https://github.com/xorbitsai/inference/pull/4964
* BUG: fix jina-embeddings-v2-base-zh deployment dependencies by @m199369309 in https://github.com/xorbitsai/inference/pull/4970
* fix(monitor): capture vLLM/SGLang GPU workers via deferred PID tattoo by @m199369309 in https://github.com/xorbitsai/inference/pull/4977
* fix(vllm): remove best_of for v0.21.0 by @llyycchhee in https://github.com/xorbitsai/inference/pull/4979
* bug: Adapt vLLM LoRA request path parameter by @amumu96 in https://github.com/xorbitsai/inference/pull/4980
* fix(logging): strip all CSI escapes and route flush() through sampling by @m199369309 in https://github.com/xorbitsai/inference/pull/4983
* fix(vllm): read json_schema from `schema_` so guided decoding applies by @m199369309 in https://github.com/xorbitsai/inference/pull/4985
* bug: fix llama.cpp streaming tool call edge cases by @qinxuye in https://github.com/xorbitsai/inference/pull/4988
* fix: Fix GPU info probe on GB10 / DGX Spark (NVML v2 memory-info fallback) by @tbraun96 in https://github.com/xorbitsai/inference/pull/4990
* fix(worker): limit concurrent model launches with semaphore to prevent heartbeat timeouts by @m199369309 in https://github.com/xorbitsai/inference/pull/4992
* fix(venv): allow user override by @llyycchhee in https://github.com/xorbitsai/inference/pull/4993
* fix(venv): evaluate CUDA version markers dynamically by @llyycchhee in https://github.com/xorbitsai/inference/pull/4958
## Documentation
* DOC: add v2.9.0 release notes by @qinxuye in https://github.com/xorbitsai/inference/pull/4941
* doc: Add AI agent guidance for the project by @qinxuye in https://github.com/xorbitsai/inference/pull/4920
* doc: remove WeChat QR links from docs site by @qinxuye in https://github.com/xorbitsai/inference/pull/4967
## Others
* chattts: set weights_only=True for torch.load speaker embedding by @tonghuaroot in https://github.com/xorbitsai/inference/pull/4956

## New Contributors
* @tonghuaroot made their first contribution in https://github.com/xorbitsai/inference/pull/4956
* @tbraun96 made their first contribution in https://github.com/xorbitsai/inference/pull/4990

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v2.9.0...v2.10.0