v4.0.0

# What's new in 4.0.0 (2026-10-10)

## Highlights

Xinference 4.0 brings Xavier to more inference engines, enables heterogeneous prefill/decode deployments across engines and hardware, and lets you reconfigure running models without reloading their weights.

- **Xavier across vLLM, SGLang, and MLX.** Xavier now supports all three engines, extending prefix KV cache sharing across compatible replicas. Reuse previously computed context across requests and replicas to avoid repeated prefill work.

- **Cross-engine and cross-hardware P/D separation.** Run prefill and decode with different inference engines, including vLLM ↔ SGLang and bidirectional NVIDIA ↔ Apple silicon deployments. Choose engines and hardware independently for each role through Xinference’s unified serving interface.

- **Reload engines without reloading model weights.** Opt-in GPU weight caching lets vLLM and SGLang retain weights while rebuilding the inference engine. Adjust supported concurrency, context, memory, and CUDA graph settings while preserving the model UID.

- **Distributed vLLM V1 inference.** Combine tensor and pipeline parallelism across Xinference workers through the xoscar executor, with improved pipeline scheduling and failure handling.

- **One-command installation and service management.** Install and start Xinference on Linux, macOS, and Windows in an isolated Python environment. Optional system services support background operation and automatic startup, with rollback for failed managed service upgrades.

- **OpenAI Responses API support.** Serve Responses-based clients through the new stateless `/v1/responses` endpoint, including streaming and tool calls. Cache-hit token reporting across supported engines makes prompt reuse more visible.

## New features
* feat(image): add jina-ocr-v1 OCR support by @Minamiyama in https://github.com/xorbitsai/inference/pull/5588
* feat(api): add stateless OpenAI Responses API (/v1/responses) by @bluefish-08 in https://github.com/xorbitsai/inference/pull/5590
* feat(tts): support Confucius4-TTS by @Minamiyama in https://github.com/xorbitsai/inference/pull/5591
* feat(logs): add runtime log center by @Minamiyama in https://github.com/xorbitsai/inference/pull/5592
* feat(auth): require globally unique API key names by @m199369309 in https://github.com/xorbitsai/inference/pull/5597
* feat: add native NIXL backend for vLLM PD deployments by @qinxuye in https://github.com/xorbitsai/inference/pull/5604
* feat: correlate audit and runtime logs by request ID by @m199369309 in https://github.com/xorbitsai/inference/pull/5610
* feat: add GPU-first tiered Xavier snapshot storage by @qinxuye in https://github.com/xorbitsai/inference/pull/5615
* feat: enable GPU-first Xavier caching with NIXL transfers by @qinxuye in https://github.com/xorbitsai/inference/pull/5616
* feat(webui): unify runtime and historical log views by @m199369309 in https://github.com/xorbitsai/inference/pull/5635
* FEAT: support SGLang Xavier KV sharing and native GPU PD by @qinxuye in https://github.com/xorbitsai/inference/pull/5631
* feat: retain GPU weights across vLLM and SGLang engine reloads by @qinxuye in https://github.com/xorbitsai/inference/pull/5640
* feat: add MLX Xavier KV cache sharing and PD separation by @qinxuye in https://github.com/xorbitsai/inference/pull/5638
* FEAT: support Xavier P/D across vLLM and SGLang by @qinxuye in https://github.com/xorbitsai/inference/pull/5639
* FEAT: support bidirectional Xavier P/D between NVIDIA and MLX by @qinxuye in https://github.com/xorbitsai/inference/pull/5641
* feat(llm): support Ling-3.0-tiny with vLLM and SGLang by @Minamiyama in https://github.com/xorbitsai/inference/pull/5645
* feat(embedding): add EmbeddingGemma 2 support by @Minamiyama in https://github.com/xorbitsai/inference/pull/5644
* FEAT: report cache hit tokens across vLLM, SGLang and MLX by @qinxuye in https://github.com/xorbitsai/inference/pull/5647
* feat: add cross-platform installation and local service management by @OliverBryant in https://github.com/xorbitsai/inference/pull/5178
* FEAT: Add distribution extension points for API composition and worker admission by @qinxuye in https://github.com/xorbitsai/inference/pull/5650
* feat: add configurable network interface MAC address lookup by @llyycchhee in https://github.com/xorbitsai/inference/pull/5654
## Enhancements
* ENH: Support Qwen3.5 recurrent-state PD handoff by @qinxuye in https://github.com/xorbitsai/inference/pull/5630
* ENH: Complete vLLM V1 pipeline parallel execution with xoscar by @qinxuye in https://github.com/xorbitsai/inference/pull/5637
* perf: batch Xavier KV reads and add transfer diagnostics by @qinxuye in https://github.com/xorbitsai/inference/pull/5606
* perf: batch Xavier KV reads across layers by @qinxuye in https://github.com/xorbitsai/inference/pull/5614
* perf: preserve BF16 bits in Xavier KV transfers by @qinxuye in https://github.com/xorbitsai/inference/pull/5613
* perf: avoid gathering already published Xavier GPU snapshots by @qinxuye in https://github.com/xorbitsai/inference/pull/5620
* perf: reduce Xavier GPU transfer padding with persistent small slabs by @qinxuye in https://github.com/xorbitsai/inference/pull/5621
* perf: batch Xavier KV loads across scheduled requests by @qinxuye in https://github.com/xorbitsai/inference/pull/5622
* PERF: overlap Xavier KV loads with vLLM decoding by @qinxuye in https://github.com/xorbitsai/inference/pull/5623
* PERF: reduce Xavier GPU snapshot eviction overhead by @qinxuye in https://github.com/xorbitsai/inference/pull/5624
* PERF: avoid repeated Xavier snapshot size scans by @qinxuye in https://github.com/xorbitsai/inference/pull/5625
* perf: reduce Xavier cold-prefill snapshot overhead by @qinxuye in https://github.com/xorbitsai/inference/pull/5627
* perf: default Xavier PD to GPU handoff with tiered KV history by @qinxuye in https://github.com/xorbitsai/inference/pull/5628
* PERF: reduce Xavier V1 cache overhead and allow CUDA Graph by @qinxuye in https://github.com/xorbitsai/inference/pull/5643
* REF: extract shared Xavier core and Torch backend by @qinxuye in https://github.com/xorbitsai/inference/pull/5629
## Bug fixes
* fix: address recurring Windows and Metal CI failures by @qinxuye in https://github.com/xorbitsai/inference/pull/5593
* fix(llm): keep text outside Qwen tool call tags as content by @MohammadHijjawi97 in https://github.com/xorbitsai/inference/pull/5595
* fix(core): name the model in the load-retry warning by @David-Wu1119 in https://github.com/xorbitsai/inference/pull/5596
* fix(ui): restore GPU selection for recommended model types by @m199369309 in https://github.com/xorbitsai/inference/pull/5598
* fix(monitoring): map audit request metadata in Elasticsearch by @m199369309 in https://github.com/xorbitsai/inference/pull/5601
* fix: constrain missing host TorchCodec to compatible Torch versions by @qinxuye in https://github.com/xorbitsai/inference/pull/5599
* fix: load parent editable installs in model virtual environments by @qinxuye in https://github.com/xorbitsai/inference/pull/5600
* fix(llm): keep parallel DeepSeek V3.1 tool calls with different arguments by @MohammadHijjawi97 in https://github.com/xorbitsai/inference/pull/5602
* fix(llm): return every DeepSeek-R1 tool call in a tool calls block by @MohammadHijjawi97 in https://github.com/xorbitsai/inference/pull/5603
* fix: use public SentenceTransformer device property by @m199369309 in https://github.com/xorbitsai/inference/pull/5608
* fix: align synchronous RPC structured logging by @m199369309 in https://github.com/xorbitsai/inference/pull/5612
* fix: reuse Xavier actor RPC loop across EngineCore connectors by @qinxuye in https://github.com/xorbitsai/inference/pull/5607
* fix: retry Router Agent bootstrap operations by @m199369309 in https://github.com/xorbitsai/inference/pull/5609
* fix: preserve launch history device semantics by @m199369309 in https://github.com/xorbitsai/inference/pull/5611
* fix(llm): preserve DeepSeek V3.2 tool argument types by @SulimanAbdulrazzaq in https://github.com/xorbitsai/inference/pull/5618
* fix: release request limit when metrics RPC is cancelled by @tuanzirwar in https://github.com/xorbitsai/inference/pull/5626
* fix: validate Xavier KV compatibility before accepting remote hits by @qinxuye in https://github.com/xorbitsai/inference/pull/5619
* fix(llm): keep finish_reason and empty content for tool responses without calls by @MohammadHijjawi97 in https://github.com/xorbitsai/inference/pull/5633
* fix(audit): skip successful metrics scrape events by @m199369309 in https://github.com/xorbitsai/inference/pull/5636
* fix: resolve SGLang JIT cache paths before runtime startup by @qinxuye in https://github.com/xorbitsai/inference/pull/5642
* BUG: make log rotation safe across Windows processes by @Ricardo-M-L in https://github.com/xorbitsai/inference/pull/5303
* fix(api): honour encoding_format=base64 in /v1/embeddings by @MohammadHijjawi97 in https://github.com/xorbitsai/inference/pull/5646
* fix(frontend): remove flags from language selector by @maoyuehui in https://github.com/xorbitsai/inference/pull/5648
* fix(llm): stream text held back as a possible tool call tag by @hpdkhoa in https://github.com/xorbitsai/inference/pull/5651
* FIX: start weight cache watcher without importing model engines by @qinxuye in https://github.com/xorbitsai/inference/pull/5656
* FIX: preflight offline mirror dependencies before release builds by @qinxuye in https://github.com/xorbitsai/inference/pull/5655
## Documentation
* DOC: update new models for v3.5.0 by @qinxuye in https://github.com/xorbitsai/inference/pull/5587
* doc: add v3.5.0 to release notes index and locale catalogs by @martinma51 in https://github.com/xorbitsai/inference/pull/5594
* doc: clarify Xinference Enterprise, Cloud, and Model API offerings by @qinxuye in https://github.com/xorbitsai/inference/pull/5605
* doc: discuss Docker Swarm deployment considerations by @qiulang in https://github.com/xorbitsai/inference/pull/5617
* doc: simplify README quick start by @qinxuye in https://github.com/xorbitsai/inference/pull/5652
* doc: refresh community installation and 4.0 release guides by @qinxuye in https://github.com/xorbitsai/inference/pull/5653
## Others
* TEST: Cover multi-prefill and multi-decode GPU routing by @qinxuye in https://github.com/xorbitsai/inference/pull/5632

## New Contributors
* @MohammadHijjawi97 made their first contribution in https://github.com/xorbitsai/inference/pull/5595
* @SulimanAbdulrazzaq made their first contribution in https://github.com/xorbitsai/inference/pull/5618
* @tuanzirwar made their first contribution in https://github.com/xorbitsai/inference/pull/5626
* @hpdkhoa made their first contribution in https://github.com/xorbitsai/inference/pull/5651

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v3.5.0...v4.0.0