v0.6.0

# Release Notes

## 📌 Release Summary

- Improvements to **gateway routing and traffic management** for LLM inference services.
- Enhancements to **distributed serving and orchestration**, enabling more flexible multi-node deployments.
- Updates to **batch request processing and OpenAI-compatible APIs**.
- Better **metrics and observability** support.
- Various **bug fixes, stability improvements, and CI/CD updates**.

Overall: This release focuses on improving routing, scalability, and operational stability for running vLLM-based LLM services in Kubernetes.


## 🚀 New Feature Highlights

### **Expanded OpenAI-Compatible API Surface**

This release significantly expands the OpenAI-compatible API capabilities supported by AIBrix.

* **Audio & Classification APIs**: Added support for OpenAI-style audio endpoints (/v1/audio/transcriptions, /v1/audio/translations) along with the new /v1/classify inference API. (#1859, #1905)
* **Image Generation Endpoint**: Introduced OpenAI-compatible generation APIs for images and videos (/v1/images/generations, /v1/video/generations), allowing multimodal generation workloads to run through the same gateway. (#1867)
* **Rerank Model Support**: Added support for rerank models via the /v1/rerank endpoint, enabling improved ranking and retrieval pipelines. (#1837)

These additions further strengthen AIBrix as a unified gateway for diverse AI workloads.

---

### **Advanced Gateway & Routing Capabilities**

Major improvements were made to the gateway routing layer to enable smarter and more flexible inference traffic management.

* **Session Affinity Routing**: Add plugin-based session affinity routing strategies for sticky workloads. (#1751, #1823)
* **Advanced Routing Filters**: Support external header filters for advanced routing scenarios. (#1804)
* **Custom HTTPRoute Paths**: Allow custom path configuration through annotations. (#1841)
* **Multi-Deployment Router Support**: Enable router configuration across multiple deployment targets. (#1835)
* **Least-Request Routing Strategy**: Introduce distributed DP API server routing using least-request algorithms. (#1866)

Together these improvements provide better flexibility when deploying large-scale inference workloads.

---

### **Prefill/Decode (P/D) Disaggregation Improvements**

This release continues to expand support for Prefill/Decode disaggregated inference architectures.

* **PD + KVCache Routing Compatibility**: Enable routing that supports both KVCache and P/D disaggregation within a single runtime image. (#1781)
* **Prefix Cache Optimizations**: Introduce asynchronous prefix cache updates and shared indexers across routing strategies. (#1914, #1939)
* **Combined Routing Strategy with P/D**: Enables intelligent routing across both PD-optimized pods (prefill/decode disaggregated) and combined pods (non-PD) within the same deployment, allowing mixed serving strategies and improved resource utilization. (#1911)

These changes improve the scalability and flexibility of large LLM inference clusters.

---

### **StormService & Control Plane Enhancements**

The StormService controller received multiple upgrades to improve reliability and visibility.

* **Role Revision Tracking**: Add per-role revision tracking to StormService for improved upgrade visibility. (#1731)
* **Role Status Aggregation**: Implement role-level status aggregation and improvements to reconcile logic. (#1761, #1767)
* **Periodic Reconciliation**: Introduce periodic reconciliation for ModelAdapter resources. (#1824)
* **Dynamic Discovery Provider Updates**: Enable discovery providers to dynamically update runtime state. (#1908)
* **Improved Scheduling Integration**: Enhance PodGroup and scheduling strategy handling. (#1889, #1795)

These changes strengthen the orchestration layer used for distributed inference deployments.

---

### **LoRA & Model Adapter Lifecycle Management**

Model adapter and LoRA workflows are improved for reliability and runtime control.

* **Dynamic LoRA Load/Unload for SGLang**: Support runtime loading and unloading of LoRA adapters. (#1853)
* **Artifact Preparation Improvements**: Delegate artifact preparation to LoRA downloader components. (#1898)
* **Improved Adapter Failure Handling**: Transition LoRA resources to `Failed` state when pods are not recoverable. (#1884)

These updates improve multi-adapter inference reliability in production.

---

### **KVCache Framework Improvements**

The AIBrix KVCache framework continues to evolve with new optimizations.

* **Block-First KVCache Layout**: Introduce block-first KVCache layout support. (#1947)
* **CUDA Kernel Improvements**: Add padding token support in KVCache CUDA kernels. (#1958)
* **New KV Connector for PD Reuse**: Add `aibrix_pd_reuse_connector` to support combined PD reuse workflows. (#1852)

These improvements further optimize memory efficiency and performance of KVCache-based inference.

---

## 📊 Observability & Metrics

Major improvements were introduced to monitoring and observability across gateway and runtime layers.

* **Gateway Metrics Support**: Introduce metrics collection directly from the gateway layer. (#1907, #1922)
* **Inference Request Metrics**: Add granular metrics tracking inference request behavior. (#1926)
* **Prometheus Integration Enhancements**:
  * Prometheus authentication support via Kubernetes secrets (#1949)
  * Query queueing support for Prometheus requests (#1964)
* **Routing & Cache Metrics Updates**: Improve routing and cache metrics definitions and naming. (#1968)

A new **SGLang gateway metrics dashboard** was also added. (#1959)

---

## 📦 Installation, Deployment & Platform Support

Deployment workflows and platform support continue to improve.

* **Docker Compose Installation**: Simplify standalone installation with improved docker-compose configuration. (#1871, #1878)
* **Gateway Plugin Standalone Mode**: Allow gateway plugin to run without Kubernetes. (#1873)
* **Envoy Sidecar Support**: Add support for running Envoy as a sidecar alongside the gateway-plugin. (#1931)
* **Flexible Docker Builds**: Improve Docker builds to support multiple platforms and architectures. (#1942)
* **Custom Registry Support**: Support registry addresses containing port values. (#1919)

Additional samples were also added for **Ascend hardware deployments**. (#1935)

---

## 📚 Documentation Improvements

A large set of documentation updates and guides were added.

Highlights include:

* Envoy AI Gateway integration guide (#1733)
* Session affinity routing documentation (#1823)
* Prefill/Decode disaggregation examples (#1811)
* LoRA adapter documentation updates (#1813)
* AIBrix container images for vLLM/SGLang (#1792)
* AWS Trainium2 / Neuron support documentation (#1894)
* Prometheus gateway configuration documentation (#1954)

Numerous README and documentation improvements were also included.

---

## 🐞 Critical Bug Fixes

A number of stability and correctness fixes were implemented across routing, metrics, and runtime systems.

Key fixes include:

* Resolve RDMA issues affecting SGLang and vLLM in P/D disaggregation setups. (#1783)
* Fix Redis authentication handling in Helm charts. (#1806)
* Prevent divide-by-zero errors in APA autoscaling logic. (#1879)
* Fix envoy extension policy paths and gateway service configuration issues. (#1921, #1932)
* Resolve inconsistent label cardinality panic when emitting metrics. (#1977)
* Fix CGO build failures caused by mismatched builder/runtime environments. (#1925)

Additional fixes improve controller stability, metrics correctness, and routing behavior.

---

## 🧪 Testing & Developer Experience

Testing coverage and development workflows were expanded.

* Add E2E tests for Batch API using OpenAI client. (#1743)
* Improve client metrics testing. (#1727)
* Add fake-client based PodGroup unit tests. (#1790)
* Improve benchmark script dependencies and testing utilities. (#1909)

---


## What's Changed
* [Misc] Added test for client metrics by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1727
* [Docs] v0.5.0 KVCache docs and samples by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1745
* [Bug]: fix infinistore(rdma) exists/delete to avoid TCP interleaving by @sherlockkenan in https://github.com/vllm-project/aibrix/pull/1748
* [Chore] fix links in bug report template by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1750
* [Test] Add E2E test for batch API using open AI client. by @zhangjyr in https://github.com/vllm-project/aibrix/pull/1743
* fix(api): correct API comment of RayClusterReplicaSetStatus.Replicas by @zhixian82 in https://github.com/vllm-project/aibrix/pull/1754
* docs(router): clarify parameter and return descriptions by @googs1025 in https://github.com/vllm-project/aibrix/pull/1755
* [Docs]: feature: envoy ai gateway integration by @googs1025 in https://github.com/vllm-project/aibrix/pull/1733
* [feat] Support per-role revision tracking in Stormservice by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1731
* [Docs] added v0.5.0 entry in README by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1757
* [Docs] seperate section for talks in README by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1758
* [Misc] upgrade sidecar webhook image from v0.4.0 to v0.5.0 by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1762
* [Feat] Implement role status aggregation for StormService by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1761
* [fix] Consider revision in role status aggregation by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1767
* [feat] Add affinity values for chart by @my-git9 in https://github.com/vllm-project/aibrix/pull/1763
* [Misc] fix sequence for dev-install-in-kind make target by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1764
* refactor(metrics): use Subsystem for aibrix metrics instead of embedding in Name by @googs1025 in https://github.com/vllm-project/aibrix/pull/1756
* [Bug]: add miss webhook config in helm chart by @googs1025 in https://github.com/vllm-project/aibrix/pull/1776
* [Integration] auto detect torch version in dockerfiles by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1782
* Fix sglang/vllm NIXL RDMA issues with PD disaggregation by @dczhu in https://github.com/vllm-project/aibrix/pull/1783
* [Bug]: add validation in stormservice webhook by @googs1025 in https://github.com/vllm-project/aibrix/pull/1778
* Support both AIBrix KVCache and PD disaggregation routing in one image by @dczhu in https://github.com/vllm-project/aibrix/pull/1781
* Update Aibrix version in installation instructions by @googs1025 in https://github.com/vllm-project/aibrix/pull/1791
* fix(controller): shorten resource name by @zhixian82 in https://github.com/vllm-project/aibrix/pull/1777
* [Docs] Add documentation for AIBrix vLLM/SGLang container images by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1792
* add Podgroup util ut cases using fake-client by @DengHom in https://github.com/vllm-project/aibrix/pull/1790
* fix(controller): correct volcano podgroup annotation key by @zhixian82 in https://github.com/vllm-project/aibrix/pull/1795
* [Feat] KVCache: change Pris to PrisKV by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1807
* [Bug] use redis auth in helm chart if redis pwd is enabled by @sceneryback in https://github.com/vllm-project/aibrix/pull/1806
* [Misc] Shorten hash length from 10 to 6 chars to mitigate 63-char name limit by @Deepam02 in https://github.com/vllm-project/aibrix/pull/1789
* Fix envoy, gateway, and quickstart pd-model images by @dczhu in https://github.com/vllm-project/aibrix/pull/1812
* [Docs]: docs(vllm, sglang): update P/D disaggregation examples Docs by @googs1025 in https://github.com/vllm-project/aibrix/pull/1811
* [Feature]: Add external-filter in Header for advanced routing by @rayne-Li in https://github.com/vllm-project/aibrix/pull/1804
* [Docs] updated lora adapter doc for replica feature redesign by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1813
* feature: add simple session affinity plugins in gateway plugin by @googs1025 in https://github.com/vllm-project/aibrix/pull/1751
* [Docs] Fix typo at samples/disaggregation/vllm/README.md by @n0gu-furiosa in https://github.com/vllm-project/aibrix/pull/1821
* docs(routing): add documentation for session-affinity routing strategy by @googs1025 in https://github.com/vllm-project/aibrix/pull/1823
* Add vLLM Lora testing scripts by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1829
* [Misc] Improve Lora edge cases test coverage by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1830
* [Docs] Add a line break to resolve the issue of code not rendering. by @rayne-Li in https://github.com/vllm-project/aibrix/pull/1833
* [helm]: Include default helpers, rename using fullname by @cabrinha in https://github.com/vllm-project/aibrix/pull/1828
* [Feat] Implement periodic reconcile for ModelAdapter by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1824
* [feat]: support rerank models by @sanmuny in https://github.com/vllm-project/aibrix/pull/1837
* [Feature] add custom path for httpRoute by annotation (#1840) by @rayne-Li in https://github.com/vllm-project/aibrix/pull/1841
* [bugfix] ReqBody receive stream=true when calling /v1/completion by @DengHom in https://github.com/vllm-project/aibrix/pull/1850
* [Feat] Support LoRA loading/unloading for SGLang by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1853
* feat:router based on multi deployment configs by @erictanjn in https://github.com/vllm-project/aibrix/pull/1835
* Feat: Audio endpoint support by @dittops in https://github.com/vllm-project/aibrix/pull/1859
* feat(docker-compose): polish simplified installation setup by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1871
* Support OpenAI compatible image generation endpoint by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1867
* [Feature] support LWS in model-router controller (#1839) by @rayne-Li in https://github.com/vllm-project/aibrix/pull/1851
* [Bugfix] change errMsg for GVK and use ctx.Background instead (#1839) by @rayne-Li in https://github.com/vllm-project/aibrix/pull/1872
* [MISC]: move constants to types.go by @omerap12 in https://github.com/vllm-project/aibrix/pull/1875
* feat: Support gateway plugin running without Kubernetes by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1873
* feat: Add OpenAI-compatible and vLLM-specific endpoints in mocked app by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1877
* fix: standalone docker-compose configuration issues by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1878
* [bug] Fix the simulator import issue by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1883
* [Feature] kv_transfer/kv_connector: Add aibrix_pd_reuse_connector to support PD + reuse by @dczhu in https://github.com/vllm-project/aibrix/pull/1852
* Fix msgpack-based events decoder/encoder from map type to list type by @autopear in https://github.com/vllm-project/aibrix/pull/1848
* [fix] Transition Lora to Failed state when no pods are retriable by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1884
* [Misc] follow up changes for LoRA load/unload in SGLang by @nurali-techie in https://github.com/vllm-project/aibrix/pull/1863
* [Feat]: update PodGroup when RoleSet.SchedulingStrategy changed by @fungaren in https://github.com/vllm-project/aibrix/pull/1889
* Update metrics to align with latest version of vllm by @varungup90 in https://github.com/vllm-project/aibrix/pull/1892
* Replace json package with sonic for better performance by @varungup90 in https://github.com/vllm-project/aibrix/pull/1891
* Fix GetPercentile method by @varungup90 in https://github.com/vllm-project/aibrix/pull/1893
* [Bug] Add roleset-index to annotations by @sceneryback in https://github.com/vllm-project/aibrix/pull/1901
* [feat] add distribute-dp api server least_request route by @paranoidRick in https://github.com/vllm-project/aibrix/pull/1866
* feature: add v1/classify endpoint support by @dittops in https://github.com/vllm-project/aibrix/pull/1905
* [Feat] Add metrics support in the gateway by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1907
* [Misc] Add missing dependency for benchmark script by @pbillaut in https://github.com/vllm-project/aibrix/pull/1909
* [Misc] Set leader election namespace according to release namespace by @pbillaut in https://github.com/vllm-project/aibrix/pull/1910
* Enable combined strategy routing along with PD by @varungup90 in https://github.com/vllm-project/aibrix/pull/1911
* refactor metrics collector by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1912
* [refactor] only update readyPodsMap of prefixCacheRouter in the imbalanced situation by @DengHom in https://github.com/vllm-project/aibrix/pull/1906
* Add async prefix cache update in PD disaggregation by @varungup90 in https://github.com/vllm-project/aibrix/pull/1914
* Feat: Discovery Provider support dynamic update by @penfree in https://github.com/vllm-project/aibrix/pull/1908
* [Bug] Refactor downloader and artifact_service to be non-blocking by @xieus in https://github.com/vllm-project/aibrix/pull/1895
* [Bug] Prevent div by zero in APA by @alpe in https://github.com/vllm-project/aibrix/pull/1879
* [Fix] Add sidecar injection webhook to helm chart by @pbillaut in https://github.com/vllm-project/aibrix/pull/1916
* fix: envoy extension policy path by @pbillaut in https://github.com/vllm-project/aibrix/pull/1921
* Feat: delegate artifacts preparation to lora downloader by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1898
* fix: resolve CGO build failure by aligning builder and runtime env by @varungup90 in https://github.com/vllm-project/aibrix/pull/1925
* feat: add gateway metrics by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1922
* Add metrics to track inference request granular details by @varungup90 in https://github.com/vllm-project/aibrix/pull/1926
* Add gateway configuration for tuning concurrent streams by @varungup90 in https://github.com/vllm-project/aibrix/pull/1918
* feat: emit error metric on failure to read metrics from prometheus or LLM engine by @varungup90 in https://github.com/vllm-project/aibrix/pull/1927
* fix: add missed docker copy cmd from PR #1925 by @varungup90 in https://github.com/vllm-project/aibrix/pull/1928
* fix: allow envoy gateway service customization by @pbillaut in https://github.com/vllm-project/aibrix/pull/1932
* add support for custom registry addr contains ':' by @rayne-Li in https://github.com/vllm-project/aibrix/pull/1919
* fix:validation logic for combined and PD separated deployment by @erictanjn in https://github.com/vllm-project/aibrix/pull/1933
* fix: set default http route and extension policy values explicitly by @pbillaut in https://github.com/vllm-project/aibrix/pull/1937
* Samples for deploying aibrix on ascend by @liangdong1201 in https://github.com/vllm-project/aibrix/pull/1935
* fix: bump up setup-envtest version by @varungup90 in https://github.com/vllm-project/aibrix/pull/1941
* fix: use a shared prefix cache indexer across routing-strategies by @varungup90 in https://github.com/vllm-project/aibrix/pull/1939
* Feat: make the gateway docker flexible to platform by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1942
* fix lock amd64 build by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1945
* [Bugfix] add APA support to stormService without podGroupSize (#1913) by @rayne-Li in https://github.com/vllm-project/aibrix/pull/1917
* revert back the original build way by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1946
* Feat: load prom auth from secret by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1949
* [Docs][API] Add AWS Neuron/Trainium2 support for disaggregated inference by @yahavb in https://github.com/vllm-project/aibrix/pull/1894
* feat: add feature to support envoy as a sidecar to gateway-plugin by @varungup90 in https://github.com/vllm-project/aibrix/pull/1931
* [fix] Add missing model name in unit test by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1950
* [Feature] KVCache support block-first layout by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1947
* [Doc] add gateway prometheus config by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1954
* [feat] Add apps/ structure with chat web portal by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1956
* [Feature] kvcache cuda kernel supports padding tokens by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1958
* feat: add queue for prometheus query by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1964
* Feat: Support vllm new kvevent format by @penfree in https://github.com/vllm-project/aibrix/pull/1962
* feat: add sglang gateway metrics and gateway dashboard by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1959
* fix: update metric name for routing-algorithms by @varungup90 in https://github.com/vllm-project/aibrix/pull/1968
* fix: add shared path for the downloaded artifacts by @scarlet25151 in https://github.com/vllm-project/aibrix/pull/1972
* fix: panic inconsistent label cardinality in emit metrics by @varungup90 in https://github.com/vllm-project/aibrix/pull/1977

## New Contributors
* @sherlockkenan made their first contribution in https://github.com/vllm-project/aibrix/pull/1748
* @dczhu made their first contribution in https://github.com/vllm-project/aibrix/pull/1783
* @sceneryback made their first contribution in https://github.com/vllm-project/aibrix/pull/1806
* @Deepam02 made their first contribution in https://github.com/vllm-project/aibrix/pull/1789
* @rayne-Li made their first contribution in https://github.com/vllm-project/aibrix/pull/1804
* @n0gu-furiosa made their first contribution in https://github.com/vllm-project/aibrix/pull/1821
* @cabrinha made their first contribution in https://github.com/vllm-project/aibrix/pull/1828
* @sanmuny made their first contribution in https://github.com/vllm-project/aibrix/pull/1837
* @erictanjn made their first contribution in https://github.com/vllm-project/aibrix/pull/1835
* @fungaren made their first contribution in https://github.com/vllm-project/aibrix/pull/1889
* @paranoidRick made their first contribution in https://github.com/vllm-project/aibrix/pull/1866
* @pbillaut made their first contribution in https://github.com/vllm-project/aibrix/pull/1909
* @alpe made their first contribution in https://github.com/vllm-project/aibrix/pull/1879
* @liangdong1201 made their first contribution in https://github.com/vllm-project/aibrix/pull/1935
* @yahavb made their first contribution in https://github.com/vllm-project/aibrix/pull/1894

**Full Changelog**: https://github.com/vllm-project/aibrix/compare/v0.5.0...v0.6.0