v0.4.0

## 🚀  New Features Highlights
- **Prefill/Decode (P/D) Disaggregation Support**: Introduces StormService and RoleSet CRDs to enable fine-grained orchestration of P/D roles, along with routing to unlock disaggregated inference at scale. (#1209, #1226, #1229, #1256, #1258, #1259, #1268, #1280, #1309, #1311, #1354, #1355, #1377, #1399, #1402)
- **KVCache V1 Connector Optimizations**: Delivers a major refactor with v1 Connector integration, CUDA kernel separation from vllm downstream, compact memory layout, connector integration for PrisDB and InfiniStore(/w TCP), tunable block sizes, RDMA auto-detection support and few performance optimizations to boost throughput and deployment density. ( #1174, #1194, #1247, #1274, #1276, #1278, #1286, #1287, #1288, #1295, #1303, #1312, #1318)
- **KV Event Synchronization**: Introduces remote tokenizer support to ensure tokenization consistency between client and server and implements a comprehensive KV cache event synchronization system that shares KV cache state between vLLM instances and aibrix gateway for improved prefix caching efficiency  (#1307, #1328, #1349, #1362) 
- **Multi-Engine Deployment Support**: Adds unified regression test suites and Helm values to support heterogeneous backends including vLLM, SGLang, and Dynamo, enabling flexible model deployment across engines. (#1293, #1319, #1322, #1341, #1346)

## 📊 Feature Enhancements

### 🌐 Gateway Enhancements
- SLO-aware router with profile support (#1192, #1305, #1368) 
- Adds custom inference port and metrics port support (#1140, #1313).
-  Make httproute timeout configurable and checks missing httproute before request start(#1212, #1344).
- Adds metrics server support and adds ready-to-use sample dashboard (#1211).

### ☁️ Control Plane Improvements
- Enhance the CRD existence check and improve webhook support (#1170, #1187).
- Ensure cache sync before starting controller reconcile and resync object on component restarts (#1146, #1219).
- Use worker pool management for periodic metrics update (#1096) 

### 📦 Installation & Tooling & CI
- Adds Helm Chart support with helm standard labels and probes (#1323, #1331, #1343).
- Supports multi-arch (AMD, ARM) Docker builds and refactors release pipelines (#1315, #1317, #1324, #1325).
- Improves kind development workflow and supports port-forward via Makefile, support override IMAGE_TAG and disable docker push workflow in forked repo(#1210, #1274, #1301).

## 🐞 Bug Fixes
- Fixes incorrect request count, out-of-index errors, and race conditions in AIBrix router(#1246, #1262, #1305).
- Fix Prefix cache chained hashing issue and optimize to O(N) via block-hash. (#1218, #1262)
- Fixes completion body parsing and complex content bugs (#1145, #1160).
- Fixes legacy autoscaling annotation misconfigurations (#1173).
- Fixes image replacement issues in Kustomize (#1165).
- Fixes e2e test flakiness with wait.PollUntilContextTimeout (#1214).
- Add read lock for h.histogram (#1147)

## 📚 Documentation Updates
- Adds v0.4.0 new features documentation including P/D disaggregation, multi-engine, KVCache Offloading and SLO routing documentation (#1279, #1285, #1341, #1356, #1368).
- Fixes broken links, typos, and dashboard URLs (#1190, #1193, #1237, #1270, #1271).
- Refactors component design docs into structured architecture folders (#1224, #1236, #1250).
- Refactors local development and quickstart guides (#1193, #1339, #1172).
- Improve installation commands and add more deployment examples  (#1128, #1136, #1230, #1379, #1395)

## New Contributors
* @dittops made their first contribution in https://github.com/vllm-project/aibrix/pull/1128
* @yyzxw made their first contribution in https://github.com/vllm-project/aibrix/pull/1139
* @firebook made their first contribution in https://github.com/vllm-project/aibrix/pull/1145
* @windsonsea made their first contribution in https://github.com/vllm-project/aibrix/pull/1150
* @emmanuel-ferdman made their first contribution in https://github.com/vllm-project/aibrix/pull/1161
* @MondayCha made their first contribution in https://github.com/vllm-project/aibrix/pull/1165
* @learner0810 made their first contribution in https://github.com/vllm-project/aibrix/pull/1170
* @jiahuipaung made their first contribution in https://github.com/vllm-project/aibrix/pull/1172
* @didier-durand made their first contribution in https://github.com/vllm-project/aibrix/pull/1190
* @gcalmettes made their first contribution in https://github.com/vllm-project/aibrix/pull/1193
* @justadogistaken made their first contribution in https://github.com/vllm-project/aibrix/pull/1218
* @ModiCodeCraftsman made their first contribution in https://github.com/vllm-project/aibrix/pull/1217
* @haitwang-cloud made their first contribution in https://github.com/vllm-project/aibrix/pull/1230
* @ae86zhizhi made their first contribution in https://github.com/vllm-project/aibrix/pull/1262
* @nicole-lihui made their first contribution in https://github.com/vllm-project/aibrix/pull/1270
* @omerap12 made their first contribution in https://github.com/vllm-project/aibrix/pull/1282
* @li-rongzhi made their first contribution in https://github.com/vllm-project/aibrix/pull/1285
* @rudeigerc made their first contribution in https://github.com/vllm-project/aibrix/pull/1301
* @Yaegaki1Erika made their first contribution in https://github.com/vllm-project/aibrix/pull/1313
* @elizabetht made their first contribution in https://github.com/vllm-project/aibrix/pull/1339
* @autopear made their first contribution in https://github.com/vllm-project/aibrix/pull/1362
* @Epsilon314 made their first contribution in https://github.com/vllm-project/aibrix/pull/1402

## What's Changed
**Full Changelog**: https://github.com/vllm-project/aibrix/compare/v0.3.0...v0.4.0

* [Docs] Update typo for installation command by @dittops in https://github.com/vllm-project/aibrix/pull/1128
* [Docs] fix: update example yaml ai runtime tag to v0.3.0 by @yyzxw in https://github.com/vllm-project/aibrix/pull/1139
* [Bug]: fix: README.md docs install error by @googs1025 in https://github.com/vllm-project/aibrix/pull/1136
* [Bug] fix: error when parse stop param in completion body by @firebook in https://github.com/vllm-project/aibrix/pull/1145
* Resync model adapters on gateway restart by @dittops in https://github.com/vllm-project/aibrix/pull/1146
* [Docs]add management user link by @yyzxw in https://github.com/vllm-project/aibrix/pull/1141
* [Bug] Add read lock for h.histogram by @runzhen in https://github.com/vllm-project/aibrix/pull/1147
* [Doc] Improve samples/volcano-engine/README.md by @windsonsea in https://github.com/vllm-project/aibrix/pull/1150
* feature: use worker pool management for periodic metrics update by @googs1025 in https://github.com/vllm-project/aibrix/pull/1096
* [Misc] [gpu_optimizer] add namespace info for log by @googs1025 in https://github.com/vllm-project/aibrix/pull/1149
* Add support for custom inference engine port by @varungup90 in https://github.com/vllm-project/aibrix/pull/1140
* Modernize logger interface by @emmanuel-ferdman in https://github.com/vllm-project/aibrix/pull/1161
* Add unit test code coverage by @varungup90 in https://github.com/vllm-project/aibrix/pull/1156
* feat: simplfy router interface by @Xunzhuo in https://github.com/vllm-project/aibrix/pull/1163
* [Bug] Fix image replacements in Kustomize files to support installation by @MondayCha in https://github.com/vllm-project/aibrix/pull/1165
* [Bug]: fix(aibrix kvcache): ObjectPool  by @googs1025 in https://github.com/vllm-project/aibrix/pull/1162
* [Bug]: Fix legacy misconfigurations of autoscaling annotations. by @zhangjyr in https://github.com/vllm-project/aibrix/pull/1173
* Enhance the CRD existence check by @learner0810 in https://github.com/vllm-project/aibrix/pull/1170
* [Misc]: add unit test for aibrix metrics collector by @googs1025 in https://github.com/vllm-project/aibrix/pull/1153
* [Docs] Add vllm-cpu local deployment guide to Quickstart by @jiahuipaung in https://github.com/vllm-project/aibrix/pull/1172
* fix: gateway benchmark info by @Xunzhuo in https://github.com/vllm-project/aibrix/pull/1180
* [Bug] fix: error when parse complex content in completion body by @firebook in https://github.com/vllm-project/aibrix/pull/1160
* Add race condition check in unit-test CI and add test-coverage cmd in Makefile by @varungup90 in https://github.com/vllm-project/aibrix/pull/1169
* Supporting Mooncake Traces in Workload Generator by @happyandslow in https://github.com/vllm-project/aibrix/pull/1182
* Recover Client Implementation by @happyandslow in https://github.com/vllm-project/aibrix/pull/1191
* Docs: fixing various text issues by @didier-durand in https://github.com/vllm-project/aibrix/pull/1190
* [Docs] update development instructions to new make commands by @gcalmettes in https://github.com/vllm-project/aibrix/pull/1193
* feat: make preble configurable and rename by @Xunzhuo in https://github.com/vllm-project/aibrix/pull/1189
* [Misc] Add deepseek-r1 tp8 pp2 example by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1195
* [Misc] Update the latest news in README.md by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1196
* [Feature] Add RDMA auto-detection for kvcache by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1194
* [Tooling]: port-forward support, Makefile changes for easier dev workflow in kind by @Venkat2811 in https://github.com/vllm-project/aibrix/pull/1210
* Multiple fixes and adding workload merging tool by @happyandslow in https://github.com/vllm-project/aibrix/pull/1213
* Add configurable httproute timeout by @varungup90 in https://github.com/vllm-project/aibrix/pull/1212
* [Bug] fix(e2e flaky): replace for-loop with wait.PollUntilContextTimeout in validateAllPodsAreReady by @googs1025 in https://github.com/vllm-project/aibrix/pull/1214
* [Misc] chore: use constant var for gpu_busy_time_ratio metrics by @googs1025 in https://github.com/vllm-project/aibrix/pull/1215
* fix prefix hash incorrect sometimes by @justadogistaken in https://github.com/vllm-project/aibrix/pull/1218
* [FIX]: vtc-basic router constructor config init, enable e2e tests & add benchmark results only by @Venkat2811 in https://github.com/vllm-project/aibrix/pull/1222
* [Doc] Add maintainer guidelines and contributor promotion criteria by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1224
* [PD] Add RoleSet and StormService API skeleton for disaggregation orchestration by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1209
* [Misc]: ensure cache sync before starting controller reconcile by @googs1025 in https://github.com/vllm-project/aibrix/pull/1219
* [FEATURE]: metrics server support for gateway plugins & dashboard by @Venkat2811 in https://github.com/vllm-project/aibrix/pull/1211
* Adding callback patterns for generator client by @happyandslow in https://github.com/vllm-project/aibrix/pull/993
* Add RoleSet and StormService detail spec by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1226
* Add RoleSet and StormService controller implementation by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1229
* Add new test cases for gateway server by @ModiCodeCraftsman in https://github.com/vllm-project/aibrix/pull/1217
* [Refactor] New memory layout for AIBrix KVCache  by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1174
* [Docs]: add advanced kubernetes deployment examples by @haitwang-cloud in https://github.com/vllm-project/aibrix/pull/1230
* [Misc] SLO-aware router with profile support by @zhangjyr in https://github.com/vllm-project/aibrix/pull/1192
* [Misc] Fix storm service rbac issue by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1235
* Support standalone stormservice deployment by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1239
* [Docs]fix: example docs error by @yyzxw in https://github.com/vllm-project/aibrix/pull/1237
* [Docs]refactor: change architecture to stand-alone directories by @yyzxw in https://github.com/vllm-project/aibrix/pull/1236
* [Lint] KVCache uses pre-commit lint by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1243
* [Misc] feature: use kvcache webhook by @googs1025 in https://github.com/vllm-project/aibrix/pull/1187
* [Feature] kvcache cuda kernel by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1247
* Update stormservice controller DefaultRequeueAfter to 15s by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1253
* Set Storm Service default update strategy by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1256
* [docs] Move aibrix component design doc to separate architecture folder by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1250
* Ignore StormService NotFound error during deletion by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1257
* [Misc] Use domain-qualified finalizer name by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1258
* [Bug] Optimize prefix cache hashing to O(N) via block-hash same as vllm by @ae86zhizhi in https://github.com/vllm-project/aibrix/pull/1262
* [Misc]: add ROLE_TEMPLATE_HASH info to container env by @googs1025 in https://github.com/vllm-project/aibrix/pull/1268
* Support /scale sub resource for replica mode by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1259
* [Docs]fix: observability docs dashboard link 404 by @nicole-lihui in https://github.com/vllm-project/aibrix/pull/1270
* [Docs] fix after reorganize incorrect file path by @nicole-lihui in https://github.com/vllm-project/aibrix/pull/1271
* [Doc] KVCache: add section for env vars by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1279
* [Misc] Improve the unit test coverage of stormservice controller by @omerap12 in https://github.com/vllm-project/aibrix/pull/1282
* [Misc] Support role replica index in pod labels by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1280
* [Fix] KVCache: enhance rdma auto-detection by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1276
* [Feature] KVCache: enhance profiling by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1278
* [Fix] KVCache: change cuda kernel's namespace by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1286
* [Feature] KVCache: optimize token list iteration and key building by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1287
* [Feature] KVCache: optimize allocator for compact layout by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1288
* [CI] Support custom IMAGE_TAG to override build tags by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1274
* [Docs] Add documentation for StormService by @li-rongzhi in https://github.com/vllm-project/aibrix/pull/1285
* [Fix] KVCache: fix requirements by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1294
* Improve UT coverage for stormservice controller by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1283
* [Integration] vLLM V1 Connector integration by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1295
* [Misc] extract hashfunc as a field to allow injection by @vie-serendipity in https://github.com/vllm-project/aibrix/pull/1297
* [Misc] Add unit test code coverage of rolesyncer by @vie-serendipity in https://github.com/vllm-project/aibrix/pull/1296
* [Bug] fix incorrect request count by @firebook in https://github.com/vllm-project/aibrix/pull/1246
* [Fix] KVCache: enhance status by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1304
* [CI] Disable docker push images workflow in forked repositories by @rudeigerc in https://github.com/vllm-project/aibrix/pull/1301
* [Docs] Update stormservice docs and link to index page by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1299
* Adding help flag to benchmark script  by @happyandslow in https://github.com/vllm-project/aibrix/pull/1302
* [Feature] KVCache: add Pris connector by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1303
* [feat] Support dynamic metrics port via model.aibrix.ai/metric-port label by @Yaegaki1Erika in https://github.com/vllm-project/aibrix/pull/1313
* [CI] Support multi-arch build in main branch by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1315
* [Misc] enhance headless service sync with update logic by @omerap12 in https://github.com/vllm-project/aibrix/pull/1311
* [Bug] Fix simple queue out of index error in unit test. by @zhangjyr in https://github.com/vllm-project/aibrix/pull/1305
* Fix nil slice issue and add UT coverage for stormservice utils by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1314
* [Misc]: use ctx instead of context by @googs1025 in https://github.com/vllm-project/aibrix/pull/1316
* [Fix] KVCache: InfiniStore connector w/ TCP by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1312
* [CI] Enable multi-arch parallel build by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1317
* [Feature] KVCache: support configurable block size by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1318
* [Feature] Adding raw metrics name conversion in metrics by @happyandslow in https://github.com/vllm-project/aibrix/pull/1293
* [Misc] Add SGLang P/D disaggregation examples by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1319
* [feat] Support generic remote tokenizer by @ae86zhizhi in https://github.com/vllm-project/aibrix/pull/1307
* [feat] Add prefill-decode disaggregation support in aibrix router by @varungup90 in https://github.com/vllm-project/aibrix/pull/1309
* [Feature] Supporting new policies for xLLM  by @happyandslow in https://github.com/vllm-project/aibrix/pull/1322
* [CI] rebuilt kuberay operator with multi-arch support by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1324
* [Fix] KVCache: fix release workflow by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1327
* [CI] Refactor the multi-arch image build in release pipeline by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1325
* [feat] Support Helm Chart by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1323
* [Misc] Support helm chart values for VKE by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1331
* [Misc] Move release test to test/regression folder by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1332
* [Docs] Refactor local development from quick start by @elizabetht in https://github.com/vllm-project/aibrix/pull/1339
* Add initial v0.4.0 regression test yamls by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1337
* feat: Add Helm standard labels and health probes by @omerap12 in https://github.com/vllm-project/aibrix/pull/1343
* Add Qwen3-32b stormservice benchmark manifests by @nwangfw in https://github.com/vllm-project/aibrix/pull/1342
* [Docs] Adding multi engine support documentation by @happyandslow in https://github.com/vllm-project/aibrix/pull/1341
* [Misc] Add vLLM disaggregation samples by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1346
* Switch disaggregation-transfer-backend to mooncake in regression manifests by @nwangfw in https://github.com/vllm-project/aibrix/pull/1348
* Bug fixes for PD disaggregation routing by @varungup90 in https://github.com/vllm-project/aibrix/pull/1354
* Adding v0.4.0 vllm, sglang and dynamo test yamls by @nwangfw in https://github.com/vllm-project/aibrix/pull/1352
* [Docs] KVCache: v0.4.0 release by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1356
* feat: add vLLM remote tokenizer with engine integration by @ae86zhizhi in https://github.com/vllm-project/aibrix/pull/1328
* Add check for missing httproute before request start by @varungup90 in https://github.com/vllm-project/aibrix/pull/1344
* Bug fix streaming in PD disaggregation and add unit tests by @varungup90 in https://github.com/vllm-project/aibrix/pull/1355
* Cut v0.4.0-rc.1 release by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1358
* [Fix] KVCache: fix no-space-left issue of release-build action by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1361
* [bench] Add vLLM disagg_proxy_server.py with xPyD Support by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1360
* [Fix] Correct default Helm values for health check port by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1365
* [Fix] Skip adding common labels in  EnvoyProxy by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1367
* [Docs] Added docs for slo routing under Heterogeneous GPU Inference section by @zhangjyr in https://github.com/vllm-project/aibrix/pull/1368
* [Misc] Update v0.4.0 regression benchmark yamls by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1369
* feat: Add KV cache event synchronization system by @ae86zhizhi in https://github.com/vllm-project/aibrix/pull/1349
* Cut v0.4.0-rc.2 release by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1373
* [Fix] Use dynamic versioning for AIBrix python package by @DwyaneShi in https://github.com/vllm-project/aibrix/pull/1380
* [fix] Use t.Setenv to resolve racing problem by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1382
* [Doc] Add documentation for pd routing by @varungup90 in https://github.com/vllm-project/aibrix/pull/1381
* [Docs] Update the release and kv event docs by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1379
* Make prefill request timeout configurable by @varungup90 in https://github.com/vllm-project/aibrix/pull/1377
* Cut release v0.4.0-rc.3 by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1384
* [fix] Fix logic bug in prefix cache load balancing causing race test failures by @ae86zhizhi in https://github.com/vllm-project/aibrix/pull/1386
* Replace some hard-coded strings with pkg/constants/model.go  by @autopear in https://github.com/vllm-project/aibrix/pull/1362
* [CI] Install poetry-dynamic-versioning plugin in release workflow by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1389
* Cut release v0.4.0-rc.4 by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1390
* [Bug] fix: kvcache_webhook in integration test and webhook config by @googs1025 in https://github.com/vllm-project/aibrix/pull/1392
* Use random as fallback if cache miss in P/D router by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1393
* Revert "Use random as fallback if cache miss in P/D router (#1393)" by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1394
* [Docs] add pd-model deployment example in quickstart doc by @nwangfw in https://github.com/vllm-project/aibrix/pull/1395
* Add @googs1025 as AIBrix maintainer by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1397
* [API] Add PodGroupSize to form a minimum role instance by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1400
* Use random strategy as fallback on cache miss for PD routing by @varungup90 in https://github.com/vllm-project/aibrix/pull/1399
* [Bug] ensure retry not ready roleset by @Epsilon314 in https://github.com/vllm-project/aibrix/pull/1402
* Cut release v0.4.0 by @Jeffwan in https://github.com/vllm-project/aibrix/pull/1398