v0.6.4

## Highlights
* Significant progress in V1 engine core refactor (#9826, #10135, #10288, #10211, #10225, #10228, #10268, #9954, #10272, #9971, #10224, #10166, #9289, #10058, #9888, #9972, #10059, #9945, #9679, #9871, #10227, #10245, #9629, #10097, #10203, #10148). You can checkout more details regarding the design and plan ahead in our recent [meetup slides](https://docs.google.com/presentation/d/1e3CxQBV3JsfGp30SwyvS3eM_tW-ghOhJ9PAJGK6KR54/edit#slide=id.g31455c8bc1e_2_130)
* Signficant progress in `torch.compile` support. Many models now support torch compile with TorchInductor. You can checkout our [meetup slides](https://docs.google.com/presentation/d/1e3CxQBV3JsfGp30SwyvS3eM_tW-ghOhJ9PAJGK6KR54/edit#slide=id.g31455c8bc1e_0_443) for more details. (#9775, #9614, #9639, #9641, #9876, #9946, #9589, #9896, #9637, #9300, #9947, #9138, #9715, #9866, #9632, #9858, #9889)

### Model Support
* New LLMs and VLMs: Idefics3 (#9767), H2OVL-Mississippi (#9747), Qwen2-Audio (#9248), Pixtral models in the HF Transformers format (#9036), FalconMamba (#9325), Florence-2 language backbone (#9555)
* New encoder-decoder embedding models: BERT (#9056), RoBERTa & XLM-RoBERTa (#9387)
* Expanded task support: Llama embeddings (#9806), Math-Shepherd (Mistral reward modeling) (#9697), Qwen2 classification (#9704), Qwen2 embeddings (#10184), VLM2Vec (Phi-3-Vision embeddings) (#9303), E5-V (LLaVA-NeXT embeddings) (#9576), Qwen2-VL embeddings (#9944)
    * Add user-configurable `--task` parameter for models that support both generation and embedding (#9424)
    * Chat-based Embeddings API (#9759)
* Tool calling parser for Granite 3.0 (#9027), Jamba (#9154), granite-20b-functioncalling (#8339)
* LoRA support for Granite 3.0 MoE (#9673), Idefics3 (#10281), Llama embeddings (#10071), Qwen (#9622), Qwen2-VL (#10022)
* BNB quantization support for Idefics3 (#10310), Mllama (#9720), Qwen2 (#9467, #9574), MiniCPMV (#9891)
* Unified multi-modal processor for VLM (#10040, #10044)
* Simplify model interface (#9933, #10237, #9938, #9958, #10007, #9978, #9983, #10205)

### Hardware Support
* Gaudi: Add Intel Gaudi (HPU) inference backend (#6143)
* CPU: Add embedding models support for CPU backend (#10193)
* TPU: Correctly profile peak memory usage & Upgrade PyTorch XLA (#9438)
* Triton: Add Triton implementation for scaled_mm_triton to support fp8 and int8 SmoothQuant, symmetric case (#9857)

### Performance
* Combine chunked prefill with speculative decoding (#9291)
* `fused_moe` Performance Improvement (#9384)

### Engine Core
* Override HF `config.json` via CLI (#5836)
* Add goodput metric support (#9338)
* Move parallel sampling out from vllm core, paving way for V1 engine (#9302)
* Add stateless process group for easier integration with RLHF and disaggregated prefill (#10216, #10072)

### Others
* Improvements to the pull request experience with DCO, mergify, stale bot, etc. (#9436, #9512, #9513, #9259, #10082, #10285, #9803)
* Dropped support for Python 3.8 (#10038, #8464)
* Basic Integration Test For TPU (#9968)
* Document the class hierarchy in vLLM (#10240), explain the integration with Hugging Face (#10173).
* Benchmark throughput now supports image input (#9851)



## What's Changed
* [TPU] Fix TPU SMEM OOM by Pallas paged attention kernel by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9350
* [Frontend] merge beam search implementations by @LunrEclipse in https://github.com/vllm-project/vllm/pull/9296
* [Model] Make llama3.2 support multiple and interleaved images by @xiangxu-google in https://github.com/vllm-project/vllm/pull/9095
* [Bugfix] Clean up some cruft in mamba.py by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9343
* [Frontend] Clarify model_type error messages by @stevegrubb in https://github.com/vllm-project/vllm/pull/9345
* [Doc] Fix code formatting in spec_decode.rst by @mgoin in https://github.com/vllm-project/vllm/pull/9348
* [Bugfix] Update InternVL input mapper to support image embeds by @hhzhang16 in https://github.com/vllm-project/vllm/pull/9351
* [BugFix] Fix chat API continuous usage stats by @njhill in https://github.com/vllm-project/vllm/pull/9357
* pass ignore_eos parameter to all benchmark_serving calls by @gracehonv in https://github.com/vllm-project/vllm/pull/9349
* [Misc] Directly use compressed-tensors for checkpoint definitions by @mgoin in https://github.com/vllm-project/vllm/pull/8909
* [Bugfix] Fix vLLM UsageInfo and logprobs None AssertionError with empty token_ids by @CatherineSue in https://github.com/vllm-project/vllm/pull/9034
* [Bugfix][CI/Build] Fix CUDA 11.8 Build by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/9386
* [Bugfix] Molmo text-only input bug fix by @mrsalehi in https://github.com/vllm-project/vllm/pull/9397
* [Misc] Standardize RoPE handling for Qwen2-VL by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9250
* [Model] VLM2Vec, the first multimodal embedding model in vLLM by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9303
* [CI/Build] Test VLM embeddings by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9406
* [Core] Rename input data types by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/8688
* [Misc] Consolidate example usage of OpenAI client for multimodal models by @ywang96 in https://github.com/vllm-project/vllm/pull/9412
* [Model] Support SDPA attention for Molmo vision backbone by @Isotr0py in https://github.com/vllm-project/vllm/pull/9410
* Support mistral interleaved attn by @patrickvonplaten in https://github.com/vllm-project/vllm/pull/9414
* [Kernel][Model] Improve continuous batching for Jamba and Mamba by @mzusman in https://github.com/vllm-project/vllm/pull/9189
* [Model][Bugfix] Add FATReLU activation and support for openbmb/MiniCPM-S-1B-sft by @streaver91 in https://github.com/vllm-project/vllm/pull/9396
* [Performance][Spec Decode] Optimize ngram lookup performance by @LiuXiaoxuanPKU in https://github.com/vllm-project/vllm/pull/9333
* [CI/Build] mypy: Resolve some errors from checking vllm/engine by @russellb in https://github.com/vllm-project/vllm/pull/9267
* [Bugfix][Kernel] Prevent integer overflow in fp8 dynamic per-token quantize kernel by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9425
* [BugFix] [Kernel] Fix GPU SEGV occurring in int8 kernels by @rasmith in https://github.com/vllm-project/vllm/pull/9391
* Add notes on the use of Slack by @terrytangyuan in https://github.com/vllm-project/vllm/pull/9442
* [Kernel] Add Exllama as a backend for compressed-tensors  by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/9395
* [Misc] Print stack trace using `logger.exception` by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9461
* [misc] CUDA Time Layerwise Profiler by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/8337
* [Bugfix] Allow prefill of assistant response when using `mistral_common` by @sasha0552 in https://github.com/vllm-project/vllm/pull/9446
* [TPU] Call torch._sync(param) during weight loading by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9437
* [Hardware][CPU] compressed-tensor INT8 W8A8 AZP support  by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/9344
* [Core] Deprecating block manager v1 and make block manager v2 default by @KuntaiDu in https://github.com/vllm-project/vllm/pull/8704
* [CI/Build] remove .github from .dockerignore, add dirty repo check by @dtrifiro in https://github.com/vllm-project/vllm/pull/9375
* [Misc] Remove commit id file by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9470
* [torch.compile] Fine-grained CustomOp enabling mechanism by @ProExpertProg in https://github.com/vllm-project/vllm/pull/9300
* [Bugfix] Fix support for dimension like integers and ScalarType by @bnellnm in https://github.com/vllm-project/vllm/pull/9299
* [Bugfix] Add random_seed to sample_hf_requests in benchmark_serving script by @wukaixingxp in https://github.com/vllm-project/vllm/pull/9013
* [Bugfix] Print warnings related to `mistral_common` tokenizer only once by @sasha0552 in https://github.com/vllm-project/vllm/pull/9468
* [Hardwware][Neuron] Simplify model load for transformers-neuronx library by @sssrijan-amazon in https://github.com/vllm-project/vllm/pull/9380
* Support `BERTModel` (first `encoder-only` embedding model) by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/9056
* [BugFix] Stop silent failures on compressed-tensors parsing by @dsikka in https://github.com/vllm-project/vllm/pull/9381
* [Bugfix][Core] Use torch.cuda.memory_stats() to profile peak memory usage by @joerunde in https://github.com/vllm-project/vllm/pull/9352
* [Qwen2.5] Support bnb quant for Qwen2.5 by @blueyo0 in https://github.com/vllm-project/vllm/pull/9467
* [CI/Build] Use commit hash references for github actions by @russellb in https://github.com/vllm-project/vllm/pull/9430
* [BugFix] Typing fixes to RequestOutput.prompt and beam search by @njhill in https://github.com/vllm-project/vllm/pull/9473
* [Frontend][Feature] Add jamba tool parser by @tomeras91 in https://github.com/vllm-project/vllm/pull/9154
* [BugFix] Fix and simplify completion API usage streaming by @njhill in https://github.com/vllm-project/vllm/pull/9475
* [CI/Build] Fix lint errors in mistral tokenizer by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9504
* [Bugfix] Fix offline_inference_with_prefix.py by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9505
* [Misc] benchmark: Add option to set max concurrency by @russellb in https://github.com/vllm-project/vllm/pull/9390
* [Model] Add user-configurable task for models that support both generation and embedding by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9424
* [CI/Build] Add error matching config for mypy by @russellb in https://github.com/vllm-project/vllm/pull/9512
* [Model] Support Pixtral models in the HF Transformers format by @mgoin in https://github.com/vllm-project/vllm/pull/9036
* [MISC] Add lora requests to metrics by @coolkp in https://github.com/vllm-project/vllm/pull/9477
* [MISC] Consolidate cleanup() and refactor offline_inference_with_prefix.py by @comaniac in https://github.com/vllm-project/vllm/pull/9510
* [Kernel] Add env variable to force flashinfer backend to enable tensor cores by @tdoublep in https://github.com/vllm-project/vllm/pull/9497
* [Bugfix] Fix offline mode when using `mistral_common` by @sasha0552 in https://github.com/vllm-project/vllm/pull/9457
* :bug: fix torch memory profiling by @joerunde in https://github.com/vllm-project/vllm/pull/9516
* [Frontend] Avoid creating guided decoding LogitsProcessor unnecessarily by @njhill in https://github.com/vllm-project/vllm/pull/9521
* [Doc] update gpu-memory-utilization flag docs by @joerunde in https://github.com/vllm-project/vllm/pull/9507
* [CI/Build] Add error matching for ruff output by @russellb in https://github.com/vllm-project/vllm/pull/9513
* [CI/Build] Configure matcher for actionlint workflow by @russellb in https://github.com/vllm-project/vllm/pull/9511
* [Frontend] Support simpler image input format by @yue-anyscale in https://github.com/vllm-project/vllm/pull/9478
* [Bugfix] Fix missing task for speculative decoding by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9524
* [Model][Pixtral] Optimizations for input_processor_for_pixtral_hf by @mgoin in https://github.com/vllm-project/vllm/pull/9514
* [Bugfix] Pass json-schema to GuidedDecodingParams and make test stronger by @heheda12345 in https://github.com/vllm-project/vllm/pull/9530
* [Model][Pixtral] Use memory_efficient_attention for PixtralHFVision by @mgoin in https://github.com/vllm-project/vllm/pull/9520
* [Kernel] Support sliding window in flash attention backend by @heheda12345 in https://github.com/vllm-project/vllm/pull/9403
* [Frontend][Misc] Goodput metric support by @Imss27 in https://github.com/vllm-project/vllm/pull/9338
* [CI/Build] Split up decoder-only LM tests by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9488
* [Doc] Consistent naming of attention backends by @tdoublep in https://github.com/vllm-project/vllm/pull/9498
* [Model] FalconMamba Support by @dhiaEddineRhaiem in https://github.com/vllm-project/vllm/pull/9325
* [Bugfix][Misc]: fix graph capture for decoder by @yudian0504 in https://github.com/vllm-project/vllm/pull/9549
* [BugFix] Use correct python3 binary in Docker.ppc64le entrypoint by @varad-ahirwadkar in https://github.com/vllm-project/vllm/pull/9492
* [Model][Bugfix] Fix batching with multi-image in PixtralHF by @mgoin in https://github.com/vllm-project/vllm/pull/9518
* [Frontend] Reduce frequency of client cancellation checking by @njhill in https://github.com/vllm-project/vllm/pull/7959
* [doc] fix format by @youkaichao in https://github.com/vllm-project/vllm/pull/9562
* [BugFix] Update draft model TP size check to allow matching target TP size by @njhill in https://github.com/vllm-project/vllm/pull/9394
* [Frontend] Don't log duplicate error stacktrace for every request in the batch by @wallashss in https://github.com/vllm-project/vllm/pull/9023
* [CI] Make format checker error message more user-friendly by using emoji by @KuntaiDu in https://github.com/vllm-project/vllm/pull/9564
* :bug: Fixup more test failures from memory profiling by @joerunde in https://github.com/vllm-project/vllm/pull/9563
* [core] move parallel sampling out from vllm core by @youkaichao in https://github.com/vllm-project/vllm/pull/9302
* [Bugfix]: serialize config instances by value when using --trust-remote-code by @tjohnson31415 in https://github.com/vllm-project/vllm/pull/6751
* [CI/Build] Remove unnecessary `fork_new_process` by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9484
* [Bugfix][OpenVINO] fix_dockerfile_openvino by @ngrozae in https://github.com/vllm-project/vllm/pull/9552
* [Bugfix]: phi.py get rope_theta from config file by @Falko1 in https://github.com/vllm-project/vllm/pull/9503
* [CI/Build] Replaced some models on tests for smaller ones by @wallashss in https://github.com/vllm-project/vllm/pull/9570
* [Core] Remove evictor_v1 by @KuntaiDu in https://github.com/vllm-project/vllm/pull/9572
* [Doc] Use shell code-blocks and fix section headers by @rafvasq in https://github.com/vllm-project/vllm/pull/9508
* support TP in qwen2 bnb by @chenqianfzh in https://github.com/vllm-project/vllm/pull/9574
* [Hardware][CPU] using current_platform.is_cpu by @wangshuai09 in https://github.com/vllm-project/vllm/pull/9536
* [V1] Implement vLLM V1 [1/N] by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9289
* [CI/Build][LoRA] Temporarily fix long context failure issue by @jeejeelee in https://github.com/vllm-project/vllm/pull/9579
* [Neuron] [Bugfix] Fix neuron startup by @xendo in https://github.com/vllm-project/vllm/pull/9374
* [Model][VLM] Initialize support for Mono-InternVL model by @Isotr0py in https://github.com/vllm-project/vllm/pull/9528
* [Bugfix] Eagle: change config name for fc bias by @gopalsarda in https://github.com/vllm-project/vllm/pull/9580
* [Hardware][Intel CPU][DOC] Update docs for CPU backend by @zhouyuan in https://github.com/vllm-project/vllm/pull/6212
* [Frontend] Support custom request_id from request by @guoyuhong in https://github.com/vllm-project/vllm/pull/9550
* [BugFix] Prevent exporting duplicate OpenTelemetry spans by @ronensc in https://github.com/vllm-project/vllm/pull/9017
* [torch.compile] auto infer dynamic_arg_dims from type annotation by @youkaichao in https://github.com/vllm-project/vllm/pull/9589
* [Bugfix] fix detokenizer shallow copy by @aurickq in https://github.com/vllm-project/vllm/pull/5919
* [Misc] Make benchmarks use EngineArgs by @JArnoldAMD in https://github.com/vllm-project/vllm/pull/9529
* [Bugfix] Fix spurious "No compiled cutlass_scaled_mm ..." for W8A8 on Turing by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/9487
* [BugFix] Fix metrics error for --num-scheduler-steps > 1 by @yuleil in https://github.com/vllm-project/vllm/pull/8234
* [Doc]: Update tensorizer docs to include vllm[tensorizer] by @sethkimmel3 in https://github.com/vllm-project/vllm/pull/7889
* [Bugfix] Generate exactly input_len tokens in benchmark_throughput by @heheda12345 in https://github.com/vllm-project/vllm/pull/9592
* [Misc] Add an env var VLLM_LOGGING_PREFIX, if set, it will be prepend to all logging messages by @sfc-gh-zhwang in https://github.com/vllm-project/vllm/pull/9590
* [Model] Support E5-V by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9576
* [Build] Fix `FetchContent` multiple build issue by @ProExpertProg in https://github.com/vllm-project/vllm/pull/9596
* [Hardware][XPU] using current_platform.is_xpu by @MengqingCao in https://github.com/vllm-project/vllm/pull/9605
* [Model] Initialize Florence-2 language backbone support by @Isotr0py in https://github.com/vllm-project/vllm/pull/9555
* [VLM] Post-layernorm override and quant config in vision encoder by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9217
* [Model] Add min_pixels / max_pixels to Qwen2VL as mm_processor_kwargs by @alex-jw-brooks in https://github.com/vllm-project/vllm/pull/9612
* [Bugfix] Fix `_init_vision_model` in NVLM_D model by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9611
* [misc] comment to avoid future confusion about baichuan by @youkaichao in https://github.com/vllm-project/vllm/pull/9620
* [Bugfix] Fix divide by zero when serving Mamba models by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9617
* [Misc] Separate total and output tokens in benchmark_throughput.py by @mgoin in https://github.com/vllm-project/vllm/pull/8914
* [torch.compile] Adding torch compile annotations to some models by @CRZbulabula in https://github.com/vllm-project/vllm/pull/9614
* [Frontend] Enable Online Multi-image Support for MLlama by @alex-jw-brooks in https://github.com/vllm-project/vllm/pull/9393
* [Model] Add Qwen2-Audio model support by @faychu in https://github.com/vllm-project/vllm/pull/9248
* [CI/Build] Add bot to close stale issues and PRs by @russellb in https://github.com/vllm-project/vllm/pull/9436
* [Bugfix][Model] Fix Mllama SDPA illegal memory access for batched multi-image by @mgoin in https://github.com/vllm-project/vllm/pull/9626
* [Bugfix] Use "vision_model" prefix for MllamaVisionModel by @mgoin in https://github.com/vllm-project/vllm/pull/9628
* [Bugfix]: Make chat content text allow type content by @vrdn-23 in https://github.com/vllm-project/vllm/pull/9358
* [XPU] avoid triton import for xpu by @yma11 in https://github.com/vllm-project/vllm/pull/9440
* [Bugfix] Fix PP for ChatGLM and Molmo, and weight loading for Qwen2.5-Math-RM by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9422
* [V1][Bugfix] Clean up requests when aborted by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9629
* [core] simplify seq group code by @youkaichao in https://github.com/vllm-project/vllm/pull/9569
* [torch.compile] Adding torch compile annotations to some models by @CRZbulabula in https://github.com/vllm-project/vllm/pull/9639
* [Kernel] add kernel for FATReLU by @jeejeelee in https://github.com/vllm-project/vllm/pull/9610
* [torch.compile] expanding support and fix allgather compilation by @CRZbulabula in https://github.com/vllm-project/vllm/pull/9637
* [Doc] Move additional tips/notes to the top by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9647
* [Bugfix]Disable the post_norm layer of the vision encoder for LLaVA models by @litianjian in https://github.com/vllm-project/vllm/pull/9653
* Increase operation per run limit for "Close inactive issues and PRs" workflow by @hmellor in https://github.com/vllm-project/vllm/pull/9661
* [torch.compile] Adding torch compile annotations to some models by @CRZbulabula in https://github.com/vllm-project/vllm/pull/9641
* [CI/Build] Fix VLM test failures when using transformers v4.46 by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9666
* [Model] Compute Llava Next Max Tokens / Dummy Data From Gridpoints by @alex-jw-brooks in https://github.com/vllm-project/vllm/pull/9650
* [Log][Bugfix] Fix default value check for `image_url.detail` by @mgoin in https://github.com/vllm-project/vllm/pull/9663
* [Performance][Kernel] Fused_moe Performance Improvement by @charlifu in https://github.com/vllm-project/vllm/pull/9384
* [Bugfix] Remove xformers requirement for Pixtral by @mgoin in https://github.com/vllm-project/vllm/pull/9597
* [ci/Build] Skip Chameleon for transformers 4.46.0 on broadcast test #9675 by @khluu in https://github.com/vllm-project/vllm/pull/9676
* [Model] add a lora module for granite 3.0 MoE models by @willmj in https://github.com/vllm-project/vllm/pull/9673
* [V1] Support sliding window attention by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9679
* [Bugfix] Fix compressed_tensors_moe bad config.strategy by @mgoin in https://github.com/vllm-project/vllm/pull/9677
* [Doc] Improve quickstart documentation by @rafvasq in https://github.com/vllm-project/vllm/pull/9256
* [Bugfix] Fix crash with llama 3.2 vision models and guided decoding by @tjohnson31415 in https://github.com/vllm-project/vllm/pull/9631
* [Bugfix] Steaming continuous_usage_stats default to False by @samos123 in https://github.com/vllm-project/vllm/pull/9709
* [Hardware][openvino] is_openvino --> current_platform.is_openvino by @MengqingCao in https://github.com/vllm-project/vllm/pull/9716
* Fix: MI100 Support By Bypassing Custom Paged Attention by @MErkinSag in https://github.com/vllm-project/vllm/pull/9560
* [Frontend] Bad words sampling parameter by @Alvant in https://github.com/vllm-project/vllm/pull/9717
* [Model] Add classification Task with Qwen2ForSequenceClassification  by @kakao-kevin-us in https://github.com/vllm-project/vllm/pull/9704
* [Misc] SpecDecodeWorker supports profiling by @Abatom in https://github.com/vllm-project/vllm/pull/9719
* [core] cudagraph output with tensor weak reference by @youkaichao in https://github.com/vllm-project/vllm/pull/9724
* [Misc] Upgrade to pytorch 2.5 by @bnellnm in https://github.com/vllm-project/vllm/pull/9588
* Fix cache management in "Close inactive issues and PRs" actions workflow by @hmellor in https://github.com/vllm-project/vllm/pull/9734
* [Bugfix] Fix load config when using bools by @madt2709 in https://github.com/vllm-project/vllm/pull/9533
* [Hardware][ROCM] using current_platform.is_rocm by @wangshuai09 in https://github.com/vllm-project/vllm/pull/9642
* [torch.compile] support moe models by @youkaichao in https://github.com/vllm-project/vllm/pull/9632
* Fix beam search eos by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/9627
* [Bugfix] Fix ray instance detect issue by @yma11 in https://github.com/vllm-project/vllm/pull/9439
* [CI/Build] Adopt Mergify for auto-labeling PRs by @russellb in https://github.com/vllm-project/vllm/pull/9259
* [Model][VLM] Add multi-video support for LLaVA-Onevision by @litianjian in https://github.com/vllm-project/vllm/pull/8905
* [torch.compile] Adding "torch compile" annotations to some models by @CRZbulabula in https://github.com/vllm-project/vllm/pull/9758
* [misc] avoid circular import by @youkaichao in https://github.com/vllm-project/vllm/pull/9765
* [torch.compile] add deepseek v2 compile by @youkaichao in https://github.com/vllm-project/vllm/pull/9775
* [Doc] fix third-party model example by @russellb in https://github.com/vllm-project/vllm/pull/9771
* [Model][LoRA]LoRA support added for Qwen by @jeejeelee in https://github.com/vllm-project/vllm/pull/9622
* [Doc] Specify async engine args in docs by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9726
* [Bugfix] Use temporary directory in registry by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9721
* [Frontend] re-enable multi-modality input in the new beam search implementation by @FerdinandZhong in https://github.com/vllm-project/vllm/pull/9427
* [Model] Add BNB quantization support for Mllama by @Isotr0py in https://github.com/vllm-project/vllm/pull/9720
* [Hardware] using current_platform.seed_everything by @wangshuai09 in https://github.com/vllm-project/vllm/pull/9785
* [Misc] Add metrics for request queue time, forward time, and execute time by @Abatom in https://github.com/vllm-project/vllm/pull/9659
* Fix the log to correct guide user to install modelscope by @tastelikefeet in https://github.com/vllm-project/vllm/pull/9793
* [Bugfix] Use host argument to bind to interface by @svenseeberg in https://github.com/vllm-project/vllm/pull/9798
* [Misc]: Typo fix: Renaming classes (casualLM -> causalLM) by @yannicks1 in https://github.com/vllm-project/vllm/pull/9801
* [Model]  Add LlamaEmbeddingModel as an embedding Implementation of LlamaModel by @jsato8094 in https://github.com/vllm-project/vllm/pull/9806
* [CI][Bugfix] Skip chameleon for transformers 4.46.1 by @mgoin in https://github.com/vllm-project/vllm/pull/9808
* [CI/Build] mergify: fix rules for ci/build label by @russellb in https://github.com/vllm-project/vllm/pull/9804
* [MISC] Set label value to timestamp over 0, to keep track of recent history  by @coolkp in https://github.com/vllm-project/vllm/pull/9777
* [Bugfix][Frontend] Guard against bad token ids by @joerunde in https://github.com/vllm-project/vllm/pull/9634
* [Model] tool calling support for ibm-granite/granite-20b-functioncalling by @wseaton in https://github.com/vllm-project/vllm/pull/8339
* [Docs] Add notes about Snowflake Meetup by @simon-mo in https://github.com/vllm-project/vllm/pull/9814
* [Bugfix] Fix prefix strings for quantized VLMs by @mgoin in https://github.com/vllm-project/vllm/pull/9772
* [core][distributed] fix custom allreduce in pytorch 2.5 by @youkaichao in https://github.com/vllm-project/vllm/pull/9815
* Update README.md by @LiuXiaoxuanPKU in https://github.com/vllm-project/vllm/pull/9819
* [Bugfix][VLM] Make apply_fp8_linear work with >2D input by @mgoin in https://github.com/vllm-project/vllm/pull/9812
* [ci/build] Pin CI dependencies version with pip-compile  by @khluu in https://github.com/vllm-project/vllm/pull/9810
* [Bugfix] Fix multi nodes TP+PP for XPU by @yma11 in https://github.com/vllm-project/vllm/pull/8884
* [Doc] Add the DCO to CONTRIBUTING.md by @russellb in https://github.com/vllm-project/vllm/pull/9803
* [torch.compile] rework compile control with piecewise cudagraph by @youkaichao in https://github.com/vllm-project/vllm/pull/9715
* [Misc] Specify minimum pynvml version by @jeejeelee in https://github.com/vllm-project/vllm/pull/9827
* [TPU] Correctly profile peak memory usage & Upgrade PyTorch XLA by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9438
* [CI/Build] VLM Test Consolidation by @alex-jw-brooks in https://github.com/vllm-project/vllm/pull/9372
* [Model] Support math-shepherd-mistral-7b-prm model by @Went-Liang in https://github.com/vllm-project/vllm/pull/9697
* [Misc] Add chunked-prefill support on FlashInfer. by @elfiegg in https://github.com/vllm-project/vllm/pull/9781
* [Bugfix][core] replace heartbeat with pid check by @joerunde in https://github.com/vllm-project/vllm/pull/9818
* [Doc] link bug for multistep guided decoding by @joerunde in https://github.com/vllm-project/vllm/pull/9843
* [Neuron] Update Dockerfile.neuron to fix build failure by @hbikki in https://github.com/vllm-project/vllm/pull/9822
* [doc] update pp support by @youkaichao in https://github.com/vllm-project/vllm/pull/9853
* [CI/Build] Simplify exception trace in api server tests by @CRZbulabula in https://github.com/vllm-project/vllm/pull/9787
* [torch.compile] upgrade tests by @youkaichao in https://github.com/vllm-project/vllm/pull/9858
* [Misc][OpenAI] deprecate max_tokens in favor of new max_completion_tokens field for chat completion endpoint by @gcalmettes in https://github.com/vllm-project/vllm/pull/9837
* Revert "[Bugfix] Use host argument to bind to interface (#9798)" by @khluu in https://github.com/vllm-project/vllm/pull/9852
* [Model] Support quantization of Qwen2VisionTransformer for Qwen2-VL by @mgoin in https://github.com/vllm-project/vllm/pull/9817
* [Misc] Remove deprecated arg for cuda graph capture by @ywang96 in https://github.com/vllm-project/vllm/pull/9864
* [Doc] Update Qwen documentation by @jeejeelee in https://github.com/vllm-project/vllm/pull/9869
* [CI/Build] Add Model Tests for Qwen2-VL by @alex-jw-brooks in https://github.com/vllm-project/vllm/pull/9846
* [CI/Build] Adding a forced docker system prune to clean up space by @Alexei-V-Ivanov-AMD in https://github.com/vllm-project/vllm/pull/9849
* [Bugfix] Fix `illegal memory access` error with chunked prefill, prefix caching, block manager v2 and xformers enabled together by @sasha0552 in https://github.com/vllm-project/vllm/pull/9532
* [BugFix][Kernel] Fix Illegal memory access in causal_conv1d in H100 by @mzusman in https://github.com/vllm-project/vllm/pull/9838
* [ci/build] Configure dependabot to update pip dependencies  by @khluu in https://github.com/vllm-project/vllm/pull/9811
* [Bugfix][Frontend] Reject guided decoding in multistep mode by @joerunde in https://github.com/vllm-project/vllm/pull/9892
* [torch.compile] directly register custom op by @youkaichao in https://github.com/vllm-project/vllm/pull/9896
* [Bugfix] Fix layer skip logic with bitsandbytes by @mgoin in https://github.com/vllm-project/vllm/pull/9887
* [torch.compile] rework test plans by @youkaichao in https://github.com/vllm-project/vllm/pull/9866
* [Model] Support bitsandbytes for MiniCPMV by @mgoin in https://github.com/vllm-project/vllm/pull/9891
* [torch.compile] Adding torch compile annotations to some models by @CRZbulabula in https://github.com/vllm-project/vllm/pull/9876
* [Doc] Update multi-input support by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9906
* [Frontend] Chat-based Embeddings API by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9759
* [CI/Build] Add Model Tests for PixtralHF by @mgoin in https://github.com/vllm-project/vllm/pull/9813
* [Frontend] Use a proper chat template for VLM2Vec by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9912
* [Bugfix] Fix edge cases for MistralTokenizer by @tjohnson31415 in https://github.com/vllm-project/vllm/pull/9625
* [Core] Refactor: Clean up unused argument preemption_mode in Scheduler._preempt by @andrejonasson in https://github.com/vllm-project/vllm/pull/9696
* [torch.compile] use interpreter with stable api from pytorch by @youkaichao in https://github.com/vllm-project/vllm/pull/9889
* [Bugfix/Core] Remove assertion for Flashinfer k_scale and v_scale by @pavanimajety in https://github.com/vllm-project/vllm/pull/9861
* [1/N] pass the complete config from engine to executor by @youkaichao in https://github.com/vllm-project/vllm/pull/9933
* [Bugfix] PicklingError on RayTaskError by @GeneDer in https://github.com/vllm-project/vllm/pull/9934
* Bump the patch-update group with 10 updates by @dependabot in https://github.com/vllm-project/vllm/pull/9897
* [Core][VLM] Add precise multi-modal placeholder tracking by @petersalas in https://github.com/vllm-project/vllm/pull/8346
* [ci/build] Have dependabot ignore pinned dependencies by @khluu in https://github.com/vllm-project/vllm/pull/9935
* [Encoder Decoder] Add flash_attn kernel support for encoder-decoder models by @sroy745 in https://github.com/vllm-project/vllm/pull/9559
* [torch.compile] fix cpu broken code by @youkaichao in https://github.com/vllm-project/vllm/pull/9947
* [Docs] Update Granite 3.0 models in supported models table by @njhill in https://github.com/vllm-project/vllm/pull/9930
* [Doc] Updated tpu-installation.rst with more details by @mikegre-google in https://github.com/vllm-project/vllm/pull/9926
* [2/N] executor pass the complete config to worker/modelrunner by @youkaichao in https://github.com/vllm-project/vllm/pull/9938
* [V1] Fix `EngineArgs` refactor on V1 by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/9954
* [bugfix] fix chatglm dummy_data_for_glmv by @youkaichao in https://github.com/vllm-project/vllm/pull/9955
* [3/N] model runner pass the whole config to model by @youkaichao in https://github.com/vllm-project/vllm/pull/9958
* [CI/Build] Quoting around > by @nokados in https://github.com/vllm-project/vllm/pull/9956
* [torch.compile] Adding torch compile annotations to vision-language models by @CRZbulabula in https://github.com/vllm-project/vllm/pull/9946
* [bugfix] fix tsts by @youkaichao in https://github.com/vllm-project/vllm/pull/9959
* [V1] Support per-request seed by @njhill in https://github.com/vllm-project/vllm/pull/9945
* [Model] Add support for H2OVL-Mississippi models by @cooleel in https://github.com/vllm-project/vllm/pull/9747
* [V1] Fix Configs by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/9971
* [Bugfix] Fix MiniCPMV and Mllama BNB  bug by @jeejeelee in https://github.com/vllm-project/vllm/pull/9917
* [Bugfix]Using the correct type hints by @gshtras in https://github.com/vllm-project/vllm/pull/9885
* [Misc] Compute query_start_loc/seq_start_loc on CPU by @zhengy001 in https://github.com/vllm-project/vllm/pull/9447
* [Bugfix] Fix E2EL mean and median stats by @daitran2k1 in https://github.com/vllm-project/vllm/pull/9984
* [Bugfix][OpenVINO] Fix circular reference #9939 by @MengqingCao in https://github.com/vllm-project/vllm/pull/9974
* [Frontend] Multi-Modality Support for Loading Local Image Files by @chaunceyjiang in https://github.com/vllm-project/vllm/pull/9915
* [4/N] make quant config first-class citizen by @youkaichao in https://github.com/vllm-project/vllm/pull/9978
* [Misc]Reduce BNB static variable by @jeejeelee in https://github.com/vllm-project/vllm/pull/9987
* [Model] factoring out MambaMixer out of Jamba by @mzusman in https://github.com/vllm-project/vllm/pull/8993
* [CI] Basic Integration Test For TPU by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/9968
* [Bugfix][CI/Build][Hardware][AMD] Shard ID parameters in AMD tests running parallel jobs by @hissu-hyvarinen in https://github.com/vllm-project/vllm/pull/9279
* [Doc] Update VLM doc about loading from local files by @ywang96 in https://github.com/vllm-project/vllm/pull/9999
* [Bugfix] Fix `MQLLMEngine` hanging by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/9973
* [Misc] Refactor benchmark_throughput.py by @lk-chen in https://github.com/vllm-project/vllm/pull/9779
* [Frontend] Add max_tokens prometheus metric by @tomeras91 in https://github.com/vllm-project/vllm/pull/9881
* [Bugfix] Upgrade to pytorch 2.5.1 by @bnellnm in https://github.com/vllm-project/vllm/pull/10001
* [4.5/N] bugfix for quant config in speculative decode by @youkaichao in https://github.com/vllm-project/vllm/pull/10007
* [Bugfix] Respect modules_to_not_convert within awq_marlin by @mgoin in https://github.com/vllm-project/vllm/pull/9895
* [Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9994
* [Core] Make encoder-decoder inputs a nested structure to be more composable by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9604
* [Bugfix] Fixup Mamba by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/10004
* [BugFix] Lazy import ray by @GeneDer in https://github.com/vllm-project/vllm/pull/10021
* [Misc] vllm CLI flags should be ordered for better user readability by @chaunceyjiang in https://github.com/vllm-project/vllm/pull/10017
* [Frontend] Fix tcp port reservation for api server by @russellb in https://github.com/vllm-project/vllm/pull/10012
* Refactor TPU requirements file and pin build dependencies by @richardsliu in https://github.com/vllm-project/vllm/pull/10010
* [Misc] Add logging for CUDA memory by @yangalan123 in https://github.com/vllm-project/vllm/pull/10027
* [CI/Build] Limit github CI jobs based on files changed by @russellb in https://github.com/vllm-project/vllm/pull/9928
* [Model] Support quantization of PixtralHFTransformer for PixtralHF by @mgoin in https://github.com/vllm-project/vllm/pull/9921
* [Feature] Update benchmark_throughput.py to support image input by @lk-chen in https://github.com/vllm-project/vllm/pull/9851
* [Misc] Modify BNB parameter name by @jeejeelee in https://github.com/vllm-project/vllm/pull/9997
* [CI] Prune tests/models/decoder_only/language/* tests by @mgoin in https://github.com/vllm-project/vllm/pull/9940
* [CI] Prune back the number of tests in tests/kernels/* by @mgoin in https://github.com/vllm-project/vllm/pull/9932
* [bugfix] fix weak ref in piecewise cudagraph and tractable test by @youkaichao in https://github.com/vllm-project/vllm/pull/10048
* [Bugfix] Properly propagate trust_remote_code settings by @zifeitong in https://github.com/vllm-project/vllm/pull/10047
* [Bugfix] Fix pickle of input when async output processing is on by @wallashss in https://github.com/vllm-project/vllm/pull/9931
* [Bugfix][SpecDecode] kv corruption with bonus tokens in spec decode by @llsj14 in https://github.com/vllm-project/vllm/pull/9730
* [v1] reduce graph capture time for piecewise cudagraph by @youkaichao in https://github.com/vllm-project/vllm/pull/10059
* [Misc] Sort the list of embedding models by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10037
* [Model][OpenVINO] Fix regressions from #8346 by @petersalas in https://github.com/vllm-project/vllm/pull/10045
* [Bugfix] Fix edge-case crash when using chat with the Mistral Tekken Tokenizer by @tjohnson31415 in https://github.com/vllm-project/vllm/pull/10051
* [Bugfix] Gpt-j-6B patch kv_scale to k_scale path  by @arakowsk-amd in https://github.com/vllm-project/vllm/pull/10063
* [Bugfix] Remove CustomChatCompletionContentPartParam multimodal input type by @zifeitong in https://github.com/vllm-project/vllm/pull/10054
* [V1] Integrate Piecewise CUDA graphs by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10058
* [distributed] add function to create ipc buffers directly by @youkaichao in https://github.com/vllm-project/vllm/pull/10064
* [CI/Build] drop support for  Python 3.8 EOL by @aarnphm in https://github.com/vllm-project/vllm/pull/8464
* [CI/Build] Fix large_gpu_mark reason by @Isotr0py in https://github.com/vllm-project/vllm/pull/10070
* [Hardware][Intel-Gaudi] Add Intel Gaudi (HPU) inference backend by @kzawora-intel in https://github.com/vllm-project/vllm/pull/6143
* [Hotfix] Fix ruff errors by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10073
* [Model][LoRA]LoRA support added for LlamaEmbeddingModel by @jeejeelee in https://github.com/vllm-project/vllm/pull/10071
* [Model] Add Idefics3 support by @jeejeelee in https://github.com/vllm-project/vllm/pull/9767
* [Model][LoRA]LoRA support added for Qwen2VLForConditionalGeneration by @ericperfect in https://github.com/vllm-project/vllm/pull/10022
* Remove ScaledActivation for AWQ by @mgoin in https://github.com/vllm-project/vllm/pull/10057
* [CI/Build] Drop Python 3.8 support by @russellb in https://github.com/vllm-project/vllm/pull/10038
* [CI/Build] change conflict PR comment from mergify by @russellb in https://github.com/vllm-project/vllm/pull/10080
* [V1] Make v1 more testable by @joerunde in https://github.com/vllm-project/vllm/pull/9888
* [CI/Build] Always run the ruff workflow by @russellb in https://github.com/vllm-project/vllm/pull/10092
* [core][distributed] add stateless_init_process_group by @youkaichao in https://github.com/vllm-project/vllm/pull/10072
* [Bugfix] Fix FP8 torch._scaled_mm fallback for torch>2.5 with CUDA<12.4 by @mgoin in https://github.com/vllm-project/vllm/pull/10095
* [Misc][XPU] Upgrade to Pytorch 2.5 for xpu backend by @yma11 in https://github.com/vllm-project/vllm/pull/9823
* [Frontend] Adjust try/except blocks in API impl by @njhill in https://github.com/vllm-project/vllm/pull/10056
* [Hardware][CPU] Update torch 2.5 by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/9911
* [doc] add back Python 3.8 ABI by @youkaichao in https://github.com/vllm-project/vllm/pull/10100
* [V1][BugFix] Fix Generator construction in greedy + seed case by @njhill in https://github.com/vllm-project/vllm/pull/10097
* [Misc] Consolidate ModelConfig code related to HF config by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10104
* [CI/Build] re-add codespell to CI by @russellb in https://github.com/vllm-project/vllm/pull/10083
* [Doc] Improve benchmark documentation by @rafvasq in https://github.com/vllm-project/vllm/pull/9927
* [Core][Distributed] Refactor ipc buffer init in CustomAllreduce by @hanzhi713 in https://github.com/vllm-project/vllm/pull/10030
* [CI/Build] Improve mypy + python version matrix by @russellb in https://github.com/vllm-project/vllm/pull/10041
* Adds method to read the pooling types from model's files by @flaviabeo in https://github.com/vllm-project/vllm/pull/9506
* [Frontend] Fix multiple values for keyword argument error (#10075) by @DIYer22 in https://github.com/vllm-project/vllm/pull/10076
* [Hardware][CPU][bugfix] Fix half dtype support on AVX2-only target by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/10108
* [Bugfix] Make image processor respect `mm_processor_kwargs` for Qwen2-VL by @li-plus in https://github.com/vllm-project/vllm/pull/10112
* [Misc] Add Gamma-Distribution Request Generation Support for Serving Benchmark. by @spliii in https://github.com/vllm-project/vllm/pull/10105
* [Frontend] Tool calling parser for Granite 3.0 models by @maxdebayser in https://github.com/vllm-project/vllm/pull/9027
* [Feature] [Spec decode]: Combine chunked prefill with speculative decoding by @NickLucche in https://github.com/vllm-project/vllm/pull/9291
* [CI/Build] Always run mypy by @russellb in https://github.com/vllm-project/vllm/pull/10122
* [CI/Build] Add shell script linting using shellcheck by @russellb in https://github.com/vllm-project/vllm/pull/7925
* [CI/Build] Automate PR body text cleanup by @russellb in https://github.com/vllm-project/vllm/pull/10082
* Bump actions/setup-python from 5.2.0 to 5.3.0 by @dependabot in https://github.com/vllm-project/vllm/pull/9745
* Online video support for VLMs by @litianjian in https://github.com/vllm-project/vllm/pull/10020
* Bump actions/checkout from 4.2.1 to 4.2.2 by @dependabot in https://github.com/vllm-project/vllm/pull/9746
* [Misc] Add environment variables collection in collect_env.py tool by @ycool in https://github.com/vllm-project/vllm/pull/9293
* [V1] Add all_token_ids attribute to Request by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10135
* [V1] Prefix caching (take 2) by @comaniac in https://github.com/vllm-project/vllm/pull/9972
* [CI/Build] Give PR cleanup job PR write access by @russellb in https://github.com/vllm-project/vllm/pull/10139
* [Doc] Update FAQ links in spec_decode.rst by @whyiug in https://github.com/vllm-project/vllm/pull/9662
* [Bugfix] Add error handling when server cannot respond any valid tokens by @DearPlanet in https://github.com/vllm-project/vllm/pull/5895
* [Misc] Fix ImportError causing by triton by @MengqingCao in https://github.com/vllm-project/vllm/pull/9493
* [Doc] Move CONTRIBUTING to docs site by @russellb in https://github.com/vllm-project/vllm/pull/9924
* Fixes a typo about 'max_decode_seq_len' which causes crashes with cuda graph. by @sighingnow in https://github.com/vllm-project/vllm/pull/9285
* Add hf_transfer to testing image by @mgoin in https://github.com/vllm-project/vllm/pull/10096
* [Misc] Fix typo in #5895 by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10145
* [Bugfix][XPU] Fix xpu tp by introducing XpuCommunicator by @yma11 in https://github.com/vllm-project/vllm/pull/10144
* [Model] Expose size to Idefics3 as mm_processor_kwargs by @Isotr0py in https://github.com/vllm-project/vllm/pull/10146
* [V1]Enable APC by default only for text models by @ywang96 in https://github.com/vllm-project/vllm/pull/10148
* [CI/Build] Update CPU tests to include all "standard" tests by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/5481
* Fix edge case Mistral tokenizer by @patrickvonplaten in https://github.com/vllm-project/vllm/pull/10152
* Disable spec-decode + chunked-prefill for draft models with tensor parallelism > 1 by @sroy745 in https://github.com/vllm-project/vllm/pull/10136
* [Misc] Improve Web UI by @rafvasq in https://github.com/vllm-project/vllm/pull/10090
* [V1] Fix non-cudagraph op name by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10166
* [CI/Build] Ignore .gitignored files for shellcheck by @ProExpertProg in https://github.com/vllm-project/vllm/pull/10162
* Rename vllm.logging to vllm.logging_utils by @flozi00 in https://github.com/vllm-project/vllm/pull/10134
* [torch.compile] Fuse RMSNorm with quant by @ProExpertProg in https://github.com/vllm-project/vllm/pull/9138
* [Bugfix] Fix SymIntArrayRef expected to contain only concrete integers by @bnellnm in https://github.com/vllm-project/vllm/pull/10170
* [Kernel][Triton] Add Triton implementation for scaled_mm_triton to support fp8 and int8 SmoothQuant, symmetric case by @rasmith in https://github.com/vllm-project/vllm/pull/9857
* [CI/Build] Adding timeout in CPU CI to avoid CPU test queue blocking by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/6892
* [0/N] Rename `MultiModalInputs` to `MultiModalKwargs` by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10040
* [Bugfix] Ignore GPTQ quantization of Qwen2-VL visual module by @mgoin in https://github.com/vllm-project/vllm/pull/10169
* [CI/Build] Fix VLM broadcast tests `tensor_parallel_size` passing by @Isotr0py in https://github.com/vllm-project/vllm/pull/10161
* [Doc] Adjust RunLLM location by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10176
* [5/N] pass the whole config to model by @youkaichao in https://github.com/vllm-project/vllm/pull/9983
* [CI/Build] Add run-hpu-test.sh script by @xuechendi in https://github.com/vllm-project/vllm/pull/10167
* [Bugfix] Enable some fp8 and quantized fullgraph tests by @bnellnm in https://github.com/vllm-project/vllm/pull/10171
* [bugfix] fix broken tests of mlp speculator by @youkaichao in https://github.com/vllm-project/vllm/pull/10177
* [doc] explaining the integration with huggingface by @youkaichao in https://github.com/vllm-project/vllm/pull/10173
* bugfix: fix the bug that stream generate not work by @caijizhuo in https://github.com/vllm-project/vllm/pull/2756
* [Frontend] add `add_request_id` middleware by @cjackal in https://github.com/vllm-project/vllm/pull/9594
* [Frontend][Core] Override HF `config.json` via CLI by @KrishnaM251 in https://github.com/vllm-project/vllm/pull/5836
* [CI/Build] Split up models tests by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10069
* [ci][build] limit cmake version by @youkaichao in https://github.com/vllm-project/vllm/pull/10188
* [Doc] Fix typo error in CONTRIBUTING.md by @FuryMartin in https://github.com/vllm-project/vllm/pull/10190
* [doc] Polish the integration with huggingface doc by @CRZbulabula in https://github.com/vllm-project/vllm/pull/10195
* [Misc] small fixes to function tracing file path by @ShawnD200 in https://github.com/vllm-project/vllm/pull/9543
* [misc] improve cloudpickle registration and tests by @youkaichao in https://github.com/vllm-project/vllm/pull/10202
* [Doc] Fix typo error in vllm/entrypoints/openai/cli_args.py by @yansh97 in https://github.com/vllm-project/vllm/pull/10196
* [doc] improve debugging code by @youkaichao in https://github.com/vllm-project/vllm/pull/10206
* [6/N] pass whole config to inner model by @youkaichao in https://github.com/vllm-project/vllm/pull/10205
* Bump the patch-update group with 5 updates by @dependabot in https://github.com/vllm-project/vllm/pull/10210
* [Hardware][CPU] Add embedding models support for CPU backend by @Isotr0py in https://github.com/vllm-project/vllm/pull/10193
* [LoRA][Kernel] Remove the unused libentry module by @jeejeelee in https://github.com/vllm-project/vllm/pull/10214
* [V1] Allow `tokenizer_mode` and `trust_remote_code` for Detokenizer by @ywang96 in https://github.com/vllm-project/vllm/pull/10211
* [Bugfix][Hardware][CPU] Fix broken encoder-decoder CPU runner by @Isotr0py in https://github.com/vllm-project/vllm/pull/10218
* [Metrics] add more metrics by @HarryWu99 in https://github.com/vllm-project/vllm/pull/4464
* [Doc] fix doc string typo in block_manager `swap_out` function by @yyccli in https://github.com/vllm-project/vllm/pull/10212
* [core][distributed] add stateless process group by @youkaichao in https://github.com/vllm-project/vllm/pull/10216
* Bump actions/setup-python from 5.2.0 to 5.3.0 by @dependabot in https://github.com/vllm-project/vllm/pull/10209
* [V1] Fix detokenizer ports by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10224
* [V1] Do not use inductor for piecewise CUDA graphs by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10225
* [v1][torch.compile] support managing cudagraph buffer by @youkaichao in https://github.com/vllm-project/vllm/pull/10203
* [V1] Use custom ops for piecewise CUDA graphs by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10227
* Add docs on serving with Llama Stack by @terrytangyuan in https://github.com/vllm-project/vllm/pull/10183
* [misc][distributed] auto port selection and disable tests by @youkaichao in https://github.com/vllm-project/vllm/pull/10226
* [V1] Enable custom ops with piecewise CUDA graphs by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10228
* Make shutil rename in python_only_dev by @shcheglovnd in https://github.com/vllm-project/vllm/pull/10233
* [V1] `AsyncLLM` Implementation by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/9826
* [doc] update debugging guide by @youkaichao in https://github.com/vllm-project/vllm/pull/10236
* [Doc] Update help text for `--distributed-executor-backend` by @russellb in https://github.com/vllm-project/vllm/pull/10231
* [1/N] torch.compile user interface design by @youkaichao in https://github.com/vllm-project/vllm/pull/10237
* [Misc][LoRA] Replace hardcoded cuda device with configurable argument  by @jeejeelee in https://github.com/vllm-project/vllm/pull/10223
* Splitting attention kernel file by @maleksan85 in https://github.com/vllm-project/vllm/pull/10091
* [doc] explain the class hierarchy in vLLM by @youkaichao in https://github.com/vllm-project/vllm/pull/10240
* [CI][CPU]refactor CPU tests to allow to bind with different cores by @zhouyuan in https://github.com/vllm-project/vllm/pull/10222
* [BugFix] Do not raise a `ValueError` when `tool_choice` is set to the supported `none` option and `tools` are not defined. by @gcalmettes in https://github.com/vllm-project/vllm/pull/10000
* [Misc]Fix Idefics3Model argument by @jeejeelee in https://github.com/vllm-project/vllm/pull/10255
* [Bugfix] Fix QwenModel argument by @DamonFool in https://github.com/vllm-project/vllm/pull/10262
* [Frontend] Add per-request number of cached token stats by @zifeitong in https://github.com/vllm-project/vllm/pull/10174
* [V1] Use pickle for serializing EngineCoreRequest & Add multimodal inputs to EngineCoreRequest by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10245
* [Encoder Decoder] Update Mllama to run with both FlashAttention and XFormers by @sroy745 in https://github.com/vllm-project/vllm/pull/9982
* [LoRA] Adds support for bias in LoRA by @followumesh in https://github.com/vllm-project/vllm/pull/5733
* [V1] Enable Inductor when using piecewise CUDA graphs by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10268
* [doc] fix location of runllm widget by @youkaichao in https://github.com/vllm-project/vllm/pull/10266
* [doc] improve debugging doc by @youkaichao in https://github.com/vllm-project/vllm/pull/10270
* Revert "[ci][build] limit cmake version" by @youkaichao in https://github.com/vllm-project/vllm/pull/10271
* [V1] Fix CI tests on V1 engine by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10272
* [core][distributed] use tcp store directly by @youkaichao in https://github.com/vllm-project/vllm/pull/10275
* [V1] Support VLMs with fine-grained scheduling by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9871
* Bump to compressed-tensors v0.8.0 by @dsikka in https://github.com/vllm-project/vllm/pull/10279
* [Doc] Fix typo in arg_utils.py by @xyang16 in https://github.com/vllm-project/vllm/pull/10264
* [Model] Add support for Qwen2-VL video embeddings input & multiple image embeddings input with varied resolutions by @imkero in https://github.com/vllm-project/vllm/pull/10221
* [Model] Adding Support for Qwen2VL as an Embedding Model. Using MrLight/dse-qwen2-2b-mrl-v1 by @FurtherAI in https://github.com/vllm-project/vllm/pull/9944
* [Core] Flashinfer - Remove advance step size restriction by @pavanimajety in https://github.com/vllm-project/vllm/pull/10282
* [Model][LoRA]LoRA support added for idefics3 by @B-201 in https://github.com/vllm-project/vllm/pull/10281
* [V1] Add missing tokenizer options for `Detokenizer` by @ywang96 in https://github.com/vllm-project/vllm/pull/10288
* [1/N] Initial prototype for multi-modal processor by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10044
* [Bugfix] bitsandbytes models fail to run pipeline parallel by @HoangCongDuc in https://github.com/vllm-project/vllm/pull/10200
* [Bugfix] Fix tensor parallel for qwen2 classification model by @Isotr0py in https://github.com/vllm-project/vllm/pull/10297
* [misc] error early for old-style class by @youkaichao in https://github.com/vllm-project/vllm/pull/10304
* [Misc] format.sh: Simplify tool_version_check by @russellb in https://github.com/vllm-project/vllm/pull/10305
* [Frontend] Pythonic tool parser by @mdepinet in https://github.com/vllm-project/vllm/pull/9859
* [BugFix]: properly deserialize `tool_calls` iterator before processing by mistral-common when MistralTokenizer is used by @gcalmettes in https://github.com/vllm-project/vllm/pull/9951
* [Model] Add BNB quantization support for Idefics3 by @B-201 in https://github.com/vllm-project/vllm/pull/10310
* [ci][distributed] disable hanging tests by @youkaichao in https://github.com/vllm-project/vllm/pull/10317
* [CI/Build] Fix CPU CI online inference timeout by @Isotr0py in https://github.com/vllm-project/vllm/pull/10314
* [CI/Build] Make shellcheck happy by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10285
* [Docs] Publish meetup slides by @WoosukKwon in https://github.com/vllm-project/vllm/pull/10331
* Support Roberta embedding models by @maxdebayser in https://github.com/vllm-project/vllm/pull/9387
* [Perf] Reduce peak memory usage of llama by @andoorve in https://github.com/vllm-project/vllm/pull/10339
* [Bugfix] use AF_INET6 instead of AF_INET for OpenAI Compatible Server by @jxpxxzj in https://github.com/vllm-project/vllm/pull/9583
* [Tool parsing] Improve / correct mistral tool parsing by @patrickvonplaten in https://github.com/vllm-project/vllm/pull/10333
* [Bugfix] Fix unable to load some models by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10312
* [bugfix] Fix static asymmetric quantization case by @ProExpertProg in https://github.com/vllm-project/vllm/pull/10334
* [Misc] Change RedundantReshapesPass and FusionPass logging from info to debug by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/10308
* [Model] Support Qwen2 embeddings and use tags to select model tests by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10184
* [Bugfix]  Qwen-vl output is inconsistent in speculative decoding by @skylee-01 in https://github.com/vllm-project/vllm/pull/10350
* [Misc] Consolidate pooler config overrides by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/10351
* [Build] skip renaming files for release wheels pipeline by @simon-mo in https://github.com/vllm-project/vllm/pull/9671

## New Contributors
* @gracehonv made their first contribution in https://github.com/vllm-project/vllm/pull/9349
* @streaver91 made their first contribution in https://github.com/vllm-project/vllm/pull/9396
* @wukaixingxp made their first contribution in https://github.com/vllm-project/vllm/pull/9013
* @sssrijan-amazon made their first contribution in https://github.com/vllm-project/vllm/pull/9380
* @coolkp made their first contribution in https://github.com/vllm-project/vllm/pull/9477
* @yue-anyscale made their first contribution in https://github.com/vllm-project/vllm/pull/9478
* @dhiaEddineRhaiem made their first contribution in https://github.com/vllm-project/vllm/pull/9325
* @yudian0504 made their first contribution in https://github.com/vllm-project/vllm/pull/9549
* @ngrozae made their first contribution in https://github.com/vllm-project/vllm/pull/9552
* @Falko1 made their first contribution in https://github.com/vllm-project/vllm/pull/9503
* @wangshuai09 made their first contribution in https://github.com/vllm-project/vllm/pull/9536
* @gopalsarda made their first contribution in https://github.com/vllm-project/vllm/pull/9580
* @guoyuhong made their first contribution in https://github.com/vllm-project/vllm/pull/9550
* @JArnoldAMD made their first contribution in https://github.com/vllm-project/vllm/pull/9529
* @yuleil made their first contribution in https://github.com/vllm-project/vllm/pull/8234
* @sethkimmel3 made their first contribution in https://github.com/vllm-project/vllm/pull/7889
* @MengqingCao made their first contribution in https://github.com/vllm-project/vllm/pull/9605
* @CRZbulabula made their first contribution in https://github.com/vllm-project/vllm/pull/9614
* @faychu made their first contribution in https://github.com/vllm-project/vllm/pull/9248
* @vrdn-23 made their first contribution in https://github.com/vllm-project/vllm/pull/9358
* @willmj made their first contribution in https://github.com/vllm-project/vllm/pull/9673
* @samos123 made their first contribution in https://github.com/vllm-project/vllm/pull/9709
* @MErkinSag made their first contribution in https://github.com/vllm-project/vllm/pull/9560
* @Alvant made their first contribution in https://github.com/vllm-project/vllm/pull/9717
* @kakao-kevin-us made their first contribution in https://github.com/vllm-project/vllm/pull/9704
* @madt2709 made their first contribution in https://github.com/vllm-project/vllm/pull/9533
* @FerdinandZhong made their first contribution in https://github.com/vllm-project/vllm/pull/9427
* @svenseeberg made their first contribution in https://github.com/vllm-project/vllm/pull/9798
* @yannicks1 made their first contribution in https://github.com/vllm-project/vllm/pull/9801
* @wseaton made their first contribution in https://github.com/vllm-project/vllm/pull/8339
* @Went-Liang made their first contribution in https://github.com/vllm-project/vllm/pull/9697
* @andrejonasson made their first contribution in https://github.com/vllm-project/vllm/pull/9696
* @GeneDer made their first contribution in https://github.com/vllm-project/vllm/pull/9934
* @mikegre-google made their first contribution in https://github.com/vllm-project/vllm/pull/9926
* @nokados made their first contribution in https://github.com/vllm-project/vllm/pull/9956
* @cooleel made their first contribution in https://github.com/vllm-project/vllm/pull/9747
* @zhengy001 made their first contribution in https://github.com/vllm-project/vllm/pull/9447
* @daitran2k1 made their first contribution in https://github.com/vllm-project/vllm/pull/9984
* @chaunceyjiang made their first contribution in https://github.com/vllm-project/vllm/pull/9915
* @hissu-hyvarinen made their first contribution in https://github.com/vllm-project/vllm/pull/9279
* @lk-chen made their first contribution in https://github.com/vllm-project/vllm/pull/9779
* @yangalan123 made their first contribution in https://github.com/vllm-project/vllm/pull/10027
* @llsj14 made their first contribution in https://github.com/vllm-project/vllm/pull/9730
* @arakowsk-amd made their first contribution in https://github.com/vllm-project/vllm/pull/10063
* @kzawora-intel made their first contribution in https://github.com/vllm-project/vllm/pull/6143
* @DIYer22 made their first contribution in https://github.com/vllm-project/vllm/pull/10076
* @li-plus made their first contribution in https://github.com/vllm-project/vllm/pull/10112
* @spliii made their first contribution in https://github.com/vllm-project/vllm/pull/10105
* @flozi00 made their first contribution in https://github.com/vllm-project/vllm/pull/10134
* @xuechendi made their first contribution in https://github.com/vllm-project/vllm/pull/10167
* @caijizhuo made their first contribution in https://github.com/vllm-project/vllm/pull/2756
* @cjackal made their first contribution in https://github.com/vllm-project/vllm/pull/9594
* @KrishnaM251 made their first contribution in https://github.com/vllm-project/vllm/pull/5836
* @FuryMartin made their first contribution in https://github.com/vllm-project/vllm/pull/10190
* @ShawnD200 made their first contribution in https://github.com/vllm-project/vllm/pull/9543
* @yansh97 made their first contribution in https://github.com/vllm-project/vllm/pull/10196
* @yyccli made their first contribution in https://github.com/vllm-project/vllm/pull/10212
* @shcheglovnd made their first contribution in https://github.com/vllm-project/vllm/pull/10233
* @maleksan85 made their first contribution in https://github.com/vllm-project/vllm/pull/10091
* @followumesh made their first contribution in https://github.com/vllm-project/vllm/pull/5733
* @imkero made their first contribution in https://github.com/vllm-project/vllm/pull/10221
* @B-201 made their first contribution in https://github.com/vllm-project/vllm/pull/10281
* @HoangCongDuc made their first contribution in https://github.com/vllm-project/vllm/pull/10200
* @mdepinet made their first contribution in https://github.com/vllm-project/vllm/pull/9859
* @jxpxxzj made their first contribution in https://github.com/vllm-project/vllm/pull/9583
* @skylee-01 made their first contribution in https://github.com/vllm-project/vllm/pull/10350

**Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.6.3...v0.6.4