v0.7.2

vllm-project/vllmv0.7.2Feb 6, 2025by github-actions[bot]

AI Summary

Introduces Qwen2.5-VL support, `transformers` backend support, and performance enhancements for DeepSeek models via MLA and FP8 kernels.

Key Highlights

  • Qwen2.5-VL support
  • Add `transformers` backend support
  • DeepSeek performance enhancements (MLA, FP8 kernels)
  • AMD support for DeepSeek models
  • Enable FusedSDPA support for Intel Gaudi

New Features

  • DeepSeek V3 fp8_w8a8 configs for B200
  • LoRA Support for Ultravox model
  • LLaMA 3.2 support upstreaming

Full Release Notes

## Highlights

* Qwen2.5-VL is now supported in vLLM. Please note that it requires a source installation from Hugging Face `transformers` library at the moment (#12604)
* Add `transformers` backend support via `--model-impl=transformers`. This allows vLLM to be ran with arbitrary Hugging Face text models (#11330, #12785, #12727). 
* Performance enhancement to DeepSeek models. 
	* Align KV caches entries to start 256 byte boundaries, yielding 43% throughput enhancement (#12676)
	* Apply `torch.compile` to fused_moe/grouped_topk, yielding 5% throughput enhancement (#12637)
	* Enable MLA for DeepSeek VL2 (#12729)
	* Enable DeepSeek model on ROCm (#12662)

### Core Engine
* Use `VLLM_LOGITS_PROCESSOR_THREADS` to speed up structured decoding in high batch size scenarios (#12368)

### Security Update
* Improve hash collision avoidance in prefix caching (#12621)
* Add SPDX-License-Identifier headers to python source files (#12628)

### Other
* Enable FusedSDPA support for Intel Gaudi (HPU) (#12359)

## What's Changed
* Apply torch.compile to fused_moe/grouped_topk by @mgoin in https://github.com/vllm-project/vllm/pull/12637
* doc: fixing minor typo in readme.md by @vicenteherrera in https://github.com/vllm-project/vllm/pull/12643
* [Bugfix] fix moe_wna16 get_quant_method by @jinzhen-lin in https://github.com/vllm-project/vllm/pull/12648
* [Core] Silence unnecessary deprecation warnings by @russellb in https://github.com/vllm-project/vllm/pull/12620
* [V1][Minor] Avoid frequently creating ConstantList by @WoosukKwon in https://github.com/vllm-project/vllm/pull/12653
* [Core][v1] Unify allocating slots in prefill and decode in KV cache manager by @ShawnD200 in https://github.com/vllm-project/vllm/pull/12608
* [Hardware][Intel GPU] add XPU bf16 support by @jikunshang in https://github.com/vllm-project/vllm/pull/12392
* [Misc] Add SPDX-License-Identifier headers to python source files by @russellb in https://github.com/vllm-project/vllm/pull/12628
* [doc][misc] clarify VLLM_HOST_IP for multi-node inference by @youkaichao in https://github.com/vllm-project/vllm/pull/12667
* [Doc] Deprecate Discord by @zhuohan123 in https://github.com/vllm-project/vllm/pull/12668
* [Kernel] port sgl moe_align_block_size kernels by @chenyang78 in https://github.com/vllm-project/vllm/pull/12574
* make sure mistral_common not imported for non-mistral models by @youkaichao in https://github.com/vllm-project/vllm/pull/12669
* Properly check if all fused layers are in the list of targets by @eldarkurtic in https://github.com/vllm-project/vllm/pull/12666
* Fix for attention layers to remain unquantized during moe_wn16 quant by @srikanthsrnvs in https://github.com/vllm-project/vllm/pull/12570
* [cuda] manually import the correct pynvml module by @youkaichao in https://github.com/vllm-project/vllm/pull/12679
* [ci/build] fix gh200 test by @youkaichao in https://github.com/vllm-project/vllm/pull/12681
* [Model]: Add `transformers` backend support by @ArthurZucker in https://github.com/vllm-project/vllm/pull/11330
* [Misc] Fix improper placement of SPDX header in scripts by @russellb in https://github.com/vllm-project/vllm/pull/12694
* [Bugfix][Kernel] Fix per-token/per-channel quantization for Hopper scaled mm by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/12696
* Squelch MLA warning for Compressed-Tensors Models by @kylesayrs in https://github.com/vllm-project/vllm/pull/12704
* [Model] Add Deepseek V3 fp8_w8a8 configs for B200 by @kushanam in https://github.com/vllm-project/vllm/pull/12707
* [MISC] Remove model input dumping when exception by @comaniac in https://github.com/vllm-project/vllm/pull/12582
* [V1] Revert `uncache_blocks` and support recaching full blocks by @comaniac in https://github.com/vllm-project/vllm/pull/12415
* [Core] Improve hash collision avoidance in prefix caching by @russellb in https://github.com/vllm-project/vllm/pull/12621
* Support Pixtral-Large HF by using llava multimodal_projector_bias config by @mgoin in https://github.com/vllm-project/vllm/pull/12710
* [Doc] Replace ibm-fms with ibm-ai-platform by @tdoublep in https://github.com/vllm-project/vllm/pull/12709
* [Quant] Fix use_mla TypeError and support loading pure-sparsity Compressed Tensors configs by @kylesayrs in https://github.com/vllm-project/vllm/pull/12711
* [AMD][ROCm] Enable DeepSeek model on ROCm by @hongxiayang in https://github.com/vllm-project/vllm/pull/12662
* [Misc] Add BNB quantization for Whisper by @jeejeelee in https://github.com/vllm-project/vllm/pull/12381
* [VLM] Merged multi-modal processor for InternVL-based models by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/12553
* [V1] Remove constraints on partial requests by @WoosukKwon in https://github.com/vllm-project/vllm/pull/12674
* [VLM] Implement merged multimodal processor and V1 support for idefics3 by @Isotr0py in https://github.com/vllm-project/vllm/pull/12660
* [Model] [Bugfix] Fix loading of fine-tuned models based on Phi-3-Small  by @mgtk77 in https://github.com/vllm-project/vllm/pull/12689
* Avoid unnecessary multi-modal input data copy when len(batch) == 1 by @imkero in https://github.com/vllm-project/vllm/pull/12722
* [Build] update requirements of no-device for plugin usage by @sducouedic in https://github.com/vllm-project/vllm/pull/12630
* [Bugfix] Fix CI failures for InternVL and Mantis models by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/12728
* [V1][Metrics] Add request_success_total counter, labelled with finish reason by @markmc in https://github.com/vllm-project/vllm/pull/12579
* [Perf] Mem align KV caches for CUDA devices (MLA perf improvement) by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/12676
* [Core] add and implement `VLLM_LOGITS_PROCESSOR_THREADS` by @akeshet in https://github.com/vllm-project/vllm/pull/12368
* [ROCM][AMD][TRITON] Halving warps number for fw_prefill to reduce spilling by @maleksan85 in https://github.com/vllm-project/vllm/pull/12713
* Refactor `Linear` handling in `TransformersModel` by @hmellor in https://github.com/vllm-project/vllm/pull/12727
* [VLM] Add MLA with pure RoPE support for deepseek-vl2 models by @Isotr0py in https://github.com/vllm-project/vllm/pull/12729
* [Misc] Bump the compressed-tensors version by @dsikka in https://github.com/vllm-project/vllm/pull/12736
* [Model][Quant] Fix GLM, Fix fused module mappings for quantization by @kylesayrs in https://github.com/vllm-project/vllm/pull/12634
* [Doc] Update PR Reminder with link to Developer Slack by @mgoin in https://github.com/vllm-project/vllm/pull/12748
* [Bugfix] Fix OpenVINO model runner by @hmellor in https://github.com/vllm-project/vllm/pull/12750
* [V1][Misc] Shorten `FinishReason` enum and use constant strings by @njhill in https://github.com/vllm-project/vllm/pull/12760
* [Doc] Remove performance warning for auto_awq.md by @mgoin in https://github.com/vllm-project/vllm/pull/12743
* [Bugfix] Fix 'ModuleNotFoundError: No module named 'intel_extension_for_pytorch'' for --tensor-parallel-size more than 1  by @Akashcodes732 in https://github.com/vllm-project/vllm/pull/12546
* [core][distributed] exact ray placement control by @youkaichao in https://github.com/vllm-project/vllm/pull/12732
* [Kernel] Use self.kv_cache and forward_context.attn_metadata in Attention.forward by @heheda12345 in https://github.com/vllm-project/vllm/pull/12536
* [Hardware][Intel-Gaudi] Enable FusedSDPA support for Intel Gaudi (HPU) by @SanjuCSudhakaran in https://github.com/vllm-project/vllm/pull/12359
* Add: Support for Sparse24Bitmask Compressed Models by @rahul-tuli in https://github.com/vllm-project/vllm/pull/12097
* [VLM] Use shared field to pass token ids to model by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/12767
* [Docs] Drop duplicate [source] links by @russellb in https://github.com/vllm-project/vllm/pull/12780
* [VLM] Qwen2.5-VL by @ywang96 in https://github.com/vllm-project/vllm/pull/12604
* [VLM] Update compatibility with transformers 4.49 by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/12781
* Quantization and MoE configs for GH200 machines  by @arvindsun in https://github.com/vllm-project/vllm/pull/12717
* [ROCm][Kernel] Using the correct warp_size value by @gshtras in https://github.com/vllm-project/vllm/pull/12789
* [Bugfix] Better FP8 supported defaults by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/12796
* [Misc][Easy] Remove the space from the file name by @houseroad in https://github.com/vllm-project/vllm/pull/12799
* [Model] LoRA Support for Ultravox model by @thedebugger in https://github.com/vllm-project/vllm/pull/11253
* [Bugfix] Fix the test_ultravox.py's license by @houseroad in https://github.com/vllm-project/vllm/pull/12806
* Improve `TransformersModel` UX by @hmellor in https://github.com/vllm-project/vllm/pull/12785
* [Misc] Remove duplicated DeepSeek V2/V3 model definition by @mgoin in https://github.com/vllm-project/vllm/pull/12793
* [Misc] Improve error message for incorrect pynvml by @youkaichao in https://github.com/vllm-project/vllm/pull/12809

## New Contributors
* @vicenteherrera made their first contribution in https://github.com/vllm-project/vllm/pull/12643
* @chenyang78 made their first contribution in https://github.com/vllm-project/vllm/pull/12574
* @srikanthsrnvs made their first contribution in https://github.com/vllm-project/vllm/pull/12570
* @ArthurZucker made their first contribution in https://github.com/vllm-project/vllm/pull/11330
* @mgtk77 made their first contribution in https://github.com/vllm-project/vllm/pull/12689
* @sducouedic made their first contribution in https://github.com/vllm-project/vllm/pull/12630
* @akeshet made their first contribution in https://github.com/vllm-project/vllm/pull/12368
* @arvindsun made their first contribution in https://github.com/vllm-project/vllm/pull/12717
* @thedebugger made their first contribution in https://github.com/vllm-project/vllm/pull/11253

**Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.7.1...v0.7.2