v0.7.2
vllm-project/vllmv0.7.2Feb 6, 2025by github-actions[bot]
AI Summary
Introduces Qwen2.5-VL support, `transformers` backend support, and performance enhancements for DeepSeek models via MLA and FP8 kernels.
Key Highlights
- Qwen2.5-VL support
- Add `transformers` backend support
- DeepSeek performance enhancements (MLA, FP8 kernels)
- AMD support for DeepSeek models
- Enable FusedSDPA support for Intel Gaudi
New Features
- DeepSeek V3 fp8_w8a8 configs for B200
- LoRA Support for Ultravox model
- LLaMA 3.2 support upstreaming
Full Release Notes
## Highlights * Qwen2.5-VL is now supported in vLLM. Please note that it requires a source installation from Hugging Face `transformers` library at the moment (#12604) * Add `transformers` backend support via `--model-impl=transformers`. This allows vLLM to be ran with arbitrary Hugging Face text models (#11330, #12785, #12727). * Performance enhancement to DeepSeek models. * Align KV caches entries to start 256 byte boundaries, yielding 43% throughput enhancement (#12676) * Apply `torch.compile` to fused_moe/grouped_topk, yielding 5% throughput enhancement (#12637) * Enable MLA for DeepSeek VL2 (#12729) * Enable DeepSeek model on ROCm (#12662) ### Core Engine * Use `VLLM_LOGITS_PROCESSOR_THREADS` to speed up structured decoding in high batch size scenarios (#12368) ### Security Update * Improve hash collision avoidance in prefix caching (#12621) * Add SPDX-License-Identifier headers to python source files (#12628) ### Other * Enable FusedSDPA support for Intel Gaudi (HPU) (#12359) ## What's Changed * Apply torch.compile to fused_moe/grouped_topk by @mgoin in https://github.com/vllm-project/vllm/pull/12637 * doc: fixing minor typo in readme.md by @vicenteherrera in https://github.com/vllm-project/vllm/pull/12643 * [Bugfix] fix moe_wna16 get_quant_method by @jinzhen-lin in https://github.com/vllm-project/vllm/pull/12648 * [Core] Silence unnecessary deprecation warnings by @russellb in https://github.com/vllm-project/vllm/pull/12620 * [V1][Minor] Avoid frequently creating ConstantList by @WoosukKwon in https://github.com/vllm-project/vllm/pull/12653 * [Core][v1] Unify allocating slots in prefill and decode in KV cache manager by @ShawnD200 in https://github.com/vllm-project/vllm/pull/12608 * [Hardware][Intel GPU] add XPU bf16 support by @jikunshang in https://github.com/vllm-project/vllm/pull/12392 * [Misc] Add SPDX-License-Identifier headers to python source files by @russellb in https://github.com/vllm-project/vllm/pull/12628 * [doc][misc] clarify VLLM_HOST_IP for multi-node inference by @youkaichao in https://github.com/vllm-project/vllm/pull/12667 * [Doc] Deprecate Discord by @zhuohan123 in https://github.com/vllm-project/vllm/pull/12668 * [Kernel] port sgl moe_align_block_size kernels by @chenyang78 in https://github.com/vllm-project/vllm/pull/12574 * make sure mistral_common not imported for non-mistral models by @youkaichao in https://github.com/vllm-project/vllm/pull/12669 * Properly check if all fused layers are in the list of targets by @eldarkurtic in https://github.com/vllm-project/vllm/pull/12666 * Fix for attention layers to remain unquantized during moe_wn16 quant by @srikanthsrnvs in https://github.com/vllm-project/vllm/pull/12570 * [cuda] manually import the correct pynvml module by @youkaichao in https://github.com/vllm-project/vllm/pull/12679 * [ci/build] fix gh200 test by @youkaichao in https://github.com/vllm-project/vllm/pull/12681 * [Model]: Add `transformers` backend support by @ArthurZucker in https://github.com/vllm-project/vllm/pull/11330 * [Misc] Fix improper placement of SPDX header in scripts by @russellb in https://github.com/vllm-project/vllm/pull/12694 * [Bugfix][Kernel] Fix per-token/per-channel quantization for Hopper scaled mm by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/12696 * Squelch MLA warning for Compressed-Tensors Models by @kylesayrs in https://github.com/vllm-project/vllm/pull/12704 * [Model] Add Deepseek V3 fp8_w8a8 configs for B200 by @kushanam in https://github.com/vllm-project/vllm/pull/12707 * [MISC] Remove model input dumping when exception by @comaniac in https://github.com/vllm-project/vllm/pull/12582 * [V1] Revert `uncache_blocks` and support recaching full blocks by @comaniac in https://github.com/vllm-project/vllm/pull/12415 * [Core] Improve hash collision avoidance in prefix caching by @russellb in https://github.com/vllm-project/vllm/pull/12621 * Support Pixtral-Large HF by using llava multimodal_projector_bias config by @mgoin in https://github.com/vllm-project/vllm/pull/12710 * [Doc] Replace ibm-fms with ibm-ai-platform by @tdoublep in https://github.com/vllm-project/vllm/pull/12709 * [Quant] Fix use_mla TypeError and support loading pure-sparsity Compressed Tensors configs by @kylesayrs in https://github.com/vllm-project/vllm/pull/12711 * [AMD][ROCm] Enable DeepSeek model on ROCm by @hongxiayang in https://github.com/vllm-project/vllm/pull/12662 * [Misc] Add BNB quantization for Whisper by @jeejeelee in https://github.com/vllm-project/vllm/pull/12381 * [VLM] Merged multi-modal processor for InternVL-based models by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/12553 * [V1] Remove constraints on partial requests by @WoosukKwon in https://github.com/vllm-project/vllm/pull/12674 * [VLM] Implement merged multimodal processor and V1 support for idefics3 by @Isotr0py in https://github.com/vllm-project/vllm/pull/12660 * [Model] [Bugfix] Fix loading of fine-tuned models based on Phi-3-Small by @mgtk77 in https://github.com/vllm-project/vllm/pull/12689 * Avoid unnecessary multi-modal input data copy when len(batch) == 1 by @imkero in https://github.com/vllm-project/vllm/pull/12722 * [Build] update requirements of no-device for plugin usage by @sducouedic in https://github.com/vllm-project/vllm/pull/12630 * [Bugfix] Fix CI failures for InternVL and Mantis models by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/12728 * [V1][Metrics] Add request_success_total counter, labelled with finish reason by @markmc in https://github.com/vllm-project/vllm/pull/12579 * [Perf] Mem align KV caches for CUDA devices (MLA perf improvement) by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/12676 * [Core] add and implement `VLLM_LOGITS_PROCESSOR_THREADS` by @akeshet in https://github.com/vllm-project/vllm/pull/12368 * [ROCM][AMD][TRITON] Halving warps number for fw_prefill to reduce spilling by @maleksan85 in https://github.com/vllm-project/vllm/pull/12713 * Refactor `Linear` handling in `TransformersModel` by @hmellor in https://github.com/vllm-project/vllm/pull/12727 * [VLM] Add MLA with pure RoPE support for deepseek-vl2 models by @Isotr0py in https://github.com/vllm-project/vllm/pull/12729 * [Misc] Bump the compressed-tensors version by @dsikka in https://github.com/vllm-project/vllm/pull/12736 * [Model][Quant] Fix GLM, Fix fused module mappings for quantization by @kylesayrs in https://github.com/vllm-project/vllm/pull/12634 * [Doc] Update PR Reminder with link to Developer Slack by @mgoin in https://github.com/vllm-project/vllm/pull/12748 * [Bugfix] Fix OpenVINO model runner by @hmellor in https://github.com/vllm-project/vllm/pull/12750 * [V1][Misc] Shorten `FinishReason` enum and use constant strings by @njhill in https://github.com/vllm-project/vllm/pull/12760 * [Doc] Remove performance warning for auto_awq.md by @mgoin in https://github.com/vllm-project/vllm/pull/12743 * [Bugfix] Fix 'ModuleNotFoundError: No module named 'intel_extension_for_pytorch'' for --tensor-parallel-size more than 1 by @Akashcodes732 in https://github.com/vllm-project/vllm/pull/12546 * [core][distributed] exact ray placement control by @youkaichao in https://github.com/vllm-project/vllm/pull/12732 * [Kernel] Use self.kv_cache and forward_context.attn_metadata in Attention.forward by @heheda12345 in https://github.com/vllm-project/vllm/pull/12536 * [Hardware][Intel-Gaudi] Enable FusedSDPA support for Intel Gaudi (HPU) by @SanjuCSudhakaran in https://github.com/vllm-project/vllm/pull/12359 * Add: Support for Sparse24Bitmask Compressed Models by @rahul-tuli in https://github.com/vllm-project/vllm/pull/12097 * [VLM] Use shared field to pass token ids to model by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/12767 * [Docs] Drop duplicate [source] links by @russellb in https://github.com/vllm-project/vllm/pull/12780 * [VLM] Qwen2.5-VL by @ywang96 in https://github.com/vllm-project/vllm/pull/12604 * [VLM] Update compatibility with transformers 4.49 by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/12781 * Quantization and MoE configs for GH200 machines by @arvindsun in https://github.com/vllm-project/vllm/pull/12717 * [ROCm][Kernel] Using the correct warp_size value by @gshtras in https://github.com/vllm-project/vllm/pull/12789 * [Bugfix] Better FP8 supported defaults by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/12796 * [Misc][Easy] Remove the space from the file name by @houseroad in https://github.com/vllm-project/vllm/pull/12799 * [Model] LoRA Support for Ultravox model by @thedebugger in https://github.com/vllm-project/vllm/pull/11253 * [Bugfix] Fix the test_ultravox.py's license by @houseroad in https://github.com/vllm-project/vllm/pull/12806 * Improve `TransformersModel` UX by @hmellor in https://github.com/vllm-project/vllm/pull/12785 * [Misc] Remove duplicated DeepSeek V2/V3 model definition by @mgoin in https://github.com/vllm-project/vllm/pull/12793 * [Misc] Improve error message for incorrect pynvml by @youkaichao in https://github.com/vllm-project/vllm/pull/12809 ## New Contributors * @vicenteherrera made their first contribution in https://github.com/vllm-project/vllm/pull/12643 * @chenyang78 made their first contribution in https://github.com/vllm-project/vllm/pull/12574 * @srikanthsrnvs made their first contribution in https://github.com/vllm-project/vllm/pull/12570 * @ArthurZucker made their first contribution in https://github.com/vllm-project/vllm/pull/11330 * @mgtk77 made their first contribution in https://github.com/vllm-project/vllm/pull/12689 * @sducouedic made their first contribution in https://github.com/vllm-project/vllm/pull/12630 * @akeshet made their first contribution in https://github.com/vllm-project/vllm/pull/12368 * @arvindsun made their first contribution in https://github.com/vllm-project/vllm/pull/12717 * @thedebugger made their first contribution in https://github.com/vllm-project/vllm/pull/11253 **Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.7.1...v0.7.2