v0.8.1

vllm-project/vllmv0.8.1Mar 19, 2025by github-actions[bot]

AI Summary

A patch release addressing critical bug fixes for the V1 engine, including sampling parameter fixes and TPU support improvements.

Key Highlights

  • Ensure using int64 for sampled token ids in V1
  • Fix long dtype in topk sampling
  • Refactor Structured Output for multiple backends
  • Fix chunked prefill with padding for TPU
  • Re-enable Gemma3 for V1

New Features

  • Embedding model support LoRA
  • Add support for Zamba2 models

Full Release Notes

This release contains important bug fixes for v0.8.0. We highly recommend upgrading! 

* V1 Fixes
	* Ensure using int64 for sampled token ids (#15065)
	* Fix long dtype in topk sampling (#15049)
	* Refactor Structured Output for multiple backends (#14694)
	* Fix size calculation of processing cache (#15114)
	* Optimize Rejection Sampler with Triton Kernels (#14930)
	* Fix oracle for device checking (#15104)

* TPU
	* Fix chunked prefill with padding (#15037)
	* Enhanced CI/CD (#15054, 14974)

* Model
	* Re-enable Gemma3 for V1 (#14980)
	* Embedding model support LoRA (#14935)
	* Pixtral: Remove layer instantiation duplication (#15053)


## What's Changed
* [Bugfix] Fix interface for Olmo2 on V1 by @ywang96 in https://github.com/vllm-project/vllm/pull/14976
* [CI/Build] Use `AutoModelForImageTextToText` to load image models in tests by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/14945
* [V1] Guard Against Main Thread Usage by @robertgshaw2-redhat in https://github.com/vllm-project/vllm/pull/14972
* [V1] TPU - Fix CI/CD runner for V1 and remove V0 tests by @alexm-redhat in https://github.com/vllm-project/vllm/pull/14974
* [Bugfix] Fix bnb quantization for models with both HF-format and Mistral-format weights by @tristanleclercq in https://github.com/vllm-project/vllm/pull/14950
* [Neuron] trim attention kernel tests to fit trn1.2x instance by @liangfu in https://github.com/vllm-project/vllm/pull/14988
* [Doc][V1] Fix V1 APC doc by @shen-shanshan in https://github.com/vllm-project/vllm/pull/14920
* [Kernels] LoRA - Retire SGMV and BGMV Kernels by @varun-sundar-rabindranath in https://github.com/vllm-project/vllm/pull/14685
* [Mistral-Small 3.1] Update docs and tests by @patrickvonplaten in https://github.com/vllm-project/vllm/pull/14977
* [Misc] Embedding model support LoRA by @jeejeelee in https://github.com/vllm-project/vllm/pull/14935
* [Bugfix] torchrun compatibility by @hiyouga in https://github.com/vllm-project/vllm/pull/14899
* [Bugfix][Frontend] Fix validation of `logprobs` in `ChatCompletionRequest` by @schoennenbeck in https://github.com/vllm-project/vllm/pull/14352
* [Misc][Docs] fix the comments of KV_T and CACHE_T in CALL_RESHAPE_AND_CACHE_XX macros by @yangsijia-serena in https://github.com/vllm-project/vllm/pull/14347
* [Bugfix] Loosen type check to avoid errors in V1 by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15021
* [Bugfix] Register serializers for V0 MQ Engine by @simon-mo in https://github.com/vllm-project/vllm/pull/15009
* [TPU][V1][Bugfix] Fix chunked prefill with padding by @NickLucche in https://github.com/vllm-project/vllm/pull/15037
* MI325 configs, fused_moe_kernel bugfix by @ekuznetsov139 in https://github.com/vllm-project/vllm/pull/14987
* [MODEL] Add support for Zamba2 models by @yury-tokpanov in https://github.com/vllm-project/vllm/pull/13185
* [Bugfix] Fix broken CPU quantization due to triton import by @Isotr0py in https://github.com/vllm-project/vllm/pull/15038
* [Bugfix] Fix LoRA extra vocab size by @jeejeelee in https://github.com/vllm-project/vllm/pull/15047
* [V1] Refactor Structured Output for multiple backends by @russellb in https://github.com/vllm-project/vllm/pull/14694
* [V1][Spec Decode] Optimize Rejection Sampler with Triton Kernels by @WoosukKwon in https://github.com/vllm-project/vllm/pull/14930
* [V1] TPU - CI/CD use smaller model by @alexm-redhat in https://github.com/vllm-project/vllm/pull/15054
* fix long dtype in topk sampling by @chujiezheng in https://github.com/vllm-project/vllm/pull/15049
* [Doc] Minor v1_user_guide update by @JenZhao in https://github.com/vllm-project/vllm/pull/15064
* [Misc][V1] Skip device checking if not available by @comaniac in https://github.com/vllm-project/vllm/pull/15061
* [Model] Pixtral: Remove layer instantiation duplication by @juliendenize in https://github.com/vllm-project/vllm/pull/15053
* [Model] Remove duplicated message check in Mistral chat completion request by @b8zhong in https://github.com/vllm-project/vllm/pull/15069
* [Core] Update dtype detection and defaults by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/14858
* [V1] Ensure using int64 for sampled token ids by @WoosukKwon in https://github.com/vllm-project/vllm/pull/15065
* [Bugfix] Re-enable Gemma3 for V1 by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/14980
* [CI][Intel GPU] update XPU dockerfile and CI script by @jikunshang in https://github.com/vllm-project/vllm/pull/15109
* [V1][Bugfix] Fix oracle for device checking by @ywang96 in https://github.com/vllm-project/vllm/pull/15104
* [Misc] Avoid unnecessary HF `do_rescale` warning when passing dummy data by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15107
* [Bugfix] Fix size calculation of processing cache by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/15114
* [Doc] Update tip info on using latest transformers when creating a custom Dockerfile  by @MarcCote in https://github.com/vllm-project/vllm/pull/15070
* [Misc][Benchmark] Add support for different `tokenizer_mode` by @aarnphm in https://github.com/vllm-project/vllm/pull/15040
* [Bugfix] Adjust mllama to regional compilation by @jkaniecki in https://github.com/vllm-project/vllm/pull/15112
* [Doc] Update the "the first vLLM China Meetup" slides link to point to the first page by @imkero in https://github.com/vllm-project/vllm/pull/15134
* [Frontend] Remove custom_cache_manager by @fulvius31 in https://github.com/vllm-project/vllm/pull/13791
* [V1] Minor V1 async engine test refactor by @andoorve in https://github.com/vllm-project/vllm/pull/15075

## New Contributors
* @tristanleclercq made their first contribution in https://github.com/vllm-project/vllm/pull/14950
* @hiyouga made their first contribution in https://github.com/vllm-project/vllm/pull/14899
* @ekuznetsov139 made their first contribution in https://github.com/vllm-project/vllm/pull/14987
* @yury-tokpanov made their first contribution in https://github.com/vllm-project/vllm/pull/13185
* @juliendenize made their first contribution in https://github.com/vllm-project/vllm/pull/15053
* @MarcCote made their first contribution in https://github.com/vllm-project/vllm/pull/15070
* @jkaniecki made their first contribution in https://github.com/vllm-project/vllm/pull/15112
* @fulvius31 made their first contribution in https://github.com/vllm-project/vllm/pull/13791

**Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.8.0...v0.8.1