v0.6.1.post1

vllm-project/vllmv0.6.1.post1Sep 13, 2024by github-actions[bot]

AI Summary

A bugfix release addressing Pixtral model issues, implementing multistep scheduling, and fixing tool use logic and multi-image support for Qwen-VL.

Key Highlights

  • Bug fixes for Pixtral models
  • Multistep scheduling implementation
  • Tool use logic fixes
  • Support multiple images for qwen-vl
  • Removes engine_use_ray
  • Add engine option to return only deltas or final output
  • Add bitsandbytes support for Gemma2

New Features

  • Pixtral model fixes (max position embeddings, multiple images, chunked prefill disabled)
  • Multistep scheduling implementation
  • Tool use logic fixes
  • Support multiple images for qwen-vl
  • Removes engine_use_ray
  • Add engine option to return only deltas or final output
  • Add bitsandbytes support for Gemma2

Full Release Notes

## Highlights
This release features important bug fixes and enhancements for 
- Pixtral models. (#8415, #8425, #8399, #8431)
	- Chunked scheduling has been turned off for vision models. Please replace `--max_num_batched_tokens 16384` with `--max-model-len 16384`
- Multistep scheduling. (#8417, #7928, #8427)
- Tool use. (#8423, #8366)

Also
* support multiple images for qwen-vl (#8247)
* removes `engine_use_ray` (#8126)
* add engine option to return only deltas or final output (#7381)
* add bitsandbytes support for Gemma2 (#8338)


## What's Changed
* [MISC] Dump model runner inputs when crashing by @comaniac in https://github.com/vllm-project/vllm/pull/8305
* [misc] remove engine_use_ray by @youkaichao in https://github.com/vllm-project/vllm/pull/8126
* [TPU] Use Ray for default distributed backend by @WoosukKwon in https://github.com/vllm-project/vllm/pull/8389
* Fix the AMD weight loading tests by @mgoin in https://github.com/vllm-project/vllm/pull/8390
* [Bugfix]: Fix the logic for deciding if tool parsing is used by @tomeras91 in https://github.com/vllm-project/vllm/pull/8366
* [Gemma2] add bitsandbytes support for Gemma2 by @blueyo0 in https://github.com/vllm-project/vllm/pull/8338
* [Misc] Raise error when using encoder/decoder model with cpu backend by @kevin314 in https://github.com/vllm-project/vllm/pull/8355
* [Misc] Use RoPE cache for MRoPE by @WoosukKwon in https://github.com/vllm-project/vllm/pull/8396
* [torch.compile] hide slicing under custom op for inductor by @youkaichao in https://github.com/vllm-project/vllm/pull/8384
* [Hotfix][VLM] Fixing max position embeddings for Pixtral by @ywang96 in https://github.com/vllm-project/vllm/pull/8399
* [Bugfix] Fix InternVL2 inference with various num_patches by @Isotr0py in https://github.com/vllm-project/vllm/pull/8375
* [Model] Support multiple images for qwen-vl by @alex-jw-brooks in https://github.com/vllm-project/vllm/pull/8247
* [BugFix] lazy init _copy_stream to avoid torch init wrong gpu instance by @lnykww in https://github.com/vllm-project/vllm/pull/8403
* [BugFix] Fix Duplicate Assignment of Class Variable in Hermes2ProToolParser by @vegaluisjose in https://github.com/vllm-project/vllm/pull/8423
* [Bugfix] Offline mode fix by @joerunde in https://github.com/vllm-project/vllm/pull/8376
* [multi-step] add flashinfer backend by @SolitaryThinker in https://github.com/vllm-project/vllm/pull/7928
* [Core] Add engine option to return only deltas or final output by @njhill in https://github.com/vllm-project/vllm/pull/7381
* [Bugfix] multi-step + flashinfer: ensure cuda graph compatible  by @alexm-neuralmagic in https://github.com/vllm-project/vllm/pull/8427
* [Hotfix][Core][VLM] Disable chunked prefill by default and prefix caching for multimodal models by @ywang96 in https://github.com/vllm-project/vllm/pull/8425
* [CI/Build] Disable multi-node test for InternVL2 by @ywang96 in https://github.com/vllm-project/vllm/pull/8428
* [Hotfix][Pixtral] Fix multiple images bugs by @patrickvonplaten in https://github.com/vllm-project/vllm/pull/8415
* [Bugfix] Fix weight loading issue by rename variable. by @wenxcs in https://github.com/vllm-project/vllm/pull/8293
* [Misc] Update Pixtral example by @ywang96 in https://github.com/vllm-project/vllm/pull/8431
* [BugFix] fix group_topk by @dsikka in https://github.com/vllm-project/vllm/pull/8430
* [Core] Factor out input preprocessing to a separate class by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/7329
* [Bugfix] Mapping physical device indices for e2e test utils by @ShangmingCai in https://github.com/vllm-project/vllm/pull/8290
* [Bugfix] Bump fastapi and pydantic version by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/8435
* [CI/Build] Update pixtral tests to use JSON by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/8436
* [Bugfix] Fix async log stats by @alexm-neuralmagic in https://github.com/vllm-project/vllm/pull/8417
* [bugfix] torch profiler bug for single gpu with GPUExecutor by @SolitaryThinker in https://github.com/vllm-project/vllm/pull/8354
* bump version to v0.6.1.post1 by @simon-mo in https://github.com/vllm-project/vllm/pull/8440

## New Contributors
* @blueyo0 made their first contribution in https://github.com/vllm-project/vllm/pull/8338
* @lnykww made their first contribution in https://github.com/vllm-project/vllm/pull/8403
* @vegaluisjose made their first contribution in https://github.com/vllm-project/vllm/pull/8423

**Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.6.1...v0.6.1.post1