v0.6.3.post1

vllm-project/vllmv0.6.3.post1Oct 17, 2024by github-actions[bot]

AI Summary

A bugfix release addressing critical issues with Pixtral models, chat API statistics, and Molmo text-only input, alongside the addition of new models like Ministral 3B/8B and VLM2Vec.

Key Highlights

  • New Models: Ministral 3B and Ministral 8B via interleaved attention
  • New Models: Support multiple and interleaved images for Llama3.2
  • New Models: VLM2Vec, the first multimodal embedding model in vLLM
  • Important bug fix: Fix chat API continuous usage stats
  • Important bug fix: Fix vLLM UsageInfo and logprobs None AssertionError
  • Important bug fix: Fix Molmo text-only input bug
  • Important bug fix: Fix CUDA 11.8 Build
  • Other Enhancements: Remove block manager v1 and make block manager v2 default
  • Other Enhancements: Spec Decode Optimize ngram lookup performance

New Features

  • Support Ministral 3B and Ministral 8B via interleaved attention
  • Support multiple and interleaved images for Llama3.2
  • Support VLM2Vec, the first multimodal embedding model in vLLM
  • Fix chat API continuous usage stats
  • Fix vLLM UsageInfo and logprobs None AssertionError with empty token_ids
  • Fix Molmo text-only input bug
  • Fix CUDA 11.8 Build
  • Remove block manager v1 and make block manager v2 default
  • Spec Decode Optimize ngram lookup performance

Full Release Notes

## Highlights

### New Models
* Support Ministral 3B and Ministral 8B via interleaved attention (#9414)
* Support multiple and interleaved images for Llama3.2 (#9095)
* Support VLM2Vec, the first multimodal embedding model in vLLM (#9303)

### Important bug fix
* Fix chat API continuous usage stats (#9357)
* Fix vLLM UsageInfo and logprobs None AssertionError with empty token_ids (#9034)
* Fix Molmo text-only input bug (#9397)
* Fix CUDA 11.8 Build (#9386)
* Fix `_version.py` not found issue (#9375)

### Other Enhancements
* Remove block manager v1 and make block manager v2 default (#8704)
* Spec Decode Optimize ngram lookup performance (#9333)


## What's Changed
* [TPU] Fix TPU SMEM OOM by Pallas paged attention kernel by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9350
* [Frontend] merge beam search implementations by @LunrEclipse in https://github.com/vllm-project/vllm/pull/9296
* [Model] Make llama3.2 support multiple and interleaved images by @xiangxu-google in https://github.com/vllm-project/vllm/pull/9095
* [Bugfix] Clean up some cruft in mamba.py by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9343
* [Frontend] Clarify model_type error messages by @stevegrubb in https://github.com/vllm-project/vllm/pull/9345
* [Doc] Fix code formatting in spec_decode.rst by @mgoin in https://github.com/vllm-project/vllm/pull/9348
* [Bugfix] Update InternVL input mapper to support image embeds by @hhzhang16 in https://github.com/vllm-project/vllm/pull/9351
* [BugFix] Fix chat API continuous usage stats by @njhill in https://github.com/vllm-project/vllm/pull/9357
* pass ignore_eos parameter to all benchmark_serving calls by @gracehonv in https://github.com/vllm-project/vllm/pull/9349
* [Misc] Directly use compressed-tensors for checkpoint definitions by @mgoin in https://github.com/vllm-project/vllm/pull/8909
* [Bugfix] Fix vLLM UsageInfo and logprobs None AssertionError with empty token_ids by @CatherineSue in https://github.com/vllm-project/vllm/pull/9034
* [Bugfix][CI/Build] Fix CUDA 11.8 Build by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/9386
* [Bugfix] Molmo text-only input bug fix by @mrsalehi in https://github.com/vllm-project/vllm/pull/9397
* [Misc] Standardize RoPE handling for Qwen2-VL by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9250
* [Model] VLM2Vec, the first multimodal embedding model in vLLM by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9303
* [CI/Build] Test VLM embeddings by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9406
* [Core] Rename input data types by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/8688
* [Misc] Consolidate example usage of OpenAI client for multimodal models by @ywang96 in https://github.com/vllm-project/vllm/pull/9412
* [Model] Support SDPA attention for Molmo vision backbone by @Isotr0py in https://github.com/vllm-project/vllm/pull/9410
* Support mistral interleaved attn by @patrickvonplaten in https://github.com/vllm-project/vllm/pull/9414
* [Kernel][Model] Improve continuous batching for Jamba and Mamba by @mzusman in https://github.com/vllm-project/vllm/pull/9189
* [Model][Bugfix] Add FATReLU activation and support for openbmb/MiniCPM-S-1B-sft by @streaver91 in https://github.com/vllm-project/vllm/pull/9396
* [Performance][Spec Decode] Optimize ngram lookup performance by @LiuXiaoxuanPKU in https://github.com/vllm-project/vllm/pull/9333
* [CI/Build] mypy: Resolve some errors from checking vllm/engine by @russellb in https://github.com/vllm-project/vllm/pull/9267
* [Bugfix][Kernel] Prevent integer overflow in fp8 dynamic per-token quantize kernel by @tlrmchlsmth in https://github.com/vllm-project/vllm/pull/9425
* [BugFix] [Kernel] Fix GPU SEGV occurring in int8 kernels by @rasmith in https://github.com/vllm-project/vllm/pull/9391
* Add notes on the use of Slack by @terrytangyuan in https://github.com/vllm-project/vllm/pull/9442
* [Kernel] Add Exllama as a backend for compressed-tensors  by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/9395
* [Misc] Print stack trace using `logger.exception` by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/9461
* [misc] CUDA Time Layerwise Profiler by @LucasWilkinson in https://github.com/vllm-project/vllm/pull/8337
* [Bugfix] Allow prefill of assistant response when using `mistral_common` by @sasha0552 in https://github.com/vllm-project/vllm/pull/9446
* [TPU] Call torch._sync(param) during weight loading by @WoosukKwon in https://github.com/vllm-project/vllm/pull/9437
* [Hardware][CPU] compressed-tensor INT8 W8A8 AZP support  by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/9344
* [Core] Deprecating block manager v1 and make block manager v2 default by @KuntaiDu in https://github.com/vllm-project/vllm/pull/8704
* [CI/Build] remove .github from .dockerignore, add dirty repo check by @dtrifiro in https://github.com/vllm-project/vllm/pull/9375

## New Contributors
* @gracehonv made their first contribution in https://github.com/vllm-project/vllm/pull/9349
* @streaver91 made their first contribution in https://github.com/vllm-project/vllm/pull/9396

**Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.6.3...v0.6.3.post1