v0.4.0.post1

vllm-project/vllmv0.4.0.post1Apr 2, 2024by github-actions[bot]

AI Summary

A hotfix release restoring sm70/75 support which was missing in v0.4.0. It includes performance optimizations for layernorm, adds CPU inference backend, and introduces lookahead scheduling for speculative decoding.

Key Highlights

  • Restoration of sm70/75 support
  • Layernorm performance optimization
  • CPU inference backend support
  • Speculative decoding lookahead scheduling

New Features

  • sm70/75 support
  • Layernorm optimization
  • CPU backend
  • Speculative decoding lookahead

Full Release Notes

## Highlight

v0.4.0 lacks support for sm70/75 support. We did a hotfix for it.

## What's Changed
* [Kernel] Layernorm performance optimization by @mawong-amd in https://github.com/vllm-project/vllm/pull/3662
* [Doc] Update installation doc for build from source and explain the dependency on torch/cuda version by @youkaichao in https://github.com/vllm-project/vllm/pull/3746
* [CI/Build] Make Marlin Tests Green by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/3753
* [Misc] Minor fixes in requirements.txt by @WoosukKwon in https://github.com/vllm-project/vllm/pull/3769
* [Misc] Some minor simplifications to detokenization logic by @njhill in https://github.com/vllm-project/vllm/pull/3670
* [Misc] Fix Benchmark TTFT Calculation for Chat Completions by @ywang96 in https://github.com/vllm-project/vllm/pull/3768
* [Speculative decoding 4/9] Lookahead scheduling for speculative decoding by @cadedaniel in https://github.com/vllm-project/vllm/pull/3250
* [Misc] Add support for new autogptq checkpoint_format by @Qubitium in https://github.com/vllm-project/vllm/pull/3689
* [Misc] [CI/Build] Speed up block manager CPU-only unit tests ~10x by opting-out of GPU cleanup by @cadedaniel in https://github.com/vllm-project/vllm/pull/3783
* [Hardware][Intel] Add CPU inference backend by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/3634
* [HotFix] [CI/Build] Minor fix for CPU backend CI by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/3787
* [Frontend][Bugfix] allow using the default middleware with a root path by @A-Mahla in https://github.com/vllm-project/vllm/pull/3788
* [Doc] Fix vLLMEngine Doc Page by @ywang96 in https://github.com/vllm-project/vllm/pull/3791
* [CI/Build] fix TORCH_CUDA_ARCH_LIST in wheel build by @youkaichao in https://github.com/vllm-project/vllm/pull/3801
* Fix crash when try torch.cuda.set_device in worker by @leiwen83 in https://github.com/vllm-project/vllm/pull/3770
* [Bugfix] Add `__init__.py` files for `vllm/core/block/` and `vllm/spec_decode/` by @mgoin in https://github.com/vllm-project/vllm/pull/3798
* [CI/Build] 0.4.0.post1, fix sm 7.0/7.5 binary by @youkaichao in https://github.com/vllm-project/vllm/pull/3803

## New Contributors
* @mawong-amd made their first contribution in https://github.com/vllm-project/vllm/pull/3662
* @Qubitium made their first contribution in https://github.com/vllm-project/vllm/pull/3689
* @bigPYJ1151 made their first contribution in https://github.com/vllm-project/vllm/pull/3634
* @A-Mahla made their first contribution in https://github.com/vllm-project/vllm/pull/3788

**Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.4.0...v0.4.0.post1