v0.4.0.post1
vllm-project/vllmv0.4.0.post1Apr 2, 2024by github-actions[bot]
AI Summary
A hotfix release restoring sm70/75 support which was missing in v0.4.0. It includes performance optimizations for layernorm, adds CPU inference backend, and introduces lookahead scheduling for speculative decoding.
Key Highlights
- Restoration of sm70/75 support
- Layernorm performance optimization
- CPU inference backend support
- Speculative decoding lookahead scheduling
New Features
- sm70/75 support
- Layernorm optimization
- CPU backend
- Speculative decoding lookahead
Full Release Notes
## Highlight v0.4.0 lacks support for sm70/75 support. We did a hotfix for it. ## What's Changed * [Kernel] Layernorm performance optimization by @mawong-amd in https://github.com/vllm-project/vllm/pull/3662 * [Doc] Update installation doc for build from source and explain the dependency on torch/cuda version by @youkaichao in https://github.com/vllm-project/vllm/pull/3746 * [CI/Build] Make Marlin Tests Green by @robertgshaw2-neuralmagic in https://github.com/vllm-project/vllm/pull/3753 * [Misc] Minor fixes in requirements.txt by @WoosukKwon in https://github.com/vllm-project/vllm/pull/3769 * [Misc] Some minor simplifications to detokenization logic by @njhill in https://github.com/vllm-project/vllm/pull/3670 * [Misc] Fix Benchmark TTFT Calculation for Chat Completions by @ywang96 in https://github.com/vllm-project/vllm/pull/3768 * [Speculative decoding 4/9] Lookahead scheduling for speculative decoding by @cadedaniel in https://github.com/vllm-project/vllm/pull/3250 * [Misc] Add support for new autogptq checkpoint_format by @Qubitium in https://github.com/vllm-project/vllm/pull/3689 * [Misc] [CI/Build] Speed up block manager CPU-only unit tests ~10x by opting-out of GPU cleanup by @cadedaniel in https://github.com/vllm-project/vllm/pull/3783 * [Hardware][Intel] Add CPU inference backend by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/3634 * [HotFix] [CI/Build] Minor fix for CPU backend CI by @bigPYJ1151 in https://github.com/vllm-project/vllm/pull/3787 * [Frontend][Bugfix] allow using the default middleware with a root path by @A-Mahla in https://github.com/vllm-project/vllm/pull/3788 * [Doc] Fix vLLMEngine Doc Page by @ywang96 in https://github.com/vllm-project/vllm/pull/3791 * [CI/Build] fix TORCH_CUDA_ARCH_LIST in wheel build by @youkaichao in https://github.com/vllm-project/vllm/pull/3801 * Fix crash when try torch.cuda.set_device in worker by @leiwen83 in https://github.com/vllm-project/vllm/pull/3770 * [Bugfix] Add `__init__.py` files for `vllm/core/block/` and `vllm/spec_decode/` by @mgoin in https://github.com/vllm-project/vllm/pull/3798 * [CI/Build] 0.4.0.post1, fix sm 7.0/7.5 binary by @youkaichao in https://github.com/vllm-project/vllm/pull/3803 ## New Contributors * @mawong-amd made their first contribution in https://github.com/vllm-project/vllm/pull/3662 * @Qubitium made their first contribution in https://github.com/vllm-project/vllm/pull/3689 * @bigPYJ1151 made their first contribution in https://github.com/vllm-project/vllm/pull/3634 * @A-Mahla made their first contribution in https://github.com/vllm-project/vllm/pull/3788 **Full Changelog**: https://github.com/vllm-project/vllm/compare/v0.4.0...v0.4.0.post1