vllm Releases

This release marks the introduction of pipeline parallelism and support for Gemma 2, Jamba, Deepseek-V2, and Phi3-Vision. It also adds initial TPU integration and an OpenVINO backend. A significant breaking change affects VLM arguments.

Jul 5, 2024

A post-release patch that addresses critical bugs, specifically adding initial TPU integration and fixing crashes related to FlashAttention and device count mismatches.

Jun 14, 2024

Major features include support for Falcon, IBM Granite, Snowflake arctic, and dynamic RoPE scaling. The release also introduces the Embedding API, OpenAI batch format support, and significant performance optimizations for FP8, Marlin, and Speculative Decoding.

Jun 1, 2024

Highlights include the ready-for-testing chunked prefill feature, speculative decoding functionalities (logprobs, ngram), and the addition of Phi-3-mini support. The release also expands Marlin kernel support and integrates FlashInfer as an attention backend.

May 5, 2024

A hotfix release restoring sm70/75 support which was missing in v0.4.0. It includes performance optimizations for layernorm, adds CPU inference backend, and introduces lookahead scheduling for speculative decoding.

Apr 2, 2024

This release adds StarCoder2 support, Gemma performance optimizations and LoRA support, and 2/3/8-bit GPTQ support. It integrates Marlin kernels for Int4 GPTQ inference, optimizes MoE kernels, and adds experimental AWS Inferentia2 support.

Mar 1, 2024

Adds support for the OLMo and Gemma models, introduces the seed parameter, and enables multi-LoRA support in the OpenAI server.

Feb 21, 2024

This version fixes a major memory leak in distributed execution by using CuPY for collective communication and adds Python 3.8 support. It also includes various bug fixes and metric improvements.

Feb 16, 2024

This release focuses on performance optimization, achieving up to 70% throughput improvement in distributed inference by removing serialization overheads, while also fixing critical tensor parallelism support for Mixtral models.

Jan 4, 2024

This release introduces fast model execution capabilities using CUDA/HIP graphs and adds support for W4A16 GPTQ quantization, alongside several bug fixes for Mixtral weight loading and memory profiling.

Dec 17, 2023

Performance optimization for Mixtral models via expert parallelism and fixes for long context input positions are the main highlights, alongside a new lazy model loader and PyTorch upgrade.

Dec 14, 2023

This release marks the official addition of Mixtral 8x7B support and introduces AMD GPU (ROCm) support, significantly expanding the platform's compatibility for both new models and hardware.

Dec 11, 2023

A major refactoring of core components like Worker and Attention improves internal architecture, while explicitly fixing tensor parallelism support for AWQ models and adding Prometheus metrics.

Dec 3, 2023

This release features a significant upgrade to PyTorch 2.1 and CUDA 12.1, extensive refactoring for better quantization support, and the addition of several new models including Yi, ChatGLM, and Phi.

Nov 19, 2023

An emergency release specifically designed to address and fix a critical bug affecting tensor parallelism support.

Oct 17, 2023

A minor release focused on bug fixes to ensure stability across various models, specifically addressing issues in ALiBi, Falcon-40B, and Code Llama.

Sep 11, 2023

An emergency release to revert a previous breaking API change that would have caused existing code using AsyncLLMServer to stop working.

Sep 8, 2023

This release stabilizes the AsyncLLMEngine with a background loop, aligns beam search behavior with HuggingFace, and adds support for CodeLLaMA.

Sep 7, 2023

v0.1.4 significantly improves usability by shipping pre-built CUDA binaries, eliminating the need for users to compile CUDA kernels locally. The release introduces support for three new models and includes optimizations for CUDA kernels and sampling parameters.

Aug 25, 2023