Starstruk Logo MarkStarstruk

v0.16.0

xorbitsai/inferencev0.16.0Oct 18, 2024by XprobeBot

AI Summary

This update introduces dynamic batching for FLUX.1 text-to-image, AWQ/GPTQ vLLM inference for VisionModels like Qwen2-VL, and MLX support for qwen2.5-instruct.

Key Highlights

  • Implemented dynamic batching for FLUX.1 text-to-image.
  • Added AWQ/GPTQ vLLM inference support for VisionModels (e.g., Qwen2-VL).
  • Added MLX support for qwen2.5-instruct.

New Features

  • Adding support for awq/gptq vLLM inference to VisionModel such as Qwen2-VL
  • Dynamic batching for the state-of-the-art FLUX.1 `text_to_image` interface
  • added MLX for qwen2.5-instruct

Full Release Notes

# What's new in 0.16.0 (2024-10-18)
These are the changes in inference v0.16.0.
## New features
* FEAT: Adding support for awq/gptq vLLM inference to VisionModel such as Qwen2-VL by @cyhasuka in https://github.com/xorbitsai/inference/pull/2445
* FEAT: Dynamic batching for the state-of-the-art FLUX.1 `text_to_image` interface by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2380
* FEAT: added MLX for qwen2.5-instruct by @qinxuye in https://github.com/xorbitsai/inference/pull/2444
## Enhancements
* ENH: Speed up cli interaction by @frostyplanet in https://github.com/xorbitsai/inference/pull/2443
* REF: Enable continuous batching for LLM with transformers engine by default by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2437
## Documentation
* DOC: update readme & docs by @qinxuye in https://github.com/xorbitsai/inference/pull/2435

## New Contributors
* @cyhasuka made their first contribution in https://github.com/xorbitsai/inference/pull/2445

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.15.4...v0.16.0