Starstruk Logo MarkStarstruk

v1.4.1

xorbitsai/inferencev1.4.1Apr 3, 2025by XprobeBot

AI Summary

Feature release focusing on distributed inference, DeepSeek-VL2 support, and internal quantization library updates.

Key Highlights

  • Distributed inference support for vLLM
  • DeepSeek-VL2 model support
  • Replacement of auto-gptq with gptqmodel

New Features

  • Fin-R1 model support
  • Distributed inference for vLLM
  • GPTQ (int4, int8) and FP8 support for Fin-R1
  • Fixed quantization parameter in vLLM engine
  • SGLang vision support
  • max_completion_tokens support
  • DeepSeek-VL2 support
  • Qwen2.5-vl-32b support
  • SGLang GPTQ int8 quantization
  • Qwen2.5-vl-32b-awq support
  • GPTQModel support to replace auto-gptq

Full Release Notes

# What's new in 1.4.1 (2025-04-03)
These are the changes in inference v1.4.1.
## New features
* FEAT: Support Fin-R1 model by @Jun-Howie in https://github.com/xorbitsai/inference/pull/3116
* FEAT: distributed inference for vLLM by @qinxuye in https://github.com/xorbitsai/inference/pull/3120
* FEAT: Support gptq(int4, int8) and fp8 for Fin-R1 model by @Jun-Howie in https://github.com/xorbitsai/inference/pull/3157
* feat: fix the quantization parameter in the vLLM engine cannot work by @amumu96 in https://github.com/xorbitsai/inference/pull/3159
* FEAT: sglang vision by @Minamiyama in https://github.com/xorbitsai/inference/pull/3150
* FEAT: support max_completion_tokens by @amumu96 in https://github.com/xorbitsai/inference/pull/3168
* FEAT: support DeepSeek-VL2 by @Jun-Howie in https://github.com/xorbitsai/inference/pull/3179
## Enhancements
* ENH: support for qwen2.5-vl-32b by @Minamiyama in https://github.com/xorbitsai/inference/pull/3119
* ENH: sglang supports gptq int8 quantization now by @Minamiyama in https://github.com/xorbitsai/inference/pull/3149
* ENH: Add validation of n_worker by @rexjm in https://github.com/xorbitsai/inference/pull/3166
* ENH: add qwen2.5-vl-32b-awq supported, and fix 7b-awq download hub typo by @Minamiyama in https://github.com/xorbitsai/inference/pull/3169
* BLD: use gptqmodel to replace auto-gptq by @qinxuye in https://github.com/xorbitsai/inference/pull/3147
* BLD: resolve docker fail by @amumu96 in https://github.com/xorbitsai/inference/pull/3164
## Bug fixes
* BUG: Fix PyTorch TypeError: Make _ModelWrapper Inherit from nn.Module by @JamesFlare1212 in https://github.com/xorbitsai/inference/pull/3131
* BUG: fix llm stream response by @amumu96 in https://github.com/xorbitsai/inference/pull/3115
* BUG: prevent potential stop hang for distributed vllm inference by @qinxuye in https://github.com/xorbitsai/inference/pull/3180
## Documentation
* DOC: update models by @qinxuye in https://github.com/xorbitsai/inference/pull/3146

## New Contributors
* @JamesFlare1212 made their first contribution in https://github.com/xorbitsai/inference/pull/3131
* @rexjm made their first contribution in https://github.com/xorbitsai/inference/pull/3166

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v1.4.0...v1.4.1