Starstruk Logo MarkStarstruk

v1.3.0.post1

xorbitsai/inferencev1.3.0.post1Feb 21, 2025by XprobeBot

AI Summary

Major feature release adding DeepSeek V3 and R1 support, distributed inference for sglang, and InternVL 2.5. It also updates dependencies (vLLM, sglang) and enhances the UI with parameter conversion and worker management.

Key Highlights

  • Support for DeepSeek V3 and R1 models.
  • Distributed inference support for sglang.
  • InternVL 2.5 support.
  • UI improvements: Parameter tips, featured models filtering, form/command line conversion, n_worker parameter.
  • Updated vLLM to v0.7.2 and sglang to v0.4.2.post4.

Breaking Changes

  • Xavier now supports `vLLM >= 0.7.0`, drops support for older versions.

New Features

  • Support qwen-2.5-instruct-1m
  • support deepseek-v3 and deepseek-r1
  • [UI] additional parameter tip function
  • [UI] add featured models filtering function
  • [UI] support form parameters and command line conversion
  • support distributed inference for sglang
  • [UI] add n_worker parameter for model launch
  • InternVL 2.5
  • support vllm reasoning content

Full Release Notes

# What's new in 1.3.0.post1 (2025-02-21)
These are the changes in inference v1.3.0.post1.
## New features
* FEAT: Support qwen-2.5-instruct-1m by @Jun-Howie in https://github.com/xorbitsai/inference/pull/2841
* FEAT: support deepseek-v3 and deepseek-r1 by @qinxuye in https://github.com/xorbitsai/inference/pull/2864
* FEAT: [UI] additional parameter tip function. by @yiboyasss in https://github.com/xorbitsai/inference/pull/2876
* FEAT: [UI] add featured models filtering function. by @yiboyasss in https://github.com/xorbitsai/inference/pull/2871
* FEAT: [UI] support form parameters and command line conversion. by @yiboyasss in https://github.com/xorbitsai/inference/pull/2850
* FEAT: support distributed inference for sglang by @qinxuye in https://github.com/xorbitsai/inference/pull/2877
* FEAT: [UI] add n_worker parameter for model launch. by @yiboyasss in https://github.com/xorbitsai/inference/pull/2889
* FEAT: InternVL 2.5 by @Minamiyama in https://github.com/xorbitsai/inference/pull/2776
* FEAT: support vllm reasoning content by @amumu96 in https://github.com/xorbitsai/inference/pull/2905
## Enhancements
* enh: add gpu utilization info by @amumu96 in https://github.com/xorbitsai/inference/pull/2852
* ENH: Update Kokoro model by @codingl2k1 in https://github.com/xorbitsai/inference/pull/2843
* ENH: cmdline supports --n-worker, add --model-path and make it compatible with --model_path by @qinxuye in https://github.com/xorbitsai/inference/pull/2890
* BLD: update sglang to v0.4.2.post4 and vllm to v0.7.2 by @qinxuye in https://github.com/xorbitsai/inference/pull/2838
* BLD: fix flashinfer installation in dockerfile by @qinxuye in https://github.com/xorbitsai/inference/pull/2844
## Bug fixes
* BUG: Fix whisper CI by @codingl2k1 in https://github.com/xorbitsai/inference/pull/2822
* BUG: fix FLUX when a scheduler is specified which is incompatible. by @shuaiqidezhong in https://github.com/xorbitsai/inference/pull/2897
* BUG: [UI] fix the bug of missing hint during model running. by @yiboyasss in https://github.com/xorbitsai/inference/pull/2904
* BUG: Clear dependency by @codingl2k1 in https://github.com/xorbitsai/inference/pull/2910
## Tests
* TST: Pin CI transformers<4.49 by @codingl2k1 in https://github.com/xorbitsai/inference/pull/2883
* TST: fix lint error by @amumu96 in https://github.com/xorbitsai/inference/pull/2911
## Documentation
* DOC: distributed inference by @qinxuye in https://github.com/xorbitsai/inference/pull/2908
## Others
* CHORE: Xavier now supports `vLLM >= 0.7.0`, drops support for older versions by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2886

## New Contributors
* @shuaiqidezhong made their first contribution in https://github.com/xorbitsai/inference/pull/2897

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v1.2.2...v1.3.0.post1