Starstruk Logo MarkStarstruk

v1.6.1

xorbitsai/inferencev1.6.1May 30, 2025by XprobeBot

AI Summary

Feature release focusing on llama.cpp backend improvements, adding multimodal support and new Deepseek models.

Key Highlights

  • llama.cpp backend now supports multimodal models
  • Auto NGL support for llama.cpp backend
  • Added Deepseek-R1-0528 and Deepseek-R1-0528-qwen3 support

New Features

  • llama.cpp backend multimodal support
  • Auto NGL for llama.cpp backend
  • UI hints for common parameters with custom input support
  • Additional Paraformer series models
  • Deepseek-R1-0528 support
  • Deepseek-R1-0528-qwen3 support
  • Modelscope support for Kolors
  • HybridCache support in transformers library for Gemma3
  • Virtualenv support for ChatTTS
  • FlashInfer version update

Full Release Notes

# What's new in 1.6.1 (2025-05-30)
These are the changes in inference v1.6.1.
## New features
* FEAT: llama.cpp backend support multimodal by @codingl2k1 in https://github.com/xorbitsai/inference/pull/3442
* FEAT: Auto ngl for llama.cpp backend by @codingl2k1 in https://github.com/xorbitsai/inference/pull/3518
* FEAT: [UI] add hint for common parameters with support for custom input. by @yiboyasss in https://github.com/xorbitsai/inference/pull/3521
* FEAT: add some other paraformer series models by @leslie2046 in https://github.com/xorbitsai/inference/pull/3536
* FEAT: support Deepseek-R1-0528 by @Jun-Howie in https://github.com/xorbitsai/inference/pull/3539
* FEAT: support deepseek-r1-0528-qwen3 by @Jun-Howie in https://github.com/xorbitsai/inference/pull/3552
## Enhancements
* ENH: [rerank] add instruction for minicpm-reranker by @llyycchhee in https://github.com/xorbitsai/inference/pull/3453
* ENH: pass extra arguments for speech2text API. by @leslie2046 in https://github.com/xorbitsai/inference/pull/3516
* ENH: add modelscope support for kolors by @qinxuye in https://github.com/xorbitsai/inference/pull/3534
* ENH: remove check when specified GPU index for vllm by @kota-iizuka in https://github.com/xorbitsai/inference/pull/3527
* ENH: Supports `HybridCache` in `transformers` lib, mainly for `gemma3` chat model by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/3538
* ENH: support virtualenv for chattts by @qinxuye in https://github.com/xorbitsai/inference/pull/3541
* BLD: fix setup.cfg by @qinxuye in https://github.com/xorbitsai/inference/pull/3467
* BLD: update  flashinfer version by @amumu96 in https://github.com/xorbitsai/inference/pull/3549
* REF: Refactor for multimodal llm models by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/3462
## Bug fixes
* BUG: fix input for jina clip by @llyycchhee in https://github.com/xorbitsai/inference/pull/3440
* BUG: [ui] delete cache file white screen bug. by @yiboyasss in https://github.com/xorbitsai/inference/pull/3482
* BUG: fix import_submodules, ignore test files by @Gmgge in https://github.com/xorbitsai/inference/pull/3545
## Documentation
* DOC: remove llama-cpp-python related doc & refine model_ability parts by @qinxuye in https://github.com/xorbitsai/inference/pull/3519
* DOC: Update doc about cosyvoice-2.0 stream and auto NGL by @codingl2k1 in https://github.com/xorbitsai/inference/pull/3547

## New Contributors
* @kota-iizuka made their first contribution in https://github.com/xorbitsai/inference/pull/3527

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v1.6.0...v1.6.1