Starstruk Logo MarkStarstruk

v1.8.0

xorbitsai/inferencev1.8.0Jul 20, 2025by XprobeBot

AI Summary

This release introduces llama.cpp backend support for embeddings, non-stream tool calling for sglang, and migration support for custom models.

Key Highlights

  • Added embedding support for llama.cpp backend
  • Added non-stream tool calling for sglang
  • Added support for migrating from v1 to v2 for custom models

New Features

  • llama.cpp backend support for embeddings
  • Non-stream tool calling for sglang
  • Migration support from v1 to v2 for custom models
  • FLUX.1-Kontext-dev model support
  • ERNIE 4.5 model support
  • Jina-embeddings-v4 model support
  • GLM-4.1v-thinking model support

Full Release Notes

# What's new in 1.8.0 (2025-07-20)
These are the changes in inference v1.8.0.
## New features
* FEAT: Embedding support llama.cpp backend by @codingl2k1 in https://github.com/xorbitsai/inference/pull/3730
* FEAT: non-stream tool calling for sglang by @aniya105 in https://github.com/xorbitsai/inference/pull/3760
* FEAT: support migrate from v1 to v2 for custom models by @qinxuye in https://github.com/xorbitsai/inference/pull/3810
* FEAT: FLUX.1-Kontext-dev by @qinxuye in https://github.com/xorbitsai/inference/pull/3728
* FEAT: support ERNIE 4.5 by @qinxuye in https://github.com/xorbitsai/inference/pull/3812
* FEAT: [embedding] add support for jina-embeddings-v4 model by @Minamiyama in https://github.com/xorbitsai/inference/pull/3814
* FEAT: [model] support glm-4.1v-thinking by @llyycchhee in https://github.com/xorbitsai/inference/pull/3756
## Enhancements
* ENH: Pin xllamacpp>=0.1.23 by @codingl2k1 in https://github.com/xorbitsai/inference/pull/3780
* ENH: add modelscope for fish speech 1.5 by @qinxuye in https://github.com/xorbitsai/inference/pull/3750
* REF: [V2 BREAK] Merge multiple JSON files into one for difference model download sources by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/3765
## Bug fixes
* BUG: disable flash_attn for qwen3 embedding & rerank when no gpu available by @qinxuye in https://github.com/xorbitsai/inference/pull/3739
* BUG: Fix bugs in del async_client by @zhcn000000 in https://github.com/xorbitsai/inference/pull/3753
* BUG: add message preprocessing to ensure that content is not null by @amumu96 in https://github.com/xorbitsai/inference/pull/3791
* BUG: pre check to prevent from list index out of range for FunASR family models by @leslie2046 in https://github.com/xorbitsai/inference/pull/3809
* BUG: resolve issue where AI output was lost when no tool was selected for function call #3767 by @aniya105 in https://github.com/xorbitsai/inference/pull/3768
* BUG: fix error in `content` output at `reasoning_content`, when using `enable_thinking` in `chat_template_kwargs` by @amumu96 in https://github.com/xorbitsai/inference/pull/3794
## Documentation
* DOC: fix links by @qinxuye in https://github.com/xorbitsai/inference/pull/3774
* DOC: update info in docs by @qinxuye in https://github.com/xorbitsai/inference/pull/3779
* DOC: update models by @qinxuye in https://github.com/xorbitsai/inference/pull/3815


**Full Changelog**: https://github.com/xorbitsai/inference/compare/v1.7.1...v1.8.0