Starstruk Logo MarkStarstruk

v1.4.0

xorbitsai/inferencev1.4.0Mar 21, 2025by XprobeBot

AI Summary

This release introduces support for Gemma-3 text and vision models, along with DeepSeek V3 function calling. It includes various UI enhancements, bug fixes for different backends (vLLM, xllamacpp, MLX), and documentation updates.

Key Highlights

  • Support for Gemma-3 text and vision models (Gemma-3-it).
  • DeepSeek V3 function calling support.
  • xllamacpp backend exception handling improvements.
  • UI updates including dark mode fixes and GPU count terminology changes.
  • Fixes for vLLM prefix caching and DeepSeek stream API.

New Features

  • Support gemma-3 text part
  • Gemma-3-it that supports vision
  • add deepseek v3 function calling

Full Release Notes

# What's new in 1.4.0 (2025-03-21)
These are the changes in inference v1.4.0.
## New features
* FEAT: Support gemma-3 text part by @zky001 in https://github.com/xorbitsai/inference/pull/3077
* FEAT: Gemma-3-it that supports vision by @qinxuye in https://github.com/xorbitsai/inference/pull/3102
* FEAT: add deepseek v3 function calling by @rogercloud in https://github.com/xorbitsai/inference/pull/3103
## Enhancements
* ENH: xllamacpp backend raise exception if failed by @codingl2k1 in https://github.com/xorbitsai/inference/pull/3053
* ENH: [UI] change 'GPU Count' to 'GPU Count per Replica'. by @yiboyasss in https://github.com/xorbitsai/inference/pull/3078
## Bug fixes
* BUG: [UI] fix dark mode bugs. by @yiboyasss in https://github.com/xorbitsai/inference/pull/3028
* BUG: fix Internvl2.5-mpo awq, fix model card info typo by @Minamiyama in https://github.com/xorbitsai/inference/pull/3067
* BUG: fix max_tokens for MLX VL models. by @qinxuye in https://github.com/xorbitsai/inference/pull/3072
* BUG:fix vLLM parameter "enable_prefix_caching" by @Gmgge in https://github.com/xorbitsai/inference/pull/3081
* BUG: fix first token error and support deepseek stream api by @amumu96 in https://github.com/xorbitsai/inference/pull/3090
## Documentation
* DOC: add auth usage guide for http request by @Minamiyama in https://github.com/xorbitsai/inference/pull/3065
* DOC: add xllamacpp related docs by @qinxuye in https://github.com/xorbitsai/inference/pull/3088
## Others
* FIX: [UI] remove the restriction of model_format on n_gpu for llama.cpp by @yiboyasss in https://github.com/xorbitsai/inference/pull/3050

## New Contributors
* @Gmgge made their first contribution in https://github.com/xorbitsai/inference/pull/3081
* @zky001 made their first contribution in https://github.com/xorbitsai/inference/pull/3077
* @rogercloud made their first contribution in https://github.com/xorbitsai/inference/pull/3103

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v1.3.1...v1.4.0