Starstruk Logo MarkStarstruk

v0.14.2

xorbitsai/inferencev0.14.2Aug 16, 2024by XprobeBot

AI Summary

Adds support for Gemma-2-it and InternLM2.5, enables FP8 support for vLLM and SGLang, and enhances InternVL2 video support. Removes some old builtin models.

Key Highlights

  • Gemma-2-it and InternLM2.5 support
  • FP8 support for vLLM & SGLang engine
  • InternVL2 and stream support

New Features

  • Gemma-2-it support
  • InternLM2.5 support
  • FP8 support for vLLM & SGLang
  • InternVL2 and stream support

Full Release Notes

# What's new in 0.14.2 (2024-08-16)
These are the changes in inference v0.14.2.
## New features
* FEAT: add gemma-2-it 2b & internlm2.5-chat 1.8b and 20b & update video and sglang docs by @qinxuye in https://github.com/xorbitsai/inference/pull/2080
* FEAT: support FP8 for vllm & sglang engine by @qinxuye in https://github.com/xorbitsai/inference/pull/2069
* Feat: Support internvl2 and internvl stream by @amumu96 in https://github.com/xorbitsai/inference/pull/2079
## Enhancements
* ENH: make MiniCPM v2.6 support video by @Minamiyama in https://github.com/xorbitsai/inference/pull/2068
* REF: Remove some builtin old models and `ggmlv3` model format by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2086
## Bug fixes
* BUG: limit AutoAWQ version to fix docker issue by @qinxuye in https://github.com/xorbitsai/inference/pull/2067
* BUG: Fix custom glm4 & remove tool calls of ChatGLM3 by @codingl2k1 in https://github.com/xorbitsai/inference/pull/2081
* BUG: Infinited loop with login by @WalkerWang731 in https://github.com/xorbitsai/inference/pull/2039
## Documentation
* DOC: update readme & add tips for large image models by @qinxuye in https://github.com/xorbitsai/inference/pull/2056

## New Contributors
* @WalkerWang731 made their first contribution in https://github.com/xorbitsai/inference/pull/2039

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.14.1...v0.14.2