Starstruk Logo MarkStarstruk

v0.10.1

xorbitsai/inferencev0.10.1Apr 12, 2024by XprobeBot

AI Summary

This release adds support for Qwen1.5 32B chat and Qwen MoE models, with improvements to streaming tool calls and multi-GPU support.

Key Highlights

  • Support for Qwen1.5 32B chat and Qwen MoE
  • Streaming in tool calls for Qwen
  • Multi-GPU support for vision models
  • Refactored Web UI launch model interface

New Features

  • add support for qwen1.5 32B chat model
  • Support Qwen MoE model for huggingface and modelscope
  • Enable streaming in tool calls for Qwen when using vllm
  • support multi gpus for qwen-vl and yi-vl
  • support llamacpp multiple gpu

Full Release Notes

# What's new in 0.10.1 (2024-04-12)
These are the changes in inference v0.10.1.
## New features
* FEAT: add support for qwen1.5 32B chat model by @mikeshi80 in https://github.com/xorbitsai/inference/pull/1249
* FEAT: Support Qwen MoE model for huggingface and modelscope by @xiaodouzi666 in https://github.com/xorbitsai/inference/pull/1263
* FEAT: Enable streaming in tool calls for Qwen when using vllm by @zhanghx0905 in https://github.com/xorbitsai/inference/pull/1215
## Enhancements
* ENH: make function create_embedding could receive extra args by @amumu96 in https://github.com/xorbitsai/inference/pull/1224
* ENH: support more GPTQ and AWQ format for some models by @xiaodouzi666 in https://github.com/xorbitsai/inference/pull/1243
* ENH: support multi gpus for qwen-vl and yi-vl by @qinxuye in https://github.com/xorbitsai/inference/pull/1236
* ENH: support llamacpp multiple gpu by @amumu96 in https://github.com/xorbitsai/inference/pull/1229
* ENH: UI: paper material for cards by @Minamiyama in https://github.com/xorbitsai/inference/pull/1261
* REF: Refactor launch model for Web UI by @yiboyasss in https://github.com/xorbitsai/inference/pull/1254
* REF: Remove ctransformers supports by @mujin2 in https://github.com/xorbitsai/inference/pull/1267
## Bug fixes
* BUG: Fix docker cpu build by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1213
* BUG: Fix cannot start xinference in docker due to `cv2` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1217
* BUG: Cannot start xinference in docker by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1219
* BUG: Fix `opencv` issue in docker container by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1227
* BUG: Fix the launch bug of OmnilMM 12B. by @hainaweiben in https://github.com/xorbitsai/inference/pull/1241
* BUG: style spell error by @Minamiyama in https://github.com/xorbitsai/inference/pull/1247
* BUG: Fix issue with supervisor not clearing information after worker exit by @hainaweiben in https://github.com/xorbitsai/inference/pull/1231
* BUG: custom models on the web ui by @yiboyasss in https://github.com/xorbitsai/inference/pull/1259
* BUG: fix system prompts for chatglm3 and internlm2 pytorch by @qinxuye in https://github.com/xorbitsai/inference/pull/1271
* BUG: Fix authority and jump issue by @yiboyasss in https://github.com/xorbitsai/inference/pull/1276
* BUG: fix custom vision model by @qinxuye in https://github.com/xorbitsai/inference/pull/1280
## Tests
* TST: Fix tests due to `llama-cpp-python` `v0.2.58` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1242
## Documentation
* DOC: auto gen vllm doc & add chatglm3-{32k, 128k} support for vllm by @qinxuye in https://github.com/xorbitsai/inference/pull/1234
* DOC: update models doc by @qinxuye in https://github.com/xorbitsai/inference/pull/1246
* DOC: update readme by @qinxuye in https://github.com/xorbitsai/inference/pull/1268

## New Contributors
* @amumu96 made their first contribution in https://github.com/xorbitsai/inference/pull/1224
* @xiaodouzi666 made their first contribution in https://github.com/xorbitsai/inference/pull/1243
* @yiboyasss made their first contribution in https://github.com/xorbitsai/inference/pull/1254

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.10.0...v0.10.1