Starstruk Logo MarkStarstruk

v0.15.0

xorbitsai/inferencev0.15.0Sep 6, 2024by XprobeBot

AI Summary

Adds streaming support for CosyVoice, introduces Qwen2-VL-instruct, includes openai-whisper, and enhances MiniCPM-V-2.6 with continuous batching. Also includes various bug fixes for Docker startup and rerank calculations.

Key Highlights

  • CosyVoice model support for streaming reply
  • Qwen2-VL-instruct model support
  • MiniCPM-V-2.6 continuous batching support
  • Refinement of request log and optional request_id

New Features

  • CosyVoice streaming reply support
  • Qwen2-VL-instruct support
  • OpenAI Whisper integration

Full Release Notes

# What's new in 0.15.0 (2024-09-06)
These are the changes in inference v0.15.0.
## New features
* FEAT: cosyvoice model support streaming reply by @wuminghui-coder in https://github.com/xorbitsai/inference/pull/2192
* FEAT: support qwen2-vl-instruct by @Minamiyama in https://github.com/xorbitsai/inference/pull/2205
## Enhancements
* ENH: include openai-whisper into thirdparty by @qinxuye in https://github.com/xorbitsai/inference/pull/2232
* ENH: `MiniCPM-V-2.6` Supports continuous batching with transformers engine by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2238
* ENH: unpad for image2image/inpainting model by @wxiwnd in https://github.com/xorbitsai/inference/pull/2229
* ENH: Refine request log and add optional request_id by @frostyplanet in https://github.com/xorbitsai/inference/pull/2173
* REF: Use `chat_template` for LLM instead of `prompt_style` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2193
## Bug fixes
* BUG: Fix docker image startup issue due to entrypoint by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2207
* BUG: fix init xinference fail when custom path is fault by @amumu96 in https://github.com/xorbitsai/inference/pull/2208
* BUG: use `default_uid` to replace `uid` of actors which may override the xoscar actor's uid property by @qinxuye in https://github.com/xorbitsai/inference/pull/2214
* BUG: fix rerank max length by @qinxuye in https://github.com/xorbitsai/inference/pull/2219
* BUG: logger bug of function using generator decoration by @wxiwnd in https://github.com/xorbitsai/inference/pull/2215
* BUG: fix rerank calculation of tokens number by @qinxuye in https://github.com/xorbitsai/inference/pull/2228
* BUG: fix embedding token calculation & optimize memory by @qinxuye in https://github.com/xorbitsai/inference/pull/2221
## Documentation
* DOC: Modify the installation documentation to change single quotes to double quotes for Windows compatibility. by @nikelius in https://github.com/xorbitsai/inference/pull/2211
## Others
* Revert "EHN: clean cache for VL models (#2163)" by @qinxuye in https://github.com/xorbitsai/inference/pull/2230
* CHORE: Docker image is only pushed to aliyun when releasing version  by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2216
* CHORE: Compatible with `openai >= 1.40` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2231

## New Contributors
* @nikelius made their first contribution in https://github.com/xorbitsai/inference/pull/2211
* @wuminghui-coder made their first contribution in https://github.com/xorbitsai/inference/pull/2192

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.14.4...v0.15.0