Starstruk Logo MarkStarstruk

v0.11.0

xorbitsai/inferencev0.11.0May 11, 2024by XprobeBot

AI Summary

A major update requiring users to specify a `model_engine` when launching models. It introduces support for Mixtral-8x22b, Phi-3, and Ascend hardware, along with significant performance improvements and refactoring.

Key Highlights

  • Mandatory `model_engine` parameter for launching models
  • Support for Mixtral-8x22b-instruct-v0.1 and Phi-3-mini
  • Ascend hardware support
  • Enhanced streaming outputs for vision models
  • Removal of speculative decoding code

Breaking Changes

  • v0.11.0 introduced break change when launching model that `model_engine` should be specified

New Features

  • Support Mixtral-8x22b-instruct-v0.1
  • add phi-3-mini series
  • Add Starling model
  • support qwen1.5 110b
  • Support query engine with cmdline
  • Ascend support
  • Audio support verbose_json and timestamp
  • [UI] Add engine option when launching LLM

Full Release Notes

# What's new in 0.11.0 (2024-05-11)
These are the changes in inference v0.11.0.
## Break Changes
v0.11.0 introduced break change when launching model that `model_engine` should be specified, refer to [Model Engine](https://inference.readthedocs.io/en/latest/getting_started/using_xinference.html#about-model-engine) for more information
## New features
* FEAT: Support Mixtral-8x22b-instruct-v0.1 by @qinxuye in https://github.com/xorbitsai/inference/pull/1340
* feat: add phi-3-mini series by @orangeclk in https://github.com/xorbitsai/inference/pull/1379
* FEAT: add Starling model by @boy-hack in https://github.com/xorbitsai/inference/pull/1384
* FEAT: support qwen1.5 110b by @qinxuye in https://github.com/xorbitsai/inference/pull/1388
* FEAT: Support query engine with cmdline by @Ago327 in https://github.com/xorbitsai/inference/pull/1380
* FEAT: Ascend support by @qinxuye in https://github.com/xorbitsai/inference/pull/1408
* FEAT: Audio support verbose_json and timestamp by @codingl2k1 in https://github.com/xorbitsai/inference/pull/1402
* FEAT: [UI] Add engine option when launching LLM by @yiboyasss in https://github.com/xorbitsai/inference/pull/1456
## Enhancements
* ENH: add custom image model by @amumu96 in https://github.com/xorbitsai/inference/pull/1312
* ENH: Support more quantization with VLLM by @amumu96 in https://github.com/xorbitsai/inference/pull/1372
* ENH: Update chatglm3 6b model version by @codingl2k1 in https://github.com/xorbitsai/inference/pull/1401
* ENH: make qwen_vl support streaming output by @Minamiyama in https://github.com/xorbitsai/inference/pull/1425
* ENH: Removed the max tokens limitation and boost performance by avoid unnecessary repeated cuda device detection. by @mikeshi80 in https://github.com/xorbitsai/inference/pull/1429
* ENH: Improve benchmark and add long context generate by @frostyplanet in https://github.com/xorbitsai/inference/pull/1423
* ENH: make yi_vl support streaming output by @Minamiyama in https://github.com/xorbitsai/inference/pull/1443
* ENH: Some minor changes by @frostyplanet in https://github.com/xorbitsai/inference/pull/1453
* ENH: make deepseek_vl support streaming output by @Minamiyama in https://github.com/xorbitsai/inference/pull/1444
* ENH: Rename `model_engine` for more clear inference backend by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1466
* BLD: Use self-hosted aws machine to build docker image by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1405
* CLN: Remove actor client by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1436
* CLN: Remove all speculative-related codes by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1435
* REF: Query for engine by @Ago327 in https://github.com/xorbitsai/inference/pull/1342
* REF: [UI] Refactor register model by @yiboyasss in https://github.com/xorbitsai/inference/pull/1368
* REF: Add the `model_engine` parameter for launching process by @hainaweiben in https://github.com/xorbitsai/inference/pull/1367
## Bug fixes
* BUG: Fix llama3-instruct 70B filename error by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1370
* BUG: no role:user msg or content empty got an error. by @liuzhenghua in https://github.com/xorbitsai/inference/pull/1378
* BUG: fix file template of andrewcanis/c4ai-command-r-v01-GGUF by @emulated24 in https://github.com/xorbitsai/inference/pull/1389
* BUG: Fix using extra gpus due to match in `__init__` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1400
* BUG: Fix qwen tool call paramerter empty issue by @codingl2k1 in https://github.com/xorbitsai/inference/pull/1381
* BUG: Fix tool calls return invalid usage by @codingl2k1 in https://github.com/xorbitsai/inference/pull/1420
* BUG: Fix tools ability by @mikeshi80 in https://github.com/xorbitsai/inference/pull/1447
* BUG: Install error on MacOS due to `auto-gptq` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1457
* BUG: fix some issues in query engine interface by @Ago327 in https://github.com/xorbitsai/inference/pull/1442
## Tests
* TST: Pin `huggingface-hub` to pass CI since it has some break changes by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1427
## Documentation
* DOC: update readme & fix Mac CI by @qinxuye in https://github.com/xorbitsai/inference/pull/1385
* DOC: worker address should be specified for `xinference-worker` by @amumu96 in https://github.com/xorbitsai/inference/pull/1397
* DOC: update docker doc in using xinference by @qinxuye in https://github.com/xorbitsai/inference/pull/1417
* DOC: add the missing backslash in shell command by @mikeshi80 in https://github.com/xorbitsai/inference/pull/1451
* DOC: Usage about `model_engine` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1468
## Others
* BUG：Fix mertics is empty when call `/v1/chat/completions` by @amumu96 in https://github.com/xorbitsai/inference/pull/1406

## New Contributors
* @liuzhenghua made their first contribution in https://github.com/xorbitsai/inference/pull/1378
* @emulated24 made their first contribution in https://github.com/xorbitsai/inference/pull/1389
* @orangeclk made their first contribution in https://github.com/xorbitsai/inference/pull/1379
* @boy-hack made their first contribution in https://github.com/xorbitsai/inference/pull/1384
* @frostyplanet made their first contribution in https://github.com/xorbitsai/inference/pull/1423

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.10.3...v0.11.0