v0.12.2

xorbitsai/inferencev0.12.2Jun 21, 2024by XprobeBot

AI Summary

This update focuses on tool support for Qwen MOE models and continuous batching for all models using the transformers backend. It also improves UI capabilities for custom models and rerank token usage.

Key Highlights

Tools support for Qwen MOE models
Continuous batching for all transformers models
Rerank model token input/output usage
UI custom model management improvements

New Features

Add Tools Support for Qwen Series MOE Models
Modify the deletion function of a custom model
Custom model presents JSON data and modifies it
Add Rerank model token input/output usage
Continuous batching supports all the models with transformers backend

Full Release Notes

# What's new in 0.12.2 (2024-06-21)
These are the changes in inference v0.12.2.
## New features
* FEAT: Add Tools Support for Qwen Series MOE Models by @zhanghx0905 in https://github.com/xorbitsai/inference/pull/1642
* FEAT: [UI]Modify the deletion function of a custom model. by @yiboyasss in https://github.com/xorbitsai/inference/pull/1656
* FEAT: [UI]Custom model presents JSON data and modifies it. by @yiboyasss in https://github.com/xorbitsai/inference/pull/1670
* FEAT: Add Rerank model token input/output usage by @wxiwnd in https://github.com/xorbitsai/inference/pull/1657
## Enhancements
* ENH: Continuous batching supports all the models with `transformers` backend by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1659
## Bug fixes
* BUG: show error when user launch quantized model without device supported by @Minamiyama in https://github.com/xorbitsai/inference/pull/1645
* BUG: Fix default rerank type by @codingl2k1 in https://github.com/xorbitsai/inference/pull/1649
* BUG: chat_completion not response while error appears more than 100 by @liuzhenghua in https://github.com/xorbitsai/inference/pull/1663
## Tests
* TST: Fix CI due to `tenacity` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1660
## Others
* CHORE: [pre-commit] Add exclude thirdparty rules by @frostyplanet in https://github.com/xorbitsai/inference/pull/1678


**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.12.1...v0.12.2