Starstruk Logo MarkStarstruk

v0.13.0

xorbitsai/inferencev0.13.0Jul 5, 2024by XprobeBot

AI Summary

A major update that introduces the MLX engine for Apple Silicon, GGUF file support for Qwen2, and continuous batching for vision models. It also adds Aliyun Docker image support and various stability improvements.

Key Highlights

  • MLX engine support
  • GGUF files for Qwen2
  • Continuous batching for vision models
  • Aliyun Docker image support

New Features

  • Support MLX engine
  • Add gemma-2-it
  • Added gguf files for qwen2
  • Continuous batching supports vision model ability
  • Supports Aliyun docker image

Full Release Notes

# What's new in 0.13.0 (2024-07-05)
These are the changes in inference v0.13.0.
## New features
* FEAT: support MLX engine by @qinxuye in https://github.com/xorbitsai/inference/pull/1765
* FEAT: add gemma-2-it by @qinxuye in https://github.com/xorbitsai/inference/pull/1774
## Enhancements
* ENH: added gguf files for qwen2 by @qinxuye in https://github.com/xorbitsai/inference/pull/1745
* ENH: Add more log modules by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1771
* ENH: Continuous batching supports `vision` model ability by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1724
* ENH: Add guard for model launching by @frostyplanet in https://github.com/xorbitsai/inference/pull/1680
* BLD: Supports Aliyun docker image by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1753
* BLD: GPU docker use `vllm` image as base by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1759
* BLD: Pin `llama-cpp-python` to `v0.2.77` in Docker for stability by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1767
## Bug fixes
* BUG: Fix glm4 tool call by @codingl2k1 in https://github.com/xorbitsai/inference/pull/1747
* BUG: [UI] Fix authentication mode related bugs by @yiboyasss in https://github.com/xorbitsai/inference/pull/1772
* BUG: Fix python client returns documents for rerank task by default by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1780
* BUG: Fix LLM based reranker may raise a TypeError by @codingl2k1 in https://github.com/xorbitsai/inference/pull/1794
* BUG: fix deepseek-vl-chat by @qinxuye in https://github.com/xorbitsai/inference/pull/1795
## Tests
* TST: Fix `llama-cpp-python` issue in CI by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1763
## Documentation
* DOC: Update continuous batching and docker usage by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1785


**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.12.3...v0.13.0