Starstruk Logo MarkStarstruk

v0.14.0

xorbitsai/inferencev0.14.0Aug 2, 2024by XprobeBot

AI Summary

Major update adding support for launching models via model_path, gte-Qwen2-7B-instruct multi-GPU deployment, SGLang support for Llama 3 and Qwen 2, and image-to-image capabilities.

Key Highlights

  • Support model_path input when launching models
  • gte-Qwen2-7B-instruct multi-GPU support
  • SGLang support for Llama 3 and Qwen 2
  • Image-to-image support

New Features

  • model_path input support
  • gte-Qwen2-7B-instruct multi-GPU support
  • SGLang support
  • Image-to-image support

Full Release Notes

# What's new in 0.14.0 (2024-08-02)
These are the changes in inference v0.14.0.
## New features
* FEAT: Supports model_path input when launching models by @Valdanitooooo in https://github.com/xorbitsai/inference/pull/1918
* FEAT: Support gte-Qwen2-7B-instruct and multi gpu deploy by @amumu96 in https://github.com/xorbitsai/inference/pull/1994
## Enhancements
* ENH: Add support of sglang for llama 3 qwen 2 by @luweizheng in https://github.com/xorbitsai/inference/pull/1947
* ENH: add cache_limit_gb option for MLX by @qinxuye in https://github.com/xorbitsai/inference/pull/1954
* ENH: [benchmark] Add api-key support by @frostyplanet in https://github.com/xorbitsai/inference/pull/1961
* ENH: Support for Gemma 2 and Llama 3.1 Models for vllm & sglang by @vikrantrathore in https://github.com/xorbitsai/inference/pull/1929
* ENH: [K8s] worker log dir name by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1997
* ENH: support image_to_image by @qinxuye in https://github.com/xorbitsai/inference/pull/1986
* REF: enable sglang by default by @qinxuye in https://github.com/xorbitsai/inference/pull/1953
## Bug fixes
* BUG: Fix GLM chat by @codingl2k1 in https://github.com/xorbitsai/inference/pull/1966
* BUG: fix match for transformers from model registered by @qinxuye in https://github.com/xorbitsai/inference/pull/1955
* BUG: Load llama.so failed in docker image by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1974
* BUG: [UI]Modifying 'model format' again resulted in an error message. by @yiboyasss in https://github.com/xorbitsai/inference/pull/1990
* BUG: fix loading multiple gguf parts by @qinxuye in https://github.com/xorbitsai/inference/pull/1987
## Documentation
* DOC: ascend support by @qinxuye in https://github.com/xorbitsai/inference/pull/1978
* DOC: add CosyVoice doc by @qinxuye in https://github.com/xorbitsai/inference/pull/1980
* DOC: Documents for K8s by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/2004

## New Contributors
* @vikrantrathore made their first contribution in https://github.com/xorbitsai/inference/pull/1929
* @Valdanitooooo made their first contribution in https://github.com/xorbitsai/inference/pull/1918

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.13.3...v0.14.0