Starstruk Logo MarkStarstruk

v0.9.2

xorbitsai/inferencev0.9.2Mar 8, 2024by XprobeBot

AI Summary

This release focuses on distributed inference capabilities and model flexibility. It introduces new features for GGUF file handling and LoRA support, alongside UI improvements for GPU layer management.

Key Highlights

  • Added LoRA support for LLM and image models
  • Implemented a Docker Compose distributed example with multiple workers
  • Enhanced llama.cpp integration with `n_gpu_layers` parameter and UI controls
  • Added support for downloading and merging multiple parts of GGUF files

New Features

  • Command/SDK interface to query available models
  • Docker-compose-distributed example
  • Support for downloading and merging multiple parts of GGUF files
  • Supports LoRA for LLM and image models

Full Release Notes

# What's new in 0.9.2 (2024-03-08)
These are the changes in inference v0.9.2.
## New features
* FEAT: Add a command / SDK interface to query which models are able to… by @hainaweiben in https://github.com/xorbitsai/inference/pull/1076
* FEAT: add a docker-compose-distributed example with multiple workers by @bufferoverflow in https://github.com/xorbitsai/inference/pull/1064
* FEAT: Support download and merge multiple parts of gguf files by @notsyncing in https://github.com/xorbitsai/inference/pull/1075
* FEAT: Supports LoRA for LLM and image models by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1080
## Enhancements
* ENH: Supports `n_gpu_layers` parameter for `llama-cpp-python` by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1070
* ENH: Add a dropdown to the web UI to support adjusting GPU offload layers for llama.cpp loader by @notsyncing in https://github.com/xorbitsai/inference/pull/1073
* ENH: [UI] Show `replica` on running model page by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1093
* ENH: Add "[DONE]" to the end of stream generation for better openai SDK compatibility by @ZhangTianrong in https://github.com/xorbitsai/inference/pull/1062
* ENH: [UI] Support setting `CPU` when selecting n_gpu by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1096
## Documentation
* DOC: Extra parameters for launching models by @aresnow1 in https://github.com/xorbitsai/inference/pull/1077
* DOC: contribution doc by @Ago327 in https://github.com/xorbitsai/inference/pull/1092
* DOC: doc for lora by @ChengjieLi28 in https://github.com/xorbitsai/inference/pull/1103
## Others
* Update llm_family.json to correct the context length of glaive coder by @mikeshi80 in https://github.com/xorbitsai/inference/pull/1083

## New Contributors
* @mikeshi80 made their first contribution in https://github.com/xorbitsai/inference/pull/1083
* @bufferoverflow made their first contribution in https://github.com/xorbitsai/inference/pull/1064
* @Ago327 made their first contribution in https://github.com/xorbitsai/inference/pull/1092

**Full Changelog**: https://github.com/xorbitsai/inference/compare/v0.9.1...v0.9.2