inference Releases

This release adds support for Qwen2.5-Instruct-1M, Moonlight-16B-A3B, and QwQ-32B models, along with InternVL 2.5. It introduces the `reasoning_content` parameter across all engines and enhances the UI with filtering and copy buttons.

Mar 9, 2025

A brief patch release fixing a command line bug.

Feb 22, 2025

Major feature release adding DeepSeek V3 and R1 support, distributed inference for sglang, and InternVL 2.5. It also updates dependencies (vLLM, sglang) and enhances the UI with parameter conversion and worker management.

Feb 21, 2025

This release adds support for Qwen2.5-VL-instruct, InternLM3, DeepSeek-R1-Distill-Llama, Kokoro-82M, and updates vLLM support for Qwen2.5-VL-instruct. It also fixes compatibility issues with llama.cpp quantizations and transformers continuous batching.

Feb 8, 2025

This release introduces MeloTTS support, DeepSeek-R1-Distill-Qwen, and Cline-style messages for all backends. It also includes enhancements for Whisper model config and CosyVoice2 SFT speakers.

Jan 24, 2025

This release adds HunyuanVideo and Hunyuan-DiT text-to-image support, Cline support for vLLM, and UI theme switching/i18n support. It also supports Qwen2VL on Ascend NPU and introduces KV cache sharing between VLLM replicas.

Jan 10, 2025

This release adds support for F5-TTS-MLX, Qwen2.5-Coder-Instruct, MinicPM-4B on vLLM, QvQ-72B-Preview, and SD3.5 series models. It also includes enhancements for guided decoding and F5-TTS resampling.

Dec 27, 2024

This release introduces F5 TTS support, VL model support for MLX, CLIP model support, and Llama-3.3-Instruct support. It also enhances error messages and unifies prompt text for speech models.

Dec 13, 2024

This release introduces new model support including Fish speech streaming, Whisper for Mac MLX, and various LLMs like QwQ-32B-Preview and GLM-edge-chat. It also includes bug fixes for embeddings and streaming interfaces.

Nov 29, 2024

This is a major version 1.0.0 release featuring the addition of Qwen2.5-Coder variants, basic cancel support for image models, and kvcache support in multi-round chat for MLX.

Nov 15, 2024

This update focuses on new vision capabilities, specifically adding support for Llama 3.2-Vision models, along with bug fixes for ChatTTS compatibility and logging.

Nov 8, 2024

This release introduces OpenMind Hub download support and drops Python 3.8 support in favor of Python 3.12, alongside bug fixes for rerank models and documentation updates.

Nov 1, 2024

This release adds support for GPTQ format of Qwen2.5-Coder-7B-Instruct, GOT-OCR2_0, and MLX support for Flux.1, alongside enhancements to ChatTTS and concurrent request handling.

Oct 25, 2024

This update introduces dynamic batching for FLUX.1 text-to-image, AWQ/GPTQ vLLM inference for VisionModels like Qwen2-VL, and MLX support for qwen2.5-instruct.

Oct 18, 2024

This release focuses on tool calling capabilities for Llama 3.1 and Qwen2.5, adds Whisper-large-v3-turbo audio model, and introduces progress tracking for image models.

Oct 12, 2024

This update adds support for Jina-embedding-v3, DeepCache with Stable Diffusion models, and Minicpm-reranker, alongside fixes for UI bugs and vLLM defaults.

Sep 30, 2024

This release adds support for Qwen 2.5 models and variants, fixes UI registration page bugs, and resolves issues with CosyVoice output and stable diffusion tools.

Sep 20, 2024

This release introduces extensive new features including SD API support (txt2img, img2img, inpainting), support for Minicpm-4B and Yi-coder-chat, and updates for DeepSeek V2/2.5 and Qwen2 audio.

Sep 14, 2024

Adds streaming support for CosyVoice, introduces Qwen2-VL-instruct, includes openai-whisper, and enhances MiniCPM-V-2.6 with continuous batching. Also includes various bug fixes for Docker startup and rerank calculations.

Sep 6, 2024

A hotfix release addressing Docker image startup issues and initialization failures with custom paths observed in v0.14.4. Includes documentation updates for Windows compatibility.

Sep 3, 2024