3.5.0

# MNN 3.5.0 版本发布说明

**发布日期**: 2026年4月

## 📌 版本概述

MNN 3.5.0 版本聚焦于 **多后端 LLM 推理能力扩展**、**高性能量化与采样优化** 以及 **端侧语音交互体验增强** 三大核心主题：

*   **多后端 LLM 推理**: Vulkan 后端新增 LLM 推理支持；摩尔线程 MUSA 后端全新接入；QNN 后端支持 Attention 算子和更多 LLM 算子；RISC-V 向量扩展（RVV）支持落地。
*   **高性能量化与采样优化**: 全新 TurboQuant TQ3/TQ4 KV Cache 量化方案；Sampler 重构为 Pipeline 架构并新增 Penalty 支持；多端 TopKV2 算子优化；多轮对话 Prompt Cache 支持。
*   **端侧语音交互增强**: 异步三阶段 Token2Wav 流水线（DiT ∥ Vocoder）；智能打断（AEC Barge-in）支持；实时视觉语音交互能力；TTS 模型支持（supertonic-tts-mnn）。

---

## 🚀 版本亮点

*   **Vulkan LLM 推理**: Vulkan 后端全面支持 LLM 推理，拓展 GPU 推理覆盖面
*   **MUSA 后端支持**: 全新接入摩尔线程 MUSA GPU 后端，支持国产 GPU 推理
*   **RISC-V RVV 支持**: CPU 后端新增 RISC-V 向量扩展支持，拓展 RISC-V 生态
*   **TurboQuant TQ3/TQ4**: 全新 KV Cache 量化方案，降低内存占用的同时保持推理质量
*   **Sampler Pipeline 重构**: 采样器重构为流水线架构，支持 Penalty 等高级采样策略
*   **多轮对话 Prompt Cache**: 文本级别 Prompt Cache，显著加速多轮对话首 Token 延迟
*   **异步 Token2Wav 三阶段流水线**: DiT 与 Vocoder 并行的三阶段流水线，大幅提升语音合成吞吐
*   **智能语音打断（AEC Barge-in）**: 支持回声消除的智能语音打断，提升语音交互自然度
*   **Tokenizer 全面重构**: minja → jinja 模板引擎迁移，二进制 tokenizer 格式（tokenizer.mtok），加载速度 20x+ 提升
*   **GLM-OCR / LFM 模型支持**: 新增 GLM-OCR 和 LFM（Liquid Foundation Model）模型推理支持
*   **Metal/CUDA/OpenCL TopKV2**: 多端 TopKV2 算子实现与优化

---

## ✨ 新功能

### LLM/VLM
*   **多轮对话 Prompt Cache**: 新增文本级别 Prompt Cache，多轮对话场景下显著加速首 Token 生成
*   **GLM-OCR 支持**: 新增 GLM-OCR 模型推理支持
*   **LFM 支持**: 新增 Liquid Foundation Model (LFM) 推理支持
*   **Qwen3.5 Smooth/Omni 导出**: 支持 Qwen3.5 的 Smooth 量化和 Omni 模态导出
*   **Sampler Pipeline 重构**: 采样器重构为流水线架构，新增 Penalty 机制支持
*   **异步 Token2Wav 三阶段流水线**: 将 Token 到音频的转换拆分为 DiT 和 Vocoder 并行的三阶段流水线
*   **异步 Token2Wav 多运行时支持**: Talker 的 LLM 和处理器使用不同运行时时支持异步 Token2Wav
*   **Tokenizer 重构**: 全面重构 Tokenizer 系统：minja → jinja 模板引擎迁移；新增二进制 tokenizer 格式（tokenizer.mtok）替代文本格式；统一 JSON 处理（ujson.hpp）；加载速度提升 20x+，编解码性能数倍提升；对齐 HuggingFace Transformers 实现
*   **Linear Attention 磁盘存储**: 支持 Linear Attention 循环状态的磁盘读写
*   **mmap_size 配置**: 新增 mmap_size 配置项，支持控制内存映射大小
*   **llm_bench Profile**: llm_bench 新增 Profile 功能

### GPU 后端
*   **Vulkan LLM 推理**: Vulkan 后端全面支持 LLM 模型推理
*   **MUSA 后端**: 全新接入摩尔线程 MUSA GPU 后端
*   **Metal TopKV2**: Metal 后端新增 TopKV2 算子实现
*   **OpenCL TopKV2**: OpenCL 后端新增 TopKV2 算子实现
*   **CUDA TopKV2 优化**: 使用堆排序替代插入排序并结合循环展开，大幅提升 TopKV2 性能
*   **Metal While/BatchMatMul 4x4 分块**: 通过 4x4 寄存器分块优化 While 和 BatchMatMul 算子
*   **Metal gemv/Binary/Scale 优化**: 优化 gemv localsize、Binary 平面调度和 Scale 无偏置路径；移除 defer encoder
*   **OpenCL LinearAttention 优化**: 优化 OpenCL LinearAttention 算子性能
*   **Metal LinearAttention 优化**: 优化 Metal LinearAttention 算子性能
*   **OpenCL 内核创建保护**: 为所有 OpenCL 算子添加内核创建保护机制
*   **QNN Attention 支持**: QNN 后端支持 LLM Attention 算子，含 max_history_token 配置
*   **QNN 新增算子**: QNN 后端新增 Transpose/UnPack/Pack 算子支持
*   **CUDA Qwen3.5 优化**: CUDA 后端针对 Qwen3.5 模型进行优化

### CPU 后端
*   **RISC-V RVV 支持**: CPU 后端新增 RISC-V 向量扩展（RVV）支持
*   **TurboQuant TQ3/TQ4**: 新增 TQ3/TQ4 KV Cache 量化方案
*   **LinearAttention NEON 优化**: 使用 NEON 汇编和 FP16 优化 LinearAttention 算子

### 应用与工具
*   **智能语音打断（AEC Barge-in）**: MNNChat 支持基于回声消除的智能语音打断
*   **实时视觉语音交互**: MNNChat 实现交互式语音聊天的实时视觉能力
*   **TTS 模型支持**: Android MNNChat 支持 supertonic-tts-mnn 模型
*   **繁体中文支持**: Android 应用新增繁体中文（zh-rTW）语言支持
*   **F-Droid 上架**: 新增 Fastlane 元数据，支持 F-Droid 应用商店上架
*   **CDN 自动上传**: 新增基于 ali-oss 的 CDN 自动上传和市场配置上传脚本

---

## 🐛 缺陷修复

*   **Prefix Disk Cache**: 修复首次响应后前缀磁盘缓存未加载的问题
*   **Jinja 渲染上下文**: 修复 enable_thinking 等 Jinja 上下文未传递到模板渲染的问题
*   **Jinja 数组拼接**: 修复 Jinja 二元加法处理数组时的错误
*   **mmap 权重死锁**: 增强状态验证，避免部分初始化的 mmap 权重导致死锁
*   **KV Cache 污染**: 修复 OmniQuant 校准前向传播中的 KV Cache 污染问题
*   **LLM 文件加载检查**: Llm::load 中新增文件存在性和权限检查
*   **LLM 状态检查**: 修复 LLM 运行状态检查逻辑
*   **MoE 专家属性获取**: 修复 transformers@5.3.0 中 MoE 专家 getattr 兼容性问题
*   **llmexport Windows 崩溃**: 修复 Windows 下使用 os.dup2 导致的 llmexport 崩溃
*   **llm_bench 修复**: 使用 context 中的 n_gen 修复 llm_bench 统计
*   **Qwen3-Embedding QNN 导出**: 修复 Qwen3-Embedding 的 QNN 导出流程
*   **离线量化余弦距离**: 修复离线量化中余弦距离计算错误
*   **int4 Embedding 计算**: 修复 int4 Embedding 计算 size 错误
*   **KV Cache 元数据拷贝**: 元数据不一致时不再复制 KV Cache
*   **GeometryGather 崩溃**: 修复 GPU 场景下 const indice 的 GeometryGather 崩溃
*   **Metal 外部偏置**: 修复读取 float 外部偏置的错误
*   **Metal Attention 测试**: 修复 Attention 算子测试问题
*   **MNNConvert NC4HW4**: 修复部分 Shape 算子计算 ONNX NC4HW4 输入错误
*   **MNNDump2Json 溢出**: 修复 MNNDump2Json 中的整数溢出
*   **ONNX Shape start/end**: 支持 ONNX Shape 算子的 start/end 参数并保持 OpParameter 兼容性
*   **Pipeline::execute 编译错误**: 修复 info.group 导致的编译错误
*   **Windows 编译错误**: 修复 LLM Windows 编译错误
*   **MNNChat 多项修复**: 修复 Android 流式步骤终止、模型设置路径重置、添加本地模型崩溃、聊天自动滚动等多项问题

---

## 📚 其他改进

*   **代码规范**: 新增 `.clang-format` 配置和 pre-commit hooks；CI 新增格式化检查
*   **安全策略**: 新增 `SECURITY.md` 安全策略文档和 PR 模板
*   **文档重构**: 重构文档结构，改善导航和新手体验
*   **Zread 徽章**: 所有 README 文件新增 Zread 徽章
*   **Demo 增强**: Demo 新增输入张量维度类型检测，支持 CAFFE 和 TENSORFLOW 格式
*   **CI**: MNN 发布工作流忽略 Android 标签

---

## 📊 LLM 性能测试

### Android 平台

<img width="2400" height="2000" alt="mnn-3 5 0-anrdoid-bench" src="https://github.com/user-attachments/assets/34b41a3d-2dbb-4cd8-a0c8-a7eeea8f38af" />

### iOS 平台

<img width="2400" height="2000" alt="mnn-3 5 0-ios-bench" src="https://github.com/user-attachments/assets/38cb1f5a-d5d7-4400-bea8-5e6b445042ff" />

---

## 🙏 致谢

衷心感谢所有贡献者对本版本的宝贵贡献：

*   **@jxt1234** - Metal/OpenCL TopKV2、Metal 性能优化、QNN Attention/算子支持、GeometryGather 修复、MNNConvert 修复
*   **@Jinde.Song** - MNNChat Android 多项功能与修复、CDN 自动上传、版本发布稳定性
*   **@EricMoin** - 异步 Token2Wav 三阶段流水线、Talker 运行时管理重构
*   **@ᡠᠵᡠᡳ ᡠᠵᡠ ᠮᠠᠨᡩ᠋ᠠᠨ** - F-Droid 上架、CI 工作流改进
*   **@YinHanke** - Qwen3.5 Smooth/Omni 导出、Qwen3-Embedding QNN 修复、ONNX Shape 兼容性
*   **@Jed Lee(lijunde)** - 实时视觉语音交互、智能语音打断（AEC）、TTS 模型支持
*   **@rainy liu** - CUDA TopKV2 堆排序优化、Metal While/BatchMatMul 4x4 分块优化
*   **@hebangwen** - Pipeline::execute 编译修复、llm_bench 修复
*   **@Jan Slominski** - 多轮对话 Prompt Cache
*   **@dongyang-mt** - 摩尔线程 MUSA 后端支持
*   **@Sherlockzhangjinge** - RISC-V RVV 向量扩展支持
*   **@JackWinterCN** - 离线量化余弦距离修复
*   **@luckysmg** - Prefix Disk Cache 修复
*   **@Riverspring996** - mmap 权重死锁修复
*   **@N Wang** - KV Cache 污染修复
*   **@Xiaochao Fang** - 添加本地模型崩溃修复
*   **@EdwardWu** - 繁体中文支持
*   **@wk** - Demo 输入张量维度类型检测

---

**完整变更日志**: https://github.com/alibaba/MNN/compare/3.4.1...3.5.0

---

# MNN 3.5.0 Release Notes

**Release Date**: April 2026

## 📌 Overview

MNN 3.5.0 focuses on three core themes: **Multi-Backend LLM Inference Expansion**, **High-Performance Quantization & Sampling Optimization**, and **On-Device Voice Interaction Enhancement**:

*   **Multi-Backend LLM Inference**: Vulkan backend adds LLM inference support; Moore Threads MUSA backend newly integrated; QNN backend supports Attention operator and more LLM operators; RISC-V Vector extension (RVV) support landed.
*   **High-Performance Quantization & Sampling**: New TurboQuant TQ3/TQ4 KV Cache quantization; Sampler refactored to pipeline architecture with new penalty support; Multi-backend TopKV2 operator optimization; Multi-turn prompt cache support.
*   **On-Device Voice Interaction Enhancement**: Async 3-stage Token2Wav pipeline (DiT ∥ Vocoder); Smart barge-in with AEC support; Real-time vision for voice chat; TTS model support (supertonic-tts-mnn).

---

## 🚀 Highlights

*   **Vulkan LLM Inference**: Full LLM inference support on Vulkan backend, expanding GPU inference coverage
*   **MUSA Backend Support**: New Moore Threads MUSA GPU backend integration for domestic GPU inference
*   **RISC-V RVV Support**: CPU backend adds RISC-V Vector extension support, expanding the RISC-V ecosystem
*   **TurboQuant TQ3/TQ4**: New KV Cache quantization scheme that reduces memory usage while maintaining inference quality
*   **Sampler Pipeline Refactor**: Sampler refactored to pipeline architecture with penalty and advanced sampling strategies
*   **Multi-Turn Prompt Cache**: Text-level prompt cache that significantly accelerates time-to-first-token in multi-turn conversations
*   **Async Token2Wav 3-Stage Pipeline**: DiT and Vocoder parallel 3-stage pipeline for improved speech synthesis throughput
*   **Smart Barge-in (AEC)**: Smart voice interruption with echo cancellation for natural voice interactions
*   **Tokenizer Refactor**: minja → jinja template engine migration, binary tokenizer format (tokenizer.mtok), 20x+ loading speedup
*   **GLM-OCR / LFM Model Support**: New inference support for GLM-OCR and Liquid Foundation Model (LFM)
*   **Metal/CUDA/OpenCL TopKV2**: Multi-backend TopKV2 operator implementation and optimization

---

## ✨ New Features

### LLM/VLM
*   **Multi-Turn Prompt Cache**: Text-level prompt cache for significantly faster first-token generation in multi-turn conversations
*   **GLM-OCR Support**: New GLM-OCR model inference support
*   **LFM Support**: New Liquid Foundation Model (LFM) inference support
*   **Qwen3.5 Smooth/Omni Export**: Support Qwen3.5 smooth quantization and omni-modal export
*   **Sampler Pipeline Refactor**: Sampler refactored to pipeline architecture with new penalty mechanism support
*   **Async Token2Wav 3-Stage Pipeline**: Split token-to-audio conversion into a 3-stage pipeline with DiT and Vocoder running in parallel
*   **Async Token2Wav Multi-Runtime**: Support async Token2Wav when Talker's LLM and processor use different runtimes
*   **Tokenizer Refactor**: Complete Tokenizer system refactor: minja → jinja template engine migration; new binary tokenizer format (tokenizer.mtok) replacing text format; unified JSON handling via ujson.hpp; 20x+ loading speedup with multi-fold encode/decode boost; aligned with HuggingFace Transformers implementation
*   **Linear Attention Disk Storage**: Support disk load/store for Linear Attention recurrent states
*   **mmap_size Configuration**: New mmap_size config option for controlling memory-mapped file size
*   **llm_bench Profile**: Added profiling support to llm_bench

### GPU Backends
*   **Vulkan LLM Inference**: Full LLM model inference support on Vulkan backend
*   **MUSA Backend**: New Moore Threads MUSA GPU backend integration
*   **Metal TopKV2**: New TopKV2 operator implementation for Metal backend
*   **OpenCL TopKV2**: New TopKV2 operator implementation for OpenCL backend
*   **CUDA TopKV2 Optimization**: Replaced insertion sort with heap sort and loop unrolling for significant TopKV2 speedup
*   **Metal While/BatchMatMul 4x4 Blocking**: Optimized While and BatchMatMul via 4x4 register blocking
*   **Metal gemv/Binary/Scale Optimization**: Optimized gemv localsize, binary plane dispatch, and scale no-bias path; removed defer encoder
*   **OpenCL LinearAttention Optimization**: Improved OpenCL LinearAttention operator performance
*   **Metal LinearAttention Optimization**: Improved Metal LinearAttention operator performance
*   **OpenCL Kernel Protection**: Added kernel creation protection for all OpenCL operators
*   **QNN Attention Support**: QNN backend supports LLM Attention operator with max_history_token configuration
*   **QNN New Operators**: QNN backend adds Transpose/UnPack/Pack operator support
*   **CUDA Qwen3.5 Optimization**: CUDA backend optimizations for Qwen3.5 models

### CPU Backend
*   **RISC-V RVV Support**: CPU backend adds RISC-V Vector extension (RVV) support
*   **TurboQuant TQ3/TQ4**: New TQ3/TQ4 KV Cache quantization scheme
*   **LinearAttention NEON Optimization**: LinearAttention optimized with NEON assembly and FP16 support

### Apps & Tools
*   **Smart Barge-in (AEC)**: MNNChat supports smart voice interruption with acoustic echo cancellation
*   **Real-Time Vision Voice Chat**: MNNChat implements real-time vision capabilities for interactive voice chat
*   **TTS Model Support**: Android MNNChat supports supertonic-tts-mnn model
*   **Traditional Chinese Support**: Android apps add Traditional Chinese (zh-rTW) locale support
*   **F-Droid Listing**: Added Fastlane metadata for F-Droid app store listing
*   **CDN Auto-Upload**: New ali-oss based CDN auto-upload and market config upload scripts

---

## 🐛 Bug Fixes

*   **Prefix Disk Cache**: Fixed prefix disk cache not loaded after first response
*   **Jinja Rendering Context**: Fixed enable_thinking and other jinja context not passed to template rendering
*   **Jinja Array Concatenation**: Fixed jinja binary add operation error when handling arrays
*   **mmap Weights Deadlock**: Enhanced state validation to avoid deadlock from partially initialized mmap weights
*   **KV Cache Contamination**: Fixed KV cache contamination in OmniQuant calibration forward pass
*   **LLM File Load Checks**: Added file existence and permission checks in Llm::load
*   **LLM Status Check**: Fixed LLM runtime status check logic
*   **MoE Expert getattr**: Fixed MoE experts getattr compatibility with transformers@5.3.0
*   **llmexport Windows Crash**: Fixed llmexport crash on Windows caused by os.dup2
*   **llm_bench Fix**: Use n_gen from context to fix llm_bench statistics
*   **Qwen3-Embedding QNN Export**: Fixed Qwen3-Embedding QNN export pipeline
*   **Offline Quantization Cosine Distance**: Fixed cosine distance calculation error in offline quantization
*   **int4 Embedding Size**: Fixed int4 embedding compute size error
*   **KV Cache Meta Copy**: Skip KV cache copy when metadata is inconsistent
*   **GeometryGather Crash**: Fixed GPU case GeometryGather crash with const indices
*   **Metal External Bias**: Fixed reading float external bias error
*   **Metal Attention Test**: Fixed attention operator test issue
*   **MNNConvert NC4HW4**: Fixed shape op compute error for ONNX NC4HW4 inputs
*   **MNNDump2Json Overflow**: Fixed integer overflow in MNNDump2Json
*   **ONNX Shape start/end**: Support ONNX Shape operator start/end parameters while preserving OpParameter compatibility
*   **Pipeline::execute Compile Error**: Fixed info.group compile error in Pipeline::execute
*   **Windows Compile Error**: Fixed LLM Windows compile error
*   **MNNChat Multiple Fixes**: Fixed Android streaming step termination, model settings path reset, add local model crash, chat auto-scroll, and more

---

## 📚 Other Improvements

*   **Code Style**: Added `.clang-format` configuration and pre-commit hooks; CI format check integration
*   **Security Policy**: Added `SECURITY.md` security policy document and PR template
*   **Documentation Restructure**: Restructured documentation for better navigation and newcomer experience
*   **Zread Badge**: Added Zread badge to all README files
*   **Demo Enhancement**: Demo adds input tensor dimension type detection supporting both CAFFE and TENSORFLOW formats
*   **CI**: MNN release workflow ignores Android tags

---

## 📊 LLM Benchmark

### Android

<img width="2400" height="2000" alt="mnn-3 5 0-anrdoid-bench" src="https://github.com/user-attachments/assets/34b41a3d-2dbb-4cd8-a0c8-a7eeea8f38af" />

### iOS

<img width="2400" height="2000" alt="mnn-3 5 0-ios-bench" src="https://github.com/user-attachments/assets/38cb1f5a-d5d7-4400-bea8-5e6b445042ff" />

---

## 🙏 Acknowledgements

We sincerely thank all contributors for their valuable contributions to this release:

*   **@jxt1234** - Metal/OpenCL TopKV2, Metal performance optimization, QNN Attention/operator support, GeometryGather fix, MNNConvert fix
*   **@Jinde.Song** - MNNChat Android features & fixes, CDN auto-upload, release stabilization
*   **@EricMoin** - Async Token2Wav 3-stage pipeline, Talker runtime manager refactor
*   **@ᡠᠵᡠᡳ ᡠᠵᡠ ᠮᠠᠨᡩ᠋ᠠᠨ** - F-Droid listing, CI workflow improvements
*   **@YinHanke** - Qwen3.5 Smooth/Omni export, Qwen3-Embedding QNN fix, ONNX Shape compatibility
*   **@Jed Lee(lijunde)** - Real-time vision voice chat, smart barge-in (AEC), TTS model support
*   **@rainy liu** - CUDA TopKV2 heap sort optimization, Metal While/BatchMatMul 4x4 blocking
*   **@hebangwen** - Pipeline::execute compile fix, llm_bench fix
*   **@Jan Slominski** - Multi-turn prompt cache
*   **@dongyang-mt** - Moore Threads MUSA backend support
*   **@Sherlockzhangjinge** - RISC-V RVV support
*   **@JackWinterCN** - Offline quantization cosine distance fix
*   **@luckysmg** - Prefix disk cache fix
*   **@Riverspring996** - mmap weights deadlock fix
*   **@N Wang** - KV cache contamination fix
*   **@Xiaochao Fang** - Add local model crash fix
*   **@EdwardWu** - Traditional Chinese support
*   **@wk** - Demo input tensor dimension type detection

---

**Full Changelog**: https://github.com/alibaba/MNN/compare/3.4.1...3.5.0

---

*本文由 Claude 协助生成 / This document was generated with assistance from Claude*