Starstruk Logo MarkStarstruk

2.2.0

alibaba/MNN2.2.0Oct 31, 2022by jxt1234

AI Summary

MNN 2.2.0 is a significant release focused on performance optimization and framework extensibility. Key additions include ARMv8.6-A instruction support (smmla/bfmmla) achieving near-theoretical performance gains, a new NNAPI backend for Android NPU/APU/DSP acceleration, and CUDA optimizations. The release also expands training capabilities with new derivative operators and adds support for A16/M2 CPU families.

Key Highlights

  • ARMv8.6-A instruction support with smmla and bfmmla instructions, delivering 83.90% and 91.53% performance improvements for GemmInt8 and GemmBF16 respectively
  • New NNAPI backend enabling Android device NPU/APU/DSP acceleration with significant performance gains over CPU
  • CUDA performance optimizations including Depthwise convolution, Raster, and Im2Col improvements
  • Training module expansion with Selu, StridedSlice, TopKV2, Scale, and Broadcastto derivative operators
  • Added A16 and M2 CPU family support and Interp3D/ConvTranspose3D operators

New Features

  • ARMv8.6-A instruction support (smmla, bfmmla)
  • NNAPI backend for Android NPU/APU/DSP
  • Assembly pre-processing script for .inst directive conversion
  • A16 and M2 CPU family support
  • New training operators: Selu, StridedSlice, TopKV2, Scale, Broadcastto derivatives
  • Interp3D and ConvTranspose3D support
  • BinaryRelu-Fuse optimization for resnet models
  • CUDA optimizations (Depthwise, Raster, Im2Col)
  • Code refactoring for complex multi-SubModule models
  • Tflite Prelu model conversion support (constant slope only)

Full Release Notes

# 一、框架通用性
- 新增对`ARMv8.6-A`指令支持，支持了`smmla`和`bfmmla`指令的使用
- 新增`NNAPI`后端，能够利用Android设备上的NPU/APU/DSP进行计算
- 新增汇编预处理脚本，能够将汇编指令转换为`.inst`指令，降低新指令对编译器的依赖
- 新增A16和M2 CPU family支持
- 训练模块新增Selu / StridedSlice / TopKV2 / Scale / Broadcastto 等求导算子
- 新增Interp3D / ConvTranspose3D支持（感谢开源群伙伴贡献）

# 二、性能优化

- 新增ARMv8.6指令支持后，GemmInt8, GemmBF16性能提升
   - `smmla`实现的`GemmInt8`实测性能在矩阵规模为`[1024, 1024, 1024]`时，性能相比`sdot`提升为83.90%（图中`33x33`项），接近理论性能(100%)；模型性能提升20%左右。
   - `bfmmla`实现的`GemmBF16`实测性能在规模为`[1024, 1024, 1024]`时，性能相比fp16`fmla`提升为91.53%（图中`1024,1024,1024`项），接近理论性能；模型性能相比原来的bf16提升一倍以上。
![image](https://user-images.githubusercontent.com/5484403/198917531-57c1f45c-f8a0-4139-906a-666bf54c2d6f.png)
![image](https://user-images.githubusercontent.com/5484403/198917547-383d5644-b478-441f-b0df-3af44c7732a4.png)
![image](https://github.com/alibaba/MNN/blob/master/docs/_static/images/intro/releases/2_2_0_bfmmla1.png)
![image](https://github.com/alibaba/MNN/blob/master/docs/_static/images/intro/releases/2_2_0_bfmmla2.png)

- MNN-NNAPI后端支持，NNAPI使用accelerator设备进行推理，在中端和高端设备上相比CPU单线程均有性能优势；在高端设备上相比CPU 4线程仍有性能优势。
![image](https://user-images.githubusercontent.com/5484403/198917581-0ebff1f5-110a-48b7-a56d-6edff6cceefd.png)
![image](https://user-images.githubusercontent.com/5484403/198917598-11155306-5cd3-40b0-805f-f6d8969259a1.png)


- CUDA性能优化，Depthwise卷积、Raster快速计算、Im2Col等优化，MobileNet/Squeezenet等模型性能提升
- 新增BinaryRelu-Fuse和对应的各后端实现，resnet模型性能提升
![image](https://user-images.githubusercontent.com/5484403/198917629-e308a565-6b73-4457-b066-b3728be22fb6.png)
![image](https://user-images.githubusercontent.com/5484403/198917642-57c88993-b7d6-48ee-bf72-fb0ae84f0ae8.png)


# 三、其他

- 进行了部分代码重构（包括但不限于） 
   - 对于包含多个SubModule的复杂模型, 复用子模型间共性tensor，重新构建计算图和指令队列，显著降低大内存操作的耗时
- 修复了如下 Bug（包括但不限于） 
   - Onnx Resize 在指定 scale 且输入输出无法整除时，计算错误
   - 修复在不支持SSE 4.1 ，但支持SSE3的设备上打开SSE执行Crash的问题
   - 修复部分情况下多输入Conv转换错误
   - 修复ARM82后端GridSampler在Linux上的编译错误
   - 修复Conv1dSqueezeMove在Squeeze双输入时计算出错的问题
   - 修复输入为NC4HW4时，stride计算错误的问题
   - 修复HIAI后端编译错误，Binary BUG
   - 新增Tflite Prelu 模型转换支持（仅支持slope为常量情况）