b9784

ggml-org/llama.cppb9784Jun 24, 2026by github-actions[bot]same day as b9782

What changed since b9782

medium risk

Hexagon backend gets a major MUL_MAT/MUL_MAT_ID rework with 32x32 tiled weight repacking, graph caching, and VTCM/DMA tuning.

New features

  • Hexagon MUL_MAT and MUL_MAT_ID kernels were reworked around a new 32x32 tiled weight repack layout.
  • Graph caching support was added to the Hexagon backend, reducing per-inference setup overhead.
  • Legacy architecture restrictions on the Hexagon path were lifted, widening the set of supported devices.

Improvements

  • Kernel parameter computation was moved to the host and the matmul optimizer now accounts for activation thread count.
  • VTCM allocations were unified under vtcm_seq_alloc and the qweight scratch buffer was eliminated in favor of vtcm_weight.
  • Dynamic quantization for q8_0/q8_1 was optimized and vec_dots unrolled for better HVX register utilization.
  • Activation processing now uses DMA prefetch, and weight dequant tiles are aligned with DMA transfers.
  • Common matmul code was factored into htp/matmul-ops.h and obsolete code paths were removed.

Bug fixes

  • Fixed HMX/HVX fallback logic and MUL_MAT_ID allocation, which unbreaks OLMoE on Hexagon.
  • Restored pipelined mode in the HMX matmul path after the rework.

Upgrade notes

This is a backend-internal overhaul of the Hexagon (Qualcomm) path; no CLI flags, config, or public API changes are mentioned, so a rebuild of the Hexagon binaries should suffice. Non-Hexagon platforms are effectively unaffected by the delta from b9782. If you run OLMoE or other MUL_MAT_ID-heavy models on Hexagon, re-validate outputs and performance after upgrading.

Analyzed Nov 1, 2026

Upgrading?

Release notes summary

Major performance and architectural overhaul of the Hexagon (Qualcomm) backend, introducing tiled weight repacking (32x32), improved kernel parameters, graph caching, and VTCM optimizations.

Key Highlights

  • Hexagon tiled weight repack (32x32)
  • Kernel parameter optimization
  • Graph caching support
  • VTCM and DMA optimizations
  • Legacy arch restrictions removed

Release Notes (excerpt)

<details open> hexagon: MUL_MAT and MUL_MAT_ID rework : 32x32 tiled weight repack, kernel-params, cached graphs (#24954) * hex-mm: new weight layout and fusion updates * hvx-mm: unroll the new tiled vec_dots to optimize hvx register util * hex-mm: optimize dyn.quant format for q8_0 and q8_1 to reduce overhead in vec_dots. * hvx-mm: parallel quantizer per block for large rows * hvx-mm: simplify and futher optimize dyn.quant and vec_dots…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories