b9767

ggml-org/llama.cppb9767Jun 23, 2026by github-actions[bot]same day as b9763

What changed since b9763

low risk

WebGPU inference for small batches now uses a mat-vec path, speeding up MTP decoding; no API changes since b9763.

Improvements

  • WebGPU backend now routes small-batch decoding through a matrix-vector path, improving multi-token prediction (MTP) throughput.
  • Added a barrier inside the NUM_COLS loop of the mul-mat-vec WebGPU kernel to keep the new path correct.
  • Prebuilt binaries refreshed across macOS, Linux, and Windows backends to include the WebGPU change.

Upgrade notes

This is a drop-in update from b9763 with no config, API, or model-format changes. Only WebGPU users doing small-batch or speculative/MTP decoding will see a measurable difference; other backends are effectively unchanged. Swap in the new binaries if you rely on WebGPU.

Analyzed Nov 1, 2026

Upgrading?

Release notes summary

This release optimizes WebGPU inference for small batches by utilizing a matrix-vector path. It includes updated binaries for supported platforms and backends.

Key Highlights

  • Improve MTP inference for small batches in WebGPU
  • macOS/iOS binaries (Apple Silicon, Intel, XCFramework)
  • Linux binaries (CPU, Vulkan, ROCm, OpenVINO, SYCL)
  • Windows binaries (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP)

New Features

  • Improve MTP inference for small batches in WebGPU

Release Notes (excerpt)

<details open> ggml-webgpu: improve MTP inference by using mat-vec path for small batches (#24811) * ggml-webgpu: improve small batches decoding * Add barrier to the NUM_COLS loop in mul-mat-vec </details> **macOS/iOS:** - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework **Linux:** - Ubuntu x64 (CPU) - Ubuntu arm64 (CPU) - Ubuntu s390x (CPU) - Ubuntu x64 (Vulkan) - Ubuntu arm64 (Vulkan) - Ubuntu x64 (ROCm 7.2)…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories