b10330

ggml-org/llama.cppb10330Aug 8, 2026by github-actions[bot]same day as b10329

What changed since b10329

low risk

b10330 adds a CUDA kernel fusion for rms_norm + mul + RoPE, cutting memory traffic; no server/UI changes beyond b10329.

New features

  • CUDA backend now fuses rms_norm, mul, and RoPE (optionally with view and set_rows) into a single kernel launch.
  • Added test coverage for broadcast weight cases in the fused rms_norm_mul_rope path.

Improvements

  • Reduced memory traffic and kernel launch overhead on CUDA by combining normalization, scaling, and rotary embedding steps.
  • Memory range checks were added before applying the rms_norm/RoPE and rope/set_rows fusions to guard correctness.

Upgrade notes

This is a CUDA-only performance change with no API, config, or CLI changes; a drop-in binary swap is sufficient. Users on non-CUDA backends will see no behavioral difference. Validate output quality on models using fused norm+RoPE patterns, since the fusion is guarded by new memory-range checks.

Analyzed Nov 1, 2026

Upgrading?

Release notes summary

Optimizes CUDA performance by fusing RMS normalization, multiplication, and RoPE operations into a single kernel launch.

Key Highlights

  • Implements kernel fusion for `rms_norm + mul + rope (+ view + set_rows)` to reduce memory traffic.
  • Includes memory range checks to ensure correctness of the fused operations.
  • Adds tests for broadcast weight cases in the rms_norm_mul_rope fusion.

Release Notes (excerpt)

<details open> CUDA: fuse rms_norm + mul + rope (+ view + set_rows) (#26767) * CUDA: fuse rms_norm + mul + rope (+ view + set_rows) * tests: add broadcast weight case to rms_norm_mul_rope * CUDA: check memory ranges before rms_norm rope fusion * CUDA: check memory ranges in rope set_rows fusion </details> **Website:** - <https://llama.app> **macOS/iOS:** - macOS Apple Silicon (arm64) - macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED - macOS Intel (x64) - iOS XCFramework…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories