b9866

ggml-org/llama.cppb9866Jul 3, 2026by github-actions[bot]same day as b9864

What changed since b9864

low risk

Adds CUDA topk-MoE fusion for 288-expert models, yielding ~2.4% faster decode with no API changes.

New features

  • CUDA topk-MoE fusion now supports models with 288 experts (e.g. Step-3.7-Flash), which previously fell back to unfused per-layer routing.

Improvements

  • Decode throughput improves roughly 2.4% at shallow context for 288-expert MoE models on CUDA.
  • Adds the missing kernel template instantiation for 288 expert counts, since 288 is a multiple of warp size.
  • Extends the fusion eligibility check to accept 288 alongside power-of-2 counts and the special-cased 576.
  • Backend op tests were updated to cover the new expert-count path.

Upgrade notes

No configuration or API changes; the SSE ping behavior from the previous build is unchanged. Rebuild with CUDA to pick up the new kernel instantiation. Gains apply only to decode on 288-expert MoE models and diminish at deep context where attention dominates.

Analyzed Oct 30, 2026

Upgrading?

Release notes summary

Enables topk-moe fusion for 288 experts in CUDA, previously only supported for power-of-2 counts or 576. This improves decode performance for large MoE models.

Key Highlights

  • Enables topk-moe fusion for 288 expert models (e.g., Step-3.7-Flash)
  • Adds missing template instantiation for 288 expert counts
  • Measured ~2.4% decode speedup at shallow context
  • Fusion only affects decode routing, not prompt processing

Release Notes (excerpt)

<details open> cuda: enable topk-moe fusion for 288 experts (#25267) * cuda: enable topk-moe fusion for 288 experts The topk-moe fusion only accepted power-of-2 expert counts (or the special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash) fell back to the unfused per-layer routing chain: softmax/sigmoid, argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that is ~330 extra tiny graph nodes per token.…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories