b9866
ggml-org/llama.cppb9866Jul 3, 2026by github-actions[bot]same day as b9864
What changed since b9864
Adds CUDA topk-MoE fusion for 288-expert models, yielding ~2.4% faster decode with no API changes.
New features
- CUDA topk-MoE fusion now supports models with 288 experts (e.g. Step-3.7-Flash), which previously fell back to unfused per-layer routing.
Improvements
- Decode throughput improves roughly 2.4% at shallow context for 288-expert MoE models on CUDA.
- Adds the missing kernel template instantiation for 288 expert counts, since 288 is a multiple of warp size.
- Extends the fusion eligibility check to accept 288 alongside power-of-2 counts and the special-cased 576.
- Backend op tests were updated to cover the new expert-count path.
Upgrade notes
No configuration or API changes; the SSE ping behavior from the previous build is unchanged. Rebuild with CUDA to pick up the new kernel instantiation. Gains apply only to decode on 288-expert MoE models and diminish at deep context where attention dominates.
Analyzed Oct 30, 2026
Upgrading?
Release notes summary
Enables topk-moe fusion for 288 experts in CUDA, previously only supported for power-of-2 counts or 576. This improves decode performance for large MoE models.
Key Highlights
- Enables topk-moe fusion for 288 expert models (e.g., Step-3.7-Flash)
- Adds missing template instantiation for 288 expert counts
- Measured ~2.4% decode speedup at shallow context
- Fusion only affects decode routing, not prompt processing
Release Notes (excerpt)
<details open> cuda: enable topk-moe fusion for 288 experts (#25267) * cuda: enable topk-moe fusion for 288 experts The topk-moe fusion only accepted power-of-2 expert counts (or the special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash) fell back to the unfused per-layer routing chain: softmax/sigmoid, argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that is ~330 extra tiny graph nodes per token.…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system