b9745
ggml-org/llama.cppb9745Jun 21, 2026by github-actions[bot]same day as b9744
What changed since b9744
Adds Flash MTP3 speculative decoding for Step 3.5/3.7 models plus two new public C APIs for MTP layer handling.
New features
- Flash Multi-Head Speculative Decoding (MTP3) support for Step 3.5 and Step 3.7 models.
- New llama_set_mtp_layer_offset API for configuring MTP layer offsets.
- New llama_model_n_nextn_layer API to query the number of next-N layers in a model.
- Speculative multi-head processing and drafting paths, including multi-sequence handling.
Improvements
- MTP layer offset and next-N flags are now included in graph reuse decisions.
- Draft logic consolidated by merging draft_multi_head into the main draft() routine.
- Outputs are gathered via inp_out_ids for more efficient multi-head result collection.
Bug fixes
- Corrected multi-sequence handling in the speculative multi-head path.
- Fixed core logic issues found during review of the MTP3 implementation.
Upgrade notes
This is a feature-only increment over b9744 with no reported breaking changes; existing grammar/GBNF behavior is untouched. Integrators using the C API can optionally adopt llama_set_mtp_layer_offset and llama_model_n_nextn_layer to enable MTP3 speculative decoding for Step 3.5/3.7 models. Rebuild against the new headers if you link the shared library, since new exported symbols were added.
Analyzed Nov 1, 2026
Upgrading?
Release notes summary
Adds support for Flash Multi-Head Speculative Decoding (MTP3) for Step 3.5 and 3.7 models, including new APIs for layer offset management.
Key Highlights
- Flash MTP3 support for Step 3.5 and 3.7
- Adds llama_set_mtp_layer_offset and llama_model_n_nextn_layer APIs
- Supports speculative multi-head process and draft
Release Notes (excerpt)
<details open> spec : Support Step3.5/3.7 flash mtp3 (#24340) * add mtp_layer_offset + include nextn flags in graph reuse * add llama_set_mtp_layer_offset + llama_model_n_nextn_layer API * offset head select + require all MTP blocks * speculative multi-head process() * speculative multi-head draft() * gather outputs via inp_out_ids * cleanup * fix core * minor cleanup * merged draft_multi_head into draft() * mtp rename nextn * Apply suggestions from code review…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system