b9745

ggml-org/llama.cppb9745Jun 21, 2026by github-actions[bot]same day as b9744

What changed since b9744

medium risk

Adds Flash MTP3 speculative decoding for Step 3.5/3.7 models plus two new public C APIs for MTP layer handling.

New features

  • Flash Multi-Head Speculative Decoding (MTP3) support for Step 3.5 and Step 3.7 models.
  • New llama_set_mtp_layer_offset API for configuring MTP layer offsets.
  • New llama_model_n_nextn_layer API to query the number of next-N layers in a model.
  • Speculative multi-head processing and drafting paths, including multi-sequence handling.

Improvements

  • MTP layer offset and next-N flags are now included in graph reuse decisions.
  • Draft logic consolidated by merging draft_multi_head into the main draft() routine.
  • Outputs are gathered via inp_out_ids for more efficient multi-head result collection.

Bug fixes

  • Corrected multi-sequence handling in the speculative multi-head path.
  • Fixed core logic issues found during review of the MTP3 implementation.

Upgrade notes

This is a feature-only increment over b9744 with no reported breaking changes; existing grammar/GBNF behavior is untouched. Integrators using the C API can optionally adopt llama_set_mtp_layer_offset and llama_model_n_nextn_layer to enable MTP3 speculative decoding for Step 3.5/3.7 models. Rebuild against the new headers if you link the shared library, since new exported symbols were added.

Analyzed Nov 1, 2026

Upgrading?

Release notes summary

Adds support for Flash Multi-Head Speculative Decoding (MTP3) for Step 3.5 and 3.7 models, including new APIs for layer offset management.

Key Highlights

  • Flash MTP3 support for Step 3.5 and 3.7
  • Adds llama_set_mtp_layer_offset and llama_model_n_nextn_layer APIs
  • Supports speculative multi-head process and draft

Release Notes (excerpt)

<details open> spec : Support Step3.5/3.7 flash mtp3 (#24340) * add mtp_layer_offset + include nextn flags in graph reuse * add llama_set_mtp_layer_offset + llama_model_n_nextn_layer API * offset head select + require all MTP blocks * speculative multi-head process() * speculative multi-head draft() * gather outputs via inp_out_ids * cleanup * fix core * minor cleanup * merged draft_multi_head into draft() * mtp rename nextn * Apply suggestions from code review…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories