b9820

ggml-org/llama.cppb9820Jun 26, 2026by github-actions[bot]same day as b9817

What changed since b9817

medium risk

Newer build shifts focus to scheduler performance with fewer syncs and async CPU-to-CUDA copies.

New features

  • Asynchronous CPU-to-CUDA tensor copy path for supported backends.
  • Generalized opt-in mechanism letting backends such as Vulkan relax host-to-device synchronization.
  • Synchronization handling for multi-GPU setups that can treat single-GPU splits more efficiently.

Improvements

  • Cut synchronization frequency between tokens on CUDA during split compute.
  • Reduced scheduler overhead by relaxing sync requirements where backends support it.
  • Reworked backend detection to avoid linking conflicts in mixed builds.
  • Kept HIP/MUSA outside the async-copy optimization to reduce risk.

Bug fixes

  • Corrected initialization of the backend sync mode during scheduler split setup.
  • Restored a stricter CPU-to-CUDA async copy check via the CPU device type.

Upgrade notes

No configuration or API changes are required; the scheduler optimizations are transparent. Rebuild from source to pick up the async copy path, and note that only CUDA currently opts into relaxed synchronization while HIP/MUSA remain excluded. Validate multi-GPU and Vulkan workloads if you rely on custom backend sync assumptions.

Analyzed Nov 1, 2026

Upgrading?

Release notes summary

Performance improvements to the scheduler by reducing synchronization overhead during split compute and adding async CPU-to-GPU copy capabilities.

Key Highlights

  • Reduced synchronizations between tokens on CUDA
  • Added CPU-to-CUDA async copy capability
  • General relaxation of sync requirements for backends like Vulkan
  • Exclusion of HIP/MUSA from copy optimization for safety

Release Notes (excerpt)

<details open> sched : reintroduce less synchronizations during split compute (#20793) * CUDA: Improve performance via less synchronizations between token (#17795) * Adds CPU-to-CUDA copy capability to ggml_backend_cuda_cpy_tensor_async() * Adds function to relax sync requirements between input copies on supported backends (CUDA for now) * Exchanges synchronous copy with async copy function. * Adds macro guards to allow compilation in non-CUDA builds…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories