b9788

ggml-org/llama.cppb9788Jun 25, 2026by github-actions[bot]same day as b9787

What changed since b9787

medium risk

b9788 adds SYCL tensor parallelism with a BF16-compressed all-reduce path for dual-GPU setups, following b9787's SYCL conv_3d test fix.

New features

  • SYCL backend now supports --split-mode tensor for dual-GPU tensor parallelism via the comm_init/comm_free/comm_allreduce_tensor trio.
  • Large tensors (nelem >= 32768) use a BF16-compressed all-reduce that halves PCIe traffic, while smaller tensors take an FP32 memcpy-plus-ADD path.
  • Each device allocates a single persistent communication buffer, avoiding repeated allocations in the SYCL pool.

Improvements

  • Reported throughput gains of roughly 20% to 79% across several models on Intel Arc Pro B70 dual-GPU configurations.
  • All-reduce threshold and BF16 scheme mirror the CUDA NCCL implementation for consistency across backends.
  • Unsupported configurations fall back to the meta-backend's generic butterfly all-reduce rather than failing.

Bug fixes

  • Carried over from b9787: SYCL conv_3d unit test failures are resolved.

Upgrade notes

No config migration is required; tensor parallelism activates only when you pass --split-mode tensor on a supported SYCL dual-GPU system. The initial implementation covers N=2 with FP32 contiguous tensors only, so other shapes or GPU counts silently use the generic fallback. Per-call synchronization is intentionally omitted, relying on in-order queue semantics.

Analyzed Oct 31, 2026

Upgrading?

Release notes summary

Adds SYCL tensor parallelism support for dual-GPU setups, implementing a BF16-compressed all-reduce path for large tensors and achieving significant performance improvements on Intel Arc Pro B70.

Key Highlights

  • SYCL tensor parallelism (--split-mode tensor) for dual-GPU
  • BF16-compressed all-reduce path for large tensors (nelem >= 32768)
  • Performance improvements: +20.2% to +78.6% on various models
  • Implements comm_init/comm_free/comm_allreduce_tensor trio

Release Notes (excerpt)

<details open> sycl : support --split-mode tensor (#24152) * Sycl tp stage1 (#1) * SYCL: tensor parallelism (--split-mode tensor) for dual-GPU Adds the comm_init/comm_free/comm_allreduce_tensor trio that the meta-backend queries via get_proc_address to enable backend-specific all-reduce, mirroring the pattern used by ggml-cuda.cu. For N=2 (the common dual-GPU case) implements a degenerate ring all-reduce with two size-branched paths:…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories