b9788
ggml-org/llama.cppb9788Jun 25, 2026by github-actions[bot]same day as b9787
What changed since b9787
b9788 adds SYCL tensor parallelism with a BF16-compressed all-reduce path for dual-GPU setups, following b9787's SYCL conv_3d test fix.
New features
- SYCL backend now supports --split-mode tensor for dual-GPU tensor parallelism via the comm_init/comm_free/comm_allreduce_tensor trio.
- Large tensors (nelem >= 32768) use a BF16-compressed all-reduce that halves PCIe traffic, while smaller tensors take an FP32 memcpy-plus-ADD path.
- Each device allocates a single persistent communication buffer, avoiding repeated allocations in the SYCL pool.
Improvements
- Reported throughput gains of roughly 20% to 79% across several models on Intel Arc Pro B70 dual-GPU configurations.
- All-reduce threshold and BF16 scheme mirror the CUDA NCCL implementation for consistency across backends.
- Unsupported configurations fall back to the meta-backend's generic butterfly all-reduce rather than failing.
Bug fixes
- Carried over from b9787: SYCL conv_3d unit test failures are resolved.
Upgrade notes
No config migration is required; tensor parallelism activates only when you pass --split-mode tensor on a supported SYCL dual-GPU system. The initial implementation covers N=2 with FP32 contiguous tensors only, so other shapes or GPU counts silently use the generic fallback. Per-call synchronization is intentionally omitted, relying on in-order queue semantics.
Analyzed Oct 31, 2026
Upgrading?
Release notes summary
Adds SYCL tensor parallelism support for dual-GPU setups, implementing a BF16-compressed all-reduce path for large tensors and achieving significant performance improvements on Intel Arc Pro B70.
Key Highlights
- SYCL tensor parallelism (--split-mode tensor) for dual-GPU
- BF16-compressed all-reduce path for large tensors (nelem >= 32768)
- Performance improvements: +20.2% to +78.6% on various models
- Implements comm_init/comm_free/comm_allreduce_tensor trio
Release Notes (excerpt)
<details open> sycl : support --split-mode tensor (#24152) * Sycl tp stage1 (#1) * SYCL: tensor parallelism (--split-mode tensor) for dual-GPU Adds the comm_init/comm_free/comm_allreduce_tensor trio that the meta-backend queries via get_proc_address to enable backend-specific all-reduce, mirroring the pattern used by ggml-cuda.cu. For N=2 (the common dual-GPU case) implements a degenerate ring all-reduce with two size-branched paths:…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system