Starstruk Logo MarkStarstruk

b10782

Pre-release

ggml-org/llama.cppb10782Sep 3, 2026by github-actions[bot]

AI Summary

Fixes a CUDA multi-GPU issue where graph optimization was skipped due to concurrent stream handling. The fix allows concurrent streams per split and explicitly sets the device to prevent CUDA events from sticking to the wrong GPU.

Key Highlights

  • Allow concurrent streams per split for multi-GPU
  • Fix CUDA event creation sticking to GPU 0
  • Enable CUDA graph optimization for multi-GPU setups

New Features

  • Multi-GPU CUDA graph optimization fix

Full Release Notes

<details open>

CUDA: Allow concurrent streams per split for multi-GPU (#28198)

* CUDA: Allow CUDA optimization per split for multi-GPU.

Previous guard caused multi-GPU to skip the graph optimization.  The
graph is already split per device and the optimization doesnt run
over the whole model but once per split, and thus should be allowed.
However, the CUDA event ggml_cuda_concurrent_event belongs to
whichever GPU was "current" when created. If the pass ran while
GPU 0 was current, it would stick and during event creation for the
second GPU it would land on GPU 0.

The fix: set the device explicitly ggml_cuda_set_device(cuda_ctx->device);
Default behaviour remains unchanged, only active for GGML_CUDA_GRAPH_OPT=1.
Explicit device setting pattern re-used from ggml_backend_cuda_graph_compute.

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Aman Gupta <amangupta052@gmail.com>

---------

Co-authored-by: tannerbruhn <tannerbruhn@users.noreply.github.com>
Co-authored-by: Aman Gupta <amangupta052@gmail.com>

</details>

**Website:**
- <https://llama.app>

**Attestations:**
- <https://github.com/ggml-org/llama.cpp/attestations/44959687>

**macOS/iOS:**
- [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-macos-arm64.tar.gz)
- macOS Apple Silicon (arm64, KleidiAI enabled) [DISABLED](https://github.com/ggml-org/llama.cpp/pull/23780)
- [macOS Intel (x64)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-macos-x64.tar.gz)
- [iOS XCFramework](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-xcframework.zip)

**Linux:**
- [Ubuntu x64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-x64.tar.gz)
- [Ubuntu arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-arm64.tar.gz)
- [Ubuntu s390x (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-s390x.tar.gz)
- [Ubuntu x64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-vulkan-x64.tar.gz)
- [Ubuntu arm64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-vulkan-arm64.tar.gz)
- [Ubuntu x64 (ROCm 10.0)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-rocm-10.0-x64.tar.gz)
- [Ubuntu x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-openvino-2026.3.1-x64.tar.gz)
- [Ubuntu x64 (SYCL FP32)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-sycl-fp32-x64.tar.gz)
- [Ubuntu x64 (SYCL FP16)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-ubuntu-sycl-fp16-x64.tar.gz)

**Android:**
- [Android arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-android-arm64.tar.gz)

**Windows:**
- [Windows x64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-cpu-x64.zip)
- [Windows arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-cpu-arm64.zip)
- [Windows arm64 (OpenCL Adreno)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-opencl-adreno-arm64.zip)
- [Windows x64 (CUDA 12)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-cuda-12.4-x64.zip) - [CUDA 12.4 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/b10782/cudart-llama-bin-win-cuda-12.4-x64.zip)
- [Windows x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-cuda-13.3-x64.zip) - [CUDA 13.3 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/b10782/cudart-llama-bin-win-cuda-13.3-x64.zip)
- [Windows arm64 (CUDA 13) (preview)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-cuda-13.4-arm64.zip) - [CUDA 13.4 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/b10782/cudart-llama-bin-win-cuda-13.4-arm64.zip)
- [Windows x64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-vulkan-x64.zip)
- [Windows x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-openvino-2026.3.1-x64.zip)
- [Windows x64 (SYCL)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-sycl-x64.zip)
- [Windows x64 (ROCm 10.0)](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-bin-win-rocm-10.0-x64.zip)

**openEuler:**
- [DISABLED](https://github.com/ggml-org/llama.cpp/pull/23705)
- openEuler x86 (310p)
- openEuler x86 (910b, ACL Graph)
- openEuler aarch64 (310p)
- openEuler aarch64 (910b, ACL Graph)

**UI:**
- [UI](https://github.com/ggml-org/llama.cpp/releases/download/b10782/llama-b10782-ui.tar.gz)