b9948

ggml-org/llama.cppb9948Jul 10, 2026by github-actions[bot]same day as b9947

What changed since b9947

low risk

CUDA top-k and argsort now process data in chunks, cutting temporary buffer memory use.

Improvements

  • CUDA ggml_top_k() and ggml_argsort() now process data in smaller chunks to reduce temporary buffer memory usage.
  • tmp_dst is allocated once before the processing loop instead of repeatedly.
  • Chunked processing applied consistently across both CUDA CUB top-k and argsort implementations.

Upgrade notes

This is a CUDA-backend-only internal optimization with no CLI, API, or config changes; existing commands and flags (including the --output option from the prior build) continue to work. Rebuild or swap in the new binaries to benefit from lower temporary memory usage on CUDA devices.

Analyzed Oct 24, 2026

Upgrading?

Release notes summary

Optimized CUDA top-k and argsort implementations to reduce temporary buffer memory usage.

Key Highlights

  • Process data in smaller chunks in CUDA top-k and argsort implementations
  • Reduces temporary buffers memory usage significantly
  • Allocates tmp_dst only once before the processing loop
  • Uses chunked processing in both CUDA CUB top-k and argsort implementations

New Features

  • CUDA memory optimization
  • Reduced temporary buffer usage

Release Notes (excerpt)

<details open> ggml : process data in smaller chunks in CUDA ggml_top_k() and ggml_argsort() to reduce temporary buffers memory usage (#24776) * ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage * ggml : allocate tmp_dst only only once before the loop * chore : whitespaces Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * ggml : use chunked processing in both CUDA CUB top-k and argsort implementations…

Published by ggml-org on GitHub.

Read the full release notes

Related Repositories