b10236
ggml-org/llama.cppb10236Aug 3, 2026by github-actions[bot]same day as b10235
What changed since b10235
b10236 adds a Metal Lightning Indexer kernel for DeepSeek V4, boosting long-context prompt processing by up to ~47%.
New features
- New GGML_OP_LIGHTNING_INDEXER Metal implementation for DeepSeek V4 attention, targeting 128-dim, 64-head inputs.
- Lightning Indexer now accepts F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1 and Q8_0 key caches rather than F16 only.
- Tiled plus tail kernels handle KV lengths around 8- and 64-element boundaries.
Improvements
- K tiles are staged and dequantized into F16 threadgroup memory before simdgroup matrix loads, unifying the numerical path across KV segments.
- Prefix-pass throughput at 10k context rose from roughly 74 to 87-88 t/s, with gains growing at longer contexts.
- At 30k context, pp512 improved from about 33 to 49 t/s; token generation also improved modestly.
- Partial tiles are zero-filled and stores guarded so all KV segments follow the same code path.
Upgrade notes
This is an additive Metal-backend change; no config, model-format, or CLI changes are required. Users running DeepSeek V4 with flash attention on Apple Silicon should see the largest gains at long context, and quantized K caches are now usable with the indexer. The prior release's Metal F32 SILU_BACK support remains in place.
Analyzed Oct 31, 2026
Upgrading?
Release notes summary
Implemented a Lightning Indexer for DeepSeek V4 in Metal, optimized with tiled kernels and staged memory access, resulting in significant performance improvements for large context lengths.
Key Highlights
- Implemented GGML_OP_LIGHTNING_INDEXER for DeepSeek V4.
- Performance improvements of up to 20% for prefix pass at 10k context length.
- Supports various quantization types (F32, F16, BF16, Q4, Q5, Q8) for K caches.
Release Notes (excerpt)
<details open> metal: implement DSv4 Lightning Indexer (#25893) * metal: implement F16 Lightning Indexer - Implement GGML_OP_LIGHTNING_INDEXER for 128-dimensional, 64-head inputs with F32 queries and weights plus F16 keys and masks. - Add tiled and tail kernels and test KV lengths around 8- and 64-element boundaries. llama-bench (--mmap 1, -fa 1, -p 512, -n 128; d=0/10k/20k/30k): Before: - pp512: 153.73 ± 0.87 t/s - tg128: 8.91 ± 0.04 t/s - pp512 @ d10000: 73.90 ± 0.39 t/s…
Published by ggml-org on GitHub.
Read the full release notesRelated Repositories
:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS
Godot Engine – Multi-platform 2D and 3D game engine
ClickHouse® is a real-time analytics database management system
Self-hosted game stream host for Moonlight.
DuckDB is a high-performance analytical database system