An upstream llama.cpp fix cuts the extra GPU memory of a quantized KV cache
I built upstream llama.cpp PR 29827, which caps the flash-attention convert buffer for a quantized KV cache, and compared load-time GPU memory against stock llama.cpp on Qwen3.5-4B. At 512K context, VRAM went from 9,862 to 8,572 MiB. At 1M with the cache in system RAM, from 9,579 to 5,547 MiB. Load-only numbers; retrieval at these lengths is not tested yet.
| Experiment | E21B |
| Date | 2026-10-28 |
| Benchmark | Load-time GPU memory A/B: VRAM used (nvidia-smi) and per-process shared GPU memory (Windows counter) after loading the server once per config |
| Model | Qwen3.5-4B, Q4_K_M GGUF, flash attention on, q4_0 KV cache (one f16 control row) |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
Stock llama.cpp versus my build of upstream llama.cpp PR 29827, "CUDA: cap FA convert buffer for quantized KV" (not my work, unmerged when tested). Stock llama.cpp turns a whole layer's quantized K and V into f16 in a scratch buffer for batched prefill, and that buffer grows with context. The PR caps it. Each config was loaded once per binary at 128K, 512K and 1M context.
Pass bar, set before the run
None. This was a measurement of an upstream fix, not a pass/fail test of a method.
Result
| Config | VRAM, PR build | VRAM, stock | Shared, PR build | Shared, stock |
|---|---|---|---|---|
| 128K, q4_0 KV, ub512 | 4,732 MiB | 5,180 MiB | 228 MiB | 228 MiB |
| 128K, f16 KV, ub512 | 7,626 MiB | 7,626 MiB | 228 MiB | 228 MiB |
| 512K, q4_0 KV, ub512 | 8,572 MiB | 9,862 MiB | 612 MiB | 1,306 MiB |
| 512K, q4_0 KV, ub128 | 8,142 MiB | 9,909 MiB | 212 MiB | 396 MiB |
| 1M, q4_0 KV in system RAM, ub512 | 5,547 MiB | 9,579 MiB | 1,130 MiB | 1,130 MiB |
The f16 row does not change, as expected: the fix only affects a quantized KV cache.
With the PR build at 512K and ub128, the 4B uses about 8.1 GB and shared memory is 212 MiB, against 132 MiB for an earlier 32K run. Close to fully resident, but not zero extra spill.
At 1M the KV cache was in system RAM on purpose (--no-kv-offload), so the 1,130 MiB shared figure is expected in both builds.
The results file says loaded=false on every row. That was a bug in my log check, which looked for the wrong text. The servers did load, in about 2 s, so the memory numbers are valid.
Load only. These numbers show the model fits; they do not show it can still find anything at that length. A needle test at about 450K and 900K tokens (E21c) is running.
What happened
Most of the extra GPU memory a q4_0 KV cache cost on this setup was one oversized conversion scratch buffer, not the cache itself. The upstream fix removes most of it: 1,290 MiB less at 512K (ub512), 1,767 MiB less at 512K (ub128) and 4,032 MiB less at 1M. Credit goes to the authors of llama.cpp PR 29827. Retrieval at these lengths is the next test.
How to rerun
powershell -ExecutionPolicy Bypass -File e21b_chain.ps1Code
All code for E21B: README.md, e21b_chain.ps1, results.jsonl.
Prior work
No novelty claim. This measures an existing upstream llama.cpp pull request.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.