The shared-memory formula predicted 5 new settings to within 3 MiB, but a smaller ubatch does not push the 4B past about 704K fully on the GPU
E21q found that shared GPU memory for the 4B model equals 82 MiB plus context x ubatch x 2 bytes until the 10 GB card runs out. I wrote down predictions for five new settings before running them, and all five landed within 3 MiB. I then tried to use a very small ubatch to fit more context on the card. It did not help: at 768K the spill was the same 298 MiB at ubatch 16 and ubatch 8, and prompt processing there was about 20 to 37 times slower than at 704K.
| Experiment | E21R |
| Date | 2026-10-29 |
| Benchmark | Memory and speed measurement only, no accuracy test. llama-server loaded at each setting, per-process Dedicated and Shared Usage read from the Windows GPU Process Memory counters (max of 5 samples), then one 2K-token prompt timed. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, K and V cache at q4_0, flash attention on, YaRN context extension above 256K |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
Prediction points: 256K ub128, 384K ub16, 512K ub128, 512K ub32, 704K ub8. Residency points: 736K ub16, 768K ub16 and ub8, 800K ub16, 832K ub16 (capped by llama.cpp at 851,967 tokens, one below the requested size). K = 1024 tokens, so 704K is 720,896 tokens. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Rule set before the run: a point counts as fully on the GPU if measured shared memory is within 20 MiB of 82 + ctx x ub x 2 bytes; more than that is spill.
Result
| Context | Ubatch | Shared (measured) | Predicted | Over prediction | Dedicated | 2K prompt speed |
|---|---|---|---|---|---|---|
| 256K | 128 | 148 MiB | 146 MiB | +2 | 5,356 MiB | 2,275 tok/s |
| 384K | 16 | 92 MiB | 94 MiB | -2 | 6,443 MiB | 1,139 tok/s |
| 512K | 128 | 212 MiB | 210 MiB | +2 | 7,724 MiB | 2,754 tok/s |
| 512K | 32 | 112 MiB | 114 MiB | -2 | 7,617 MiB | 1,647 tok/s |
| 704K | 8 | 90 MiB | 93 MiB | -3 | 9,321 MiB | 652 tok/s |
| 736K | 16 | 222 MiB | 105 MiB | +117 (spill) | 9,502 MiB | 1,179 tok/s |
| 768K | 16 | 404 MiB | 106 MiB | +298 (spill) | 9,612 MiB | 32 tok/s |
| 768K | 8 | 392 MiB | 94 MiB | +298 (spill) | 9,598 MiB | 18 tok/s |
| 800K | 16 | 660 MiB | 107 MiB | +553 (spill) | 9,645 MiB | 33 tok/s |
| 832K | 16 | 982 MiB | 108 MiB | +874 (spill) | 9,615 MiB | 34 tok/s |
All five prediction points were within 3 MiB, so the 82 MiB + ctx x ub x 2 bytes model now holds at 16 settings across E21q and E21r. It is still attributed by size match only; llama.cpp's buffer-size lines were not in these logs.
Above about 704K, dedicated memory stops near 9.5 to 9.65 GB and the rest goes to shared memory. Shrinking the ubatch only shrinks the small mask term, so it cannot win back hundreds of MiB.
736K went 117 MiB over prediction but its 2K prompt speed was normal. 768K and up were 20 to 37 times slower than 704K on the same prompt. Only a 2K prompt was timed; a full-length prompt was not.
Small ubatch costs prompt speed: 704K at ubatch 8 ran 652 tok/s against 1,000 to 2,700 tok/s at larger ubatch and smaller context.
Reading: for this model and cache format the card holds about 720,896 tokens. Reaching 1,048,576 tokens fully on the GPU needs real memory savings, not settings. E21s measures how much memory each token costs per cache format to size that gap.
What happened
The memory formula from E21q held on every new point I predicted ahead of time. The hope was that a tiny ubatch would free enough room to go past 704K, but the card was already full: from 736K up, dedicated memory stayed pinned near its limit and the overflow grew by about 180 to 320 MiB per extra 32K tokens, the same at ubatch 16 and 8. Once the spill reached a few hundred MiB, prompt processing dropped from hundreds or thousands of tokens per second to 18 to 34. So the practical ceiling on this card for the 4B with a 4-bit cache is about 704K, and 1M needs a smaller cache or smaller buffers. The memory saving that makes these sizes fit at all is upstream llama.cpp PR 29827, so credit goes to its authors.
How to rerun
powershell -ExecutionPolicy Bypass -File e21r_chain.ps1Code
All code for E21R: e21r_chain.ps1, results.jsonl.
Prior work
No novelty claim. This is a memory accounting measurement of standard llama.cpp behavior.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.