Almost all per-token GPU memory is KV cache, so 1M tokens on the 4B needs about 3.2 bits per cache value
I loaded the 4B model at 14 context sizes with three KV cache formats (q4_0, q8_0, f16) and fit a straight line to dedicated GPU memory for each. The lines are almost perfectly straight. Per-token cost is 9,277 bytes at q4_0, 17,540 at q8_0 and 32,789 at f16, which works out to about 16,384 cache values per token and only about 119 bytes per token of anything else. The fixed cost is about 2.95 GB. At q4_0, 1,048,576 tokens would need about 12.2 GB, so the cache has to shrink to about 3.2 bits per value to fit on the 10 GB card.
| Experiment | E21S |
| Date | 2026-10-29 |
| Benchmark | Memory measurement only, no accuracy test. llama-server loaded at each setting; per-process Dedicated and Shared Usage read from the Windows GPU Process Memory counters (max of 5 samples), then one short prompt. Straight-line fit per cache format. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, flash attention on, ubatch 8, YaRN context extension above 256K |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
q4_0 K/V at 4K, 128K, 256K, 384K, 512K, 640K, 704K; q8_0 K/V at 4K, 128K, 256K, 384K; f16 K/V at 4K, 128K, 192K. K = 1024 tokens, so 704K is 720,896 tokens. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Planned analysis, written before the run: one least-squares line per cache format; slope is bytes per token, intercept is fixed cost; the slope differences between formats separate KV bytes from other per-token memory.
Result
| Cache format | Bits per value | Bytes per token (fit) | Intercept | Largest point measured | Dedicated there |
|---|---|---|---|---|---|
| q4_0 | 4.5 | 9,277 | 2,950 MiB | 704K | 9,321 MiB |
| q8_0 | 8.5 | 17,540 | 2,939 MiB | 384K | 9,507 MiB |
| f16 | 16 | 32,789 | 2,913 MiB | 192K | 9,061 MiB |
Fit residuals: f16 under 1 MiB, q8_0 up to 19 MiB, q4_0 up to 25 MiB. Shared memory stayed at 78 to 90 MiB at every point, so nothing spilled.
Fitting bytes per token = other + values x bits / 8 across the three formats gives about 16,344 cache values per token (16,384 would be 8 attention layers x 4 KV heads x 256 dims x K and V) and about 119 bytes per token of non-cache growth.
Prediction at q4_0 for 1,048,576 tokens: about 12.2 GB dedicated, against roughly 9.5 to 9.6 GB usable on this card.
To fit 1,048,576 tokens with the weights and fixed buffers unchanged, the cache must average about 6,575 to 6,675 bytes per token, which is 3.16 to 3.21 bits per value including any scale overhead. q4_0 is 4.5. The other route is cutting about 2.6 GB from the fixed cost, or a mix of both.
This only says what must fit. Whether a 3-bit cache keeps answers right is a separate test.
What happened
Every context size loaded and stayed fully on the GPU. The memory lines were close to perfectly straight, and their slopes lined up with the bits per value of each cache format, which means the per-token cost on this setup is almost entirely the KV cache itself. That turns the 1M-token goal into one clear number: about 3.2 bits per cache value, against 4.5 for the 4-bit format I use now. The memory saving that lets these sizes load at all is upstream llama.cpp PR 29827, so credit goes to its authors.
How to rerun
powershell -ExecutionPolicy Bypass -File e21s_chain.ps1
python3 e21s_fit.pyCode
All code for E21S: e21s_chain.ps1, e21s_fit.py, results.csv.
Prior work
No novelty claim. This is a memory accounting measurement of standard llama.cpp behavior.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.