Does the cost of quantized keys grow with context? 16K, 32K and 64K, 4B model
E22c saw KV cache error rise with context length, but that was on the flash-attention path, which E22e to E22g showed adds its own shift. Here the same six spans are scored with 16K, 32K and 64K of context against an f16 flash-attention-off reference. With flash attention off, the extra error from 4-bit keys stays flat across lengths. The growth shows up in the full q4_0 cache on the flash-attention path.
| Experiment | E22H |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity with two small patches: skip the first N tokens of the file, and score only the last 4,096 tokens of the single chunk. The same six WikiText-2 test spans as E22c to E22g (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K), each scored with 16,384, 32,768 and 65,536 tokens of context before it. At each length the reference is an f16 KV cache with flash attention off. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
At each span and context length, against the f16 flash-attention-off reference: q8_0 keys with flash attention off (V f16), q4_0 keys with flash attention off (V f16), q4_0 keys and values with flash attention on (the usual deployable setting), and f16 with flash attention on as a path-only control. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Set before the run: if the mean KLD of q4_0 keys with flash attention off rises at least 1.5x from 16K to 64K and rises in at least 5 of 6 spans, the growth of cache error with context is real on the flash-attention-off path. Otherwise the growth seen with flash attention on is mostly a property of that path.
Result
| Cache and attention path | Mean KLD 16K / 32K / 64K | 64K vs 16K | Spans where 64K > 16K | Mean PPL vs reference 16K / 32K / 64K | Same top token 64K |
|---|---|---|---|---|---|
| q8_0 keys, V f16, flash attention off | 0.0058 / 0.0085 / 0.0095 | 1.64x | 5/6 | +0.22 / +0.38 / +0.10% | 97.24% |
| q4_0 keys, V f16, flash attention off | 0.0098 / 0.0134 / 0.0128 | 1.31x | 4/6 | +0.14 / +0.40 / -0.01% | 96.22% |
| q4_0 keys and values, flash attention on | 0.0130 / 0.0179 / 0.0264 | 2.02x | 5/6 | -0.21 / -0.50 / -2.32% | 95.20% |
| f16, flash attention on (control) | 0.0055 / 0.0098 / 0.0090 | 1.63x | 4/6 | -0.07 / +0.32 / +0.30% | 97.14% |
The pre-set bar is not met: q4_0 keys with flash attention off rise 1.31x from 16K to 64K, in 4 of 6 spans.
The f16 control, which only switches the attention path, also rises 1.63x. Part of the length trend is the reference itself drifting from any other path, not cache compression.
Subtracting the f16 control span by span, the extra KLD from q4_0 keys with flash attention off is 0.0043 / 0.0036 / 0.0039 at 16K / 32K / 64K, flat. For q8_0 keys it is 0.0003 / -0.0013 / 0.0005, about zero. For the full q4_0 cache with flash attention on it is 0.0075 / 0.0081 / 0.0174, more than double at 64K.
That last arm also quantizes V, which cannot be done with flash attention off in llama.cpp, so this run cannot say whether the growth comes from the flash-attention path or from quantized values. Both remain possible.
The 64K numbers repeat E22f and E22g exactly (q4_0 cache 0.0264, q4_0 keys 0.0128, q8_0 keys 0.0095, f16 control 0.0090).
Per-span numbers are in the code folder (results.csv).
What happened
Eighteen span and length pairs took about 32 minutes. On the flash-attention-off path, 4-bit keys add a roughly constant amount of error over the f16 path-switch control at every length, and 8-bit keys add almost none. The usual deployable setting, a q4_0 keys and values cache with flash attention on, is the one whose extra error doubles between 16K and 64K, and it is also the one whose perplexity drops at 64K. For the 1M-context goal, this means key compression tested on the flash-attention-off path should not get worse just because the context is longer, while the growth to watch is in values or the quantized flash-attention route.
How to rerun
powershell -ExecutionPolicy Bypass -File e22h_chain.ps1Code
All code for E22H: e22h_chain.ps1, results.csv.
Prior work
No novelty claim. This is a quality check of existing llama.cpp cache formats and attention paths at three context lengths with a fixed-token scoring setup.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.