KV cache quantization hurts more at longer context: 4B model, 16K vs 64K
I repeated the KV cache quality test at a 64K context. Compared with the full f16 cache, the damage from a quantized cache roughly doubles or triples: KL divergence for q4_0 goes from 0.0166 at 16K to 0.0347 at 64K, and for q8_0 from 0.0072 to 0.0207. So a cache format for 1M tokens has to be judged at long context, not 16K. The 128K run did not happen because the measuring tool ran out of system memory.
| Experiment | E21X |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity on the WikiText-2 raw test split, context 65,536, first 2 chunks. The f16 KV cache run is saved as the reference, and every other run reports perplexity, mean KL divergence from the reference and how often the top token matches. 16K numbers are from E21v on the same text. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, flash attention on, batch 2048, ubatch 512 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
K/V cache formats q8_0, q4_0, K q8_0 with V q4_0, and an f16 repeat as the noise floor, at context 65,536 and 131,072. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Planned before the run: if q4_0 KL divergence at long context is more than about 1.5x its 16K value (0.0166), treat the error as growing with context.
Result
| K / V format | KLD at 16K | KLD at 64K | Same top token 16K / 64K | PPL vs f16 at 64K |
|---|---|---|---|---|
| f16 / f16 (repeat) | 0.0000 | 0.0000 | 99.998% / 99.997% | +0.16% |
| q8_0 / q8_0 | 0.0072 | 0.0207 ± 0.0014 | 97.30% / 96.43% | -3.11% |
| q8_0 / q4_0 | 0.0100 | 0.0254 ± 0.0016 | 96.69% / 95.80% | -3.19% |
| q4_0 / q4_0 | 0.0166 | 0.0347 ± 0.0020 | 95.85% / 95.25% | -2.91% |
q4_0 KL divergence at 64K is 2.1x its 16K value and q8_0 is 2.9x, both above the 1.5x line set before the run. The order of the formats stays the same, and 8-bit keys with 4-bit values is still clearly better than full q4_0.
At 64K every quantized cache gave about 3% lower perplexity than the f16 cache (8.27 to 8.28 vs 8.53), while the f16 repeat matched the reference. I don't have an explanation yet and am not counting it as a gain. KL divergence and top-token agreement both say the quantized caches moved further from the reference.
The 131,072-token run failed for every format. The f16 reference run crashed right after starting the chunk, most likely because llama-perplexity keeps every position's logits in system RAM (context times vocabulary times 4 bytes, roughly 130 GB at 128K, more than the rig's 128 GB). It is a limit of the measuring tool, not of the GPU or the cache.
Only 2 chunks at 64K, so the 64K numbers rest on fewer scored tokens than the 16K ones. A 32K run to fill in the curve is next.
What happened
The 64K runs all finished and show that quantization damage grows with context length for every format tested. That raises the bar for the 1M-token goal: a smaller cache format has to hold up at long context, and its quality should be checked there. The 128K runs failed because the measuring tool needs more system memory than the rig has at that length.
How to rerun
powershell -ExecutionPolicy Bypass -File e21x_chain.ps1Code
All code for E21X: e21x_chain.ps1, results.csv.
Prior work
No novelty claim. This is a standard llama.cpp quality measurement of existing cache formats at a longer context.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.