KV cache error vs context, filled in at 32K: 4B model, 16K, 32K and 64K
I added a 32K point to the KV cache quality curve and two more formats at 64K. KL divergence from the full f16 cache rises at every step for every format, with most of the rise between 32K and 64K. Keys stay the more sensitive half at every length. 8-bit q8_0 at 64K (0.0207) is already worse than 4-bit q4_0 at 32K (0.0186).
| Experiment | E21Y |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity on the WikiText-2 raw test split, context 32,768 (4 chunks) and 65,536 (2 chunks), both covering the same first ~131K tokens. The f16 KV cache run is the reference; every other run reports perplexity, mean KL divergence from the reference and how often the top token matches. 16K numbers are from E21v and the other 64K numbers from E21x, same text and settings. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, flash attention on, batch 2048, ubatch 512 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
K/V cache formats q8_0, q5_0, q4_0, K q8_0 with V q4_0, K q4_0 with V q8_0, and an f16 repeat at 32K; q5_0 and K q4_0 with V q8_0 at 64K. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Expected before the run: KL divergence should rise monotonically from 16K to 32K to 64K for each format.
Result
| K / V format | KLD at 16K | KLD at 32K | KLD at 64K | Same top token 32K | PPL vs f16 at 32K |
|---|---|---|---|---|---|
| f16 / f16 (repeat) | 0.0000 | 0.0000 | 0.0000 | 99.998% | +0.10% |
| q8_0 / q8_0 | 0.0072 | 0.0102 ± 0.0008 | 0.0207 | 97.05% | -1.04% |
| q8_0 / q4_0 | 0.0100 | 0.0128 ± 0.0009 | 0.0254 | 96.37% | -0.88% |
| q5_0 / q5_0 | 0.0098 | 0.0131 ± 0.0009 | 0.0248 ± 0.0016 | 96.65% | -0.97% |
| q4_0 / q8_0 | 0.0145 | 0.0152 ± 0.0009 | 0.0282 ± 0.0016 | 96.18% | -0.85% |
| q4_0 / q4_0 | 0.0166 | 0.0186 ± 0.0010 | 0.0347 | 95.52% | -0.73% |
The expected monotonic rise held for all five formats. The step from 32K to 64K (about 1.9x to 2.0x) is much larger than the step from 16K to 32K (1.05x to 1.42x).
4-bit keys with 8-bit values is worse than 8-bit keys with 4-bit values at 16K, 32K and 64K, so keys are the half to protect at every length tested.
Quantized caches again gave slightly lower perplexity than f16: about 0.7 to 1.0% lower at 32K versus about 3% at 64K, so this gap also grows with length. I still can't explain it and don't count it as a gain. The next run checks whether the f16 reference itself is the cause by comparing it with bf16, f32 and flash attention off.
The 32K and 64K points use different chunk counts over the same text, and llama-perplexity scores the second half of each chunk, so the longer runs score tokens that sit further into their context.
What happened
All runs finished in about 14 minutes. The curve confirms the E21x finding with an extra point: cache quantization damage grows with context for every format, and grows fastest between 32K and 64K. For the 1M-token goal this means a smaller cache format has to be judged at long context, and keys need more care than values.
How to rerun
powershell -ExecutionPolicy Bypass -File e21y_chain.ps1Code
All code for E21Y: e21y_chain.ps1, results.csv.
Prior work
No novelty claim. This is a standard llama.cpp quality measurement of existing cache formats at several context lengths.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.