Does KV cache error really grow with context? Same scored tokens at 16K, 32K and 64K, 4B model

Earlier runs mixed context length with which tokens got scored. This time I scored the exact same 4,096 WikiText tokens with 16K, 32K and 64K of context before them, at six places in the text. Average KL divergence from f16 still rises with length for every quantized cache (q8_0 0.0062, 0.0101, 0.0181). My stricter pre-set bar was not met because q4_0 rose from 32K to 64K in only 4 of 6 spans. The lower perplexity with quantized caches is also a length effect: none at 16K, about 1% at 32K, about 2.7% at 64K, on the same tokens.

Experiment	E22C
Date	2026-10-30
Benchmark	llama-perplexity with two small patches: skip the first N tokens of the file, and score only the last 4,096 tokens of the single chunk. Six target spans in WikiText-2 test, each the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K or 252K. Each span is scored with exactly 16,384, 32,768 and 65,536 tokens of context before it. Reference is an f16 KV cache with flash attention on. Each run reports perplexity on the target span, mean KL divergence from the reference and how often the top token matches.
Model	Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512, flash attention on
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM
Verdict	BASELINE

What I tested

q8_0, q4_0, K q8_0 with V q4_0, K q4_0 with V q8_0, and bf16 as a noise floor. llama.cpp built with upstream PR 29827.

Pass bar, set before the run

None (supporting measurement). Set before the run: if KL(64K)/KL(32K) is above 1 for both q4_0 and q8_0 in at least 5 of 6 spans, the growth of cache error with context is a real length effect; otherwise the earlier rise was caused by scoring different tokens at each length.

Result

KV format	Mean KLD 16K	Mean KLD 32K	Mean KLD 64K	Spans where 64K > 32K	PPL vs f16 at 16K / 32K / 64K
q8_0	0.0062	0.0101	0.0181	6 of 6	-0.1% / -1.0% / -2.7%
q4_0	0.0127	0.0178	0.0255	4 of 6	-0.1% / -0.7% / -2.5%
K q8_0, V q4_0	0.0079	0.0136	0.0216	6 of 6	0.0% / -1.1% / -2.7%
K q4_0, V q8_0	0.0093	0.0135	0.0237	6 of 6	-0.1% / -0.9% / -2.6%
bf16 (noise floor)	0.0052	0.0062	0.0085	3 of 6	+0.1% / -0.1% / -0.1%

Means are over the six spans. Per-span numbers are in the code folder (results.csv).

The two q4_0 spans that did not rise from 32K to 64K were at 0.93x and 0.96x. Spans vary a lot: q8_0 KL divergence at 64K ranges from 0.009 to 0.041 depending on where in the text the span is.

Because the scored tokens are identical across lengths, the rise in average error with context is not just a text-mixing artifact, but with only six spans the per-span result is noisy.

At 64K every quantized cache gave lower perplexity than f16 on all six spans (ratios 0.961 to 0.992), and bf16 never did. At 16K there is no drop. So on WikiText the drop grows with how much context sits in the quantized cache. On C++ code at 64K (E22b) it did not appear. I do not have an explanation yet.

Next (E22d, running): the same six spans at 64K with only K or only V quantized, to see which half causes the drop.

What happened

The full grid of 90 quantized runs plus 18 references took about 31 minutes. The question was whether KV cache error really grows with context or only looked that way because longer contexts scored different tokens. On identical tokens the average error still rises with length for every quantized format, but q4_0 missed my 5 of 6 per-span bar, so I count the length effect as likely but not confirmed. The lower perplexity with quantized caches turned out to depend on context length too.

How to rerun

powershell -ExecutionPolicy Bypass -File e22c_chain.ps1

Code

All code for E22C: e22c_chain.ps1, results.csv.

Prior work

No novelty claim. This is a quality check of existing llama.cpp cache formats with a fixed-token scoring setup.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.