Is the 64K quantized KV cache perplexity drop caused by K or V? 4B model
At 64K context a quantized KV cache gives about 2.7% lower perplexity than f16 on the same tokens. I quantized only K or only V to see which half causes it. Neither does on its own: K-only q8_0 gives about 60% of the drop and V-only q8_0 about 69%, in all six spans. q4_0 halves give nearly the same drops as q8_0 halves even though q4_0 adds far more rounding error.
| Experiment | E22D |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity with two small patches: skip the first N tokens of the file, and score only the last 4,096 tokens of the single chunk. The same six WikiText-2 test spans as E22c (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K), each with exactly 65,536 tokens of context before it. Reference is an f16 KV cache with flash attention on. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512, flash attention on |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
K q8_0 with V f16, K f16 with V q8_0, K q4_0 with V f16, K f16 with V q4_0, and q8_0 for both as a repeat of E22c. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Set before the run: if K-only arms reproduce at least 70% of the q8_0 drop in at least 5 of 6 spans and V-only arms less than 30%, the drop is a key-side effect; the reverse points at V; both partial means the two interact or share a cause.
Result
| KV format | Mean PPL vs f16 | Share of q8_0 drop per span | Mean share | Mean KLD | Same top token |
|---|---|---|---|---|---|
| K q8_0, V f16 | -1.66% | 71 / 55 / 70 / 66 / 39 / 58% | 60% | 0.0139 | 96.76% |
| K f16, V q8_0 | -1.93% | 82 / 74 / 70 / 68 / 58 / 64% | 69% | 0.0134 | 96.70% |
| K q4_0, V f16 | -1.53% | 62 / 49 / 78 / 29 / 40 / 69% | 55% | 0.0182 | 95.91% |
| K f16, V q4_0 | -1.87% | 70 / 69 / 73 / 77 / 44 / 69% | 67% | 0.0183 | 96.07% |
| K q8_0, V q8_0 (repeat) | -2.71% | 100% by definition | 100% | 0.0181 | 96.22% |
Shares are (f16 PPL minus arm PPL) divided by (f16 PPL minus q8_0/q8_0 PPL), per span, in span order 72K to 252K. Per-span numbers are in the code folder (results.csv).
The q8_0 repeat matches E22c: mean -2.71% here vs -2.7% there, with per-span drops from 1.5% to 3.6%.
Every arm gave lower perplexity than f16 in all six spans.
The halves add up to more than the full drop (60% + 69%), so the effect is not split cleanly between K and V.
A hypothesis that rounding error on K sharpens attention like a lower temperature is ruled out by its own pre-set rule, because V-only q8_0 gave at least half the drop in 6 of 6 spans. That planned follow-up was dropped.
My working guess, not tested yet: because q8_0 and q4_0 give similar drops, the cause may be the different flash-attention code path used for quantized caches rather than the rounding itself. E22e (running) checks this by quantizing K with flash attention off.
What happened
Thirty quantized runs plus six references took about 16 minutes. The K-side explanation failed my pre-set reading: K-only q8_0 reached 70% of the drop in only 1 of 6 spans, while V-only q8_0 gave at least 58% in every span. Both halves each produce most of the drop, so the cause is shared or interacting. Since the drop barely depends on how coarse the rounding is, the next test separates the flash-attention code path from the quantization itself.
How to rerun
powershell -ExecutionPolicy Bypass -File e22d_chain.ps1Code
All code for E22D: e22d_chain.ps1, results.csv.
Prior work
No novelty claim. This is a quality check of existing llama.cpp cache formats with a fixed-token scoring setup.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.