The 64K quantized KV cache perplexity drop comes from the flash-attention path, not the rounding. 4B model
At 64K context a quantized KV cache gave lower perplexity than f16 on the same tokens. I ran the same q8_0 keys with flash attention off. With flash attention off the drop disappears in all six spans, so it comes from the kernel path the quantized cache takes, not from quantization making the model better.
| Experiment | E22E |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity with two small patches: skip the first N tokens of the file, and score only the last 4,096 tokens of the single chunk. The same six WikiText-2 test spans as E22c and E22d (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K), each with exactly 65,536 tokens of context before it. Reference is an f16 KV cache with flash attention on. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
f16 cache with flash attention off, K q8_0 with V f16 and flash attention off, K q8_0 with V f16 and flash attention on (repeat of E22d), and an f32 cache with flash attention on. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Set before the run: quantization effect if K q8_0 with flash attention off reproduces at least 70% of the flash-attention-on K q8_0 drop in at least 5 of 6 spans; kernel-path effect if it reproduces less than 30% in at least 5 of 6 spans; otherwise mixed.
Result
| KV cache and attention path | PPL vs f16 per span (72K to 252K) | Mean PPL vs f16 | Mean KLD | Same top token |
|---|---|---|---|---|
| f16, flash attention off | -0.48 / +0.79 / -0.70 / -0.15 / -0.06 / -0.23% | -0.14% | 0.0089 | 97.14% |
| K q8_0, V f16, flash attention off | -0.03 / -0.20 / -0.26 / -0.36 / +0.01 / +0.09% | -0.12% | 0.0092 | 97.21% |
| K q8_0, V f16, flash attention on | -2.50 / -1.99 / -2.00 / -1.48 / -0.60 / -1.43% | -1.66% | 0.0139 | 96.76% |
| f32, flash attention on | identical to f16 | 0% | 0.0000 | 100% |
Share of the flash-attention-on K q8_0 drop that survives with flash attention off, per span: 1 / 10 / 13 / 24 / -2 / -6%. Under 30% in all six spans.
The flash-attention-on K q8_0 arm reproduces E22d exactly (mean -1.66%, KLD 0.0139).
For that arm llama.cpp logs that it has no vector kernel for mixed q8_0 and f16 and converts K and V to f16 instead, so the quantized cache goes through a different kernel route than the f16 reference.
The f32 control is not informative. It gave KL divergence of exactly zero and the same top token everywhere, so it ran the same way as f16. It did show that the stored reference perplexity reads about 0.07% low, which does not change any reading.
Turning flash attention off with an f16 cache moves the outputs about as much as q8_0 keys do with it off (KLD 0.0089 vs 0.0092), so part of what earlier tests measured as cache quantization error is really attention-path difference.
Per-span numbers are in the code folder (results.csv).
What happened
Four arms over six spans took about 16 minutes. By the rule set before the run this is a kernel-path effect: the same q8_0 keys lower perplexity by about 1.7% through the flash-attention path and by about 0.1% without it. So the lower perplexity from quantized caches at 64K, seen in E22b to E22d, is not a quality gain from quantization. The next test (E22f, running) measures the real cost of q8_0 and q4_0 caches with the attention path held fixed, which becomes the honest bar for new cache compression methods.
How to rerun
powershell -ExecutionPolicy Bypass -File e22e_chain.ps1Code
All code for E22E: e22e_chain.ps1, results.csv.
Prior work
No novelty claim. This is a quality check of existing llama.cpp cache formats and attention paths with a fixed-token scoring setup.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.