Quantized keys only, with flash attention off and on, at 64K. 4B model
llama.cpp will not run a quantized V cache with flash attention off, so E22f could not measure the honest cost of a quantized cache. Here V stays f16 and only the K cache is quantized, which lets flash attention be turned off. Against an f16 flash-attention-off reference, q8_0 and q4_0 keys with flash attention off barely move perplexity, while the same keys with flash attention on lower it by about 1.4%.
| Experiment | E22G |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity with two small patches: skip the first N tokens of the file, and score only the last 4,096 tokens of the single chunk. The same six WikiText-2 test spans as E22c to E22f (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K), each with exactly 65,536 tokens of context before it. Reference is an f16 KV cache with flash attention off. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
Against the f16 flash-attention-off reference, V cache always f16: q8_0 keys with flash attention off, q4_0 keys with flash attention off, q8_0 keys with flash attention on, q4_0 keys with flash attention on, and f16 keys with flash attention on as a path-only control. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Set before the run: if the flash-attention-off key-only arms stay within 0.5% PPL of the reference in at least 5 of 6 spans, quantizing keys alone costs about nothing on the flash-attention-off path. Also report the KLD ranking of q8_0 against q4_0.
Result
| K cache and attention path (V f16) | PPL vs f16 FA-off per span (72K to 252K) | Mean PPL vs reference | Mean KLD | Same top token | Spans within 0.5% |
|---|---|---|---|---|---|
| q8_0 keys, flash attention off | +0.60 / -0.71 / +0.46 / -0.18 / +0.07 / +0.39% | +0.10% | 0.0095 | 97.24% | 4/6 |
| q4_0 keys, flash attention off | -0.23 / -0.15 / -0.28 / +0.32 / +0.23 / +0.05% | -0.01% | 0.0128 | 96.22% | 6/6 |
| q8_0 keys, flash attention on | -1.88 / -2.49 / -1.29 / -1.31 / -0.54 / -1.14% | -1.44% | 0.0145 | 96.74% | 0/6 |
| q4_0 keys, flash attention on | -1.57 / -2.29 / -1.54 / -0.48 / -0.56 / -1.41% | -1.31% | 0.0182 | 95.96% | 1/6 |
| f16 keys, flash attention on (control) | +0.78 / -0.31 / +0.75 / +0.19 / +0.06 / +0.33% | +0.30% | 0.0090 | 97.14% | 4/6 |
The pre-set bar is met for q4_0 keys (6 of 6 spans within 0.5%) but not for q8_0 keys (4 of 6). The f16 control, which changes only the attention path and not the cache, is also 4 of 6, so a 0.5% band is about the size of the run-to-run difference between the two attention paths. I read the q8_0 miss as noise, not a real cost, but the bar as written is not met.
With flash attention off, q8_0 keys move the outputs about as much as switching f16 to flash attention does (KLD 0.0095 vs 0.0090). q4_0 keys add a little more (0.0128).
Turning flash attention on with quantized keys adds about 0.005 KLD on top and brings back the 1.3 to 1.4% perplexity drop seen in E22c to E22f. So the drop needs both quantized keys and the flash-attention route.
KLD ranks q4_0 above q8_0 on both paths, as expected, while perplexity does not. KLD against the flash-attention-off f16 reference is the score I will use for cache compression tests.
A follow-up (E22h, running) checks whether the cost of quantized keys on the flash-attention-off path grows with context, at 16K, 32K and 64K.
Per-span numbers are in the code folder (results.csv).
What happened
Six spans took about 20 minutes. Keeping V in f16 let the flash-attention-off arms run this time. On that path, 8-bit and 4-bit keys leave perplexity within a fraction of a percent of the f16 reference, and their KL divergence is close to the small shift from just changing the attention path. With flash attention on, the same keys lower perplexity by 1.3 to 1.4% and move the outputs further. This backs up E22e: the lower perplexity from quantized caches is a property of the quantized flash-attention route, not a sign that quantized keys help the model.
How to rerun
powershell -ExecutionPolicy Bypass -File e22g_chain.ps1Code
All code for E22G: e22g_chain.ps1, results.csv.
Prior work
No novelty claim. This is a quality check of existing llama.cpp cache formats and attention paths with a fixed-token scoring setup.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.