Quantized KV cache against a flash-attention-off f16 reference at 64K. 4B model

E22e showed the lower perplexity from quantized KV caches at 64K comes from the flash-attention path. Here I used an f16 cache with flash attention off as the reference and measured q8_0 and q4_0 caches. The planned flash-attention-off quantized arms could not run, because llama.cpp refuses a quantized V cache without flash attention, so the pre-set rule could not be checked. The flash-attention-on arms did run.

Experiment	E22F
Date	2026-10-30
Benchmark	llama-perplexity with two small patches: skip the first N tokens of the file, and score only the last 4,096 tokens of the single chunk. The same six WikiText-2 test spans as E22c to E22e (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K), each with exactly 65,536 tokens of context before it. Reference is an f16 KV cache with flash attention off.
Model	Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM
Verdict	BASELINE

What I tested

Against the f16 flash-attention-off reference: q8_0 K and V with flash attention off, q4_0 K and V with flash attention off (both failed to start), f16 with flash attention on, q8_0 K and V with flash attention on, q4_0 K and V with flash attention on. llama.cpp built with upstream PR 29827.

Pass bar, set before the run

None (supporting measurement). Set before the run: the drop is a flash-attention-path artifact if the flash-attention-off quantized arms stay within 0.5% PPL of the reference in at least 5 of 6 spans while flash-attention-on q8_0 drops at least 1% in at least 5 of 6.

Result

KV cache and attention path	PPL vs f16 FA-off per span (72K to 252K)	Mean PPL vs reference	Mean KLD	Same top token
f16, flash attention on	+0.78 / -0.31 / +0.75 / +0.19 / +0.06 / +0.33%	+0.30%	0.0090	97.14%
q8_0 K and V, flash attention on	-2.89 / -4.14 / -2.16 / -2.08 / -1.46 / -2.17%	-2.48%	0.0174	96.17%
q4_0 K and V, flash attention on	-2.77 / -4.13 / -2.55 / -1.96 / -0.74 / -1.75%	-2.32%	0.0264	95.20%
q8_0 or q4_0 K and V, flash attention off	did not run	n/a	n/a	n/a

The flash-attention-off quantized arms stopped at startup with the llama.cpp error: quantized V cache requires flash_attn to be enabled. So the first half of the pre-set rule could not be checked, and this run gives no verdict on it.

Flash-attention-on q8_0 lowered perplexity by at least 1% in all six spans, the second half of the rule.

Measured against f16 with flash attention on, q8_0 is 2.78% lower on average and q4_0 2.62% lower, larger than E22c's 2.7% for q8_0 against the flash-attention-on reference.

q4_0 has about 1.5 times the KL divergence of q8_0 (0.0264 vs 0.0174) but about the same perplexity drop, so perplexity is not tracking how far the outputs move.

A follow-up (E22g, running) keeps V in f16 so flash attention can be turned off, and measures q8_0 and q4_0 keys with it off and on.

Per-span numbers are in the code folder (results.csv).

What happened

Six spans took about 13 minutes. The plan was to measure the true cost of q8_0 and q4_0 caches with the attention path held fixed, but llama.cpp will not run a quantized V cache with flash attention off, so those arms failed in two seconds each. The arms that ran show that with flash attention on, q8_0 and q4_0 caches both score about 2.4% lower perplexity than the f16 flash-attention-off reference, while f16 with flash attention on scores 0.3% higher. Together with E22e this points again at the quantized flash-attention route, but it does not yet give an honest cost number for a quantized V cache.

How to rerun

powershell -ExecutionPolicy Bypass -File e22f_chain.ps1

Code

All code for E22F: e22f_chain.ps1, results.csv.

Prior work

No novelty claim. This is a quality check of existing llama.cpp cache formats and attention paths with a fixed-token scoring setup.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.