Is the f16 KV reference at fault? bf16, flash attention off and f32 at 64K, 4B model
Quantized KV caches gave about 3% lower perplexity than the f16 cache at 64K, so I checked whether the f16 reference itself was the problem. It isn't: bf16 and f16 with flash attention off land within 0.4% of f16, while q8_0 again lands 3.1% lower. A side finding: just changing the number format or attention path already moves the output by a KL divergence of about 0.011, roughly half of what q8_0 does at 64K.
| Experiment | E21Z |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity on the WikiText-2 raw test split, context 65,536, 2 chunks (same text and settings as E21x and E21y). The f16 KV cache run with flash attention on is the reference; every other run reports perplexity, mean KL divergence from the reference and how often the top token matches. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
q8_0 cache repeat (stability check), f16 cache with flash attention off, bf16 cache with flash attention on, f32 cache with flash attention on. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Set before the run: if bf16, f32 or flash attention off land near q8_0's perplexity (about 8.27), the f16 reference is degraded at long context and the earlier KL bars must be redone; if they land near f16 (about 8.53), the drop comes from quantization itself.
Result
| K / V format | Flash attention | Mean PPL | PPL vs f16 | KLD vs f16 | Same top token |
|---|---|---|---|---|---|
| f16 / f16 (reference) | on | 8.532 | |||
| q8_0 / q8_0 (repeat) | on | 8.266 | -3.11% ± 0.12% | 0.0207 ± 0.0014 | 96.43% |
| f16 / f16 | off | 8.519 | -0.15% ± 0.09% | 0.0115 ± 0.0009 | 97.22% |
| bf16 / bf16 | on | 8.500 | -0.38% ± 0.09% | 0.0107 ± 0.0008 | 97.20% |
| f32 / f32 | on | 8.545 | +0.16% ± 0.03% | 0.0000 | 99.997% |
The q8_0 repeat matched E21x exactly (KLD 0.0207, same top token 96.43%), so the run is stable.
bf16 and flash attention off both sit next to f16, not next to q8_0. So the f16 reference is not degraded at 64K, and the KL bars from E21v, E21x and E21y stand. The lower perplexity is something the quantized formats do, and I still don't count it as a gain.
bf16 and flash attention off each move the output by KLD about 0.011 even though neither loses meaningful information compared with f16. That is about half of q8_0's 0.0207 at 64K, so any new cache format scored at 64K should be read against a floor of roughly 0.011, not zero.
The f32 arm gave KLD of exactly zero, identical to f16. Most likely it ran through the same f16 flash attention path, so I treat it as uninformative rather than as evidence.
What happened
All five runs finished in about 8 minutes. The question was whether the f16 cache, used as the reference in all the recent KV tests, was itself off at 64K. It wasn't. The next run checks whether the lower perplexity with quantized caches shows up on other text too, or only on these WikiText chunks.
How to rerun
powershell -ExecutionPolicy Bypass -File e21z_chain.ps1Code
All code for E21Z: e21z_chain.ps1, results.csv.
Prior work
No novelty claim. This is a standard llama.cpp quality check of existing cache formats and attention paths.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.