Does the lower perplexity with quantized KV show up on other text? 4B model at 64K
At 64K, quantized KV caches gave about 3% lower perplexity than f16 on the first WikiText chunks. On a different part of WikiText the drop is still there but about half as big: q8_0 and q4_0 both sit 1.5% below f16, while bf16 and an f16 repeat sit at f16. The second text I planned (my calibration file) was too short to run at 64K.
| Experiment | E22A |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity at context 65,536, 2 chunks, on WikiText-2 raw test starting at character 600,000 (a segment the earlier 64K runs did not score). The f16 KV cache with flash attention on is the reference; each run reports perplexity, mean KL divergence from the reference and how often the top token matches. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512, flash attention on |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
q8_0, q4_0 and bf16 KV caches, plus an f16 repeat as a stability check. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Set before the run: if quantized caches again land clearly below f16 while bf16 and the f16 repeat land at f16, the drop is a real property of the quantized formats and not one WikiText chunk.
Result
| K / V format | Mean PPL | PPL vs f16 | KLD vs f16 | Same top token |
|---|---|---|---|---|
| f16 / f16 (reference) | 9.074 | |||
| q8_0 / q8_0 | 8.937 | -1.51% ± 0.08% | 0.0113 ± 0.0008 | 96.75% |
| q4_0 / q4_0 | 8.940 | -1.48% ± 0.11% | 0.0186 ± 0.0011 | 95.66% |
| bf16 / bf16 | 9.083 | +0.10% ± 0.07% | 0.0074 ± 0.0007 | 97.37% |
| f16 / f16 (repeat) | 9.079 | +0.05% ± 0.02% | 0.0000 | 99.99% |
The lower perplexity with quantized caches replicates on a second WikiText segment, but at -1.5% instead of the -3.1% seen in E21x and E21z. Both quantized formats land at almost the same perplexity even though q4_0 moves the output much more (KLD 0.019 vs 0.011).
bf16 and the f16 repeat sit at f16, as in E21z, so this is something the quantized formats do. I still don't treat it as a quality gain; KL divergence from f16 is the score that counts.
q8_0's KLD here (0.0113) is about half of what it was on the first segment (0.0207), and q4_0's (0.0186) is about the same as q4_0 at 32K. The error at 64K depends noticeably on the text.
The second text, my calibration file, tokenizes to 66,338 tokens. llama-perplexity needs twice the context (131,072) to score at 64K, so that arm was skipped. The next run uses a long code text and scores the calibration file at 16K.
What happened
The WikiText runs finished in about 7 minutes. The question was whether the lower perplexity with quantized caches at 64K was a quirk of one WikiText chunk. It shows up again on a different segment, at about half the size. The next run checks a non-WikiText text (C++ source code) to see whether it holds outside Wikipedia prose.
How to rerun
powershell -ExecutionPolicy Bypass -File e22a_chain.ps1Code
All code for E22A: e22a_chain.ps1, results.csv.
Prior work
No novelty claim. This is a standard llama.cpp quality check of existing cache formats.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.