Does the lower perplexity with quantized KV show up outside WikiText? 4B model on code and calibration text
On WikiText at 64K, quantized KV caches gave 1.5% to 3% lower perplexity than f16. On C++ source code at 64K that drop is gone: q8_0 and q4_0 sit 0.09% and 0.16% above f16, about where bf16 sits. On my calibration text at 16K they sit 0.1% and 0.2% below f16, inside the error bars. So far the drop is a WikiText effect.
| Experiment | E22B |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity, f16 KV cache with flash attention on as the reference. Text 1: about 900,000 characters of llama.cpp C++ source, context 65,536, 2 chunks. Text 2: my calibration file (bartowski calibration_datav3), context 16,384, 2 chunks. Each run reports perplexity, mean KL divergence from the reference and how often the top token matches. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, batch 2048, ubatch 512, flash attention on |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM |
| Verdict | BASELINE |
What I tested
q8_0, q4_0 and bf16 KV caches, plus an f16 repeat as a stability check. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Set before the run: if quantized caches land clearly below f16 on non-WikiText text while bf16 and the f16 repeat land at f16, the drop is a general property of the quantized formats; if not, it is tied to WikiText.
Result
| Text | K / V format | Mean PPL | PPL vs f16 | KLD vs f16 | Same top token |
|---|---|---|---|---|---|
| Code, 64K | f16 (reference) | 1.6868 | |||
| Code, 64K | q8_0 | 1.6883 | +0.09% ± 0.06% | 0.0060 ± 0.0007 | 98.99% |
| Code, 64K | q4_0 | 1.6895 | +0.16% ± 0.08% | 0.0106 ± 0.0009 | 98.48% |
| Code, 64K | bf16 | 1.6883 | +0.09% ± 0.05% | 0.0043 ± 0.0005 | 99.08% |
| Code, 64K | f16 (repeat) | 1.6873 | +0.03% ± 0.02% | 0.0000 | 100.00% |
| Calibration, 16K | f16 (reference) | 9.7430 | |||
| Calibration, 16K | q8_0 | 9.7338 | -0.10% ± 0.12% | 0.0068 ± 0.0011 | 97.06% |
| Calibration, 16K | q4_0 | 9.7237 | -0.20% ± 0.18% | 0.0143 ± 0.0018 | 95.52% |
| Calibration, 16K | bf16 | 9.7426 | -0.00% ± 0.13% | 0.0049 ± 0.0007 | 97.39% |
| Calibration, 16K | f16 (repeat) | 9.7490 | +0.06% ± 0.05% | 0.0000 | 99.99% |
On code at 64K the quantized caches do not lower perplexity. They are slightly above f16, in the same place as bf16, which is what normal rounding noise looks like.
On the calibration text at 16K the quantized caches are 0.1% to 0.2% below f16, but inside the error bars. The WikiText drop at 32K was 0.7% to 1.0% and at 64K 1.5% to 3.1%.
KL divergence at 64K on code (q8_0 0.0060, q4_0 0.0106) is about half of WikiText segment B and under a third of segment A. Cache error at a given length depends a lot on the text, so earlier claims that error grows with context need a test that holds the scored tokens fixed. That test (E22c) is running now.
Code is very predictable for this model (perplexity 1.69), which leaves less room for any change in perplexity. This is one code text and one calibration text, not a broad survey.
What happened
Both texts ran in about 9 minutes. The question was whether the lower perplexity with quantized caches seen on WikiText also appears on other kinds of text. It does not appear on C++ source at 64K and is within noise on the calibration text at 16K. I now treat it as a WikiText-specific effect and keep using KL divergence from f16 as the quality score.
How to rerun
powershell -ExecutionPolicy Bypass -File e22b_chain.ps1Code
All code for E22B: e22b_chain.ps1, results.csv.
Prior work
No novelty claim. This is a standard llama.cpp quality check of existing cache formats.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.