How much each KV cache format changes the 4B model's predictions, and the quality bar a 3-bit cache has to meet
I ran the 4B model over the WikiText-2 test text at a 16K context with eight different KV cache formats and compared every next-token distribution to the full f16 cache. Perplexity barely moves for any of them. KL divergence does: 0.0072 for q8_0, about 0.015 for the 4-bit formats and 0.0166 for q4_0, the format I use now. Keeping keys at 8 bits and values at 4 bits (0.0100) beats the reverse (0.0145), so keys are the more sensitive half. A cache small enough for 1M tokens has to stay near q4_0's numbers.
| Experiment | E21V |
| Date | 2026-10-30 |
| Benchmark | llama-perplexity on the WikiText-2 raw test split, context 16,384, first 8 chunks (65,536 scored tokens). The f16 KV cache run is saved as the reference, and every other run reports perplexity, mean KL divergence from the reference and how often the top token matches. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, flash attention on, batch 2048, ubatch 512 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
K/V cache formats q8_0, q5_0, q4_1, iq4_nl, q4_0, K q8_0 with V q4_0, K q4_0 with V q8_0, and an f16 repeat as the noise floor. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None (supporting measurement). Planned before the run: report mean perplexity, mean KL divergence and same-top-token rate for each format against the f16 cache, to set the quality a sub-4-bit cache must match.
Result
| K / V format | Mean KLD | Same top token | PPL vs f16 | RMS change in top prob |
|---|---|---|---|---|
| f16 / f16 (repeat) | 0.0000 | 99.998% | +0.08% | 0.001% |
| q8_0 / q8_0 | 0.0072 ± 0.0005 | 97.30% | -0.13% | 2.48% |
| q5_0 / q5_0 | 0.0098 ± 0.0007 | 97.00% | -0.37% | 2.73% |
| q8_0 / q4_0 | 0.0100 ± 0.0006 | 96.69% | -0.01% | 2.70% |
| q4_0 / q8_0 | 0.0145 ± 0.0010 | 96.35% | -0.27% | 3.20% |
| q4_1 / q4_1 | 0.0146 ± 0.0008 | 96.05% | +0.13% | 3.24% |
| iq4_nl / iq4_nl | 0.0149 ± 0.0009 | 96.03% | -0.75% | 3.27% |
| q4_0 / q4_0 | 0.0166 ± 0.0010 | 95.85% | +0.06% | 3.33% |
Reference perplexity with the f16 cache is 7.790. Every format lands within -0.75% to +0.13% of it, so perplexity alone hides the differences. KL divergence and top-token agreement show them.
The f16 repeat gives KL divergence 0.0000 and 99.998% top-token agreement, so run-to-run noise is far below the gaps between formats.
iq4_nl has no flash-attention GPU kernel for the cache in this build, so it fell back to the CPU and took about 25 minutes instead of about 1. Its quality numbers are still valid.
This is one model, one text and one context length. Whether the error grows with context length is the next run.
Target for a cache small enough for 1M tokens (about 3.2 bits per value, see E21s): KL divergence at or below about 0.017 and top-token agreement at or above about 95.8% here.
What happened
All nine runs finished. The 8-bit cache is closest to the full-precision one, the 4-bit formats are all close to each other, and q4_0 is the weakest of them, though still within 0.1% on perplexity. The split runs show that 4-bit keys hurt more than 4-bit values, which matches the earlier bit-level measurements. These numbers are now the bar any smaller cache format has to clear before it is worth testing on long-context retrieval.
How to rerun
powershell -ExecutionPolicy Bypass -File e21v_chain.ps1Code
All code for E21V: e21v_chain.ps1, results.csv.
Prior work
No novelty claim. This is a standard llama.cpp quality measurement of existing cache formats.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.