A bit-level look at real KV cache from the 4B: no single cheap redundancy gets it to 3 bits, but key channels have stable scales
I saved the full f16 KV cache from the 4B model after it read three real texts (two WikiText slices and a math-solutions text, 27K to 30K tokens each) and measured what is actually in it: raw entropy, how low-rank it is, how similar neighbouring tokens are, how much error common 2, 3 and 4 bit roundings cause, and whether any of this repeats across texts. The goal is to find structure a new cache format could use to get from 4.5 bits per value down to about 3.2, the target from E21s.
Correction (Oct 30, 2026): the 19-41% last-layer V error used a per-channel centered-variance normalization that is much harsher for high-mean blocks. Under the standard relative-error metric, layer 7's V is ~3% at 3-bit rotated, the same as other blocks, so it is not an outlier. See E21w.
| Experiment | E21U |
| Date | 2026-10-30 |
| Benchmark | Offline measurement of saved KV cache, no accuracy test. Error is squared error divided by per-channel variance, on the first 8,192 tokens of each layer. Medians over 3 texts x 8 attention layers. |
| Model | Qwen3.5-4B, Q4_K_M GGUF weights, KV cache saved in f16 (and once in q4_0) with llama-server slot save, flash attention on |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB for the capture, CPU for the analysis |
| Verdict | BASELINE |
What I tested
Texts: wikiA (WikiText test, 28,952 tokens), wikiB (a later WikiText slice, 27,083 tokens), mathA (math solution text, 30,287 tokens). Cache layout found in the saved file: 8 full-attention layers, K then V, 4 KV heads x 256 dims each, f16. Rounding methods (payload bits only, groups of 32): per-token absmax, per-token min-max, per-token min-max after a 256-dim Hadamard rotation, per-channel min-max over 32-token groups.
Pass bar, set before the run
None (supporting measurement, analysis plan written before the capture). Replication check: a structure counts only if it shows up in both WikiText slices and in the math text.
Result
| Measure | K (median, range) | V (median, range) |
|---|---|---|
| Raw f16 entropy, bits per value | 13.63 (13.56-13.71) | 13.63 (13.53-13.92) |
| Dims for 90% of variance, of 256 per head | 123 (87-152) | 147 (109-186) |
| Neighbour token cosine | 0.70 | 0.56 |
| Neighbour difference variance vs signal | 73% | 120% |
| 3-bit error, per-token absmax | 6.4% | 6.3% |
| 3-bit error, per-token min-max | 4.6% | 4.5% |
| 3-bit error, rotated per-token min-max | 3.6% | 3.6% |
| 3-bit error, per-channel min-max | 1.6% | 2.6% |
| 4-bit error, rotated per-token min-max | 0.78% | 0.78% |
| 2-bit error, rotated per-token min-max | 19.7% | 19.8% |
| Stored q4_0 error (wikiA) | 1.0-1.7% | 0.9-1.9%, last layer 6.3% |
| Per-channel spread match, wiki vs wiki / wiki vs math | r 0.99 / 0.94 | r 0.97 / 0.72 |
Lossless coding alone cannot get close: the raw values carry about 13.6 bits of order-0 entropy out of 16.
The cache is not low-rank enough for big savings: about half of each head's dims are needed for 90% of the variance, and about 230 of 256 for 99%.
Neighbouring tokens are only loosely similar. Coding differences between neighbours would help K a little and would make V worse.
Scaling each key channel separately cut 3-bit error by about 2.2x versus the best per-token method, and the per-channel key scales match closely across texts and even across domains, so they could be set once ahead of time instead of stored with the cache. Value channels are less stable across domains.
One block is the outlier: the V cache of the last attention layer had 19 to 41% error at 3 bits versus about 3.5 to 6% elsewhere, with heavy tails (kurtosis about 11).
Correction (Oct 30, 2026): the 19-41% last-layer V error used a per-channel centered-variance normalization that is much harsher for high-mean blocks. Under the standard relative-error metric, layer 7's V is ~3% at 3-bit rotated, the same as other blocks, so it is not an outlier. See E21w.
First q4_0 comparison was wrong: this llama.cpp build stores the 4-bit K cache rotated by a 256-dim Hadamard transform and V by a 64-dim one, so comparing against unrotated f16 gave nonsense (about 240% error). Applying the same rotation gives the 1 to 2% shown. The note is in the code folder.
The rounding errors above count payload bits only. Per-group scales add about 0.5 to 1 bit per value at group size 32, which matters against a 3.2-bit budget.
These are reconstruction errors, not answer quality. E21v (running now) measures how cache error maps to model output quality.
What happened
Capturing the cache took under a minute per text using llama-server's slot save. The analysis found no single cheap trick that gets from 4.5 to about 3.2 bits: entropy, rank and token-to-token similarity each leave most of the bits in place. Two measured leads did replicate across texts: key channels have stable, predictable scales, and almost all of the worst error sits in one block, the last layer's V cache. Both have prior art (per-channel key quantization and calibrated scales are published, for example KIVI and KVQuant), so any method built on them has to go through a prior-art search and a separate novelty check before I spend rig time on it.
How to rerun
powershell -ExecutionPolicy Bypass -File e21t_capture_chain.ps1
python e21u_analyze.pyCode
All code for E21U: NOTE.txt, e21t_capture_chain.ps1, e21t_capture_results.jsonl, e21u_analyze.py.
Prior work
No novelty claim. This is a measurement of a standard KV cache with standard rounding methods.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.