PROVISIONAL: known 3-bit KV cache methods on the 4B fall well short of 4-bit quality (proxy queries)
PROVISIONAL, measured with proxy queries. I compared four known 3-bit ways to round the 4B model's real KV cache: plain, Hadamard-rotated, per-channel key scaling, and scaling plus rotation. Per-channel key scaling lowers key error but makes attention output worse once rotation is on, and the best mix of bit widths at about 3.2 bits has about 2.2x the attention error and about 12x the output drift of today's rotated 4-bit cache. This also withdraws E21u's claim that the last layer's V is an outlier.
| Experiment | E21W |
| Date | 2026-10-30 |
| Benchmark | Offline CPU measurement on saved f16 KV cache, no accuracy test. PROVISIONAL: the dumps had no queries, so attention output and KL use post-RoPE keys at 128 random positions in tokens 3,000 to 6,000 as proxy queries (causal over earlier keys). K/V error is squared error over signal energy. Attention error is relative error of the attention output, KL is the divergence of the attention distribution from f16. |
| Model | Qwen3.5-4B, Q4_K_M GGUF weights, KV cache from the E21t f16 slot saves |
| Hardware | One of my dedicated test rigs for the capture (RTX 3080 10 GB), CPU only for this analysis |
| Verdict | BASELINE |
What I tested
Texts: wikiA, wikiB, mathA (the E21t captures), first 6,000 tokens of all 8 full-attention layers, 4 KV heads each (32 units per text). Variants at 3 and 4 bits, groups of 32: A plain per-token min-max, B rotated by a 256-dim Hadamard (as this llama.cpp build does), C per-channel K scaling (calibrated offline on wikiA), D per-channel K scaling first, then rotation, scales divided back out after dequantizing. Then a per-layer K/V bit allocation (2 to 4 bits, groups of 128) under a 3.2-bit average, fit on wikiA and checked on all three texts.
Pass bar, set before the run
None (supporting measurement). Question set before the run: does per-channel K scaling add anything on top of rotation, and how close does the best known allocation at about 3.2 bits get to rotated q4_0.
Result
| Measure | wikiA | wikiB | mathA |
|---|---|---|---|
| D vs B: K error ratio (lower is better for D) | 0.87 | 0.89 | 0.92 |
| D vs B: attention error ratio | 1.05 | 1.05 | 1.05 |
| D vs B: KL ratio | 1.35 | 1.33 | 1.37 |
| Units where B beats D on attention error, of 32 | 28 | 29 | 29 |
| 3-bit attention error A / B / C / D | 0.189 / 0.154 / 0.181 / 0.162 | 0.192 / 0.156 / 0.186 / 0.164 | 0.193 / 0.158 / 0.191 / 0.167 |
| Rotated q4_0 (B 4-bit): attention error / KL | 0.076 / 0.0091 | 0.076 / 0.0086 | 0.077 / 0.0083 |
| Best rotated allocation, 3.19 bits avg: attention error / KL | 0.166 / 0.112 | 0.170 / 0.104 | 0.171 / 0.098 |
| Last layer V error at 3-bit rotated (all blocks ~3%) | 3.1% | 3.1% | 3.0% |
PROVISIONAL: keys used as stand-in queries can rank methods differently from real queries. A matched Q/K/V capture from the same run is next, and B, D and the 3.19-bit allocation will be rerun on it first.
Per-channel key scaling on its own (C) cuts K error a lot, but rotation (B) already removes most of the outlier-channel problem, and on top of rotation the scaling moves error into directions that hurt attention.
The 3.19-bit allocation (rotated, groups of 128) uses 2 bits for K in layers 2 and 7, 4 bits for V in layers 1, 3 and 4, and 3 bits elsewhere. It has about 2.2x the attention error and about 12x the KL of rotated q4_0.
Correction to E21u: the 19 to 41% last-layer V error there was normalized by per-channel centered variance, which is much harsher for blocks with large channel means. Under the standard relative error used here, layer 7's V is about 3% at 3-bit rotated, the same as the other blocks. E21u's outlier claim is withdrawn.
Group scales are counted in the bit totals (3.5 bits per value for 3-bit at group 32, 4.5 for 4-bit).
Prior art: KVQuant and KIVI (per-channel key quantization, calibrated scales), QServe SmoothAttention (scale then rotate), KVTuner (per-layer K/V bit widths), QuaRot (Hadamard rotation). Everything tested here is from that published work.
What happened
The comparison ran on CPU in about 8 minutes and a rerun matched byte for byte. On these numbers, no known 3-bit method gets near today's rotated 4-bit cache, which would mean reaching about 3.2 bits for 1M context needs something beyond the published methods. That conclusion waits on two things: the matched Q/K/V rerun, and the retrieval breakpoint test, which shows whether this gap actually costs the model right answers. A first slicing attempt had a script bug that duplicated layer 0 and zeroed V; it was caught by checking per-layer spreads, fixed, and re-sliced before these numbers.
How to rerun
python3 e21w.py
python3 analyze.pyCode
All code for E21W: README.md, analyze.py, e21w.py, results.json.
Prior work
No novelty claim. This compares published methods (KVQuant, KIVI, QServe SmoothAttention, KVTuner, QuaRot) on our model's cache.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.