Can a 3-bit KV cache with dictionary scales match 4-bit? Simulated, 4B model, 64K

A 3-bit KV cache where each group of 32 values picks its scale from a small frozen dictionary uses about 3.2 bits per value, against 4.5 for q4_0-style 4-bit. Simulated on a real model at 64K context, it does not match 4-bit in any of six spans. Its KL is about 1.6x higher. Hadamard rotation is what makes 3-bit usable at all, and the dictionary scales help a little on top of it while using fewer bits.

Experiment	E22I
Date	2026-10-30
Benchmark	Next-token KL against the same model with an unquantized cache, plus top-1 agreement. Six WikiText-2 test spans (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K of the Hugging Face tokenization), each scored with 65,536 tokens of context before it, chunked prefill.
Model	Qwen3.5-4B in Hugging Face transformers, NF4 weights (bitsandbytes), bf16 compute. Reference is the same model with an unquantized bf16 cache.
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM
Verdict	FAIL

What I tested

Quantize then dequantize post-RoPE keys and values as they enter the cache, on the 8 full-attention layers only. Arms: q4_0-style 4-bit, group 32, fp16 absmax scale (4.5 bits per value); plain 3-bit group 32 (3.5 bits); plain 3-bit with a 256-wide Hadamard rotation; and 3-bit codes whose group scale is chosen by exhaustive search over a frozen 32-entry (5-bit, 3.16 bits per value) or 64-entry (6-bit, 3.19 bits per value) dictionary, with and without rotation. Dictionaries were fit per layer and per K or V on 8,192 tokens of separate calibration text, never on the scored spans.

Pass bar, set before the run

Set before the run: PASS if dictionary-scale 3-bit with the 6-bit dictionary has KL at or below the simulated 4-bit cache in at least 5 of 6 spans.

Result

Cache format	Bits per value	Mean KL	Per-span KL (72K to 252K)	Same top token
4-bit q4_0-style, group 32	4.5	0.0251	0.0302 / 0.0460 / 0.0272 / 0.0137 / 0.0074 / 0.0263	95.86%
3-bit, group 32	3.5	0.0612	0.0747 / 0.0869 / 0.0858 / 0.0437 / 0.0309 / 0.0453	91.86%
3-bit, group 32, rotated	3.5	0.0430	0.0470 / 0.0661 / 0.0564 / 0.0277 / 0.0176 / 0.0430	93.27%
3-bit, 5-bit dictionary scale, rotated	3.16	0.0398	0.0409 / 0.0630 / 0.0474 / 0.0270 / 0.0189 / 0.0418	93.55%
3-bit, 6-bit dictionary scale, rotated	3.19	0.0411	0.0371 / 0.0736 / 0.0466 / 0.0265 / 0.0187 / 0.0442	93.64%
3-bit, 5-bit dictionary scale, no rotation	3.16	0.0673	0.0656 / 0.0989 / 0.0907 / 0.0528 / 0.0360 / 0.0598	91.74%
3-bit, 6-bit dictionary scale, no rotation	3.19	0.0655	0.0606 / 0.0994 / 0.0869 / 0.0514 / 0.0342 / 0.0606	91.96%

The pre-set bar is not met: the 6-bit dictionary version is below 4-bit in 0 of 6 spans, with KL 1.23x to 2.53x the 4-bit value per span.

Rotation is the main lever. Without it, dictionary scales are no better than plain 3-bit.

With rotation, the 5-bit dictionary beats plain rotated 3-bit in 5 of 6 spans (mean 0.0398 vs 0.0430) while using about 0.34 fewer bits per value. The 6-bit dictionary beats it in only 3 of 6, so the larger dictionary did not help here.

This is a simulation in Hugging Face transformers, not a llama.cpp cache type, and the reference is the same NF4 model with an unquantized cache. Numbers are not directly comparable to the llama.cpp KLD in E22c to E22h.

A smoke test of the harness, chunked against one-shot prefill, gave KL 0.0013, the noise floor here.

Per-span numbers, top-1 agreement, p99 KL and timing are in the code folder (results.jsonl).

What happened

Six spans with eight arms each took about 52 minutes. The idea came from a CPU test that suggested a frozen scale dictionary could make 3-bit codes close to 4-bit quality at fewer bits. On a real model with 64K of context it closes only part of the gap: rotated 3-bit with dictionary scales sits at about 1.6x the KL of 4-bit. The useful result for the 1M-context goal is that rotation before quantizing is worth most of the gain at 3 bits, and a small 5-bit scale dictionary is a cheap way to drop the scale cost without losing quality. Neither is enough on its own to replace 4-bit. The next run splits the error between keys and values at 16K, 32K and 64K to see which one to work on.

How to rerun

python e22i.py smoke
python e22i.py

Code

All code for E22I: e22i.py, results.jsonl.

Prior work

No novelty claim. Hadamard rotation and searched or double-quantized group scales are known techniques; this is a quality check of one combination on a real model.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.