More bits for keys, fewer for values, at the same total? Simulated KV cache, 4B model, 64K

What I tested

Quantize then dequantize post-RoPE keys and values as they enter the cache, on the 8 full-attention layers only. Absmax group-32 quantization with an fp16 scale per group; 'rotated' means a 256-wide Hadamard rotation before quantizing. Arms with uneven bits for keys and values: 5-bit keys plus rotated 3-bit values (with and without key rotation), the reversed split (rotated 3-bit keys plus 5-bit values) as a control, rotated 4-bit for both, plus two off-budget arms: rotated 4-bit keys with rotated 3-bit values (4.0 bits) and 6-bit keys with rotated 3-bit values (5.0 bits).

Pass bar, set before the run

Set before the run: PASS if 5-bit keys plus rotated 3-bit values (either key variant) has lower mean KL than plain 4-bit keys and values in at least 5 of 6 spans at the same 4.5 bits per value, AND the reversed control (rotated 3-bit keys, 5-bit values) is worse than that winning arm in at least 5 of 6 spans. Secondary: report whether rotated 4-bit keys plus rotated 3-bit values (4.0 bits) is at or below plain 4-bit in at least 5 of 6 spans.

Result

Main bar: rotated 5-bit keys plus rotated 3-bit values beat plain 4-bit in 5 of 6 spans (it lost only on the 216K span), and the reversed split was worse than it in 6 of 6. PASS as pre-registered.

The fair comparison the bar did not include is rotated 4-bit for both at the same 4.5 bits. It has the lowest mean KL of the equal-budget arms (0.0173) and the best top-1 agreement (96.34%). The uneven rotated split beat it in only 3 of 6 spans. So rotation is doing most of the work here, and spending extra bits on keys adds little on top of it.

The uneven splits have lower top-1 agreement than plain 4-bit (95.3% to 95.4% vs 95.86%) even though their mean KL is lower, because 3-bit values add small errors on many tokens.

Secondary reading: rotated 4-bit keys plus rotated 3-bit values (4.0 bits, 11% fewer than 4.5) was at or below plain 4-bit in only 3 of 6 spans, so it did not meet the secondary bar, though its mean KL (0.0236) is slightly lower.

The plain 4-bit arm reproduces E22i and E22j exactly (0.0251), so the harness is stable between runs.

This is a simulation in Hugging Face transformers, not llama.cpp, so it says nothing yet about speed or real memory use.

Per-span numbers for every arm, with p99 KL, top-1 agreement, perplexity and timing, are in results.csv in the code folder.

What happened

Seven arms over six spans at 64K took about 31 minutes. E22j showed keys are about twice as sensitive as values, so this run moved a bit from values to keys while keeping the same 4.5 bits per value. That split beat plain 4-bit and the reversed split clearly lost, which confirms the direction. The more useful finding is that the gain is mostly from the Hadamard rotation: rotating both keys and values at 4 bits did better than any uneven split at the same budget. For the 1M-token goal the budget has to fall to about 3.1 bits per value, so the next run tests rotated 4-bit keys with rotated 2-bit values and larger groups.

How to rerun

python e22k.py smoke
python e22k.py

Code

All code for E22K: e22k.py, results.csv.

Prior work

No novelty claim. This is a supporting measurement; uneven key and value bit widths (for example KIVI and mixed-precision KV cache papers) and Hadamard rotation before quantization are both known.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.