Which hurts more when quantized, keys or values? Simulated 4-bit KV cache, 4B model, 16K to 64K

Quantizing only the keys to 4 bits costs about twice as much quality as quantizing only the values, at every context length tested and in all 18 span and length pairs. Value error does not grow much with context (1.15x from 16K to 64K), so the growth seen in earlier llama.cpp runs is not explained by rounding the values themselves. Keys are the part to work on for a 3-bit cache.

Experiment	E22J
Date	2026-10-30
Benchmark	Next-token KL against the same model with an unquantized cache, plus top-1 agreement. Six WikiText-2 test spans (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K of the Hugging Face tokenization), each scored with 16,384, 32,768 and 65,536 tokens of context before it, chunked prefill.
Model	Qwen3.5-4B in Hugging Face transformers, NF4 weights (bitsandbytes), bf16 compute. Reference is the same model with an unquantized bf16 cache.
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM
Verdict	BASELINE

What I tested

Quantize then dequantize post-RoPE keys and/or values as they enter the cache, on the 8 full-attention layers only, with no flash-attention kernel involved. Arms: keys only, values only, and both, each in q4_0-style 4-bit (group 32, fp16 absmax scale, 4.5 bits per value); plus keys only and values only with the rotated 6-bit-dictionary 3-bit format from E22i (3.19 bits per value).

Pass bar, set before the run

Set before the run, two readings. (1) Value growth: if values-only 4-bit mean KL at 64K is at least 1.5x its 16K value and higher at 64K in at least 5 of 6 spans, value rounding explains the error growth seen in E22h; otherwise that growth is put down to the llama.cpp flash-attention path. (2) Whichever of keys-only or values-only has the larger KL at 64K in at least 5 of 6 spans is the target for the next 3-bit work.

Result

Arm	Bits per value	Mean KL 16K	Mean KL 32K	Mean KL 64K	Per-span KL at 64K (72K to 252K)
Keys only, 4-bit	4.5	0.0159	0.0147	0.0185	0.0243 / 0.0311 / 0.0250 / 0.0103 / 0.0050 / 0.0152
Values only, 4-bit	4.5	0.0080	0.0097	0.0092	0.0074 / 0.0176 / 0.0125 / 0.0043 / 0.0044 / 0.0088
Keys and values, 4-bit	4.5	0.0177	0.0197	0.0251	0.0302 / 0.0460 / 0.0272 / 0.0137 / 0.0074 / 0.0263
Keys only, 3-bit dictionary, rotated	3.19	0.0244	0.0302	0.0302	0.0283 / 0.0593 / 0.0355 / 0.0190 / 0.0097 / 0.0293
Values only, 3-bit dictionary, rotated	3.19	0.0199	0.0173	0.0185	0.0190 / 0.0287 / 0.0218 / 0.0123 / 0.0099 / 0.0196

Reading (1): values-only 4-bit goes from 0.0080 at 16K to 0.0092 at 64K (1.15x) and is higher at 64K in only 3 of 6 spans, so the bar is not met. Value rounding by itself does not grow much with context.

Reading (2): keys-only is worse than values-only in 6 of 6 spans at 64K, and in 18 of 18 span and length pairs at 4 bits. With the 3-bit dictionary format, keys are worse in 15 of 18.

The 3-bit dictionary format is worse than 4-bit for keys in 18 of 18 pairs and for values in 18 of 18 pairs, in line with E22i.

The keys-and-values 4-bit arm at 64K reproduces E22i's 4-bit number exactly (0.0251), so the harness is stable between runs.

Keys and values together grow 1.42x from 16K to 64K while each alone grows less (keys 1.16x, values 1.15x), so some of the growth comes from the two errors combining.

This is a simulation in Hugging Face transformers, not llama.cpp, so the numbers are not directly comparable to the llama.cpp KLD in E22c to E22h.

Per-span numbers for every arm and length, with p99 KL, top-1 agreement, perplexity and timing, are in results.csv in the code folder.

What happened

Six spans at three context lengths with six arms each took about 45 minutes. E22h left open whether the growing error of a 4-bit llama.cpp cache came from the flash-attention path or from rounding the values. With no flash-attention kernel in the loop, values-only error stays nearly flat as context grows, which points back at the kernel path for most of that growth. The clearer result is that keys are roughly twice as sensitive as values at the same bit width. For fitting long context into 10 GB, that suggests spending bits unevenly: more on keys and fewer on values. The next run tests exactly that at equal average bits.

How to rerun

python e22j.py smoke
python e22j.py

Code

All code for E22J: e22j.py, results.csv.

Prior work

No novelty claim. This is a supporting measurement; it is already known that keys are usually more sensitive to quantization than values.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.