Can 4-bit keys and 2-bit values hit a 1M-token memory budget? Simulated KV cache, 4B model, 64K

What I tested

Quantize then dequantize post-RoPE keys and values as they enter the cache, on the 8 full-attention layers only, absmax quantization with an fp16 scale per group. 'Rotated' means a 256-wide Hadamard rotation before quantizing. Fitting a 1M-token cache on a 10 GB card needs roughly 3.1 bits per value, so the arms use larger groups (64 or 128 values per scale) to cut scale overhead: A rotated 4-bit keys and 2-bit values, group 128 (3.125 bits); B the same with group 64 (3.25); C 4-bit keys with rotated 2-bit values, group 32 (3.5); D rotated 4-bit keys and 3-bit values, group 128 (3.625); plus a keys-only arm to isolate the key side. Bits per value are averaged over keys and values and include the scales.

Pass bar, set before the run

Set before the run: the 1M budget counts as reachable with these known methods if arm A has mean KL at or below plain 4-bit keys and values (group 32, 4.5 bits) in at least 5 of 6 spans. If A fails but B or C passes, report the bits needed. If all fail, the next round targets values below 3 bits.

Result

Arm A was worse than plain 4-bit in all 6 spans, about 3.8 times the KL, so the pre-set bar was not met. B and C also lost in all 6, so none of these known setups reaches the 1M budget.

The keys-only arm (rotated 4-bit keys, group 128, values left unquantized) had mean KL 0.0165, below plain 4-bit in 6 of 6. Larger key groups are cheap once keys are rotated, so the loss comes from the 2-bit values.

Even 3-bit rotated values with group-128 scales (arm D) lost in all 6 spans, by a small margin in the 72K and 108K spans and by more elsewhere.

The 2-bit grid here is the ggml-style signed absmax grid (levels -2, -1, 0, 1 times the scale), which leaves one side with fewer levels. The next run checks whether a symmetric grid or Gaussian Lloyd-Max levels on rotated values close the gap.

The plain 4-bit arm reproduces E22i, E22j and E22k exactly (0.0251).

This is a simulation in Hugging Face transformers, not llama.cpp, so it says nothing yet about speed or real memory use.

Per-span numbers for every arm, with p99 KL, top-1 agreement, perplexity and timing, are in results.csv in the code folder.

What happened

Seven arms over six spans at 64K took about 28 minutes. The goal was to see whether plain known tricks (rotation, uneven key and value bits, bigger quantization groups) get the cache down to the roughly 3.1 bits per value a 1M-token context needs on a 10 GB card. They don't: every 2-bit value arm roughly tripled or quadrupled the KL of plain 4-bit and dropped top-1 agreement from about 96% to 89% to 91%. Keys at 4 bits with large groups were fine once rotated, so the problem is narrowed to the values. The next run tests better 2-bit and 3-bit value grids before any new method is tried.

How to rerun

python e22l.py smoke
python e22l.py

Code

All code for E22L: e22l.py, results.csv.

Prior work

No novelty claim. This is a baseline; uneven key and value bit widths (for example KIVI and KVTuner), group-wise scales and Hadamard rotation before quantization are all known.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.