Can known tricks rescue 2-bit values? Zero points, no rotation and kept outliers, 4B model, 64K

No. Keeping the two largest values per group at full precision (3.30 bits per value) and a rotated zero-point grid at group 32 (3.56) each closed about a third of the gap between the plain 2-bit value cache and plain 4-bit, but every 2-bit arm stayed far above plain 4-bit keys and values (0.0251) in all six spans. Zero points at group 128 and skipping the rotation made things worse.

Experiment	E22N
Date	2026-10-30
Benchmark	Next-token KL against the same model with an unquantized cache, plus top-1 agreement. Six WikiText-2 test spans (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K of the Hugging Face tokenization), each scored with 65,536 tokens of context before it, chunked prefill.
Model	Qwen3.5-4B in Hugging Face transformers, NF4 weights (bitsandbytes), bf16 compute. Reference is the same model with an unquantized bf16 cache.
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM
Verdict	FAIL

What I tested

Same setup as E22l and E22m: quantize then dequantize post-RoPE keys and values as they enter the cache, 8 full-attention layers only, keys rotated 4-bit with group 128 in every value arm. Three known levers for 2-bit values: (a) an asymmetric min/max grid with an fp16 scale and fp16 zero point per group (KIVI-style), (b) skipping the Hadamard rotation, and (c) keeping the top 2 values by magnitude in each group exactly in fp16 (dense-and-sparse outliers, as in KVQuant and SqueezeLLM). Bits per value are averaged over keys and values and include scales, zero points and outlier indices.

Pass bar, set before the run

Set before the run: the roughly 3.3-bit 1M budget counts as reachable with known methods if any arm at or below 3.3 bits per value (signed rotated G128, zero-point rotated G128, rotated G128 with outliers, unrotated G128 with outliers) has mean KL at or below plain 4-bit keys and values (group 32, 4.5 bits) in at least 5 of 6 spans.

Result

Arm	Bits per value	Mean KL	Top-1 agreement	At or below plain 4-bit	Gap to plain 4-bit closed	Per-span KL (72K to 252K)
Plain 4-bit keys and values (group 32)	4.5	0.0251	95.86%	-	-	0.0302 / 0.0460 / 0.0272 / 0.0137 / 0.0074 / 0.0263
Keys only: rotated 4-bit, group 128	-	0.0165	96.44%	6 of 6	-	0.0207 / 0.0281 / 0.0204 / 0.0084 / 0.0052 / 0.0164
2-bit values, signed grid, rotated, group 128 (E22l arm A repeat)	3.125	0.0959	88.88%	0 of 6	0%	0.0956 / 0.1233 / 0.1119 / 0.0830 / 0.0689 / 0.0928
2-bit values, zero-point grid, rotated, group 128	3.19	0.1096	87.69%	0 of 6	-19%	0.1088 / 0.1615 / 0.1259 / 0.0891 / 0.0772 / 0.0949
2-bit values, zero-point grid, unrotated, group 32 (KIVI-style)	3.56	0.1015	88.44%	0 of 6	-8%	0.1023 / 0.1442 / 0.1111 / 0.0843 / 0.0731 / 0.0939
2-bit values, zero-point grid, rotated, group 32	3.56	0.0725	90.50%	0 of 6	33%	0.0791 / 0.0977 / 0.0844 / 0.0647 / 0.0457 / 0.0635
2-bit values, signed grid, rotated, group 128, top-2 outliers kept	3.30	0.0733	90.66%	0 of 6	32%	0.0765 / 0.1039 / 0.0880 / 0.0558 / 0.0450 / 0.0706
2-bit values, signed grid, unrotated, group 128, top-2 outliers kept	3.30	0.1042	88.26%	0 of 6	-11%	0.1064 / 0.1462 / 0.1188 / 0.0893 / 0.0621 / 0.1025
2-bit values, zero-point grid, unrotated, group 128, top-2 outliers kept	3.37	0.1277	85.52%	0 of 6	-45%	0.1168 / 0.1459 / 0.1487 / 0.1294 / 0.0930 / 0.1321
3-bit values, zero-point grid, rotated, group 128	3.69	0.0320	94.08%	1 of 6	90%	0.0411 / 0.0427 / 0.0394 / 0.0200 / 0.0157 / 0.0333

No 2-bit arm reached plain 4-bit in any span, so the pre-set bar was not met. Gap closed is measured per span against the plain signed 2-bit arm (0%) and plain 4-bit (100%), then averaged.

The two levers that helped, kept outliers and a zero point with small groups, each recovered about a third of the gap, but both cost extra bits and still left KL about three times plain 4-bit.

Rotation helped every time here: each unrotated arm was worse than its rotated version.

3-bit rotated zero-point values (3.69 bits) got within 90% of the gap but beat plain 4-bit in only 1 of 6 spans, about the same as the plain signed 3-bit grid in E22l (0.0324).

Not tested: a full-precision window of recent tokens (KIVI keeps the last 128), because this harness quantizes at cache insert.

This is a simulation in Hugging Face transformers, not llama.cpp, so it says nothing yet about speed or real memory use.

Per-span numbers for every arm, with p99 KL, top-1 agreement, perplexity and timing, are in results.csv in the code folder.

What happened

Eleven arms over six spans at 64K took about 44 minutes. After E22l and E22m showed that 2-bit values are what breaks the 1M memory budget, this run tried the standard published fixes for low-bit values. None was enough. The next run measures how much room is left at all: giving more bits only to the tokens that get the most attention, aligning value bits with the output projection, an error-feedback encoder, and a simple per-layer gain correction.

How to rerun

python e22n.py smoke
python e22n.py

Code

All code for E22N: e22n.py, results.csv.

Prior work

No novelty claim. This is a baseline; asymmetric zero-point low-bit value caches (KIVI), dense-and-sparse outlier retention (KVQuant, SqueezeLLM) and Hadamard rotation before quantization (QuaRot-style) are all known.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.