Where is the headroom for 2-bit values? Attention-weighted bits, output-aligned bits, an error ledger and a gain fix, 4B model, 64K

What I tested

Same harness as E22l to E22n: post-RoPE keys and values are quantized then dequantized as they enter the cache, 8 full-attention layers only, keys rotated 4-bit with group 128 in every value arm. Four ideas for where 2-bit values lose quality. (1) Spend extra bits on the tokens that get the most attention: 1a picks the top 6% or 18.5% of tokens by attention received over the whole run, including future queries (meant as an upper bound); 1b is causal, keeping the last 512 tokens at 4 bits and, as tokens leave that window, keeping those with the highest attention so far at 4 bits and dropping the rest to 2 bits. (2) Give 3 bits to the 16 or 80 directions of each head that matter most to the output projection (W_o), against the same split on a Hadamard basis as a control. (3) Carry each token's rounding error forward to later tokens whose keys fall in the same group (by key sign pattern or k-means), against random groups, after first checking whether tokens with similar keys are actually attended together. (4) A per-layer gain on 2-bit values, set by variance matching or by a small grid search on held-out calibration text. Bits per value include scales and flags.

Pass bar, set before the run

Set before the run: an idea stays in the brief if it closes at least 50% of the gap between rotated 2-bit (G128) and rotated 4-bit (G128) values at about 2.25 bits per value in most spans. Attention-weighted bits are dropped if even the future-attention upper bound (1a) closes under 25% at 2.25 bits. The error ledger only goes ahead if tokens in the same key group are attended together clearly more than random groups (a margin of at least 0.1 in some layer). Matching plain 4-bit keys and values (group 32) is the overall goal.

Result

No arm at about 2 to 2.5 bits per value reached plain 4-bit keys and values in any span, so the overall goal was not met.

Idea 1 is a contradiction I am reporting as is. The causal version (1b) passed the 50% bar in all six spans, but the future-attention version (1a), meant as the upper bound, closed only 19% at 2.25 bits, which under the pre-set rule drops attention-weighted bits. The difference is that 1b also keeps the most recent 512 tokens at 4 bits, which is where the scored tokens look most. The likely explanation is that the recent window, not attention ranking, does the work. The next check is a recent-window-only arm at the same bits.

The error ledger's premise failed: tokens with similar keys were not attended together much more than random groups (margins 0.044 to 0.095 for k-means groups and 0.015 to 0.037 for sign groups, against a pre-set 0.1, in all 8 layers and 6 spans). The ledger itself was worse than no ledger in 6 of 6 spans and no better than random groups, so the follow-up ledger run was skipped.

For idea 2, measuring the value error before and after the output projection showed the projection mostly made the relative error larger, not smaller, in all but one layer.

Gap closed is measured per span against rotated 2-bit (0%) and rotated 4-bit (100%) at group 128, then averaged. The 2.06-bit arms use one scale per 256 values, so they start slightly behind the 2.125-bit reference.

Peak GPU memory was 6.98 GiB. This is a simulation in Hugging Face transformers, not llama.cpp, so it says nothing about speed or real memory use.

Per-span numbers for every arm are in results.tsv in the code folder.

What happened

The run took about 2 hours 50 minutes on the dedicated test rig after two out-of-memory crashes on the arm that keeps a full-precision copy of the values for comparison; moving that copy to pinned CPU memory fixed it. The goal was to find where 2-bit values leave room for improvement before inventing anything. Three of the four ideas failed outright, including the error ledger, whose premise did not hold on real cache data. The one arm that did well mostly points at a known idea, keeping recent tokens at higher precision, so the next step is to measure that window on its own.

How to rerun

python e22o.py smoke
python e22o.py

Code

All code for E22O: __pycache__, e22o.py, results.tsv.

Prior work

No novelty claim. This was a headroom measurement before any invention. Mixed precision by token importance (for example MiKV and ZipCache) and keeping recent tokens at higher precision (KIVI) are known; the W_o-aligned bit split, the key-group error ledger and the per-layer gain were screened here and failed, so no prior-art search was run for them.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.