Do better 2-bit grids fix the value cache? Symmetric and Lloyd-Max levels, 4B model, 64K

No. Swapping the 2-bit value grid for a symmetric grid or Gaussian Lloyd-Max levels did not help: Lloyd-Max at 3.125 bits per value had mean KL 0.0960, the same as the plain grid (0.0959), and the symmetric grid was worse (0.1329). Plain 4-bit keys and values stayed at 0.0251. At 3 bits, both new grids were also slightly worse than the plain grid from the last run.

Experiment	E22M
Date	2026-10-30
Benchmark	Next-token KL against the same model with an unquantized cache, plus top-1 agreement. Six WikiText-2 test spans (the 4,096 tokens ending at token 72K, 108K, 144K, 180K, 216K and 252K of the Hugging Face tokenization), each scored with 65,536 tokens of context before it, chunked prefill.
Model	Qwen3.5-4B in Hugging Face transformers, NF4 weights (bitsandbytes), bf16 compute. Reference is the same model with an unquantized bf16 cache.
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, 128 GB system RAM
Verdict	FAIL

What I tested

Same setup as E22l: quantize then dequantize post-RoPE keys and values as they enter the cache, 8 full-attention layers only, fp16 scale per group, keys rotated 4-bit with group 128 in every value arm. The last run used the ggml-style signed 2-bit grid (levels -2, -1, 0, 1 times the scale), which leaves one side short a level. This run replaces it on rotated values with (s) a symmetric mid-rise grid (levels at plus or minus 0.5 and 1.5 steps, no wasted level) and (n) Gaussian Lloyd-Max levels scaled by each group's RMS, since rotated values are close to Gaussian. On Gaussian test data Lloyd-Max cut 2-bit squared error from 0.303 (symmetric) to 0.116. Arms at 2 bits use group 128 (3.125 bits per value averaged over keys and values) and one Lloyd-Max arm uses group 32 (3.5). Two 3-bit arms (3.625) test the same grids one bit up.

Pass bar, set before the run

Set before the run: the roughly 3.1-bit 1M budget counts as reachable with known grids if the symmetric or Lloyd-Max 2-bit arm (group 128) has mean KL at or below plain 4-bit keys and values (group 32, 4.5 bits) in at least 5 of 6 spans.

Result

Arm	Bits per value	Mean KL	Top-1 agreement	At or below plain 4-bit	Per-span KL (72K to 252K)
Plain 4-bit keys and values (group 32)	4.5	0.0251	95.86%	-	0.0302 / 0.0460 / 0.0272 / 0.0137 / 0.0074 / 0.0263
Keys only: rotated 4-bit, group 128	-	0.0165	96.44%	6 of 6	0.0207 / 0.0281 / 0.0204 / 0.0084 / 0.0052 / 0.0164
2-bit values, signed grid, group 128 (E22l arm A repeat)	3.125	0.0959	88.88%	0 of 6	0.0956 / 0.1233 / 0.1119 / 0.0830 / 0.0689 / 0.0928
2-bit values, symmetric grid, group 128	3.125	0.1329	86.26%	0 of 6	0.1245 / 0.1776 / 0.1543 / 0.1122 / 0.0988 / 0.1302
2-bit values, Lloyd-Max levels, group 128	3.125	0.0960	89.28%	0 of 6	0.1003 / 0.1232 / 0.1156 / 0.0903 / 0.0600 / 0.0864
2-bit values, Lloyd-Max levels, group 32	3.5	0.0910	89.71%	0 of 6	0.0894 / 0.1264 / 0.1070 / 0.0803 / 0.0564 / 0.0863
3-bit values, symmetric grid, group 128	3.625	0.0353	93.62%	0 of 6	0.0372 / 0.0575 / 0.0407 / 0.0254 / 0.0195 / 0.0316
3-bit values, Lloyd-Max levels, group 128	3.625	0.0361	93.83%	0 of 6	0.0400 / 0.0503 / 0.0418 / 0.0276 / 0.0209 / 0.0362

Neither new 2-bit grid came close to plain 4-bit in any span, so the pre-set bar was not met. Lloyd-Max at group 128 closed none of the gap between the plain 2-bit grid and plain 4-bit; at group 32 it closed about 7% for 0.375 more bits per value. The symmetric grid made things worse.

Lower squared error on Gaussian data did not turn into lower model KL. Both grids that worsened things have no exact zero level, which may matter for values; that is a guess, not something this run tested.

At 3 bits both new grids (0.0353 and 0.0361) were slightly worse than the plain signed 3-bit grid from E22l (0.0324), so the grid shape is not where the 2-bit loss comes from.

The plain 4-bit, keys-only and repeated arm A numbers match E22l exactly.

This is a simulation in Hugging Face transformers, not llama.cpp, so it says nothing yet about speed or real memory use.

Per-span numbers for every arm, with p99 KL, top-1 agreement, perplexity and timing, are in results.csv in the code folder.

What happened

Nine arms over six spans at 64K took about 37 minutes. The question was whether the 2-bit value failure in E22l came from a wasteful grid rather than from 2 bits itself. It did not: the textbook better grids either matched the plain grid or did worse, and top-1 agreement stayed around 86% to 90% against about 96% for plain 4-bit. The next run tries other known levers for 2-bit values: a zero-point (asymmetric) grid, skipping the rotation, and keeping the two largest values in each group at full precision.

How to rerun

python e22m.py smoke
python e22m.py

Code

All code for E22M: e22m.py, results.csv.

Prior work

No novelty claim. This is a baseline; Lloyd-Max and NF-style non-uniform levels, Hadamard rotation before quantization (QuaRot-style) and low-bit value caches (KIVI) are all known.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.