Ten paired trials: q8_0 and q4_0 KV caches gave the same multikey needle answer 9 times out of 10 for Qwen3.5-4B at 256K
I repeated the KV cache precision check with 10 fresh paired trials at 256K, each with new passcodes, vault colors and needle depth. The 8-bit cache got 7 of 10 and the 4-bit cache got 6 of 10. Nine pairs gave the identical answer, and the one disagreement is far from significant (McNemar exact p = 1.0). The 4-bit cache used about 2 GB less VRAM.
| Experiment | E21M |
| Date | 2026-10-29 |
| Benchmark | RULER-style multikey needle: one target passcode sentence plus three decoy passcodes for other vault colors, in shuffled paragraphs from the MATH training set, filled to about 256K tokens. Question only after the document. Greedy decoding, 64-token answer limit. A run passes only if the first code in the answer is the target code. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, flash attention on, K and V cache at q8_0 or q4_0, ubatch 64 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
10 trials, each with its own seed (2100 to 2109), random passcodes, random target color and a target depth from 10% to 90%. Each trial's prompt was built once and run with both cache types on the identical text, with the run order alternating between trials. f16 was left out because it does not fit at 256K on this card (E21l). llama.cpp built with upstream PR 29827.
Pass bar, set before the run
None. Supporting control measurement, plan written down before the run: report per-pair outcomes and McNemar exact test on the discordant pairs.
Result
| Trial | Depth | Prompt tokens | q8_0 answer | q4_0 answer | Pair |
|---|---|---|---|---|---|
| 1 | 10% | 259,526 | yellow code (pass) | yellow code (pass) | same |
| 2 | 90% | 256,769 | orange code (pass) | orange code (pass) | same |
| 3 | 30% | 257,213 | black decoy (fail) | black decoy (fail) | same |
| 4 | 70% | 253,235 | red code (pass) | red code (pass) | same |
| 5 | 50% | 257,475 | purple code (pass) | purple code (pass) | same |
| 6 | 15% | 257,592 | purple decoy (fail) | purple decoy (fail) | same |
| 7 | 85% | 256,598 | black code (pass) | black code (pass) | same |
| 8 | 40% | 255,707 | yellow code (pass) | yellow code (pass) | same |
| 9 | 60% | 257,823 | silver code (pass) | yellow decoy (fail) | different |
| 10 | 20% | 256,495 | red decoy (fail) | red decoy (fail) | same |
| Total | 7 of 10 pass | 6 of 10 pass | 9 same, 1 different |
Peak VRAM was 7,724 to 7,734 MiB with q8_0 and 5,676 to 5,686 MiB with q4_0. Shared (system) memory stayed at 114 MiB in every run. Prefill ran at about 1,340 tok/s (q8_0) and 1,365 tok/s (q4_0); each run took about 3.3 minutes.
The only disagreement was trial 9 at 60% depth, where q8_0 found the silver code and q4_0 gave the yellow decoy. With one discordant pair, McNemar's exact test gives p = 1.0, so there is no measurable difference at this sample size.
The three misses shared by both caches picked the same wrong decoy code each time, so those misses do not come from 4-bit rounding. E21k showed that putting the question before the document fixed this kind of miss on the configs it tried.
Ten trials can only catch a large effect. A small accuracy cost from the 4-bit cache is still possible.
The embedding table stays in system RAM in this setup, so the runs are not 100% on the GPU; shared memory stayed at 114 MiB.
What happened
The earlier three-prompt check (E21l) found no difference between 8-bit and 4-bit KV caches but was too small to mean much, so I ran ten matched pairs with fresh passcodes, colors and depths. The two caches agreed on nine of ten pairs, including the same wrong decoy on all three shared misses. The 8-bit cache won the one remaining pair. That is no evidence that the 4-bit cache hurts retrieval here, and it saves about 2 GB of VRAM, which is what lets longer contexts fit on the 10 GB card. The memory saving that makes these context sizes fit is upstream llama.cpp PR 29827, so credit goes to its authors. Next, E21n runs the same kind of paired test for question-first versus question-last prompts at 512K on unselected trials.
How to rerun
powershell -ExecutionPolicy Bypass -File e21m_chain.ps1Code
All code for E21M: e21m_chain.ps1, results.jsonl.
Prior work
No novelty claim. This is a control measurement of standard llama.cpp KV cache types on an existing upstream pull request with a standard RULER-style multikey needle test.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.