KV cache precision check: q8_0 and q4_0 gave identical multikey needle answers for Qwen3.5-4B at 256K, and f16 does not fit

To see whether the 4-bit KV cache causes the 4B's decoy picks, I reran the question-last multikey needle test at 256K with f16, q8_0 and q4_0 caches on the same text. f16 did not fit on the 10 GB card. q8_0 and q4_0 gave the same answer in all three runs (2 of 3 correct, same decoy on the miss), while q4_0 used 2,048 MiB less VRAM.

Experiment	E21L
Date	2026-10-29
Benchmark	RULER-style multikey needle: one target passcode sentence at 10%, 50% or 90% depth plus three decoy passcodes for other vault colors (near 25%, 60% and 80% depth), in shuffled paragraphs from the MATH training set. Question only after the document. Greedy decoding, 512-token answer limit. A run passes only if the first code in the answer is the target code.
Model	Qwen3.5-4B, Q4_K_M GGUF, flash attention on, K and V cache at f16, q8_0 or q4_0, ubatch 64
Hardware	One of my dedicated test rigs: RTX 3080 10 GB
Verdict	BASELINE

What I tested

The same 256K question-last prompts and haystack seeds as E21i/E21j, run three times with only the K and V cache type changed (f16, q8_0, q4_0). A config counted as not fitting if the loaded server already used more than 300 MiB of shared (system) memory before any prompt. Also tried 512K with YaRN x2 and a q8_0 cache. llama.cpp built with upstream PR 29827.

Pass bar, set before the run

None. Supporting control measurement, plan written down before the run.

Result

Cache	Depth	Prompt tokens	Answer	Peak VRAM	Shared	Prefill	Time
f16	10%, 50%, 90%	n/a	did not fit (1,738 MiB shared at load)	n/a	1,738 MiB	n/a	n/a
q8_0	10%	256,534	green decoy (fail)	7,734 MiB	114 MiB	1,343 tok/s	3.5 min
q8_0	50%	255,660	red code (pass)	7,734 MiB	114 MiB	1,340 tok/s	3.5 min
q8_0	90%	255,925	orange code (pass)	7,734 MiB	114 MiB	1,338 tok/s	3.5 min
q4_0	10%	256,534	green decoy (fail)	5,686 MiB	114 MiB	1,361 tok/s	3.5 min
q4_0	50%	255,660	red code (pass)	5,686 MiB	114 MiB	1,365 tok/s	3.5 min
q4_0	90%	255,925	orange code (pass)	5,686 MiB	114 MiB	1,363 tok/s	3.5 min
q8_0 at 512K (YaRN x2)	10%	n/a	did not fit (2,366 MiB shared at load)	n/a	2,366 MiB	n/a	n/a

Three paired runs per cache type is a small sample. It shows no difference on these prompts, not that there is none. A 10-trial paired version with fresh passcodes, colors and depths per trial (E21m) started right after this one.

The 10% miss happened with both caches and picked the same decoy, so on this prompt the miss does not come from 4-bit cache rounding. E21k already showed that asking the question before the document fixes it.

q4_0 saved 2,048 MiB of VRAM at 256K with no change in answers here, which is why the longer runs use it.

The embedding table stays in system RAM in this setup, so even the runs that fit are not fully on the GPU; shared memory stayed at 114 MiB.

What happened

The 4B sometimes gives a decoy vault's passcode when the question comes only after a very long document. One possible cause was the 4-bit KV cache losing detail. Running the exact same three 256K prompts with an 8-bit cache gave the exact same three answers as the 4-bit cache, including the same wrong decoy at 10% depth, while the 4-bit cache used about 2 GB less memory. A 16-bit cache does not fit on the 10 GB card at 256K, and neither does an 8-bit cache at 512K. The memory saving that makes these context sizes fit is upstream llama.cpp PR 29827, so credit goes to its authors.

How to rerun

powershell -ExecutionPolicy Bypass -File e21l_chain.ps1

Code

All code for E21L: e21l_chain.ps1, results.jsonl.

Prior work

No novelty claim. This is a control measurement of standard llama.cpp KV cache types on an existing upstream pull request with a standard RULER-style multikey needle test.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.