Where the decoys win: Qwen3.5-4B multikey needle from 128K to 512K, plus Qwen3.5-9B at 480K

Same decoy passcode test as the last round, run on the 4B at shorter contexts to find where it starts picking the wrong vault. It was right at 128K and 384K and in 2 of 3 runs at 256K, but wrong in both runs at 512K. The 9B answered correctly at 480,544 tokens, its longest multikey run so far.

Experiment	E21J
Date	2026-10-29
Benchmark	RULER-style multikey needle: one target passcode sentence at 10%, 50% or 90% depth plus three decoy passcodes for other vault colors (near 25%, 60% and 80% depth), in shuffled paragraphs from the MATH training set. Greedy decoding, 512-token answer limit. A run passes only if the first code in the final answer, after any thinking block, is the target code.
Model	Qwen3.5-9B and Qwen3.5-4B, Q4_K_M GGUF, flash attention on, q4_0 K and V cache, ubatch 64
Hardware	One of my dedicated test rigs: RTX 3080 10 GB
Verdict	BASELINE

What I tested

My build of upstream llama.cpp PR 29827, "CUDA: cap FA convert buffer for quantized KV" (not my work, unmerged when tested). 4B at 128K and 256K (native), 384K with YaRN x1.5 and 512K with YaRN x2. 9B at 480K with YaRN x1.875. Prompts were filled to the context size minus 1,500 tokens.

Pass bar, set before the run

None. Supporting measurement, plan written down before the run. A run passes if the target code is the answer; decoy picks are recorded.

Result

Model	Context	Depth	Prompt tokens	Answer	Peak VRAM	Shared	Prefill	Time
4B	128K (native)	50%	129,329	green code (pass)	4,514 MiB	98 MiB	1,969 tok/s	1.3 min
4B	256K (native)	10%	256,534	asked blue, gave the green decoy (fail)	5,686 MiB	114 MiB	1,361 tok/s	3.5 min
4B	256K (native)	50%	254,587	green code (pass)	5,686 MiB	114 MiB	1,370 tok/s	3.3 min
4B	256K (native)	90%	255,925	orange code (pass)	5,686 MiB	114 MiB	1,364 tok/s	3.5 min
4B	384K (YaRN x1.5)	50%	384,466	green code (pass)	6,858 MiB	130 MiB	1,048 tok/s	6.5 min
4B	512K (YaRN x2)	10%	513,414	asked blue, gave the red decoy (fail)	8,026 MiB	146 MiB	851 tok/s	10.5 min
4B	512K (YaRN x2)	90%	514,162	asked orange, gave the green decoy (fail)	8,026 MiB	146 MiB	850 tok/s	10.5 min
9B	480K (YaRN x1.875)	50%	480,544	green code (pass)	9,876 MiB	264 MiB	808 tok/s	10.3 min

Counting the last round too, the 4B is now 0 of 3 at 512K and 1 of 3 at 704K on this test, against 4 of 5 at 384K and below. The one failure below 512K was at 256K with the target near the start of the prompt.

All three 4B failures in this round happened when the target was at 10% or 90% depth; every 50% run passed. Seven runs are too few to call that a pattern.

The 9B at 480K used 9,876 MiB of the 10 GB card and 264 MiB of shared memory, the most so far and close to the 300 MiB line I use for counting a run as on the GPU. That is about as far as the 9B goes on this card with this setup.

After answering, both models kept writing extra question and answer pairs or copied math text from the haystack. The grader only reads the first code.

Next round: the same failing 4B configs with the question also stated before the long document (a known prompting technique), to see if that stops the decoy picks.

What happened

This round mapped where the 4B stops being able to pick the right fact out of a long prompt full of similar facts. Up to 384K it mostly could. At 512K it failed both times, each time giving a decoy's passcode. So the 4B fits 704K on the card but only uses roughly 384K reliably on this test. The 9B kept passing at 480K, near the limit of what fits on the card. The memory saving that makes these runs fit is upstream PR 29827, so credit goes to its authors.

How to rerun

powershell -ExecutionPolicy Bypass -File e21j_chain.ps1

Code

All code for E21J: README.md, e21j_chain.ps1, results.jsonl.

Prior work

No novelty claim. This measures an existing upstream llama.cpp pull request with a standard RULER-style multikey needle test.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.