Decoy passcodes: Qwen3.5-9B stays right at 454K tokens, Qwen3.5-4B mostly picks a decoy
I made the needle test harder. The haystack is now real math text instead of a repeated filler sentence, and besides the target passcode there are three decoy passcodes for other vault colors. The question asks for one color. The 9B answered correctly in all 4 runs, up to 454,278 tokens. The 4B was right in 1 of 4: at 704K and 512K it gave a decoy's passcode 3 times, even though it passed a single-needle test at 704K before.
| Experiment | E21I |
| Date | 2026-10-29 |
| Benchmark | RULER-style multikey needle: one target passcode sentence at 10%, 50% or 90% depth plus three decoy passcodes for other vault colors (near 25%, 60% and 80% depth), in shuffled paragraphs from the MATH training set. Greedy decoding, 512-token answer limit. A run passes only if the first code in the final answer, after any thinking block, is the target code. |
| Model | Qwen3.5-9B and Qwen3.5-4B, Q4_K_M GGUF, flash attention on, q4_0 K and V cache, ubatch 64 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
My build of upstream llama.cpp PR 29827, "CUDA: cap FA convert buffer for quantized KV" (not my work, unmerged when tested). 9B at native 262,144 context (target at 50%) and 448K with YaRN x1.75 (10%, 50%, 90%). 4B at 704K with YaRN x2.75 (10%, 50%, 90%) and 512K with YaRN x2 (50%). Prompts were filled to the context size minus 1,500 tokens.
Pass bar, set before the run
None. Supporting measurement, plan written down before the run. A run passes if the target code is the answer; decoy picks are recorded.
Result
| Model | Context | Depth | Prompt tokens | Answer | Peak VRAM | Shared | Prefill | Time |
|---|---|---|---|---|---|---|---|---|
| 9B | 262K (native) | 50% | 255,660 | red code (pass) | 7,946 MiB | 116 MiB | 1,196 tok/s | 3.9 min |
| 9B | 448K (YaRN x1.75) | 10% | 447,206 | blue code (pass) | 9,702 MiB | 140 MiB | 850 tok/s | 9.2 min |
| 9B | 448K (YaRN x1.75) | 50% | 454,278 | green code (pass) | 9,702 MiB | 140 MiB | 845 tok/s | 9.4 min |
| 9B | 448K (YaRN x1.75) | 90% | 449,169 | orange code (pass) | 9,704 MiB | 140 MiB | 852 tok/s | 9.2 min |
| 4B | 704K (YaRN x2.75) | 10% | 709,568 | asked blue, gave the red decoy (fail) | 9,784 MiB | 170 MiB | 661 tok/s | 18.4 min |
| 4B | 704K (YaRN x2.75) | 50% | 709,792 | asked green, gave the orange decoy (fail) | 9,786 MiB | 170 MiB | 661 tok/s | 18.4 min |
| 4B | 704K (YaRN x2.75) | 90% | 709,515 | orange code (pass) | 9,784 MiB | 170 MiB | 661 tok/s | 18.4 min |
| 4B | 512K (YaRN x2) | 50% | 514,510 | asked red, gave the orange decoy (fail) | 8,028 MiB | 146 MiB | 848 tok/s | 10.5 min |
Fitting a long context in memory is not the same as using it. The 4B passed a single passcode at 704K in the previous round, but with three decoys around it, it gave the wrong vault's code in 3 of 4 runs.
In two of the three 4B failures it gave the orange decoy, which sits near 80% depth, the latest decoy in the prompt. Four runs are too few to call that a pattern.
After answering, the 9B kept writing extra question and answer pairs. It listed the decoy codes correctly and then made up codes for yellow and purple vaults that are not in the prompt. The 4B drifted into copying math text from the haystack.
The 704K prompts reuse the training paragraphs about 2 to 3 times (in a different order each pass), so the haystack has some repetition.
Shared GPU memory stayed between 116 and 170 MiB, under the 300 MiB line I use for counting a run as on the GPU. llama.cpp keeps the token embedding table and a small staging buffer in system memory.
Next round: the same test on the 4B at 128K to 512K to find where it starts to pick decoys, plus the 9B at 480K.
What happened
This round asked whether the models can pick the right fact out of a long prompt when similar facts compete with it. The 9B could: it found the asked-for passcode every time at 448K tokens, at the start, middle and end of the prompt. The 4B could not reliably: at 704K and 512K it mostly returned a passcode for a different vault. So for the 4B, the context that fits on the card is much longer than the context it can actually use. The memory saving that makes these runs fit is upstream PR 29827, so credit goes to its authors.
How to rerun
powershell -ExecutionPolicy Bypass -File e21i_chain.ps1Code
All code for E21I: README.md, e21i_chain.ps1, results.jsonl.
Prior work
No novelty claim. This measures an existing upstream llama.cpp pull request with a standard RULER-style multikey needle test.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.