Ten paired trials at 512K: asking the question before the document took Qwen3.5-4B from 2 of 10 to 9 of 10 on a multikey needle test
I tested whether stating the question before a long document, as well as after it, helps a small model pick the right passcode out of four at about 515K tokens. On 10 fresh, unselected trials with identical documents per pair, question-first got 9 of 10 and question-last got 2 of 10. All seven disagreements favored question-first (McNemar exact p = 0.016), so the pre-registered bar was met. This is a known prompting technique, measured here on my own setup.
| Experiment | E21N |
| Date | 2026-10-29 |
| Benchmark | RULER-style multikey needle: one target passcode sentence plus three decoy passcodes for other vault colors, in shuffled paragraphs from the MATH training set, filled to about 515K tokens. Greedy decoding, 64-token answer limit. A run passes only if the first code in the answer is the target code. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, YaRN context extension x2 to 524,288 tokens, flash attention on, K and V cache at q4_0, ubatch 64 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | PASS |
What I tested
10 trials with seeds 2200 to 2209, random passcodes, random target color and target depths from 10% to 90%. Each trial's document was built once and run twice on identical text: question-last (question only after the document) and question-first (the same question also stated before the document). Run order alternated between trials. llama.cpp built with upstream PR 29827.
Pass bar, set before the run
Set before the run: question-first wins on the discordant pairs with McNemar exact p < 0.1.
Result
| Trial | Depth | Prompt tokens | Question-first | Question-last | Pair |
|---|---|---|---|---|---|
| 1 | 10% | 516,673 | blue code (pass) | black decoy (fail) | question-first only |
| 2 | 90% | 516,170 | green code (pass) | green code (pass) | both pass |
| 3 | 30% | 514,262 | blue code (pass) | no match (fail) | question-first only |
| 4 | 70% | 517,332 | silver code (pass) | purple decoy (fail) | question-first only |
| 5 | 50% | 517,993 | red code (pass) | orange decoy (fail) | question-first only |
| 6 | 15% | 514,432 | green code (pass) | red decoy (fail) | question-first only |
| 7 | 85% | 514,561 | black code (pass) | black code (pass) | both pass |
| 8 | 40% | 514,829 | silver decoy (fail) | black decoy (fail) | both fail |
| 9 | 60% | 513,078 | black code (pass) | orange decoy (fail) | question-first only |
| 10 | 20% | 515,028 | yellow code (pass) | purple decoy (fail) | question-first only |
| Total | 9 of 10 pass | 2 of 10 pass | 7 question-first only, 0 question-last only |
McNemar exact test on the 7 to 0 discordant pairs gives two-sided p = 0.016, which meets the bar.
Question-last only passed when the target was near the end of the document (85% and 90% depth). Everywhere else it answered with a decoy code, or with another code that was not the target.
The one question-first miss was trial 8 at 40% depth, where it gave the silver decoy. Question-last missed that trial too.
Peak VRAM was 8,016 to 8,072 MiB. Shared (system) memory stayed at 146 MiB in every run, the same as at idle, because the embedding table stays in system RAM in this setup, so the runs are not 100% on the GPU. Prefill ran at about 850 tok/s and each run took about 10.2 minutes.
Ten trials on one synthetic task. It says nothing yet about real documents or other models.
What happened
E21k showed question-first fixing five configs that had failed with question-last, but those configs were picked because they failed, which can exaggerate an effect. So I ran ten new trials that were not selected in any way, with the same document used for both prompt styles in each pair. Question-first won 9 to 2, and every one of the seven disagreements went its way. For this 4B model, much of the gap between the context it can hold (over 700K tokens on this card) and the context it can actually use with decoys comes down to where the question sits. The memory saving that makes these context sizes fit is upstream llama.cpp PR 29827, so credit goes to its authors. Next, E21o tests question-first at 704K on ten more fresh trials.
How to rerun
powershell -ExecutionPolicy Bypass -File e21n_chain.ps1Code
All code for E21N: e21n_chain.ps1, results.jsonl.
Prior work
No novelty claim. Stating the question before a long context is a known prompting technique; this is a paired measurement of it on a standard RULER-style multikey needle test.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.