Question-last control at 704K: on the same ten documents, the 4B found the right passcode in 3 of 10, against 9 of 10 with the question first

E21o got 9 of 10 multikey needle trials right at about 710K tokens with the question stated before and after the document. Here I reran the exact same ten documents with the question only at the end. It passed 3 of 10. Question-first won all 6 pairs where the two prompts disagreed (McNemar exact p = 0.031), meeting the bar set before the run. Every question-last miss answered with a decoy code. This is a known prompting technique, measured here on my own setup.

Experiment	E21P
Date	2026-10-29
Benchmark	RULER-style multikey needle: one target passcode sentence plus three decoy passcodes for other vault colors, in shuffled paragraphs from the MATH training set, about 706K to 713K tokens. Greedy decoding, 64-token answer limit. A run passes only if the first code in the answer is the target code.
Model	Qwen3.5-4B, Q4_K_M GGUF, YaRN context extension x2.75 to 720,896 tokens, flash attention on, K and V cache at q4_0, ubatch 64
Hardware	One of my dedicated test rigs: RTX 3080 10 GB
Verdict	PASS

What I tested

The same 10 saved documents, targets and depths as E21o (seeds 2300 to 2309, depths 10% to 90%), with the question stated only after the document. Compared pair by pair with the E21o question-first results. llama.cpp built with upstream PR 29827.

Pass bar, set before the run

Set before the run: question-first wins on the pairs where the two prompts disagree, McNemar exact test p below 0.1.

Result

Trial	Depth	Prompt tokens	Question-first (E21o)	Question-last (E21p)	Peak VRAM	Time
1	10%	711,332	pass	blue decoy (fail)	9,868 MiB	18.3 min
2	90%	707,385	pass	orange code (pass)	9,856 MiB	18.1 min
3	30%	712,936	pass	silver decoy (fail)	9,854 MiB	18.3 min
4	70%	712,989	pass	yellow code (pass)	9,839 MiB	18.3 min
5	50%	705,887	pass	green decoy (fail)	9,839 MiB	17.9 min
6	15%	706,711	pass	black decoy (fail)	9,839 MiB	17.9 min
7	85%	708,226	pass	orange code (pass)	9,868 MiB	18.1 min
8	40%	711,424	fail (black decoy)	red decoy (fail)	9,836 MiB	18.1 min
9	60%	706,848	pass	purple decoy (fail)	9,836 MiB	17.9 min
10	20%	708,082	pass	silver decoy (fail)	9,846 MiB	18.1 min
Total			9 of 10	3 of 10	9,836 to 9,868 MiB

Pairs: 6 won by question-first only, 0 by question-last only, 3 both pass, 1 both fail (trial 8, 40% depth). McNemar exact two-sided p = 0.031.

Question-last only passed when the target sat late in the document (70%, 85%, 90% depth). Every miss returned one of the decoy codes.

The same pattern showed up at 512K in E21n (7 to 0, p = 0.016), so it holds across both context sizes I tested.

Shared (system) memory read 170 MiB in every run, the same as E21o. A separate measurement (E21q) is checking whether that is spill or driver overhead.

Ten pairs is a small sample, and both prompts are greedy single runs.

What happened

E21o showed question-first working at about 710K tokens, but without a same-document comparison it could not say how much of that came from the prompt. This round fed the model the identical ten documents with the question only at the end. Accuracy dropped from 9 of 10 to 3 of 10, and the model only got it right when the target passcode was near the end of the document. So at this context size, where the question sits matters far more than anything else I have measured so far, including the 4-bit cache (E21m). The memory saving that makes this context fit on a 10 GB card is upstream llama.cpp PR 29827, so credit goes to its authors.

How to rerun

powershell -ExecutionPolicy Bypass -File e21p_chain.ps1

Code

All code for E21P: e21p_chain.ps1, results.jsonl.

Prior work

No novelty claim. Stating the question before a long context is a known prompting technique; this is a paired control measurement of it on a standard RULER-style multikey needle test.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.