Asking first fixes the decoys: Qwen3.5-4B multikey needle 5/5 at up to 709,596 tokens with the question before the document
The 4B kept giving a decoy's passcode at 512K and 704K when the question came only after the long document. Stating the question before the document too, a known prompting technique, made all five previously failing configs pass, including a 709,596-token prompt on a 10 GB RTX 3080.
| Experiment | E21K |
| Date | 2026-10-29 |
| Benchmark | RULER-style multikey needle: one target passcode sentence at 10%, 50% or 90% depth plus three decoy passcodes for other vault colors (near 25%, 60% and 80% depth), in shuffled paragraphs from the MATH training set. Greedy decoding, 512-token answer limit. A run passes only if the first code in the final answer, after any thinking block, is the target code. |
| Model | Qwen3.5-4B, Q4_K_M GGUF, flash attention on, q4_0 K and V cache, ubatch 64 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
The E21j test with one prompt change: the question is also stated once before the document ("You will be asked: ... keep that question in mind"), as well as after it. Same haystack seeds as the earlier runs, so each result pairs with an earlier question-last run on the same text. 4B at 256K (native), 512K with YaRN x2 and 704K with YaRN x2.75, on llama.cpp built with upstream PR 29827. Prompts filled to the context size minus 1,500 tokens.
Pass bar, set before the run
None. Supporting measurement, plan written down before the run. A run passes if the target code is the answer.
Result
| Context | Depth | Prompt tokens | Question last (E21i/E21j) | Question first (this run) | Peak VRAM | Shared | Prefill | Time |
|---|---|---|---|---|---|---|---|---|
| 256K (native) | 10% | 256,529 | green decoy (fail) | blue code (pass) | 5,688 MiB | 114 MiB | 1,359 tok/s | 3.5 min |
| 512K (YaRN x2) | 10% | 513,364 | red decoy (fail) | blue code (pass) | 8,028 MiB | 146 MiB | 850 tok/s | 10.5 min |
| 512K (YaRN x2) | 50% | 514,477 | orange decoy (fail) | red code (pass) | 8,026 MiB | 146 MiB | 850 tok/s | 10.5 min |
| 512K (YaRN x2) | 90% | 514,116 | green decoy (fail) | orange code (pass) | 8,026 MiB | 146 MiB | 851 tok/s | 10.5 min |
| 704K (YaRN x2.75) | 10% | 709,596 | red decoy (fail) | blue code (pass) | 9,784 MiB | 170 MiB | 662 tok/s | 18.4 min |
Five of five passed with the question first; the same five configs and haystack seeds had all failed with the question only at the end. One run per config, so this is a strong signal on a small sample, not a measured rate.
Stating the question before the document is a known technique for long-context prompts. The point here is how large the effect was for this small model at 512K and 704K.
After answering, the 4B still repeated its answer or copied math text from the haystack until the 512-token limit. The grader only reads the first code.
The embedding table stays in system RAM in this setup, so the runs are not fully on the GPU; shared memory stayed at 170 MiB or less.
Next round: a cache precision check (q8_0 vs q4_0 K and V cache at 256K, question last) to see whether the 4-bit cache contributes to the decoy picks.
What happened
In the last two rounds the 4B gave another vault's passcode in every run at 512K and in two of three at 704K. Putting the question in front of the long document as well as after it fixed every one of those cases in this round, up to a 709,596-token prompt that used 9,784 MiB of the 10 GB card. A likely reason is that the model reads the document already knowing which vault matters, instead of having to look back across half a million tokens at the end. The memory saving that makes these context sizes fit is upstream llama.cpp PR 29827, so credit goes to its authors.
How to rerun
powershell -ExecutionPolicy Bypass -File e21k_chain.ps1Code
All code for E21K: README.md, e21k_chain.ps1, results.jsonl.
Prior work
No novelty claim. Question-before-document is a known prompting technique; this measures it on an existing upstream llama.cpp pull request with a standard RULER-style multikey needle test.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.