Stricter needle grading: Qwen3.5-9B passes up to 484K tokens and Qwen3.5-4B at 715K on one RTX 3080
I reran the long-context needle tests with a stricter grader that only counts the passcode if it is in the final answer, after any thinking, and with room for thinking to finish (1,024 tokens). All six runs passed. The 9B at its native 262K with the needle at 10% depth, which had failed twice before, passed after finishing its reasoning. The 9B passed at 448K (10% and 90% depth) and at 480K (50%), and the 4B passed at 704K at 10% and 90% depth. The 4B answered correctly but then kept copying the filler text until the token limit.
| Experiment | E21H |
| Date | 2026-10-29 |
| Benchmark | Needle in a haystack: one passcode sentence at 10%, 50% or 90% depth in repeated filler text, greedy decoding, 1,024-token answer limit. A run passes only if 7319-4826 appears in the final answer, after any thinking block. |
| Model | Qwen3.5-9B and Qwen3.5-4B, Q4_K_M GGUF, flash attention on, q4_0 K and V cache, ubatch 64 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
My build of upstream llama.cpp PR 29827, "CUDA: cap FA convert buffer for quantized KV" (not my work, unmerged when tested). 9B at native 262,144 context (needle at 10%), 448K with YaRN x1.75 (10% and 90%), 480K with YaRN x1.875 (50%), and 4B at 704K with YaRN x2.75 (10% and 90%). A run is skipped if the server uses more than 300 MiB of shared GPU memory right after loading.
Pass bar, set before the run
None. Supporting measurement. A pass means 7319-4826 was in the final answer.
Result
| Model | Context | Depth | Prompt tokens | Final answer | Peak VRAM | Shared | Prefill | Time |
|---|---|---|---|---|---|---|---|---|
| 9B | 262K (native) | 10% | 256,843 | 7319-4826 after a full thinking block (pass) | 7,946 MiB | 116 MiB | 1,193 tok/s | 3.9 min |
| 9B | 448K (YaRN x1.75) | 10% | 451,243 | 7319-4826 (pass) | 9,692 MiB | 140 MiB | 847 tok/s | 9.0 min |
| 9B | 448K (YaRN x1.75) | 90% | 451,243 | 7319-4826 (pass) | 9,692 MiB | 140 MiB | 847 tok/s | 9.0 min |
| 9B | 480K (YaRN x1.875) | 50% | 483,643 | 7319-4826 (pass) | 9,864 MiB | 264 MiB | 804 tok/s | 10.1 min |
| 4B | 704K (YaRN x2.75) | 10% | 715,243 | 7319-4826, then repeated filler (pass) | 9,784 MiB | 170 MiB | 656 tok/s | 19.2 min |
| 4B | 704K (YaRN x2.75) | 90% | 715,243 | 7319-4826, then repeated filler (pass) | 9,784 MiB | 170 MiB | 653 tok/s | 19.4 min |
The earlier 9B 262K failure at 10% depth was a limit problem, not a recall problem: with 96 tokens the model had not finished thinking. With 1,024 tokens it reasoned, closed the thinking block and answered correctly.
Not everything is in GPU memory. llama.cpp keeps the token embedding table in system memory as a lookup, plus a small host staging buffer; attention and the whole KV cache are on the GPU. With no model loaded the card shows 20 MiB of shared memory.
The 9B at 480K used 264 MiB of shared memory, more than the 140 MiB at 448K, and 9,864 MiB of the card. That is close to the limit for this model on this card.
The 4B at 704K printed the correct passcode first and then kept repeating the haystack sentences until the 1,024-token limit. The grader counts it as a pass because the passcode is the answer; in real use it would need a stop rule.
One needle in simple repeated filler is an easy haystack. This is not a RULER score. The next round uses real text and decoy passcodes.
What happened
With a grader that ignores the passcode when it only shows up inside the model's thinking, every run still passed. The 9B now holds a 484K-token prompt on a 10 GB card and finds the passcode, and the 4B does the same at 715K tokens at both ends of the prompt. Prefill slows as the context grows, from about 1,190 tokens per second at 262K to about 650 at 704K, so a full 704K prompt takes around 19 minutes. The memory saving that makes these fit is upstream PR 29827, so credit goes to its authors.
How to rerun
powershell -ExecutionPolicy Bypass -File e21h_chain.ps1Code
All code for E21H: README.md, e21h_chain.ps1, results.jsonl.
Prior work
No novelty claim. This measures an existing upstream llama.cpp pull request with a standard needle test.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.