Qwen3.5-9B finds a needle in a 451K-token prompt on one RTX 3080
With upstream llama.cpp PR 29827 and a q4_0 KV cache, Qwen3.5-9B found a single needle at 50% depth at 320K, 384K and 448K context (YaRN x1.25 to x1.75). The 448K run used 9,692 MiB peak VRAM and 140 MiB of shared memory. The rerun of the 9B at its native 262K with the needle at 10% depth still did not pass: twice the model began a thinking block that quoted the prompt and gave no final answer within 96 tokens.
| Experiment | E21G |
| Date | 2026-10-29 |
| Benchmark | Needle in a haystack: one passcode sentence at 10% or 50% depth in repeated filler text, greedy decoding, 96-token answer limit. A run passes only if 7319-4826 appears as the answer, not inside an unfinished thinking block. |
| Model | Qwen3.5-9B, Q4_K_M GGUF, flash attention on, q4_0 K and V cache, ubatch 64 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
My build of upstream llama.cpp PR 29827, "CUDA: cap FA convert buffer for quantized KV" (not my work, unmerged when tested). 9B at native 262,144 context with the needle at 10% depth (two runs), then 320K (YaRN x1.25), 384K (x1.5) and 448K (x1.75) at 50% depth. A run is skipped if the server uses more than 300 MiB of shared GPU memory right after loading.
Pass bar, set before the run
None. Supporting measurement. A pass means the answer was 7319-4826.
Result
| Context | Depth | Prompt tokens | Answer | Peak VRAM | Shared | Prefill | Time |
|---|---|---|---|---|---|---|---|
| 262K (native), run 1 | 10% | 256,843 | unfinished thinking block, no answer in 96 tokens (not passed) | 7,936 MiB | 116 MiB | 1,192 tok/s | 3.7 min |
| 262K (native), run 2 | 10% | 256,843 | same as run 1 (not passed) | 7,936 MiB | 116 MiB | 1,185 tok/s | 3.7 min |
| 320K (YaRN x1.25) | 50% | 321,643 | 7319-4826 (pass) | 8,532 MiB | 124 MiB | 1,046 tok/s | 5.4 min |
| 384K (YaRN x1.5) | 50% | 386,443 | 7319-4826 (pass) | 9,106 MiB | 132 MiB | 937 tok/s | 7.0 min |
| 448K (YaRN x1.75) | 50% | 451,243 | 7319-4826 (pass) | 9,692 MiB | 140 MiB | 849 tok/s | 9.0 min |
At 448K the q4_0 KV cache is 4,032 MiB and the 9B weights on the GPU are 4,861 MiB.
Not everything is in GPU memory. llama.cpp keeps the token embedding table (546 MiB for this model) in system memory as a lookup, plus a 60 MiB host staging buffer; attention and the whole KV cache are on the GPU. With no model loaded the card shows 20 MiB of shared memory, so the 116 to 140 MiB measured here is used by the server itself.
The script's automatic pass flag checked for the passcode anywhere in the reply, so it marked both 10% runs as passes even though the model only quoted the prompt while thinking. I graded them by hand. The next round requires the passcode in the final answer and allows 1,024 tokens so thinking can finish.
448K was the largest context tried in this round. The next round tries 480K.
One needle in simple repeated filler is an easy haystack. This is not a RULER score.
What happened
The 9B model held a 451K-token prompt on the card and still pulled out the passcode, using 9.7 of 10 GB. Each step from 320K to 448K added roughly 580 MiB and cost some prefill speed, from 1,046 down to 849 tokens per second. The rerun at native 262K with the needle near the start did not pass either time: the model started reasoning, restated the question, and hit the 96-token limit before answering. The memory saving that made these fit is upstream PR 29827, so credit goes to its authors.
How to rerun
powershell -ExecutionPolicy Bypass -File e21g_chain.ps1Code
All code for E21G: README.md, e21g_chain.ps1, results.jsonl.
Prior work
No novelty claim. This measures an existing upstream llama.cpp pull request with a standard needle test.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.