A 4B model finds a fact in a 504K-token prompt on one RTX 3080
With upstream llama.cpp PR 29827 and a q4_0 KV cache, Qwen3.5-4B found a single needle at 50% depth in prompts of 247K, 504K and 126K tokens on one 10 GB RTX 3080. Peak VRAM was 8,073 MiB at 512K, with 146 MiB still showing as shared GPU memory, so this is not yet zero spill. One needle, one depth, one run each.
| Experiment | E21D |
| Date | 2026-10-28 |
| Benchmark | Needle in a haystack: one passcode sentence at 50% depth in repeated filler text, greedy decoding, pass if the reply contains 7319-4826 |
| Model | Qwen3.5-4B, Q4_K_M GGUF, flash attention on, q4_0 K and V cache, ubatch 64 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
My build of upstream llama.cpp PR 29827, "CUDA: cap FA convert buffer for quantized KV" (not my work, unmerged when tested). Needle: "The secret passcode for the vault is 7319-4826." Three configs: 256K native context; 512K with YaRN x2 and the context length overridden to 524,288; and the same 512K server with a shorter 126K-token prompt.
Pass bar, set before the run
None. Baseline measurement. A pass on a row means the answer contained 7319-4826.
Result
| Config | Prompt tokens | Answer | Peak VRAM | Shared | Prefill | Time |
|---|---|---|---|---|---|---|
| 256K native | 247,243 | 7319-4826 (pass) | 5,735 MiB | 114 MiB | 1,403 tok/s | 3.1 min |
| 512K, YaRN x2 | 504,043 | 7319-4826 (pass) | 8,073 MiB | 146 MiB | 861 tok/s | 10 min |
| 512K, YaRN x2, shorter prompt | 126,043 | 7319-4826 (pass) | 8,055 MiB | 146 MiB | 1,955 tok/s | 1.1 min |
About 146 MiB still shows as shared GPU memory at 512K, so this does not meet the strict zero-spill bar yet. Part of that may be the normal Windows baseline; not separated yet.
One needle, one depth (50%), one run each. Depth sweeps and repeat runs are next.
The filler is simple repeated sentences, which is an easy haystack. This is not a RULER score.
1M is still not solved for the 4B: the q4_0 KV cache alone is about 9.2 GB at 1M. That needs a new KV compression method, which is the actual research goal.
What happened
The 4B model really did retrieve a fact from about 504K tokens on one 10 GB card, with nearly everything resident on the GPU. The memory saving that made it fit is upstream PR 29827, so credit goes to its authors. This is a supporting measurement for the 1M work, not a new method.
How to rerun
powershell -ExecutionPolicy Bypass -File e21d_chain.ps1Code
All code for E21D: README.md, e21d_chain.ps1, results.jsonl.
Prior work
No novelty claim. This measures an existing upstream llama.cpp pull request with a standard needle test.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.