Needle recall at three depths: a 2B model at about 1M tokens and a 4B at 512K on one RTX 3080
With upstream llama.cpp PR 29827 and a q4_0 KV cache, Qwen3.5-2B found a single needle in a 996,043-token prompt at 10%, 50% and 90% depth using 5,547 MiB of VRAM, and Qwen3.5-4B found it in a 504,043-token prompt at 10%, 90% and on a repeat at 50%, using 8,069 MiB. All six runs passed. Shared GPU memory stayed at 210 MiB (2B) and 146 MiB (4B), under the 300 MiB spill line we set, but not zero.
| Experiment | E21E |
| Date | 2026-10-28 |
| Benchmark | Needle in a haystack: one passcode sentence at 10%, 50% or 90% depth in repeated filler text, greedy decoding, pass if the reply contains 7319-4826 |
| Model | Qwen3.5-2B and Qwen3.5-4B, Q4_K_M GGUF, flash attention on, q4_0 K and V cache, ubatch 64 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB |
| Verdict | BASELINE |
What I tested
My build of upstream llama.cpp PR 29827, "CUDA: cap FA convert buffer for quantized KV" (not my work, unmerged when tested). 2B at 1,048,576 context with YaRN x4; 4B at 524,288 context with YaRN x2. One prompt per run, no prompt cache.
Pass bar, set before the run
None. Baseline measurement. A pass on a row means the answer contained 7319-4826.
Result
| Model and context | Depth | Prompt tokens | Answer | Peak VRAM | Shared | Prefill | Time |
|---|---|---|---|---|---|---|---|
| 2B, 1M (YaRN x4) | 50% | 996,043 | 7319-4826 (pass) | 5,547 MiB | 210 MiB | 1,244 tok/s | 13.5 min |
| 2B, 1M (YaRN x4) | 10% | 996,043 | 7319-4826 (pass) | 5,547 MiB | 210 MiB | 1,240 tok/s | 13.5 min |
| 2B, 1M (YaRN x4) | 90% | 996,043 | 7319-4826 (pass) | 5,547 MiB | 210 MiB | 1,240 tok/s | 13.5 min |
| 4B, 512K (YaRN x2) | 10% | 504,043 | 7319-4826 (pass) | 8,069 MiB | 146 MiB | 860 tok/s | 10 min |
| 4B, 512K (YaRN x2) | 90% | 504,043 | 7319-4826 (pass) | 8,069 MiB | 146 MiB | 861 tok/s | 10 min |
| 4B, 512K (YaRN x2), repeat | 50% | 504,043 | 7319-4826 (pass) | 8,069 MiB | 146 MiB | 861 tok/s | 10 min |
Shared GPU memory is under the 300 MiB line we set before the runs, but it is not zero. Part of it may be the normal Windows baseline; that has not been separated out yet.
One needle per prompt and simple repeated filler, which is an easy haystack. This is not a RULER score.
The 4B at 1M is still not solved: its q4_0 KV cache alone is about 9.2 GB at 1M. That needs a new KV compression method, which is the actual research goal.
With the 4B 50% run from E21d, the 4B has now passed at all three depths at 512K, plus one repeat.
What happened
Both models retrieved the needle every time, at the start, middle and end of the prompt. The 2B used about 5.5 GB for a 996K-token prompt, which leaves room on the card. The memory saving that made these fit is upstream PR 29827, so credit goes to its authors. This is a supporting measurement for the 1M work, not a new method.
How to rerun
powershell -ExecutionPolicy Bypass -File e21e_chain.ps1Code
All code for E21E: README.md, e21e_chain.ps1, results.jsonl.
Prior work
No novelty claim. This measures an existing upstream llama.cpp pull request with a standard needle test.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.