How far a 4B model's context fits on one RTX 3080: 704K tokens, and a 9B at 262K

With upstream llama.cpp PR 29827 and a q4_0 KV cache, Qwen3.5-4B found a single needle in a 715,243-token prompt (704K context, YaRN x2.75) at 9,827 MiB peak VRAM with 170 MiB shared. At 768K the model loaded already spilling 582 MiB into shared memory, so it was skipped. Qwen3.5-9B at its native 262K passed at 50% and 90% depth (7,936 MiB). At 10% depth it started a thinking block and gave no passcode within the 24-token limit, so that run did not pass.

Experiment	E21F
Date	2026-10-29
Benchmark	Needle in a haystack: one passcode sentence at 10%, 50% or 90% depth in repeated filler text, greedy decoding, 24-token answer limit, pass if the answer text contains 7319-4826
Model	Qwen3.5-4B and Qwen3.5-9B, Q4_K_M GGUF, flash attention on, q4_0 K and V cache, ubatch 64
Hardware	One of my dedicated test rigs: RTX 3080 10 GB
Verdict	BASELINE

What I tested

My build of upstream llama.cpp PR 29827, "CUDA: cap FA convert buffer for quantized KV" (not my work, unmerged when tested). 4B at 640K (YaRN x2.5), 704K (x2.75) and 768K (x3); 9B at its native 262,144 context. A run is skipped if the server uses more than 300 MiB of shared GPU memory right after loading.

Pass bar, set before the run

None. Supporting measurement. A pass on a row means the answer text contained 7319-4826.

Result

Model and context	Depth	Prompt tokens	Answer	Peak VRAM	Shared	Prefill	Time
4B, 640K (YaRN x2.5)	50%	648,043	7319-4826 (pass)	9,241 MiB	162 MiB	712 tok/s	15.3 min
4B, 704K (YaRN x2.75)	50%	715,243	7319-4826 (pass)	9,827 MiB	170 MiB	657 tok/s	18.3 min
4B, 768K (YaRN x3)	50%	not run	skipped, does not fit	9,930 MiB idle	582 MiB idle
9B, 262K (native)	50%	256,843	7319-4826 (pass)	7,936 MiB	116 MiB	1,184 tok/s	3.7 min
9B, 262K (native)	10%	256,843	began a thinking block, no passcode in 24 tokens (not passed)	7,936 MiB	116 MiB	1,183 tok/s	3.7 min
9B, 262K (native)	90%	256,843	7319-4826 (pass)	7,936 MiB	116 MiB	1,183 tok/s	3.7 min

704K is the largest 4B context that stayed under our 300 MiB shared-memory line with a stock q4_0 cache. At 768K the q4_0 KV cache alone is 6,912 MiB.

The script's automatic pass flag checked the whole server reply instead of the answer text, so it marked the 9B 10% run as a pass. I regraded every needle run from E21c to E21f on the answer text; this was the only one affected. The next round uses the fixed check and a 96-token limit to rerun it.

Shared GPU memory is under the 300 MiB line but not zero.

Correction: "fully resident" in this post's address overstates it. llama.cpp keeps the token embedding table and a small staging buffer in system memory; attention and the whole KV cache are on the GPU. With no model loaded the card shows 20 MiB of shared memory, so the 162 to 170 MiB here is the server's own.

One needle in simple repeated filler is an easy haystack. This is not a RULER score.

The 4B at 1M still needs a new KV compression method, which is the actual research goal.

What happened

The 4B model kept a 715K-token prompt on the card, with only 170 MiB in shared memory, and still found the needle, using 9.8 of 10 GB. One step further, 768K, spilled into system memory on load, so the stock q4_0 cache tops out around 704K for this model. The 9B found the needle at 50% and 90% depth at 262K, but at 10% it started reasoning out loud and ran out of answer tokens, so that one counts as not passed until the rerun. The memory saving that made these fit is upstream PR 29827, so credit goes to its authors.

How to rerun

powershell -ExecutionPolicy Bypass -File e21f_chain.ps1

Code

All code for E21F: README.md, e21f_chain.ps1, results.jsonl.

Prior work

No novelty claim. This measures an existing upstream llama.cpp pull request with a standard needle test.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.