E21: long-context needle recall and GPU fit on one RTX 3080

I ran Qwen3.5 2B, 4B and 9B at 128K to 1M context fully on a 10 GB RTX 3080 and checked whether they could find three hidden facts. Nothing past about 256K fits on the GPU without spilling into system memory, and recall was mediocre even where it fit. Fail.

Experiment	E21
Date	2026-10-28
Benchmark	3-needle needle-in-a-haystack recall (literal, code and paraphrase needles at 10%, 50% and 90% depth) at 128K, 256K, 512K and 1M tokens, plus measured VRAM and shared-memory use
Model	Qwen3.5-2B, 4B and 9B, Q4_K_M GGUF (llama.cpp llama-server), all layers on GPU, flash attention, KV cache q4_0 (9B: K q8_0, V q4_0), YaRN above 262K
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM
Verdict	FAIL

What I tested

Each model was loaded at each context length with every layer on the GPU. A run only counts if it stays fully resident on the GPU with zero spill into Windows shared system memory. For runs that fit, three needles were hidden in filler text and the model was asked for each one. Afterwards I measured how much VRAM the 4B actually has left for the KV cache at 512K and 1M.

Pass bar, set before the run

Fully resident on the GPU (zero shared-memory spill) with usable recall at 512K and 1M.

Result

Model	Context	Recall	VRAM / status
2B	128K	0.444	fits
2B	256K	0.333	fits
2B	512K	0.333	fits
2B	1M	0.222	INVALID, spilled 1120 MB to shared memory
4B	128K	0.667	5.3 GB
4B	256K	0.667	7.1 GB
4B	512K	none	FAILED, overflowed and spilled
9B	128K	0.667	8.1 GB
9B	262K	none	SKIPPED, 9893 of 10240 MiB used at load

Literal and code needles were found. Paraphrase needles were missed at every size and length.

4B at 512K: about 9073 MiB is usable. Weights take 2603 MiB, activations about 2620 MiB and fixed state 50 MiB, which leaves 3800 MiB for the KV cache. Only 8 of 32 layers hold a cache, so that is 8.59 billion values and a budget of 3.71 bits per value (3.42 with a 300 MiB margin). q4_0 needs 4.5 bits, 4608 MiB, about 808 MiB too much.

A smaller ubatch barely helps: at -ub 128 activations drop only to 2191 MiB, so the budget rises to 4.13 bits per value. Plain q4_0 KV does not fit at 512K at any ubatch.

At 1M the activation buffer alone is 5.5 to 6.3 GB, leaving 87 to 900 MiB for 17.2 billion KV values (0.04 to 0.44 bits per value). At 1M the activations are the main problem, more than the KV bits.

What happened

Stock llama.cpp cannot hold these models fully on a 10 GB card past about 256K. The two limits are clear and measured: at 512K the KV cache has to get under about 3.4 to 3.7 bits per value, and at 1M the activation buffer has to shrink before KV compression even matters. That is what the compression work has to beat. Recall was also weak where things fit, since no model found the paraphrased needle.

How to rerun

# Sweep (2B/4B, 9B uses the same launch with -ctk q8_0):
python e21_longctx.py
# Each point launches, for example:
llama-server -m Qwen3.5-4B-Q4_K_M.gguf -ngl 99 -fa on -c 524288 -np 1 -ctk q4_0 -ctv q4_0 -t 8 --no-mmap --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144
# Budget runs: add -v and -ub 512, 256 or 128; for 1M add --no-kv-offload to measure activations alone

Code

All code for E21: README.md, e21_4b_512k_vram_budget.md, e21_4b_ub_and_1m_budget.md, e21_longctx.py.

Prior work

No novelty claim. This is a capacity and recall measurement of existing models and llama.cpp.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.