# E21: VRAM budget for the KV cache, Qwen3.5-4B Q4_K_M at 512K (dedicated test rig, RTX 3080 10GB)
Measured 2026-10-28 ~16:55 ET. llama.cpp llama-server, -ngl 99 (33/33 layers on GPU), -fa on, -np 1, YaRN x2 (orig 262144), default ubatch 512.
Raw logs: C:\AGI\agi-research\e21_budget\*.log on dedicated test rig.

## Original 4B-512K sweep point: FAILED-OVERFLOW
Launched 15:32 ET with q4_0 K and V. Stalled for 77+ minutes, server log 0 bytes, card at 9881/10240 MiB used (173 MiB free), spilling into shared memory. Stopped at ~16:50 ET along with the sweep driver. The 9B configs never ran. The point is logged as FAILED-OVERFLOW in results/quality/e21_longctx.jsonl and bench/E21.log and is not a recall result.

## Card
- Total 10240 MiB. With the GPU idle, the desktop/OS used 553 MiB (nvidia-smi showed 9501 MiB free).
- **llama.cpp sees 9073 MiB free on CUDA0.** This is the real usable ceiling and is the figure used below.

## Architecture (from the GGUF)
32 blocks, hybrid. full_attention_interval = 4, so **only 8 layers carry a KV cache**. The other 24 are Gated-DeltaNet recurrent layers with a fixed-size state.
n_kv_heads = 4, head_dim = 256 for both K and V.
KV elements at 524,288 tokens = 8 x 2 x 4 x 256 x 524,288 = **8,589,934,592 values** (f16 = 16,384 MiB; q4_0 = 4,608 MiB).

## Measured allocations (MiB, llama.cpp memory breakdown, CUDA0)
| item | MiB |
|---|---|
| (a) weights (model buffer) | **2603** |
| recurrent state (R+S, constant in context length) | 50.25 |
| output buffer | ~1 (host) |
| (b) compute/activations at 512K, KV kept off the GPU (--no-kv-offload) | **2612** |
| compute at 512K with q4_0 KV on the GPU | 2620 |
| weights + compute + recurrent state (KV excluded) | **~5266-5274** |

Per-process dedicated/shared memory from Windows counters:
- weights only, c=512: 2999 MiB dedicated, 100 MiB shared
- 512K with KV off the GPU: 5474 dedicated, 618 shared
- 512K with q4_0 KV on the GPU: 9382 dedicated, **1370 shared (spill)**

The baseline "shared" figure of about 100 MiB even at c=512 is WDDM-mapped host buffers, not overflow.

## (c) KV budget at 512K
9073 - 2603 - 2620 - 50 = **3800 MiB = 3,984,588,800 bytes**
-> **3.71 bits/value**, all scales and metadata included, over 8.59e9 KV values.
If we leave a 300 MiB safety margin for WDDM/fragmentation, that becomes 3500 MiB, or **3.42 bits/value**.

q4_0 (4.5 bits/value, 4608 MiB) is about 808 MiB over budget, which matches the measured spill.
Fully resident 512K is possible: weights plus activations with an empty KV come to about 5.27 GB of the 9.07 GB free. The KV cache is what doesn't fit.

## Notes
- The compute buffer (~2.6 GB) grows with n_ctx at ubatch 512. A smaller ubatch should shrink it and widen the KV budget. A run with -ub 128 was started but had not logged its buffer sizes within 25 s, so it is **unmeasured**.
- 1M on the 4B: the KV element count doubles (17.18e9 values) and compute probably grows further, so the budget will be much tighter. Not measured here.