Real-query attention is far less concentrated than the proxy suggested
I captured the real queries of Qwen3.5-4B and measured how many keys hold most of the attention. At 32K, 90% of the mass needs about 17% of the keys, roughly 280 times more than the earlier proxy measurement suggested. Negative result for compression ideas that keep only a few tokens precise.
| Experiment | E22 |
| Date | 2026-10-28 |
| Benchmark | Attention concentration with real captured queries: fraction of keys holding 90% and 99% of attention mass, top-k mass, self, first-4 and last-64 token mass, at 4K and 32K context |
| Model | Qwen3.5-4B, HF bf16 weights loaded as bitsandbytes NF4 (not Q4_K_M), compute bf16 |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, GPU memory capped at 6 GiB |
| Verdict | FAIL |
What I tested
WikiText-2 text was prefilled in chunks of 2048. At 64 query positions I captured the real post-RoPE queries for all 16 query heads on the 8 full-attention layers (3, 7, ..., 31) and computed attention over the full causal prefix. Numbers are means over 8,192 rows. An earlier proxy, which used key rows as stand-in queries at 131K, had put 90% of the mass on about 0.06% of keys.
Pass bar, set before the run
None. This was a check of the proxy figure, not a pass/fail test of a method.
Result
| Context | Keys for 90% of mass | Keys for 99% | Top-1 / top-8 / top-64 mass | Self / first-4 / last-64 mass |
|---|---|---|---|---|
| 4K | 12.6% (339 keys) | 40.0% | 0.182 / 0.430 / 0.704 | 0.104 / 0.028 / 0.361 |
| 32K | 17.1% (3,228 keys) | 47.8% | 0.088 / 0.21 / 0.40 | 0.041 / 0.0065 / 0.15 |
| 64K | none | none | FAILED, CUDA out of memory under the 6 GiB cap | not run |
Against the proxy: 90% of mass on ~0.06% of keys (proxy, 131K) vs ~17.1% (real queries, 32K), about 280x less concentrated.
Later in the context is weaker. At 32K, 90% of mass needs 13.2% of keys in the early third, 16.4% in the middle third and 21.4% in the late third.
By layer at 32K: layer 7 is the most concentrated (90% at 7.1%, top-64 0.60). Layers 11 (22.8%) and 23 (21.6%) are the most spread out. Layer 31 puts 0.115 of its mass on the current token.
Attention sinks are weak: the first 4 tokens hold under 1% at 32K. A local window of 64 tokens holds about 15%.
Nothing spilled: shared GPU memory stayed flat at about 0.5 to 0.6 GB, its idle level. Peak VRAM was 4,354 MiB at 4K and 5,248 MiB at 32K.
Caveats: NF4 weights, not the Q4_K_M GGUF used in E21. 32K is the longest context measured (64K ran out of memory, 128K was not attempted). Only the 8 full-attention layers. One corpus (WikiText-2) and 64 query positions.
What happened
Negative result. On this model, with real queries, attention is spread over a wide set of keys and gets more spread out deeper into the context. Compression or eviction schemes that keep only a small top-k set of tokens at high precision are not supported by these numbers. Any KV compression for long context here has to keep a wide spread of keys usable, especially late in the context.
How to rerun
python qcap.py 4096
python qcap.py 32768Code
All code for E22: README.md, attn_concentration_real_32k.json, attn_concentration_real_4k.json, qcap.py.
Prior work
No novelty claim. This is a measurement of an existing model's attention.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.