The number to beat is 13.28 tokens a second

llama.cpp's Q6_K runs Qwen2.5-14B at 13.28 tok/s on my RTX 3080 and only loses 0.72% quality. That's the bar now.

I ran a bunch of llama.cpp compression levels on Qwen2.5-14B, on one RTX 3080 with 10 GB of memory. I wanted the fastest setup that stays within my rule: no more than 1% worse than the full model.

Q6_K won. It stores each weight in about 6 bits and fits 40 of the model's 48 layers on the GPU.

Setup	Speed	Quality vs F16
Q6_K, 40/48 layers on GPU	13.28 tok/s	+0.72% perplexity · pass
Q8_0	5.17 tok/s	+0.64% · pass
F16, 14 layers on GPU	2.05 tok/s	reference
Q5_K	~33 tok/s	+3.7% · fail
Q4	~66 tok/s	+11.4% · fail

Q5 and Q4 are a lot faster. They also make the model noticeably worse, so they don't count.

For comparison, my own methods are way behind on a single user. Split-plane residency does 1.31 tok/s and layer streaming does 1.19. Both keep quality perfect, but that doesn't matter much if they're 10× slower.

I also tried n-gram speculative decoding on top of Q6_K. It got 13.72 tok/s, but the guesses were only accepted 0–3% of the time. That's basically no speedup.

So 13.28 at under 1% loss is the bar. Anything I build has to beat that, or win somewhere else, like many users on one GPU.