Research log
Everything I've tried, in order
Short dated notes. Wins, flops, and stuff I'm still figuring out.
A tiny draft model made Q6_K about 35% faster
Speculative decoding with Qwen2.5-0.5B as the draft took Q6_K from 14.13 to 19.09 tok/s on my RTX 3080. Quality held. N-gram guessing did nothing, so it's dropped.
Read →Why is 32 users slower than 16?
Full-size F16 in llama.cpp gets 28.4 tok/s total at 16 users, 19.1 at 32, then 35.3 at 64. That dip shouldn't be there.
Read →The number to beat is 13.28 tokens a second
llama.cpp's Q6_K runs Qwen2.5-14B at 13.28 tok/s on my RTX 3080 and only loses 0.72% quality. That's the bar now.
Read →Perfect quality, 29× slower. Dropping SSX.
My exact-correction idea matched the full model's quality exactly. It was also way too slow, so it's gone.
Read →Ideas I didn't build because someone already did
Before building anything I search the papers. Several of my ideas were already out there, so I dropped them.
Read →