Some background first
If you already know how these models work, skip this. If not, here's what you need to follow the rest of the site.
What I'm actually running
A language model is what's behind ChatGPT. Under the hood it's a giant file of numbers. It guesses the next word, then the next one, until it's written an answer. The one I test is Qwen2.5-14B. It has about 14 billion of those numbers.
How I measure speed
In tokens per second. A token is about a word, sometimes part of one. At 1 tok/s it's like watching a slow typist. 5–10 tok/s is around how fast you read. Once you're past 13 tok/s it feels smooth.
How I measure quality
Mostly with perplexity. It's how surprised the model is by real text, and lower is better. When I say "+0.72% worse", the shrunk model scored 0.72% worse than the original. More than 1% worse and I call it a fail.
Why it's hard
At full size this model is roughly 28–30 GB (14 billion numbers × 2 bytes each). Most gaming GPUs have 8–12 GB. Mine has 10.
So it doesn't fit. You either shrink it and risk it getting dumber, or you keep moving pieces in and out of the GPU and it gets slow. That tradeoff is basically my whole project.
Why I care
Mostly money. Serving AI is expensive, and the cost per token decides whether a company running it is profitable. If a model runs well on cheaper hardware, that cost drops.
It's also nice for regular people. A normal PC with 32 GB of RAM and one graphics card is something a lot of people already own, and nothing you type has to leave it.