Open hypotheses

How I pick what to test

I don't test random tweaks. Each idea starts as an educated guess based on how these models actually work. Then I search papers, patents and forums. If someone's already done it, I drop it. I only call something "never tested" when the search came up empty. Otherwise it's "not found in prior work yet."

What the guesses are based on

Memory speed is usually what holds it back. To write each token, the model has to read all of its weights from memory. Usually the GPU spends more time waiting on that than doing the actual math.
The GPU is too small for the model. A 10 GB card can't hold a model that's roughly 28–30 GB at full size, so something has to give.
More users share each read. When you serve lots of people at once, every weight you read gets used for all of them. That's where cost per token can really drop.
Conversations get heavier. The model keeps notes on everything said so far (the KV cache), and that keeps growing as the conversation gets longer. It eats memory too.

Why is 32 users slower than 16?

TESTING

The idea: Full-size F16 in llama.cpp did 28.4 tok/s total at 16 users, dropped to 19.1 at 32, then climbed to 35.3 at 64. My guess: somewhere around 32 users the batch crosses a size threshold that changes where the work runs (GPU vs CPU), and that makes 32 slower than 16.
Why it should cut cost: If that's what's happening and it can be fixed, the GPU stays busy at every user count. More users per GPU means lower cost per token.
Prior art: Not found in prior work yet.
Status: Testing now. Result goes here when I have it.