Why is 32 users slower than 16?

Full-size F16 in llama.cpp gets 28.4 tok/s total at 16 users, 19.1 at 32, then 35.3 at 64. That dip shouldn't be there.

Serving lots of people on one GPU is where cost per token can really drop. Every weight you read from memory gets used for everybody at once. So total speed should go up as you add users.

Mostly it does. Here's full-size F16 in llama.cpp on my RTX 3080:

Users	Total speed
16	28.4 tok/s
32	19.1 tok/s
64	35.3 tok/s

32 is slower than 16. Then 64 is faster than both. Weird.

My guess: around 32 users the batch crosses some size threshold that changes where the work runs, GPU vs CPU. If that's right, 32 is getting stuck on the slow path.

Why care? If the dip can be fixed, the GPU stays busy at every user count. More users per GPU means lower cost per token.

I haven't found this in prior work yet. Testing it now. I'll post what I find, even if the answer is boring.