One llama.cpp flag more than doubled my 32-user speed

Update: raising GGML_OP_OFFLOAD_MIN_BATCH to 128 works even better, no downsides found yet. Originally: turning off op offload took Qwen2.5-14B Q6_K from 60.2 to 135.0 tok/s at 32 users on my RTX 3080. Single user didn't change. It's a setting, not a new method.

Remember the weird dip at 32 users? I think I found it. Round E05.

At batch 32, llama.cpp turns on something called op offload. For layers that live on the CPU, it copies their weights over to the GPU to do the math there. With only part of the model on the GPU, all that copying costs more than it saves.

So I turned it off with --no-op-offload. Qwen2.5-14B Q6_K, 40 of 48 layers on the GPU, RTX 3080:

Test	Default	No op offload
32 users, generation	60.2 tok/s	135.0 tok/s
64 users, generation	95.5 tok/s	148.5 tok/s
1 user	14.6 tok/s	14.4 tok/s
Prompt reading	~294 tok/s	335 tok/s
Total throughput, 32 users	165 tok/s	258 tok/s
Energy per token, 32 users	~1.72 J	~1.28 J

That's 2.24x on 32-user generation and 56% more total throughput, with one user basically the same. The energy numbers are rough, from GPU plus CPU package power.

Why I care: same GPU, more users served, less energy per token. That's what makes serving cheaper.

To be clear, this isn't a new method. It's an existing llama.cpp knob that happens to be set wrong for this kind of setup.

It's not a free win everywhere either. On full-size F16 with only 14 layers on the GPU, turning it off removes the dip (19.4 to 39.2 tok/s), but prompt reading drops to about 43 tok/s. So the real fix is probably a better threshold, not just off. Next round, E06, sweeps GGML_OP_OFFLOAD_MIN_BATCH to find it.

Update: a better fix (round E06)

Turning op offload off fixed Q6_K but hurt F16. Turns out there's a better option, and it's just an existing environment variable. No code change.

llama.cpp has GGML_OP_OFFLOAD_MIN_BATCH, which sets the batch size where op offload kicks in. I swept it, and 128 is the sweet spot.

Q6_K, 40/48 layers on GPU	Default	MIN_BATCH=128
32 users	57.3 tok/s	135.2 tok/s
64 users	n/a	150.7 tok/s
Energy per token, 32 users	~1.59 J	~1.23 J
Electricity per million tokens	~$0.066	~$0.051

That's 2.36x at 32 users. Single-user speed and prompt speed didn't change (about 186 tok/s prompt alone, about 290 batched). Energy is approximate, and the dollar figure assumes $0.15/kWh.

The bigger deal is F16, where turning offload off had wrecked prompt speed. With 14 layers on the GPU at 32 users it goes from 19.1 to 39.4 tok/s, and single-user prompt stays around 75 tok/s.

Why not higher or lower? At 256 or 512, single-user prompt on F16 drops to about 41. At 64, the dip just comes back at 64 users. So 128.

Same as before, this is an existing knob, not something I invented. I'm just finding where it should be set for a gaming GPU with part of the model in system RAM.

Next up, E07: checking long prompts and 96 users to make sure 128 holds up.