Tests

Everything I've tried, ranked

This page is every test, good and bad. For just the takeaways you can use, see Findings.

Same model (Qwen2.5-14B), same machine, best at the top. Bars are speed for one user. Longer is faster. The badge tells you if quality stayed within 1%.

RTX 3080 · 10 GBRyzen 9 5950X128 GB RAM (test rig)Target: consumer hardware, 32 GB RAM + 1 GPU
PASS within 1% FAIL worse than 1% DROPPED too slow My method vs Existing tool
Key takeaway

Honestly, right now an existing tool is winning. llama.cpp's Q6_K runs this model at 13.28 tokens (roughly words) per second on one RTX 3080 and only loses 0.72% quality. My own methods keep quality perfect. They're also about 10× slower for a single user. That stings. Where they might still win is lots of people sharing one GPU, so that's what I'm testing next.

  1. 1

    Raising the op offload threshold to 128 (32 users)

    Existing llama.cpp setting, measured on my test rig
    2.36x
    Speed (32 users)
    Single user
    Single user and prompt speed unchanged

    At batch 32 llama.cpp copies CPU-held weights to the GPU. With 40 of 48 layers on the GPU that costs more than it saves. Round E06: setting GGML_OP_OFFLOAD_MIN_BATCH=128, no code change. 64 users hit 150.7 tok/s. Energy per token at 32 users about 1.59 J to 1.23 J, roughly $0.066 to $0.051 of electricity per million tokens at $0.15/kWh. F16 at 32 users goes 19.1 to 39.4 with prompt speed kept.

    An existing setting, not a new method. Beats turning offload off (E05), which hurt F16 prompt speed. E07 checks long prompts and 96 users. Post

  2. 2

    Q6_K + 0.5B draft model (speculative decoding)

    Existing llama.cpp feature, measured on my test rig
    PASS
    Speed (one user)
    Quality vs full model
    GSM8K 0.71875 vs 0.6875 (same within noise, 32 problems)

    Qwen2.5-0.5B at Q8_0 guesses the next few tokens and the 14B model checks them in one pass. Round E04, Q6_K went from 14.13 to 19.09 tok/s.

    Fastest so far, about 35% faster than Q6_K alone. Not my method, but it's a cheap win for serving on a consumer GPU today.

  3. 3

    Q6_K compression

    Existing tool · llama.cpp
    PASS
    Speed (one user)
    Quality vs full model
    +0.72% worse

    Stores the model at about 6 bits per number instead of 16, and fits 40 of its 48 layers on the GPU.

    Best plain compression so far. In round E04 it matched F16: MMLU 0.772 vs 0.770 (500 questions), GSM8K 0.6875 both, at 14.13 vs 1.98 tok/s.

  4. 4

    Q6_K + n-gram guessing

    Existing technique, tested on Q6_K
    DROPPED
    Speed (one user)
    Quality vs full model
    GSM8K 0.719

    Guesses the next few words by looking for repeats earlier in the text, so it can skip some work.

    The guesses were right only 0–3% of the time. No real speedup, so it's dropped.

  5. 5

    Q8_0 compression

    Existing tool · llama.cpp
    PASS
    Speed (one user)
    Quality vs full model
    +0.64% worse

    Stores each number in 8 bits. You barely lose anything.

    Great quality. Less than half as fast as Q6_K though.

  6. 6

    Full-size F16 (reference)

    Existing tool · llama.cpp
    REFERENCE
    Speed (one user)
    Quality vs full model
    0% (reference)

    The original model, uncompressed. 14 layers on the GPU, the rest on the CPU.

    This is what I compare everything else's quality to.

  7. 7

    Split-plane residency (N1)

    My method
    LOSSLESS
    Speed (one user)
    Quality vs full model
    Bit-identical

    Half of each number lives on the GPU. The other half streams in when it's needed. The output matches the full model exactly.

    Perfect quality. Still slower than the existing tools, so not a win yet.

  8. 8

    Layer streaming (H1)

    My method
    LOSSLESS
    Speed (one user)
    Quality vs full model
    Full quality

    Moves the model's layers from regular memory to the GPU one at a time. With 16 users sharing it, it hits 19.1 tok/s total.

    About 3× faster than Hugging Face's standard offloading. Still behind llama.cpp for one user.

  9. 9

    Exact CPU correction (SSX)

    My method
    DROPPED
    Speed (one user)
    2.3× slower than its baseline
    Quality vs full model
    Identical (MMLU 0.772, perplexity 5.9394)

    The GPU does a rough version and the CPU fixes the errors exactly.

    Quality matched exactly. It was also 2.3× slower for one user and 29× slower for 16. Dropped it.

  10. 10

    Q5 compression

    Existing tool · llama.cpp
    FAIL
    Speed (one user)
    Quality vs full model
    +3.7% worse

    Squeezes each number down to about 5 bits.

    About 2.5× faster than Q6_K (from my E03 log). The answers get noticeably worse though.

  11. 11

    Simple rounding (RTN)

    Existing baseline method
    FAIL
    Speed (one user)
    Speed not the issue: quality failed
    Quality vs full model
    −12.5 points on coding test

    Just rounds every number off to fewer bits. The simplest thing you can do.

    It lost 12.5 points on HumanEval, which is a coding test.

  12. 12

    Q4 compression

    Existing tool · llama.cpp
    FAIL
    Speed (one user)
    Quality vs full model
    +11.4% worse

    Squeezes each number down to about 4 bits.

    Fastest thing on the list (from my E03 log). Way too much quality loss.

Many users on one GPU

Total speed with lots of people on one GPU (full-size F16)

Serve more people at once and total output goes up. Mostly. That dip at 32 users makes no sense to me yet, so I'm digging into it now.

16 users
32 users
64 users

My layer streaming got 19.1 tok/s total with 16 users, for comparison.

Already done by someone else

Ideas I didn't build

I always check whether an idea's been done before I build it. These had. Somebody already published them, so I didn't build or test them, and they're not in the results above.

  • GPU low-bit weights + exact CPU correction in parallel: Keep a rough copy of the model on the GPU and fix its errors on the CPU at the same time. Already published: HCInfer, DecDEC
  • Switching each layer between CPU and GPU by batch size: Decide layer by layer whether the CPU computes it or it gets streamed to the GPU, depending on how many users are active. Already published: Pipelined sharding, ATSInfer, LIA
  • Compressed 8-bit weights decoded on the fly by the CPU: Store the model compressed and unpack it as needed. Decoding also turned out slower than just reading memory. Already published: EntroLLM, Huff-LLM
  • Skipping near-zero work on CPU layers: Skip the calculations that barely affect the answer on layers running on the CPU (“activation sparsity”). Already published: TEAL, PowerInfer
  • A general CPU/GPU work scheduler: Hands out work between the CPU and GPU. Already published: NEO, Fiddler
  • Four more ideas: Nested-plane self-speculation, device-priced mixed precision, GPU prefix-tree pruning, and using the idle GPU to draft guesses with a low-bit copy of the model. Already published
Method

Approach

01

I look before I build

Papers, patents, forums. If someone's already done it, I move on.

02

Standard tests

MMLU, GSM8K, HumanEval and perplexity. That way you can compare my numbers to anyone else's.

03

1% or it doesn't count

Faster is pointless if the model gets dumber. More than 1% worse is a fail.

04

I post the failures

Knowing what doesn't work saves the next person a week.

Writing

Publications

Coming soon

I'm working on the first write-up and the code. Both go up here and on X.