I'm making frontier-size LLMs run on one consumer GPU without losing quality.
I'm Andrew Rose. I work on getting large language models to run well on hardware people actually own, and on the efficiency questions that matter in data centers: cost per token. This site is my open lab notebook, with the results, the baselines, and the things that didn't work.
Research · lab notebook, in progress
Test rig: one RTX 3080 (10 GB), Ryzen 9 5950X, 128 GB RAM. Model: Qwen2.5-14B. These numbers are early and will change as I go.
Layer streaming
The model doesn't fit in 10 GB of VRAM, so I stream layers from system RAM to the GPU as they're needed.
Split-plane residency
I keep the top byte of every BF16 weight resident on the GPU and stream only the low byte over PCIe, then recombine them in a fused Triton kernel. Output is bit-identical to the full model.
Being honest about the competition
Here's how existing tools do on the same machine. Right now llama.cpp's 8-bit mode is the number to beat.
| Method | Speed (batch 1) | Quality |
|---|---|---|
| My layer streaming | 1.19 tok/s | Full BF16 |
| My split-plane residency | 1.31 tok/s | Bit-identical |
| llama.cpp F16, partial offload | 2.05 tok/s | Full F16 |
| llama.cpp Q8_0 | 5.17 tok/s | +0.64% perplexity |
What didn't work
- Significance-split CPU correction: quality was identical, but it ran 2.3× slower. Dropped.
- Ideas that already existed: several of my ideas turned out to be published already when I did the prior-art search, so I dropped them rather than redo someone else's work.
Publications
The first write-up and code are in progress. They'll be posted here and on X.
Approach
- Check prior art first. Before I build anything, I search papers, patents, and community work. If it's been done, I drop it.
- Use standard benchmarks. MMLU, GSM8K, HumanEval, and perplexity, so the results are comparable.
- Set a ≤1% degradation bar. Speed only counts if quality stays within 1% of the full model.
- Publish negative results. What failed is as useful as what worked.
Contact
I'm open to research roles and collaborations in LLM inference and efficiency.