Ideas I didn't build because someone already did
Before building anything I search the papers. Several of my ideas were already out there, so I dropped them.
I check prior work before I write any code. If an idea's been published, I'd just be redoing someone else's research. Here's what got cut that way.
- GPU low-bit weights + exact CPU correction in parallel. Already done in HCInfer and DecDEC.
- Switching each layer between CPU and GPU by batch size. Already done: pipelined sharding, ATSInfer, LIA.
- Compressed 8-bit weights decoded on the fly by the CPU. Already done in EntroLLM and Huff-LLM. Decoding was also slower than just reading the memory.
- Skipping near-zero work on CPU layers (activation sparsity). Already done in TEAL and PowerInfer.
- A general CPU/GPU work scheduler. Already done in NEO and Fiddler.
- Four more: nested-plane self-speculation, device-priced mixed precision, GPU prefix-tree pruning, and using the idle GPU to draft guesses with a low-bit copy of the model. All previously published.
It's a little annoying when an idea you're excited about turns out to be taken. Still beats spending a month on it.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.