← All posts

Ideas I didn't build because someone already did

Before building anything I search the papers. Several of my ideas were already out there, so I dropped them.

I check prior work before I write any code. If an idea's been published, I'd just be redoing someone else's research. Here's what got cut that way.

  • GPU low-bit weights + exact CPU correction in parallel. Already done in HCInfer and DecDEC.
  • Switching each layer between CPU and GPU by batch size. Already done: pipelined sharding, ATSInfer, LIA.
  • Compressed 8-bit weights decoded on the fly by the CPU. Already done in EntroLLM and Huff-LLM. Decoding was also slower than just reading the memory.
  • Skipping near-zero work on CPU layers (activation sparsity). Already done in TEAL and PowerInfer.
  • A general CPU/GPU work scheduler. Already done in NEO and Fiddler.
  • Four more: nested-plane self-speculation, device-priced mixed precision, GPU prefix-tree pruning, and using the idle GPU to draft guesses with a low-bit copy of the model. All previously published.

It's a little annoying when an idea you're excited about turns out to be taken. Still beats spending a month on it.

Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.