Idea C: draft model as a difficulty signal (failed)

Use how often the speculative draft model's guesses get rejected to decide which problems deserve extra samples. It was no better than the model's own confidence, which is free.

Experiment	IDEA-C
Date	2026-10-27
Benchmark	GSM8K (128 problems), greedy, 320-token cap
Model	Qwen2.5-14B-Instruct, Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp). Qwen2.5-0.5B Q8_0 as a draft model, up to 16 drafted tokens
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM
Verdict	FAIL

What I tested

Whether a low draft acceptance rate flags wrong answers, and whether sending the worst problems by that signal to an 8-sample majority vote gains more right answers than sending them by the model's own average token probability or at random.

Pass bar, set before the run

Set before the run: continue only if AUROC is above 0.65 and routing by acceptance beats routing by token probability for the same extra compute.

Result

Signal	AUROC	Gain routing worst 43	Gain routing worst 32
Low draft acceptance	0.60	+6	+2
Low average token probability	0.65	+6	+5
Answer hit the 320-token cap	0.90	+11	+8
Random (expected)	n/a	+2.7	+2.0

Mean draft acceptance was 0.515. The first run was void because this llama.cpp build needs --spec-type draft-simple to actually draft; the rerun fixed that.

What happened

Draft acceptance was a weak signal, below the bar and no better than token probability, which costs nothing. The surprise was the trivial check: whether the answer hit the token cap. Of 29 wrong answers, 28 had been cut off. That led straight to E16.

How to rerun

llama-server --port 8093 -c 4096 -np 1 -t 16 -fa on -m q14-Q4_K_M-imat.gguf -ngl 40 --spec-type draft-simple -md qwen2.5-0.5b-instruct-q8_0.gguf -ngld 99 --spec-draft-n-max 16 --spec-draft-n-min 1
python c_specstats.py
python c_export.py
python c_analyze2.py   # reads e15_compact.jsonl, c_specstats_compact.jsonl, c_pairs.txt

Code

All code for IDEA-C: c_analyze2.py, c_export.py, c_specstats.py, c_specstats_run.ps1.

Prior work

No novelty claim. My prior-art search found no prior work in this search, but a second model review rated the idea close to existing work, and it failed.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.