Two reasoning ideas that didn't work

Update, Oct 27: every GSM8K number in this post was limited by a 320-token output cap. With a 1024-token cap (round E16), one greedy answer scored 122 of 128 (95.3%), with 289 tokens per answer on average and only 6 real errors. The Idea A and Idea C verdicts below were measured under the old cap. GSM8K is now too easy for this model to tell methods apart, so my next tests move to MATH-500. Details below.

Speed is only half of it. A model that runs on hardware people already own is only useful if it gets the answer right. So this week I started testing ways to get more correct answers out of Qwen2.5-14B for the same amount of compute.

First I needed baselines any new idea has to beat. Round E15, Qwen2.5-14B at Q4_K_M with an importance matrix, 40 layers on the GPU, on one of my dedicated RTX 3080 test rigs. Nothing here is mine. These are standard methods. All of these numbers were cut off at 320 tokens.

Majority voting is the known self-consistency trick. It works. Picking the sample the model was most confident in (highest average token probability) actually did worse than just taking one answer. The last row is a cheat. It counts a problem as solved if any of the 8 samples got it, so it shows how much room a smarter picker could have.

On coding (HumanEval, 164 problems) one answer got 78.7%, the most confident of 4 got 80.5%, and the best possible pick of 4 was 90.2%.

Idea A: vote against the small model

When the 14B samples disagree, look at what a tiny 0.5B model answers on the same problem. If the tiny model lands on an answer a lot, maybe it's the easy wrong answer, so count it a bit less. I fixed the scoring rule before looking at any results. Measured under the 320-token cap.

It made things worse at both sizes. At 8 samples it won 2 problems and lost 8. My best guess is that the small model also gets the easy problems right, so this ends up punishing correct answers. Dropped.

Idea C: let the draft model say when a problem is hard

Speculative decoding already runs a small draft model that guesses tokens for the big one. When lots of guesses get rejected, maybe the problem is hard. Then you could spend extra samples only on those problems instead of on everything. Also measured under the 320-token cap.

Mean acceptance was 0.515. As a signal for which answers were wrong, it scored an AUROC of 0.60. The model's own average token probability, which you get for free, scored 0.65.

Sending the worst third of problems (43 of 128) to an 8-sample vote gained 6 right answers using acceptance and 6 using token probability. Random picks would gain about 2.7. At a quarter of the problems, acceptance gained 2, the same as random. Not good enough. Dropped.

The real lesson: my token cap

While digging through Idea C I checked whether each answer hit the 320-token output limit. That simple check scored an AUROC of 0.90. Of the 29 wrong greedy answers in that run, 28 had hit the cap. Only 1 of the 86 answers that finished on their own was wrong.

So my GSM8K numbers so far mostly measure answers getting cut off, not reasoning. That affects the baselines above and both ideas. If you run your own evals, give the model enough room to finish.

The rerun

Round E16 reran the same 128 problems with one greedy answer and a 1024-token cap. It got 122 right (95.3%), up from 76.6%. Answers averaged 289 tokens, and only 6 of the misses were real reasoning errors.

That means the baselines above, the E13 quant comparison, and both idea verdicts were all measured under the cutoff. It also means GSM8K is now too easy for this model. With only 6 misses left there isn't room to show one method beating another, so I'm moving my reasoning tests to MATH-500, which is harder.

On prior work: before building either idea I searched for it and had a second model review it. It rated both as close to existing work, so I'm not claiming either one was new.