Idea A: voting against a small model (failed)
When the big model's samples disagree, count answers the tiny model also likes a bit less. It lost to plain majority voting at both sample sizes.
| Experiment | IDEA-A |
| Date | 2026-10-27 |
| Benchmark | GSM8K (128 problems), same samples as E15, 320-token cap |
| Model | Qwen2.5-14B-Instruct, Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp), plus 16 samples per problem from Qwen2.5-0.5B Q8_0 |
| Hardware | Two of my dedicated test rigs (RTX 3080). The 0.5B samples ran fully on the GPU of the second rig |
| Verdict | FAIL |
What I tested
On problems where the 14B samples split, score each answer by how often the 14B gave it minus a weight times how often the 0.5B gave it. Weight 1.0 was the main setting and 0.5 the backup. Compared on the exact same samples as plain majority voting.
Pass bar, set before the run
Fixed before seeing results: beat majority voting by at least 2 points on the same samples. Scoring rule and weights were frozen in the analysis script first.
Result
| Samples, weight | Majority vote | Idea A | Difference |
|---|---|---|---|
| 4, 1.0 | 78.9% | 76.6% | -2.3 pts (3 wins, 6 losses, p=0.51) |
| 8, 1.0 | 83.6% | 78.9% | -4.7 pts (2 wins, 8 losses, p=0.11) |
| 8, 0.5 | 83.6% | 80.5% | -3.1 pts (p=0.29) |
The 0.5B model got 31.7% of its own samples right.
What happened
It made things worse at every setting. The likely reason is that the tiny model also gets the easy problems right, so the rule ends up punishing correct answers. Dropped. This was measured under the 320-token cap, but both arms used the same samples, so the comparison is fair even if the absolute numbers are low.
How to rerun
# Needs E15's GSM8K samples (compact JSONL) and 16 samples per problem from the 0.5B with the E15 prompt,
# temperature 0.7, top_k 40, top_p 0.95, 320 max tokens, seeds 5000+j
python a_analyze.py e15_compact.jsonl a_small_q05.jsonlCode
All code for IDEA-A: a_analyze.py.
Prior work
No novelty claim. A second model review rated the idea close to existing work, and it failed anyway.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.