E19: four samples and a vote on MATH-500
Sample four answers per problem and keep the most common one. It beat a single greedy answer by 3.2 points on MATH-500, at about four times the tokens. This is now the bar new reasoning methods have to clear.
| Experiment | E19 |
| Date | 2026-10-28 |
| Benchmark | MATH-500, 2048-token cap per sample, grader v2 |
| Model | Qwen2.5-14B-Instruct, Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp), q8_0 KV cache |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM |
| Verdict | BASELINE |
What I tested
Standard self-consistency: four samples per problem at temperature 0.7, top-p 0.95, majority vote over the final boxed answers (ties go to the first sample). Compared item by item with the greedy run from E17, scored by the same grader.
Pass bar, set before the run
None. This is a baseline. It sets the matched-compute bar: a new method using about four samples of compute must beat it by at least 3 points with McNemar p below 0.1.
Result
| Method | Correct | Accuracy | Generated tokens | GPU time |
|---|---|---|---|---|
| Greedy, one answer (E17) | 396/500 | 79.2% | 294,238 | 1.5 h |
| One sample at T=0.7 (average of 4) | 389/500 | 77.8% | about 287,500 | n/a |
| Self-consistency, 4 samples | 412/500 | 82.4% | 1,150,073 | 7.1 h |
| Any of the 4 samples right | 440/500 | 88.0% |
Paired against greedy: 32 problems only the vote got right, 16 only greedy got right, exact McNemar p = 0.029.
Tokens per solved problem: 743 for greedy, 2,791 for the vote. GPU seconds per solved problem: 13.9 and 62.3.
Geometry did not move at all (26 of 41 both ways). Precalculus gained the most (37 to 42 of 56).
In 28 problems a correct answer was among the four samples but lost the vote. That gap is what a smarter way of choosing could go after.
What happened
Voting helps, but it is expensive: four times the tokens for 3.2 points. Most of the gain is on mid-difficulty algebra and precalculus, where one sample slips and the others do not. Geometry errors are the same in every sample, so voting cannot fix them. The run was paused twice for another experiment and once by a rig reboot; it resumes by skipping finished problems, so every problem was sampled exactly once with the same seeds.
How to rerun
# Start the server and sample (edit the model path in run_e19.ps1):
powershell -ExecutionPolicy Bypass -File run_e19.ps1
# Score with grader v2 and compare with the greedy E17 output:
python e19_analyze.pyCode
All code for E19: e19_analyze.py, e19_math500_sc.py, math_grade.py, math_grade_v2.py, run_e19.ps1.
Prior work
No novelty claim. Self-consistency is a published method (Wang et al., 2022). This is a baseline measurement.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.