Baselines any reasoning idea has to beat

Standard ways to get more right answers from the same model, measured on GSM8K and HumanEval. Majority voting helped, picking the most confident answer did not. The GSM8K numbers were later found to be cut off by a 320-token cap.

Experiment	E15
Date	2026-10-27
Benchmark	GSM8K (128 problems) and HumanEval (164 problems)
Model	Qwen2.5-14B-Instruct, Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp)
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM, 8 requests at once, GGML_OP_OFFLOAD_MIN_BATCH=128
Verdict	BASELINE

What I tested

Known methods only: one greedy answer, majority vote over 4 and 8 samples (temperature 0.7), picking the sample with the highest average token probability, and a best-possible-pick reference that peeks at the answer key.

Pass bar, set before the run

None. These are the reference numbers new ideas are compared against at the same compute.

Result

Method	GSM8K	Tokens per problem
One answer (greedy)	76.6%	266
Majority vote, 4 samples	78.9%	1068
Majority vote, 8 samples	83.6%	2134
Most confident of 4	72.7%	1068
Most confident of 8	74.2%	2134
Best possible pick of 8 (reference)	90.6%	2134

HumanEval: one answer 78.7% (161 tokens), most confident of 4 80.5% (642 tokens), best possible pick of 4 90.2% (reference).

Correction: every GSM8K number here used a 320-token output cap, and E16 later showed the cap caused most of the errors. Read the GSM8K rows as cut-off-limited.

What happened

Majority voting gave a real gain on GSM8K and picking the most confident sample did worse than one answer. The gap to the best possible pick suggested room for a smarter picker. Later, E16 showed the 320-token cap was cutting off most wrong answers, so these GSM8K baselines measure truncation more than reasoning. The HumanEval numbers were not affected the same way.

How to rerun

# Start the server (see e15_run.ps1), then:
set GGML_OP_OFFLOAD_MIN_BATCH=128
llama-server --port 8090 -c 8192 -np 8 -t 16 -fa on -m q14-Q4_K_M-imat.gguf -ngl 40
python e15_baselines.py --task gsm8k --tag q4km_imat_ngl40 --n_items 128 --n_samples 8 --workers 8 --resume
python e15_baselines.py --task humaneval --tag q4km_imat_ngl40 --n_items 164 --n_samples 4 --workers 8 --resume

Code

All code for E15: e15_baselines.py, e15_run.ps1.

Prior work

No novelty claim. This is a standard evaluation of existing methods.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.