E17: MATH-500 baseline, 79.2%

One greedy answer per problem on MATH-500 scores 396 of 500 (79.2%) with a fixed grader. The first grader said 380, but 16 of its misses were right answers in a different format.

Experiment	E17
Date	2026-10-27
Benchmark	MATH-500, greedy, 2048-token cap
Model	Qwen2.5-14B-Instruct, Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp). Qwen2.5-0.5B Q8_0 as a draft model for speed only, which does not change greedy output
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM
Verdict	BASELINE

What I tested

The plain baseline on a harder math set, one answer per problem, scored by the last \boxed{} answer.

Pass bar, set before the run

None. This is the number new reasoning methods on MATH-500 have to beat.

Result

Grader	Correct	Score
Original grader (v1)	380 / 500	76.0%
Fixed grader (v2)	396 / 500	79.2%

The 16 extra answers were correct but written differently, like C instead of (C), a units label, or a fraction written another way. The fixed grader accepts everything the old one did and was checked to not accept wrong answers. 5 answers hit the 2048 cap.

What happened

MATH-500 leaves real room to improve, unlike GSM8K. The first score of 76.0% was low because the grader rejected correct answers over formatting. After fixing it the baseline is 79.2%, and every later MATH-500 comparison uses the fixed grader.

How to rerun

llama-server --port 8093 -c 6144 -np 1 -t 16 -fa on -m q14-Q4_K_M-imat.gguf -ngl 40 --spec-type draft-simple -md qwen2.5-0.5b-instruct-q8_0.gguf -ngld 99 --spec-draft-n-max 16 --spec-draft-n-min 1
python e17_math500.py
python e17_rescore.py e17_math500_greedy.jsonl   # prints v1 380, v2 396

Code

All code for E17: e17_math500.py, e17_rescore.py, e17_run.ps1, math_grade.py, math_grade_v2.py.

Prior work

No novelty claim. This is a standard evaluation of existing methods.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.