Why the model misses MATH-500 problems

I sorted all 120 problems the first grader marked wrong in E17. 16 were actually right. Of the real misses, most were slips and unchecked shortcuts, not missing knowledge.

Experiment	E17-TAXONOMY
Date	2026-10-27
Benchmark	MATH-500, the 120 E17 answers marked wrong by the original grader
Model	Qwen2.5-14B-Instruct, Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp)
Hardware	Analysis only, no GPU. Labels assigned by hand from the model output
Verdict	BASELINE

What I tested

One main cause per wrong answer, plus where in the solution the first error shows up.

Pass bar, set before the run

None. This is analysis to pick what to work on next.

Result

Cause	Count	Share
Algebra or arithmetic slip	34	28%
Wrong method or plan	32	27%
Geometry or diagram	17	14%
Grader false negative (actually right)	16	13%
Counting or casework miss	7	6%
Cut off at 2048 tokens	5	4%
Gave up or unrelated final answer	4	3%
Misread the problem	3	2%
Right work, wrong quantity boxed	2	2%

Most misses are on the hardest problems: 58 at level 5 and 38 at level 4. Intermediate Algebra had the most (42). The first real error was usually in the middle third of the solution (50 of 104).

What happened

Almost half the real misses do not need more math knowledge, just catching mistakes: dropped signs, wrong squares, and shortcuts the model guesses and never checks. The 16 grader misses led to the fixed grader used in E17. The labels are my judgment from reading each output, so treat the split between nearby categories as approximate.

How to rerun

# Per-item labels are in e17_failures.jsonl. To reproduce the grader split:
python e17_rescore.py e17_math500_greedy.jsonl

Code

All code for E17-TAXONOMY: e17_failures.jsonl, e17_failures.md, e17_rescore.py, math_grade.py, math_grade_v2.py.

Prior work

No novelty claim. This is a standard evaluation of existing methods.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.