E18: rewriting math before solving it (failed)
Rewrite polynomial expressions in a problem into a simpler form before the model solves it. Only 7 of 500 MATH-500 problems qualified, and it changed nothing.
| Experiment | E18 |
| Date | 2026-10-27 |
| Benchmark | MATH-500, greedy, 2048-token cap, original grader |
| Model | Qwen2.5-14B-Instruct, Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp). Qwen2.5-0.5B Q8_0 as a draft model for speed only, which does not change greedy output |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM |
| Verdict | FAIL |
What I tested
Three versions of each problem, one answer each: the original (from E17), a fixed normal form (expanded, one side zero, cleared denominators) as the control, and a selector that picks the simplest of up to 16 safe rewrites by a fixed score. Rewrites run on the CPU with SymPy and are checked to mean the same thing.
Pass bar, set before the run
Set before the run: keep the selector only if it beats both the original and the normal form by at least 3 points on all 500, with Holm-corrected p below 0.05, and the gain is not explained by answers that hit the token cap.
Result
| Version | All 500 | Eligible 7 |
|---|---|---|
| Original (E17) | 76.0% | 6 / 7 |
| Normal form (control) | 75.8% | 5 / 7 |
| Selector | 76.0% | 6 / 7 |
Selector vs normal form: 1 win, 0 losses. Selector vs original: no change. Holm p = 1. Rerunning the originals changed 0 answers, so the setup was stable.
What happened
The restricted parser only found 7 problems it could rewrite, so the idea had almost nothing to act on. A pre-run coverage check had predicted this. Killed with no follow-up variants, as planned. These numbers use the original grader; the fixed grader came after.
How to rerun
python precond.py math500.jsonl plan_check.json
llama-server --port 8094 -c 6144 -np 1 -t 16 -fa on -m q14-Q4_K_M-imat.gguf -ngl 40 --spec-type draft-simple -md qwen2.5-0.5b-instruct-q8_0.gguf -ngld 99 --spec-draft-n-max 16 --spec-draft-n-min 1
python e18_precond.py
python e18_analyze.py # expects e17_math500_greedy.jsonl and e18_precond.jsonlCode
All code for E18: e18_analyze.py, e18_precond.py, e18_run.ps1, math_grade.py, plan_math500.json, precond.py.
Prior work
No novelty claim. A prior-art search and a second model review both rated the idea close to existing work.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.