E20: an 8B model trained on its own correct answers

I let an 8B model solve math training problems, kept only the answers that came out right, and fine-tuned it on them with a small LoRA on one 10 GB GPU. On MATH-500 it gained 1.6 points, which is not enough to count. Fail.

Experiment	E20
Date	2026-10-28
Benchmark	MATH-500, greedy, 2048-token cap, grader v2
Model	Qwen3-8B, Q4_K_M GGUF (llama.cpp), thinking off; LoRA trained with 4-bit QLoRA
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM
Verdict	FAIL

What I tested

Standard self-training (STaR / rejection-sampling fine-tuning). On 1,500 MATH train problems the model sampled 4 answers at temperature 0.8 on the ones it missed greedily, kept up to 2 that matched the gold answer, and mixed them 1:1 with its own correct greedy answers. One epoch of QLoRA (rank 16, all linear layers, max length 1024) on the same GPU. Then base and base plus LoRA were run greedy on MATH-500 and compared item by item.

Pass bar, set before the run

Set before training: tuned minus base at least +3.0 points and exact McNemar p below 0.1.

Result

Model	Correct	Accuracy	Generated tokens	Hit the 2048 cap
Qwen3-8B base	392/500	78.4%	372,497	57
Qwen3-8B + self-trained LoRA	400/500	80.0%	354,267	49

Paired: 26 problems only the tuned model got right, 18 only the base got right, exact McNemar p = 0.29.

Most of the gain was Intermediate Algebra (60 to 67 of 97). Geometry went down (29 to 27 of 41) and Number Theory went down (54 to 51 of 62).

The tuned model wrote about 5% fewer tokens and hit the length cap less often.

Training took about 13 minutes and the MATH-500 eval about 25 minutes. The first training attempt crashed the GPU driver, so this run was done with the GPU power limit set to 240 W.

What happened

One round of training on its own correct answers made the 8B a little better and a little shorter, but the gain is small enough that it could be noise, and it does nothing for geometry. A longer run, more problems or several rounds might do better; this run does not show it.

How to rerun

# Build the training set from the model's own sampled answers, then train and evaluate:
powershell -ExecutionPolicy Bypass -File phaseA.ps1
powershell -ExecutionPolicy Bypass -File phaseB.ps1
# Paired comparison and per-subject table:
python analyze.py

Code

All code for E20: analyze.py, build_sft.py, gen.py, grade.py, math_grade_v2.py, night5.ps1, phaseA.ps1, phaseB.ps1, prep.py, train.py.

Prior work

No novelty claim. This is standard self-training (Zelikman et al., 2022 STaR; Yuan et al., 2023 rejection-sampling fine-tuning).

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.