E16: my token cap was cutting off answers
Rerunning GSM8K with a 1024-token cap instead of 320 took one greedy answer from 76.6% to 95.3%. Earlier GSM8K numbers mostly measured truncation.
| Experiment | E16 |
| Date | 2026-10-27 |
| Benchmark | GSM8K (128 problems, same as E15), greedy, 1024-token cap |
| Model | Qwen2.5-14B-Instruct, Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp). Qwen2.5-0.5B Q8_0 as a draft model for speed only, which does not change greedy output |
| Hardware | One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM |
| Verdict | BASELINE |
What I tested
The same 128 problems and prompt as before, with room to finish. Nothing else changed.
Pass bar, set before the run
None. A correction measurement to check whether the 320-token cap explained the errors.
Result
| Setting | Correct | Score |
|---|---|---|
| 320-token cap (E15) | 98 / 128 | 76.6% |
| 1024-token cap (E16) | 122 / 128 | 95.3% |
No answer hit the 1024 cap. Answers averaged 289 tokens. The 6 remaining errors are real reasoning mistakes. Total run time 789 seconds.
What happened
The cap caused about 19 points of error. So the E15 GSM8K baselines and the Idea A and C comparisons were all measured on cut-off answers. With only 6 misses left, GSM8K can no longer tell methods apart for this model, so reasoning tests moved to MATH-500. If you run your own evals, give the model room to finish.
How to rerun
llama-server --port 8093 -c 4096 -np 1 -t 16 -fa on -m q14-Q4_K_M-imat.gguf -ngl 40 --spec-type draft-simple -md qwen2.5-0.5b-instruct-q8_0.gguf -ngld 99 --spec-draft-n-max 16 --spec-draft-n-min 1
python e16_cap1024.pyCode
All code for E16: e16_cap1024.py, e16_run.ps1.
Prior work
No novelty claim. This is a standard evaluation of existing methods.
Written by Andrew Rose with help from AI tools. Every number comes from the run logs.
Questions or ideas? me@andrewroseit.com or @AndrewRoseIT.