# E17 failure taxonomy: Qwen2.5-14B Q4_K_M-imat, MATH-500 greedy, 2048 cap

Source: Kylei-PC `C:\AGI\agi-research\results\quality\e17_math500_greedy.jsonl` (500 rows). Graded with `precond/math_grade.py` (the same grader as E18); it marks 120 wrong, matching the reported 380/500.
Each wrong answer was classified by hand from the model output. One primary cause per item. 'First error' is a hand estimate of where the first wrong step appears in the text (early, middle or late third). Per-item labels: `e17_failures.jsonl`.

## Counts by category

| Category | Count | Share |
|---|---|---|
| algebra or arithmetic slip | 34 | 28% |
| wrong method or plan | 32 | 27% |
| geometry or diagram | 17 | 14% |
| grader false negative (correct answer, format) | 16 | 13% |
| counting or casework miss | 7 | 6% |
| truncated / hit 2048 cap | 5 | 4% |
| gave up / non-sequitur final answer | 4 | 3% |
| misread the problem | 3 | 2% |
| bad final answer (right work, boxed wrong quantity) | 2 | 2% |

## Category by subject

| Category | Algebra | Counting & Probability | Geometry | Intermediate Algebra | Number Theory | Prealgebra | Precalculus |
|---|---|---|---|---|---|---|---|
| SL | 3 | 4 | 2 | 14 | 3 | 2 | 6 |
| WM | 0 | 0 | 1 | 21 | 1 | 0 | 9 |
| GE | 0 | 1 | 11 | 0 | 0 | 5 | 0 |
| FN | 2 | 0 | 1 | 2 | 3 | 2 | 6 |
| CC | 1 | 3 | 0 | 1 | 1 | 1 | 0 |
| T | 0 | 0 | 1 | 3 | 0 | 0 | 1 |
| GU | 0 | 0 | 0 | 1 | 0 | 0 | 3 |
| MR | 1 | 1 | 0 | 0 | 0 | 1 | 0 |
| AF | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| total | 8 | 9 | 16 | 42 | 8 | 12 | 25 |

## Category by level

| Category | L1 | L2 | L3 | L4 | L5 |
|---|---|---|---|---|---|
| SL | 0 | 1 | 3 | 14 | 16 |
| WM | 0 | 1 | 4 | 8 | 19 |
| GE | 0 | 2 | 0 | 4 | 11 |
| FN | 1 | 3 | 5 | 5 | 2 |
| CC | 0 | 0 | 0 | 3 | 4 |
| T | 0 | 0 | 0 | 2 | 3 |
| GU | 0 | 1 | 1 | 1 | 1 |
| MR | 0 | 1 | 0 | 0 | 2 |
| AF | 0 | 1 | 0 | 1 | 0 |
| total | 1 | 10 | 13 | 38 | 58 |

Codes: SL = algebra or arithmetic slip, WM = wrong method or plan, GE = geometry or diagram, FN = grader false negative (correct answer, format), CC = counting or casework miss, T = truncated / hit 2048 cap, GU = gave up / non-sequitur final answer, MR = misread the problem, AF = bad final answer (right work, boxed wrong quantity)

## Where the first error occurs (excluding FN, which has no error)

- early (first third): 21
- middle third: 50
- late (last third): 33

By category (E/M/L):
- SL: 2/18/14
- WM: 6/20/6
- GE: 8/9/0
- CC: 2/3/2
- T: 0/0/5
- GU: 0/0/4
- MR: 3/0/0
- AF: 0/0/2

## Token length

- Wrong answers: median 766.0 tokens; 5 hit the 2048 cap (all 5 are loops: 2 search loops, 3 degenerate repetition of one line).
- SL: median 980.5 tokens
- WM: median 812.5 tokens
- GE: median 572 tokens
- FN: median 526.0 tokens
- CC: median 495 tokens
- T: median 2048 tokens
- GU: median 979.5 tokens
- MR: median 522 tokens
- AF: median 496.5 tokens

## Examples (real items) for the top 3 categories

### algebra or arithmetic slip
- MATH-500 `test/precalculus/990.json` (Precalculus L5): Let $z = 2 + \sqrt{2} - (3 + 3 \sqrt{2})i$, and let $c = 2 - 3i$. Let $w$ be the result when $z$ is rotated around $c$ by $\frac{\pi}{4}$ counter-clockwise. [as... Gold `6 - 5i`, model `-5i`. sign slip: -3sqrt2 i * i sqrt2/2 taken as -3 not +3.
- MATH-500 `test/intermediate_algebra/1388.json` (Intermediate Algebra L5): The function $f$ satisfies the functional equation \[f(x) + f(y) = f(x + y) - xy - 1\]for all real numbers $x$ and $y.$ If $f(1) = 1,$ then find all integers $n... Gold `1,-2`, model `0`. recurrence f(k+1)=f(k)+k+2 then wrong closed form n^2-1 (inductive step actually failed).

### wrong method or plan
- MATH-500 `test/intermediate_algebra/1994.json` (Intermediate Algebra L5): Define \[p = \sum_{k = 1}^\infty \frac{1}{k^2} \quad \text{and} \quad q = \sum_{k = 1}^\infty \frac{1}{k^3}.\]Find a way to write \[\sum_{j = 1}^\infty \sum_{k ... Gold `p - q`, model `\frac{1}{2} q (1 + p)`. hallucinated 'known identity' for double zeta sum.
- MATH-500 `test/number_theory/1032.json` (Number Theory L3): What is the least positive integer multiple of 30 that can be written with only the digits 0 and 2?... Gold `2220`, model `222`. treated digit 0 as ensuring divisibility by 5/10; accepted 222.

### geometry or diagram
- MATH-500 `test/prealgebra/1646.json` (Prealgebra L5): In quadrilateral $ABCD$, angle $BAD$ and angle $CDA$ are trisected as shown. What is the degree measure of angle $AFD$? [asy] size(150); pair A , B, C, D; A = (... Gold `80`, model `110`. diagram: angles taken as 110/100 and wrong angle chase.
- MATH-500 `test/geometry/183.json` (Geometry L5): A hexagon is inscribed in a circle: [asy] pair pA, pB, pC, pD, pE, pF, pO; pO = (0, 0); pA = pO + dir(-10); pB = pO + dir(60); pC = pO + dir(130); pD = pO + dir... Gold `145^\circ`, model `75`. used 'opposite angles supplementary' for cyclic hexagon.

## Fixable without new knowledge vs knowledge or reasoning gaps

- Fixable with no new model knowledge (FN + AF + SL + T): 57/120 = 48%. FN 16 and AF 2 are pure answer-format issues; SL 34 are local slips a checker could catch; T 5 are loops.
- Plausibly fixable by checking/search (CC 7, MR 3, GU 4): 14.
- Real knowledge or strategy gaps (WM 32 + GE 17): 49/120 = 41%.
- If the 16 grader false negatives are counted correct, the E17 score is 396/500 = 79.2%.

Caveats: one rater (me), judged from the text; several items mix causes (for example a slip followed by a guessed final answer), and the primary label is the first decisive error.

## Example check (2026-10-27)

Every numeric claim in the example notes was rechecked in Python against the raw outputs in `e17_wrong_raw.json`. All of them correctly describe what the model wrote; none were our miscalculations. The notes on six items (357, 1072, 662, 765, 1065, 966) were reworded to say plainly "model wrote X; correct value is Y", because the short form could be read as our own claim. Category counts are unchanged.