Does compression level change coding scores?

HumanEval-164 with three compressed versions of Qwen2.5-14B. All three landed within 3 problems of each other, which is noise.

Experiment	E14
Date	2026-10-27
Benchmark	HumanEval, all 164 problems, pass@1, greedy
Model	Qwen2.5-14B-Instruct at Q6_K, Q5_K_M and Q4_K_M with an importance matrix, 40 of 48 layers on the GPU (llama.cpp)
Hardware	One of my dedicated test rigs: RTX 3080 10 GB, Ryzen 9 5950X, 128 GB RAM
Verdict	BASELINE

What I tested

Whether the cheaper, faster compression levels lose coding ability. Each version writes one answer per problem, and the answer is run against the problem's tests in a separate process with a 15 second timeout.

Pass bar, set before the run

None. This is a measurement to pick which compression level to use for later tests.

Result

Version	Solved	Score
Q5_K_M	132 / 164	80.5%
Q6_K	131 / 164	79.9%
Q4_K_M + importance matrix	129 / 164	78.7%
F16 (full size)	not finished	skipped

One problem is worth 0.6 points, so a 3 problem spread is within noise. The full-size F16 run was stopped after 13 problems because it ran at about 2 tokens per second and would have taken about 2 more hours.

What happened

All three versions scored about the same on coding, so the smallest and fastest one, Q4_K_M with an importance matrix, stays the default for reasoning tests. That matters for keeping capable AI usable on hardware people already own: the cheaper version did not give up measurable coding ability here.

How to rerun

# For each model file, start a server, then run the eval (see e14_run.ps1)
llama-server --port 8090 -c 4096 -np 1 -t 16 -fa on -m q14-Q4_K_M-imat.gguf -ngl 40
python humaneval_eval.py --tag q4km_imat_ngl40_he

Code

All code for E14: e14_run.ps1, humaneval_eval.py.

Prior work

No novelty claim. This is a standard evaluation of existing methods.

Written by Andrew Rose with help from AI tools. Every number comes from the run logs.