An AI That Grades Its Own Code Keeps Its Own Misreadings
A popular way to improve an AI model is to let it practise on itself: it writes tasks, runs code to get the answers, and trains on its own solutions that match. Because a computer produced those answers, they feel trustworthy. We tested that on a small open coding model, and found the catch. Running code tells you what the code does, not what it was supposed to do. When the model's misreading of a requirement had reached its code, training on that code's answers never fixed it. An outside answer key helped, then brought a new mistake with it.
01 — the question
What happens when the code shares the mistake?
Picture a rule that says orders of at most $500 skip approval. A coding assistant reads that as under $500 and writes the check that way. Run that code on a $500 order and it will correctly report what the code does: approval needed. The computer made no error. The misunderstanding happened earlier, and now it looks like a verified result.
Self-improvement loops have the same weak point, because one model often writes both the task and the program that produces its answer. We wanted to know which wrong beliefs a loop like this removes, which it leaves alone, and what a correction actually teaches.
02 — how we tested it
Plant known errors, then change only where the answers come from
You cannot measure a fix unless you know exactly what is wrong, so we trained two misreadings into the model on purpose: at most 7 read as below 7, and the 2nd smallest counted from zero. We also tracked two mistakes the model makes by itself about how Python rounds and divides.
Each round, the model got 3,000 requests to write a task. It attempted each valid task eight times, and we trained it only on tasks it solved between one and seven times. Across the runs, we changed where the answer key came from: the model's own code, or code written by the same model from before we planted the errors. A control group never saw the affected tasks. We scored the model on separate test questions it never trained on.
03 — what we found
Whether a belief got fixed depended on where the error lived
Its own code could not correct a misreading in that code. We audited 1,302 "at most" tasks where the answer could be checked. On every one, the model's code gave the wrong count. Training on those answers left it right just 0–3% of the time. That is close to guaranteed once the error is in the code, and it is the point: running the code added no second opinion.
Answers from a model without the error did help. The score rose to 23–58% after three rounds, in all four runs, and to 88% in the one run we extended to six rounds. This is a best case, because we knew the old model did not share the error. In real work you will not know that.
A correct answer key was not always enough. The model counted the 2nd smallest from zero on 99–100% of attempts, so it almost never matched a correct key, and our filter dropped tasks it never solved. In one first round the key was right on 395 of 428 such tasks, yet only a handful reached training. Over six rounds, one run barely moved, from 100% to 96%. A second six-round run fell to 61%, and we do not know why.
Running code did fix mistakes the code itself does not make. A model can wrongly expect round(54.5) to give 55, but Python still returns 54, so here the answer key was right. The rounding mistake fell from about 61% to 4%, and the division mistake from about 41% to 6–10%, far below the control group (33–56%). On those two mistakes, code execution earned its reputation.
04 — what the fix broke
The model learned a different wrong rule
Every correction said the same thing: also count the value that equals the limit. That fits the right rule, but it also fits count up to the nearest value at or above the limit. When the limit was not in the list at all, the model began counting the next value up. In one sample it wrote that "the prices at most 16 are 4, 5, 9, 6, 13, 9, and 17".
A test of the fix alone would have missed this. In the six-round run, the score on the original error reached 88%. Over the same rounds, the model's single best answer on the neighbouring cases dropped from 37 of 50 right to 23 of 50. Counting too many reached 46% of attempts.
Similar shifts appeared elsewhere, less consistently. After learning how Python divides negative numbers, the model often answered -90 % 9 with 9 instead of 0: 29–38% of attempts, up from 10%, though one control run picked up some of this too. A comparable rounding mistake appeared in only one of two repeats. We cannot yet say what causes these rules. The training data was lopsided, and some of the reasoning we trained on reached the right answer by faulty steps. Both fit.
Asking for the missing cases did not work. We told the model to write tasks where the limit is not in the list. In 2,061 checkable tasks, the planted model never did. A version without the planted error managed it 27% of the time. Both often changed the wording instead, adding lines such as "the cap itself does not count", which makes the wrong answer the correct one. So our remedy never got a fair test: the runs contained no true examples of the missing case.
05 — what we apply now
Four practical lessons
These come from one small model in a short loop, so treat them as things to check on your own system rather than settled rules.
1. Get answers from something that does not share the error. If one model writes both the code and the tests, passing tests may only show that the two agree. A separate source helped here, but it is not automatically independent, so measure where it fails too. Ours shared the counting-from-zero error on 5–34% of those tasks.
2. Check generated data directly. If you ask for a particular kind of case, read the inputs to see whether you got it. Our indirect check for the missing cases was wrong on every checkable task it selected, and we found out only by inspecting the inputs after the runs.
3. Look at what your filter throws away. Training only on what a model sometimes gets right gives it almost nothing to learn from when it is nearly always wrong. Extra attempts, hints or worked examples may be needed to start the fix, though we did not test them.
4. Score every fix on the neighbouring cases. Test where the old behaviour was right, not only where it was wrong, and read some of the reasoning you are about to train on. Otherwise you may ship a new error along with a good-looking score.
06 — the conclusion
Verified by code is not the same as correct
Running code proves what the code does. It cannot tell you whether the code understood the request, and a fix is not finished until you have checked what it did to the cases around it.
The limits matter. This was one small model, with up to six rounds and only two repeats at each planting strength. The results vary widely between runs, several single-run effects did not repeat, and the planted errors are simpler than real ones. The study was not pre-registered, and a loop at production scale may behave differently. What did hold in all four main runs: its own code left the planted "at most" error in place, independent answers moved it, and the new counting mistake came with the fix.
This connects to our earlier work on what cheap verifiers miss. The full paper, with every run, table and caveat, is on arXiv.