Three tests compare computing the mistake with asking a model.
Summary
For a problem that is two numbers and an operation, the Harness applies each of 66 faulty procedures to the problem
and names the one that gives exactly what the learner wrote (src/diagnose.js). When none
does, it names nothing, and the turn goes on as it would have without a name.
- Code names the right procedure as often as the best of three models asked the same question (70 answers in 100 against 71), and names a wrong one half as often (10 against 21).
- On wrong answers generated by another team's library, code names 61 in 100 of those it can read, and the same procedure of theirs gets the same name of ours 88 times in 100.
- On that library's own task, predicting a learner's next answer, a prediction from code is right about nine times in ten. With code first and a model asked only where code has no prediction, a small open-weight model goes from 33 right in 100 to 60.
| Test | Cases | Models asked |
|---|---|---|
| Against a model asked the same question | 80 wrong answers, the test half of the scorecard's set | gpt-6-luna, gpt-oss-120b, gemma-4-31b-it |
| On wrong answers from another library | MalruleLib's 4,991 problems, of which code takes the 547 that are two numbers and one operation | None |
| On predicting a learner's next answer | MalruleLib's 12,706 pairs, of which 779 are in scope and 200 were put to models | gemma-4-31b-it, gpt-oss-120b |
Against a model asked the same question
The cases are the 80 wrong answers in the test half of wrong-answers.json,
each labelled with the faulty procedure behind it. Six of them are slips that no procedure explains. Code computes.
Each model is shown the problem, the learner's answer and the same list of 66 procedures, and is told to work a
procedure through before choosing it, or to answer "none".
| Names the labelled procedure | Names another | Gives a slip a procedure | |
|---|---|---|---|
| Code, no model | 70% (56 of 80; 59 to 79) | 10% (8; 5 to 19) | 0 of 6 |
gpt-6-luna, asked |
71% (57; 61 to 80) | 21% (17; 14 to 31) | 2 of 6 |
gpt-oss-120b, asked |
63% (50; 52 to 72) | 34% (27; 24 to 45) | 3 of 6 |
gemma-4-31b-it, asked |
44% (35; 33 to 55) | 56% (45; 45 to 67) | 6 of 6 |
Each range is a 95% interval. Code and the models answer the same 80 cases, so the differences are taken case by case:
| Code against | Names the labelled procedure, points | Names another, points |
|---|---|---|
gpt-6-luna |
-1.3 (-6.3 to 3.8) | -11.3 (-18.8 to -5.0) |
gpt-oss-120b |
+7.5 (0.0 to 16.3) | -23.8 (-33.8 to -13.8) |
gemma-4-31b-it |
+26.3 (16.3 to 37.5) | -46.3 (-57.5 to -35.0) |
Code finds as many as the best of the three models. It names a wrong procedure on 11 fewer answers in 100 than that
model, 24 fewer than the larger open-weight model and 46 fewer than the smaller. The difference lies in what happens
when there is nothing to find: code says nothing, and the models name something. gemma-4-31b-it, of a size a
district could run on its own server, never answered "none" and was wrong more often than right.
A wrong name is the costlier error, since a tutor told the wrong misconception addresses a mistake the learner didn't make.
On wrong answers from another library
MalruleLib (Chen, Liu & Sonkar, 2026, arXiv 2601.03217) holds 101 faulty procedures in school mathematics as code, and 4,991 generated problems with the wrong answer each procedure gives. None of the Harness's procedures were written from it. Nothing is matched by hand: a script counts how many of their problems are in scope, how many of their wrong answers get a name, and whether one of their procedures keeps getting the same name of ours.
| Result | |
|---|---|
| Their problems that are two numbers and one operation, and read correctly | 547 |
| Wrong answers to those that code names | 333 of 547 (60.9%) |
| Named answers that carry the name most often given to their procedure | 88.3% |
About one problem in nine is in scope. Most of MalruleLib is algebra, exponents, radicals, statistics and geometry.
The test also found a fault. The code that reads a problem took 94 problems for two-number arithmetic that weren't:
|3x - 4| when x = -3 for 3 × -4, √(3+1) for 3 + 1, and a word problem with three numbers for the two inside its
brackets. A wrong reading gives a wrong right answer, and then a confident name for a mistake nobody made. The reader
was changed to refuse a problem with a root, a power, an absolute value, an equals sign before the numbers, a letter beside
them, or any other number in it. It then misreads 2 of the same problems and takes 540 as in scope.
Four more changes came from the answers that went unnamed. A decimal cut short is read as the longer one (7.11 for 64/9). An answer that is one of the problem's own numbers counts when it stands alone or ends a line of working (7/9 for 1/9 × 7/9). Two procedures from the classic lists of arithmetic bugs were added (Brown & Burton, 1978): the carried digit never added when multiplying, and the points left out of a sum. With these, code names 361 of those 540 wrong answers (66.9%). That figure was reached by reading this same data, so it shows only that the gaps found are closed, and 60.9% is the estimate for cases the procedures weren't written from.
Still without a name are MalruleLib's borrowing variants in column subtraction, misaligned rows in multi-digit multiplication, and carrying the wrong digit when adding.
On predicting the next answer
MalruleLib's paper asks models to infer a misconception from one worked mistake and predict the same student's answer to another problem. Across nine models it reports 66% on solving the problems and 40% on predicting the answer when the second problem is from another template. Two problems share a template when the same generator of the library made both.
A procedure that is code can do this directly: name the procedure behind the first answer, then apply it to the
second problem (applyProcedure). Of the library's 12,706 pairs, 779 have both problems in
scope. On those, code predicts the next answer exactly on 62.5% of the 504 same-template pairs and 40.0% of the 275
cross-template pairs. It predicts another answer on 3.0% and 7.6%, and makes no prediction on the rest.
Two open-weight models were asked the same question on 200 of those pairs, 100 of each kind:
| On the same 200 pairs | Same template: right, wrong, none | Cross template: right, wrong, none |
|---|---|---|
| Code | 60%, 6%, 34% | 45%, 7%, 48% |
gemma-4-31b-it, asked |
58%, 42%, 0% | 33%, 65%, 2% |
gpt-oss-120b, asked |
79%, 19%, 2% | 56%, 41%, 3% |
Code, then gemma-4-31b-it where code has no prediction |
73%, 27%, 0% | 60%, 39%, 1% |
Code, then gpt-oss-120b where code has no prediction |
79%, 19%, 2% | 63%, 34%, 3% |
When code makes a prediction it is right about nine times in ten, and it makes one for half to two thirds of the pairs. The larger model gets more pairs right than code and is wrong three to six times as often. Putting code first helps the small model clearly: on cross-template pairs it goes from 33% right to 60%. The larger goes from 56% to 63%, which is within what chance could give, since with 100 pairs in each cell a difference under about 14 points could be chance. Both make fewer wrong predictions.
The Harness uses a diagnosis the same way in a turn. Its instructions say what the answer shows only when code found a procedure, and otherwise leave the model to work it out.
Limits
The 80 wrong answers of the first test were written by an AI model together with their labels, and no teacher has checked them. MalruleLib's wrong answers are generated by its procedures. No learner's answers are in any of the three tests, which measure whether a name or a prediction matches a label.
Print the tables again
Every answer a model gave is in runs/, by case and by pair. With no key set, these commands read the kept
answers and print the tables without a model:
node evals/diagnosis/ask-a-model.mjs --models gpt-6-luna,gpt-oss-120b,gemma-4-31b-it --kept evals/diagnosis/runs/asked.jsonl
MalruleLib's license allows research and open-source use, including evaluating models whose weights are public, and forbids commercial use, so none of it is in this repository. Clone it beside this one for the other two tests:
git clone https://github.com/sonkar-lab/malrulelib && git -C malrulelib checkout 8ddedfb
node evals/diagnosis/malrulelib.mjs malrulelib/data/raw_data.csv --rules
node evals/diagnosis/predict-next.mjs malrulelib/data/experiment_pairs.csv --models gemma-4-31b-it,gpt-oss-120b --sample 200 --kept evals/diagnosis/runs/predicted.jsonl
To ask a model again, set OPENAI_BASE_URL and OPENAI_API_KEY. New answers are added to the file --kept names.
This page comes from the open repository: harness/evals/diagnosis/README.md. To change it, change it there.
