It calls no model and reaches no network, so anyone gets the same numbers in under a second:
npx @viable-lab/harness scorecard
Guarantees are things the code does every time, such as never copying a learner's words into what the model is told.
The measures say how often code reads a message, a wrong answer or a conversation the way its label says, and how
much review, a check for people building a tool for learners, finds of what was planted.
The tables below are what the command prints. A test fails if this page, the figure and a run ever differ.
| Guarantee | Checks | Failed |
|---|---|---|
| What a learner writes changes what is chosen for a turn, and is never copied into what the model is told. | 1,800 | 0 |
| How far the teacher lets the AI go is in every turn's instructions, whatever the learner wrote. | 1,800 | 0 |
| A record of a turn holds neither the learner's words nor the reply, unless it was asked to. | 360 | 0 |
| Nothing that names a learner goes on to the model's endpoint through the gateway: not its own header, not the fields a request names an end user in. | 60 | 0 |
| A tool the teacher switched off is taken out of the request, not left in with a note asking the model not to use it. | 72 | 0 |
| An answer key goes to the model that reviews a reply and never to the one that writes or rewrites it. | 30 | 0 |
| A reply that states the answer key’s number is failed by code before any reviewer is asked, and what the reply model is then told never holds the answer. (Where the question itself holds that number, the reviewer decides.) | 158 | 0 |
| A worked example that code chooses the problem for has none of the learner’s numbers in it and never their answer, so working it through can’t give theirs away. | 159 | 0 |
| Reading the moment, by rules | Messages | Read as labelled |
|---|---|---|
| Test half, English | 150 | 83.3% (125 of 150; 95% interval 76.6 to 88.4) |
| Test half, Spanish | 15 | 86.7% (13 of 15; 95% interval 62.1 to 96.3) |
| Test half, Chinese | 15 | 86.7% (13 of 15; 95% interval 62.1 to 96.3) |
| Development half, English | 150 | 94.7% (142 of 150; 95% interval 89.8 to 97.3) |
| Development half, Spanish | 15 | 80% (12 of 15; 95% interval 54.8 to 93) |
| Development half, Chinese | 15 | 86.7% (13 of 15; 95% interval 62.1 to 96.3) |
review, on projects with problems planted |
Projects | Planted problems found | Pointed at, and not planted |
|---|---|---|---|
| Test half | 30 (10 with nothing planted) | 86.8% (46 of 53; 95% interval 75.2 to 93.5) | 0 |
| Development half | 30 (10 with nothing planted) | 100% (59 of 59; 95% interval 93.9 to 100) | 0 |
| The procedure behind a wrong answer, by code | Wrong answers | Named as labelled | Named as something else | Slips named |
|---|---|---|---|---|
| Test half | 80 of 80 read as arithmetic | 70% (56 of 80; 95% interval 59.2 to 78.9) | 10% (8 of 80; 95% interval 5.2 to 18.5) | 0 of 6 |
| Development half | 80 of 80 read as arithmetic | 77.5% (62 of 80; 95% interval 67.2 to 85.3) | 5% (4 of 80; 95% interval 2 to 12.2) | 0 of 14 |
| A policy in code, by rules | Conversations | The step decided is the one the turn requires | A right answer is seen | Seen where the learner gave none |
|---|---|---|---|---|
| Test half | 60 | 89.9% (310 of 345; 95% interval 86.2 to 92.6) | 90% (18 of 20; 95% interval 69.9 to 97.2) | 0 of 301 turns |
| Development half | 60 | 94.2% (325 of 345; 95% interval 91.2 to 96.2) | 100% (18 of 18; 95% interval 82.4 to 100) | 0 of 274 turns |
How to read it
Guarantees
Guarantees are claims this repository makes in plain words, each turned into a check that has to hold on every case: a single failure fails the command. They are run on the 360 messages below, each with a line added that tries to pass for an instruction, at each of the five levels of help. They cover what the code does. They say nothing of what a model then does with its instructions: that is what VETTING and agent tests are for.
Reading the moment
Reading the moment is the first thing a turn does without a model: is this an answer, a request for the answer, being
stuck, another try after two, or something else? The rules are scored on 360 messages (300 in English, 30 in Spanish, 30 in
Chinese) across mathematics, science, reading, writing, history and programming, for grades 3 to 12, each with its
conversation. A small picker model (pickerModel) reads the moment instead of the rules where one is set, and isn't scored
here, since scoring it calls a model.
The procedure behind a wrong answer
The procedure behind a wrong answer is worked out by code when the problem is one piece of arithmetic. Each of 66
faulty procedures a teacher of the topic knows is applied to the numbers, and one that gives exactly what the learner
wrote is named (src/diagnose.js). It is scored on 160 wrong answers, 20 of them slips that no
procedure explains. "Named as labelled" compares the first procedure named with the one a second reader matched the
author's words to. "Named as something else" also counts any name given to a slip, or to a procedure the list doesn't
hold. One wrong answer often fits two procedures (-5 for -4 + 9 is the first number's sign kept, or the size right and
the sign wrong), and then a turn's instructions give both. The same code is run on
another library's wrong answers.
A policy in code
A policy in code is a level of help whose steps code decides: a question first, a hint after one try, a worked example after two (policy.md). Code has to read the conversation to do that. On 120 scripted conversations, with the tutor's turns replaced by one neutral line, it is scored on whether the step it decides is the one each turn requires, and on whether it sees that a learner has written the answer. What a model then does with the step takes models to measure, and has its own page.
review
review is the check for someone building a tool for learners (for builders). It is scored on 60
small projects in about thirty stacks: 40 with one to four of its sixteen problems planted, 112 in all, and 20 that do the
same things properly, several with look-alikes such as a public Firebase key.
How the cases were made
- Written apart from the code. Each set was written by an AI model given only definitions: the five moments in the words the picker is told, the sixteen problems in a sentence each, the kinds of wrong answer, and the kinds of turn. It was told not to open this repository. Every message and project is invented, with no learner's data and no real key.
- Split before anything was read. Each set is in two halves, fixed by case number before any result was looked at. The development half is the one that was read while the code was changed. The test half was then run once, and is what to quote.
What to keep in mind when using the numbers
The labels are a model's. A model wrote each message, wrong answer and conversation together with its label, about one message in six was written to be arguable, and no teacher has gone through them. The test half is the number to quote, and it is a number about invented cases.
The rules count whatever a learner says back to a tutor who left them something to do as an answer, unless it is a question, a plea, being stuck or small talk. On the test half's English messages, 30 of each moment:
| Moment | Read as labelled |
|---|---|
| An answer or some working | 29 |
| Asks for the answer | 23 |
| Stuck | 28 |
| Another try after two | 29 |
| Something else | 16 |
So the rules lean toward reading a message as an answer. Of the 30 messages that were small talk or a question about the task, 9 are read as answers, because "we did this in class yesterday" looks like one after a tutor's question. Such a turn gets the move for an answer, and the teacher's limit as always.
Spanish and Chinese rest on 15 test messages each, so their intervals are wide. A multilingual class should set a picker model.
review reads patterns in text. It misses about one planted problem in eight on projects it hasn't seen, and it can't
see whether a budget works or what a teacher would notice, so a clean result still needs a person's look.
A wrong answer is named only when a listed procedure gives it exactly. Three in ten wrong answers in the test half get no name or another one, and the turn then goes on as it would have without a name.
How replies come out on models is measured separately: what a reply gains from the teacher's settings in what has been measured, and whether a tutor holds a teacher's policy in the policy check.
The files
learner-moments.json, classroom-trips.json,
wrong-answers.json and
tutoring-conversations.json, described in
scorecard/README.md, with the runner in src/scorecard.js. Score your own
reader of moments on the same cases: each has message, history, moment and split.
This page comes from the open repository: harness/docs/scorecard.md. To change it, change it there.
