OhMyVet is a prototype we are building that reads veterinary lab reports. From a case's documents it proposes a ranked list of possible diagnoses, a differential, and cites every item to the line of the report it rests on. It is research software under construction. It is not for use on real patients and not a substitute for a veterinarian's judgement.
Until this month it had a critic loop. After the model wrote its list, two critic models from other providers than the main model read it and raised objections, and a third call rewrote the list to answer them. On 1 November we deleted it. It did not break anything. A rule we had written down in August said to delete it if two numbers did not move, and they did not move.
The rule every component has to pass
OhMyVet has a build rule, R13: a component that does not beat the baseline is deleted. That only works if the baseline is a recorded number, so we recorded one before building anything on top: one model call per case, no tools, no critic. Every model reply is saved as a cassette, so any run replays later for $0 and gives the same numbers.
Each new part is then measured against that baseline on the same cases, with the same scorers. The graded set is 30 test cases that no veterinarian has reviewed yet. Because of that, we do not publish OhMyVet's diagnosis accuracy. Any figure scored against unreviewed cases would be a draft, and a draft on a public page reads as a claim. What we can publish is the comparison: what the critic changed, against the same cases without it.
What the critic loop did
Each round, two critics read the same ranked list (docs/DECISIONS.md, T-E3-4(a) and (b)):
- Every objection had to quote its evidence. The subject of an objection had to appear word for word in the case record or the list, or it was refused. An objection nobody can point at is worse than a bad citation, because it gets acted on: the reviser would rewrite a list to answer a finding that does not exist.
- Objections were pooled, then the whole list was rewritten. If either critic raised anything, a reviser wrote the list again.
- At most two rounds. Then the answer went out.
It was built to do two jobs. Catch contradictions the main model missed, and close a citation gap: 150 of 151 diagnoses in the baseline resolved to a line in the record, and the gate asks for all of them.
The first reading was partial, so we wrote the rule
The first measurement ran on 28 August and stopped early. Recording halted on a 402 Payment Required from OpenRouter, with 22 of the 30 cases and 6 of the 12 planted errors recorded
(docs/BASELINE-DELTAS.md, T-E3-4(c)).
On those 22 cases, not one changed result on either of the two diagnosis checks we track, the top answer and the top five. Cost per case went from $0.035778 to $0.157050.
That looked like enough to delete it. We did not, for one reason: 8 cases had never run. Deleting a component on evidence that could not have detected its effect is the same mistake as shipping it on that evidence. So before the missing data existed, we wrote the decision down:
Delete the arm if
dx_top5_hithas not overtaken the baseline andcitation_resolution_ratehas not closed to 1.00. Keep it if either has moved.
We priced the rest of the run at $2.60. Writing the rule first matters because the alternative is choosing a rule after seeing the numbers, which means choosing whichever one reads better on the day.
The full run
The remaining 16 recordings ran on 1 November and cost $2.623550 (docs/BASELINE-DELTAS.md,
T-E3-4(d); commit 2fbcd5f).
| Measure | Without critics | With critics |
|---|---|---|
| Diagnosis checks, 30 cases | Not published | No case changed result |
| Citations resolved | 150/151 = 0.9934 | 152/154 = 0.9870 |
| Cost per case | $0.035577 | $0.148067 (+316.2%) |
| Planted errors caught | No critic to catch them | 7/12 = 0.58, floor 0.8 |
Neither condition in the rule moved. The diagnosis checks stayed where they were, and the citation
rate went down instead of up. The arm was deleted in the next commit (b07b4bc).
What the critics actually did
The per-case records show why it did not help.
They rarely agreed. Of 60 objections, 4 were raised by both critics. Two critics bought more coverage, not confirmation.
They missed planted errors that should have been easy. We keep 20 deliberately broken copies of
test cases, each with one defect written in by a generator, and every make check confirms the
defect is still there. Twelve of them carry the kind of error a critic is there to catch. In the
first reading, one copy had a glucose value raised to three times the
top of its printed range. Another had band neutrophils at 900 against a range of 0–300. Both went
past two critics without a comment.
They made citations slightly worse. In the first reading, the one diagnosis the baseline could not cite was still uncitable after review, and the rewrite added a second one.
Cost alone was not the argument. $0.148 a case is still under the $0.50 per-case ceiling the project set. Four times the price for no measured change is the argument.
What we deleted and what we kept
Deleted: three modules, their prompts, the eval target, five trend scorers, the recording script and 313 recorded replies. All of it can be restored from the commit before the deletion.
Kept: the planted-error probe and its 20 broken cases. They are not part of the product. They tell us whether the test suite still notices a defect, and the next component that claims to catch errors will be measured against them.
The broken cases also run in a tier of their own, outside the main test set. Adding them to the main set would have changed every figure computed over it, and no test would have failed. The numbers would just have been different, with the same sentences still describing them.
What this does not show
This result covers one design: two critics reviewing a list one model wrote. It does not tell us whether several models each writing their own list, then merging them, does better. That is the next component on OhMyVet's roadmap, and it gets the same treatment: a recorded baseline, a rule written before the data, and a delete if it does not move.
For a team adding an AI review step
Record a baseline first. Without a number from before the reviewer, there is nothing to compare it with.
Write the delete rule before the data. Name the metrics that have to move and by how much.
Do not decide on a partial run. Price the rest and finish it. Ours cost $2.62.
Make the reviewer quote its evidence. An objection that cannot point at a line becomes a rewrite nobody asked for.
Delete the component, keep the instrument. The probe outlives the critic.
We wrote about the wider version of this in Harness Engineering and Making an Agent Show Receipts.
Common questions
- Why delete a component that did not make results worse?
- It cost four times as much per case and changed no diagnosis result over 30 test cases. Citations got slightly worse, and it caught 7 of 12 planted errors against a floor of 0.8. A part that only adds cost and latency still has to be maintained and kept working by every later change.
- Why not delete the critic loop after the first reading?
- The first reading covered 22 of 30 cases, because recording stopped when our API credit ran out. A partial run can miss a real effect. So we wrote the delete rule first, priced the rest of the run at $2.60, and decided only on the full data.
- Does this mean AI critic loops do not work?
- It means this one did not, here: two critics reviewing one model's ranked list. It says nothing about several models each writing their own list and merging them. OhMyVet tests that next as a separate component, measured against the same baseline.
- AI
- Evals
- Agents
