The agent reports that it fixed the bug. The tests were never run. Nobody notices until a customer does.
This is the most expensive class of agent failure, and it is not a reasoning failure. The model did work, formed a conclusion, and reported it. What was missing was anything that checked.
In the harness model this is primitive 10 — verification and observability — and its one-line description is the whole idea: the harness asks for receipts. And it records the run.
Two jobs in one layer. Verification catches the failure. Observability tells you why.
Self-report is not evidence
A language model asked whether it completed a task will answer. That answer correlates with success, and the correlation is nowhere near strong enough to build on.
The fix is unglamorous: the harness, not the model, decides whether work succeeded. The agent claims; something external confirms.
What counts as a receipt depends on the work:
- Code changed → the test suite ran and passed, the types check
- UI changed → a screenshot rendered, and something looked at it
- Data transformed → row counts reconcile, constraints hold
- A document produced → an eval scored it against a rubric
The pattern is the same in each case. There is an assertion, and there is an artifact, and the artifact is produced by something that has no stake in the answer.
An agent that cannot produce the artifact has not failed the task. It has failed to demonstrate the task, which the harness should treat identically — otherwise you are back to trusting the self-report, one step removed.
The record is the other half
Verification tells you that something failed. It rarely tells you why, and "why" is what determines whether you fix a prompt, a retrieval rule, a tool schema or a permission boundary.
The minimum useful run record:
- What went in — the request, and the fully assembled context, not a summary of it
- Every tool call, with arguments and returned result
- What the model concluded at each step
- What came out
That is enough to answer the question you will be asked constantly: why did it do that? Without it, the honest answer is a shrug, and debugging becomes re-running the thing and hoping it misbehaves again.
Why deferring it is more expensive than it looks
Almost every team defers this layer. The reasoning is reasonable — it produces no user-visible feature, and there is always something that does.
The cost is structural rather than proportional. Retrofitting observability means changing every other component to emit something: the context assembler has to log what it assembled, the tool layer has to log calls, the orchestrator has to correlate them into a run. Added at the start, that is a decision. Added in month six, it is a refactor touching everything, performed under the pressure of an incident nobody can explain.
The asymmetry is the argument. Building it early costs a little and you might not need it. Building it late costs a lot and you definitely do.
It is also where improvement comes from
There is a second-order reason this layer matters more than its description suggests.
Primitive 11 in the harness model is evolution — failures become infrastructure. A missed context becomes a retrieval rule. That loop is only available if failures are legible. Without run records you cannot see patterns, only individual incidents, and you fix each one by hand forever.
With them, the same failure appearing three times stops being three bugs and becomes one missing rule. That is the difference between a system that gets better and a system that gets maintained.
The smallest version worth having
If you have nothing today, you do not need a platform. Start with:
- One verification gate on the riskiest action. Usually: no change is accepted unless the test suite ran.
- Structured run logs to a file. Inputs, tool calls, outputs. Not a dashboard. A file you can grep is worth more than a dashboard you have not built.
- A weekly read of three failed runs. Not automated analysis — a person, reading. The patterns show up faster than anyone expects.
That is a day of work and it changes what every subsequent debugging session costs.
The full specification covers this layer alongside the other ten, with a symptom-to-layer diagnostic for working backwards from a failure to the thing that caused it.
Common questions
- What does verification mean for an AI agent?
- Requiring evidence that work actually succeeded rather than accepting the agent's own report — tests that pass, types that check, a screenshot that renders, an eval that scores. The agent asserts; the harness confirms.
- Why is observability expensive to add later?
- Because retrofitting it means changing every other component to emit something. Added at the start it costs a decision; added in month six it costs a refactor across the whole system.
- What is the minimum useful run record?
- Inputs, every tool call with its arguments and result, the assembled context, and the final output. That is enough to answer "why did it do that?" which is the question you will be asked most.
- AI
- Agents
- Observability
