Two things were true across 2025 and into 2026. Models got dramatically more capable. And practitioners at QCon London cited reports that as many as 80% of firms saw no tangible benefit from their AI initiatives.
Both cannot be explained by the same cause. If capability were the constraint, rising capability would have closed the gap. It did not, which means the constraint was somewhere else.
It was. The teams in the successful minority were not using better models than everyone else. They were building four things first, none of which demo well.
1. An evaluation set, before the launch rather than after
The common failure is shipping on spot checks. Someone tries twelve prompts, the output looks good, it goes out. Quality regressions are then discovered in the field, by customers, one at a time.
The teams that succeed invert this. They build the evaluation pipeline before the production deployment, not after, which sounds obvious and almost nobody does, because an eval set is work with no visible output.
What makes it worth it is not the score. It is that you can now answer "did that change help?" Without evals, every prompt tweak, model swap and retrieval adjustment is a guess dressed as an improvement, and teams spend months making confident changes with no idea of their direction.
Two kinds are needed. Automated evals chew through thousands of examples per hour, which is what gives you velocity — but they only catch failure modes already represented in the set. Human review catches the novel ones. Teams that run only the first are fast and blind; teams that run only the second are careful and slow.
2. A context strategy, not a retrieval pipeline
Most teams treat "give the model the right information" as a retrieval problem, solve it with a vector database, and stop.
Retrieval is one input. Context engineering is the broader practice of deciding what the full context the model sees actually contains — and a team can run a state-of-the-art RAG pipeline and still ship something that fails because it has no memory layer, an overloaded tool set, or bad token-budget hygiene.
The distinction that matters in practice is between getting material in and deciding what earns the room. Those are different jobs. In the harness model they are separate primitives — context delivery and context management — precisely because teams that conflate them build good retrieval and then watch quality fall as the window fills.
3. Permissions at the retrieval layer, not the application layer
This one is a security incident waiting to be scheduled.
The natural implementation indexes everything and filters at the application layer. It is easier, and it works in testing. Then similarity matches start returning chunks the user should not see, because a filter applied after retrieval is a filter the retrieval already ignored.
Chunk-level access control is significantly more annoying to build and is not optional for anything touching customer data. Retrofitting it means re-indexing, which is why it belongs in the first version.
The same logic applies to what the agent can do. An execution environment with unbounded filesystem and network scope is fine until the first mistaken command, at which point it is the only thing anyone wants to talk about.
4. Observability, before you need it
Verification and observability is the layer teams defer most often and pay for most heavily.
The reason it is expensive late is structural: retrofitting observability means changing every other component to emit something. Adding it at the start costs a decision. Adding it in month six costs a refactor.
What you want is boring. Every run recorded: what went in, what tools were called, what came back, what the model concluded. Without it, "the agent did something weird yesterday" is unanswerable — and it will be asked constantly.
The pattern
None of the four is a model decision. All four are infrastructure, all four are invisible in a demo, and all four are dramatically cheaper before launch than after.
Which is the actual explanation for the 80%. The work that separates a demo from a product was never the model. It was the system around it, and that system does not build itself while you wait for the next release.
If you have an AI feature that works in a demo and not in production, the diagnostic in the harness piece is a faster route to the cause than another model upgrade.
Common questions
- Why do most AI initiatives fail to show benefit?
- Because the work that makes an agent reliable sits outside the model — evaluation, context, permissions and observability. Teams that buy model access and skip that work get demos that do not survive contact with real inputs.
- Does a better model fix a failing AI feature?
- Rarely. If the failure is a missing evaluation loop, an overloaded context or an unbounded execution environment, a stronger model fails the same way slightly later. Upgrade the model after you can measure the effect.
- What should a team build before shipping an agent?
- An evaluation set with real failure cases, a defined context strategy, a bounded execution environment, and run-level observability. All four are cheaper before launch than after.
- AI
- Agents
- Strategy
