For two years, the agent demo followed one script. A model is handed a goal, it narrates a plan, it clicks through a few steps, and the room applauds. Then the lights come up and nobody ships it. That gap closed in 2025, and not mainly because the models got smarter, though they did. It closed because the unglamorous layer around the model finally caught up: retries, memory, tool contracts, evaluation, and the discipline of scoping a task narrowly enough to be reliable.
The demo lied about the hard part
The hard part of an agent was never the reasoning. It was everything that happens when the reasoning is wrong, which is often. There is now a clean number for this. The tau-bench benchmark, introduced in 2024 to test agents on realistic tool-and-user tasks, found that even a state-of-the-art function-calling model, GPT-4o, succeeded on fewer than half the tasks, and when its consistency was measured, whether it could pass the same task on all eight tries, its success rate in the retail domain fell below 25 percent. Read that twice. The capability per attempt was decent; the reliability across attempts was not. That single result is the whole reason the scaffolding exists. A production agent is mostly an error-handling and consistency system that occasionally calls a model. The model is the cheap part. The expensive part is deciding what to do when a tool returns garbage, when a step half-succeeds, or when the goal was ambiguous to begin with.
What actually changed
The teams who got past the demo internalized a distinction Anthropic drew in its widely-read guide: most useful systems are workflows, where the model and tools move along predefined paths, not open-ended agents that direct themselves, and you should reach for the agent only when the steps genuinely cannot be predicted. From there, three shifts did the work. Narrow scope beat general autonomy: the things in production own one workflow, like triage a ticket or reconcile an invoice, with sharp edges and a human gate. Tools became contracts, not suggestions; the same guide argues the agent-computer interface deserves as much design care as a human-computer interface, so teams constrained and validated every call. And evaluation became infrastructure. LangChain’s late-2025 survey of more than 1,300 practitioners found 57 percent now run agents in production, 89 percent have some observability in place, and about half run offline evaluations against a test set on every change, the way the rest of us run unit tests.
The teams shipping agents reliably stopped asking “can the model do this?” and started asking “what happens the third time it gets it wrong?”
The reckoning that proves the point
The other half of the evidence is the failure rate, and it is steep. Gartner predicts that more than 40 percent of agentic-AI projects will be cancelled by the end of 2027, citing rising costs, unclear value, and weak risk controls, and it estimates that of the thousands of vendors claiming agentic products, only around 130 actually offer the capability, the rest being “agent washing.” That figure is a forecast off an early-2025 poll, so read it as a directional warning rather than a measurement. The same gap shows in the business numbers: an MIT study of enterprise generative AI found that about 95 percent of pilots delivered no measurable bottom-line impact, with the divide driven not by model quality but by whether the tool was built to fit a real workflow. The projects that die mostly skipped the boring layer. The ones that live treated reliability as the product.
The pattern that ships
The shape that survives looks less like a robot and more like a well-supervised intern. It does a bounded job, it shows its work, it escalates when unsure, and it is measured constantly. None of that is exciting in a demo. All of it is why the thing is still running ninety days later. The encouraging part, for a small team, is that none of it requires the frontier model or a research budget. It requires a narrow problem, a tight tool contract, and an eval set you trust. The plumbing is the product now, and plumbing is learnable.