Where agents actually run today
Across the organisations we surveyed, production agent deployments cluster tightly around internal operations. Invoice reconciliation, support ticket triage, procurement compliance checks and data quality remediation account for the large majority of running systems.
These tasks share a profile: repetitive, high volume, well-specified, and expensive in human hours. Crucially, they also fail safely. A mis-triaged ticket is recoverable; a mis-sent customer communication is not.
Customer-facing autonomous agents remain a small minority of deployments, and most of those operate with a human approving any outbound action.
The evaluation divide
The single strongest predictor of a successful deployment is whether the team maintains a private evaluation set drawn from their own traffic. Teams with one reported roughly half the production incident rate of teams relying on vendor benchmarks and spot checks.
The evaluation sets that work are unglamorous: one hundred to three hundred real tasks with known-correct outcomes, scored automatically on every change, with a weekly human review of a random sample.
Teams without this discipline consistently describe the same trajectory: strong pilot, gradual quality erosion, loss of stakeholder confidence, quiet decommissioning around month five.
Escalation is a feature, not a failure
Deployments that survive past six months almost universally include an explicit, low-friction escalation path. When the agent is uncertain, handing off is treated as a correct outcome rather than a miss.
This has a measurable cultural effect. Operations staff who trust that the system will ask for help engage with it as a colleague rather than policing it as a threat, and adoption metrics reflect that within weeks.
Teams that penalised escalation in their internal metrics saw the predictable result: agents that pressed forward confidently into situations they should have deferred.
Measuring what finance will accept
Token consumption is an engineering metric. Cost per resolved task is a business metric, and it is the only one that survives contact with a finance review.
The organisations reporting the clearest return calculate a fully loaded baseline — human minutes, error rate, rework cost — before deployment, then compare against the same measure afterwards. Those without a baseline struggle to defend renewal budgets even when the system plainly works.
Median reported cost per resolved task in reconciliation workflows fell by roughly a third year on year, driven more by architectural improvement than by model pricing.
What breaks in year two
The most common second-year problem is drift in the surrounding systems rather than in the model. A schema changes, a form gains a field, an upstream team renames a status value, and agent accuracy degrades quietly.
Contract testing against the tools an agent uses catches most of this. It is standard practice in distributed systems and remains surprisingly rare in agent deployments.
Outlook
We expect back-office autonomy to keep compounding through next year while customer-facing deployments stay cautious and heavily supervised. The reliability required for the latter is a different order of magnitude.
The organisations that will move fastest are the ones investing now in evaluation infrastructure rather than in additional pilots.
Comments (0)
Discussion is opening soon. Be the first to comment.