The Queue That Made Automation Worth Testing
The support operation handled roughly 18,000 tickets a month for a business software platform used by small retailers. Volume had risen 42 per cent in a year, but the service team had added only four people, taking total headcount to 31. Median first-response time had slipped from 46 minutes to three hours and 12 minutes. On Mondays, when failed weekend integrations and billing queries arrived together, more than 2,000 tickets could be waiting by midday.
The queue was not uniformly difficult. A six-week sample showed that 57 per cent of tickets fell into eight recurring categories: password resets, invoice requests, subscription changes, data-export questions, integration status checks, duplicate charges, user-permission changes and basic troubleshooting. Agents often spent four or five minutes finding an account, checking policy and composing a response for work that required little judgement. Meanwhile, complex cases involving lost transactions or disputed payments waited beside them.
The goal was therefore narrower than “replace support with AI”. The project team set three measurable targets for a 12-week pilot: reduce median first-response time below one hour, cut manual handling time by 25 per cent and keep incorrect or policy-breaking responses below 1 per cent. The system would begin with low-risk work, operate inside the existing helpdesk and escalate whenever evidence was incomplete. That framing mattered. It treated automation as queue redesign rather than a demonstration of conversational fluency.
From Chatbot to Agentic Workflow
The first prototype looked like a conventional chatbot. It classified a ticket, searched the knowledge base and drafted a reply for an employee to approve. This improved writing speed, but saved less than expected: an average of 74 seconds per ticket. Agents still had to open the customer record, verify subscription status, inspect recent events and perform any required action. The language model could explain a refund policy, but it could not determine whether a particular payment qualified or initiate the refund.
The second version was agentic because it could plan and execute a limited sequence of tool calls. For an invoice request, it identified the customer, matched the account, retrieved the correct invoice, checked that the requester had billing permission, attached the document and sent a templated response. For an integration-status query, it checked service health, inspected the customer’s connection logs and either provided a known fix or routed the case to a specialist with the diagnostic evidence attached.
That capability was deliberately constrained. The agent could read from 14 internal systems but write to only four. It could reset a test connection, update ticket fields, send approved messages and issue account credits up to £20 under explicit rules. It could not cancel contracts, alter bank details, erase data or refund card payments. Every action carried a structured record containing the customer identifier, retrieved evidence, policy version, tool output and final response. Autonomy was granted task by task, not as a blanket permission.
The Prototype Failed in Useful Ways
Early testing exposed errors that polished demos had concealed. In one case, the agent saw the phrase “charged twice” and prepared a duplicate-payment credit. The account actually contained two valid subscriptions for different shop locations. In another, a user asked to “remove James”, meaning revoke a former employee’s access; the system interpreted the request as deleting the user’s historical activity. Neither action was completed because write permissions were disabled, but both failures showed that correct language classification did not guarantee correct operational intent.
Knowledge retrieval created a second problem. The company had three articles describing subscription pauses, each written at a different time. The model combined elements from all three and produced a plausible but invalid policy. The remedy was not a better prompt. Content owners archived obsolete documents, added effective dates and assigned one authoritative source to each policy. Retrieval was also changed to favour approved procedures over semantically similar forum posts. The resulting answer set became smaller but more reliable.
The team then built a test suite of 1,240 historical tickets, including ambiguous requests, hostile phrasing, missing account details and attempts to manipulate the agent. Performance was scored at the level of individual decisions: identity match, intent, evidence selection, policy application, tool choice and customer message. A response counted as failed if any safety-critical step was wrong, even when the prose was excellent. On the first run, end-to-end accuracy was 71 per cent. After six iterations, it reached 93 per cent for the restricted pilot categories, still insufficient for unsupervised handling of high-impact cases.
Choosing What the Agent Could Resolve
The rollout used a risk matrix rather than a simple confidence score. Tasks were ranked by reversibility, financial impact, privacy sensitivity and ambiguity. Sending a previously issued invoice was highly reversible and low impact; changing an account owner was neither. The agent could close a case automatically only when the intent matched an approved category, required evidence was present, every tool call succeeded and the estimated risk stayed below a fixed threshold. Confidence alone never overruled a prohibited action.
Three operating modes emerged. About 34 per cent of incoming tickets qualified for autonomous resolution, chiefly document requests, known-error troubleshooting and service-status questions. Another 41 per cent received an AI-prepared response and action plan for human approval. The remaining 25 per cent went directly to people because they involved complaints, cancellation threats, security concerns, vulnerable customers or unclear identity. These proportions shifted over time, but the categories were more important than the headline automation rate.
A practical example was duplicate billing. The agent could automatically explain two pending authorisations, because these normally disappear without intervention and no money had settled. It could prepare a refund recommendation when two settled charges had the same amount, card token and subscription identifier within five minutes. Yet a human had to approve the refund and contact the customer if chargeback language appeared. One label covered several materially different situations; the workflow reflected those differences instead of pretending all billing queries were equivalent.
What Changed During Rollout
The pilot began with five agents and 10 per cent of eligible traffic. For two weeks, every autonomous decision ran in shadow mode: the system completed its analysis and proposed actions, but a person handled the live ticket. Reviewers found that the agent was strong during normal operating conditions and weak around exceptions, particularly regional tax rules and accounts migrating from an older pricing plan. Those cases were excluded until the necessary policy data could be exposed through tools.
Traffic increased in stages to 25, 50 and finally 100 per cent of eligible tickets. Each stage required at least 500 reviewed cases, a critical-error rate below 0.5 per cent and no unresolved security incident. The team also monitored recontact within seven days. A fast answer was not considered successful if the customer returned because the underlying issue remained. During the first month, autonomous responses had a recontact rate of 11.8 per cent, compared with 8.6 per cent for human-handled cases. Adding account-specific checks and clearer next steps reduced it to 7.9 per cent.
Operational ownership proved as important as model selection. Support managers defined policy and escalation rules; engineers maintained tools and observability; security reviewed permissions; legal approved data handling; and a three-person quality group sampled 150 automated cases each week. When a policy changed, its machine-readable rule and customer-facing article had to be updated together. Without that discipline, the agent would have reproduced organisational inconsistency at machine speed.
The Gains Were Real, but Uneven
After four months, median first-response time had fallen from three hours and 12 minutes to 38 minutes. Average manual handling time dropped 29 per cent, from 9.4 to 6.7 minutes, and the backlog older than 24 hours shrank by 61 per cent. The system autonomously resolved 36 per cent of tickets and prepared usable drafts for a further 39 per cent. The remaining suggestions were discarded or substantially rewritten, most often because the customer’s real concern was emotional rather than procedural.
Savings varied sharply by category. Invoice retrieval fell from 6.2 minutes of staff time to 42 seconds of automated processing. Integration troubleshooting improved by only 18 per cent because logs often revealed several possible causes. Cancellation cases became slightly slower: agents had to review an AI summary as well as read enough of the original conversation to understand tone and commercial context. The team removed automated drafting from some of these cases after discovering that superficially helpful retention language could frustrate customers who had already explained a serious service failure.
The financial case remained positive. Including model usage, engineering support, monitoring and quality review, the system cost about £1.18 per automated resolution. The estimated fully loaded cost of a manual resolution was £6.40. However, multiplying that difference by every automated ticket would have overstated the benefit: some demand would have been absorbed through self-service, and fixed staffing costs did not disappear immediately. The credible annual saving was calculated at £410,000, around 46 per cent lower than the original business-case estimate.
Why Human Support Became More Important
Automation removed repetitive work but concentrated complexity. Agents handled fewer password and invoice requests, yet saw more payment disputes, migration failures and angry customers. Average cognitive load rose even as ticket volume per person fell. Team leaders responded by changing schedules, limiting consecutive escalations and introducing specialist rotations. The role moved from rapid transaction processing towards investigation, negotiation and exception management.
Human reviewers also became teachers of the system, though not through indiscriminate feedback. A thumbs-down button had produced noisy signals: employees disliked some correct responses because of tone, while accepting polished drafts that contained subtle policy errors. The quality team instead tagged failures against a controlled taxonomy, such as missing evidence, wrong account, outdated policy, unsafe action or poor empathy. This allowed engineers and content owners to fix the relevant layer rather than continually adjusting the model prompt.
The strongest human contribution appeared where policy met circumstance. One customer requested a refund after a shop fire had halted trading. The account did not meet standard refund criteria, and an automated system could only reject or escalate. A support specialist recognised the exceptional context, arranged a three-month credit and coordinated data access for the customer’s insurer. The case took 37 minutes and could not be justified by efficiency alone. It protected a long-standing relationship and illustrated why service quality cannot be reduced to average handling time.
The Durable Lessons for Agent Deployment
The first lesson is to automate a bounded workflow, not an entire job. Reliable agents need explicit tools, authoritative data, narrow permissions and visible stop conditions. They also need evaluation against real operational outcomes: correct action, policy compliance, recontact and customer harm. Measuring whether a response sounds convincing is dangerously inadequate when the system can change records or move money.
The second lesson is that exception handling defines the product. Teams should expect ambiguous identities, contradictory policies, failed tool calls and customers whose situation does not fit the standard process. Escalation must preserve context rather than forcing people to reconstruct the case. In this rollout, pre-populating the ticket with evidence, attempted actions and the reason for escalation saved human agents an average of 2.1 minutes, even when the agent could not solve the problem itself.
The final lesson is organisational. Agentic automation exposes weak documentation, unclear authority and inconsistent service rules. That can feel like a technology failure, but it is often an operational diagnosis. The successful rollout did not come from maximising autonomy. It came from deciding precisely where machines were dependable, making their work auditable and preserving human discretion where consequences, ambiguity or empathy demanded it. The result was a faster queue without the fiction that every customer problem is merely a task waiting to be automated.
Comments (0)
Discussion is opening soon. Be the first to comment.