Most “AI agent” demos answer every question with total confidence. That is exactly the behavior you do not want in customer support, because the model is sometimes confidently wrong, and a wrong answer sent under your name costs more than the ten right ones saved. So this walkthrough builds the unglamorous version: a system that triages support tickets, drafts replies for a human, and, most importantly, knows which tickets it should not touch. It is scoped for a small business, it uses off-the-shelf parts, and it is built eval-first, because the evaluation is the part that makes the rest safe to ship.
Scope the job down to something measurable
The failure mode is starting with “replace support.” Start instead with a single, checkable task: read an incoming ticket, assign it one of a fixed set of intents (billing, shipping, returns, technical, other), attach a confidence, and draft a suggested reply. No autonomous sending, no open-ended chat, no promise to resolve everything. This matters because a narrow task has a right answer you can grade, and a system you can grade is a system you can improve. The broad version (“handle support”) has no score, so it never gets better, it just gets defended in meetings.
Build the eval before you build the bot
Write the test set first. Anthropic’s January 2026 guidance on evals is blunt about the order: build the evaluation that defines the capability before the agent can fulfill it, then iterate until it can, and treat owning that eval as routine as maintaining unit tests. Hamel Husain, who has shipped this pattern across dozens of teams, names the inverse as the single most common cause of failed LLM products: no robust evaluation system. The starting size is smaller than you expect. Anthropic suggests 20 to 50 tasks drawn from real failures is a great start. So pull 50 real past tickets, label each by hand with the correct intent and a note on what a good reply contains, and freeze that as your scorecard. This is the step teams skip: LangChain’s 2025 survey found only about 52 percent run offline evaluations even as 89 percent have observability. Skipping it is why so many pilots feel fine in the demo and fall over in production. The eval is not bureaucracy. It is the only thing that tells you whether a change helped.
The model is the cheap part
Now pick a model, and resist the urge to reach for the most powerful one. Triage and drafting are well within the range of small, cheap models. As of June 2026, Claude Haiku 4.5 runs about $1 per million input tokens and $5 per million output; Google’s Gemini 2.5 Flash-Lite is roughly $0.10 and $0.40 for the same. Put that against the work. A triage-and-draft call of, say, 500 input tokens and 150 output tokens (an illustrative size; yours will vary) costs on the order of $0.0001 to $0.0013 per ticket depending on the model. A human-handled contact runs around $6 to $7 on a common global baseline. The token cost is three to four orders of magnitude smaller, which means the economics are not decided by which model you pick. They are decided by how often the system is wrong and how expensive each wrong answer is. That reframes the whole build: optimize for correct escalation, not for cents per call.
Teach it when to escalate
Here is the number that should set your expectations. On real deployed support data, not the clean synthetic benchmarks where models score in the high nineties, the best LLMs reached about 0.74 F1 on intent classification (Arora, Jain and Merugu, EMNLP 2024). A system that acts on every prediction is therefore confidently wrong on a meaningful slice of tickets. The fix is not a better model; it is a confidence threshold. The same paper shows that an uncertainty-based router, auto-handling the confident predictions and escalating the rest to a human, recovers accuracy within about 2 percent of the full model at half the latency. So the design is a gate: above the threshold, the draft goes to the agent’s queue ready to send; below it, the ticket is routed to a human with the model’s best guess attached as a starting point, never as an answer. One caveat that matters in practice: a model’s raw confidence score is not the same as its accuracy, so do not trust the number at face value. Tune the threshold against your frozen eval set, watching the accuracy of what you auto-handle, until the auto-handled slice is reliably good and the escalated slice is where the genuinely hard tickets land.
What “good” actually looks like
Be honest about the ceiling, because the vendors will not be. Marketing pages advertise “up to 65 percent” end-to-end resolution; those are self-reported, usually a single customer’s best case, and they are not what you should plan around. The defensible, independent number is quieter and more useful: a large field study of over 5,000 support agents found AI assistance raised issues resolved per hour by about 14 percent on average, and 34 percent for the newest agents, with no drop in quality (Brynjolfsson, Li and Raymond, NBER). That is the realistic first-year shape of this: not a support team replaced, but a support team that clears the easy third of its queue faster and spends its attention on the hard two-thirds. Grade the live system against the same eval you built in step two: track how much it correctly auto-handles, how clean those drafts are, and, the metric that actually protects you, how often it escalates something it should have escalated. A high escalation rate is not failure. Sending a wrong answer is.
Ship the third you can prove
The version that ships does not deflect 65 percent of tickets on day one. It correctly handles the easy third, escalates the rest with a draft already written, and never makes the company look careless in front of a customer. That is a genuinely useful system, and it is small enough to build in a couple of weeks with bought parts. The bigger numbers come later, earned by feeding real escalations back into the eval and lifting the threshold only when the score says you can. Build the eval, ship the third you can prove, and let the system grow into the rest. The order is the whole trick.