Support Triage Agent
A support agent that sorts tickets, drafts replies citing the help center, verifies every citation in code, and hands risky tickets to a person.
- Agent Lab · sample data for Parcelly
- Design & Engineering
- Agent Lab · 2026
Overview
Most support bots either answer everything, including what they shouldn't, or cite sources nobody checks. This agent classifies each ticket, retrieves passages from a 14-article help center, and drafts a reply in the customer's language where every factual sentence points at a source. Code verifies the citations before anyone sees the draft, and code, not the model, decides when a person must take over: large refunds, legal threats, security reports, or low confidence. An eval suite measures classification, escalation, citation validity and groundedness on a golden set.
The Problem
A support team wants help with the repetitive 70% of tickets without an AI promising refunds it can't give or inventing policy. The hard part is not generating a reply; it is knowing the reply is grounded and knowing when not to reply at all.
- ▸Citations from a language model can point at passages that don't say what the sentence claims.
- ▸Refund requests, legal threats and account-security reports must always reach a person.
- ▸Without measurement, nobody can tell whether a prompt change made things better or worse.
The Solution
A pipeline that separates what the model does well from what must be guaranteed. The model classifies, searches and drafts; code checks citations, scores confidence and applies the escalation rules, and an eval harness scores the whole thing against expected outcomes.
- ✓Heading-aware chunking and embeddings over 64 help-center passages, with a keyword boost for exact product terms.
- ✓Every number, price, time frame or app location in a draft must cite a retrieved passage; failing drafts are repaired once.
- ✓Escalation rules in code: refunds over $100, legal threats, security reports, confidence below 0.6, with a keyword backstop.
- ✓Evals: category accuracy, escalation precision and recall, citation validity, and an LLM judge for groundedness.
Architecture
Classify, retrieve, draft, verify, decide: each stage typed and testable, with a helpdesk adapter at the edge.
Knowledge index
Markdown help center chunked by heading, embedded with Gemini, committed as a small vector index so the demo needs no ingest.
Triage pipeline
Structured classification, top-5 retrieval, drafting in the customer's language, and one citation repair. Two tickets in parallel; one failure never stops the inbox.
Rules in code
Citation checks, confidence caps and escalation rules as pure functions with boundary tests at exactly $100 and 0.6.
Evals & helpdesk
A golden set scored on every change, and a helpdesk adapter (a mock of Zendesk or Help Scout) where escalated drafts can't be sent.
Key Features
Verified citations
Click a citation to see the exact help-center passage behind the sentence. Uncited claims fail validation.
Escalation you can trust
Refunds over $100, legal threats and security reports always go to the right team, with the reason shown.
Replies in the customer's language
A Spanish ticket gets a Spanish reply, still grounded in the English help center.
Eval report built in
Scores for every ticket on classification, escalation, citations and groundedness, in the product itself.
Technology Stack
Design & Engineering
Designed the agent, its rules and its evals on the shared Agent Lab runtime, and directed the build.
- ▪Specified the citation contract and the escalation policy the agent must never break.
- ▪Wrote a realistic help center and inbox covering refunds, legal, security, another language and an unanswerable question.
- ▪Built the eval harness and the in-product report so quality is measured, not assumed.
- ▪Reviewed the pipeline and interface against the spec before release.
Product UI
Technical Challenges
Citations that actually support the claim
Challenge: Models cite sources loosely, and a citation marker proves nothing on its own.
Solution: Code requires a marker on every sentence with a number, price, time frame or app location, checks each marker against the retrieved set, and an LLM judge scores groundedness in evals.
Knowing when not to answer
Challenge: Model confidence is poorly calibrated, and similarity scores for covered and uncovered questions overlap.
Solution: Confidence is capped in code when citations fail or retrieval is weak, and hard rules for refunds, legal and security override everything.
Shared rate limits
Challenge: Parallel tickets hit per-minute quotas and failed mid-inbox.
Solution: The model chain waits as long as the provider asks, and a failing ticket is reported without stopping the rest.
Outcomes & Highlights
On the golden set, every ticket was classified correctly, every escalation was caught with no false alarms, and every draft's citations held up.
- ✓Category accuracy 12 of 12; escalation precision and recall 100%.
- ✓Citations valid on 12 of 12 drafts; groundedness 1.00 over 46 cited claims.
- ✓The judge was checked with a planted false claim and marked it unsupported.
- ✓Inbox of 12 tickets triaged in about 20 seconds.
- 12/12
- Tickets classified correctly
- 100%
- Escalation precision & recall
- 1.00
- Groundedness
Golden set
Refund, legal, security, low confidence
46 cited claims judged
