The client situation

The client runs an AI assistant that handles internal IT support requests. Employees ask it things in plain language, and behind the scenes it looks up information, updates support tickets, and answers questions.

The team had plenty of recorded conversations: 136 real support conversations, broken down into 533 individual actions the assistant took along the way. What they didn’t have was any way to tell which of those actions were good ones.

The problem

Grading an AI assistant normally works like grading a school test: someone marks the right answers in advance, and you check the AI against them. Those marked answers are the labeled examples — and here, nobody has any.

Real support requests can’t have the right answers marked down in advance — every ticket, every employee, every situation is different. There is not a single labeled example to check against. The usual testing approach simply doesn’t apply.

Worse, a support assistant can give a plausible final answer that looks fine to a casual reader, while having made sloppy moves along the way — looked things up it didn’t need to, invented a detail, or wandered away from what the employee actually asked.

The question we were hired to answer: can we judge the quality of an AI assistant’s work when nobody — not even the client — has labeled a single example of right and wrong?


Our approach

Step 1 — Grade the process, not just the answer

Instead of checking final answers, we scored every single move the assistant made — each lookup, each ticket update, each reply — against four plain-language quality checks:

What we measured The question we asked
Groundedness Did the assistant stick to the facts in front of it, or did it make things up (names, ticket statuses, dates)?
Efficiency Did it avoid doing the same lookup twice, or making calls it didn’t need?
Adaptability When something failed or came back empty, did it recover sensibly — try a different approach — or just give up?
Goal alignment Was each move actually a step toward what the employee asked for?

These checks are drawn from published academic research on evaluating AI agents, adapted for an assistant that doesn’t show its “thinking out loud” — we judge the actions themselves against what the assistant actually knew at that moment.

Step 2 — Use a panel of independent judges, not one

A single automated grader has blind spots and biases. So every action was scored independently by three different AI judge models, from different vendors. Where at least two of three agreed, we accepted the verdict.

This mattered immediately: on “goal alignment,” the three judges split sharply on roughly a quarter of the actions. Rather than quietly picking a number, we flagged exactly which actions the judges disagreed on and handed that list to human reviewers. A disagreement you can see is far more valuable than a wrong number you can’t.

Step 3 — Test the same conversations under three different setups

The assistant can run on different underlying AI models. We replayed all 136 original conversations, word for word, through a safe non-production copy of the assistant in three configurations:

  • “Smart router” — the system picks a model for each moment (a mix of three models)
  • “Luna” — one fixed, budget-priced model
  • “Terra” — one fixed, speed-focused model

Same conversations, same questions, three different engines — plus the original production run for reference. We measured quality, cost, and response time for each.


What we found

There is no single winner — and that’s the point

Configuration Quality (problem conversations) Cost to run all 136 Typical response time
Original production run Best accuracy & focus $23.31 21 sec
Smart router Best of the three $4.34 11 sec
Luna (budget model) Weakest $0.48 — about 9× cheaper 8.7 sec
Terra (fast model) Similar to Luna $4.44 8.3 sec — fastest

Three clear headlines:

  1. The smart router delivered the best quality — the fewest conversations with made-up facts (11% vs 15% for fixed models) and by far the fewest wasted actions (2% vs up to 7%). Its edge came from exactly what it’s designed to do: when one model started struggling, the router steered the next step to a different one and rescued the conversation.
  2. The budget model is startlingly cheap — the same workload for under fifty cents. If cost is the constraint, this is a serious option, provided its weaker accuracy is acceptable.
  3. The fast model is consistently quickest — and by far the least wasteful of the three fixed-model runs, with almost no repeated or unnecessary lookups.

The trade-offs are real, and now they’re quantified. The client can choose based on what they actually care about, instead of vendor claims.

The full scoreboard

The same comparison in detail — defect rates, failure recovery (judged by panel majority), latency, and cost, configuration by configuration:

Configuration Ungrounded Redundant Tool errors → recovered Misaligned Median turn latency Median step latency Median turn cost
Production (main · gpt-5.2) 4% 20% 21 → 10 recovered / 11 not 21% 21.1s 3.0s $0.1199
Router (head · mixed 5.6) 11% 2% 0 errors 38% 11.0s 4.9s $0.0279
Luna (head · fixed) 15% 7% 7 → 4 recovered / 3 not 48% 8.7s 4.4s $0.0028
Terra (head · fixed) 15% 1% 5 → 1 recovered / 4 not 49% 8.3s 3.4s $0.0277

The numbers behind the headlines: production was the most grounded and best aligned, but slowest and most expensive per turn — and one in five of its actions was redundant. The router never hit a tool error at all. Luna’s median turn cost is two orders of magnitude below production’s.

The judges exposed a specific, fixable defect

The clearest quality problem wasn’t vague “bad AI” — it was concrete: in the replay environment the assistant sometimes addressed the wrong person, replying to the test identity instead of the employee who had actually asked. That’s a specific engineering fix, not a research problem. This is what process-level evaluation buys you: named defects, not vague scores.

Recovery from failure differed sharply

When a lookup failed mid-conversation, the production agent recovered about half the time (10 of 21 failures). In the replayed configurations, the router never hit a failure at all, while the fixed models recovered 4-of-7 (Luna) and 1-of-5 (Terra). How an assistant behaves when things break is invisible in normal testing — and it varied more than almost any other measure.


What we told the client honestly

We deliberately publish the limits of this study, because a case study that hides them is marketing, not measurement:

  • Not a single example was human-labeled, so final answers were not checked against a trusted source. A future strengthening we recommended: have the evaluation independently re-check the assistant’s factual claims against the live systems it claims to have read.
  • Judge disagreement on “goal alignment” is unresolved — the strict and lenient judges differ by ~2× on that metric. Until the flagged actions are human-reviewed, that one number deserves an asterisk.
  • The replay runs are a fair comparison among themselves, but not a perfectly clean comparison against the original production run — the production agent was a different software version handling live multi-turn sessions. We separated those comparisons in the report so no one over-reads them.

The outcome

The client ended up with:

  • A repeatable scoring method that works even without a single labeled example, and can run on any future batch of conversations
  • A quantified, honest comparison of three run-time configurations across quality, cost, and speed
  • A short, concrete defect list (wrong-name replies, failed recoveries) ranked and ready for engineering
  • A human-review worklist of exactly the ~60 actions where automated judges disagreed

You don’t need labeled examples to hold an AI assistant accountable. You need the right questions, independent judges, and the honesty to publish what you still don’t know.


Methodology adapted from published research on evaluating tool-using AI agents: “Beyond the Final Answer” (TRACE, ICML 2026), “What Is Your Agent’s GPA?” (goal–plan–action alignment), and the Agent-as-a-Judge literature on independent verification. Dataset: 136 real support conversations, 533 recorded assistant actions, judged action-by-action by a three-model independent panel.

Want this for your agent?

Tell us what it does and what decision you're facing — or send a batch of production traces. We'll reply within two business days with how we would approach it.

Get in touch