Scoring

After each call, a judge model reviews it against the scenario's success criteria. Duet keeps the judge on a short leash: it picks a band from 1 to 5 for each criterion, and fixed rules turn those bands into points, verdicts and an overall score. That keeps scores stable from one judgement to the next.

Criteria, weights and must criteria#

Each scenario has between 1 and 30 criteria. Every criterion has:

• Text: one thing a good call does, like “The agent verifies identity before acting.”
• Kind: Must (essential: if it fails, the whole call counts as failed, whatever the score) or Should (it matters, but only through its weight).
• Evidence source: what the judge looks at. Transcript for what was said, Timing for how fast the agent answered, Trace for what happened inside the agent (tool calls, models, errors), which needs a Langfuse connection on the agent.
• Weight: how much it counts. The weights of a scenario must add up to exactly 100.

Bands, points and verdicts#

For every criterion the judge picks a band. The band decides the points and the verdict:

A criterion the judge could not check gets Not observable instead, with no points (see below).

The overall score#

The overall score, from 0 to 100, is the weighted average of the criteria that could be observed. Criteria that were not observable are left out and the others are re-weighted, so a missing trace neither helps nor hurts.

In the example above: (50 × 50 + 100 × 30 + 70 × 20) ÷ 100 = 69. And because the first criterion is a must criterion that failed, the report also shows Must criterion failed. If the second criterion had not been observable, the score would be (50 × 50 + 70 × 20) ÷ 70 ≈ 56.

Failure tags and caps#

When a criterion is not perfect, the judge labels what went wrong with up to 5 failure tags from a fixed list. The report counts them across the call, so repeated problems stand out.

When a criterion is not observable#

Three samples, one median#

Language models can answer differently each time. So the judge looks at every call three times, on exactly the same evidence, and each criterion keeps the median band. The scorecard shows the three samples as small dots.

Timing and latency#

Latency is how long the agent takes to start answering: from the moment the caller stops speaking to the agent's first sound. A greeting has no latency. The report shows the median (p50) and the slowest reply.

As a rough guide for phone calls, a median under about 1.5 seconds feels responsive and replies over about 3 seconds feel broken. But the judge only applies what your timing criteria say, so write the number you expect, like “Every agent reply starts in under 2 seconds.”

Timing for text agents

A text agent has no sound, so its timings start when the user message is sent. The judge sees, for every reply, the time to the first text and to the full reply, and the tools the agent ran. In Stream mode latency is the time to the first text; in Invoke mode the whole reply arrives at once, so the first text and the full reply are the same. A reply that broke off (the agent failed before finishing it) is marked and never counts as a measured reply. Write timing criteria for what the user sees, like “Every reply starts within 3 seconds.”

Evidence#

Every verdict comes with the evidence the judge relied on: exact quotes, and references to where they come from.

Reading the results#

Looks good

No must criterion failed, a score of 90 or more, nothing capped, every criterion observable, and samples that agree.

Needs work

A must criterion failed, a capped criterion, a score under about 70, or the same tag (like slow response or incomplete task) coming back across runs.

Inconclusive

Many criteria not observable, or a call that ended early (failed, or too long without speech). Fix the setup before drawing conclusions about the agent.

Suggestions and the brief#

The judge also writes suggestions: concrete changes, each with an area (prompt, tools, flow, latency, voice or other), a priority (high, medium or low) and its evidence. The Handoff tab turns everything into a brief: a Markdown note meant to be pasted, on its own, into a coding agent. It covers the problems found, what could not be checked, the changes to make (most important first), what worked, and how to verify the fix.

Writing good criteria#

• Make each one checkable from the call: “Confirms the card is blocked”, not “Is helpful”.
• One idea per criterion. Split “verifies identity and blocks the card” into two.
• Put the numbers in timing criteria (“under 2 seconds”).
• Keep must criteria for what truly makes or breaks the call.
• If the samples often disagree on a criterion (the report says Judges disagreed), its wording is probably ambiguous: make it more concrete.
• Keep a criterion when you edit it, rather than deleting and re-adding it, so results stay comparable over time.