How it works
What happens between pressing Launch and reading the report, and the states a run goes through.
Behind every report there is a simple chain: a scenario becomes runs, each run is a real conversation (a call with a voice agent, a chat with a text agent), the conversation leaves a recording or transcript behind, and a judge turns it into a report.
- ScenarioYou describe one kind of call: who the caller is, what they want, what they know and how the call is scored.
- BatchLaunching one or more scenarios creates a batch: one run per scenario and repetition, queued in order.
- Run: a real callThe evaluator phones your agent over its WebSocket and plays the caller until the call ends.Evaluator (caller)talks withYour agentover WebSocket audio
- ArtifactsEverything is recorded: both voices as audio, the transcript, per-turn timings and an event log.
- JudgeA judge model reads the transcript, the timings and (when linked) the agent's Langfuse traces, and scores every criterion.
- ReportYou get a score, a verdict per criterion, suggestions and a brief you can hand to a coding agent.
From scenario to batch
When you press Launch, Duet creates a batch: one run for every scenario you picked, times the number of runs each. Launching 2 scenarios with 3 runs each makes a batch of 6 runs. An optional label helps you find the batch later, like “after prompt fix”.
- In parallel sets how many runs of this batch may call at the same time. The server also has its own limit across all batches (3 by default), because each call is a paid session.
- Each run keeps a snapshot of everything it was launched with: the scenario, the agent and the evaluator setup. Editing a scenario later never changes past runs, so comparisons stay fair.
- Each run opens its own connection with a fresh thread ID. It is also the agent's Langfuse session id, which is how Duet finds the agent's traces for that call.
The call: evaluator and agent
When a run starts, Duet opens your agent's WebSocket exactly like a real client and waits until the agent says its session is ready. Then the conversation flows turn by turn:
- The agent speaks (often a greeting first). Duet records its audio and its text.
- The evaluator decides what the caller says next, following the persona, the goal, the instructions and the facts. It only reveals a fact when the agent asks for it.
- The evaluator speaks with a synthetic voice, and the agent answers. Duet times every reply.
- The call ends when the caller reaches its goal or says goodbye (depending on the scenario), when the agent closes the session, when the time limit is reached, or after too long without speech.
You can follow all of this from Live sessions: the transcript appears as people speak, and you can listen in. The details of interruptions and evaluator modes are on Interruptions and the evaluator.
Text agents: a chat instead of a call
With a text agent there is no audio. Duet reaches the agent over Vivi's channel API, and the evaluator writes first:
- The evaluator writes the user's message, following the same persona, goal, instructions and facts, and Duet sends it with an HTTP request:
/stream(the reply arrives in pieces as it is written) or/invoke(the whole reply in one response), depending on the agent's mode. - Every message of the run carries the same thread ID (a UUID), so the agent sees one conversation. It is also the Langfuse session id, as for calls.
- Duet times each reply from the moment the message is sent: the time to the first text (the latency in Stream mode) and to the full reply (the latency in Invoke mode), and notes the tools the agent ran.
- The conversation ends when the evaluator reaches its goal or says goodbye, after the scenario's Max turns user messages, at the time limit, or when the evaluator has nothing more to say. If the agent fails mid-reply (an HTTP error, or a stream that stops without its end-of-reply event), the run fails and the partial reply is kept, marked as broken off.
What is recorded
| Artifact | What it holds |
|---|---|
| Audio | Voice agents only: both sides of the call, plus one mixed recording with both voices on the call's timeline, pauses and interruptions included. |
| Transcript | Every turn in order, with when it started and ended. Interrupted agent replies show what was heard and what was not. |
| Timings | Per agent turn: how long it took to start answering (latency), how long it spoke, and how the turn ended. Text agents: when the message was sent, the first text and the full reply arrived, and the tools used. |
| Events | A detailed log of the call (status changes, decisions, interruptions, warnings), used for the live view and the report. |
| Summary | Duration, turn counts, latencies, the protocol your agent spoke (or the text mode), and any warnings. |
Run lifecycle
Every run moves through the same states. The status chip in the runs table and the live view show where it is.
- QueuedWaiting for a free slot
- ConnectingDialling the agent
- RunningThe call is in progress
Ends as one of
- CompletedThe call ended normally: a hang-up, the agent closing, or a time or silence limit.
- FailedSomething broke, such as the agent being unreachable. The run shows a short reason.
- CancelledSomeone stopped it (Cancel run or Stop all).
- InterruptedThe server stopped or restarted while the call was in progress.
Why a run ended
Next to the status, the report says why the call ended. The most common reasons:
| Reason | Status | What it means |
|---|---|---|
| Evaluator hung up | Completed | The caller reached its goal or said goodbye. |
| Agent ended the call | Completed | Your agent closed the session. |
| Reached the maximum duration | Completed | The scenario's time limit was hit. |
| Too long without speech | Completed | Nobody spoke for the scenario's maximum silence. |
| Reached the maximum turns | Completed | Text agents: the evaluator sent the scenario's maximum number of messages. |
| The evaluator had nothing to say | Completed | Text agents: the evaluator ended without a message to send. |
| The agent's reply failed | Failed | Text agents: an HTTP error, a timeout, or a stream that stopped without its end-of-reply event. |
| Could not connect to the agent | Failed | The WebSocket URL could not be reached. |
| The agent never became ready | Failed | The connection opened but the session never started. |
| Protocol error | Failed | Duet could not recognise or follow the agent's voice protocol. |
| Speech synthesis failed / The evaluator model failed | Failed | A problem with the caller's voice or brain: check the scenario's connections. |
| Cancelled | Cancelled | Someone stopped the run. |
| The server stopped / restarted | Interrupted | Duet itself was stopped during the call. |
Judging
A run is judged automatically when it ends Completed, or Failed after the agent spoke at least once. Cancelled and interrupted runs are not judged. The judgement goes through its own short lifecycle:
| State | What is happening |
|---|---|
| Queued | Waiting for the judge. |
| Collecting traces | Only with a Langfuse connection: Duet waits until the agent's traces are complete, checking every 30 seconds, for up to 20 minutes. |
| Judging | The judge model scores every criterion, three times, on the same evidence. |
| Completed | The score, scorecard, suggestions and brief are ready. |
| Failed | The judge could not finish; the report says why and offers Retry. |
The judge reads the transcript, the timings and, when available, the agent's traces (tool calls, models and errors). How it scores is explained in Scoring.
The report
The run page brings everything together: the verdict at the top, the call recording with audio lanes (for a text agent, the conversation metrics), and four tabs: Transcript & scorecard, Langfuse trace, Suggestions and Handoff. From there you can also Re-run the scenario (a new, paid call with the scenario as it is now) or Re-judge the call.