Skip to content
DuetAgent evals
Section · Overview

Overview

What Duet is, who it is for, and the idea behind it: a simulated user calls or chats with your agent.

Duet tests agents the way your customers meet them: on a call or in a chat. A simulated user, the evaluator, phones or writes to your agent under test, plays a person with a goal, and every conversation is recorded and then scored by a judge.

What is Duet?

Building a voice agent is the easy part. Knowing whether it still works after you change a prompt, a tool or a model is hard: you would have to call it yourself, again and again, and listen carefully every time. Duet does those calls for you.

You write down the kinds of calls you care about (a customer who lost a card, someone in a hurry, a caller who keeps interrupting). Duet places those calls for real, against your agent's own voice connection, and gives you back the recording, the transcript, timings and a report that says what went well, what went wrong and what to change.

Voice and text agents

Duet tests two kinds of agents with the same scenarios, judge and reports:

  • Voice agents are called over their realtime WebSocket. The evaluator speaks, listens and can interrupt; you get a recording, a transcript and response latencies.
  • Text agents are reached over Vivi's channel API (HTTP). The evaluator writes each message as the user and reads the reply; you get a chat transcript with the tools the agent ran and how long each reply took to start and to finish. There is no audio.

You choose the kind when you add the agent; scenarios written for it adapt (a text scenario has a maximum number of turns instead of silence and interruption settings).

Who is it for?

  • Teams building voice or chat agents who want to catch regressions before their users do.
  • Product and QA people who want to listen to calls, read chats and clear reports without touching code.
  • Developers and AI coding agents who want a ready-made brief of what to fix, or who drive Duet from scripts and CI through its API.

The idea: the evaluator is the user

Most test tools send text to a model. Duet instead acts like a phone caller. The evaluator connects to your agent over the same WebSocket voice protocol a real client uses, speaks with a real synthetic voice, listens to the answers and decides what to say next, following a persona and a goal. It can even cut the agent off mid-sentence, like people do. With a text agent it does the same in writing: it sends each message through the agent's channel API and keeps the whole chat in one thread.

Throughout the app,blue is your agentandamber is the evaluator

The big picture

Every evaluation follows the same path, from a written scenario to a report you can act on.

  1. ScenarioYou describe one kind of call: who the caller is, what they want, what they know and how the call is scored.
  2. BatchLaunching one or more scenarios creates a batch: one run per scenario and repetition, queued in order.
  3. Run: a real callThe evaluator phones your agent over its WebSocket and plays the caller until the call ends.Evaluator (caller)talks withYour agentover WebSocket audio
  4. ArtifactsEverything is recorded: both voices as audio, the transcript, per-turn timings and an event log.
  5. JudgeA judge model reads the transcript, the timings and (when linked) the agent's Langfuse traces, and scores every criterion.
  6. ReportYou get a score, a verdict per criterion, suggestions and a brief you can hand to a coding agent.
From scenario to report: what happens after you press Launch.

The page How it works walks through each step in detail.

What you get from each run

  • A recording of the whole conversation (voice agents), both voices on one timeline, with pauses and interruptions as they happened.
  • A transcript with timings: how long the agent took to start answering each time. For a text agent it is a chat, with the time to the first text and to the full reply, and the tools each reply used.
  • A score from 0 to 100 and a verdict for each of your success criteria, with quotes as evidence.
  • Suggestions ranked by priority, and a brief: a ready-to-paste note for a coding agent that explains what to change.

Where to go next

New to Duet? Follow Getting started for your first run. Looking for a specific screen? See The modules. Puzzled by a score? Read Scoring.