The modules
Every screen of the app: what it is for, its key fields and a few tips.
The sidebar of the app lists the main modules; Connections, API keys and Model prices are reached from Settings, which also holds your theme and language. This page explains each one: what it is for, what you will find there, and a few tips. Jump to one:
Overview
The home page: how your agents did over the last 7 days, at a glance. It is empty until the first run, and then shows:
- Key metrics: runs in the last 7 days (compared with the week before), the average judge score, goal completion (the share of judged runs whose must criteria all passed) and the agent's median reply latency.
- Live now: runs in progress, with Listen in for calls and Watch for text conversations.
- Recent runs, filtered by All, Failed or Needs review (a must criterion failed, or the score is under 60).
- Weakest criteria, the criteria with the lowest average scores, and Suggestions ready for handoff.
Live sessions
Watch runs as they happen, and listen to calls. Each run in progress is a card (calls show their recent audio levels); after a launch you land on the batch, with its progress. A text agent's run shows the chat streaming in instead: each reply as it is written, the tools it runs, and its time to the first text.
- Live transcript: what each side says, as it is said. Interruptions and barge-ins are marked.
- Evaluator intent: what the caller is about to say and why, before it says it. If the agent speaks again first, the decision is shown as discarded. The realtime evaluator decides as it speaks: its words appear in its bubble as they are said, and the full text replaces them once it has finished.
- Evaluator goals: the plan the judge will check after the call.
- Call metrics: average and slowest agent latency, turns and interruptions (for a text agent: time to the first text, messages and tools used).
- Listen (calls only) with an audio mix (both, agent only or evaluator only) and a volume control.
- The run's full thread ID under its name, with a copy button, to look the call up in your agent's logs while it runs.
Turn on Follow and listen automatically to have each run open as it starts and, for a call, play on its own. With runs in parallel it stays on the current one until it ends, then moves to the next run still in progress; you can still pick another card. Your browser remembers the choice. After a reload, the browser may need one click (on Listen) before it plays audio.
Runs & reports
Every run ever made, with its transcript, timings and (for calls) audio. Filter by Status or Scenario, and switch between Newest first and Oldest first beside the filters; the filters and the order are kept in the URL, so a link or a reload shows the same list. Recent batches shows how each launch is progressing.
On wide screens (1280 px and up) the runs are a compact table. The Scenario cell holds the scenario name, with the agent, its kind (Voice or Text) and the persona under it; then come Duration, Turns (caller / agent), p50 latency and Cost; Outcome shows the score and the result, and the status only when the run did not complete normally (still in progress, failed, cancelled or interrupted); Started says when it began. Live and Cancel appear only for runs still in progress. On narrower screens each run is a card with the same figures, and the card keeps the full run ID with a copy button. Click a scenario name to open the run's report:
69/100
3/ 4
Missed: The agent blocks the cardThe agent verified the caller politely but never confirmed the card was blocked before the call ended.
Incomplete task: 1 criterion| Part | What it shows |
|---|---|
| Verdict | Overall score, goals met, the judge's summary, failure tags and strengths. |
| Call recording | The whole conversation as one recording (both voices on one timeline), audio lanes with issue markers, Download conversation .wav and the separate tracks. |
| Conversation (text agents) | Instead of the recording: the mode, user messages and agent replies, time to the first text (median and slowest), full reply time, tools used and why the conversation ended. |
| Transcript & scorecard | The conversation next to one row per criterion, with verdicts and links to the turns used as evidence. A text agent's transcript is a chat, with each reply's timings, its tools and replies that broke off. |
| Langfuse trace | The agent's model calls, tool calls and errors during the call, as a timeline. |
| Suggestions | Concrete changes to make, each with an area (prompt, tools, flow, latency, voice) and a priority. |
| Handoff | A brief for a coding agent: copy it or download it as Markdown. |
Scenarios
Your library of conversations. A scenario describes the user the evaluator plays, what it knows and wants, and how the judge scores the call. Filter by agent or tag, select several to Launch selected, or click a scenario's title to open it and edit it. Each scenario has three icon buttons, with a tooltip naming each: launch (play), duplicate (copy) and delete (trash, after a confirmation).
| Section | Key fields |
|---|---|
| 1 · Target agent and context | Name, tags, the agent under test, and context added on top of the agent's shared context. |
| 2 · Persona the evaluator plays | Persona name, locale (the caller speaks that language), voice override, temperament, notes. |
| 3 · Goal and instructions | Goal (the judge checks it), how to behave, and facts the caller knows (only revealed when asked). |
| 4 · Success criteria for the judge | Up to 30 criteria: text, Must or Should, evidence source (transcript, timing or trace) and weight. Weights total 100. |
| 5 · When the call ends | Max duration (30–1800 s, default 360), max silence (5–120 s, default 15), and hang up when: goal met or goodbye said, goodbye said, or max duration reached. For a text agent: max turns (1–100 user messages, default 20) instead of max silence. |
| 6 · Interruptions | Voice agents only. Up to 10: which agent reply to cut and after how many seconds of it. |
| 7 · Evaluator | Pipeline or speech-to-speech, the realtime model, the evaluator voice connection and the evaluator model connection. For a text agent (section 6): only the evaluator model connection, which is required. |
Agents
The agents you test: how to reach them and what they do. Each one is a voice agent (called over its WebSocket) or a text agent (reached over Vivi's channel API); the kind is chosen when the agent is added and cannot change. Scenarios are written against an agent, so add one first.
| Field | What it is |
|---|---|
| Name | Unique; scenarios and runs refer to the agent by it. |
| What this agent does | Shared by all its scenarios; the evaluator and the judge read it before every run. |
| Agent kind | Voice or Text, chosen when the agent is added. |
| Protocol | Voice: Auto-detect (default), Live or Realtime. |
| WebSocket URL | The agent's address, wss://, without a query string. ws:// is allowed for local addresses. |
| Channel ID, User ID | Voice: sent when the call opens, like your own client does. |
| Base URL, Channel ID | Text: the address before /channelsApi (https; http only for local hosts) and the channel's UUID. |
| Mode | Text: Stream (latency to the first text) or Invoke (latency to the full reply). |
| Authentication | Text: None for a WEB channel such as the portal's testing channel, or API key (sent as vivi-api-key, write-only). |
| Origin (optional) | Sent as the Origin header when your agent checks it. |
| Langfuse connection | Optional. The agent's traces are fetched after each run and handed to the judge. |
Connections
Under Settings → Connections: the credentials Duet uses to reach providers. Secrets are write-only: they are stored encrypted and only their last 4 characters are ever shown.
| Provider | Used for |
|---|---|
| OpenAI | The caller's voice, the caller's chat model, the realtime evaluator, and the judge. |
| Azure OpenAI | The same, through your deployments: a TTS deployment, a chat deployment for the evaluator, and realtime deployments for speech-to-speech. |
| Azure AI Speech | Can be stored, but cannot be used as the caller's voice yet. |
| Langfuse | Reads your agent's traces so the judge can check tool calls, models and errors. |
API keys
Under Settings → API keys: keys let scripts, CI or an AI agent use Duet without signing in. A key acts as you.
- Create API key: give it a name (where it is used, like “Nightly CI”) and an expiry (never, 30, 90 or 365 days).
- The full key is shown only once, right after you create it. Copy it into your CI secrets.
- The table shows when each key was created, last used and when it expires.
- The trash icon on a key's row revokes it: after you confirm with Revoke key, the key stops working immediately. This cannot be undone.
Model prices
Under Settings → Model prices: the token prices Duet uses to estimate what each evaluation costs. Each run report shows the Evaluation cost (the evaluator plus the judge, never your agent) and the runs list a compact Cost column.
- Prices are USD per 1M tokens, per model and kind of token (text or audio, input, cached input or output). They start as OpenAI's published list prices; edit them to match your contract. Azure OpenAI may differ, so costs are approximate.
- The judge counts every sample of every judgement, re-judges included. Pipeline calls do not price the TTS audio (the speech endpoint reports no usage), so their cost ends with a
+. - Runs from before costs were recorded show a dash.
API docs and agent skill
API docs in the sidebar opens the Swagger reference at /api/docs: every endpoint, its fields and its responses, readable without signing in. The Agent skill card on the API keys page gives an AI agent everything it needs to use the API: concepts, flows and how to read results.
How to put these together is explained in Automation.