What it is
τ²-Bench evaluates conversational agents in a dual-control setting, where the agent and a simulated user can affect the environment. Its scenarios combine conversation, tools and domain policies.
Finishing a support task requires both correct tool use and cooperation with the user. A plausible answer alone does not establish task completion.
How an agent works in it
Sees
User dialogue, domain policies and tool responses.
Does
Respond to the user and invoke domain-specific tools.
Feedback
Domain task evaluators assess the required outcomes.
Example task (illustrative): Help a customer resolve a service issue while following the domain policy.
Setup
Run the project’s agent and user-simulator harness. Configure model access and the domain/task split according to the upstream instructions.
Use it with BoltzLabs
BoltzLabs does not host this project. To record an agent working on one of its tasks:
- Prepare one task from the project: its starter files, instructions, dependencies and check command.
- In Evals, choose New eval, upload the starter folder or enter its repository, then set the task and check command.
- Choose BoltzLabs models or your own endpoint. Each run gets its own machine.
- When the run finishes, open it to read the agent’s trace, file changes and check result.
Projects that need a desktop or phone, self-hosted websites, a game or a simulator need that runtime set up separately; an eval machine does not include it. Machine time uses account credits; hosted model calls are billed separately.