What it is
Terminal-Bench evaluates autonomous agents on command-line workflows. Tasks supply an environment, an instruction and verification, spanning software and system-level work.
A terminal agent needs to combine commands, files and debugging into a completed workflow. Task verification measures the resulting work.
How an agent works in it
Sees
Terminal output, files and the task instruction.
Does
Run shell commands, inspect artifacts and edit files.
Feedback
Task-specific verification checks the final artifacts or state.
Example task (illustrative): Diagnose a broken build and produce the required working artifact.
Setup
Follow the current Harbor/Terminal-Bench instructions and select a pinned dataset version and compatible sandbox provider.
Use it with BoltzLabs
BoltzLabs does not host this project. To record an agent working on one of its tasks:
- Prepare one task from the project: its starter files, instructions, dependencies and check command.
- In Evals, choose New eval, upload the starter folder or enter its repository, then set the task and check command.
- Choose BoltzLabs models or your own endpoint. Each run gets its own machine.
- When the run finishes, open it to read the agent’s trace, file changes and check result.
Projects that need a desktop or phone, self-hosted websites, a game or a simulator need that runtime set up separately; an eval machine does not include it. Machine time uses account credits; hosted model calls are billed separately.