What it is
OSWorld evaluates multimodal agents on open-ended tasks in real computer environments. It includes application and operating-system tasks with execution-based evaluation.
Desktop work combines visual understanding, actions and application state. The benchmark measures whether the final state satisfies the task.
How an agent works in it
Sees
Desktop screenshots and supported computer-state observations.
Does
Mouse, keyboard and supported computer-control actions.
Feedback
Task-specific evaluators inspect the completed computer state.
Example task (illustrative): Edit a document, change an application setting, and save the result.
Setup
Prepare the virtual-machine environment and task snapshots. Desktop/VM dependencies need a compatible host; this is not a plain Python pool image.
Use it with BoltzLabs
BoltzLabs does not host this project. To record an agent working on one of its tasks:
- Prepare one task from the project: its starter files, instructions, dependencies and check command.
- In Evals, choose New eval, upload the starter folder or enter its repository, then set the task and check command.
- Choose BoltzLabs models or your own endpoint. Each run gets its own machine.
- When the run finishes, open it to read the agent’s trace, file changes and check result.
Projects that need a desktop or phone, self-hosted websites, a game or a simulator need that runtime set up separately; an eval machine does not include it. Machine time uses account credits; hosted model calls are billed separately.