SWE-Gym
Train software agents on real repositories, executable tasks and unit tests.
Explore
Coding 8
SWE-bench
Evaluate repository-level fixes for real GitHub issues.
SWE-smith
Creates many repository tasks with failing tests, for training software agents.
R2E-Gym
Procedurally built repository environments with hybrid verifiers, for training software agents.
HumanEval
Hand-written Python programming problems, each checked by unit tests.
LiveCodeBench
New problems collected over time from LeetCode, AtCoder and Codeforces, to avoid contamination.
Repo to RL tasks
Tasks written from your repository, each with a check, for evals and RL training.
Issue fixing
Real bugs from your repository’s history, each with the tests from the real fix.
Web 8
WebArena
Complete realistic, multi-step tasks across self-hosted websites.
BrowserGym
A common Gym-style interface for studying web agents across benchmarks.
WebShop
Ground shopping instructions in product search, attributes and choices.
MiniWoB++
Practice focused web interactions in small, repeatable browser tasks.
VisualWebArena
Realistic web tasks that depend on images and other visual page content.
Mind2Web
Over 2,000 tasks collected on 137 real websites, for generalist web agents.
WorkArena
Everyday knowledge-work tasks on the ServiceNow platform, run through BrowserGym.
Browser tasks
Jobs in your web app, done through a browser and checked on the app’s data.
Tool use 5
τ²-Bench
Evaluate agents that coordinate with users and act through tools.
τ-bench
Retail and airline tasks where an agent works with a simulated user, tools and policies.
ToolBench
Tool-use instructions over more than 16,000 real-world APIs.
AppWorld
Agents write code against the APIs of nine everyday apps used by simulated people.
Tool calling
Tasks that use your own APIs or tools, checked on what the calls changed.
Desktop 2
OSWorld
Work across real desktop applications using computer interaction.
Windows Agent Arena
Over 150 tasks across Windows applications, on a real Windows machine, run in parallel.
Mobile 1
AndroidWorld
Tasks across 20 real Android apps, with goals that change from run to run.
Terminal 3
Terminal-Bench
Solve practical tasks in a terminal with a reproducible task harness.
InterCode
Interactive coding in Bash, SQL and Python, with execution feedback after every action.
Terminal tasks
Command-line jobs in your project: build, debug, migrate and configure.
Embodied 2
ALFWorld
Learn household tasks through aligned text and embodied environments.
ScienceWorld
A text world for running elementary-school science experiments.
Games 2
TextWorld
Generates text-based games for training and testing reinforcement learning agents.
NetHack Learning Environment
The game NetHack as a fast, hard, procedurally generated RL environment.
Documents 3
SpreadsheetBench
Spreadsheet tasks built from 912 real questions on online Excel forums.
OfficeBench
Office workflows that need long plans and switching between several applications.
Document Q&A
Questions about your own documents, with answers checked against them.
SQL and data 4
Spider 2.0
Enterprise text-to-SQL workflows on large databases and cloud data warehouses.
BIRD
Text-to-SQL questions grounded in large, real-world databases.
DS-1000
Data science coding problems across NumPy, pandas and other Python libraries.
SQL questions
Questions about your data, answered by writing SQL against a copy of it.
Security 4
Cybench
40 professional capture-the-flag tasks from four competitions.
NYU CTF Bench
Capture-the-flag challenges from popular competitions, with an automated agent harness.
AgentDojo
Tool-using agents face prompt-injection attacks; measures both task success and security.
Security fixes
Security weaknesses in your code to find and fix, checked by tests that try them.
ML research 2
MLE-bench
75 Kaggle competitions as tasks for machine learning engineering agents.
PaperBench
Agents replicate 20 ICML 2024 papers from scratch, graded against author-checked rubrics.
Math and reasoning 2
Reasoning Gym
Over 100 generators and verifiers for reasoning tasks, for RL with verifiable rewards.
GSM8K
Grade-school math word problems with checkable final answers.
General 3
AgentBench
Eight environments, from operating systems and databases to games, for testing LLMs as agents.
GAIA
Real-world questions for general assistants that need browsing, tools and reasoning.
TheAgentCompany
A simulated small software company where agents browse, write code and message coworkers.
Research entries are public projects, not hosted by BoltzLabs. Ready to run evals come with a sample task we checked, and On your repo evals are ones we prepare from your code; both are started from Prepare eval in the BoltzLabs desktop app. Coming soon lists kinds of eval that are not available yet. Hosted environments are in RL Pools; to compare agents on your own task, use Evals.