What it is
SWE-bench pairs real software issues with repositories and tests. Agents produce patches, which are evaluated with a reproducible execution harness.
Resolving an issue requires understanding existing code and preserving behavior. Repository tests check the patch against the requested change.
How an agent works in it
Sees
Issue text, repository contents and execution feedback.
Does
Search code, edit files and execute development tools.
Feedback
The selected benchmark’s test harness determines whether the issue is resolved.
Example task (illustrative): Fix the reported behavior without breaking the existing test suite.
Setup
Use the upstream containerized evaluation harness and choose the appropriate dataset split. Published evaluation tasks should stay outside a training corpus.
Ready to run
A run passes when the regression test from the real fix passes. This sample task was checked: its check fails on the starting code and passes with the known fix. Run it from Prepare eval in the BoltzLabs desktop app.
| Task | pydicom: to_json_dict still raises with suppress_invalid_tags (pydicom__pydicom-1694) Dataset.to_json_dict(suppress_invalid_tags=True) can still raise. A data element whose stored value cannot be converted, for example an IS (integer string) element holding b"5.25", raises "TypeError: Could not convert value to integer without loss" while the dataset is converted to JSON. With suppress_invalid_tags=True such an element should be left out of the result; without it the error should still be raised. Fix this in the pydicom package. |
|---|---|
| Repository | github.com/pydicom/pydicom at v2.3.1 |
| Check | python3 - <<'EOF'
import pytest
from pydicom.dataelem import RawDataElement
from pydicom.dataset import Dataset
from pydicom.tag import Tag
ds = Dataset()
ds[0x00082128] = RawDataElement(Tag(0x00082128), "IS", 4, b"5.25", 0, True, True)
with pytest.raises(TypeError):
ds.to_json_dict()
ds_json = ds.to_json_dict(suppress_invalid_tags=True)
assert "00082128" not in ds_json
print("ok")
EOF |
| Time limit | 120 seconds |
Use it with BoltzLabs
BoltzLabs does not host this project. To record an agent working on one of its tasks:
- Prepare one task from the project: its starter files, instructions, dependencies and check command.
- In Evals, choose New eval, upload the starter folder or enter its repository, then set the task and check command.
- Choose BoltzLabs models or your own endpoint. Each run gets its own machine.
- When the run finishes, open it to read the agent’s trace, file changes and check result.
Projects that need a desktop or phone, self-hosted websites, a game or a simulator need that runtime set up separately; an eval machine does not include it. Machine time uses account credits; hosted model calls are billed separately.