What it is
We read your repository and write tasks from it. Each task is a change to make in your code, with a starting state and a check command. A task is kept only if its check fails on your current code and passes once the change is made. You can keep some tasks aside for evals, so training never sees them.
What you give us
Your repository (a link or an upload), what the tasks should exercise, and which models you want to compare.
How a run is checked
A run passes when the task’s check command exits 0 in the run’s machine.
Request it
We prepare this eval from your repository; there is nothing to set up yourself. Request it from Prepare eval in the BoltzLabs desktop app: choose Repo to RL tasks, add your code and say what the eval should measure. We reply there when it is ready, and you then run it in Evals like any other eval.