What it is
We take fixed issues or bug-fix commits from your repository and set each one up just before the fix. The model gets the issue and changes the code; the tests from the real fix decide whether it worked.
What you give us
Your repository, how to install and test it, and any issues or areas you want included.
How a run is checked
A run passes when the tests from the real fix pass and your existing tests still pass.
Request it
We prepare this eval from your repository; there is nothing to set up yourself. Request it from Prepare eval in the BoltzLabs desktop app: choose Issue fixing, add your code and say what the eval should measure. We reply there when it is ready, and you then run it in Evals like any other eval.