Tasks that use your own APIs or tools, checked on what the calls changed.
Coming soon. This kind of eval can't be run or requested yet. Today BoltzLabs runs codebase evals: each task is a repository, a job to do in it and a test that checks the result, run on its own machine.
We expose your API or command-line tools as tools a model can call, and write tasks that need them: look something up, change a record, chain several calls. Each task is checked on the state the calls leave behind, not on the wording of the reply.