What it is
We load a copy or a sample of your database into the run’s machine and write questions in plain words, each with a query that answers it. The model writes its own query, and the rows it returns are compared with the right answer.
SQL questions
Questions about your data, answered by writing SQL against a copy of it.
Coming soon. This kind of eval can't be run or requested yet. Today BoltzLabs runs codebase evals: each task is a repository, a job to do in it and a test that checks the result, run on its own machine.