Evals

Task sets

Write your tasks once, then score any models on all of them, as often as you like.

When to use one

One task tells you little about a model. A set of tasks from your own work tells you much more, and you can run the same set again when a new model comes out and compare the scores.

Make a set

  1. Open Evals, then Tasks, and press New set.
  2. Give the set a name, for example "billing bugs".
  3. Press Add task for each task, and fill in:
    • the task, in your own words;
    • its starting code: a public GitHub link with an optional branch, or empty. In the desktop app it can also be a folder on that Mac;
    • a check, such as python3 tests/test_refunds.py (optional);
    • a short name, such as "Refund rounding" (optional).
  4. Press Save. A set holds up to 50 tasks.

Each task can start from different code, so one set can cover several repositories.

Score models on it

  1. Open the set and press Evaluate models.
  2. Pick BoltzLabs models or your endpoint, how many runs each model gets on each task, and the machine size.
  3. Start. Every task becomes its own eval, and all the runs go at the same time.

Tasks × models × runs can be at most 10 at once. With 5 tasks you can run 2 models once each; with 2 tasks, 5 models. If one task cannot start, the ones after it are not started either, and the message says how many are running.

Read the scores

The set's page lists its last 10 runs. For each, every model gets a score such as "6 of 8 passed", plus how many are still running, did not finish, or had no check. Each task links to its eval, where you can open the traces.

Good to know

  • A set with a task that starts from a folder on a Mac can only be run from the desktop app on that Mac. The website shows the set but sends you to the app to run it.
  • Save your changes before you press Evaluate models.
  • Deleting a set keeps the evals it started.

See Costs and limits for what a set costs to run.