Run an eval

1. Open New eval

On the website, open Evals and press New eval. In the desktop app, press Evals on the left, then New eval.

2. Write the task

Write it the way you would ask a teammate. Say which files matter, what done looks like, and what must not change. For example:

tests/test_checkout.py has a failing test. Find the cause in shop/pricing.py and fix it without changing the tests, then run the tests and make sure they all pass.

3. Choose the starter code

Every run starts with its own copy of this code.

• GitHub link: a public repository, and a branch if you want one. Private repositories are not supported yet.
• Local folder (on the website) or Folder on this Mac (in the app): a folder from your computer. .env files and dependency folders such as node_modules are left out. On the website the folder can be up to 48 MB and 10,000 files.
• Empty: every run starts with an empty folder.

If your code has scripts that need to stay executable, use a GitHub link. A folder uploaded from the website loses that setting.

4. Add a check (optional)

The check is a command that runs in the code folder after the model is done. If it succeeds (exit code 0), the run passed. If it fails, the run failed the check. It has 5 minutes.

Each machine has Debian Linux with Python 3, Node.js and npm, and git. Good checks use what is already there, for example python3 tests/test_checkout.py, python3 -m unittest or npm test. If your check needs something else, install it in the check itself, for example npm ci && npm test.

A model can edit any file, tests included. Say in the task that the tests must not change, and look at the changes in the trace if a pass seems too easy.

Without a check, a run just says Finished, and you judge it by reading its trace.

5. Choose who does the task

• BoltzLabs models: tick up to five. Each shows its price per million tokens, in and out.
• My endpoint: up to five models from your own service. See Use your own models.

6. Set runs, name and machine

• Runs of each model: give each model the task more than once to see if a pass was skill or luck. An eval has at most 10 runs in all.
• Name: optional. Left empty, the eval is named by date.
• Machine for each run: small, medium or large, with its price per hour. Small is enough for most tasks.

Press Start. The button says how many runs will start.

What happens next

• Each run gets its own machine with internet access, and its row says Working.
• You can close the page. The runs keep going, and the page shows them when you come back.
• A run ends when the model says it is done, or at a limit: 30 minutes, and for BoltzLabs models also 400 model calls or $5 of model use.
• To end a run early, press Stop and collect on its row. What it did so far is kept and the check still runs.

Next: Results and traces.