Evals

Evals

Give the same coding task to several AI models. See which ones got it right, how long each took, what each cost, and every step each one took.

How an eval works

  1. You write a task, for example "Make the failing tests in tests/test_api.py pass."
  2. You choose the starting code: a public GitHub repository, a folder from your computer, or nothing.
  3. You pick the models. Each model gets its own machine with its own copy of the code, and works on the task alone. It reads files, runs commands and edits code until it is done.
  4. When a model is done, your check runs. A check is a command such as python3 tests/test_checkout.py. If it succeeds, the run passed.
  5. You compare the results and read what each model did, step by step.

Words you will see

  • Eval: one task, given to the models you picked.
  • Task: what you want done, in your own words.
  • Run: one model's attempt at the task, on a machine of its own. You can give the same model several runs.
  • Check: the command that says if the work is right. Optional.
  • Trace: everything a model did in a run: what it said, the commands it ran and the files it changed.

Which models you can use

BoltzLabs models are ready to use. You pay for the tokens they use from your credits. Today they are DeepSeek V4 Pro, DeepSeek V4.1 Flash, DeepSeek V4 Flash, Kimi K3, Kimi K2.7 Code, GLM 5.3, GLM 5.3 Flash, MiMo V2.6 Pro and LongCat 2.0. Each model's price, and what a whole eval costs, are on Costs and limits.

Your own models work too. If your model is behind an OpenAI-compatible API, such as a model you host yourself, a fine-tuned model or a provider you already pay, add its address and key and test it the same way. See Use your own models.

Where to run evals

  • On the website: open Evals in your dashboard.
  • In the desktop app: press Evals on the left. See the desktop app quickstart to install it.

Both show the same evals from your account, so you can start one in the app and read it on the website.

In this section