Evals

Use your own models

Test a model you host, a model you fine-tuned, or a model from a provider you already pay, on the same tasks and in the same way as BoltzLabs models.

What your service needs

  • An OpenAI-compatible chat API: it answers POST …/chat/completions in the OpenAI format. Most model servers and providers offer this.
  • Tool calls (also called function calling) and streamed answers. The model uses tools to read files, run commands and edit code, so a model without tool calls cannot do the task.
  • An address the internet can reach. Runs happen on BoltzLabs machines, not on your computer, so localhost or an address on your home or office network will not work.

Fill in My endpoint

In New eval, under Who does the task, choose My endpoint and fill in:

  • Address: the part before /chat/completions. It usually ends in /v1, for example https://api.example.com/v1.
  • API key: leave it empty if your service needs none.
  • Sent in header: leave it empty to send the key as Authorization: Bearer <key>. If your service wants the key in another header, such as x-api-key, type that header's name. The key is then sent as it is.
  • Models: the model names exactly as your service knows them, separated by commas. Up to five.

Test it before you start

Press Check it works. It sends each model one short question and shows, for each, that it answered and how fast, or what went wrong. If the address can only be reached from your own computer or network, it tells you.

This only checks that the model answers. It does not check tool calls, so start with one run of each model and read its trace before you run more.

Your key

  • The key is used to start the runs. BoltzLabs does not save it.
  • Each run's machine holds a copy only while the run is going. It is removed when the run ends.
  • The website does not keep it in your browser, so you type it again for each new eval. The desktop app keeps it only while the app is open.

What you pay

  • Your service bills you for the tokens. BoltzLabs does not charge for them.
  • BoltzLabs bills only the machines the runs use.
  • Because BoltzLabs does not know your prices, results show no model cost for your models. Time, model calls and tokens are still shown.
  • Runs still stop at 30 minutes. The 400-call and $5 limits apply only to BoltzLabs models, so set limits on your service if you need them.

Compare with BoltzLabs models

One eval uses either BoltzLabs models or your endpoint. To compare the two, run the same task twice: open the first eval, press Run again, and switch who does the task. On the website, Run again fills in your model names but not the address or the key. The Evals list shows evals with the same task, starter code and check together, so all the models sit side by side. Your models are marked your endpoint.

Next: Results and traces.