What it is
HumanEval has 164 hand-written Python problems. Each gives a function signature and docstring; the model writes the body, and the unit tests stored with the problem check it.
HumanEval
Hand-written Python programming problems, each checked by unit tests.
Coming soon. This kind of eval can't be run or requested yet. Today BoltzLabs runs codebase evals: each task is a repository, a job to do in it and a test that checks the result, run on its own machine.
Details from the linked paper and repository, checked October 2026.