What it is
GSM8K has grade-school math word problems written by people, each with a worked solution that ends in one number: about 7,500 for training and 1,319 for testing.
GSM8K
Grade-school math word problems with checkable final answers.
Coming soon. This kind of eval can't be run or requested yet. Today BoltzLabs runs codebase evals: each task is a repository, a job to do in it and a test that checks the result, run on its own machine.
Details from the linked paper and repository, checked October 2026.