How the test is defined.
one current sampled test
It specifies what can be tested, how questions are selected, and which sampled set the models receive. Within one attempt, every model is evaluated on the same sampled questions under the same scoring rules.
01
A benchmark can become difficult to interpret when the test is adjusted after model performance is already visible. Changing the question subset, prompt conditions, grader, or reporting rule can change the result. A challenge separates those choices from the score: the test is defined first, then models are evaluated against it.
02
The challenge records the eligible question set, the sampling method, how many questions are asked, the seed used for the draw, and the fingerprint of the resulting sample. The live prompts and answers remain private, but the public record still identifies which sample was used. That makes it possible to distinguish two runs that used the same questions from two runs that used different samples.
03
The challenge describes the test. An attempt is one evaluation carried out on that test. The attempt receipt records the models evaluated, their scores, the sample fingerprint, cost, token use, and timing where those values are available. Rankings summarize results across eligible public attempts; they do not replace the individual receipts.
04
The public site shows the size and structure of the question bank, the subjects and skills it covers, and display-only examples with answers. It does not publish the live prompts or gold answers. This reduces direct exposure of the exact test while still making the sampling and scoring procedure visible. Private questions do not guarantee that a model has never encountered similar material, but they avoid turning the live test itself into a public training set.
The current benchmark has one sampled challenge.
1,661 questions held · 421 in the hard set (god-mode) · 132 asked in the current draw · seed 42