Method — ISOTANTA

Words we use

A question is one problem. The question pool is every problem we hold, including easier ones used only to check coverage. The hard set is the god-mode slice: that is what models are tested on. A sampled test is a subset drawn from the hard set — twenty questions per subject when the cell is large enough, plus a sealed extra-hard ceiling pack. An attempt is one time a group of models takes that sampled test, same questions, same rules. A receipt is the public record of one attempt. The board is not that one attempt. It is the middle score across many independent attempts.

Why the protocol behind a score matters

Benchmark results can be difficult to compare when labs use different question sets, prompt wrappers, graders, scaffolds, or reporting rules. Public benchmark material can also enter training data, while cost, token use, and time are often omitted from the main result. Why these details matter. The rest of this page describes the protocol used here.

People write the questions

Questions come from people, not from the lab shipping the model. Contributors are not the systems under test. Names stay off the public pages. The question text and the correct answers stay private. What you see is a count, a subject, a skill tag, and a fingerprint of the sample — not an exam key.

How a test is drawn

The live test is a stratified sample from the hard set, using a calibration grid of subjects and difficulties. Easy, difficult, and hard questions stay in the pool so coverage can be inspected; they are not used in the sampled test. Each independent attempt uses its own seed so results can be examined across different samples. The receipt records the seed, method, version, number of questions asked, and a salt-hashed fingerprint.

How a question is scored

Every model in an attempt sees the same questions. Ceiling questions weigh more in the mean. Separately, a Wilson 95% interval is computed on the unweighted pass count. That interval is on the receipt. It is not the bar on the board.

How the board ranks models

For each public attempt, the model’s mean is converted to a percentage. The board then reports the median across attempts. A model appears only after at least 5 public attempts. The displayed score is the median, not the highest observed result. The strip shows every attempt as a bubble on the score line, and the line underneath shows the lowest-to-highest range of those attempts (not a confidence interval). The small n is the number of attempts, not the number of questions.

Cost, tokens, time

Cost is the amount OpenRouter billed for the attempt, in USD, rather than the catalog list price. Because published list rates can change, the site reports the billed amount for the actual run. Tokens are the prompt, completion, and total counts reported by the provider. On the figures, USD / question is billed cost ÷ number of questions asked. Tokens / question is total tokens ÷ that same count. Implied USD / 1M tokens is billed cost ÷ (total tokens / 1 000 000). USD / test is the billed cost of the whole attempt. Time / question is the median model time per question, then the median of those values across attempts. Queue time is reported on the receipt but is not used for the time axis.

What this is not

OpenRouter may send the same slug to more than one backend. Provider tokenizers are not a shared unit, so token counts are not directly comparable across providers in the same way billed dollars are. A hard sample on this pool measures performance on this pool; it is not a general intelligence score. Disputed attempts remain visible. Source ↗