A benchmark should make its testing conditions clear.

A benchmark score is difficult to interpret when the questions, sampling procedure, prompts, grading rules, or reported attempts are unclear. ISOTANTA is designed to make those choices visible.

The model provider can also define the evaluation

The organization publishing a model may also choose which questions count, which public sets to use, how prompts are formatted, how answers are graded, and which attempts are reported. Those choices can materially affect the result. Independent evaluation therefore requires more than applying a familiar benchmark name to a model.

Public benchmarks can enter training data

Static public benchmark sets are widely copied and redistributed, which increases the possibility that benchmark material appears in training data. As this happens, performance on a fixed public set becomes harder to interpret as evidence of performance on unseen questions.

A single attempt does not show run-to-run variation

A single draw of questions, temperature setting, routing decision, or seed can affect a result. Reporting repeated attempts makes that variation visible instead of reducing it to one selected score.

Performance should be reported with resource use

Two models can reach similar scores while using very different amounts of time, tokens, or money. Reporting accuracy without those quantities gives an incomplete account of the evaluation.

Grading rules need to be inspectable

Model-based grading can be useful, but its rubric and procedure still need to be documented. The same applies to private grading systems. A result is easier to examine and reproduce when the rules used to produce it are explicit.

The same name is not the same test

A published “MMLU score” or “coding score” can refer to zero-shot, five-shot, chain-of-thought, a tool loop, a hidden system prompt, or a different subset. Results that use different procedures should not be treated as the same measurement simply because they share a benchmark name. Tokenizers also differ across providers, so token counts are not a shared unit.

What ISOTANTA changes

People contribute questions. The public pages show how the pool is tagged, not the prompts. A sampled test is drawn from the hard set under common conditions for every model in an attempt. Each receipt records the seed, fingerprint, billed cost, token use, and other details needed to inspect the attempt. Rankings use the median across attempts, and a model appears only after at least five public attempts. The same sample can be repeated, and disputed attempts remain visible.

ISOTANTA does not claim to remove every source of bias from benchmarking. Its aim is narrower: separate the evaluated model from the test design, document the procedure, and make the result inspectable. How a score is made