Skip to content

Own benchmarks: how I measure

For my own projects I keep measuring which model is fit for which job. What is useful to others I publish here with its method and dataset. Every new run follows the same rules, so anyone can check it and repeat it.

Go to the external leaderboards

The rules for every run

  1. 01Rules fixed before measuringWhat gets measured and how it counts sits in a file before the first call, with a SHA-256 checksum and a timestamp. If a rule changes during a run, that makes a new version with an entry in the change log.
  2. 02Clean, anonymous inputsInputs are invented or public. No customer data, no personal data, nothing from client work. A judge only sees "Text 1" and "Text 2", never a model or brand name.
  3. 03Judges from at least two model familiesWherever a model judges, a model from another family judges too. One family alone is never the final verdict, because a judge can rate its own family's output differently from an outside judge.
  4. 04Controls inside the runEvery pairing runs in both orders. A known-weak entry runs alongside to show whether the judge spots it. Load-time and network measurements run at least three times.
  5. 05Open datasetEvery run comes with a CSV file holding one row per verdict or measurement, a schema, and the frozen rules with their checksums. I never publish third-party content, only identifiers.
  6. 06Date, models, costEvery run states its date, the exact model ids and what it cost.
  7. 07Numbers with uncertaintyI give counts and shares, and a 95 percent interval where the sample allows it. Small samples are marked as small.
  8. 08Limits written outEvery run says what its numbers do not show.
  9. 09Naming companiesWhen a benchmark names companies, it says whether they were asked for comment before publication.

When I re-run

Every re-run gets a new version. Old runs stay up, so changes stay visible.

Models
On every major release of the model families covered, otherwise every six months. The focus is on Claude models, which I work with every day.
Websites and companies
Every quarter.
Hosting and infrastructure
Every six months, or when a provider changes a region or runtime.

Runs so far

Each run with its date, version, article and every file needed to recompute it.

The datasets are licensed CC BY 4.0. Please credit seelig.ai when you reuse them.

What this page is not

The leaderboards on the benchmark overview come from external sources such as Arena and LiveBench. This page lists only measurements I ran myself.

Need a model picked for your use case?

Benchmarks rank models on other people's tasks. For a business workflow I test the shortlist on your own data, costs and privacy requirements, and recommend one.

AI consulting