How we measure.

Every number on this site is reproducible from raw data plus a pinned config revision. If you can't falsify it, we don't publish it.

Sampling

Probes fire continuously against every backend: fixed prompt battery, streamed and non-streamed, TTFT measured to the first content byte. Failures are classified (timeout, refusal, rate-limit, malformed) and never silently dropped.

Suites & scoring

Code tasks run under pytest in an isolated scratch dir with a 30s timeout and (when available) a network namespace. This is not a full VM sandbox. Frontend tasks get structural checks; agentic tasks validate JSON fields. Suite definitions are content-hashed, including scoring rules and sampling params, and that hash is stored on each tick.

Latency candles

Ticks aggregate into hourly/daily OHLCV buckets: open/close are first/last successful samples in the bucket, high/low the extremes. Failed requests count toward volume only — they never pollute the price series.

Cost model

Capex amortizes over duty-cycle serving life. Energy counts only marginal watts (above idle). API prices are list prices at publication time. Crossover volumes are computed, not eyeballed.

Lineage

Router config is append-only revisioned in SQLite. Every request records the exact revision that served it. Any published figure traces to raw ticks → suite hash → config rev.

How to falsify

Raw Parquet ticks, suite YAMLs, and config revisions ship alongside every report. Re-run our rollups on the raw data; if your numbers differ from ours, that's our bug — file it.