Benchmark Methodology & Statistical Evaluation

TOURNAMENT RANKING SYSTEM

Bradley-Terry Elo Rating Framework

Tabular benchmarks contain a mix of diverse tasks with incompatible units and varying entropy. We resolve the "Metric Mixing Problem" by framing every persisted evaluation split as a head-to-head tournament match under stationary paired comparison likelihoods.

⚡ IN 30 SECONDS

Tabular datasets speak different "languages" — you cannot average raw dollars ( $$ 40 k RMSE$ ) with classification probabilities ( $0.05 Log-Loss$ ).

Instead of averaging mismatched numbers, we treat every dataset test split like a competitive match:

Head-to-Head: We pit Model A against Model B on the exact same data split.
Determining the Winner: The model with lower error loss wins the match ( $1 - 0$ ).
Global Elo: An algorithm tallies all wins across the suite and assigns each model a standardized Elo rating centered at $1000$ .

CORE INTUITION

The "Chess Tournament" Mental Model

The Metric Mixing Problem: In tabular machine learning, calculating a simple arithmetic average across datasets is statistically meaningless: you cannot average continuous regression error ( $$ 40,000 + RMSE$ in dollars) with binary classification performance ( $0.05 Log-Loss$ ). However, on any individual dataset split, you can always ask a clean, decisive question: Which model won the matchup?

Step 1

The Matchup

Every persisted dataset split (489 across the current 21-dataset suite) is treated as an isolated 1-on-1 match between two competing foundation models.

Step 2

Determining the Winner

The model achieving the lower error loss ( $1 - AUC$ , $Log-Loss$ , or $RMSE$ ) receives $1.0$ win point. The loser receives $0.0$ , and exact ties award $0.5$ .

Step 3

Solving Global Ability

All 489 split matchup outcomes populate a tournament win matrix, which the Bradley-Terry solver maps into a global Elo rating centered at baseline $1000$ .

Tournament Data Pipeline Flow

21 DatasetsAdaptive Splits (489 Total)

→

Pairwise 1v1Split Loss Comparison

→

Win MatrixPairwise Matches

W_{i j}

→

Bradley-TerryMM Likelihood Solver

→

Global RankingElo

R_{i} \pm {CI}_{95 %}

EMPIRICAL WIN/LOSS MATRIX

Tournament Scorecard (Pairwise Matchup Matrix)

Every reader understands a win/loss record. The Elo score summarizes that empirical record across all persisted dataset-split matchups:

Foundation Model	vs. TabICLv2	vs. TabPFN v3	vs. Google TabFM	Total Match Record	Tournament Elo
TabICLv2 (Carla)	—	52W – 43L	83W – 12L	135 – 55 (71.1%)	1165 ± 28
TabPFN v3	43W – 52L	—	81W – 14L	124 – 66 (65.3%)	1142 ± 31
Google TabFM	12W – 83L	14W – 81L	—	26 – 164 (13.7%)	835 ± 36

WORKED MATCHUP WALKTHROUGH

Example: TabICLv2 vs. Google TabFM (Adult Census, Fold 3)

Task & Primary MetricBinary Classification (

1 - AUC

, lower is better)

TabICLv2 Fold 3 Error

0.0745

(

92.55 %

AUC) — Winner (1.0 pt)

Google TabFM Fold 3 Error

0.1240

(

87.60 %

AUC) — Loss (0.0 pt)

Expected Win ProbabilityBecause TabICLv2 is

+ 330

Elo higher, the model expected TabICLv2 to win with

\approx 87.1 %

probability. TabICLv2 gains a small rating confirmation, while an upset by TabFM would have yielded a massive rating swing.

Elo Rating Difference (ΔR) Rule-of-Thumb

What does an Elo difference of 50, 100, or 200 points mean for actual model performance?

Elo Difference ( $Δ R$ )	Win Probability $P (A > B)$	Expected Odds	Practical Benchmark Meaning
0 pts	50.0%	1 : 1	Perfect Coin-Flip: The models have statistically indistinguishable capability.
+50 pts	57.1%	4 : 3	Slight Edge: Model A wins consistently on edge-case datasets with complex features.
+100 pts	64.0%	2 : 1	Noticeable Advantage: Model A wins roughly 2 out of every 3 fold matchups.
+200 pts	76.0%	3 : 1	Decisive Superiority: Model A wins roughly 3 out of every 4 fold matchups across diverse domains.
+400 pts	90.9%	10 : 1	Complete Dominance: Model A beats Model B in more than 9 out of 10 head-to-head splits (10:1 odds).

Paired Comparison

1. Bradley-Terry Log-Odds Formulation

Statistical Reference ↗

What is this? The expected win probability that Model A beats Model B on an unseen tabular dataset fold, derived directly from their rating difference:

Win Probability (A beats B) = \frac{1}{1 + 10^{\frac{R_{B} - R_{A}}{400}}}

Formal Mathematical Derivation (Latent Log-Abilities & Logistic Mapping)

In the Bradley-Terry parametric formulation, each model's latent ability is represented by $θ_{i} \in R$ . The win probability is given by the logistic sigmoid of their log-odds ability difference:

P (M_{A} > M_{B}) = \frac{e^{θ_{A}}}{e^{θ_{A}} + e^{θ_{B}}} = \frac{1}{1 + e^{- (θ_{A} - θ_{B})}}

To anchor abilities to the standard Elo 1000-scale with a 400-point logistic scale factor, parameters are transformed via:

θ_{i} = \frac{\ln (10)}{400} (R_{i} - 1000) \Leftrightarrow R_{i} = 1000 + \frac{400}{\ln (10)} θ_{i}

P (M_{A} > M_{B})

Predicted probability that Model A achieves a higher out-of-sample fold validation score than Model B.

R_{A}, R_{B}

(Elo Ratings)Standardized scalar ratings centered at baseline 1000. Higher ratings reflect higher win rates against strong opponents.

Logistic Scale Factor (

400

)Standard Elo constant: a 400-point rating difference corresponds to expected winning odds of 10:1 (

P \approx 90.91 %

).

θ_{A}, θ_{B}

(Latent Abilities)Natural-log ability parameters:

θ_{i} = \frac{\ln (10)}{400} (R_{i} - 1000)

, solved via maximum likelihood estimation.

Numerical Solver

2. Minorization-Maximization (MM) Algorithm

Hunter (2004) Paper ↗

Optimization Strategy: Rather than updating Elo step-by-step in an arbitrary chronological order (which introduces game-order bias), we aggregate all pairwise tournament wins $w_{i j}$ into an $M \times M$ win matrix and compute the exact global maximum likelihood estimation (MLE) across all models simultaneously.

Mathematical Derivation & Stationary Equations

Let $W_{i} = \sum_{j \neq i} w_{i j}$ and total matches between models $i$ and $j$ be $N_{i j} = w_{i j} + w_{j i}$ . We solve for stationary parameters $γ_{i} = \exp (θ_{i})$ iteratively:

γ_{i}^{(t + 1)} = \frac{W_{i}}{\sum_{j \neq i} \frac{N_{i j}}{γ_{i}^{(t)} + γ_{j}^{(t)}}}, R_{i} = 1000 + \frac{400}{\ln (10)} (\ln γ_{i} - \frac{1}{M} \sum_{k = 1}^{M} \ln γ_{k})

Convergence CriteriaIterations update until parameter changes satisfy

∥ γ^{(t + 1)} - γ^{(t)} ∥_{\infty} < 10^{- 6}

.

Standardized NormalizationFinal abilities are zero-centered in log-space and anchored to baseline

R_{base} = 1000

.

Uncertainty

3. Non-Parametric Bootstrap 95% Confidence Bounds

1,000 Replicates

Definition: Single-point ratings do not convey sample variance. We execute 1,000 non-parametric bootstrap replications across dataset splits to determine if model ranking differences are statistically significant:

{CI}_{95 %} (R_{i}) = [R_{i, 0.025}^{*}, R_{i, 0.975}^{*}], \pm {Margin}_{i} = \frac{R_{i, 0.975}^{*} - R_{i, 0.025}^{*}}{2}

1. Dataset-Split ResamplingSample

N

dataset splits with replacement from the empirical evaluation pool (

B = 1000

).

2. Tournament Re-SolvingThe MM solver re-computes bootstrap ratings

R_{i}^{(b)}

for each resampled tournament matrix.

3. Percentile BoundsEmpirical 95% confidence intervals are extracted from the 2.5th and 97.5th percentiles:

[R_{i, 0.025}^{*}, R_{i, 0.975}^{*}]

.

4. Reported Margin (

\pm CI

)Reported as

R_{i} \pm {CI}_{95 %}

where

{CI}_{95 %} = \frac{R_{i, 0.975}^{*} - R_{i, 0.025}^{*}}{2}

.

SCALE-INVARIANT EVALUATION

TabArena Normalized Improvability

Tabular benchmarks contain a mix of diverse tasks with varying entropy and scale. Following TabArena's error-normalized regret formulation, Improvability is computed strictly using error loss metrics ( $1 - AUC$ for binary classification, $Log-Loss$ for multiclass, and $RMSE$ for regression) rather than uncalibrated raw accuracy.

Scale-Invariant

4. Error-Loss Regret Formulation

TabArena Standard ↗

Definition: For a foundation model $m$ evaluated on benchmark dataset $d$ with task error loss $L (m, d)$ (where lower is better), Normalized Improvability evaluates relative error regret against the empirical suite ceiling loss $L_{d}^{*}$ and a naive baseline floor $L_{dummy}$ :

Improvability (m, d) = \frac{L (m, d) - L_{d}^{*}}{L_{dummy} - L_{d}^{*}} \times 100 %

Improvability (m, d)

Normalized fraction of headroom left on the table. Lower is better (0.0% indicates achieving the empirical suite ceiling).

L (m, d)

(Model Error Loss)Out-of-sample error loss on dataset

d

:

1 - AUC

for binary classification,

Log-Loss

for multiclass, and

RMSE

for regression.

L_{d}^{*}

(Suite Ceiling Loss)Lowest error loss achieved by any tested foundation model on dataset

d

:

L_{d}^{*} = \min_{m^{'}} L (m^{'}, d)

.

L_{dummy}

(Anchor Floor)Naive uninformative baseline error loss representing zero learned pattern signal.

Anchor Floors

5. Task-Specific Loss Metrics & Dummy Anchors

Zero-Signal Floor

Domain Formulation: TabArena uses task-appropriate loss metrics that are invariant to class imbalance and scale:

\begin{array}{l} {Regret}_{bin} (m, d) = \frac{(1 - {AUC}_{m, d}) - (1 - {AUC}_{d}^{*})}{0.50 - (1 - {AUC}_{d}^{*})} = \frac{{AUC}_{d}^{*} - {AUC}_{m, d}}{{AUC}_{d}^{*} - 0.50} \\ {Regret}_{multi} (m, d) = \frac{{LogLoss}_{m, d} - {LogLoss}_{d}^{*}}{\ln (K) - {LogLoss}_{d}^{*}} \\ {Regret}_{reg} (m, d) = \frac{{RMSE}_{m, d} - {RMSE}_{d}^{*}}{σ_{y} - {RMSE}_{d}^{*}} (σ_{y} = std (y)) \end{array}

Binary Classification (

1 - AUC

)Loss

L = 1 - AUC

evaluates ranking quality across all thresholds. Dummy anchor

L_{dummy} = 0.50

(random chance). Regret =

\frac{{AUC}_{d}^{*} - {AUC}_{m, d}}{{AUC}_{d}^{*} - 0.50}

.

Multiclass Classification (

Log-Loss

)Loss

L = LogLoss

(Cross-Entropy). Evaluates well-calibrated class probability distributions. Dummy anchor

L_{dummy} = \ln (K)

(uniform prior cross-entropy).

Regression (

RMSE

)Loss

L = RMSE (m, d)

. Dummy anchor

L_{dummy} = σ_{y} = std (y)

(naive sample mean predictor). Regret =

\frac{{RMSE}_{m, d} - {RMSE}_{d}^{*}}{σ_{y} - {RMSE}_{d}^{*}}

.

Aggregation

6. Macro-Suite Improvability Aggregation

Macro-Average

Definition: Because error regret is normalized between 0.0% and 100.0% regardless of native task units or target dynamic ranges, overall suite Improvability is the unweighted macro-average across all evaluation datasets:

Improvability (m) = \frac{1}{| D |} \sum_{d \in D} Improvability (m, d)

Classification ImprovabilityMacro-averaged across all binary and multiclass classification tasks in the suite.

Regression ImprovabilityMacro-averaged across all continuous regression tasks in the suite.

EXPERIMENTAL RIGOR

Evaluation Protocols & Runtime Telemetry

Experimental integrity safeguards, data leakage controls, and in-browser WebGPU execution benchmarks.

Integrity

7. IID & Non-IID Evaluation Protocol

Beyond IID paper ↗

Protocol Constraints: Following the application-aware outer-split framework introduced with BeyondArena and DataFoundry, splits are generated once during dataset curation, serialized as positional row indices, and consumed unchanged by the Python and browser runners. Context prompts contain only training rows; test rows remain strictly out-of-sample:

\hat{μ} = \frac{1}{K} \sum_{k = 1}^{K} s_{k}, {\hat{σ}}_{SE} = \frac{1}{\sqrt{K}} \sqrt{\frac{1}{K - 1} \sum_{k = 1}^{K} {(s_{k} - \hat{μ})}^{2}} (K = R \times 3)

Adaptive Repeated IIDIID tasks use three folds: 20 repeats below 500 effective training rows, 10 below 2,500, 3 below 250,000, and one below 1,000,000. Larger datasets use a single 250,000-row holdout. Classification folds preserve class proportions.

Grouped & Temporal Non-IIDGrouped splits keep entity IDs disjoint; temporal splits sort by time and ensure the training period never follows the test period.

Deterministic SerializationRepeated IID folds use seed 4267; million-row holdouts use seed 42. Persisted indices guarantee cross-language reproducibility.

Leakage ExclusionGroup, time, and schema-flagged leakage columns are excluded from the feature matrix while row identity remains stable.

Telemetry

8. In-Browser WebGPU vs Server Execution

WASM Runtime

Architecture: Unlike server-side baselines requiring cloud GPUs or virtual environments, Carla HQ's engine (TabICLv2) executes 100% client-side via WebGPU.

T_{fold} = T_{embed} + \sum_{l = 1}^{L} T_{layer}^{(l)} + T_{head}, Throughput = \frac{N_{eval}}{T_{inference}}

In-Browser WebGPU TimingMeasured locally using high-resolution performance timers (performance.now()). Zero network round-trips.

Warm KV-Cache AccelerationTransformer KV-cache buffers are reused across inference passes to minimize memory allocations.

RELATED BENCHMARK GUIDES

Explore Additional Benchmark Resources

BENCHMARK SUITE

Statistical Benchmark Methodology

Bradley-Terry Elo Rating Framework

The "Chess Tournament" Mental Model

The Matchup

Determining the Winner

Solving Global Ability

Tournament Scorecard (Pairwise Matchup Matrix)

Example: TabICLv2 vs. Google TabFM (Adult Census, Fold 3)

Elo Rating Difference (ΔR) Rule-of-Thumb

1. Bradley-Terry Log-Odds Formulation

2. Minorization-Maximization (MM) Algorithm

3. Non-Parametric Bootstrap 95% Confidence Bounds

TabArena Normalized Improvability

4. Error-Loss Regret Formulation

5. Task-Specific Loss Metrics & Dummy Anchors

6. Macro-Suite Improvability Aggregation

Evaluation Protocols & Runtime Telemetry

7. IID & Non-IID Evaluation Protocol

8. In-Browser WebGPU vs Server Execution

Evaluate Foundation Models Directly In Google Sheets

Statistical Benchmark Methodology

Bradley-Terry Elo Rating Framework

The "Chess Tournament" Mental Model

The Matchup

Determining the Winner

Solving Global Ability

Tournament Scorecard (Pairwise Matchup Matrix)

Example: TabICLv2 vs. Google TabFM (Adult Census, Fold 3)

Elo Rating Difference (ΔR) Rule-of-Thumb

1. Bradley-Terry Log-Odds Formulation

2. Minorization-Maximization (MM) Algorithm

3. Non-Parametric Bootstrap 95% Confidence Bounds

TabArena Normalized Improvability

4. Error-Loss Regret Formulation

5. Task-Specific Loss Metrics & Dummy Anchors

6. Macro-Suite Improvability Aggregation

Evaluation Protocols & Runtime Telemetry

7. IID & Non-IID Evaluation Protocol

8. In-Browser WebGPU vs Server Execution

Explore Additional Benchmark Resources

21 Evaluation Datasets ↗

Global Matchup Matrix ↗

ML Metrics Guide ↗

Beyond IID ↗

Evaluate Foundation Models Directly In Google Sheets