# Carla HQ Statistical Benchmark Methodology

> Mathematical foundations, stationary Bradley-Terry tournament solving, bootstrap confidence intervals, and adaptive cross-validation protocols.

## 1. Bradley-Terry Elo Rating Model

In tabular benchmark evaluation, naive average ranks across heterogeneous datasets suffer from sensitivity to outliers and fail to reflect non-linear capability gaps. Carla HQ employs the **Bradley-Terry probability model** to estimate latent model skill ratings $p_i$ from paired tournament outcomes:

$$P(\text{Model } i \succ \text{Model } j) = \frac{p_i}{p_i + p_j}$$

We solve for optimal skill vector $\mathbf{p}$ via iterative Minorization-Maximization (MM):

$$p_i^{(t+1)} = \frac{W_i}{\sum_{j \ne i} \frac{N_{ij}}{p_i^{(t)} + p_j^{(t)}}}$$

Where $W_i$ is the total wins accumulated by model $i$ across all dataset splits, and $N_{ij} = W_{ij} + W_{ji}$ is the total number of head-to-head encounters between model $i$ and model $j$.

Skill parameters are mapped to human-readable Elo ratings centered at 1,000:

$$\text{Elo}_i = 1000 + \frac{400}{\ln 10} \ln p_i$$

## 2. Bootstrap Confidence Intervals

To quantify ranking stability and guard against small-sample variance, Carla runs **1,000 bootstrap resamples** with replacement over the benchmark dataset corpus. For each bootstrap iteration $b \in [1, 1000]$, the full win matrix $W^{(b)}$ is reconstructed and solved. The 95% confidence interval is extracted via the empirical 2.5th and 97.5th percentiles:

$$\text{CI}_{95} = [\text{Elo}_{2.5\%}, \text{Elo}_{97.5\%}]$$

## 3. Scale-Invariant Improvability (TabArena Regret)

Improvability quantifies how much performance remains on the table relative to the best known model on that specific dataset (the empirical ceiling $L^*_d$) versus an uninformed baseline ($L_{\text{dummy}}$):

$$\text{Improvability}(m, d) = \frac{L(m, d) - L^*_d}{L_{\text{dummy}} - L^*_d} \times 100\%$$

- **Binary Classification:** Loss $L = 1 - \text{AUC}$, $L_{\text{dummy}} = 0.50$.

- **Multiclass Classification:** Loss $L = \text{Log-Loss}$, $L_{\text{dummy}} = \ln(K)$ (where $K$ is the number of target classes).

- **Regression:** Loss $L = 1 - R^2$, $L_{\text{dummy}} = 1.00$ ($R^2 = 0.00$).

## 4. Evaluation Split Protocol

All benchmarks use repeated $k$-fold cross-validation with persisted random seeds. For tabular datasets, repeated 3-fold or 5-fold cross-validation is evaluated. For time-series tasks, rolling sequential evaluation windows without lookahead leakage are enforced.

## 5. Execution Environments & Testbeds

- **In-Browser WebGPU (Carla Engine):** Evaluated inside Chromium via WebGPU compute shaders and WASM preprocessors with zero server interaction.

- **Reference Python Foundation Models:** Evaluated on dedicated NVIDIA A100/H100 GPUs using official PyTorch model implementations.