CARLA HQ
Add to Chrome
EVALUATION METHODOLOGY

Machine Learning & In-Browser Telemetry Metrics

A comprehensive technical guide to the statistical evaluation metrics, loss formulations, and local WebGPU execution telemetry reported across Carla HQ dataset benchmarks.

Jump to section:Classification MetricsRegression MetricsExecution TelemetryTournament Methodology →Google ML References
PREDICTIVE CLASSIFICATION

Classification Metrics

Classification tasks involve predicting discrete categorical labels (e.g. spam detection, loan default, disease diagnosis). Because class distributions in real tabular data are frequently imbalanced, we evaluate models across threshold-dependent and ranking-invariant criteria.

Classification

Predictive Accuracy

Google ML Guide ↗

Definition: Predictive Accuracy is the fraction of total test instances that were correctly classified by the model across all classes.

Accuracy = (TP + TN) / (TP + TN + FP + FN) = Total Correct / Total Instances
When to useBalanced class distributions where false positives and false negatives carry equal operational consequences.
Key PitfallHighly misleading on imbalanced datasets (e.g., in a 99:1 fraud dataset, predicting all negatives yields 99% accuracy while catching zero fraud).
Classification

ROC-AUC (Area Under the Receiver Operating Characteristic)

Google ML Guide ↗

Definition: ROC-AUC measures the two-dimensional area underneath the entire ROC curve (True Positive Rate vs. False Positive Rate) across all possible classification decision thresholds (from 0.0 to 1.0).

TPR = TP / (TP + FN), FPR = FP / (FP + TN), AUC = Area under TPR(FPR) curve
InterpretationAn AUC of 1.0 represents a perfect model; 0.5 corresponds to random chance. AUC equals the probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.
Key AdvantageThreshold-invariant and scale-invariant. Evaluates ranking capability regardless of where the final decision threshold is set.
Classification

F1 Score / F-Measure

Google ML Guide ↗

Definition: The F1 Score (also known as the balanced F-measure) is the harmonic mean of Precision and Recall. Unlike the arithmetic mean, the harmonic mean heavily penalizes extreme imbalances between Precision and Recall.

F1 = 2 * (Precision * Recall) / (Precision + Recall) = 2*TP / (2*TP + FP + FN)
When to useImbalanced classification benchmarks where both false alarms and missed detections impose substantial domain costs.
Score RangeRanges from 0.0 (worst) to 1.0 (perfect precision and recall).
Classification

Precision (Positive Predictive Value)

Google ML Guide ↗

Definition: Precision answers: "Of all instances the model predicted as positive, what proportion was actually positive?"

Precision = TP / (TP + FP)
High Precision PriorityCritical in settings where false positives are costly (e.g., spam filters where legitimate customer emails must never be filtered).
Trade-offIncreasing precision typically lowers recall (the model becomes more conservative).
Classification

Recall (Sensitivity / True Positive Rate)

Google ML Guide ↗

Definition: Recall answers: "Of all instances that were actually positive, what proportion did the model successfully identify?"

Recall = TP / (TP + FN)
High Recall PriorityCritical in diagnostic and safety applications (e.g., oncology screening or defect detection where missing a true positive is catastrophic).
Trade-offMaximizing recall can increase false alarms (false positives).
Classification

Log-Loss (Cross-Entropy Loss)

Definition: Log-Loss evaluates the probabilistic calibration and confidence of predicted class probabilities rather than just the final discrete labels.

Log-Loss = - (1/N) * sum( y_i * ln(p_i) + (1 - y_i) * ln(1 - p_i) )
InterpretationLower is better (0.0 represents perfect calibration and absolute confidence). Heavily penalizes confident predictions that turn out to be wrong.
Foundation Model RelevanceDirectly measures TabICLv2's posterior predictive calibration over in-context tabular distributions.
CONTINUOUS QUANTITATIVE PREDICTION

Regression Metrics

Regression tasks involve forecasting continuous numerical values (e.g., house sales prices, employee compensation, energy demand). Metrics measure the magnitude, distribution, and scale of residual errors between predicted values and observed targets.

Regression

R² Score (Coefficient of Determination)

Definition: R² measures the proportion of variance in the target dependent variable that is explained by the predictive features in the model, relative to a baseline model that simply predicts the mean target value.

R^2 = 1 - (SS_res / SS_tot) = 1 - ( sum((y_i - y_hat_i)^2) / sum((y_i - y_mean)^2) )
InterpretationAn R² of 1.0 indicates 100% of variance explained; 0.0 indicates performance identical to predicting the constant mean; negative values indicate performance worse than the sample mean.
Scale IndependenceUnitless metric that allows standardized performance comparison across datasets with wildly different target magnitudes.
Regression

Mean Absolute Error (MAE)

Definition: MAE is the arithmetic average of the absolute differences between the predicted values and the true target values.

MAE = (1/n) * sum( |y_i - y_hat_i| )
InterpretabilityExpressed in the exact same units as the target variable (e.g., $15,000 MAE on a house price dataset means predictions are off by $15k on average).
RobustnessRobust to extreme outliers because errors are not squared.
Regression

Root Mean Squared Error (RMSE)

Definition: RMSE is the square root of the average squared differences between predicted and observed values.

RMSE = sqrt( (1/n) * sum( (y_i - y_hat_i)^2 ) )
Outlier SensitivityBecause errors are squared before averaging, large individual prediction mistakes are penalized far more heavily than in MAE.
UnitsExpressed in original target units. When RMSE is significantly larger than MAE, it indicates the presence of large variance in error distribution.
Regression

Mean Absolute Percentage Error (MAPE)

Definition: MAPE computes the average relative percentage deviation of predictions relative to actual observed ground truth.

MAPE = (100% / n) * sum( |(y_i - y_hat_i) / y_i| )
Business UtilityEasy to communicate to non-technical stakeholders (e.g. "our forecasts are within 4.2% on average").
LimitationUndefined when true target is 0, and heavily penalizes positive over-predictions more than negative under-predictions.
Regression

Root Mean Squared Logarithmic Error (RMSLE)

Definition: RMSLE measures the ratio between predicted and actual values by applying a logarithmic transformation (ln(x + 1)) prior to calculating squared error.

RMSLE = sqrt( (1/n) * sum( (ln(1 + y_hat_i) - ln(1 + y_i))^2 ) )
Exponential Scale InvarianceIdeal for targets with exponential distributions (e.g., website traffic, salaries, house values).
Asymmetric PenaltyPenalizes underestimations (predicting too low) more heavily than overestimations.
LOCAL CLIENT SANDBOX

Execution Telemetry

Carla HQ executes tabular foundation models (TabICLv2) 100% locally inside the browser's JavaScript / WebGPU execution environment. We benchmark latency and throughput to guarantee zero-lag spreadsheet interaction.

Telemetry

Split Latency / Mean Split Time

Definition: The end-to-end wall-clock time required for TabICLv2 to encode the split's in-context training rows, execute transformer self-attention and cross-attention blocks via WebGPU shaders, and generate predictions for its out-of-sample test rows.

Zero Server LatencyBecause inference executes locally via WebGPU on the user's GPU hardware, there are zero network hops, zero API queue delays, and zero server round-trip latency.
In-Context InferenceRequires zero iterative gradient training epochs. Model fits context and predicts in a single forward pass.
Protocol

IID & Non-IID Evaluation Protocol

Beyond IID paper ↗

Protocol: Split indices are generated once during data curation and persisted with each dataset. IID tasks use repeated 3-fold evaluation: 20 repeats below 500 effective training rows, 10 below 2,500, 3 below 250,000, and one above that threshold. Datasets with at least one million rows use a single 250,000-row test holdout. Classification folds are stratified; grouped and temporal tasks instead use disjoint-entity or chronological out-of-time partitions.

Cross-Runner DeterminismPython and browser runners consume the same positional row indices, so every model sees identical train and test observations.
Leakage-Safe RegimesGroup identifiers remain disjoint and temporal training rows precede test rows. Group, time, and flagged leakage columns are excluded from model features.