Time Series Forecasting Model Leaderboard & Matrix

fev-bench is an open-source empirical time-series forecasting benchmark developed by AutoGluon researchers. It evaluates 41 foundation models, deep learning architectures, gradient-boosted trees, and statistical baselines across 100 standardized, diverse real-world tasks using rolling evaluation windows and probabilistic metrics including Scaled Quantile Loss (SQL) and Mean Absolute Scaled Error (MASE).

Point forecasts (such as RMSE or MAE) do not account for forecast uncertainty. Scaled Quantile Loss (SQL) evaluates probabilistic forecasts across 9 quantile levels (p10 to p90), penalizing both point prediction deviations and quantile miscalibration, while scaling errors by the in-sample mean absolute difference to allow scale-invariant comparison across heterogeneous datasets.

Timestral-209M from Mistral AI achieves the #1 global ranking with an 87.72% aggregate win rate and 50.39% skill score relative to Seasonal Naive. Timestral-123M (Carla Engine) ranks #2 globally with an 86.85% win rate and 0.32s median latency.

Yes. Carla Engine executes Mistral's Timestral-123M 100% locally in-browser using quantized WebGPU WGSL compute shaders. Sensitive enterprise telemetry, energy meter data, financial records, and medical admissions never leave the user's machine.

While gradient boosted trees excel on static tabular data, time series forecasting requires modeling temporal dynamics and multi-step joint distributions. Without manual task-specific lag feature engineering, LightGBM achieves only an 17.8% win rate on fev-bench, whereas zero-shot foundation models (like Timestral) score 86%+ out of the box with zero feature engineering.