Skip to main content
CARLA HQ
PAIRWISE TIME SERIES BENCHMARKfev-bench • 100 TasksScaled Quantile Loss (SQL)

ci-proto vs Simons-297M Benchmark

Empirical head-to-head evaluation of ci-proto (Continuous Intelligence) versus Simons-297M (Abel AI Lab) across 100 standardized time series forecasting tasks from AutoGluon fev-bench.

HEAD-TO-HEAD WINS?Direct pairwise task matchup wins where the model achieved lower Scaled Quantile Loss (SQL) across 9 probability levels (p10–p90).
ci-proto65
:
35Simons-297M
ci-proto leads 65.0% of tasks
GLOBAL SQL WIN RATE?Average win rate (%) across competing models in the fev-bench suite (35-model official public view; 52-model full pool).
ci-proto90.0%
Simons-297M82.0%
Global benchmark win rate
SQL SKILL SCORE?Normalized percentage improvement in SQL relative to the Seasonal Naive reference baseline (0% = baseline, higher is better).
ci-proto+51.5%
Simons-297M+48.4%
Improvement vs Seasonal Naive
MEDIAN LATENCY?Median inference time per 100 series in seconds. Lower is faster.
ci-proto0.53s
Simons-297M0.94s
Per 100 time series
ARCHITECTURAL SPECIFICATIONS

Model Architecture & Execution Profiles

Comparing underlying architectures, tokenization mechanics, and in-browser deployment feasibility.

Foundation Model

ci-proto

Continuous Intelligence

Continuous Intelligence's flagship Mamba-3 state-space foundation model released under the ci-proto-license, ranked #1 on the 35-model official fev-bench view (90.00% SQL win rate; #3 overall at 84.37% across all 52 methods) and ported by Carla for local browser WebGPU execution.

ArchitectureWeight-Tied Mamba-3 State-Space Foundation Model + VariateMixer
Tokenization32-Step Continuous Patch Projections with Calendar & Differencing Channels
Context WindowUp to 32,768 steps (64× baseline; 16,384 in WebGPU v1; 2M+ upstream long ckpt)
In-Browser RuntimeBrowser WebGPU Live Demo (ONNX + Rust/WASM) / PyTorch GPU (0.53s benchmark)
Foundation Model

Simons-297M

Abel AI Lab

Flagship 297M foundation model from Abel AI Lab, ranking #5 on the official fev-bench leaderboard with 81.97% SQL win rate and 48.41 skill score.

ArchitectureCausal Decoder-Only Transformer (297M parameters)
TokenizationContinuous Multi-Frequency Patch Tokenization
Context WindowUp to 16,384 steps
In-Browser RuntimeServer / PyTorch GPU (0.94s median latency)
LicenseApache 2.0
EMPIRICAL BENCHMARK BREAKDOWN

Task-by-Task Comparison Across 100 fev Tasks

Detailed breakdown of Scaled Quantile Loss (SQL) for each task. Lower loss indicates superior probabilistic calibration. Tasks featured in Carla's interactive Google Sheets catalog are highlighted with direct sheet links.

Benchmark TaskDataset Categoryci-proto (SQL)Simons-297M (SQL)AdvantageWinnerSheet
BizITobs Cloud Infrastructure Telemetrybizitobs_l2c_5TCarla Demo0.32980.3037+7.9%Simons-297M
Boomlet Smart Meter Building Energyboomlet_1282Carla Demo0.39270.4043+2.9%ci-proto
Boomlet Smart Meter Building Energyboomlet_1676Carla Demo0.54900.5539+0.9%ci-proto
Boomlet Smart Meter Building Energyboomlet_619Carla Demo0.30110.2984+0.9%Simons-297M
Electricity Transformer TemperatureETT_15TCarla Demo0.52400.5335+1.8%ci-proto
Electricity Transformer TemperatureETT_1HCarla Demo0.82580.7744+6.2%Simons-297M
Global Energy Forecasting Competition (GEFCom 2014)proenfo_gfc14Carla Demo0.23380.3581+34.7%ci-proto
Global Sovereign Carbon Dioxide (CO2) Emissionsworld_co2_emissionsCarla Demo2.66662.6496+0.6%Simons-297M
Hospital Emergency Admissionshospital_admissions_1DCarla Demo0.55420.5540+0.0%Simons-297M
Hospital Emergency Admissionshospital_admissions_1WCarla Demo0.57800.5784+0.1%ci-proto
Jena Climate & Weather Telemetryjena_weather_1HCarla Demo0.27660.3449+19.8%ci-proto
Monash High-Density Time Series BenchmarkM_DENSE_1DCarla Demo0.66880.6413+4.1%Simons-297M
Nord Pool Day-Ahead Electricity Pricesepf_npCarla Demo0.55930.6328+11.6%ci-proto
Rohlik E-Grocery Daily Fulfillment Ordersrohlik_orders_1DCarla Demo0.87620.8721+0.5%Simons-297M
Rossmann Store Daily & Weekly Salesrossmann_1DCarla Demo0.22590.2715+16.8%ci-proto
Rossmann Store Daily & Weekly Salesrossmann_1WCarla Demo0.25150.2785+9.7%ci-proto
Solar Photovoltaic Power Output with Weathersolar_with_weather_1HCarla Demo0.67620.7127+5.1%ci-proto
UCI Air Quality Chemical Gas Sensor Telemetryuci_air_quality_1HCarla Demo0.73540.7892+6.8%ci-proto
US Macroeconomic Consumption & Disposable Incomeus_consumption_1YCarla Demo3.03063.5054+13.5%ci-proto
australian tourismaustralian_tourismfev-bench0.69360.7238+4.2%ci-proto—
bizitobs l2c 1Hbizitobs_l2c_1Hfev-bench0.30310.2847+6.1%Simons-297M—
boomlet 1062boomlet_1062fev-bench0.54020.5381+0.4%Simons-297M—
boomlet 1209boomlet_1209fev-bench0.63100.6474+2.5%ci-proto—
boomlet 1225boomlet_1225fev-bench0.18330.1831+0.1%Simons-297M—
boomlet 1230boomlet_1230fev-bench1.13801.1495+1.0%ci-proto—
boomlet 1487boomlet_1487fev-bench0.40150.3981+0.8%Simons-297M—
boomlet 1631boomlet_1631fev-bench0.56860.5685+0.0%Simons-297M—
boomlet 1855boomlet_1855fev-bench0.44590.4473+0.3%ci-proto—
boomlet 1975boomlet_1975fev-bench0.11250.1086+3.5%Simons-297M—
boomlet 2187boomlet_2187fev-bench0.70570.7088+0.4%ci-proto—
boomlet 285boomlet_285fev-bench0.26840.3243+17.2%ci-proto—
boomlet 772boomlet_772fev-bench0.27440.2764+0.7%ci-proto—
boomlet 963boomlet_963fev-bench0.71020.7023+1.1%Simons-297M—
ecdc iliecdc_ilifev-bench2.13732.2975+7.0%ci-proto—
entsoe 15Tentsoe_15Tfev-bench0.31460.4014+21.6%ci-proto—
entsoe 1Hentsoe_1Hfev-bench0.29660.3823+22.4%ci-proto—
entsoe 30Tentsoe_30Tfev-bench0.29470.3820+22.9%ci-proto—
epf beepf_befev-bench0.48010.4885+1.7%ci-proto—
epf deepf_defev-bench0.47770.4685+1.9%Simons-297M—
epf frepf_frfev-bench0.31330.3336+6.1%ci-proto—
epf pjmepf_pjmfev-bench0.41340.3450+16.5%Simons-297M—
ercot 1Dercot_1Dfev-bench0.80040.8131+1.6%ci-proto—
ercot 1Hercot_1Hfev-bench0.94510.9696+2.5%ci-proto—
ercot 1Mercot_1Mfev-bench0.73810.7448+0.9%ci-proto—
ercot 1Wercot_1Wfev-bench0.94310.9254+1.9%Simons-297M—
ETT 1DETT_1Dfev-bench1.09811.1321+3.0%ci-proto—
ETT 1WETT_1Wfev-bench2.30022.2689+1.4%Simons-297M—
favorita stores 1Dfavorita_stores_1Dfev-bench0.88860.9158+3.0%ci-proto—
favorita stores 1Mfavorita_stores_1Mfev-bench1.73343.1184+44.4%ci-proto—
favorita stores 1Wfavorita_stores_1Wfev-bench1.87924.3522+56.8%ci-proto—
favorita transactions 1Dfavorita_transactions_1Dfev-bench0.66920.8330+19.7%ci-proto—
favorita transactions 1Mfavorita_transactions_1Mfev-bench1.01121.0208+0.9%ci-proto—
favorita transactions 1Wfavorita_transactions_1Wfev-bench1.18351.1582+2.1%Simons-297M—
fred md 2025/ceefred_md_2025/ceefev-bench3.31733.4422+3.6%ci-proto—
fred md 2025/macrofred_md_2025/macrofev-bench5.10285.0608+0.8%Simons-297M—
fred qd 2025/ceefred_qd_2025/ceefev-bench1.77801.7119+3.7%Simons-297M—
fred qd 2025/macrofred_qd_2025/macrofev-bench3.38723.3605+0.8%Simons-297M—
gvargvarfev-bench0.56600.5571+1.6%Simons-297M—
hermeshermesfev-bench0.59860.6369+6.0%ci-proto—
hierarchical sales 1Dhierarchical_sales_1Dfev-bench0.54670.5474+0.1%ci-proto—
hierarchical sales 1Whierarchical_sales_1Wfev-bench0.61200.6171+0.8%ci-proto—
hospitalhospitalfev-bench0.66750.6817+2.1%ci-proto—
jena weather 10Tjena_weather_10Tfev-bench0.34230.3364+1.7%Simons-297M—
jena weather 1Djena_weather_1Dfev-bench1.09871.0869+1.1%Simons-297M—
kdd cup 2022 10Tkdd_cup_2022_10Tfev-bench0.42400.4148+2.2%Simons-297M—
kdd cup 2022 1Dkdd_cup_2022_1Dfev-bench0.71550.6999+2.2%Simons-297M—
kdd cup 2022 30Tkdd_cup_2022_30Tfev-bench0.41520.4222+1.7%ci-proto—
LOOP SEATTLE 1DLOOP_SEATTLE_1Dfev-bench0.74320.7442+0.1%ci-proto—
LOOP SEATTLE 1HLOOP_SEATTLE_1Hfev-bench0.56900.6003+5.2%ci-proto—
LOOP SEATTLE 5TLOOP_SEATTLE_5Tfev-bench0.52660.5305+0.7%ci-proto—
M DENSE 1HM_DENSE_1Hfev-bench0.52840.5363+1.5%ci-proto—
m5 1Dm5_1Dfev-bench0.70180.7006+0.2%Simons-297M—
m5 1Mm5_1Mfev-bench0.97380.9721+0.2%Simons-297M—
m5 1Wm5_1Wfev-bench0.90390.8658+4.2%Simons-297M—
proenfo gfc12proenfo_gfc12fev-bench0.48980.5974+18.0%ci-proto—
proenfo gfc17proenfo_gfc17fev-bench0.33090.4437+25.4%ci-proto—
redset 15Tredset_15Tfev-bench0.62940.6420+2.0%ci-proto—
redset 1Hredset_1Hfev-bench1.25631.2709+1.1%ci-proto—
redset 5Tredset_5Tfev-bench0.63180.6264+0.9%Simons-297M—
restaurantrestaurantfev-bench0.66840.6898+3.1%ci-proto—
rohlik orders 1Wrohlik_orders_1Wfev-bench1.18151.2388+4.6%ci-proto—
rohlik sales 1Drohlik_sales_1Dfev-bench0.90270.8775+2.8%Simons-297M—
rohlik sales 1Wrohlik_sales_1Wfev-bench1.13411.1197+1.3%Simons-297M—
solar 1Dsolar_1Dfev-bench0.61900.6044+2.4%Simons-297M—
solar 1Wsolar_1Wfev-bench0.72801.0710+32.0%ci-proto—
solar with weather 15Tsolar_with_weather_15Tfev-bench0.58870.6705+12.2%ci-proto—
SZ TAXI 15TSZ_TAXI_15Tfev-bench0.39200.3928+0.2%ci-proto—
SZ TAXI 1HSZ_TAXI_1Hfev-bench0.39120.4838+19.1%ci-proto—
uci air quality 1Duci_air_quality_1Dfev-bench1.03501.0783+4.0%ci-proto—
uk covid nation 1D/cumulativeuk_covid_nation_1D/cumulativefev-bench5.49957.1754+23.4%ci-proto—
uk covid nation 1D/newuk_covid_nation_1D/newfev-bench1.95602.0975+6.7%ci-proto—
uk covid nation 1W/cumulativeuk_covid_nation_1W/cumulativefev-bench2.64743.1310+15.4%ci-proto—
uk covid nation 1W/newuk_covid_nation_1W/newfev-bench3.78744.8572+22.0%ci-proto—
uk covid utla 1D/newuk_covid_utla_1D/newfev-bench3.49254.0308+13.4%ci-proto—
uk covid utla 1W/cumulativeuk_covid_utla_1W/cumulativefev-bench15.599514.9974+3.9%Simons-297M—
us consumption 1Mus_consumption_1Mfev-bench1.43821.4686+2.1%ci-proto—
us consumption 1Qus_consumption_1Qfev-bench1.55821.6586+6.1%ci-proto—
walmartwalmartfev-bench0.63310.6984+9.3%ci-proto—
world life expectancyworld_life_expectancyfev-bench1.06611.0704+0.4%ci-proto—
world tourismworld_tourismfev-bench2.68402.8373+5.4%ci-proto—
ZERO-SERVER CLIENT-SIDE EXECUTION

Run Time Series Forecasts Directly in Google Sheets

Carla Engine compiles Mistral's Timestral-123M foundation model into client-side WebGPU shaders. Execute probabilistic forecasts with zero cloud API keys, zero infrastructure setup, and 100% data privacy.

FREQUENTLY ASKED QUESTIONS

ci-proto vs Simons-297M FAQ

How does ci-proto compare to Simons-297M in forecasting accuracy?

Across the 100 standardized tasks in fev-bench, ci-proto achieved lower Scaled Quantile Loss (SQL) on 65 tasks, recording a head-to-head win rate of 65.0%.

What is the latency difference between ci-proto and Simons-297M?

ci-proto records a median end-to-end latency of 0.53s per 100 time series, compared to 0.94s for Simons-297M.

Can ci-proto or Simons-297M run locally in the browser?

Yes. Carla ported ci-proto to run 100% locally in the browser using stock ONNX Runtime WebGPU and Rust/WASM (chunked loop-free Mamba-3 scan + lazy multivariate VariateMixer loading), testable in our live browser demo without sending series data to a server.