Common Task Framework For a Critical Evaluation of Scientific Machine Learning Algorithms

TL;DR

The paper introduces the Common Task Framework (CTF) for scientific machine learning to enable objective, multi-metric evaluation on dynamic systems. It provides curated KS and Lorenz benchmarks, a 12-task scoring scheme, and a radar-based performance visualization, with withheld-test leaderboards to curb reporting bias. Through comprehensive comparisons across twelve methods and baselines, it demonstrates task-specific strengths and limitations, advocating for a community-driven expansion to include more datasets and control tasks. The long-term aim is to raise rigor and reproducibility in scientific ML by standardizing evaluations on hidden test sets and fostering open, transparent benchmarking.

Abstract

Machine learning (ML) is transforming modeling and control in the physical, engineering, and biological sciences. However, rapid development has outpaced the creation of standardized, objective benchmarks - leading to weak baselines, reporting bias, and inconsistent evaluations across methods. This undermines reproducibility, misguides resource allocation, and obscures scientific progress. To address this, we propose a Common Task Framework (CTF) for scientific machine learning. The CTF features a curated set of datasets and task-specific metrics spanning forecasting, state reconstruction, and generalization under realistic constraints, including noise and limited data. Inspired by the success of CTFs in fields like natural language processing and computer vision, our framework provides a structured, rigorous foundation for head-to-head evaluation of diverse algorithms. As a first step, we benchmark methods on two canonical nonlinear systems: Kuramoto-Sivashinsky and Lorenz. These results illustrate the utility of the CTF in revealing method strengths, limitations, and suitability for specific classes of problems and diverse objectives. Next, we are launching a competition around a global real world sea surface temperature dataset with a true holdout dataset to foster community engagement. Our long-term vision is to replace ad hoc comparisons with standardized evaluations on hidden test sets that raise the bar for rigor and reproducibility in scientific ML.

Figures (10)

• Figure 1: The twelve-axis radar plot characterizes a method's performance across all tasks on a dataset, and provides a visual performance profile. The axes correspond to the various tasks associated with forecasting and reconstruction with noise, limited data and parametric dependency. The chart shows the top four performing metrics on the KS and the Lorenz dataset scored against their reference baselines: constant zero and average prediction respectively.
• Figure 2: The CTF Evaluation framework scores the performance of methods on (a) the Lorenz dynamical system and (b) the Kuramoto-Sivashinsky partial differential equation. (c) Data is collected and organized into matrices which is then split into testing and training sets. RMSE errors are computed for reconstruction and short-time forecasting, while the spectral error computes the statistics of long-time forecasting (spatial or temporal). (d) Forecasting and reconstruction tasks are evaluated on noise-free, low-noise and high-noise data. Methods are also evaluated when (e) only limited data is available and (f) for reconstruction of parametrically dependent data.
• Figure 3: Ranked average scores of each model on the KS and Lorenz Dataset.
• Figure 4: Model performances for each metric on each dataset (mean $\pm$ std).
• Figure 5: Top three performing models per metric on the (a) Lorenz and (b) KS dataset. The blue baseline line here corresponds to the constant zero prediction. This baseline is not producing a score of zero in long-time predictions for the Lorenz dataset due to the different long-time evaluation methods used for KS and Lorenz. KS uses spectral $L_2$-error whereas Lorenz uses histogram $L_2$-error.