CODES: Benchmarking Coupled ODE Surrogates

TL;DR

By offering a fair and multi-faceted comparison, CODES helps researchers select the most suitable surrogate for their specific dataset and application while deepening the understanding of surrogate learning behaviour.

Abstract

We introduce CODES, a benchmark for comprehensive evaluation of surrogate architectures for coupled ODE systems. Besides standard metrics like mean squared error (MSE) and inference time, CODES provides insights into surrogate behaviour across multiple dimensions like interpolation, extrapolation, sparse data, uncertainty quantification and gradient correlation. The benchmark emphasizes usability through features such as integrated parallel training, a web-based configuration generator, and pre-implemented baseline models and datasets. Extensive documentation ensures sustainability and provides the foundation for collaborative improvement. By offering a fair and multi-faceted comparison, CODES helps researchers select the most suitable surrogate for their specific dataset and application while deepening our understanding of surrogate learning behaviour.

Figures (8)

• Figure 1: Example trajectories of a datapoint (29 chemicals, 100 timesteps) in the osu2008 dataset.
• Figure 2: Quantities over time per model, averaged across the test set. Left: Mean and median relative error. Right: Predictive uncertainty (DeepEnsemble, $n=5, 1\sigma$) and mean absolute error.
• Figure 3: Model errors for the modalities interpolation, extrapolation and sparse.
• Figure 4: Smoothed histogram plots of the distribution of the relative errors per model alongside their mean and median relative errors.
• Figure 5: Smoothed histogram plots of the relative error distribution of FullyConnected for each quantity (chemical) in the osu2008 dataset.