Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Foundation Inference Models that Learn ODEs In-Context

Maximilian Mauel, Manuel Hinz, Patrick Seifner, David Berghaus, Ramses J. Sanchez

TL;DR

The paper tackles identifying the vector field $\mathbf{f}$ in the continuous-time dynamics $\dfrac{d\mathbf{x}(t)}{dt}=\mathbf{f}(t,\mathbf{x}(t))$ from sparse and noisy time-series data. It introduces FIM-ODE, a foundation inference model that leverages a neural operator (DeepONet) with a Transformer-based branch-net to perform zero-shot, in-context inference of $\hat{\mathbf{f}}(\mathbf{x})$ from data $\mathcal{D}$. On synthetic ODEs with vector fields drawn from multivariate polynomials up to degree $3$ in dimension $D\le 3$, FIM-ODE outperforms the pretrained ODEFormer in reconstruction ($R^2=0.90$ vs $0.65$) and generalization ($R^2=0.26$ vs $0.19$), and benefits from multiple context trajectories to improve global vector-field estimates. This work highlights the potential of foundation inference models for robust, interpretable ODE identification from limited data and outlines future benchmarks (e.g., ODEBench) and applications to latent dynamics and reaction networks.

Abstract

Ordinary differential equations (ODEs) describe dynamical systems evolving deterministically in continuous time. Accurate data-driven modeling of systems as ODEs, a central problem across the natural sciences, remains challenging, especially if the data is sparse or noisy. We introduce FIM-ODE (Foundation Inference Model for ODEs), a pretrained neural model designed to estimate ODEs zero-shot (i.e., in context) from sparse and noisy observations. Trained on synthetic data, the model utilizes a flexible neural operator for robust ODE inference, even from corrupted data. We empirically verify that FIM-ODE provides accurate estimates, on par with a neural state-of-the-art method, and qualitatively compare the structure of their estimated vector fields.

Towards Foundation Inference Models that Learn ODEs In-Context

TL;DR

The paper tackles identifying the vector field in the continuous-time dynamics from sparse and noisy time-series data. It introduces FIM-ODE, a foundation inference model that leverages a neural operator (DeepONet) with a Transformer-based branch-net to perform zero-shot, in-context inference of from data . On synthetic ODEs with vector fields drawn from multivariate polynomials up to degree in dimension , FIM-ODE outperforms the pretrained ODEFormer in reconstruction ( vs ) and generalization ( vs ), and benefits from multiple context trajectories to improve global vector-field estimates. This work highlights the potential of foundation inference models for robust, interpretable ODE identification from limited data and outlines future benchmarks (e.g., ODEBench) and applications to latent dynamics and reaction networks.

Abstract

Ordinary differential equations (ODEs) describe dynamical systems evolving deterministically in continuous time. Accurate data-driven modeling of systems as ODEs, a central problem across the natural sciences, remains challenging, especially if the data is sparse or noisy. We introduce FIM-ODE (Foundation Inference Model for ODEs), a pretrained neural model designed to estimate ODEs zero-shot (i.e., in context) from sparse and noisy observations. Trained on synthetic data, the model utilizes a flexible neural operator for robust ODE inference, even from corrupted data. We empirically verify that FIM-ODE provides accurate estimates, on par with a neural state-of-the-art method, and qualitatively compare the structure of their estimated vector fields.
Paper Structure (4 sections, 1 equation, 2 figures, 1 table)

This paper contains 4 sections, 1 equation, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Foundation Inference Model
  3. Experiments
  4. Conclusions

Figures (2)

  • Figure 1: Vector field estimates of FIM-ODE from $1$, $5$ and $9$ context trajectories of the same system. The estimates improve, given more trajectories.
  • Figure 2: Comparison of vector field estimates to the ground-truth, given a single trajectory. ODEFormer predicts a complex global pattern, while FIM-ODE focuses on accurate predictions near the provided trajectory.