Generalizing Dynamics Modeling More Easily from Representation Perspective

Abstract

Learning system dynamics from observations is a critical problem in many applications over various real-world complex systems, e.g., climate, ecology, and fluid systems. Recently, neural dynamics modeling method have become a prevalent solution that embeds the object's observations into a latent space before learning dynamics using neural methods such as neural Ordinary Differential Equations (ODE). Existing dynamics modeling methods induce a specific model for each observation of different complex systems, resulting in poor generalization across systems. Inspired by the great success of pre-trained models, we conduct a generalized Pre-trained Dynamics EncoDER (PDEDER) which can embed the original state observations into a latent space where the dynamics can be captured more easily. To conduct the generalized PDEDER, we pre-train any Pre-trained Language Model (PLM) by minimizing the Lyapunov exponent objective, which constrains the chaotic behavior of governing dynamics learned in the latent space. By penalizing the divergence of embedded observations, our PDEDER promotes locally stable and well-structured latent dynamics, thereby facilitating more effective dynamics modeling than in the original observation space. In addition, we incorporate reconstruction and forecasting objectives to mitigate the risk of obtaining an over-smoothed latent space. Specifically, we collect 152 sets of real-world and synthetic observations from 23 complex systems as pre-training corpora and employ them to pre-train PDEDER. Given any future dynamic observation, we can fine-tune PDEDER with any specific dynamics modeling method. We evaluate PDEDER on 12 dynamic systems by short/long-term forecasting under both in-domain and cross-domain settings, and the empirical results indicate the effectiveness and generalizability of PDEDER.

Figures (4)

• Figure 1: The overall pipeline of pre-training (top) and fine-tuning (bottom) PDEder. (1) Pre-training of PDEder on large-scale dynamics observations: The sequences are first tokenized and normalized, followed by projection with system-specific data projection layers to align the dimension. The projected sequences are then encoded into latent space and decoded for reconstruction and forecasting. To pre-train PDEder, we employ a maximal Lyapunov exponent objective on sequence embeddings along with reconstruction and forecasting objectives. (2) Fine-tuning PDEder to approximate specific dynamics: The first token of an observation sequence is taken as the initial state and encoded into the latent space via the pre-trained encoder of PDEder. Then we approximate governing dynamics by integrating using the embedded initial states within the latent space through a GNN-based ODE module. Finally, we decode the integrated solutions by the pre-trained decoder and fine-tuning PDEder by minimizing the forecasting objective against the ground-truth observation sequence.
• Figure 2: The interacting graphs among objects of 2D Compressible Navier-Stokes (2D-CFD) and PEMS08.
• Figure 3: Forecasting visualizations on Heat dynamics over the variants of PDEder. "PDEder w/o MLE" denotes the ablative version pre-training without M-Lyapunov exponent; "PDEder w/o pre" denotes fine-tuning PDEder without pre-training; "PDEder frz" denotes fine-tuning by freezing the PLM module; "PDEder-sys" denotes the variant fine-tuning under the cross-domain setting of pre-training leaving one system out; "PDEder-para" denotes the variant fine-tuning under the cross-domain setting of pre-training learning one parameter out. The horizontal axes denote object indices, while the vertical axis represents object state values.
• Figure 4: Traffic flow forecasting visualizations on PDEder and baselines of real-world systems (a) T-Drive and (b) PEMS08. The vertical coordinate denotes the traffic flow values.