Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation Model

Derek Jollie, Jingmin Sun, Zecheng Zhang, Hayden Schaeffer

TL;DR

The paper tackles generalization gaps in time-series forecasting of spatiotemporal PDEs by leveraging an additional equation modality in a PDE foundation model. It replaces manual PROSE symbolic encoding with a SymPy-based standardization (SymPy tree) to automatically produce consistent token sequences, enabling more robust zero-shot extrapolation to new operators. A Bayesian Sequential Monte Carlo (SMC) particle-filter module is introduced to refine learned PDE coefficients, improving equation accuracy and long-term stability. Experiments across five symbolic-encoding settings demonstrate that the SymPy-based encoding yields superior robustness to term reordering and noise, while particle-filter refinement further reduces time-series errors, pointing to a scalable, automated approach for multimodal PDE-based forecasting.

Abstract

Symbolic encoding has been used in multi-operator learning as a way to embed additional information for distinct time-series data. For spatiotemporal systems described by time-dependent partial differential equations, the equation itself provides an additional modality to identify the system. The utilization of symbolic expressions along side time-series samples allows for the development of multimodal predictive neural networks. A key challenge with current approaches is that the symbolic information, i.e. the equations, must be manually preprocessed (simplified, rearranged, etc.) to match and relate to the existing token library, which increases costs and reduces flexibility, especially when dealing with new differential equations. We propose a new token library based on SymPy to encode differential equations as an additional modality for time-series models. The proposed approach incurs minimal cost, is automated, and maintains high prediction accuracy for forecasting tasks. Additionally, we include a Bayesian filtering module that connects the different modalities to refine the learned equation. This improves the accuracy of the learned symbolic representation and the predicted time-series.

Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation Model

TL;DR

The paper tackles generalization gaps in time-series forecasting of spatiotemporal PDEs by leveraging an additional equation modality in a PDE foundation model. It replaces manual PROSE symbolic encoding with a SymPy-based standardization (SymPy tree) to automatically produce consistent token sequences, enabling more robust zero-shot extrapolation to new operators. A Bayesian Sequential Monte Carlo (SMC) particle-filter module is introduced to refine learned PDE coefficients, improving equation accuracy and long-term stability. Experiments across five symbolic-encoding settings demonstrate that the SymPy-based encoding yields superior robustness to term reordering and noise, while particle-filter refinement further reduces time-series errors, pointing to a scalable, automated approach for multimodal PDE-based forecasting.

Abstract

Symbolic encoding has been used in multi-operator learning as a way to embed additional information for distinct time-series data. For spatiotemporal systems described by time-dependent partial differential equations, the equation itself provides an additional modality to identify the system. The utilization of symbolic expressions along side time-series samples allows for the development of multimodal predictive neural networks. A key challenge with current approaches is that the symbolic information, i.e. the equations, must be manually preprocessed (simplified, rearranged, etc.) to match and relate to the existing token library, which increases costs and reduces flexibility, especially when dealing with new differential equations. We propose a new token library based on SymPy to encode differential equations as an additional modality for time-series models. The proposed approach incurs minimal cost, is automated, and maintains high prediction accuracy for forecasting tasks. Additionally, we include a Bayesian filtering module that connects the different modalities to refine the learned equation. This improves the accuracy of the learned symbolic representation and the predicted time-series.
Paper Structure (9 sections, 8 equations, 5 figures, 2 tables)

This paper contains 9 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Particle Filter
  4. Numerical Experiments
  5. Conclusion
  6. Experiment Setup
  7. Dataset
  8. Evaluation Metrics
  9. Training

Figures (5)

  • Figure 1: PROSE PDE Foundation Model with Particle Filtering.
  • Figure 2: PROSE Tree Examples: The left tree is an example of a manually standardized PROSE tree for the viscous Burgers' equation $u_t + kuu_x = \frac{\epsilon}{\pi}u_{xx}$. In the experiments, to generate the randomized trees (or a tree encountered in testing), we randomly switch the order of any branch of the tree with probability $0.5$, leading to different orders of the same symbolic expressions. The right tree is an example of an altered tree for the same equation.
  • Figure 3: SymPy Tree Example: KdV equation $uu_x + u_t + \delta^2u_{xxx} = 0$. Here, $\partial(u(x,t),(x,3))$ is used in the tree structure to embed the term $u_{xxx}$ and similarly. Other derivatives are written using this notation.
  • Figure 4: Particle Filter Module: A discretized version of the Bayesian filter process.
  • Figure 5: Various examples of the symbolic modality for inviscid conservation law with sine flux. Target equation: $u_t + 0.955 \cos(u)u_x=0$. For PROSE tree, the model is trained for the order $[?]u_t + [?]\cos(u)u_x$, and for SymPy tree, the input expression is automatically uniformed into $[?]\cos(u)u_x + [?]u_t$. The generated symbols use 3 significant digits while the refinement is a standard float. Notably the SymPy tree removes the erroneous term in prediction. See Table \ref{['tab:my_label']} for error details.