Causal Time-Series Synchronization for Multi-Dimensional Forecasting

TL;DR

This work proposes a novel channel-dependent pre-training strategy that leverages synchronized cause-effect pairs to overcome challenges by breaking down the multi-dimensional time-series data into pairs of cause-effect variables and demonstrates significant improvements in forecasting accuracy and generalization capability compared to traditional training methods.

Abstract

The process industry's high expectations for Digital Twins require modeling approaches that can generalize across tasks and diverse domains with potentially different data dimensions and distributional shifts i.e., Foundational Models. Despite success in natural language processing and computer vision, transfer learning with (self-) supervised signals for pre-training general-purpose models is largely unexplored in the context of Digital Twins in the process industry due to challenges posed by multi-dimensional time-series data, lagged cause-effect dependencies, complex causal structures, and varying number of (exogenous) variables. We propose a novel channel-dependent pre-training strategy that leverages synchronized cause-effect pairs to overcome these challenges by breaking down the multi-dimensional time-series data into pairs of cause-effect variables. Our approach focuses on: (i) identifying highly lagged causal relationships using data-driven methods, (ii) synchronizing cause-effect pairs to generate training samples for channel-dependent pre-training, and (iii) evaluating the effectiveness of this approach in channel-dependent forecasting. Our experimental results demonstrate significant improvements in forecasting accuracy and generalization capability compared to traditional training methods.

Figures (3)

• Figure 1: (a) Channel Independence (CI), i.e. each variable is treated as an isolated univariate problem only indirectly learning from other channels through shared weights; (b) Channel Dependence (CD), i.e. all variables are mixed and treated as a multivariate problem directly learning from other variables; (c) Proposed hybrid model (CI+CD), i.e. target variables $\mathbf{X}^{(N)}$ are mixed with exogenous variables $\mathbf{Z}^{(N)}$ forming blocks of cause-effect pairs, directly learning cause-effect behaviour of two variables through CD and indirectly learning from other cause-effects from other variable interactions through shared weights, i.e. CI. Illustrative example adapted from Wang et al. Wang2024.
• Figure 2: Illustrative example of a non-synchronized cause-effect pair (a), the effect lags with a time delay of two time-steps and a synchronized cause-effect pair (b), where the cause is shifted $\delta_{ij}$ time-steps, i.e. a shift by two time-steps in this example, to account for the causal lag. Illustrative example adapted from Zhao et al. Zhao2024.
• Figure 3: Illustrative example of the transformed cause-effect pair for channel-dependent forecasting, where the context - $C$ denotes the context length, i.e. historic data points - contains three variables: the historic values of the actual target variable $\mathbf{X}^{(i)}$, the target-shifted, synchronized cause variable $\mathbf{\tilde{Z}}^{(j)}$, and the non-synchronized cause variable $\mathbf{Z}^{(j)}$ to counteract data loss of right shift. The future effect $\hat{X}$ is predicted over a horizon $H$.