PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction

TL;DR

PastNet tackles spatio-temporal video prediction by injecting physical inductive biases through a Fourier-domain spectral convolution and a memory-bank–driven discrete spatio-temporal path. The method combines a Fourier-based Physics-Guided (FPG) module with a Discrete Spatio-temporal (DST) module, leveraging patch embeddings, FFT-based spectral filtering, intrinsic dimensionality estimation, and memory-augmented quantization to enable accurate, scalable prediction. Empirical results show PastNet achieving state-of-the-art or competitive performance across non-natural and natural datasets, with notably faster convergence and reduced computational cost, and preliminary evidence for solving PDE-like dynamics (e.g., NSE, SWE). This work offers a practical, physics-informed framework that improves both accuracy and efficiency for high-resolution video forecasting and opens avenues for PDE-aware video analysis.

Abstract

In this paper, we investigate the challenge of spatio-temporal video prediction task, which involves generating future video frames based on historical spatio-temporal observation streams. Existing approaches typically utilize external information such as semantic maps to improve video prediction accuracy, which often neglect the inherent physical knowledge embedded within videos. Worse still, their high computational costs could impede their applications for high-resolution videos. To address these constraints, we introduce a novel framework called \underline{P}hysics-\underline{a}ssisted \underline{S}patio-\underline{t}emporal \underline{Net}work (PastNet) for high-quality video prediction. The core of PastNet lies in incorporating a spectral convolution operator in the Fourier domain, which efficiently introduces inductive biases from the underlying physical laws. Additionally, we employ a memory bank with the estimated intrinsic dimensionality to discretize local features during the processing of complex spatio-temporal signals, thereby reducing computational costs and facilitating efficient high-resolution video prediction. Extensive experiments on various widely-used spatio-temporal video benchmarks demonstrate the effectiveness and efficiency of the proposed PastNet compared with a range of state-of-the-art methods, particularly in high-resolution scenarios.

Figures (7)

• Figure 1: Performance comparison of video prediction methods on MovingMNIST. PastNet outperforms previous models in training time and image quality, achieving the lowest MSE, highest SSIM, and shortest training time over 100 epochs. The small bubble for PastNet indicates minimal training time.
• Figure 2: An overview of the proposed PastNet, which consists of a Fourier-based Physics-guided (FPG) and a Discrete Spatio-temporal (DST) module. The PFI module first divides video frames into non-overlapping patches and introduce Fourier-based spectral filter with the introduction of physical biases. Then, its also extract spatial signals with convolutional neural networks. The DST module is an encoder-decoder architecture, which introduces a memory bank to discretize local features.
• Figure 3: Prediction results on the TrafficBJ dataset. Top: input Traffic flow; Middle: future real Traffic flow; Bottom: PastNet predicted Traffic flow.
• Figure 4: Example of prediction results on the SEVIR dataset. Top: input weather sequence; Middle: future real weather sequence; Bottom: PastNet predicted weather sequence.
• Figure 5: PastNet outperforms other models in terms of efficiency and convergence rate on the MovingMNIST dataset. Specifically, it achieves the lowest LPIPS score in the shortest training time, as shown on the Left side of the figure. In addition, it achieves the highest MS-SSIM and PSNR scores within the same epochs, as depicted in the Middle and Right sides of the figure, respectively.