STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Prediction

TL;DR

This paper proposes a novel video prediction model, which has infinite-dimensional latent variables over the spatio-temporal domain, and is able to achieve temporal continuous prediction, i.e., predicting in an unsupervised way the future video frames with an arbitrarily high frame rate.

Abstract

Predicting future frames of a video is challenging because it is difficult to learn the uncertainty of the underlying factors influencing their contents. In this paper, we propose a novel video prediction model, which has infinite-dimensional latent variables over the spatio-temporal domain. Specifically, we first decompose the video motion and content information, then take a neural stochastic differential equation to predict the temporal motion information, and finally, an image diffusion model autoregressively generates the video frame by conditioning on the predicted motion feature and the previous frame. The better expressiveness and stronger stochasticity learning capability of our model lead to state-of-the-art video prediction performances. As well, our model is able to achieve temporal continuous prediction, i.e., predicting in an unsupervised way the future video frames with an arbitrarily high frame rate. Our code is available at \url{https://github.com/XiYe20/STDiffProject}.

Figures (4)

• Figure 1: Graphical model for the generation process of STDiff. Green arrows denote the temporal motion connections, and blue arrows denote the connections between latent variables of each frame at timestep $t$, i.e., the reverse image diffusion process. Red arrows denote the recurrent connection from the previous frame to each level of latent variable in the next time step. $m_{t_0}$ denotes the initial motion feature extracted from observed frames. $x_{t_0}^0$ denotes the most recent observed frame.
• Figure 2: Neural network architecture of STDiff. Difference images of past frames are encoded as motion feature $m_N$ by a ConvGRU for future motion prediction. The curved arrow denotes one step of random future motion prediction (SDE integration), i.e., the temporal motion diffusion process. Detail computation flow of the SDE solver is shown in the top left green box. The conditional image diffusion model recurrently predicts each future frame given motion feature and the previous frame.
• Figure 3: Prediction examples on the Cityscapes dataset for MCVD voleti2022, NPVP ye2023, and our model. Gray frames indicate non-existent or unpredictable frames. MCVD exhibits issues with brightness changes and lacks the ability of continuous predictions. NPVP predictions are noticeably more blurry than the other two models.
• Figure 4: Random predictions for BAIR dataset by our model and NPVP ye2023.