Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction

TL;DR

This work tackles video prediction by reframing video as a continuous multi-dimensional process rather than a sequence of discrete frames. The Continuous Video Process (CVP) defines an explicit forward path between consecutive frames with $\mathbf{x}_t = (1-t)\mathbf{x} + t\mathbf{y} - \frac{t\log(t)}{\sqrt{2}}\mathbf{z}$ and learns a reverse Gaussian process via a variational bound, achieving $75\%$ fewer sampling steps during inference. It conducts extensive experiments on KTH, BAIR, Human3.6M, and UCF101, showing state-of-the-art video prediction while eliminating external temporal constraints like temporal attention. The approach uses a U-Net backbone, a specialized noise schedule $g(t) = -t\log(t)$, and a simple training loss, enabling efficient, high-fidelity frame synthesis with competitive metrics such as FVD. The work also provides ablations on noise schedules and sampling steps and discusses limitations and broader societal impacts of advanced video synthesis.

Abstract

Diffusion models have made significant strides in image generation, mastering tasks such as unconditional image synthesis, text-image translation, and image-to-image conversions. However, their capability falls short in the realm of video prediction, mainly because they treat videos as a collection of independent images, relying on external constraints such as temporal attention mechanisms to enforce temporal coherence. In our paper, we introduce a novel model class, that treats video as a continuous multi-dimensional process rather than a series of discrete frames. We also report a reduction of 75\% sampling steps required to sample a new frame thus making our framework more efficient during the inference time. Through extensive experimentation, we establish state-of-the-art performance in video prediction, validated on benchmark datasets including KTH, BAIR, Human3.6M, and UCF101. Navigate to the project page https://www.cs.umd.edu/~gauravsh/cvp/supp/website.html for video results.

Figures (7)

• Figure 1: The figure is divided into two parts. The top portion of the figure illustrates the intermediate frames $\mathbf{x}_t$ between two consecutive frames. $\mathbf{x},\mathbf{y}$ represents consecutive frames from a video sequence where $\mathbf{y} = \mathbf{x}^{j+1}$ and $\mathbf{x} = \mathbf{x}^{j}$. $\mathbf{x}^j$ denotes some frame at timestep $j$ in the video sequence $\mathcal{V} = \{\mathbf{x}^i\}_{i=1}^{N}$. $\mathbf{z}$ denotes the white noise. The lower portion of the figure represents the directed graphical model considered in this work to represent the continuous video process.
• Figure 2: Fig. (a) demonstrates the methodology for estimating $\mathbf{x}_t$ in a single step, showcasing the specific computational process involved. Fig. (b) details the training pipeline of our Continuous Video Process (CVP) model, where $\mathbf{x}_t$ and $t$ are fed as inputs to the U-Net architecture, and the anticipated output is $\hat{\mathbf{y}}$, with $\hat{\mathbf{y}} = \mathbf{x}^{1:k+1}$ in this scenario. Fig. (c) provides an overview of the sampling pipeline utilized in our CVP method, illustrating the sequential steps to predict the next frame of the video sequence given the context frames.
• Figure 3: Figure represents qualitative results of our CVP model on the KTH dataset. The number of context frames used in the above setting is 4 for all three sequences. Every $4^{th}$ predicted future frame is shown in the figure.
• Figure 4: Figure represents qualitative results of our CVP model on the BAIR dataset. The number of context frames used in the above setting is two for both sequences. Every $6^{th}$ predicted future frame is shown in the figure.
• Figure 5: Figure represents qualitative results of our CVP model on the Human3.6M dataset. The number of context frames used in the above setting is 4 for all three sequences. Every $4^{th}$ predicted future frame is shown in the figure.