Efficient Continuous Video Flow Model for Video Prediction

TL;DR

This work tackles the latency and parameter-efficiency challenges of diffusion-based video prediction by introducing Continuous Video Flow (CVF), which treats video as a continuous latent-space process. It employs a two-stage pipeline: encode frames into a latent space via a pre-trained autoencoder, then model a continuous forward and reverse process between latent frames with a denoising objective defined in latent space, using the interpolation z_t = (1−t)z^j + t z^{j+1} − (t log t)/√2 ε, ε ∼ N(0,I). The approach achieves state-of-the-art results on KTH, BAIR, Human3.6M, and UCF101 while using far fewer parameters and sampling steps, highlighting practical benefits for efficient video prediction. Limitations include remaining sequential sampling bottlenecks and resource constraints; future work points to further reducing sampling steps and scaling to longer sequences with more powerful hardware. Overall, CVF provides a principled, efficient framework for continuous video prediction in latent space with strong empirical performance and potential for real-time applications.

Abstract

Multi-step prediction models, such as diffusion and rectified flow models, have emerged as state-of-the-art solutions for generation tasks. However, these models exhibit higher latency in sampling new frames compared to single-step methods. This latency issue becomes a significant bottleneck when adapting such methods for video prediction tasks, given that a typical 60-second video comprises approximately 1.5K frames. In this paper, we propose a novel approach to modeling the multi-step process, aimed at alleviating latency constraints and facilitating the adaptation of such processes for video prediction tasks. Our approach not only reduces the number of sample steps required to predict the next frame but also minimizes computational demands by reducing the model size to one-third of the original size. We evaluate our method on standard video prediction datasets, including KTH, BAIR action robot, Human3.6M and UCF101, demonstrating its efficacy in achieving state-of-the-art performance on these benchmarks.

Figures (9)

• Figure 1: Fig. (a) represents a naive adaptation of the diffusion model for the video prediction task. Here, the sampling process always starts from a Gaussian distribution, and sampling steps are taken in the direction of conditional distribution given by $X_\text{j+1}|X_j$. Here, $X_\text{j+1}$ denotes frame at time $j+1$. In contrast, Fig. (b) introduces our Continuous Video Flow (CVF) approach, which reimagines the problem by treating video not as a discrete sequence of frames but as a continuously evolving process. Instead of starting from a static Gaussian distribution for each sampling step, CVF models the underlying dynamics of the entire video, learning to predict changes smoothly over time. This continuous framework allows the model to better capture temporal coherence and evolution, leading to more accurate and fluid video predictions.
• Figure 2: Overview of the Continuous Video Flow (CVF) framework. (a) Stage 1 depicts the auto-encoding process where input video frames $X_j$ are passed through an encoder (Enc) and decoder (Dec) to reconstruct $\hat{X_j}$. (b) Stage 2 illustrates the forward and reverse processes in the latent space. In the forward process, latent embeddings $z_T, \dots, z_1$ are generated through a fixed process given by Eqn \ref{['eqn:fwdprocess']}. The reverse process involves sampling $z_\text{j+1}$ from $z_\text{j}$ through the learned process $p_\theta$ for continuous video frame prediction. (c) The full pipeline of CVF, showing how frame $X_j$ is passed through the encoder to obtain latent embedding $z_j$ and $\hat{z}_{j+1}$, which is then used for decoding to obtain $\hat{X}_{j+1}$ frame.
• Figure 3: (a) Illustration of the single-step estimation process for $\mathbf{z}_t$, where a pre-trained Encoder encodes a block of $k$ frames, highlighting the computational methodology employed. (b) The training pipeline of the Continuous Video Process (CVP) model, where $\mathbf{z}_t$ and $t$ serve as inputs to the U-Net architecture (details in Appendix), producing the predicted output $\hat{\mathbf{z}}^{1:k+1}$. (c) Overview of the sampling pipeline used in our approach, demonstrating the sequential prediction of the next frame in the video sequence. Given context frames in latent space$\hat{\mathbf{z}}^{0:k}$, the predicted latent $\hat{\mathbf{z}}^{k+1}$ is decoded to generate the subsequent frame $\hat{\mathbf{x}}^{k+1}$.
• Figure 4: Figure represents qualitative results of our CVF model on the KTH dataset. The number of context frames used in the above setting is 4 for all three sequences. Every $4^{th}$ predicted future frame is shown in the figure.
• Figure 5: Figure represents qualitative results of our CVF model on the BAIR dataset. The number of context frames used in the above setting is two for both sequences. Every $6^{th}$ predicted future frame is shown in the figure.