Time-Correlated Video Bridge Matching

TL;DR

This work identifies a key limitation of diffusion-based data-to-data translation in video tasks: neglecting temporal correlations between frames. It introduces Time-Correlated Video Bridge Matching (TCVBM), which extends Bridge Matching by embedding a time-aware prior into the diffusion bridge to capture inter-frame dependencies. The method provides training and inference procedures that jointly optimize a frame-aware drift via regression on the clean sequence and perform iterative denoising under a temporally structured bridge. Empirical results on frame interpolation, image-to-video generation, and video super-resolution show improved temporal coherence and reconstruction fidelity compared to DDPM, DDIM, and standard Bridge Matching. The framework offers a flexible foundation for sequential data tasks beyond video, though it currently relies on a tractable tridiagonal prior and incurs notable computational cost.

Abstract

Diffusion models excel in noise-to-data generation tasks, providing a mapping from a Gaussian distribution to a more complex data distribution. However they struggle to model translations between complex distributions, limiting their effectiveness in data-to-data tasks. While Bridge Matching (BM) models address this by finding the translation between data distributions, their application to time-correlated data sequences remains unexplored. This is a critical limitation for video generation and manipulation tasks, where maintaining temporal coherence is particularly important. To address this gap, we propose Time-Correlated Video Bridge Matching (TCVBM), a framework that extends BM to time-correlated data sequences in the video domain. TCVBM explicitly models inter-sequence dependencies within the diffusion bridge, directly incorporating temporal correlations into the sampling process. We compare our approach to classical methods based on bridge matching and diffusion models for three video-related tasks: frame interpolation, image-to-video generation, and video super-resolution. TCVBM achieves superior performance across multiple quantitative metrics, demonstrating enhanced generation quality and reconstruction fidelity.

Figures (7)

• Figure 1: Comparison of the Bridge Matching and Time-Correlated Video Bridge Matching methods. The frames of the same video in the distributions $p(\mathbf{x}_0)$ and $p(\mathbf{X}_0)$ are indicated by dots of the same color. While Bridge Matching method makes the transition from one distribution to another without considering the relationship between frames, our approach treats one video as a single $\mathbf{X}_0$ data sequence and constructs the transition taking into account the internal correlation between video frames.
• Figure 2: Frame interpolation results. The first and last frames are fixed and come from the dataset. The task is to generate the intermediate $8$ frames. TCVBM provides a smoother and more consistent transition from frame to frame and a better detalization.
• Figure 3: Image-to-Video generation results. Based on the first frame, the methods need to generate the next $9$ frames. As can be seen, TCVBM makes it possible to achieve better appearance of the generations and a lower quality degradation with increasing sequence length.
• Figure 4: Video super resolution results. The resolution is increased from $16 \times 16$ to $64 \times 64$. As can be seen, TCVBM outperforms other methods.
• Figure 5: Additional frame interpolation results.

Theorems & Definitions (6)

• Proposition 1: Correlated Process Score
• Proposition 2: Correlated Bridge Distribution
• Proposition 3: Reparameterization of the drift function
• proof : Proof of Proposition \ref{['thm:correlated-process-score']}
• proof : Proof of Proposition \ref{['thm:correlated-bridge-distribution']}
• proof : Proof of Proposition \ref{['thm:reparam']}