SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction

TL;DR

SyncVP addresses RGB-only video prediction limitations by jointly modeling RGB and depth with pre trained modality specific diffusion models. It introduces a split spatio-temporal cross-attention STCA to enable cross modal information exchange and uses a shared forward diffusion noise across modalities, improving convergence and coherence. A cross modality guidance training strategy enables robust generation under partial conditioning, allowing predictions even when one modality is missing. On Cityscapes and BAIR, SyncVP achieves state of the art FVD improvements while remaining effective when depth is unavailable, and it generalizes to other modalities such as semantic maps and climate data, highlighting strong robustness and broad applicability.

Abstract

Predicting future video frames is essential for decision-making systems, yet RGB frames alone often lack the information needed to fully capture the underlying complexities of the real world. To address this limitation, we propose a multi-modal framework for Synchronous Video Prediction (SyncVP) that incorporates complementary data modalities, enhancing the richness and accuracy of future predictions. SyncVP builds on pre-trained modality-specific diffusion models and introduces an efficient spatio-temporal cross-attention module to enable effective information sharing across modalities. We evaluate SyncVP on standard benchmark datasets, such as Cityscapes and BAIR, using depth as an additional modality. We furthermore demonstrate its generalization to other modalities on SYNTHIA with semantic information and ERA5-Land with climate data. Notably, SyncVP achieves state-of-the-art performance, even in scenarios where only one modality is present, demonstrating its robustness and potential for a wide range of applications.

Figures (14)

• Figure 1: SyncVP is a diffusion model for synchronized multi-modal video prediction. It generates multi-modal future frames like RGB and depth for a given observation that can consist of both modalities (left) or only one modality (right).
• Figure 2: Our multi-modal latent diffusion framework is trained by exploiting pre-trained weights ($\theta_R,\theta_D$) of each modality and an efficient spatio-temporal cross-attention mechanism. The same noise $\boldsymbol{\epsilon}$ is used during the forward diffusion process of each modality.
• Figure 3: SyncVP predictions on BAIR using 2 conditioning frames (yellow frame). Predicting future movements of the robotic arm is challenging due to high stochasticity, but we can appreciate the alignment between predicted RGB and depth frames.
• Figure 4: SyncVP predictions on SYNTHIA (RGB + Semantic segmentation) using 8 conditioning frames to predict the next 8 frames.
• Figure 5: Comparison of RGB and depth loss for single-modality models and SyncVP on Cityscapes.