Generating time-consistent dynamics with discriminator-guided image diffusion models

TL;DR

The paper tackles the challenge of generating time-consistent dynamics with diffusion-based models without expensive training of video diffusion models. It introduces a time-consistency discriminator that guides the sampling of pretrained image diffusion models during inference, making the approach architecture-agnostic and computationally efficient. Empirical results on 2D Navier-Stokes turbulence and ERA5 precipitation show that the guided diffusion produces temporal dynamics with comparable realism to a VDM trained from scratch, while delivering improved uncertainty calibration and reduced biases, and enabling centennial-scale climate rollouts. This technique significantly broadens the utility of pretrained IDMs for long-horizon, physics-rich applications in fluid dynamics and climate science, with potential for broader video-processing tasks.

Abstract

Realistic temporal dynamics are crucial for many video generation, processing and modelling applications, e.g. in computational fluid dynamics, weather prediction, or long-term climate simulations. Video diffusion models (VDMs) are the current state-of-the-art method for generating highly realistic dynamics. However, training VDMs from scratch can be challenging and requires large computational resources, limiting their wider application. Here, we propose a time-consistency discriminator that enables pretrained image diffusion models to generate realistic spatiotemporal dynamics. The discriminator guides the sampling inference process and does not require extensions or finetuning of the image diffusion model. We compare our approach against a VDM trained from scratch on an idealized turbulence simulation and a real-world global precipitation dataset. Our approach performs equally well in terms of temporal consistency, shows improved uncertainty calibration and lower biases compared to the VDM, and achieves stable centennial-scale climate simulations at daily time steps.

Figures (15)

• Figure 1: Overview sketch of the time-consistency discriminator guidance for generating images in a dynamically realistic sequence. The discriminator guidance $\bm{d}_{\theta}(\cdot)$ uses the current and past time frames, $\bm{x}^{n}$ and $\bm{x}^{n-1}$, to guide the denoising generation of the next $\bm{x}^{n+1}$.
• Figure 2: Time-consistency prediction of the discriminator network during sampling of vorticity fields with guidance switched on (red) or off (blue). The mean over 50 samples is given by the solid line, and the shaded area shows the standard deviation. With decreasing noise scales in the reserve diffusion process ($t_{\max}=1 \rightarrow t_{\min}=0$), the discriminator network reliably predicts whether samples are time-consistent or not.
• Figure 3: Hovmöller diagrams, often used to visualize spatiotemporal dynamics and, in particular, the propagation of waves in fluid dynamics and meteorology, are computed for the 2D vorticity simulation as the mean over a vertical band of grid columns for (from left to right) the ground truth numerical simulation, the unconditional DM, the video DM, and our guidance approach. The guidance method and video DM generate dynamics indistinguishable from the ground truth.
• Figure 4: Quantitative evaluation of 2D Navier-Stokes turbulent vorticity dynamics in terms of Wasserstein distances between consecutive rows of the Hovmöller diagram in Fig. \ref{['fig:vorticity_hovmoeller']} (a), autocorrelation function (ACF) (b), continuous ranked probability score (CRPS) (c) and running window spatial mean as solid line with the actual time series shown as shades (d), for the ground truth simulation (black), the unconditional DM (green), video DM (blue) and the guided DM (red). Note that only the guided DM achieves an unbiased representation of the vorticity.
• Figure 5: Hovmöller diagrams of the global daily precipitation simulation (from left to right) are computed for 180 days as a mean over the latitude band from $10^\circ$S to $10^\circ$N for the ground truth ERA5, unconditional DM, video DM, and our guidance approach.