Improving Temporal Consistency and Fidelity at Inference-time in Perceptual Video Restoration by Zero-shot Image-based Diffusion Models
Nasrin Rahimi, A. Murat Tekalp
TL;DR
This work tackles temporal instability in zero-shot video restoration using image-based diffusion priors. It introduces two inference-time, training-free strategies: Perceptual Straightening Guidance (PSG), which imposes a curvature-based penalty in a perceptual RetinaND–V1 space to encourage smoother temporal evolution, and Multi-Path Ensemble Sampling (MPES), which ensembles multiple diffusion trajectories to reduce stochastic variation and improve fidelity. PSG enhances temporal naturalness, especially under temporal blur, while MPES yields consistent gains in both spatial fidelity and perceptual quality, achieving a favorable perception-distortion balance. Across DAVIS and REDS4 datasets, pixel-space fusion with a moderate ensemble size (K) provides strong, generalizable improvements with manageable computational cost, offering a practical path toward temporally stable, high-fidelity zero-shot diffusion-based video restoration without retraining. These methods extend the applicability of pretrained diffusion priors to dynamic video restoration by addressing temporal coherence and variance at inference time, with potential for broader deployment in real-world degradations and diffusion architectures.
Abstract
Diffusion models have emerged as powerful priors for single-image restoration, but their application to zero-shot video restoration suffers from temporal inconsistencies due to the stochastic nature of sampling and complexity of incorporating explicit temporal modeling. In this work, we address the challenge of improving temporal coherence in video restoration using zero-shot image-based diffusion models without retraining or modifying their architecture. We propose two complementary inference-time strategies: (1) Perceptual Straightening Guidance (PSG) based on the neuroscience-inspired perceptual straightening hypothesis, which steers the diffusion denoising process towards smoother temporal evolution by incorporating a curvature penalty in a perceptual space to improve temporal perceptual scores, such as Fréchet Video Distance (FVD) and perceptual straightness; and (2) Multi-Path Ensemble Sampling (MPES), which aims at reducing stochastic variation by ensembling multiple diffusion trajectories to improve fidelity (distortion) scores, such as PSNR and SSIM, without sacrificing sharpness. Together, these training-free techniques provide a practical path toward temporally stable high-fidelity perceptual video restoration using large pretrained diffusion models. We performed extensive experiments over multiple datasets and degradation types, systematically evaluating each strategy to understand their strengths and limitations. Our results show that while PSG enhances temporal naturalness, particularly in case of temporal blur, MPES consistently improves fidelity and spatio-temporal perception--distortion trade-off across all tasks.