Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers

TL;DR

The paper tackles temporal incoherence in video frame interpolation by moving from frame-centric to holistic video modeling. It introduces LDF-VFI, an auto-regressive diffusion transformer that models entire video sequences, with a novel skip-concatenate sampling strategy to curb error accumulation, and a sparse, local attention plus tiled VAE encoding design to enable 4K-scale inference. A latent diffusion framework with a conditional VAE decoder and multi-scale video conditioning improves fidelity and coherence, while diffusion forcing enables stable autoregressive inference over long sequences. Empirically, LDF-VFI achieves state-of-the-art results on challenging long-sequence benchmarks, demonstrating improved per-frame quality and temporal consistency, especially in scenes with large motion, and provides a scalable, practical pipeline for high-resolution VFI.

Abstract

Existing video frame interpolation (VFI) methods often adopt a frame-centric approach, processing videos as independent short segments (e.g., triplets), which leads to temporal inconsistencies and motion artifacts. To overcome this, we propose a holistic, video-centric paradigm named \textbf{L}ocal \textbf{D}iffusion \textbf{F}orcing for \textbf{V}ideo \textbf{F}rame \textbf{I}nterpolation (LDF-VFI). Our framework is built upon an auto-regressive diffusion transformer that models the entire video sequence to ensure long-range temporal coherence. To mitigate error accumulation inherent in auto-regressive generation, we introduce a novel skip-concatenate sampling strategy that effectively maintains temporal stability. Furthermore, LDF-VFI incorporates sparse, local attention and tiled VAE encoding, a combination that not only enables efficient processing of long sequences but also allows generalization to arbitrary spatial resolutions (e.g., 4K) at inference without retraining. An enhanced conditional VAE decoder, which leverages multi-scale features from the input video, further improves reconstruction fidelity. Empirically, LDF-VFI achieves state-of-the-art performance on challenging long-sequence benchmarks, demonstrating superior per-frame quality and temporal consistency, especially in scenes with large motion. The source code is available at https://github.com/xypeng9903/LDF-VFI.

Figures (5)

• Figure 1: Comparison between prior works and our framework. We illustrate $2\times$ VFI for simplicity. Prior methods treat VFI as a set of independent triplets, ignoring dependencies among interpolated frames. In contrast, our framework models the entire video holistically, leading to improved temporal coherence.
• Figure 2: Model architecture overview. Our model first aligns the low-quality (LQ) video with the high-quality (HQ) ground-truth (GT) video's temporal dimension using nearest-neighbor upsampling. Both the GT video, the upsampled LQ video, and a corresponding binary mask are then processed through a spatial-tiled and temporal non-overlapping chunk encoder to generate latent representations. During training, we employ diffusion forcing on these temporal chunks, a strategy that enables auto-regressive inference. Furthermore, the model incorporates sparse attention to efficiently handle high-resolution video inputs, ensuring scalability.
• Figure 3: The architecture of conditional VAE decoder.
• Figure 4: Causal vs. skip-concatenate auto-regressive inference. A bold blue border marks the current chunk being generated, and a blue arrow indicates its context. Darker chunks signify greater error accumulation. The skip-concatenate approach suppresses error propagation by generating a skip chunk that does not depend on the preceding context, effectively resetting the error.
• Figure 5: Qualitative comparisons of $16\times$ interpolation on SNU-FILM dataset. Regions with artifacts in synthesized frames are highlighted with red boxes. Our method preserves sharper details and exhibits fewer artifacts with better temporal consistency.