LDMVFI: Video Frame Interpolation with Latent Diffusion Models

TL;DR

LDMVFI reframes video frame interpolation as conditional generation within a latent diffusion framework. It introduces a VFI-specific autoencoder, VQ-FIGAN, to map frames into a latent space and a diffusion-based denoiser operating on latent codes, enabling high-fidelity, perceptually-oriented frame synthesis. Key innovations include frame-aware decoding with neighbor-frame features via MaxViT, deformable convolution-based kernel synthesis for interpolation, and a conditional diffusion objective that uses $z^0$ and $z^1$ as conditioning signals. Experimental results across multiple benchmarks, including 4K content, show LDMVFI achieves superior perceptual quality (LPIPS, FloLPIPS, FID) and favorable subjective judgments, at the cost of higher computational demand and larger model size. The work demonstrates the potential of latent diffusion models for perception-driven VFI and outlines future directions for speeding up inference and reducing footprint while maintaining perceptual gains.

Abstract

Existing works on video frame interpolation (VFI) mostly employ deep neural networks that are trained by minimizing the L1, L2, or deep feature space distance (e.g. VGG loss) between their outputs and ground-truth frames. However, recent works have shown that these metrics are poor indicators of perceptual VFI quality. Towards developing perceptually-oriented VFI methods, in this work we propose latent diffusion model-based VFI, LDMVFI. This approaches the VFI problem from a generative perspective by formulating it as a conditional generation problem. As the first effort to address VFI using latent diffusion models, we rigorously benchmark our method on common test sets used in the existing VFI literature. Our quantitative experiments and user study indicate that LDMVFI is able to interpolate video content with favorable perceptual quality compared to the state of the art, even in the high-resolution regime. Our code is available at https://github.com/danier97/LDMVFI.

Figures (11)

• Figure 1: Overview of the diffusion processes in LDMVFI. The encoder and decoder enable projection between image and latent spaces, and the diffusion processes take place in the latent space.
• Figure 2: The architecture of the VFI autoencoding model, VQ-FIGAN. It differs from the original VQGAN esser2021taming in three aspects: (i) the use of features extracted by the encoder from neighboring frames during the decoding via MaxViT-based cross attention; (ii) use of more efficient MaxViT block instead of the vanilla self-attention; (iii) frame synthesis via adaptive deformable convolution. The kernel ($\Omega$), offset ($\alpha,\beta$), visibility ($v$) and residual ($\delta$) heads contain 3$\times${conv3x3, ReLU}.
• Figure 3: Visual examples of frames interpolated by the state-of-the-art methods and the proposed LDMVFI. Under large and complex motions, our method preserves the most high-frequency details, delivering superior perceptual quality.
• Figure 4: Results of the user study in terms of preference ratio. Error bar reflects the standard error over test sequences.
• Figure 5: The architecture of the denoising U-Net. The hyper-parameter $c$ is a base channel size, which is set to 256 in LDMVFI. In each block, the $(\cdot \rightarrow \cdot)$ indicates the input and output channels of the block.