Adapting Image-to-Video Diffusion Models for Large-Motion Frame Interpolation

TL;DR

This work tackles large-motion video frame interpolation by adapting pre-trained image-to-video diffusion models through a plug-and-play conditional encoder. It introduces a dual-branch feature extractor to capture spatial and temporal cues and a cross-frame attention mechanism to fuse information across frames, improving motion consistency. Experiments on DAVIS-7 and UCF101-7 demonstrate strong performance, particularly in Fréchet Video Distance (FVD), and ablations confirm the contributions of the temporal branch and cross-frame attention. Limitations of latent diffusion, such as reduced fine details due to down-sampling and resolution constraints, are discussed, with guidance for future improvements toward higher-resolution and more motion-rich synthesis.

Abstract

With the development of video generation models has advanced significantly in recent years, we adopt large-scale image-to-video diffusion models for video frame interpolation. We present a conditional encoder designed to adapt an image-to-video model for large-motion frame interpolation. To enhance performance, we integrate a dual-branch feature extractor and propose a cross-frame attention mechanism that effectively captures both spatial and temporal information, enabling accurate interpolations of intermediate frames. Our approach demonstrates superior performance on the Fréchet Video Distance (FVD) metric when evaluated against other state-of-the-art approaches, particularly in handling large motion scenarios, highlighting advancements in generative-based methodologies.

Figures (5)

• Figure 1: The input video data $\{I_1, I_2, \dots, I_N\}$ is first encoded into latent representations, where Gaussian noise is added to produce the noisy representations $\{z_1, z_2, \dots, z_N\}$. Subsequently, the spatial and temporal features of the first and last frames are extracted through separate branches and fed into the conditional encoder. The conditional encoder integrates these features while employing zero initialization to ensure stable training. Finally, the integrated features are incorporated into a pre-trained 3D U-Net.
• Figure 2: The first and last frames are processed through distinct feature extractors, followed by a fusion process to integrate their features.
• Figure 3: In both the spatial transformer block and the temporal transformer block, a cross-frame attention block is first integrated to enhance feature representation.
• Figure 4: Interpolation results across different styles. From the first row to the last row, the results correspond to real-world, anime, and sketch styles.
• Figure 5: The example demonstrates a large-motion scenario where traditional methods fail to predict the direction and magnitude of motion accurately, often resulting in blurred or inconsistent intermediate frames. In contrast, our approach reconstruct realistic motion trajectories, producing coherent intermediate frames that capture accurate motion dynamics.