ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler

TL;DR

ViBiDSampler tackles two-frame keyframe interpolation with diffusion models by adopting bidirectional diffusion sampling that sequentially denoises along the forward path conditioned on the start frame and along the backward path conditioned on the end frame, incorporating a single re-noising step to keep generation on the diffusion manifold. The method enhances alignment and temporal coherence with CFG++ and DDS guidance, enabling training-free, efficient generation of high-quality intermediate frames and achieving state-of-the-art performance on benchmark datasets. It demonstrates that bounded, high-resolution video interpolation can be achieved without extensive re-noising or fine-tuning, producing 25-frame sequences in about 195 seconds on a single RTX 3090. The approach also handles identical and dynamic boundary conditions, preserves object identities, and remains applicable across various I2V diffusion models, highlighting practical impact for video editing, animation, and content creation.

Abstract

Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V) diffusion models has greatly enhanced video generation, especially in terms of keyframe interpolation. However, current image-to-video diffusion models, while powerful in generating videos from a single conditioning frame, need adaptation for two-frame (start & end) conditioned generation, which is essential for effective bounded interpolation. Unfortunately, existing approaches that fuse temporally forward and backward paths in parallel often suffer from off-manifold issues, leading to artifacts or requiring multiple iterative re-noising steps. In this work, we introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning. Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames. Additionally, we incorporate advanced guidance techniques, CFG++ and DDS, to further enhance the interpolation process. By integrating these, our method achieves state-of-the-art performance, efficiently generating high-quality, smooth videos between keyframes. On a single 3090 GPU, our method can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds, establishing it as a leading solution for keyframe interpolation.

Figures (10)

• Figure 1: Keyframe interpolation results using our ViBiDSampler. (a) The images in the first and last rows are keyframes, and the intermediate frames are generated using ViBiDSampler. (b) A comparison of results with three baseline methods---FILM, TRF, and Generative Inbetweening (GI)---demonstrates that these baselines exhibit artifacts or unnatural appearances. In contrast, our method generates clear and realistic frames.
• Figure 2: Comparison of denoising processes. (a) Time Reversal Fusion method and (b) bidirectional sampling (Ours).
• Figure 3: Comparison of diffusion sampling paths. (a) Existing methods encounter off-manifold issues due to the averaging of two sample points. (b) In contrast, our bidirectional sampling sequentially connects the temporally forward and backward paths, ensuring that the process remains within the manifold.
• Figure 4: Qualitative evaluation compared to three baselines: FILM, TRF, and Generative Inbetweening. The start and end frames ($I_0$, $I_{24}$) are used as keyframes. While FILM encounters difficulties in capturing motion when there is a significant discrepancy between the two keyframes, and TRF and Generative Inbetweening experience a decline in perceptual quality due to the blurring of objects within the image, our method successfully captures motion while maintaining high fidelity in the generated images.
• Figure 5: Ablation study on the effects of CFG++ and DDS. The inclusion of CFG++ and DDS results in improved perceptual quality in the generated frames.