EGVD: Event-Guided Video Diffusion Model for Physically Realistic Large-Motion Frame Interpolation

TL;DR

This work addresses the challenge of video frame interpolation under large motion and varied lighting by integrating high-temporal-resolution event data with pre-trained stable video diffusion priors. It introduces the Multi-modal Motion Condition Generator (MMCG) to fuse RGB frames and event signals into diffusion conditioning, and adopts a two-stage training strategy with EDM-inspired normalization to enable efficient adaptation of a stable video diffusion model. Experimental results on real and simulated Event-VFI data show substantial perceptual-quality gains (e.g., LPIPS improvements of 27.4% on Prophensee and 24.1% on BSRGB) and notable PSNR gains on large-motion sequences, demonstrating strong generalization and robustness. The approach offers a practical pathway to physically realistic, high-motion interpolation and provides public code and datasets, highlighting its potential for applications in high-speed video synthesis and analysis.

Abstract

Video frame interpolation (VFI) in scenarios with large motion remains challenging due to motion ambiguity between frames. While event cameras can capture high temporal resolution motion information, existing event-based VFI methods struggle with limited training data and complex motion patterns. In this paper, we introduce Event-Guided Video Diffusion Model (EGVD), a novel framework that leverages the powerful priors of pre-trained stable video diffusion models alongside the precise temporal information from event cameras. Our approach features a Multi-modal Motion Condition Generator (MMCG) that effectively integrates RGB frames and event signals to guide the diffusion process, producing physically realistic intermediate frames. We employ a selective fine-tuning strategy that preserves spatial modeling capabilities while efficiently incorporating event-guided temporal information. We incorporate input-output normalization techniques inspired by recent advances in diffusion modeling to enhance training stability across varying noise levels. To improve generalization, we construct a comprehensive dataset combining both real and simulated event data across diverse scenarios. Extensive experiments on both real and simulated datasets demonstrate that EGVD significantly outperforms existing methods in handling large motion and challenging lighting conditions, achieving substantial improvements in perceptual quality metrics (27.4% better LPIPS on Prophesee and 24.1% on BSRGB) while maintaining competitive fidelity measures. Code and datasets available at: https://github.com/OpenImagingLab/EGVD.

Figures (4)

• Figure 1: Visual comparisons of video frame interpolation (VFI) results across diverse scenes. The top row presents synthetic data generated from a 240fps DJI Action4 video, downsampled to 30fps with simulated event data. The second row shows real-world data captured in low-light conditions, while the third row features real-world data under normal illumination with large motion. We compare RIFE huang2022real (RGB-based VFI), DualSVD wang2024generative (RGB-based VFI with a diffusion model), CBMNet kim2023event (event-based VFI), and our proposed EGVD, which integrates event information within a diffusion-based framework. Our method not only achieves superior interpolation performance, producing sharper reconstructions, but also demonstrates strong generalization and robustness to large motion. See the supplementary video for video results.
• Figure 2: The illustration of our Event-Guided Video Diffusion Model for physically realistic large-motion frame interpolation.
• Figure 3: Qualitative comparison of various VFI methods across multiple real scenes. For example, #i indicates the frame index. The three examples shown from top to bottom are from the real datasets HQEVFI, BSRGB, and ERDS, respectively. Note that all results are generated using a unified set of inference weights, without dataset-specific training. See the supplementary video for more results.
• Figure 4: Qualitative ablation study results under different component configurations. (Top: Low-Light, Bottom: Large Motion)