EventDiff: A Unified and Efficient Diffusion Model Framework for Event-based Video Frame Interpolation

TL;DR

EventDiff introduces a unified diffusion-based framework for event-based video frame interpolation by coupling a novel Event-Frame Hybrid AutoEncoder (HAE) with a Spatial-Temporal Cross Attention module and performing interpolation directly in the latent space. The model is trained in two stages: Stage I pretrains the HAE on ground-truth I-E pairs to produce a ground-truth embedding $z_{gt}$ and multi-scale hybrid features, while Stage II jointly optimizes a latent diffusion process conditioned on boundary I-E features to reconstruct $z_{gt}$ with a fast 5-step inference. This approach yields state-of-the-art PSNR/SSIM gains on synthetic and real-world datasets, and achieves up to $4.24\times$ faster inference compared to prior diffusion-based VFI methods due to the reduced iteration count and a lightweight autoencoder. The framework also demonstrates extensibility to event-based motion deblurring, underscoring its potential as a general-purpose, event-enhanced visual generation paradigm.

Abstract

Video Frame Interpolation (VFI) is a fundamental yet challenging task in computer vision, particularly under conditions involving large motion, occlusion, and lighting variation. Recent advancements in event cameras have opened up new opportunities for addressing these challenges. While existing event-based VFI methods have succeeded in recovering large and complex motions by leveraging handcrafted intermediate representations such as optical flow, these designs often compromise high-fidelity image reconstruction under subtle motion scenarios due to their reliance on explicit motion modeling. Meanwhile, diffusion models provide a promising alternative for VFI by reconstructing frames through a denoising process, eliminating the need for explicit motion estimation or warping operations. In this work, we propose EventDiff, a unified and efficient event-based diffusion model framework for VFI. EventDiff features a novel Event-Frame Hybrid AutoEncoder (HAE) equipped with a lightweight Spatial-Temporal Cross Attention (STCA) module that effectively fuses dynamic event streams with static frames. Unlike previous event-based VFI methods, EventDiff performs interpolation directly in the latent space via a denoising diffusion process, making it more robust across diverse and challenging VFI scenarios. Through a two-stage training strategy that first pretrains the HAE and then jointly optimizes it with the diffusion model, our method achieves state-of-the-art performance across multiple synthetic and real-world event VFI datasets. The proposed method outperforms existing state-of-the-art event-based VFI methods by up to 1.98dB in PSNR on Vimeo90K-Triplet and shows superior performance in SNU-FILM tasks with multiple difficulty levels. Compared to the emerging diffusion-based VFI approach, our method achieves up to 5.72dB PSNR gain on Vimeo90K-Triplet and 4.24X faster inference.

Figures (8)

• Figure 1: EventDiff compared with previous Event-based VFI methods. (a) The traditional event-based VFI methods, such as Timelens tulyakov2021time et al., use multi-component architectures with separate warping, synthesis and refinement modules. These methods rely on intermediate products like optical flow serving as manually designed auxiliary elements for fusing frames and events, which make bottlenecks and lead to suboptimal results. (b) The proposed EventDiff employs a unified and efficient autoencoder framework that directly optimizes against ground-truth frames without requiring handcrafted intermediate products. Instead, it utilizes a diffusion model to reconstruct the ground-truth embedding in the latent space, enabling more accurate VFI preformance.
• Figure 2: Illustration of the EventDiff framework. The framework consists of two training stages. In Stage 1 (left), we train the Event-Frame Hybrid AutoEncoder (HAE) using ground-truth inputs, enabling it to capture hybrid pyramid features and project the ground-truth into the latent space. In Stage 2 (right), the ground-truth inputs are removed and a diffusion model is employed to reconstruct the ground-truth embedding within the latent space.
• Figure 3: Illustration of the Event-Frame Hybrid AutoEncoder (HAE) architecture.
• Figure 4: Illustration of (a) Temporal Cross Attention (TCA) and (b) Spatial Cross Attention (SCA) modules.
• Figure 5: Illustration of the U-net with a condition encoder for diffusion model.