Event-Enhanced Blurry Video Super-Resolution

TL;DR

This work addresses blurry video super-resolution (BVSR) by introducing event signals to augment frame information, mitigating deconvolution ambiguities and high-frequency detail loss. The proposed Ev-DeblurVSR jointly fuses intra-frame and inter-frame events with RGB frames through two key modules: Reciprocal Feature Deblurring (RFD), which refines frame features using motion cues from intra-frame events while enriching event features with scene context, and Hybrid Deformable Alignment (HDA), which blends inter-frame event trajectories with optical flow to improve motion estimation for deformable alignment. An edge-enhanced loss further emphasizes high-frequency regions during training, and extensive experiments on synthetic and real-world datasets show state-of-the-art performance with superior spatial recovery and temporal consistency, including a notable real-world gain of +2.59 dB over a strong BVSR baseline and significantly faster inference. The work demonstrates the practical potential of event data to enhance BVSR in challenging scenes, offering a new direction for joint frame-event video restoration methods.

Abstract

In this paper, we tackle the task of blurry video super-resolution (BVSR), aiming to generate high-resolution (HR) videos from low-resolution (LR) and blurry inputs. Current BVSR methods often fail to restore sharp details at high resolutions, resulting in noticeable artifacts and jitter due to insufficient motion information for deconvolution and the lack of high-frequency details in LR frames. To address these challenges, we introduce event signals into BVSR and propose a novel event-enhanced network, Ev-DeblurVSR. To effectively fuse information from frames and events for feature deblurring, we introduce a reciprocal feature deblurring module that leverages motion information from intra-frame events to deblur frame features while reciprocally using global scene context from the frames to enhance event features. Furthermore, to enhance temporal consistency, we propose a hybrid deformable alignment module that fully exploits the complementary motion information from inter-frame events and optical flow to improve motion estimation in the deformable alignment process. Extensive evaluations demonstrate that Ev-DeblurVSR establishes a new state-of-the-art performance on both synthetic and real-world datasets. Notably, on real data, our method is +2.59 dB more accurate and 7.28$\times$ faster than the recent best BVSR baseline FMA-Net. Code: https://github.com/DachunKai/Ev-DeblurVSR.

Figures (13)

• Figure 1: An example (a) from a challenging blurry video enhanced by (b) SOTA methods in video deblurring zhang2024blur + VSR xu2024enhancing; (c) a SOTA BVSR method youk2024fma; and (e) our event-enhanced approach. Our method can restore the license plate with finer details, sharper edges, and less aliasing.
• Figure 2: Proposed event processing for BVSR. The exposure time of $B^{LR}_t$ is $\mathcal{T}_t$. Intra-frame events $\mathcal{E}^{LR}_{t}$ capture motion within the exposure, which is used to deblur frame features. Inter-frame events $\mathcal{E}^{LR}_{t-1\to t}$ capture motion between frames, which helps enhance temporal alignment in VSR.
• Figure 3: Overview of Ev-DeblurVSR. Intra-frame voxels are fused with blurry frames in the RFD module to deblur frame features and enhance event features with scene context. Inter-frame voxels are integrated into the HDA module, using continuous motion trajectories to guide deformable alignment. Finally, the aligned features are upsampled to reconstruct sharp HR frames.
• Figure 4: The structure of the RFD module.
• Figure 5: The structure of the HDA module.