EvEnhancer: Empowering Effectiveness, Efficiency and Generalizability for Continuous Space-Time Video Super-Resolution with Events

TL;DR

EvEnhancer tackles the challenge of continuous space-time video super-resolution by integrating asynchronous event streams with frame-based frames to enable arbitrary spatial and temporal upsampling. It introduces two key components: the Event-adapted Synthesis Module (EASM) for long-range motion modeling via event-guided alignment and bidirectional recurrence, and the Local Implicit Video Transformer (LIVT) for learning a unified continuous video INR through cross-scale spatiotemporal attention. Across synthetic and real-world datasets, EvEnhancer outperforms state-of-the-art methods in both in-distribution and out-of-distribution scales, while offering a lighter-weight variant with strong performance. The approach provides a practical pathway to high-quality, flexible video synthesis for applications requiring variable resolution and framerate with robust temporal consistency.

Abstract

Continuous space-time video super-resolution (C-STVSR) endeavors to upscale videos simultaneously at arbitrary spatial and temporal scales, which has recently garnered increasing interest. However, prevailing methods struggle to yield satisfactory videos at out-of-distribution spatial and temporal scales. On the other hand, event streams characterized by high temporal resolution and high dynamic range, exhibit compelling promise in vision tasks. This paper presents EvEnhancer, an innovative approach that marries the unique advantages of event streams to elevate effectiveness, efficiency, and generalizability for C-STVSR. Our approach hinges on two pivotal components: 1) Event-adapted synthesis capitalizes on the spatiotemporal correlations between frames and events to discern and learn long-term motion trajectories, enabling the adaptive interpolation and fusion of informative spatiotemporal features; 2) Local implicit video transformer integrates local implicit video neural function with cross-scale spatiotemporal attention to learn continuous video representations utilized to generate plausible videos at arbitrary resolutions and frame rates. Experiments show that EvEnhancer achieves superiority on synthetic and real-world datasets and preferable generalizability on out-of-distribution scales against state-of-the-art methods. Code is available at https://github.com/W-Shuoyan/EvEnhancer.

Figures (11)

• Figure 1: Performance comparison of different C-STVSR methods. (a) PSNR (dB) comparison for in-distribution (In-dist.) spatiotemporal upsampling scale (temporal scale $t=8$, spatial scale $s=4$) on the different datasets including GoPro nah2017deep, Adobe240 su2017deep and BS-ERGB tulyakov2022time. (b) PSNR (dB) comparison for different spatiotemporal upsampling scales on GoPro. (c) Visualization comparison for In-dist. scale on GoPro.
• Figure 2: The overall architecture of our EvEnhancer, an event-driven C-STVSR method which consists of an event-adapted synthesis module (EASM), and a local implicit video transformer (LIVT), where EASM contains two steps: (a) event-modulated alignment (EMA), and (b) bidirectional recurrent compensation (BRC). "CS": channel squeeze, "FFN": feed-forward neural network, "Res": residual blocks.
• Figure 3: The structure of our local implicit video transformer (LIVT), which integrates the 3D local spatiotemporal attention with implicit neural function to learn continuous video INR to reconstruct HR and HFR video frames.
• Figure 4: Qualitative comparison for In-dist. scale ($t=8,s=4$) on the GoPro dataset nah2017deep. We compare the normalized absolute difference maps (yellow boxes) for the same regions in each frame as in the GT frames.
• Figure 5: Comparison of temporal profile on the GoPro dataset nah2017deep ($t=12,s=6$). We select a column (orange dotted lines) and observe the changes across time.