EvTexture: Event-driven Texture Enhancement for Video Super-Resolution

TL;DR

EvTexture introduces an event-driven texture enhancement framework for video super-resolution, addressing the long-standing challenge of restoring fine textures. The method deploys a two-branch bidirectional recurrent architecture that combines a motion branch with a texture enhancement branch, and it employs an Iterative Texture Enhancement (ITE) module to progressively inject high-frequency information from event voxel grids into texture-rich features. Empirical results show state-of-the-art performance across four benchmarks, with substantial gains on texture-rich clips (e.g., up to +4.67 dB PSNR on Vid4) and improved temporal consistency, validating the importance of leveraging event cues for texture restoration. An extension, EvTexture+, further integrates event-driven motion cues to boost performance, underscoring the practical impact of incorporating neuromorphic signals into VSR for texture-focused restoration.

Abstract

Event-based vision has drawn increasing attention due to its unique characteristics, such as high temporal resolution and high dynamic range. It has been used in video super-resolution (VSR) recently to enhance the flow estimation and temporal alignment. Rather than for motion learning, we propose in this paper the first VSR method that utilizes event signals for texture enhancement. Our method, called EvTexture, leverages high-frequency details of events to better recover texture regions in VSR. In our EvTexture, a new texture enhancement branch is presented. We further introduce an iterative texture enhancement module to progressively explore the high-temporal-resolution event information for texture restoration. This allows for gradual refinement of texture regions across multiple iterations, leading to more accurate and rich high-resolution details. Experimental results show that our EvTexture achieves state-of-the-art performance on four datasets. For the Vid4 dataset with rich textures, our method can get up to 4.67dB gain compared with recent event-based methods. Code: https://github.com/DachunKai/EvTexture.

Figures (19)

• Figure 1: Comparative results of VSR methods on the City clip in Vid4 liu2013bayesian. It can be observed that current VSR methods, with lu2023learningkai2023video or without event signals chan2022basicvsr++, still suffer from blurry textures or jitter effects, resulting in large errors in texture regions. In contrast, our method can predict the texture regions successfully and greatly reduce errors in the restored frames.
• Figure 2: Comparisons of VSR learning process. RGB-based methods chan2022basicvsr++lin2022unsupervised usually focus on motion leaning to recover the missing details from other unaligned frames. Previous event-based methods lu2023learningkai2023video have attempted to use events to enhance the motion learning. In contrast, our method is the first to utilize events to enhance the texture restoration in VSR. The red dotted line is an optional branch, where our method can easily adapt to approaches that use events to enhance the motion learning.
• Figure 3: Network architecture of EvTexture. (a) EvTexture adopts a bidirectional recurrent network, where features are propagated forward and backward. At each timestamp, it includes a motion branch and a parallel texture branch to explicitly enhance the restoration of texture regions. (b) In the texture branch, the ITE module plays a key role. It progressively refines the feature across multiple iterations, leveraging high-frequency textural information from events along with context information from the current frame.
• Figure 4: Qualitative comparison on Vid4 liu2013bayesian. Our method can restore more vivid branches and leaves on the tulip tree.
• Figure 5: Qualitative comparison on Vimeo-90K-T xue2019video. Our method can restore more detailed stripes on clothing surfaces.