Efficient Event Stream Super-Resolution with Recursive Multi-Branch Fusion

TL;DR

An efficient Recursive Multi-Branch Information Fusion Network (RMFNet) that separates positive and negative events for complementary information extraction, followed by mutual supplementation and refinement is proposed, and Feature Fusion Modules and Feature Exchange Modules are introduced.

Abstract

Current Event Stream Super-Resolution (ESR) methods overlook the redundant and complementary information present in positive and negative events within the event stream, employing a direct mixing approach for super-resolution, which may lead to detail loss and inefficiency. To address these issues, we propose an efficient Recursive Multi-Branch Information Fusion Network (RMFNet) that separates positive and negative events for complementary information extraction, followed by mutual supplementation and refinement. Particularly, we introduce Feature Fusion Modules (FFM) and Feature Exchange Modules (FEM). FFM is designed for the fusion of contextual information within neighboring event streams, leveraging the coupling relationship between positive and negative events to alleviate the misleading of noises in the respective branches. FEM efficiently promotes the fusion and exchange of information between positive and negative branches, enabling superior local information enhancement and global information complementation. Experimental results demonstrate that our approach achieves over 17% and 31% improvement on synthetic and real datasets, accompanied by a 2.3X acceleration. Furthermore, we evaluate our method on two downstream event-driven applications, \emph{i.e.}, object recognition and video reconstruction, achieving remarkable results that outperform existing methods. Our code and Supplementary Material are available at https://github.com/Lqm26/RMFNet.

Figures (3)

• Figure 1: Compared to previous ESR methods that directly mix positive and negative events, our multi-branch approach effectively extracts and integrates features from positive and negative events, achieving a more complete and clearer details (see the green box).
• Figure 2: Architecture of our proposed Recursive Multi-Branch Information Fusion Network (RMFNet). Initially, the event frame is fused into positive and negative branches along with the previous output $O_{t-1}$ and state $h_{t-1}$ using the Feature Fusion Module (bottom left). Subsequently, each branch independently extracts features through Residual Blocks, and a Feature Exchange Module (bottom) facilitates the exchange of information between the branches. Finally, the features from the positive and negative branches are concatenated, and high-resolution event count images $O_t$ are obtained through Pixel Shuffle operation.
• Figure 3: Qualitative analysis comparison on synthetic and real datasets. The upper and lower figures represent $4\times$ SR results on the NFS-syn and EventNFS datasets, respectively. It is evident that our RMFNet excels in recovering finer details of the event streams on both datasets (see the green box), resulting in sharper edges. Positive events are in blue, negative events in red. Zoom in for the best view.