Bilateral Event Mining and Complementary for Event Stream Super-Resolution

TL;DR

The paper tackles the challenge of insufficient spatial resolution in event streams by decoupling positive and negative events and modeling their cross-polarity correlations. It introduces BMCNet, a two-stream network augmented with Bilateral Information Exchange (BIE) and Cross-Layer Interaction Representations (CIR) to exchange hierarchical global spatiotemporal context between the two event types. Training uses a segment-based mean-squared-error loss between the predicted high-resolution event counts $E_t^{SR}$ and ground-truth $E_t^{HR}$ over segments of length $T=9$. Empirically, BMCNet achieves over 11% ESR improvement on synthetic and real datasets, improves downstream tasks such as object recognition and video reconstruction, and offers favorable parameter/FLOP efficiency compared with prior ESR approaches.

Abstract

Event Stream Super-Resolution (ESR) aims to address the challenge of insufficient spatial resolution in event streams, which holds great significance for the application of event cameras in complex scenarios. Previous works for ESR often process positive and negative events in a mixed paradigm. This paradigm limits their ability to effectively model the unique characteristics of each event and mutually refine each other by considering their correlations. In this paper, we propose a bilateral event mining and complementary network (BMCNet) to fully leverage the potential of each event and capture the shared information to complement each other simultaneously. Specifically, we resort to a two-stream network to accomplish comprehensive mining of each type of events individually. To facilitate the exchange of information between two streams, we propose a bilateral information exchange (BIE) module. This module is layer-wisely embedded between two streams, enabling the effective propagation of hierarchical global information while alleviating the impact of invalid information brought by inherent characteristics of events. The experimental results demonstrate that our approach outperforms the previous state-of-the-art methods in ESR, achieving performance improvements of over 11\% on both real and synthetic datasets. Moreover, our method significantly enhances the performance of event-based downstream tasks such as object recognition and video reconstruction. Our code is available at https://github.com/Lqm26/BMCNet-ESR.

Figures (12)

• Figure 1: Illustration of various approaches in event stream super resolution for processing positive and negative events.
• Figure 2: Overall framework of our BMCNet. Blue/red represent events with positive/negative polarity, respectively. BMCNet consists of two parallel streams, each dedicated to mining the information of two events with different polarities. The inter-stream BIE is applied to exchange and complement global structures between two events. Additionally, an inner-stream BIE is embedded within each stream to model the spatio-temporal context of each event. Through the layer-wise introduction of inter- and inner-stream BIE, BMCNet can effectively model hierarchical spatio-temporal contextual correlations between different events, thereby enhancing the performance in ESR.
• Figure 3: The architecture of the proposed bilateral information exchange (BIE) module. [Best viewed with zoom-in.]
• Figure 4: Qualitative analysis comparison on synthetic and real datasets. The upper and lower sections represent the $4\times$ super-resolution results for NFS-syn and EventNFS, respectively. "GT" denotes the $4\times$ ground-truth. Our BMCNet-plain and BMCNet demonstrates superior detail recovery and clearer edges on both datasets. [Best viewed with zoom-in.]
• Figure 5: Qualitative comparison on the real dataset EventNFS-real between the mixed and decoupled paradigm.