Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow

TL;DR

This work tackles unsupervised event-based optical flow estimation by addressing the limitations of frame-based and linear-motion approaches, especially for long sequences. It introduces E-NMSTFlow, a framework combining Spatio-Temporal Motion Feature Aware (STMFA) and Adaptive Motion Feature Enhancement (AMFE) to capture rich spatio-temporal motion cues, together with a nonlinear motion compensation loss (NLMC) that accounts for complex, non-linear motion between events. The method leverages event count images as input, a ConvGRU-based backbone, and cross-attention to fuse past and present features, achieving state-of-the-art results among unsupervised methods on MVSEC and DSEC-Flow, including robust performance for long-term sequences. Overall, the approach demonstrates that explicit nonlinear motion modeling and robust spatio-temporal data associations substantially improve event-based optical flow estimation in challenging real-world scenes.

Abstract

Event cameras have the potential to capture continuous motion information over time and space, making them well-suited for optical flow estimation. However, most existing learning-based methods for event-based optical flow adopt frame-based techniques, ignoring the spatio-temporal characteristics of events. Additionally, these methods assume linear motion between consecutive events within the loss time window, which increases optical flow errors in long-time sequences. In this work, we observe that rich spatio-temporal information and accurate nonlinear motion between events are crucial for event-based optical flow estimation. Therefore, we propose E-NMSTFlow, a novel unsupervised event-based optical flow network focusing on long-time sequences. We propose a Spatio-Temporal Motion Feature Aware (STMFA) module and an Adaptive Motion Feature Enhancement (AMFE) module, both of which utilize rich spatio-temporal information to learn spatio-temporal data associations. Meanwhile, we propose a nonlinear motion compensation loss that utilizes the accurate nonlinear motion between events to improve the unsupervised learning of our network. Extensive experiments demonstrate the effectiveness and superiority of our method. Remarkably, our method ranks first among unsupervised learning methods on the MVSEC and DSEC-Flow datasets. Our project page is available at https://wynelio.github.io/E-NMSTFlow.

Figures (6)

• Figure 1: Qualitative examples on DSEC-Flow sequences. (a) Image for visualization only. (b) Events for input. (c) Predicted optical flow by our method.
• Figure 2: The overview of our proposed E-NMSTFlow. First, we extract multi-scale features with a backbone network. Next, Spatio-Temporal Motion Feature Aware (STMFA) module utilizes previous multi-scale features with current low-level features to aggregate spatio-temporal motion information. Then, Adaptive Motion Feature Enhancement (AMFE) module adaptively enhances motion features. Finally, we upsample enhanced motion features and output predicted optical flow.
• Figure 3: Details of Motion Feature Aggregation.
• Figure 4: Qualitative comparison (dt=4) on MVSEC. From left to right: indoor_flying1, indoor_flying2, indoor_flying3, and outdoor_day1.
• Figure 5: Performance comparison of state-of-the-art unsupervised methods in challenging scenes (sudden turns).