BAT: Learning Event-based Optical Flow with Bidirectional Adaptive Temporal Correlation

TL;DR

BAT introduces bidirectional adaptive temporal correlation (BTC) to convert temporally dense event cues into spatially dense representations for optical flow estimation. It further employs adaptive temporal sampling (ATS) and spatially adaptive temporal motion aggregation (SATMA) to maintain temporal consistency and selectively fuse motion features, all within an iterative ConvGRU-based update. The approach achieves state-of-the-art results on DSEC-Flow, outperforms prior methods by large margins, and can predict future optical flow using only past events while better handling occlusions. This yields sharper edges, richer details, and a strong new baseline for event-based optical flow research with practical potential for fast robotics and dynamic scene understanding.

Abstract

Event cameras deliver visual information characterized by a high dynamic range and high temporal resolution, offering significant advantages in estimating optical flow for complex lighting conditions and fast-moving objects. Current advanced optical flow methods for event cameras largely adopt established image-based frameworks. However, the spatial sparsity of event data limits their performance. In this paper, we present BAT, an innovative framework that estimates event-based optical flow using bidirectional adaptive temporal correlation. BAT includes three novel designs: 1) a bidirectional temporal correlation that transforms bidirectional temporally dense motion cues into spatially dense ones, enabling accurate and spatially dense optical flow estimation; 2) an adaptive temporal sampling strategy for maintaining temporal consistency in correlation; 3) spatially adaptive temporal motion aggregation to efficiently and adaptively aggregate consistent target motion features into adjacent motion features while suppressing inconsistent ones. Our results rank $1^{st}$ on the DSEC-Flow benchmark, outperforming existing state-of-the-art methods by a large margin while also exhibiting sharp edges and high-quality details. Notably, our BAT can accurately predict future optical flow using only past events, significantly outperforming E-RAFT's warm-start approach. Code: \textcolor{magenta}{https://github.com/gangweiX/BAT}.

Figures (9)

• Figure 1: Left: Comparison with state-of-the-art event-based flow methods on DSEC-Flow and MVSEC benchmarks. Our BAT achieves the highest accuracy among all published methods. Right: Compared to E-RAFT and TMA, our BAT shows much higher efficiency in iterative optimization. Additionally, the aforementioned methods converge at a high error rate quickly due to limited unidirectional motion cues. In contrast, our BAT fully utilizes bidirectional and rich motion cues, allowing for continuous optimization for better results.
• Figure 2: Benefiting from the proposed backward temporal correlation, our method effectively handles occlusions caused by objects moving out of the target frame.
• Figure 3: Network architecture of the proposed BAT. We first split the reference and target event streams into multiple groups and extract the corresponding features separately. Then, we perform forward and backward temporal correlations between the sequential event features. We further propose spatially adaptive temporal motion aggregation, which integrates temporally consistent target motion features into adjacent motion features while suppressing inconsistent ones.
• Figure 4: Future optical flow prediction. The first row illustrates events from timestamp $t_{i-1}$ to $t_i$, while the second row presents the optical flow results from $t_{i}$ to $t_{i+1}$. Given only past events, our method can predict future optical flow.
• Figure 5: Qualitative results of optical flow predictions on DSEC-Flow. Significant improvements are highlighted by red boxes. Images are provided for visualization only, since the optical flow is event-based. Our method accurately distinguishes subtle details and sharp edges. Zoom in for a clearer view.