AsyncBEV: Cross-modal Flow Alignment in Asynchronous 3D Object Detection

TL;DR

AsyncBEV addresses the practical challenge of sensor asynchrony in autonomous driving by introducing $\Delta$-BEVFlow, a BEV-space flow estimator that aligns multi-modal features from asynchronous sensors. It supports both token-based and grid-based detectors through detector-specific warpers and offers two flow formulations, with velocity-based BEV-VE chosen as the default for its regularization benefits. The approach yields substantial improvements on dynamic objects under large offsets (up to $0.5\,s$), while incurring minimal computational overhead, and it remains compatible with existing BEV detectors. By enabling robust multi-modal fusion under real-world timing variations, AsyncBEV enhances safety-critical perception without requiring full reengineering of detection architectures. The work also provides a pathway to extend to multi-sensor asynchronous setups and streaming fusion in future deployments.

Abstract

In autonomous driving, multi-modal perception tasks like 3D object detection typically rely on well-synchronized sensors, both at training and inference. However, despite the use of hardware- or software-based synchronization algorithms, perfect synchrony is rarely guaranteed: Sensors may operate at different frequencies, and real-world factors such as network latency, hardware failures, or processing bottlenecks often introduce time offsets between sensors. Such asynchrony degrades perception performance, especially for dynamic objects. To address this challenge, we propose AsyncBEV, a trainable lightweight and generic module to improve the robustness of 3D Birds' Eye View (BEV) object detection models against sensor asynchrony. Inspired by scene flow estimation, AsyncBEV first estimates the 2D flow from the BEV features of two different sensor modalities, taking into account the known time offset between these sensor measurements. The predicted feature flow is then used to warp and spatially align the feature maps, which we show can easily be integrated into different current BEV detector architectures (e.g., BEV grid-based and token-based). Extensive experiments demonstrate AsyncBEV improves robustness against both small and large asynchrony between LiDAR or camera sensors in both the token-based CMT and grid-based UniBEV, especially for dynamic objects. We significantly outperform the ego motion compensated CMT and UniBEV baselines, notably by $16.6$ % and $11.9$ % NDS on dynamic objects in the worst-case scenario of a $0.5 s$ time offset. Code will be released upon acceptance.

Figures (7)

• Figure 1: Multi-modal 3D object detection under sensor asynchrony. a) shows a general concept of asynchronous 3D object detection. b) shows multi-modal 3D object detection under sensor asynchrony of a pretrained CMT yan2023cmt. Predictions have a large translation error. c) shows our AsyncBEV improves CMT's robustness against sensor asynchrony and mostly corrects the spatial misalignment of predicted boxes.
• Figure 2: Overview of AsyncBEV. a) demonstrates an example pipeline of asynchronous multi-modal 3D object detection. In this setup, images are sampled at the reference timestamp $t_{1}$, while LiDAR point clouds were obtained at an earlier timestamp $t_{0}$. b) shows the $\Delta$-BEVFlow estimation, the core module of AsyncBEV. It takes asynchronous BEV features as input and predicts the velocity of each BEV cell. The BEV flow is calculated by multiplying the velocity by $\Delta t$ and supervised by the ground truth dense flow representation. Finally, the predicted flow is used to spatially align the features from $t_{0}$ to $t_{1}$ with the detector-specific warper.
• Figure 3: Detector-specific Warper. a) Token-based methods rely on 3D coordinates of tokens to provide spatial guidance. Generated $\Delta$-BEVFlow is used to correct the corresponding 3D coordinates of tokens from the asynchronous sensor. b) Grid-based methods project the multi-modal features into predefined BEV grids. Generated $\Delta$-BEVFlow is used to generate a look-up table from the reference timestamp to the asynchronous timestamp to warp the asynchronous BEV features to the reference timestamp.
• Figure 4: Performance of different methods with increasing time offsets under LiDAR asynchrony.
• Figure 5: Qualitative Results of AsyncBEV. The left two columns demonstrate the 3D object detection performance of CMT and CMT+AsyncBEV under $0.5 s$ LiDAR asynchrony. The right two columns show the output of our predicted $\Delta$-BEVFlow and the ground truth $\Delta$-BEVFlow.