BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereo

TL;DR

<3-5 sentence high-level summary> BEVStereo tackles the depth-ambiguity challenge in camera-based multi-view 3D object detection by introducing a dynamic temporal stereo framework that adaptively samples depth candidates via a depth center mu and depth range sigma. It fuses mono-depth with stereo-depth through a Weight Net and refines depth with an EM-like iterative update, enabling robust performance for moving objects and static ego-vehicles. The approach also includes a size-aware Circle NMS to improve duplicate suppression and an Efficient Voxel Pooling v2 to accelerate computation. Experiments on nuScenes demonstrate state-of-the-art camera-based results with improved robustness and memory efficiency compared to MVS-based baselines. The work advances practical, high-performance camera-only 3D detection for autonomous driving and provides code for replication.

Abstract

Bounded by the inherent ambiguity of depth perception, contemporary camera-based 3D object detection methods fall into the performance bottleneck. Intuitively, leveraging temporal multi-view stereo (MVS) technology is the natural knowledge for tackling this ambiguity. However, traditional attempts of MVS are flawed in two aspects when applying to 3D object detection scenes: 1) The affinity measurement among all views suffers expensive computation cost; 2) It is difficult to deal with outdoor scenarios where objects are often mobile. To this end, we introduce an effective temporal stereo method to dynamically select the scale of matching candidates, enable to significantly reduce computation overhead. Going one step further, we design an iterative algorithm to update more valuable candidates, making it adaptive to moving candidates. We instantiate our proposed method to multi-view 3D detector, namely BEVStereo. BEVStereo achieves the new state-of-the-art performance (i.e., 52.5% mAP and 61.0% NDS) on the camera-only track of nuScenes dataset. Meanwhile, extensive experiments reflect our method can deal with complex outdoor scenarios better than contemporary MVS approaches. Codes have been released at https://github.com/Megvii-BaseDetection/BEVStereo.

Figures (6)

• Figure 1: Framework of BEVStereo. The Depth Module uses the image feature of the reference frame and source frame as input to generate $\mu$, $\sigma$, context, and mono depth. Stereo depth is produced using $\mu$ and $\sigma$. Weight Net uses $\mu$ and the mono depth of two frames to create a weight map that is applied to the stereo depth. Mono depth and weighted stereo depth are accumulated together to create the final depth. BEV Feature is produced when context is combined with it and is used by the detecting head.
• Figure 2: Iterative process of $\mu$ and $\sigma$. The initial $\mu$ and $\sigma$ are generated using feature of the reference frame as input. For each round of iteration, $\mu$ and $\sigma$ are used for homography warping to fetch the source feature. Similarity Net takes the inner product results of warpped source feature and reference feature as input to generate depth confidence which is used to update $\mu$ and $\sigma$.
• Figure 3: Drawbacks of circle NMS. In the left part of the figure, despite having distinct IoUs, the blue boxes and red boxes share the same center distance as long as their centers coincide. In the right part of the figure, when the green box has the highest score, the red box is more likely to be suppressed since its center is closer to the green box's center which goes against our common sense.
• Figure 4: Thread mapping of point features to BEV features. Based on the point coordinates, the point features are atomically accumulated into the corresponding BEV features. Each thread block loads the point coordinates it is responsible for into the shared memory.
• Figure 5: Visualization of depth prediction. The blue area is the distribution of depth prediction, while the green line represents the depth GT produced by the point cloud. The red dotted boxes denotes the promotion of depth prediction on moving objects and the yellow dotted boxes denotes the the promotion of depth prediction on static objects.