FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection

TL;DR

FCOS3D presents a simple, single-stage monocular 3D detector by repurposing FCOS: it projects 7-DoF 3D targets to the image plane, decouples them into 2D and 3D attributes, and trains with 3D-center-based target assignment and a 2D Gaussian center-ness. It introduces 2D-guided multi-level prediction and a distance-based target assignment to robustly allocate targets across FPN levels, along with a 3D center-ness formulation aligned to the projected 3D center. The approach achieves state-of-the-art performance among vision-only methods on the nuScenes camera track and demonstrates efficiency advantages, while ablations validate the effectiveness of depth-space reformulation, center-ness, and multi-head regression. The method emphasizes simplicity and practicality, removing the need for 2D-3D priors and auxiliary sub-networks, with strong implications for accessible monocular 3D perception in autonomous systems.

Abstract

Monocular 3D object detection is an important task for autonomous driving considering its advantage of low cost. It is much more challenging than conventional 2D cases due to its inherent ill-posed property, which is mainly reflected in the lack of depth information. Recent progress on 2D detection offers opportunities to better solving this problem. However, it is non-trivial to make a general adapted 2D detector work in this 3D task. In this paper, we study this problem with a practice built on a fully convolutional single-stage detector and propose a general framework FCOS3D. Specifically, we first transform the commonly defined 7-DoF 3D targets to the image domain and decouple them as 2D and 3D attributes. Then the objects are distributed to different feature levels with consideration of their 2D scales and assigned only according to the projected 3D-center for the training procedure. Furthermore, the center-ness is redefined with a 2D Gaussian distribution based on the 3D-center to fit the 3D target formulation. All of these make this framework simple yet effective, getting rid of any 2D detection or 2D-3D correspondence priors. Our solution achieves 1st place out of all the vision-only methods in the nuScenes 3D detection challenge of NeurIPS 2020. Code and models are released at https://github.com/open-mmlab/mmdetection3d.

Figures (6)

• Figure 1: Illustration of 2D detection and monocular 3D object detection. Given an input RGB image, a 2D anchor-free detector needs to predict the distance from a foreground point to four box sides. In contrast, a monocular 3D anchor-free detector needs to predict a transformed 3D-center, 3D size, and orientation of the object.
• Figure 2: An overview of our pipeline. To leverage the well-developed 2D feature extractors, we basically follow the typical design of backbone and neck for 2D detectors. For detection head, we first reformulate the 3D targets with center-based paradigm to decouple it as multi-task learning. The strategies for multi-level target assignment and center sampling are further adjusted accordingly to equip this framework with the better capability of handling overlapped ground truths and scale variance problem.
• Figure 3: Our exploited rotation encoding scheme. Two objects with opposite orientations share the same rotation offset based on the 2-bin boundary, thus have the same $sin$ value. To distinguish them, we predict an additional direction class from the regression branch.
• Figure 4: Our proposed distance-based target assignment for dealing with ambiguity case could significantly improve the best possible recall (BPR) for each class, especially for large objects like trailers. Construction vehicle and traffic cone are abbreviated as CV and TC in this figure.
• Figure 5: Qualitative analysis of detection results. 3D bounding boxes predictions are projected onto images from six different views and bird-view, respectively. Boxes from different categories are marked with different colors. We can see that the results are reasonable except for some detection with false class predictions from the left part. Moreover, a few small objects are detected by our model while not annotated as ground truth, like barriers in the back/back right camera. However, apart from the inherent occlusion problem in this setting, depth and orientation estimations of some objects are still inaccurate, which can be observed in the visualization from bird-view.