MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders

TL;DR

Monocular 3D detector inspired by Masked Autoencoders is designed that addresses the object occlusion issue by masking and reconstructing objects in the feature space by learning enriched 3D representations that achieve superior monocular 3D detection performance qualitatively and quantitatively for both occluded and non-occluded objects.

Abstract

Monocular 3D object detection aims for precise 3D localization and identification of objects from a single-view image. Despite its recent progress, it often struggles while handling pervasive object occlusions that tend to complicate and degrade the prediction of object dimensions, depths, and orientations. We design MonoMAE, a monocular 3D detector inspired by Masked Autoencoders that addresses the object occlusion issue by masking and reconstructing objects in the feature space. MonoMAE consists of two novel designs. The first is depth-aware masking that selectively masks certain parts of non-occluded object queries in the feature space for simulating occluded object queries for network training. It masks non-occluded object queries by balancing the masked and preserved query portions adaptively according to the depth information. The second is lightweight query completion that works with the depth-aware masking to learn to reconstruct and complete the masked object queries. With the proposed object occlusion and completion, MonoMAE learns enriched 3D representations that achieve superior monocular 3D detection performance qualitatively and quantitatively for both occluded and non-occluded objects. Additionally, MonoMAE learns generalizable representations that can work well in new domains.

Figures (4)

• Figure 1: Object occlusion is pervasive and affects monocular 3D detection: Object occlusion is pervasive, e.g., 62% (17725) cars in the KITTI 3D dataset suffer from various occlusions as illustrated in (a). Prevalent monocular 3D detection techniques such as GUPNet lu2021geometry and MonoDETR zhang2023monodetr are clearly affected by object occlusions in both 3D space (3D) and the bird’s eye view (BEV) space as in (b). The proposed MonoMAE simulates and learns object occlusions by feature masking and completing which improves detection consistently for both occluded and non-occluded objects.
• Figure 2: The framework of MonoMAE training: Given a single-view image, the 3D Backbone extracts 3D object query features which are grouped into non-occluded query features and occluded query features by the Non-Occluded Query Grouping. The Depth-Aware Masking then masks the non-occluded query features to simulate object occlusions adaptively based on the object depth, and the Completion Network then learns to reconstruct the masked queries. Finally, the completed and the occluded query features are concatenated to train the 3D Detection Head for 3D predictions.
• Figure 3: Illustration of the Depth-Aware Masking. (a) Objects farther away are usually smaller capturing less visual information. (b) The Depth-Aware Masking determines the mask ratio of an object according to its depth - the closer the object is, the larger the mask ratio is applied, thereby compensating the information deficiency for objects that have larger distances from the camera.
• Figure 4: Detection visualization over the KITTI val set. Ground-truth annotations are highlighted by red boxes, and predictions by MonoMAE and two state-of-the-art methods are highlighted by green boxes. Red arrows highlight objects that have very different predictions across the compared methods. The ground truth of LiDAR point clouds is provided for visualization only, and they are not used in MonoMAE training. Best viewed in color and zoom-in.