Mono3DV: Monocular 3D Object Detection with 3D-Aware Bipartite Matching and Variational Query DeNoising

TL;DR

Mono3DV tackles the core challenge of monocular 3D object detection: the mismatch and instability caused by 2D-only bipartite matching when 3D attributes are ill-posed from single images. It introduces a 3D-Aware Bipartite Matching mechanism with a scheduler, along with 3D DeNoising and Variational Query DeNoising to stabilize training and preserve gradient flow. A forward-looking self-distillation strategy further refines predictions across decoder layers. On KITTI, Mono3DV achieves state-of-the-art monocular performance without external data, demonstrating robust 3D localization from a single view and establishing a strong baseline for transformer-based monocular 3D detection.

Abstract

While DETR-like architectures have demonstrated significant potential for monocular 3D object detection, they are often hindered by a critical limitation: the exclusion of 3D attributes from the bipartite matching process. This exclusion arises from the inherent ill-posed nature of 3D estimation from monocular image, which introduces instability during training. Consequently, high-quality 3D predictions can be erroneously suppressed by 2D-only matching criteria, leading to suboptimal results. To address this, we propose Mono3DV, a novel Transformer-based framework. Our approach introduces three key innovations. First, we develop a 3D-Aware Bipartite Matching strategy that directly incorporates 3D geometric information into the matching cost, resolving the misalignment caused by purely 2D criteria. Second, it is important to stabilize the Bipartite Matching to resolve the instability occurring when integrating 3D attributes. Therefore, we propose 3D-DeNoising scheme in the training phase. Finally, recognizing the gradient vanishing issue associated with conventional denoising techniques, we propose a novel Variational Query DeNoising mechanism to overcome this limitation, which significantly enhances model performance. Without leveraging any external data, our method achieves state-of-the-art results on the KITTI 3D object detection benchmark.

Figures (5)

• Figure 1: Limitations of 2D-only Bipartite Matching. A major limitation of using a 2D-only matching cost for bipartite assignment is that it prioritizes high-quality 2D predictions even if their associated 3D bounding boxes are poor. Conversely, superior 3D predictions are often discarded if their 2D projection is merely sufficient. This leads to suboptimal optimization because the model is trained based on 2D fidelity rather than the desired 3D accuracy.
• Figure 2: The overall of our proposed framework Mono3DV. The architecture initially extracts features from a single-view image using an image backbone, which are then fed into a Transformer encoder. The subsequent decoder utilizes both standard learnable queries and supplementary noisy queries generated by a Variational Query Generator. Finally, the loss is determined by subjecting the predictions derived from the learnable queries to 3D-Aware Bipartite Matching.
• Figure 3: The influence of the denoised query on learnable queries. The self-attention maps trend resulting from the conventional denoising method and the proposed Variational Query DeNoising approach are illustrated in (a) and (b), respectively. (c) presents the average entropy of the attention maps for both methods throughout the entire training period.
• Figure 4: Qualitative results on KITTI val set. (a) MonoDETR MonoDETR. (b) MonoDGP MonoDGP. (c) Mono3DV (Ours). For each image set, the top row presents the camera-view visualization, while the bottom row offers the corresponding bird’s-eye view. Ground-truth bounding boxes are rendered in green, and predictions are shown in order: red, yellow, blue. We also circle some objects to highlight the difference between other state of the art and our method.
• Figure 5: Qualitative results on KITTI val set. (a) MonoDETR MonoDETR. (b) MonoDGP MonoDGP. (c) Mono3DV (Ours). For each image set, the top row presents the camera-view visualization, while the bottom row offers the corresponding bird’s-eye view. Ground-truth bounding boxes are rendered in green, and predictions are shown in order: red, yellow, blue. We also circle some objects to highlight the difference between other state of the art and our method.