Systematic Evaluation of Depth Backbones and Semantic Cues for Monocular Pseudo-LiDAR 3D Detection

TL;DR

This work systematically evaluates how depth backbone choice and simple semantic cues impact monocular Pseudo-LiDAR 3D detection on KITTI. By keeping the detector and training protocol fixed, it isolates the influence of depth fidelity and auxiliary features, finding that geometry dominates: NeWCRFs outperforms Depth Anything V2 modestly, especially at strict IoU thresholds, while grayscale intensity yields the strongest but still modest gains among semantic cues. Semantics such as mask confidences or mask-guided sampling provide limited 3D improvements and can even harm localization by removing contextual geometry. The study concludes that advancing monocular 3D detection via pseudo-LiDAR hinges on producing geometrically faithful reconstructions and aligning detector architectures with the input statistics, rather than relying on semantic injections alone.

Abstract

Monocular 3D object detection offers a low-cost alternative to LiDAR, yet remains less accurate due to the difficulty of estimating metric depth from a single image. We systematically evaluate how depth backbones and feature engineering affect a monocular Pseudo-LiDAR pipeline on the KITTI validation split. Specifically, we compare NeWCRFs (supervised metric depth) against Depth Anything V2 Metric-Outdoor (Base) under an identical pseudo-LiDAR generation and PointRCNN detection protocol. NeWCRFs yields stronger downstream 3D detection, achieving 10.50\% AP$_{3D}$ at IoU$=0.7$ on the Moderate split using grayscale intensity (Exp~2). We further test point-cloud augmentations using appearance cues (grayscale intensity) and semantic cues (instance segmentation confidence). Contrary to the expectation that semantics would substantially close the gap, these features provide only marginal gains, and mask-based sampling can degrade performance by removing contextual geometry. Finally, we report a depth-accuracy-versus-distance diagnostic using ground-truth 2D boxes (including Ped/Cyc), highlighting that coarse depth correctness does not fully predict strict 3D IoU. Overall, under an off-the-shelf LiDAR detector, depth-backbone choice and geometric fidelity dominate performance, outweighing secondary feature injection.

Figures (4)

• Figure 1: Overall monocular pseudo-LiDAR pipeline.
• Figure 2: Qualitative depth comparison (same KITTI frame): Depth Anything V2 Metric-Outdoor shows banding/over-smoothing relative to NeWCRFs.
• Figure 3: Qualitative BEV comparison: Exp 2 vs Exp 5 (NeWCRFs). Blue points show the pseudo-LiDAR point cloud. Red boxes are KITTI ground-truth 3D boxes and green boxes are PointRCNN predictions (BEV). Exp 5 uses mask-guided point selection that preserves dense car points but strongly reduces background/road context compared to Exp 2 (full-scene cloud), which can destabilize localization and increase errors at high IoU.
• Figure 4: Qualitative BEV comparison (Exp 2 setting): NeWCRFs vs. Depth Anything V2 Metric-Outdoor under the same PointRCNN protocol.