Instance-Guided Radar Depth Estimation for 3D Object Detection

TL;DR

This work tackles depth ambiguity in monocular BEV 3D object detection by introducing InstaRadar, an instance segmentation-guided Radar expansion that densifies Radar data within object masks, and by integrating a Radar-guided depth estimator (RCDPT) into BEVDepth to provide explicit depth supervision. The combined approach yields consistent improvements over a BEVDepth baseline, improving depth quality and 3D detection accuracy, though it remains outperformed by dedicated Radar–camera fusion models that fuse Radar as a separate BEV feature stream. Key contributions include object-aware Radar expansion via OneFormer masks and the replacement of BEVDepth's depth module with a LiDAR-supervised, Radar-guided depth model, enabling better intermediate depth features and BEV representations. The results on nuScenes demonstrate the potential of Radar-guided depth supervision and instance-aware Radar preprocessing to enhance monocular 3D perception and guide future multimodal fusion enhancements.

Abstract

Accurate depth estimation is fundamental to 3D perception in autonomous driving, supporting tasks such as detection, tracking, and motion planning. However, monocular camera-based 3D detection suffers from depth ambiguity and reduced robustness under challenging conditions. Radar provides complementary advantages such as resilience to poor lighting and adverse weather, but its sparsity and low resolution limit its direct use in detection frameworks. This motivates the need for effective Radar-camera fusion with improved preprocessing and depth estimation strategies. We propose an end-to-end framework that enhances monocular 3D object detection through two key components. First, we introduce InstaRadar, an instance segmentation-guided expansion method that leverages pre-trained segmentation masks to enhance Radar density and semantic alignment, producing a more structured representation. InstaRadar achieves state-of-the-art results in Radar-guided depth estimation, showing its effectiveness in generating high-quality depth features. Second, we integrate the pre-trained RCDPT into the BEVDepth framework as a replacement for its depth module. With InstaRadar-enhanced inputs, the RCDPT integration consistently improves 3D detection performance. Overall, these components yield steady gains over the baseline BEVDepth model, demonstrating the effectiveness of InstaRadar and the advantage of explicit depth supervision in 3D object detection. Although the framework lags behind Radar-camera fusion models that directly extract BEV features, since Radar serves only as guidance rather than an independent feature stream, this limitation highlights potential for improvement. Future work will extend InstaRadar to point cloud-like representations and integrate a dedicated Radar branch with temporal cues for enhanced BEV fusion.

Figures (3)

• Figure 1: Visualization of raw Radar, instance segmentation, instance masks, and the proposed InstaRadar on the nuScenes dataset nuScenes. The top-left shows raw Radar points projected from 5 frames onto the reference image, while the top-right displays instance segmentation results from OneFormer OneFormer. The middle row highlights close-up views of the Radar depth points and corresponding instance masks from the boxes in the top-left image. The bottom image presents the final InstaRadar result, where Radar points are expanded within instance masks to improve resolution and spatial coverage. Radar points are enlarged for better visibility, and the color bar indicates distances ranging from 0 to 80 meters.
• Figure 2: Examples of the proposed Instance Segmentation-guided Radar expansion on the nuScenes dataset nuScenes. The left column shows raw Radar points from 5 frames, and the right column shows InstaRadar expanded within instance masks. Rows from top to bottom correspond to day, night, and rain scenes. Radar points are enlarged for clarity, with distances indicated by the 0–80 m color bar.
• Figure 3: Our proposed framework takes multi-view images and InstaRadar as input. The image backbone extracts image features, while a pre-trained RCDPT takes both image and InstaRadar as input to predict depth features under LiDAR supervision. These image and depth features are concatenated and passed through voxel pooling to generate a BEV feature map. Finally, the BEV feature map is passed through a BEV encoder and a detection head to output the final 3D detection results.