Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception

TL;DR

Doracamom addresses robust omnidirectional perception by fusing multi-view cameras with 4D radar to jointly perform 3D object detection and semantic occupancy prediction. The approach introduces three innovations: a Coarse Voxel Queries Generator that injects geometric radar priors and image semantics, a Dual-Branch Temporal Encoder that fuses temporal cues in BEV and voxel spaces, and a Cross-Modal BEV-Voxel Fusion module with auxiliary supervision. Empirical results on OmniHD-Scenes, VoD, and TJ4DRadSet establish state-of-the-art performance in both tasks, with ablations validating the contributions of each module. The method demonstrates robustness under adverse conditions and offers a practical path toward all-weather, multi-task autonomous-perception systems, with public release planned for code and models.

Abstract

3D object detection and occupancy prediction are critical tasks in autonomous driving, attracting significant attention. Despite the potential of recent vision-based methods, they encounter challenges under adverse conditions. Thus, integrating cameras with next-generation 4D imaging radar to achieve unified multi-task perception is highly significant, though research in this domain remains limited. In this paper, we propose Doracamom, the first framework that fuses multi-view cameras and 4D radar for joint 3D object detection and semantic occupancy prediction, enabling comprehensive environmental perception. Specifically, we introduce a novel Coarse Voxel Queries Generator that integrates geometric priors from 4D radar with semantic features from images to initialize voxel queries, establishing a robust foundation for subsequent Transformer-based refinement. To leverage temporal information, we design a Dual-Branch Temporal Encoder that processes multi-modal temporal features in parallel across BEV and voxel spaces, enabling comprehensive spatio-temporal representation learning. Furthermore, we propose a Cross-Modal BEV-Voxel Fusion module that adaptively fuses complementary features through attention mechanisms while employing auxiliary tasks to enhance feature quality. Extensive experiments on the OmniHD-Scenes, View-of-Delft (VoD), and TJ4DRadSet datasets demonstrate that Doracamom achieves state-of-the-art performance in both tasks, establishing new benchmarks for multi-modal 3D perception. Code and models will be publicly available.

Figures (8)

• Figure 1: Performance comparison of different methods on 3D perception benchmarks. By effectively fusing camera and 4D radar inputs, Doracamom (marked with $\star$) consistently achieves superior performance across multiple metrics, outperforming existing camera-only and camera-radar fusion approaches in both 3D object detection and occupancy prediction tasks.
• Figure 2: The overall framework of Doracamom. Initially, the camera encoder and 4D radar encoder are utilized to extract multi-camera features and radar BEV features, respectively. Subsequently, the coarse voxel query generator employs geometric priors derived from 4D radar features and semantic priors from image features to generate coarse voxel queries. These queries are then refined by a stacked voxel queries encoder to obtain local fine-grained voxel features. A dual-branch temporal encoder is employed to fuse historical BEV and voxel features with current frame features, leveraging temporal clues. The output radar BEV and image voxel features are fed into the cross-modal BEV-voxel fusion module for adaptive fusion, resulting in the final BEV and voxel representations. Finally, the obtained representations are used to predict 3D detection and semantic occupancy for the current scene.
• Figure 3: Illustration of the proposed Coarse Voxel Queries Generator. It combines radar BEV features and multi-view image features to initialize voxel queries with both geometric and semantic priors.
• Figure 4: Illustration of the proposed Dual-branch Temporal Encoder. To eliminate misalignment caused by ego motion, ego poses are used to warp the 2D and 3D reference points and sample features to the current frame. Historical features are then merged using ResNet2D/3D blocks to reduce cross-frame feature interaction and enhance efficiency. To mitigate the impact of moving objects, 2D and 3D deformable attention mechanisms are employed to adaptively fuse features from the current and historical frames.
• Figure 5: Illustration of the proposed Cross-Modal BEV-Voxel Fusion module. Complementary information from both modalities is adaptively fused to generate BEV and voxel features for downstream task decoding. Additionally, two auxiliary tasks which predict the occupied/non-occupied binary 3D occupancy probability and foreground/background BEV segmentation mask, are incorporated to enhance the quality of the generated 3D occupancy and BEV features.