Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection

Yiheng Li, Yang Yang, Zhen Lei

TL;DR

This work tackles the challenge of sparse and noisy radar data in radar-camera 3D object detection. It introduces RCTrans, a query-based transformer framework with a Radar Dense Encoder to densify radar BEV grids and a Pruning Sequential Decoder that fuses multi-modality tokens in a stepwise manner while updating object queries and position embeddings after each layer. A pruning training strategy reduces inference time while maintaining discriminative queries, and an end-to-end training objective combines classification and regression losses with Hungarian assignment. On nuScenes, RCTrans achieves state-of-the-art radar-camera performance, e.g., 64.7% NDS and 57.8% mAP on the test set, and demonstrates strong robustness to sensor drop scenarios, highlighting its practical impact for reliable, low-cost autonomous perception.

Abstract

In radar-camera 3D object detection, the radar point clouds are sparse and noisy, which causes difficulties in fusing camera and radar modalities. To solve this, we introduce a novel query-based detection method named Radar-Camera Transformer (RCTrans). Specifically, we first design a Radar Dense Encoder to enrich the sparse valid radar tokens, and then concatenate them with the image tokens. By doing this, we can fully explore the 3D information of each interest region and reduce the interference of empty tokens during the fusing stage. We then design a Pruning Sequential Decoder to predict 3D boxes based on the obtained tokens and random initialized queries. To alleviate the effect of elevation ambiguity in radar point clouds, we gradually locate the position of the object via a sequential fusion structure. It helps to get more precise and flexible correspondences between tokens and queries. A pruning training strategy is adopted in the decoder, which can save much time during inference and inhibit queries from losing their distinctiveness. Extensive experiments on the large-scale nuScenes dataset prove the superiority of our method, and we also achieve new state-of-the-art radar-camera 3D detection results. Our implementation is available at https://github.com/liyih/RCTrans.

RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection

TL;DR

This work tackles the challenge of sparse and noisy radar data in radar-camera 3D object detection. It introduces RCTrans, a query-based transformer framework with a Radar Dense Encoder to densify radar BEV grids and a Pruning Sequential Decoder that fuses multi-modality tokens in a stepwise manner while updating object queries and position embeddings after each layer. A pruning training strategy reduces inference time while maintaining discriminative queries, and an end-to-end training objective combines classification and regression losses with Hungarian assignment. On nuScenes, RCTrans achieves state-of-the-art radar-camera performance, e.g., 64.7% NDS and 57.8% mAP on the test set, and demonstrates strong robustness to sensor drop scenarios, highlighting its practical impact for reliable, low-cost autonomous perception.

Abstract

In radar-camera 3D object detection, the radar point clouds are sparse and noisy, which causes difficulties in fusing camera and radar modalities. To solve this, we introduce a novel query-based detection method named Radar-Camera Transformer (RCTrans). Specifically, we first design a Radar Dense Encoder to enrich the sparse valid radar tokens, and then concatenate them with the image tokens. By doing this, we can fully explore the 3D information of each interest region and reduce the interference of empty tokens during the fusing stage. We then design a Pruning Sequential Decoder to predict 3D boxes based on the obtained tokens and random initialized queries. To alleviate the effect of elevation ambiguity in radar point clouds, we gradually locate the position of the object via a sequential fusion structure. It helps to get more precise and flexible correspondences between tokens and queries. A pruning training strategy is adopted in the decoder, which can save much time during inference and inhibit queries from losing their distinctiveness. Extensive experiments on the large-scale nuScenes dataset prove the superiority of our method, and we also achieve new state-of-the-art radar-camera 3D detection results. Our implementation is available at https://github.com/liyih/RCTrans.

Paper Structure

This paper contains 17 sections, 7 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. BEV-Based 3D Object Detection
  4. Query-Based 3D Object Detection
  5. Radar-Camera 3D Object Detection
  6. Method
  7. Token Generator
  8. Pruning Sequential Decoder
  9. Training Head and Task Loss
  10. Experiments
  11. Datasets and metrics
  12. Implementation Details
  13. Performance Comparison
  14. Ablation Study
  15. Robustness Analysis
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Comparison between RCTrans and other camera or radar-camera 3D detection methods. The size of the circle represents the time consumption of inference. All experiments are conducted on nuScenes val set and the speed is measured on a single NVIDIA RTX3090 GPU.
  • Figure 2: The overall architecture of Radar-Camera Transformer. RCTrans can be divided into two key components: (1) Token generator which extracts multi-modality tokens and sums them with corresponding position embedding and (2) Pruning Sequential Decoder which updates the randomly initialized queries and queries position in each layer and makes more precise fusion in a step-by-step way. The final predicted results are the outputs of the last layer after pruning.
  • Figure 3: The overall architecture of Radar Dense Encoder. RDE uses a downsample-then-upsample architecture. Self-attention is used to fill the empty grids in the minimal resolution. Skip connection is used in the upsampling process.
  • Figure 4: Qualitative results of RCTrans on nuScenes val set with different backbones. Due to space constraints, we only show the corresponding front left and back left images where objects are mainly distributed in this scene.