SFGFusion: Surface Fitting Guided 3D Object Detection with 4D Radar and Camera Fusion

TL;DR

SFGFusion tackles the challenge of 3D object detection for autonomous driving when using camera data together with sparse 4D imaging radar. It introduces a surface fitting module that jointly fuses image semantics and radar geometry to predict dense object surface depth, which guides a PV-to-BEV view transformation and enables a dense surface pseudo-point cloud for enhanced BEV fusion. The architecture comprises an image branch, a radar branch, and a surface pseudo-point branch, all feeding a multi-layer BEV fusion and an anchor-based detection head; the surface fitting model is supervised by a loss $L_{sf}$ and a quadratic surface equation $d_s = a_c u_s^{2}+b_c v_s^{2}+c_c u_s v_s+d_c u_s+e_c v_s+f_c$. Empirical results on TJ4DRadSet and VoD show consistent improvements over state-of-the-art fusion methods, especially in challenging long-range and occluded scenarios, underscoring the practical value of explicit surface-guided depth for robust multi-modal fusion.

Abstract

3D object detection is essential for autonomous driving. As an emerging sensor, 4D imaging radar offers advantages as low cost, long-range detection, and accurate velocity measurement, making it highly suitable for object detection. However, its sparse point clouds and low resolution limit object geometric representation and hinder multi-modal fusion. In this study, we introduce SFGFusion, a novel camera-4D imaging radar detection network guided by surface fitting. By estimating quadratic surface parameters of objects from image and radar data, the explicit surface fitting model enhances spatial representation and cross-modal interaction, enabling more reliable prediction of fine-grained dense depth. The predicted depth serves two purposes: 1) in an image branch to guide the transformation of image features from perspective view (PV) to a unified bird's-eye view (BEV) for multi-modal fusion, improving spatial mapping accuracy; and 2) in a surface pseudo-point branch to generate dense pseudo-point cloud, mitigating the radar point sparsity. The original radar point cloud is also encoded in a separate radar branch. These two point cloud branches adopt a pillar-based method and subsequently transform the features into the BEV space. Finally, a standard 2D backbone and detection head are used to predict object labels and bounding boxes from BEV features. Experimental results show that SFGFusion effectively fuses camera and 4D radar features, achieving superior performance on the TJ4DRadSet and view-of-delft (VoD) object detection benchmarks.

Figures (9)

• Figure 1: Projection of 4D radar and LiDAR point clouds onto image. The three modalities of the same object are highlighted. The radar point cloud is sparse and irregular, while the LiDAR point cloud is denser and structured.
• Figure 2: The overall architecture of the SFGFusion. In the surface fitting model, we integrate image semantic information with point cloud spatial data to predict the fine-grained dense depth of object surfaces. In the image branch, the surface fitting dense depth and the 4D radar point cloud guide the view transformation of the image feature, enhancing depth estimation accuracy. The surface pseudo-point branch converts the depth data from surface fitting into dense pseudo-point cloud and performs feature extraction to compensate for the sparse nature of 4D radar point cloud. In the radar branch, both point cloud spatial data and radar-specific velocity and RCS features are exploited to increase feature diversity. Finally, the fused BEV features from all branches are processed through a multi-scale network, and the detection head is employed to generate 3D bounding boxes, object classifications, and confidence scores.
• Figure 3: The surface fitting model in SFGFusion. During information fusion, the depth-enhanced mask is generated by averaging the radar point depths within each 2D instance mask and then concatenated with the original image feature from the image branch, facilitating interaction between the image and radar modalities. For parameter fitting, the 2D bounding box generated from the instance mask is used to guide a lightweight MLP network in estimating the coefficients of the surface fitting equation, which is then used to calculate the depth of each pixel within the instance. The depth estimation is refined by supervising the surface fitting using the radar point cloud data, leading to fine-grained and accurate depth prediction for each instance.
• Figure 4: Illustration of image feature view transformation. The process comprises two steps: Lift and Splat. The combined feature $F_v$ integrates information from the image, point cloud, and surface fitting model.
• Figure 5: Surface fitting pseudo-point visualization. The surface fitting model generates denser 3D spatial information than LiDAR by fusing camera and radar data. Each row shows the RGB image, LiDAR point cloud, camera instance mask, 4D radar point cloud, and surface fitting pseudo-point cloud.