CaRaFFusion: Improving 2D Semantic Segmentation with Camera-Radar Point Cloud Fusion and Zero-Shot Image Inpainting

TL;DR

This work tackles the challenge of robust 2D semantic segmentation under adverse weather by fusing camera and radar data through a novel three-stage pipeline. It combines a cross-attention-based camera-radar fusion for initial masks, MobileSAM-driven pseudo-mask generation with a Noise Reduction Unit for denoising, and diffusion-based inpainting to recover occluded details, followed by dual SegFormer encoders for final predictions. The authors demonstrate improvements over camera-only and existing fusion baselines on the Waterscenes dataset, particularly in hard-weather scenarios, and provide extensive ablations to justify design choices. The approach advances the practical robustness of autonomous perception by leveraging complementary sensing modalities and generative inpainting to fill in weather-induced gaps.

Abstract

Segmenting objects in an environment is a crucial task for autonomous driving and robotics, as it enables a better understanding of the surroundings of each agent. Although camera sensors provide rich visual details, they are vulnerable to adverse weather conditions. In contrast, radar sensors remain robust under such conditions, but often produce sparse and noisy data. Therefore, a promising approach is to fuse information from both sensors. In this work, we propose a novel framework to enhance camera-only baselines by integrating a diffusion model into a camera-radar fusion architecture. We leverage radar point features to create pseudo-masks using the Segment-Anything model, treating the projected radar points as point prompts. Additionally, we propose a noise reduction unit to denoise these pseudo-masks, which are further used to generate inpainted images that complete the missing information in the original images. Our method improves the camera-only segmentation baseline by 2.63% in mIoU and enhances our camera-radar fusion architecture by 1.48% in mIoU on the Waterscenes dataset. This demonstrates the effectiveness of our approach for semantic segmentation using camera-radar fusion under adverse weather conditions.

Figures (3)

• Figure 1: Our method can segment the objects in very adverse conditions where other methods fail.
• Figure 2: Model Architecture: Our three-stage framework combines radar and camera data for robust segmentation, especially in challenging conditions. First, radar and image features are extracted and fused through cross-attention to produce an initial segmentation mask. To improve resilience, MobileSAM generates an additional mask using radar points as prompts, and the Noise Reduction Unit (NRU) refines this by filtering radar-related noise. Finally, dual Segformer Encoders further enhance the refined mask, producing a high-quality segmentation output.
• Figure 3: Qualitative results show that our inpainting technique is likely addressing missing or occluded regions in the data. It helps to fill in parts of the objects or scenes that might otherwise go undetected, thereby boosting the model’s ability to achieve higher segmentation accuracy. Columns (a), (b), and (c) visualize the original image, the inpainted image, and the Ground Truth (GT) mask of the given image. Columns (d), (e), and (f) illustrate the segmentation from the image-only baseline, the fusion model from the first stage, and the final model from the third stage, respectively.