V2X-DGW: Domain Generalization for Multi-agent Perception under Adverse Weather Conditions

TL;DR

The paper tackles the problem of domain generalization for LiDAR-based V2X multi-agent perception under unseen adverse weather. It introduces V2X-DGW, a framework built on Adaptive Weather Augmentation (AWA) to simulate weather-induced perception changes, plus Trust-region Weather-invariant Alignment (TWA) and Agent-aware Contrastive Alignment (ACA) to learn robust representations from clean-weather data. Two synthetic benchmarks, OPV2V-w and V2XSet-w, are created to evaluate generalization across Fog, Rain, and Snow. Empirical results show substantial improvements in unseen weather conditions, validating the proposed DG approach and its components, with code released for reproducibility.

Abstract

Current LiDAR-based Vehicle-to-Everything (V2X) multi-agent perception systems have shown the significant success on 3D object detection. While these models perform well in the trained clean weather, they struggle in unseen adverse weather conditions with the domain gap. In this paper, we propose a Domain Generalization based approach, named \textit{V2X-DGW}, for LiDAR-based 3D object detection on multi-agent perception system under adverse weather conditions. Our research aims to not only maintain favorable multi-agent performance in the clean weather but also promote the performance in the unseen adverse weather conditions by learning only on the clean weather data. To realize the Domain Generalization, we first introduce the Adaptive Weather Augmentation (AWA) to mimic the unseen adverse weather conditions, and then propose two alignments for generalizable representation learning: Trust-region Weather-invariant Alignment (TWA) and Agent-aware Contrastive Alignment (ACA). To evaluate this research, we add Fog, Rain, Snow conditions on two publicized multi-agent datasets based on physics-based models, resulting in two new datasets: OPV2V-w and V2XSet-w. Extensive experiments demonstrate that our V2X-DGW achieved significant improvements in the unseen adverse weathers. The code is available at https://github.com/Baolu1998/V2X-DGW.

Figures (6)

• Figure 1: Multi-agent perception system under adverse weather. Three CAVs' LiDAR point clouds are highlighted in Black, Orange and Green colors. We can discover the reduced perception range (shrinked wave circles) and degradation (highlighted by the blue arrows) for the point clouds under adverse weather.
• Figure 2: Overview framework of the proposed V2X-DGW for domain generalization of multi-agent perception system in adverse weather. During the training stage, every agent has source flow (clean weather) and augmented flow (mimicked adverse weather). Pillar features are obtained from the Pillar Feature Net (PFN) lang2019pointpillars, which is shown in bird view here.
• Figure 3: Illustration of simulation before and after Adaptive Weather Augmentation.
• Figure 4: Illustration of TWA and ACA. (a) Pillar-based weather-invariant alignment within Trust-region, (b) Agent-aware contrastive alignment in agent-level and group-level. Best view in color.
• Figure 5: 3D object detection visualization example under adverse weather.Green and red 3D bounding boxes represent the ground truth and prediction respectively. The detection errors are highlighted using black arrows. OPV2V-w fog weather is used as example here.