WeatherGen: A Unified Diverse Weather Generator for LiDAR Point Clouds via Spider Mamba Diffusion

TL;DR

WeatherGen tackles the lack of high-fidelity, diverse adverse-weather LiDAR data by introducing a unified diffusion-based framework. It combines a map-based data producer (MDP) to bootstrap realistic weather patterns, a Spider Mamba Generator (SMG) that preserves LiDAR geometry by operating along beam circles and central rays, a Latent Feature Aligner (LFA) to fuse real-world knowledge, and a Contrastive Learning-based Controller (CLC) to ground weather control signals in compact semantic knowledge via language supervision. The model is trained with a diffusion objective and auxiliary losses, enabling both unconditional and weather-conditioned LiDAR generation that improves downstream 3D perception. A mini-weather dataset is constructed to evaluate robustness under adverse weather, demonstrating practical impact for safe perception in autonomous systems.

Abstract

3D scene perception demands a large amount of adverse-weather LiDAR data, yet the cost of LiDAR data collection presents a significant scaling-up challenge. To this end, a series of LiDAR simulators have been proposed. Yet, they can only simulate a single adverse weather with a single physical model, and the fidelity of the generated data is quite limited. This paper presents WeatherGen, the first unified diverse-weather LiDAR data diffusion generation framework, significantly improving fidelity. Specifically, we first design a map-based data producer, which can provide a vast amount of high-quality diverse-weather data for training purposes. Then, we utilize the diffusion-denoising paradigm to construct a diffusion model. Among them, we propose a spider mamba generator to restore the disturbed diverse weather data gradually. The spider mamba models the feature interactions by scanning the LiDAR beam circle or central ray, excellently maintaining the physical structure of the LiDAR data. Subsequently, following the generator to transfer real-world knowledge, we design a latent feature aligner. Afterward, we devise a contrastive learning-based controller, which equips weather control signals with compact semantic knowledge through language supervision, guiding the diffusion model to generate more discriminative data. Extensive evaluations demonstrate the high generation quality of WeatherGen. Through WeatherGen, we construct the mini-weather dataset, promoting the performance of the downstream task under adverse weather conditions. Code is available: https://github.com/wuyang98/weathergen

Figures (8)

• Figure 1: (a) Previous simulation-based methods hahner2022lidarhahner2021fogkilic2021lidarcharron2018noising can only provide a single non-learnable simulator for a single weather condition. Due to the complexity of optical propagation, previous simulated data all suffer from insufficient fidelity issues. (b) Our method is the first unified generative framework. Through learning, the generated data has higher fidelity and can be more conducive to promoting downstream tasks under diverse weather.
• Figure 2: The pipeline of WeatherGen. It has three core components. An MDP to produce high-quality training data that is closer to real-world data; An SMG models denoising features in a way that conforms to the LiDAR imaging process. SMG is further followed by an LFA to transfer real-world data knowledge into the generator. A CLC, composed of a weather encoder and a CLIP text encoder, is used to generate control signals with more compact and discriminative knowledge through contrastive learning.
• Figure 3: (a) Spider mamba scans model features along the LiDAR beams-circle and central ray, excellently maintaining the physical structures of the LiDAR; (b) Convolution can only model features locally; (c) Self-attention disorderly connecting all points, disrupting the physical properties of LiDAR data.
• Figure 4: Visual comparisons with competitive generation methods on KITTI-360 liao2022kitti dataset. The results prove that WeatherGen has a better ability to generate clear object contours and small objects.
• Figure 5: Visual comparisons of real-world diverse weather LiDAR data and generated results on Seeing Through Fog bijelic2020seeing dataset.