WeatherCity: Urban Scene Reconstruction with Controllable Multi-Weather Transformation

TL;DR

The proposed WeatherCity introduces a novel weather Gaussian representation based on shared scene features and dedicated weather-specific decoders that simulates dynamic weather effects through particles and motion patterns and designs a physics-driven model that simulates dynamic weather effects through particles and motion patterns.

Abstract

Editable high-fidelity 4D scenes are crucial for autonomous driving, as they can be applied to end-to-end training and closed-loop simulation. However, existing reconstruction methods are primarily limited to replicating observed scenes and lack the capability for diverse weather simulation. While image-level weather editing methods tend to introduce scene artifacts and offer poor controllability over the weather effects. To address these limitations, we propose WeatherCity, a novel framework for 4D urban scene reconstruction and weather editing. Specifically, we leverage a text-guided image editing model to achieve flexible editing of image weather backgrounds. To tackle the challenge of multi-weather modeling, we introduce a novel weather Gaussian representation based on shared scene features and dedicated weather-specific decoders. This representation is further enhanced with a content consistency optimization, ensuring coherent modeling across different weather conditions. Additionally, we design a physics-driven model that simulates dynamic weather effects through particles and motion patterns. Extensive experiments on multiple datasets and various scenes demonstrate that WeatherCity achieves flexible controllability, high fidelity, and temporal consistency in 4D reconstruction and weather editing. Our framework not only enables fine-grained control over weather conditions (e.g., light rain and heavy snow) but also supports object-level manipulation within the scene.

Figures (17)

• Figure 1: We present WeatherCity, a novel framework for dynamic urban scene reconstruction and controllable weather editing. Given a sequence of raw images, WeatherCity seamlessly integrates 4D reconstruction with flexible weather manipulation, producing highly consistent, photorealistic, and versatile multi-weather rendering results.
• Figure 2: Overview of WeatherCity. Our framework comprises four main modules. First, the image editing module employs a text-guided video editing foundation model to adapt image weather background. Second, the scene representation module introduces a weather-aware Gaussian representation based on shared features and multi-weather decoders, which disentangles geometric-textural attributes from weather-specific appearances, thereby ensuring structural consistency across varying meteorological conditions. Subsequently, we construct RGB and content losses for consistency optimization. Finally, a physics-driven dynamic weather simulation mechanism is designed to achieve flexible and controllable editing of diverse dynamic weather effects.
• Figure 3: Qualitative results of multi-weather editing on the Waymo Open Dataset. Our method produces realistic and consistent weather editing effects while supporting dynamic weather simulation, whereas all baselines exhibit significant scene distortion.
• Figure 4: Qualitative results of multi-weather editing on the nuScenes dataset. Our method produces realistic and consistent weather editing effects while supporting dynamic weather simulation, whereas all baselines exhibit significant scene distortion.
• Figure 5: Visualization of object editing results. The text prompt is "Remove all vehicles except the red and white ones in the center and change the weather to snowy".