ClimaOoD: Improving Anomaly Segmentation via Physically Realistic Synthetic Data

TL;DR

ClimaOoD presents a physics-informed synthetic data framework (ClimaDrive) that generates semantically coherent, weather-diverse driving scenes with perspective-aware OoD object placement. Built into a large-scale benchmark (ClimaOoD), it enables robust evaluation of anomaly segmentation under open-world conditions. Across four state-of-the-art methods, training with ClimaOoD yields improvements in AUROC and AP, though robustness gaps remain under adverse weather, underscoring the ongoing value of realistic synthetic data. Overall, the work demonstrates that physically grounded, semantically guided synthesis can substantially enhance anomaly segmentation in autonomous driving while highlighting areas for further improvement in challenging environments.

Abstract

Anomaly segmentation seeks to detect and localize unknown or out-of-distribution (OoD) objects that fall outside predefined semantic classes a capability essential for safe autonomous driving. However, the scarcity and limited diversity of anomaly data severely constrain model generalization in open-world environments. Existing approaches mitigate this issue through synthetic data generation, either by copy-pasting external objects into driving scenes or by leveraging text-to-image diffusion models to inpaint anomalous regions. While these methods improve anomaly diversity, they often lack contextual coherence and physical realism, resulting in domain gaps between synthetic and real data. In this paper, we present ClimaDrive, a semantics-guided image-to-image framework for synthesizing semantically coherent, weather-diverse, and physically plausible OoD driving data. ClimaDrive unifies structure-guided multi-weather generation with prompt-driven anomaly inpainting, enabling the creation of visually realistic training data. Based on this framework, we construct ClimaOoD, a large-scale benchmark spanning six representative driving scenarios under both clear and adverse weather conditions. Extensive experiments on four state-of-the-art methods show that training with ClimaOoD leads to robust improvements in anomaly segmentation. Across all methods, AUROC, AP, and FPR95 show notable gains, with FPR95 dropping from 3.97 to 3.52 for RbA on Fishyscapes LAF. These results demonstrate that ClimaOoD enhances model robustness, offering valuable training data for better generalization in open-world anomaly detection.

Figures (10)

• Figure 1: Visual comparison of dataset limitations and ClimaOoD. (a) Existing datasets suffer from limited weather and scene diversity. (b) Recent data synthesis leads to contextual inconsistencies and physically unrealistic object placement. (c) ClimaDrive generates diverse weather scenes and places anomalies with realistic spatial arrangement, using semantic maps and text prompts for better OoD object placement.
• Figure 2: ClimaDrive Training Pipeline. (a) Multi-Scene Weather Generator: A structure-guided diffusion model creates semantically consistent driving scenes under diverse weather conditions using BDD100K inputs and scene-aware prompts. (b) AnomPlacer: A trainable module that predicts anomaly locations and synthesizes OoD objects via text-conditioned diffusion inpainting (e.g., “dog”).
• Figure 3: Anomaly pixel statistics in ClimaOoD: (a) Distribution of anomaly pixel fraction per image, indicating scale diversity; (b) Spatial distribution heatmap of anomaly pixels, showing that anomalies frequently appear near drivable regions.
• Figure 4: Visual examples from our ClimaOoD dataset showing synthetic anomaly scenes under diverse weather conditions. Each scene exhibits realistic illumination, texture variations, and weather-specific visual cues. Anomalous objects are seamlessly integrated into the environment, highlighting the controllable diversity and fidelity of our generation pipeline.
• Figure 5: Visualization of Ablation Study for OoD Detection: RPL Outputs from Diverse Training Datasets (Left to Right: Image, Original RPL Output, Clear&Street-trained, ClimaOoD-trained, Ground Truth(GT))