Open-set Anomaly Segmentation in Complex Scenarios

TL;DR

This work tackles the instability of semantic segmentation in open-set conditions by introducing the ComsAmy benchmark, which enforces performance under diverse adverse weather and open-road scenarios. It proposes DiffEEL, a combination of a diffusion-based anomaly data synthesizer and an energy-entropy learning strategy, to improve anomaly segmentation robustness. Empirical results show DiffEEL yields meaningful gains on ComsAmy and generalizes to public benchmarks, highlighting the approach's practical potential for safer autonomous systems. The work emphasizes the importance of realistic anomalous data and robust scoring in open-world settings, suggesting broad applicability as a plug-and-play enhancement for existing anomaly segmentation methods.

Abstract

Precise segmentation of out-of-distribution (OoD) objects, herein referred to as anomalies, is crucial for the reliable deployment of semantic segmentation models in open-set, safety-critical applications, such as autonomous driving. Current anomalous segmentation benchmarks predominantly focus on favorable weather conditions, resulting in untrustworthy evaluations that overlook the risks posed by diverse meteorological conditions in open-set environments, such as low illumination, dense fog, and heavy rain. To bridge this gap, this paper introduces the ComsAmy, a challenging benchmark specifically designed for open-set anomaly segmentation in complex scenarios. ComsAmy encompasses a wide spectrum of adverse weather conditions, dynamic driving environments, and diverse anomaly types to comprehensively evaluate the model performance in realistic open-world scenarios. Our extensive evaluation of several state-of-the-art anomalous segmentation models reveals that existing methods demonstrate significant deficiencies in such challenging scenarios, highlighting their serious safety risks for real-world deployment. To solve that, we propose a novel energy-entropy learning (EEL) strategy that integrates the complementary information from energy and entropy to bolster the robustness of anomaly segmentation under complex open-world environments. Additionally, a diffusion-based anomalous training data synthesizer is proposed to generate diverse and high-quality anomalous images to enhance the existing copy-paste training data synthesizer. Extensive experimental results on both public and ComsAmy benchmarks demonstrate that our proposed diffusion-based synthesizer with energy and entropy learning (DiffEEL) serves as an effective and generalizable plug-and-play method to enhance existing models, yielding an average improvement of around 4.96% in $\rm{AUPRC}$ and 9.87% in $\rm{FPR}_{95}$.

Figures (8)

• Figure 1: Visualization of anomalous segmentation results by SoTA Mask2Anomaly rai2024mask2anomaly model on existing and our benchmarks. We highlight the anomalies using yellow bounding boxes. The pure black regions denote void pixels, which are excluded from evaluation. The top row presents anomalous images from existing benchmarks, including FS-L&F blum2019fishyscapes, SMIYC-Obstacle chan2021segmentmeifyoucan, RoadAnomaly lis2019detecting, SMIYC-Anomaly chan2021segmentmeifyoucan, and FS-Static blum2019fishyscapes. The bottom row displays those from ours. Objects marked with red color are predicted as anomalies by the Mask2Former model. The comparative analysis reveals that the model exhibits significant limitations under challenging conditions, such as rainy weather and low illumination.
• Figure 2: The fraction and distribution of annotated anomalous pixels within the image over the entire dataset.
• Figure 3: The structure of the proposed diffusion-based anomaly data synthesizer and energy-entropy learning strategy. The object generator can create a large diversity of anomalous objects, and the salient object extractor will extract the precise mask of the anomalous object to enhance the variety of the anomalies. The harmonization blender will blend the anomalous object with a harmonious size, position, and illumination. The EEL strategy influences the training and inference stages, which can improve model generalization and performance in intricate open-world scenarios.
• Figure 4: The diffusion generated anomalous objects and the extracted masks.
• Figure 5: The comparison of anomaly images generated by the "copy-paste" method and ours. The anomaly is marked by the red bounding box.