Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editing

TL;DR

An automatic data generation pipeline Gen4Seg is constructed to stress-test semantic segmentation models by generating various challenging samples with different attribute changes, and suggests the potential of generative models as effective tools for automatically analyzing segmentation models.

Abstract

Semantic segmentation takes pivotal roles in various applications such as autonomous driving and medical image analysis. When deploying segmentation models in practice, it is critical to test their behaviors in varied and complex scenes in advance. In this paper, we construct an automatic data generation pipeline Gen4Seg to stress-test semantic segmentation models by generating various challenging samples with different attribute changes. Beyond previous evaluation paradigms focusing solely on global weather and style transfer, we investigate variations in both appearance and geometry attributes at the object and image level. These include object color, material, size, position, as well as image-level variations such as weather and style. To achieve this, we propose to edit visual attributes of existing real images with precise control of structural information, empowered by diffusion models. In this way, the existing segmentation labels can be reused for the edited images, which greatly reduces the labor costs. Using our pipeline, we construct two new benchmarks, Pascal-EA and COCO-EA. We benchmark a wide variety of semantic segmentation models, spanning from closed-set models to open-vocabulary large models. We have several key findings: 1) advanced open-vocabulary models do not exhibit greater robustness compared to closed-set methods under geometric variations; 2) data augmentation techniques, such as CutOut and CutMix, are limited in enhancing robustness against appearance variations; 3) our pipeline can also be employed as a data augmentation tool and improve both in-distribution and out-of-distribution performances. Our work suggests the potential of generative models as effective tools for automatically analyzing segmentation models, and we hope our findings will assist practitioners and researchers in developing more robust and reliable segmentation models.

Figures (11)

• Figure 1: Gen4Seg expands existing images with various appearance and geometry attributes changes for evaluating the robustness of semantic segmentation models.
• Figure 2: Our Gen4Seg generates appearance and geometry attributes variations to existing images, with collaboration of diffusion models, VLMs and LLMs. For appearance editing, we employ a VLM to edit text description of target object, and use proposed Mask-Guidance Diffusion model with ControlNet and designed energy function. For geometry editing, we first manipulate (move, rescale) the intent edit object, and use a VLM and ControlNet to recover background content by a diffusion-based inpainting model. At last, we apply established metrics to discard low-quality images and further employ a surrogate segmentation model to eliminate harmful regions from evaluation.
• Figure 3: Illustration of our Mask-Guidance Diffusion. We propose the Mask-Guidance Energy function to optimize cross-attention maps region of target edit token.
• Figure 4: Illustration of our Geometry Attributes Editing pipeline. We first apply a rigid transform, then employ a diffusion inpainting model and ControlNet controlnet.
• Figure 5: Examples of our generated datasets Pascal-EA and COCO-EA with diverse appearance and geometry variations.