Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating and Enhancing Segmentation Model Robustness with Metamorphic Testing

Seif Mzoughi, Mohamed Elshafeia, Foutse Khomh

TL;DR

This work introduces SegRMT, a metamorphic testing framework that uses a genetic algorithm to discover sequences of spatial and spectral perturbations under a PSNR constraint to stress test image segmentation models. Evaluated on Cityscapes with DeepLabV3, SegRMT delivers stronger adversaries (mIoU as low as $6.4\%$ at $24$ dB PSNR) than traditional gradient-based attacks and enables adversarial training that improves robustness and cross-adversarial generalization. The study provides a rigorous comparison against FGSM, PGD, and C&W, demonstrates robust gains through metamorphic testing, and highlights the importance of diverse adversarial exposure for reliable segmentation in safety-critical domains. The findings suggest SegRMT as both an effective robustness evaluator and a practical tool for improving generalization against a broad spectrum of perturbations in real-world vision systems.

Abstract

Image segmentation is critical for applications such as medical imaging, augmented reality, and video surveillance. However, segmentation models often lack robustness, making them vulnerable to adversarial perturbations from subtle image distortions. In this work, we propose SegRMT, a metamorphic testing approach that leverages genetic algorithms (GA) to optimize sequences of spatial and spectral transformations while preserving image fidelity via a predefined PSNR threshold. Using the Cityscapes dataset, our method generates adversarial examples that effectively challenge the DeepLabV3 segmentation model. Our experiments show that SegRMT reduces DeepLabV3's mean Intersection over Union (mIoU) to 6.4%, outperforming other adversarial baselines that decrease mIoU to between 8.5% and 21.7%. Furthermore, when used for adversarial training, SegRMT boosts model performance, achieving mIoU improvements up to 73% on dedicated adversarial datasets and increasing cross-adversarial mIoU to 53.8%, compared to only 2%-10% for other methods. These findings demonstrate that SegRMT not only simulates realistic image distortions but also enhances the robustness of segmentation models, making it a valuable tool for ensuring reliable performance in safety-critical applications.

Evaluating and Enhancing Segmentation Model Robustness with Metamorphic Testing

TL;DR

This work introduces SegRMT, a metamorphic testing framework that uses a genetic algorithm to discover sequences of spatial and spectral perturbations under a PSNR constraint to stress test image segmentation models. Evaluated on Cityscapes with DeepLabV3, SegRMT delivers stronger adversaries (mIoU as low as at dB PSNR) than traditional gradient-based attacks and enables adversarial training that improves robustness and cross-adversarial generalization. The study provides a rigorous comparison against FGSM, PGD, and C&W, demonstrates robust gains through metamorphic testing, and highlights the importance of diverse adversarial exposure for reliable segmentation in safety-critical domains. The findings suggest SegRMT as both an effective robustness evaluator and a practical tool for improving generalization against a broad spectrum of perturbations in real-world vision systems.

Abstract

Image segmentation is critical for applications such as medical imaging, augmented reality, and video surveillance. However, segmentation models often lack robustness, making them vulnerable to adversarial perturbations from subtle image distortions. In this work, we propose SegRMT, a metamorphic testing approach that leverages genetic algorithms (GA) to optimize sequences of spatial and spectral transformations while preserving image fidelity via a predefined PSNR threshold. Using the Cityscapes dataset, our method generates adversarial examples that effectively challenge the DeepLabV3 segmentation model. Our experiments show that SegRMT reduces DeepLabV3's mean Intersection over Union (mIoU) to 6.4%, outperforming other adversarial baselines that decrease mIoU to between 8.5% and 21.7%. Furthermore, when used for adversarial training, SegRMT boosts model performance, achieving mIoU improvements up to 73% on dedicated adversarial datasets and increasing cross-adversarial mIoU to 53.8%, compared to only 2%-10% for other methods. These findings demonstrate that SegRMT not only simulates realistic image distortions but also enhances the robustness of segmentation models, making it a valuable tool for ensuring reliable performance in safety-critical applications.

Paper Structure

This paper contains 39 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. RGB image segmentation
  4. Evaluating robustness using adversarial attacks
  5. Metamorphic testing and PSNR
  6. Problem formulation
  7. Methodology
  8. Image Transformation
  9. Spatial Distortions
  10. Spectral Distortions
  11. Robustness Criterion
  12. Genetic Algorithm for Optimizing Transformations
  13. Genetic Operators and Algorithm Configuration
  14. Integration with Metamorphic Testing
  15. Preserving Semantic Integrity
  16. ...and 24 more sections

Figures (3)

  • Figure 1: Pipeline of the proposed SegRMT for robustness assessment. The pipeline illustrates the process from initial image perturbation using various transformations and optimization using the genetic algorithm to the evaluation of segmentation model performance.
  • Figure 2: Transformation vector structure.
  • Figure 3: Violin Plot of IoU for Different Attack Methods