Benchmarking the Robustness of Instance Segmentation Models

TL;DR

This paper tackles the problem of assessing robustness and generalization in instance segmentation by conducting a broad benchmark across architectures, backbones, normalization layers, initializations, and training strategies. The authors evaluate models on real-world corruptions (ImageNet-C style) and cross-domain datasets (COCO to Cityscapes/BDD), examining the effects of single-stage versus multi-stage architectures, diverse backbones (including transformer-based ones), normalization choices (BN, SyncBN, GN), and copy-paste augmentation, as well as joint-task training. Key findings show that group normalization improves robustness to corruptions, synchronized batch normalization aids cross-dataset generalization, single-stage detectors are more robust to corruptions while multi-stage/cascaded architectures generalize better to new domains, and training from scratch can outperform ImageNet pretraining in robustness (except for JPEG corruption). The study provides practical guidance for building robust instance segmentation systems in real-world deployments and highlights promising directions for future robustness-focused development, such as normalization-aware design and backbone selection tailored to domain shift challenges.

Abstract

This paper presents a comprehensive evaluation of instance segmentation models with respect to real-world image corruptions as well as out-of-domain image collections, e.g. images captured by a different set-up than the training dataset. The out-of-domain image evaluation shows the generalization capability of models, an essential aspect of real-world applications and an extensively studied topic of domain adaptation. These presented robustness and generalization evaluations are important when designing instance segmentation models for real-world applications and picking an off-the-shelf pretrained model to directly use for the task at hand. Specifically, this benchmark study includes state-of-the-art network architectures, network backbones, normalization layers, models trained starting from scratch versus pretrained networks, and the effect of multi-task training on robustness and generalization. Through this study, we gain several insights. For example, we find that group normalization enhances the robustness of networks across corruptions where the image contents stay the same but corruptions are added on top. On the other hand, batch normalization improves the generalization of the models across different datasets where statistics of image features change. We also find that single-stage detectors do not generalize well to larger image resolutions than their training size. On the other hand, multi-stage detectors can easily be used on images of different sizes. We hope that our comprehensive study will motivate the development of more robust and reliable instance segmentation models.

Figures (9)

• Figure 1: An image from the COCO validation dataset, the original clean version, and corrupted ones by different noise models (severity=3). B.=Blur, N.=Noise, D.=Digital, and W.=Weather. Row 2 and 4 show inference results of the R50-FPN model on images from rows 1 and 3, respectively. On blur corrupted images, traffic lights and most of the persons are no longer detected. With impulse noise, the pole is detected as a tie. While changing the brightness of an image does not cause a significant accuracy drop, the parking meter is no longer detected in this example. The model does not detect persons on the bus and falsely detects a person on top of the bus on the image with JPEG compression. Snow and frost cause falsely duplicated detections on the bus, which are accurately detected on the other corrupted images.
• Figure 2: An image from the COCO validation dataset, the original clean version, and the corrupted one under mixed corruption setting. This image is corrupted with motion blur, speckle noise, jpeg compression, and spatter. We show the inference results of R50-FPN model in Row 2.
• Figure 3: An image from the COCO validation dataset, the original clean version, and the corrupted one under geometric transformation. We show the inference results of R50-FPN model in Row 2.
• Figure 4: Rows present the results of BN R50-FPN pretrained, SyncBN R50-FPN scratch, and GN R50-FPN scratch models, respectively. On the clean COCO validation image, models perform similarly. On the image corrupted with speckle noise, there are large performance gaps within models. Sync BN model from scratch outputs more accurate masks compared to BN pretrained model. GN from scratch model shows the highest robustness to the corruption and continues detecting the frisbee and the car with high confidence.
• Figure 5: Rows present the results of R50-C4, R50-DC5, R50-FPN Cascade models, and R50-FPN Deformable models, respectively. On the clean COCO validation image, models perform detections with high accuracy. On the other hand, on the Gaussian noise corrupted image, C4 architecture outputs noisy detections. DC5 based model achieves slightly better results but still outputs false detections such as the bottle. FPN deformable architectured model achieves the highest robustness in this example.