Are Synthetic Corruptions A Reliable Proxy For Real-World Corruptions?

TL;DR

The paper addresses whether synthetic corruptions can serve as reliable proxies for real-world distribution shifts in semantic segmentation. It conducts a large-scale benchmarking study across real-world corruptions from ACDC and synthetic corruptions from Cityscapes-C, evaluating multiple architectures on Cityscapes, ADE20K, and PASCAL VOC2012. It finds a strong overall correlation in mean performance, suggesting synthetic corruptions are useful for robustness evaluation, though per-corruption analysis reveals notable gaps (e.g., brightness and fog) where proxies misrepresent real-world effects. The work introduces GAM as a worst-case robustness metric, provides insights into which corruption types align, and releases open-source benchmarks to guide future OOD robustness research.

Abstract

Deep learning (DL) models are widely used in real-world applications but remain vulnerable to distribution shifts, especially due to weather and lighting changes. Collecting diverse real-world data for testing the robustness of DL models is resource-intensive, making synthetic corruptions an attractive alternative for robustness testing. However, are synthetic corruptions a reliable proxy for real-world corruptions? To answer this, we conduct the largest benchmarking study on semantic segmentation models, comparing performance on real-world corruptions and synthetic corruptions datasets. Our results reveal a strong correlation in mean performance, supporting the use of synthetic corruptions for robustness evaluation. We further analyze corruption-specific correlations, providing key insights to understand when synthetic corruptions succeed in representing real-world corruptions. Open-source Code: https://github.com/shashankskagnihotri/benchmarking_robustness/tree/segmentation_david/semantic_segmentation

Figures (9)

• Figure 1: Comparing images with weather corruptions captured in the wild (ACDC acdc) and images corrupted using synthetic corruptions commoncorruptions and the predictions using a Mask2Former cheng2021mask2former with a Swin-Base liu2021Swin backbone trained on the Cityscapes cordts2016cityscapes dataset.
• Figure 2: To empirically determine if synthetic common corruptions such as those proposed by commoncorruptions truly represent the distribution and domain shifts in the real world, we try to find correlations in evaluations on ACDC and 2D Common Corruptions. Each model is trained on the training dataset of the Cityscapes dataset. Left plot: The y-axis represents values from evaluations on the ACDC dataset, and the x-axis represents mean performance from evaluations on the Common Corruptions at severity=3. We observe a high positive correlation. Centre plot: The y-axis again represents values from evaluations on the ACDC dataset, while the x-axis represents $\mathrm{GAM}_3$, which is the worst performance of the methods across all the Common Corruptions at severity=3. We observe a slightly higher positive correlation. Right plot: serves as a sanity check, where the y-axis represents $\mathrm{GAM}_3$ and the x-axis represents mean performance from evaluations on the Common Corruptions at the same severity. We observe a very high correlation in performance. Thus, given the high positive correlations between performance on the ACDC and mean performance against all synthetic common corruption, we conclude for relative analysis that synthetic corruptions do serve as a reliable proxy for real-world corruptions.
• Figure 3: Correlation between model performance (legend as in \ref{['fig:correlation_2dcc_acdc']}) on ACDC (real-world corruptions) and 2D Common Corruptions (synthetic) for different corruption types. The left-most plot shows the correlation between mean mIoU across all 2D Common Corruptions and ACDC, with a strong Pearson correlation of 0.759, indicating that synthetic corruptions are generally a reasonable proxy for real-world robustness. The remaining plots analyze specific corruptions: brightness (synthetic) vs. night (real) with correlation 0.270, snow (synthetic) vs. snow (real) with correlation 0.867, fog (synthetic) vs. fog (real) with correlation 0.349, and frost (synthetic) vs. fog (real) with correlation 0.680. While some synthetic corruptions (e.g., snow) closely align with their real-world counterparts, others (e.g., brightness for night) exhibit weaker correlations, highlighting cases where synthetic corruptions may fail as accurate proxies.
• Figure 4: Comparison of mean accuracy across synthetic (2D Common Corruptions) and real-world (ACDC) corruptions. The top plot presents mAcc (mean class accuracy) with a stronger correlation of 0.782–0.858, while the bottom plot shows results for aAcc (all pixel accuracy) with a Pearson correlation of 0.688–0.767. These results indicate that synthetic corruptions serve as a reasonable proxy for real-world robustness, even when measured using metrics other than mIoU
• Figure 5: Illustrating changes in prediction due to different 2D Common Corruptions on a randomly chosen input image from the Cityscapes dataset, when attaching the semantic segmentation method InterImage-Base. In the subfigures with semantic segmentation mask predictions, Left: Ground Truth Mask, and Right: Predicted Mask.