Transparency Distortion Robustness for SOTA Image Segmentation Tasks

TL;DR

This work investigates how spatially varying distortions from transparent materials affect semantic segmentation. It introduces a grid distortion model to synthetically simulate local radial distortions and evaluates Swin Transformer and DINOv2 backbones, revealing substantial performance degradation, especially under grid distortions. The study finds that while larger pretraining and model capacity provide some robustness, fine-tuning on distorted data offers only limited relief, underscoring the need for intrinsic robust representations. The findings highlight a practical challenge for real-world vision systems operating through transparent surfaces and point toward future directions in distortion-aware training and model design.

Abstract

Semantic Image Segmentation facilitates a multitude of real-world applications ranging from autonomous driving over industrial process supervision to vision aids for human beings. These models are usually trained in a supervised fashion using example inputs. Distribution Shifts between these examples and the inputs in operation may cause erroneous segmentations. The robustness of semantic segmentation models against distribution shifts caused by differing camera or lighting setups, lens distortions, adversarial inputs and image corruptions has been topic of recent research. However, robustness against spatially varying radial distortion effects that can be caused by uneven glass structures (e.g. windows) or the chaotic refraction in heated air has not been addressed by the research community yet. We propose a method to synthetically augment existing datasets with spatially varying distortions. Our experiments show, that these distortion effects degrade the performance of state-of-the-art segmentation models. Pretraining and enlarged model capacities proof to be suitable strategies for mitigating performance degradation to some degree, while fine-tuning on distorted images only leads to marginal performance improvements.

Figures (5)

• Figure 1: The novel grid distortion model applied to a grid of lines (a) and a real scene (b). The bottom left crop in (b) shows the corresponding undistorted view. The simulated distortion of the real scene xie2020segmenting creates similarly curved edges as the real distorted view of the same building (c). The real transparent structure also adds other distortions and artifacts like reflection and a lower contrast to the scene, which are not part of the simulation.
• Figure 2: Results of global radial distortion (top) and grid distortion (bottom) for different parameters $K_1, K_1$ on an example image (a). Global radial distortion with positive parameters (b) leads to barrel distortion, while $K_1 < 0$ and $K_2 = 0$ (c) lead to a severe pincushion distortion effect. With grid distortion increasing values of $K_1, K_2$ lead to increasingly wavy edges (bottom, left to right).
• Figure 3: Comparison between Swin Transformer models on different intensities $\sigma$ of global distortion (a), and between Swin Transformer models and DINOv2 on grid distortion (b). Swin-T and Swin-B are the Tiny and Base Swin Transformer models pretrained on ImageNet-1K, Swin-B (22K) is the model pretrained on ImageNet-22K. Error bars show the 95% confidence interval of IoU differences.
• Figure 4: The ground truth and segmentations on the undistorted and distorted ADE20K datasets zhou2019semantic.
• Figure 5: Comparison of various retraining methods on original and distorted data.