Reducing Semantic Ambiguity In Domain Adaptive Semantic Segmentation Via Probabilistic Prototypical Pixel Contrast

TL;DR

Probabilistic prototypical pixel contrast (PPPC), a universal adaptation framework that models each pixel embedding as a probability via multivariate Gaussian distribution to fully exploit the uncertainty within them, eventually improving the representation quality of the model.

Abstract

Domain adaptation aims to reduce the model degradation on the target domain caused by the domain shift between the source and target domains. Although encouraging performance has been achieved by combining cognitive learning with the self-training paradigm, they suffer from ambiguous scenarios caused by scale, illumination, or overlapping when deploying deterministic embedding. To address these issues, we propose probabilistic proto-typical pixel contrast (PPPC), a universal adaptation framework that models each pixel embedding as a probability via multivariate Gaussian distribution to fully exploit the uncertainty within them, eventually improving the representation quality of the model. In addition, we derive prototypes from probability estimation posterior probability estimation which helps to push the decision boundary away from the ambiguity points. Moreover, we employ an efficient method to compute similarity between distributions, eliminating the need for sampling and reparameterization, thereby significantly reducing computational overhead. Further, we dynamically select the ambiguous crops at the image level to enlarge the number of boundary points involved in contrastive learning, which benefits the establishment of precise distributions for each category. Extensive experimentation demonstrates that PPPC not only helps to address ambiguity at the pixel level, yielding discriminative representations but also achieves significant improvements in both synthetic-to-real and day-to-night adaptation tasks. It surpasses the previous state-of-the-art (SOTA) by +5.2% mIoU in the most challenging daytime-to-nighttime adaptation scenario, exhibiting stronger generalization on other unseen datasets. The code and models are available at https://github.com/DarlingInTheSV/Probabilistic-Prototypical-Pixel-Contrast.

Figures (7)

• Figure 1: Illustration of the existing issues in self-training contrastive learning. (a) The decision boundary crosses the part of target pixel embeddings, leading to incorrect pseudo-label predictions. (b) The use of prototype contrast ignores the diversity of pixel embeddings, resulting in the decision boundary being unable to distinguish a few embeddings from both the source and target domains. (c) Our probabilistic prototypical pixel contrast not only better adjusts the decision boundaries but also addresses the issue of ambiguous classes that the previous two methods did not involve.
• Figure 2: Overview of our framework. The model is trained with both the supervised segmentation loss $\mathcal{L}_s$ and the unsupervised adaptation loss $\mathcal{L}_t$. Specifically, the predictions for the source and target domain $P_s$, $P_t$ are provided by the student model, and the pseudo-labels $\tilde{Y}^t$ are generated by the teacher model. Furthermore, the distribution for each pixel embedding $p(z_i |x_i^{s,t})$ is modeled by the projection head of the student model. For prototypes, we first obtain pixel-wise probabilistic embeddings $p(z_i |x_i^s)$from the source image $x^s$ via the projection head $g_p^{\prime}$ within the teacher model. Then, we estimate each composed prototype $p(\rho_c |z_c)$ with these embeddings. Lastly, to stabilize the updating of prototypes we deploy the momentum update strategy. In the embedding space, apart from conducting distributional contrast, we also utilize an additional regularization term $\mathcal{L}_{KL}$ to prevent the covariance collapse to zero. Additionally, AGC uses the ambiguity provided by the online-updated prototypes to select ambiguous scenarios dynamically.
• Figure 3: The Class distribution of the Cityscapes and SYNTHIA datasets.
• Figure 4: Overview of all our tasks. Our tasks include two synthetic-to-real tasks and one daytime-to-nighttime task. Additionally, we conduct extra generalization tests on the daytime-to-nighttime task.
• Figure 5: Qualitative results of domain adaptive semantic segmentation task GTAV $\to$ Cityscapes. Better segmentation results are highlighted in dash boxes. PPPC improves the segmentation of ambiguous scenarios such as inadequately illuminated walls, crowded persons, and challenging classes like bus.