Balanced Learning for Domain Adaptive Semantic Segmentation

TL;DR

This work addresses unsupervised domain adaptation for semantic segmentation under severe class bias caused by cross-domain data and label distribution shifts. It introduces Balanced Learning for Domain Adaptation (BLDA), which analyzes per-class logits to quantify bias, uses anchor-based post-hoc logit alignment, incorporates online logits adjustment via Gaussian Mixture Models, and leverages a cumulative density estimation as a shared bridge between domains. BLDA is designed as a plug-in to existing self-training UDA methods and yields consistent improvements, particularly for under-predicted classes, across GTA→Cityscapes and SYN→Cityscapes benchmarks. The approach emphasizes direct, distribution-informed balancing of classes and offers scalable, efficient implementation with public code.

Abstract

Unsupervised domain adaptation (UDA) for semantic segmentation aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Despite the effectiveness of self-training techniques in UDA, they struggle to learn each class in a balanced manner due to inherent class imbalance and distribution shift in both data and label space between domains. To address this issue, we propose Balanced Learning for Domain Adaptation (BLDA), a novel approach to directly assess and alleviate class bias without requiring prior knowledge about the distribution shift. First, we identify over-predicted and under-predicted classes by analyzing the distribution of predicted logits. Subsequently, we introduce a post-hoc approach to align the logits distributions across different classes using shared anchor distributions. To further consider the network's need to generate unbiased pseudo-labels during self-training, we estimate logits distributions online and incorporate logits correction terms into the loss function. Moreover, we leverage the resulting cumulative density as domain-shared structural knowledge to connect the source and target domains. Extensive experiments on two standard UDA semantic segmentation benchmarks demonstrate that BLDA consistently improves performance, especially for under-predicted classes, when integrated into various existing methods. Code is available at https://github.com/Woof6/BLDA.

Figures (10)

• Figure 1: Demonstration of factors that cause class bias. (a) The inherent class imbalance problem in segmentation datasets. (b) The differences in transfer difficulty across classes in cross-domain settings. "Oracle' represents the performance under full supervision, while "Src-only' represents training with the source domain and testing it on the target domain. (c) The differences in logits distributions predicted for each class, including "positive distribution" and "negative distribution". (d) Bias assessment for different classes via Eq.\ref{['eq.4']}. The corresponding class IDs of (a), (b), and (c) are mapped in descending order onto this figure.
• Figure 2: In UDA, class bias can be expressed as over-predicted classes and under-predicted classes. (a) Class-wise accuracy under different training settings. (b) Frequency of pseudo-labels generated by the network for different classes during training.
• Figure 3: Illustration of proposed post-hoc class balancing. (a) The logits distributions of over-predicted and under-predicted classes. (b) Reweighting/resampling strategies alleviate class imbalance by adjusting the training emphasis on different classes. (c) Our post-hoc logits adjustment method aligns the logits distributions of all classes with anchor distributions to achieve balanced prediction.
• Figure 4: Qualitative results. Note that the yellow boxes mark regions improved by BLDA.
• Figure 5: Study of the different evalution metrics with respect to scaling factor $\tau$.