Heuristic Self-Paced Learning for Domain Adaptive Semantic Segmentation under Adverse Conditions

Abstract

The learning order of semantic classes significantly impacts unsupervised domain adaptation for semantic segmentation, especially under adverse weather conditions. Most existing curricula rely on handcrafted heuristics (e.g., fixed uncertainty metrics) and follow a static schedule, which fails to adapt to a model's evolving, high-dimensional training dynamics, leading to category bias. Inspired by Reinforcement Learning, we cast curriculum learning as a sequential decision problem and propose an autonomous class scheduler. This scheduler consists of two components: (i) a high-dimensional state encoder that maps the model's training status into a latent space and distills key features indicative of progress, and (ii) a category-fair policy-gradient objective that ensures balanced improvement across classes. Coupled with mixed source-target supervision, the learned class rankings direct the network's focus to the most informative classes at each stage, enabling more adaptive and dynamic learning. It is worth noting that our method achieves state-of-the-art performance on three widely used benchmarks (e.g., ACDC, Dark Zurich, and Nighttime Driving) and shows generalization ability in synthetic-to-real semantic segmentation.

Figures (4)

• Figure 1: From design curriculum to self-paced learning. Traditional Curriculum Learning (CL) and Hard Class Mining (HCM) both rely on fixed handcrafted priors and fixed learning paths. CL adopts a static, easy-to-hard curriculum, while HCM focuses solely on difficult classes. Both strategies induce class bias, manifesting as significant accuracy discrepancies across semantic classes. Differently, we adaptively mine the most informative classes via dynamically perceived models' evolving ($S$: Source domain, $T$: Target domain, $M_{n}$: the $n$-th intermediate domain, $Cn$: the $n$-th Semantic Class).
• Figure 2: The framework of our designed Heuristic Semantic Class Mining (HeuSCM). First, a Gaussian Mixture VAE (GM-VAE) encodes high-dimensional semantic states into latent features $z_t^s$. Our SKFEN processes $z_t^s$ to distill key features reflecting the model's learning status. Conditioned on these features, ClassGen outputs ranked classes (sorted in descending order of informational value). These rankings guide the generation of mixed image and mixed label pairs, which optimize the segmentation model via SegLoss. Concurrently, we maximize the objective $J_F(\pi)$ to jointly optimize the copied GM-VAE encoder, SKFEN, and ClassGen.
• Figure 3: The qualitative comparison between our method and existing state-of-the-art methods based on DeepLabV2, DAFormer, and HRDA on the ACDC val. Compared with the existing state-of-the-art UDA method (e.g., Refign, CMA, VBLC, ACSegFormer), our method achieves better performance under the same backbone. Importantly, our method, based on HRDA, achieves the best performance, with predictions closely matching the ground truth, validating the effectiveness of our method.
• Figure 4: A qualitative comparison between our method and existing state-of-the-art approaches based on DAFormer and HRDA is conducted on the Dark Zurich val (Top two rows) and Nighttime Driving test (Bottom two rows).