Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training

TL;DR

This work addresses domain adaptation for semantic segmentation under limited unlabeled data by continual self-supervised pre-training. It introduces GLARE, a three-level framework (global, regional, local) built on a student-teacher ViT setup that trains only lightweight UniAdapter modules to mitigate forgetting. GLARE enforces global, region-aware, and patch-level consistency, including attention-guided region sampling and cross-attention region correspondence, with patch-augmentation strategies that include patch blurring. Empirical results across ADE20k, Pascal Context, Cityscapes, and LoveDA show consistent segmentation gains over baselines and alternative continual pre-training methods, and improvements transfer to classification tasks as well. The approach yields strong practical benefits for adapting foundation models to new, data-scarce domains in dense prediction tasks with modest computational overhead.

Abstract

Self-supervised learning (SSL) has emerged as a central paradigm for training foundation models by leveraging large-scale unlabeled datasets, often producing representations with strong generalization capabilities. These models are typically pre-trained on general-purpose datasets such as ImageNet and subsequently adapted to various downstream tasks through finetuning. While prior work has investigated parameter-efficient adaptation methods like adapters, LoRA, and prompt tuning, primarily targeting downstream finetuning, extending the SSL pre-training itself in a continual manner to new domains under limited data remains largely underexplored, especially for downstream dense prediction tasks like semantic segmentation. In this work, we address the challenge of adapting vision foundation models to low-data target domains through continual self-supervised pre-training, specifically targeting downstream semantic segmentation. We propose GLARE (Global Local and Regional Enforcement), a novel continual self-supervised pre-training task designed to enhance downstream semantic segmentation performance. GLARE introduces patch-level augmentations to encourage local consistency and incorporates a regional consistency constraint that leverages spatial semantics in the data. For efficient continual pre-training, we initialize Vision Transformers (ViTs) with weights from existing SSL models and update only lightweight adapter modules specifically UniAdapter - while keeping the rest of the backbone frozen. Experiments across multiple semantic segmentation benchmarks on different domains demonstrate that GLARE consistently improves downstream performance with minimal computational and parameter overhead.

Figures (5)

• Figure 1: Overview of GLARE continual pre-training framework. Given an image, two views $X$ and $X'$ are generated with image-level augmentation. Each view goes through the base and momentum encoders $f_\theta,\theta_A$ and $f_{\theta',\theta'_A}$. All the parameters are frozen except for those of the adapter on the base encoder. GLARE applies three levels of feature consistency during the pre-training. Firstly, global consistency is considered on [CLS] tokens ($L_{glob}$) of the two views \ref{['sec:global_level']}. Secondly, regional consistency is applied on sampled regions, with their representations obtained using a cross-attention module to calculate $L_{reg}$\ref{['sec:reg_cons']}. Finally, we enforce local consistency focusing on patch-augmentation consistency with distorted vs. not distorted patches of the same view ($L_{loc_1}$) and inter-view local consistency on matching patches from the two views ($L_{loc_2}$). \ref{['sec:local_level']}
• Figure 2: Attention map of the last block of a DINO dino pre-trained model over different heads on an image. The different heads have their attention directed toward specific regions in the image. Some heads focus more on the dog on the left, others on the dog on the right and also on the background.
• Figure 3: Effect of dataset scale on the performance of GLARE continual pre-training applied on LoveDA and ADE20k. The dashed gray line represent the baseline performance of UDI pre-trained model on the respective dataset.
• Figure 4: Visualization of self-attention maps obtained from DINO, UDI and GLARE continual pre-trained models from the last block of the ViT encoder starting from UDI.
• Figure 5: PCA Visualizations of the embeddings on images from COVIDx wu2023covidxcxr4expandedmultiinstitutional and Derm7pt derm7pt datasets. We take the first 3 principal components and show the results for DINO and DINO $\rightarrow$ GLARE continual pre-trained model.