Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining

TL;DR

This work targets efficient Earth observation pretraining by leveraging open land-cover-land-use annotations and vision foundation models. It introduces SoftCon, a soft, multi-label guided contrastive loss that aligns cross-scene similarity with label distributions, built on a global multi-label dataset SSL4EO-S12-ML created by matching SSL4EO-S12 with Dynamic World maps. It further demonstrates cross-domain continual pretraining using strong vision model weights and Siamese masking to adapt to multispectral and SAR data without RGB alignment, achieving state-of-the-art results on 10 of 11 downstream tasks. The approach yields compact, high-performing multispectral and SAR foundation models and provides a global multi-label benchmark dataset, with implications for scalable, cross-domain remote sensing representations.

Abstract

Self-supervised pretraining on large-scale satellite data has raised great interest in building Earth observation (EO) foundation models. However, many important resources beyond pure satellite imagery, such as land-cover-land-use products that provide free global semantic information, as well as vision foundation models that hold strong knowledge of the natural world, are not widely studied. In this work, we show these free additional resources not only help resolve common contrastive learning bottlenecks, but also significantly boost the efficiency and effectiveness of EO pretraining. Specifically, we first propose soft contrastive learning that optimizes cross-scene soft similarity based on land-cover-generated multi-label supervision, naturally solving the issue of multiple positive samples and too strict positive matching in complex scenes. Second, we revisit and explore cross-domain continual pretraining for both multispectral and SAR imagery, building efficient EO foundation models from strongest vision models such as DINOv2. Adapting simple weight-initialization and Siamese masking strategies into our soft contrastive learning framework, we demonstrate impressive continual pretraining performance even when the input modalities are not aligned. Without prohibitive training, we produce multispectral and SAR foundation models that achieve significantly better results in 10 out of 11 downstream tasks than most existing SOTA models. For example, our ResNet50/ViT-S achieve 84.8/85.0 linear probing mAP scores on BigEarthNet-10\% which are better than most existing ViT-L models; under the same setting, our ViT-B sets a new record of 86.8 in multispectral, and 82.5 in SAR, the latter even better than many multispectral models. Dataset and models are available at \url{https://github.com/zhu-xlab/softcon}.

Figures (10)

• Figure 1: A visual comparison of transfer learning performances on BigEarthNet-10%. SoftCon (ours) achieves SOTA results with lighter backbones on both linear probing and fine-tuning, and both multispectral and SAR. Our best multispectral linear result is comparable to best models' fine-tuning; our best SAR result outperforms many multispectral models. See \ref{['fig:scatter-split']} for separated views of each setting.
• Figure 2: The workflow of the multi-label dataset curation.
• Figure 3: Different contrastive learning designs. (a) The original contrastive learning performs strict instance discrimination where one anchor image has only one positive pair; (b) supervised contrastive learning allows multiple positive responses when images belong to the same class; (c) our proposed soft contrastive learning can effectively exploit multi-label annotations by assigning soft similarity scores to different pairs. SofCon is a more generic design that covers SupCon: when multi-label degrades to single-label, the soft similarities turn into binary scores, thus becoming multi-positive supervised contrastive learning.
• Figure 4: The general framework of SoftCon. Given a batch of input images, two batches of augmented views are parallelly sent through the two branches of a Siamese network. A similarity matrix is calculated based on the resulting two batches of feature vectors. A weighted sum of the contrastive and soft contrastive loss is optimized. For contrastive loss, this matrix should be close to Identity; for soft contrastive loss, this matrix should be close to the similarity matrix of the label vectors. We load vision foundation models for both the trainable (base encoder) and the frozen (momentum encoder) branches when initializing the model. During training, the weights of the momentum encoder are updated by exponential moving average (EMA) of the base encoder: $\theta_{\mathrm{momentum}} \leftarrow m \cdot \theta_{\mathrm{momentum}}+(1-m) \cdot \theta_{\mathrm{base}}$, where $m \in[0,1)$ is a momentum coefficient. Siamese masking is only used with ViT backbones, where random patches of an image are masked out and only the visible patches are sent through the trainable branch.
• Figure 5: A detailed comparison of transfer learning performances on BigEarthNet-10%. S/B/L represents ViT-small/base/large. SoftCon (ours) achieves SOTA results with lighter backbones on both linear probing and fine-tuning, and both multispectral and SAR. Our best multispectral linear result is better or comparable to many SOTA models' fine-tuning results; our best SAR result outperforms many multispectral models.