SwiMDiff: Scene-wide Matching Contrastive Learning with Diffusion Constraint for Remote Sensing Image

TL;DR

SwiMDiff tackles two key RSI SSL challenges: confusing false negatives among geographically adjacent crops and the loss of fine-grained details in traditional contrastive learning. It fuses scene-wide matching with a diffusion-model auxiliary task, optimizing a joint objective $L = \lambda_C L_C + \lambda_D L_D$ to learn representations that capture both global semantics and local textures. Across change detection and land-cover classification benchmarks (OSCD, LEVIR-CD, BigEarthNet, EuroSAT), SwiMDiff delivers state-of-the-art or competitive improvements and clear ablation benefits for each component. The approach promises richer, transferable RSI representations for downstream tasks and highlights directions for efficiency improvements in diffusion-assisted SSL.

Abstract

With recent advancements in aerospace technology, the volume of unlabeled remote sensing image (RSI) data has increased dramatically. Effectively leveraging this data through self-supervised learning (SSL) is vital in the field of remote sensing. However, current methodologies, particularly contrastive learning (CL), a leading SSL method, encounter specific challenges in this domain. Firstly, CL often mistakenly identifies geographically adjacent samples with similar semantic content as negative pairs, leading to confusion during model training. Secondly, as an instance-level discriminative task, it tends to neglect the essential fine-grained features and complex details inherent in unstructured RSIs. To overcome these obstacles, we introduce SwiMDiff, a novel self-supervised pre-training framework designed for RSIs. SwiMDiff employs a scene-wide matching approach that effectively recalibrates labels to recognize data from the same scene as false negatives. This adjustment makes CL more applicable to the nuances of remote sensing. Additionally, SwiMDiff seamlessly integrates CL with a diffusion model. Through the implementation of pixel-level diffusion constraints, we enhance the encoder's ability to capture both the global semantic information and the fine-grained features of the images more comprehensively. Our proposed framework significantly enriches the information available for downstream tasks in remote sensing. Demonstrating exceptional performance in change detection and land-cover classification tasks, SwiMDiff proves its substantial utility and value in the field of remote sensing.

Figures (11)

• Figure 1: In remote sensing datasets, images are typically cropped from large scene images. Images cropped from the same scene exhibit certain similarities in terms of color, texture details, and overall layout.
• Figure 2: Diagram of the SwiMDiff. The network architecture is bifurcated into two components: 1) A dual-branch structure for CL. 2) A diffusion model network comprising an encoder and a decoder.
• Figure 3: The forward diffusion process of the diffusion model. It's similar to a Markov chain, where noise is added based on the previous state.
• Figure 4: High-frequency components of the images and its shallow features. (a): Input Images. (b): High-frequency components extracted from input images. (c): High-frequency details of shallow features extracted from encoder pre-trained by CL. (d): High-frequency details of shallow features extracted from encoder pre-trained by model integrating with the diffusion model.
• Figure 5: The network architecture for change detection task. The images taken at different times are first processed by a pre-trained and frozen encoder $f_{q}$. It extracts two sets of features from these images. These feature sets are then passed through a difference module. The resulting difference is then input into the decoder for further processing.