Enhancing Scene Classification in Cloudy Image Scenarios: A Collaborative Transfer Method with Information Regulation Mechanism using Optical Cloud-Covered and SAR Remote Sensing Images

TL;DR

This work tackles cloud-prone remote sensing scene classification by transferring knowledge from cloud-free optical data to a target domain containing both cloudy optical and SAR data. It introduces a collaborative transfer framework that uses pseudo-labels and auxiliary models to bridge cross-modality gaps, and an Information Regulation Mechanism (IRM) to dynamically balance information from optical and SAR modalities at the sample level. IRM computes modality contributions and a discrepancy ratio to weight losses, enabling the model to leverage inferior modalities without overfitting the superior one. Experiments on simulated SEN12MS Cloud and real Hunan Cloud datasets show significant improvements over state-of-the-art methods, with IRM reducing modality imbalance and enhancing robustness in cloud-covered scenarios. The approach promises practical impact for continuous monitoring in regions with frequent cloud cover, while recognizing limitations under extreme cloud content and outlining directions for future work in multi-modality pretraining and advanced transfer strategies.

Abstract

In remote sensing scene classification, leveraging the transfer methods with well-trained optical models is an efficient way to overcome label scarcity. However, cloud contamination leads to optical information loss and significant impacts on feature distribution, challenging the reliability and stability of transferred target models. Common solutions include cloud removal for optical data or directly using Synthetic aperture radar (SAR) data in the target domain. However, cloud removal requires substantial auxiliary data for support and pre-training, while directly using SAR disregards the unobstructed portions of optical data. This study presents a scene classification transfer method that synergistically combines multi-modality data, which aims to transfer the source domain model trained on cloudfree optical data to the target domain that includes both cloudy optical and SAR data at low cost. Specifically, the framework incorporates two parts: (1) the collaborative transfer strategy, based on knowledge distillation, enables the efficient prior knowledge transfer across heterogeneous data; (2) the information regulation mechanism (IRM) is proposed to address the modality imbalance issue during transfer. It employs auxiliary models to measure the contribution discrepancy of each modality, and automatically balances the information utilization of modalities during the target model learning process at the sample-level. The transfer experiments were conducted on simulated and real cloud datasets, demonstrating the superior performance of the proposed method compared to other solutions in cloud-covered scenarios. We also verified the importance and limitations of IRM, and further discussed and visualized the modality imbalance problem during the model transfer. Codes are available at https://github.com/wangyuze-csu/ESCCS

Figures (9)

• Figure 1: Valid optical information remains in the yellow part (partially contaminated).
• Figure 2: The general framework of the proposed method. It primarily consists of two components: the Collaborative Transfer Strategy and the Information Regulation Mechanism. The former is designed to facilitate efficient transfer between heterogeneous data, while the latter tackles the issue of modality imbalance.
• Figure 3: The collaborative transfer steps of the proposed method.
• Figure 4: Architecture of the Integrated Representation Model (IRM).
• Figure 5: Numbers and example of each category in SEN12MS Cloud dataset.