Prototype-Based Information Compensation Network for Multi-Source Remote Sensing Data Classification

TL;DR

PICNet tackles land-cover classification from multi-source RS data by jointly enhancing inter-frequency coupling and global cross-modal information modeling. It introduces a Frequency Feature Interaction Encoder (FIM) to decouple and recouple high- and low-frequency information and a Prototype-based Information Compensation Module (PICM) that uses modality-specific prototypes with cross-attention to align HSI and SAR/LiDAR features. The model is trained with a composite loss combining cross-entropy and consistency terms to enforce reliable cross-modal alignment, and it demonstrates state-of-the-art performance on Augsburg, Berlin, and Houston 2018 datasets with favorable computational efficiency. The work offers a practical fusion framework with plug-and-play scalability for diverse multi-source remote sensing scenarios and sets the stage for further enhancements in multi-scale frequency modeling and hierarchical attention fusion.

Abstract

Multi-source remote sensing data joint classification aims to provide accuracy and reliability of land cover classification by leveraging the complementary information from multiple data sources. Existing methods confront two challenges: inter-frequency multi-source feature coupling and inconsistency of complementary information exploration. To solve these issues, we present a Prototype-based Information Compensation Network (PICNet) for land cover classification based on HSI and SAR/LiDAR data. Specifically, we first design a frequency interaction module to enhance the inter-frequency coupling in multi-source feature extraction. The multi-source features are first decoupled into high- and low-frequency components. Then, these features are recoupled to achieve efficient inter-frequency communication. Afterward, we design a prototype-based information compensation module to model the global multi-source complementary information. Two sets of learnable modality prototypes are introduced to represent the global modality information of multi-source data. Subsequently, cross-modal feature integration and alignment are achieved through cross-attention computation between the modality-specific prototype vectors and the raw feature representations. Extensive experiments on three public datasets demonstrate the significant superiority of our PICNet over state-of-the-art methods. The codes are available at https://github.com/oucailab/PICNet.

Figures (12)

• Figure 1: The motivation of the proposed Prototype-based Information Compensation Network (PICNet). Two sets of learnable modality-specific prototypes are introduced to store the global modality features, which can be used to compensate for the multi-source feature alignment.
• Figure 2: Comparative analysis of the efficacy between the Frequency Interaction Module (FIM) proposed in this study and conventional inter-frequency coupling methodologies. (a) Original input image, (b) Feature map before frequency interaction, (c) Feature representation after Wavelet transformation, (d) Enhanced feature map subsequent to FIM.
• Figure 3: The overall framework of the proposed PICNet. It consists of two key components: (1) The frequency feature interaction encoder which leverages the Frequency Interaction Module (FIM) to decouple the frequency information of multi-source data and use the dominant features of one modality to enhance the complementary feature of the other. (2) The Prototype-based Information Compensation Module (PICM) is responsible for compensating the missing modality-specific information, which is achieved by cross-attention computation between the prototype vectors and the raw feature representations.
• Figure 4: Pooling-based frequency feature separation.
• Figure 5: Multisource remote sensing dataset for land cover classification. (a) Augsburg dataset. (b) Berlin dataset. (c) Houston2018 dataset.