HetSSNet: Spatial-Spectral Heterogeneous Graph Learning Network for Panchromatic and Multispectral Images Fusion

TL;DR

HetSSNet tackles pansharpening by modeling spatial-spectral relationships in a non-Euclidean space using a spatial-spectral heterogeneous graph (HetSS-Graph). It introduces a Basic Relationship Pattern Generation Module to extract multiple pattern matrices and a Relationship Pattern Aggregation Module with local and global perspectives, tied together by a contrastive loss to align representations. The approach yields significant improvements over state-of-the-art methods across multiple datasets and scales to real-world full-resolution scenes, demonstrating strong generalization. This graph-based framework offers a flexible, principled way to fuse PAN and LR-MS data into high-resolution HR-MS imagery with preserved spatial detail and spectral fidelity.

Abstract

Remote sensing pansharpening aims to reconstruct spatial-spectral properties during the fusion of panchromatic (PAN) images and low-resolution multi-spectral (LR-MS) images, finally generating the high-resolution multi-spectral (HR-MS) images. In the mainstream modeling strategies, i.e., CNN and Transformer, the input images are treated as the equal-sized grid of pixels in the Euclidean space. They have limitations in facing remote sensing images with irregular ground objects. Graph is the more flexible structure, however, there are two major challenges when modeling spatial-spectral properties with graph: \emph{1) constructing the customized graph structure for spatial-spectral relationship priors}; \emph{2) learning the unified spatial-spectral representation through the graph}. To address these challenges, we propose the spatial-spectral heterogeneous graph learning network, named \textbf{HetSSNet}. Specifically, HetSSNet initially constructs the heterogeneous graph structure for pansharpening, which explicitly describes pansharpening-specific relationships. Subsequently, the basic relationship pattern generation module is designed to extract the multiple relationship patterns from the heterogeneous graph. Finally, relationship pattern aggregation module is exploited to collaboratively learn unified spatial-spectral representation across different relationships among nodes with adaptive importance learning from local and global perspectives. Extensive experiments demonstrate the significant superiority and generalization of HetSSNet.

Figures (7)

• Figure 1: The overview of HetSSNet. Our HetSSNet consists of three components: spatial-spectral heterogeneous graph construction, basic relationship pattern generation and relationship pattern aggregation. According to the provided relationship priors, we construct the spatial-spectral heterogeneous graph structure in non-Euclidean space. Based on the constructed graph, we generate a series of basic spatial-spectral relationship pattern matrices, and finally aggregate these basic relationship patterns from the local and global perspectives.
• Figure 2: Qualitative results of reduced-resolution scene on the WorldView-3 dataset. Top group: the fused results. Bottom group: the error between fused results and reference.
• Figure 3: The spatial distribution comparison between LR-MS/PAN and GT (target HR-MS) in terms of each spectral band. The band illustrated in the figure refers to the spectral band. The correlation coefficient represents the correlation between the two distribution curves, and the larger the value, the greater the correlation.
• Figure 4: The distribution comparison between LR-MS's spectral bands, and the pixel distribution comparison between GT's spectral bands. The band illustrated in the figure denotes the spectral band. The correlation coefficient represents the correlation between the two distribution curves, and the larger the value, the greater the correlation.
• Figure 5: Qualitative results of reduced-resolution scene on the QuickBird dataset. Top group: the fused results. Bottom group: the error between fused results and reference.