A General Adaptive Dual-level Weighting Mechanism for Remote Sensing Pansharpening

TL;DR

This work tackles remote sensing pansharpening by explicitly modeling feature heterogeneity and redundancy through a covariance-based approach. It introduces Correlation-Aware Covariance Weighting (CACW) to generate data-driven weights from a normalized covariance matrix and builds an Adaptive Dual-level Weighting Mechanism (ADWM) comprising Intra-Feature Weighting (IFW) and Cross-Feature Weighting (CFW) that can be plugged into existing networks. The method demonstrates state-of-the-art gains across multiple datasets (WV3, QB, GF2) and resolutions, with extensive ablations, redundancy analyses, and generality tests showing robust, architecture-agnostic improvements. The approach provides a principled, scalable framework for refining feature representations in pansharpening, with practical impact for improving spatial-spectral fidelity in satellite imagery. Key contributions include: (1) covariance-based modeling of feature correlations to guide weighting, (2) a dual-level weighting scheme (IFW for intra-feature channel emphasis, CFW for inter-layer fusion), and (3) plug-and-play integration across diverse networks, yielding consistent gains and guiding future fusion architectures toward leveraging feature correlations more effectively.

Abstract

Currently, deep learning-based methods for remote sensing pansharpening have advanced rapidly. However, many existing methods struggle to fully leverage feature heterogeneity and redundancy, thereby limiting their effectiveness. We use the covariance matrix to model the feature heterogeneity and redundancy and propose Correlation-Aware Covariance Weighting (CACW) to adjust them. CACW captures these correlations through the covariance matrix, which is then processed by a nonlinear function to generate weights for adjustment. Building upon CACW, we introduce a general adaptive dual-level weighting mechanism (ADWM) to address these challenges from two key perspectives, enhancing a wide range of existing deep-learning methods. First, Intra-Feature Weighting (IFW) evaluates correlations among channels within each feature to reduce redundancy and enhance unique information. Second, Cross-Feature Weighting (CFW) adjusts contributions across layers based on inter-layer correlations, refining the final output. Extensive experiments demonstrate the superior performance of ADWM compared to recent state-of-the-art (SOTA) methods. Furthermore, we validate the effectiveness of our approach through generality experiments, redundancy visualization, comparison experiments, key variables and complexity analysis, and ablation studies. Our code is available at https://github.com/Jie-1203/ADWM.

Figures (24)

• Figure 1: Application of our dual-level weighting mechanism within the existing methods. (a) General methods. (b) Intra-Feature Weighting (IFW): weighting different channels within a single feature. (c) Cross-Feature Weighting (CFW): weighting features at different depths to obtain the final result. (d) Our dual-level weighting combines both IFW and CFW to fully unlock the potential of the original networks.
• Figure 2: Feature heterogeneity and redundancy correspond to the covariance matrix: darker colors indicate stronger correlations and redundancy, while lighter colors suggest weaker correlations and more heterogeneity. (a) Intra-feature: different channels within a feature. (b) Cross-feature: features at different depths. (c) CACW leverages intra- and cross-feature correlations to generate weights and adjust features accordingly.
• Figure 3: The CACW structure is illustrated. First, we compute the covariance matrix $\tilde{C}$ based on the correlations among the $n$ columns of $X$. Then, this covariance matrix is passed through a nonlinear function $g$, which generates the resulting weights.
• Figure 4: The overall workflow of ADWM is comprised of two sub-modules: Intra-Feature Weighting (IFW) and Cross-Feature Weighting (CFW). In IFW, each original feature $F_i$ is adjusted to $\tilde{F}_i$ based on its internal correlations. In CFW, weights are generated based on the correlations among $F_i$ features, dynamically adjusting each $\tilde{F}_i$'s contribution to the final output.
• Figure 5: Visualization of covariance matrices, weights in IFW and CFW. (a) Channels that are multiples of six selected for clarity. (b) Lower entropy indicates higher feature redundancy.