Physics-Inspired Degradation Models for Hyperspectral Image Fusion

TL;DR

This work tackles the challenge that hyperspectral image fusion often relies on imperfect degradation models, which degrade performance in real-world scenarios. It introduces physics-inspired degradation models (PIDM) consisting of SpaDN for spatial degradation and SpeDN for spectral degradation, incorporating non-uniform effects via spatial warping and spectral modulation, along with asymmetric and parallel downsampling to align degradation with physical characteristics. A self-supervised training scheme enables plug-and-play deployment of PIDM with existing fusion methods, and extensive experiments on four real datasets and four auxiliary datasets demonstrate improved degradation modeling (lower SSIM/RMSE between degraded outputs) and consistently enhanced fusion quality (QNR) across methods and modes. The approach offers a practical, adaptable framework to improve HS fusion in realistic imaging systems and paves the way for fusion algorithms tailored to physics-based degradation models.

Abstract

The fusion of a low-spatial-resolution hyperspectral image (LR-HSI) with a high-spatial-resolution multispectral image (HR-MSI) has garnered increasing research interest. However, most fusion methods solely focus on the fusion algorithm itself and overlook the degradation models, which results in unsatisfactory performance in practical scenarios. To fill this gap, we propose physics-inspired degradation models (PIDM) to model the degradation of LR-HSI and HR-MSI, which comprises a spatial degradation network (SpaDN) and a spectral degradation network (SpeDN). SpaDN and SpeDN are designed based on two insights. First, we employ spatial warping and spectral modulation operations to simulate lens aberrations, thereby introducing non-uniformity into the spatial and spectral degradation processes. Second, we utilize asymmetric downsampling and parallel downsampling operations to separately reduce the spatial and spectral resolutions of the images, thus ensuring the matching of spatial and spectral degradation processes with specific physical characteristics. Once SpaDN and SpeDN are established, we adopt a self-supervised training strategy to optimize the network parameters and provide a plug-and-play solution for fusion methods. Comprehensive experiments demonstrate that our proposed PIDM can boost the fusion performance of existing fusion methods in practical scenarios.

Figures (6)

• Figure 1: Illustration of our proposed PIDM. We utilize SpaDN and SpeDN to represent the spatial and spectral degradation models of LR-HSI and HR-MSI, respectively.
• Figure 2: The training strategy and usage modes of degradation models. (a) Self-supervised training strategy. (b) Training samples generation mode. (c) Objective function construction mode.
• Figure 3: Framework of proposed PIDM. The structures of our designed SpaDN and SpeDN are mirror-symmetric. The implementation of SpaDN is divided into two SW-step and AD-step, and the implementation of SpeDN is divided into two SM-step and PD-step. The network parameters are optimized using a self-supervised training strategy.
• Figure 4: Examples of generated shifted anisotropic Gaussian kernels. The parameters $[\alpha_y, \alpha_x, \delta_{y}, \delta_{x}, \beta]$ of these kernels are: (a) $[0, 0, 3, 3, 0]$. (b)$[-2, -2, 2.5, 1.5, \pi/8]$. (c) $[2, 2, 2.5, 3.5, \pi/4]$. (d) $[-2, 0, 1.5, 2.5, 0]$.
• Figure 5: Visualization (with bands 1-2-3 as B-G-R) of degradation estimation for each degradation model on the HypSen dataset. We add the word suffix "-DM" on the names of fusion methods to represent their respective degradation models. Row 1 represents the spatially degraded HR-MSI $\widetilde{\textbf{Z}}$ and row 2 represents the spectrally degraded LR-HSI $\widetilde{\textbf{Y}}$. Row 3 represents the modeling errors map.