Degradation Alchemy: Self-Supervised Unknown-to-Known Transformation for Blind Hyperspectral Image Fusion

TL;DR

A novel self-supervised unknown-to-known degradation transformation framework (U2K) for blind HSI fusion, which adaptively transforms unknown degradation into the same type of degradation as those handled by pre-trained SLMs is proposed.

Abstract

Hyperspectral image (HSI) fusion is an efficient technique that combines low-resolution HSI (LR-HSI) and high-resolution multispectral images (HR-MSI) to generate high-resolution HSI (HR-HSI). Existing supervised learning methods (SLMs) can yield promising results when test data degradation matches the training ones, but they face challenges in generalizing to unknown degradations. To unleash the potential and generalization ability of SLMs, we propose a novel self-supervised unknown-to-known degradation transformation framework (U2K) for blind HSI fusion, which adaptively transforms unknown degradation into the same type of degradation as those handled by pre-trained SLMs. Specifically, the proposed U2K framework consists of: (1) spatial and spectral Degradation Wrapping (DW) modules that map HR-HSI to unknown degraded HR-MSI and LR-HSI, and (2) Degradation Transformation (DT) modules that convert these wrapped data into predefined degradation patterns. The transformed HR-MSI and LR-HSI pairs are then processed by a pre-trained network to reconstruct the target HR-HSI. We train the U2K framework in a self-supervised manner using consistency loss and greedy alternating optimization, significantly improving the flexibility of blind HSI fusion. Extensive experiments confirm the effectiveness of our proposed U2K framework in boosting the adaptability of five existing SLMs under various degradation settings and surpassing state-of-the-art blind methods.

Figures (7)

• Figure 1: Visual fusion results (bottom-left) and reconstruction error maps (top-right) under various degradation settings for the image chart_and_stuffed_toy. From left to right, the columns show the degradation operators and the results of the baseline ma2024reciprocal, baseline with C2F zhang2024unsupervised, and baseline with U2K, respectively. The baseline with U2K shows the best generalization capability.
• Figure 2: The RSNR relationship between the input data and the fusion results.
• Figure 3: The concept and training pipeline of our proposed U2K framework.
• Figure 4: Illustration of the degradations used in the experiments.
• Figure 5: Visual the HSI fusion result for the ZY1-02D satellite.