Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyperspectral Image Fusion with Spectral-Band and Fusion-Scale Agnosticism

Yu-Jie Liang, Zihan Cao, Liang-Jian Deng, Yang Yang, Malu Zhang

TL;DR

This work proposes SSA, a universal framework for MS/HS fusion with spectral-band and fusion-scale agnosticism, and introduces Matryoshka Kernel, a novel operator that enables a single model to adapt to arbitrary numbers of spectral channels.

Abstract

Current deep learning models for Multispectral and Hyperspectral Image Fusion (MS/HS fusion) are typically designed for fixed spectral bands and spatial scales, which limits their transferability across diverse sensors. To address this, we propose SSA, a universal framework for MS/HS fusion with spectral-band and fusion-scale agnosticism. Specifically, we introduce Matryoshka Kernel (MK), a novel operator that enables a single model to adapt to arbitrary numbers of spectral channels. Meanwhile, we build SSA upon an Implicit Neural Representation (INR) backbone that models the HS signal as a continuous function, enabling reconstruction at arbitrary spatial resolutions. Together, these two forms of agnosticism enable a single MS/HS fusion model that generalizes effectively to unseen sensors and spatial scales. Extensive experiments demonstrate that our single model achieves state-of-the-art performance while generalizing well to unseen sensors and scales, paving the way toward future HS foundation models.

Hyperspectral Image Fusion with Spectral-Band and Fusion-Scale Agnosticism

TL;DR

This work proposes SSA, a universal framework for MS/HS fusion with spectral-band and fusion-scale agnosticism, and introduces Matryoshka Kernel, a novel operator that enables a single model to adapt to arbitrary numbers of spectral channels.

Abstract

Current deep learning models for Multispectral and Hyperspectral Image Fusion (MS/HS fusion) are typically designed for fixed spectral bands and spatial scales, which limits their transferability across diverse sensors. To address this, we propose SSA, a universal framework for MS/HS fusion with spectral-band and fusion-scale agnosticism. Specifically, we introduce Matryoshka Kernel (MK), a novel operator that enables a single model to adapt to arbitrary numbers of spectral channels. Meanwhile, we build SSA upon an Implicit Neural Representation (INR) backbone that models the HS signal as a continuous function, enabling reconstruction at arbitrary spatial resolutions. Together, these two forms of agnosticism enable a single MS/HS fusion model that generalizes effectively to unseen sensors and spatial scales. Extensive experiments demonstrate that our single model achieves state-of-the-art performance while generalizing well to unseen sensors and scales, paving the way toward future HS foundation models.
Paper Structure (29 sections, 17 equations, 12 figures, 11 tables, 2 algorithms)

This paper contains 29 sections, 17 equations, 12 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Deep Learning for MS/HS Image Fusion
  4. Matryoshka Representation Learning and Flexible Model Design
  5. Method
  6. Problem Formulation and Model Overview
  7. Matryoshka Kernels: Spectral Agnosticism
  8. Arbitrary-Scale Image Fusion as Continuous Function
  9. Cross-Sensor Joint Training Strategy
  10. Experiments
  11. Datasets
  12. Implementation Details
  13. Main Results
  14. Quantitative Comparison
  15. Qualitative Comparison
  16. ...and 14 more sections

Figures (12)

  • Figure 1: Existing paradigms v.s. our universal approach for hyperspectral image fusion. Our method can suit different spectral bands joint training and can inference on arbitrary-scale fusion.
  • Figure 2: The overall architecture of our proposed SSA framework. This end-to-end model realizes the universal mapping function $\mathcal{F}$, which takes an LR-HSI and an HR-MSI from various sensors as input and reconstructs a high-fidelity HR-HSI at arbitrary scales.
  • Figure 3: Illustration of the MKL's encoding and decoding.
  • Figure 4: Quantitative comparison of PSNR (dB) across multiple scaling factors on all seven datasets.
  • Figure 5: The upper and lower parts respectively showcase the fused images and error maps from the CAVE and PaviaC datasets. Red rectangles depict some close-up shots.
  • ...and 7 more figures