Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network with Spatial-Spectral Manifold Learning

He Wang, Yang Xu, Zebin Wu, Zhihui Wei

TL;DR

A novel deep Tucker decomposition network is designed that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameters, and a Laplacian-based spatial–spectral manifold constraint is introduced in shared-decoders.

Abstract

Hyperspectral and multispectral image fusion aims to generate high spectral and spatial resolution hyperspectral images (HR-HSI) by fusing high-resolution multispectral images (HR-MSI) and low-resolution hyperspectral images (LR-HSI). However, existing fusion methods encounter challenges such as unknown degradation parameters, incomplete exploitation of the correlation between high-dimensional structures and deep image features. To overcome these issues, in this article, an unsupervised blind fusion method for hyperspectral and multispectral images based on Tucker decomposition and spatial spectral manifold learning (DTDNML) is proposed. We design a novel deep Tucker decomposition network that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameter. To better exploit and fuse spatial-spectral features in the data, we design a core tensor fusion network that incorporates a spatial spectral attention mechanism for aligning and fusing features at different scales. Furthermore, to enhance the capacity in capturing global information, a Laplacian-based spatial-spectral manifold constraints is introduced in shared-decoders. Sufficient experiments have validated that this method enhances the accuracy and efficiency of hyperspectral and multispectral fusion on different remote sensing datasets. The source code is available at https://github.com/Shawn-H-Wang/DTDNML.

Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network with Spatial-Spectral Manifold Learning

TL;DR

A novel deep Tucker decomposition network is designed that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameters, and a Laplacian-based spatial–spectral manifold constraint is introduced in shared-decoders.

Abstract

Hyperspectral and multispectral image fusion aims to generate high spectral and spatial resolution hyperspectral images (HR-HSI) by fusing high-resolution multispectral images (HR-MSI) and low-resolution hyperspectral images (LR-HSI). However, existing fusion methods encounter challenges such as unknown degradation parameters, incomplete exploitation of the correlation between high-dimensional structures and deep image features. To overcome these issues, in this article, an unsupervised blind fusion method for hyperspectral and multispectral images based on Tucker decomposition and spatial spectral manifold learning (DTDNML) is proposed. We design a novel deep Tucker decomposition network that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameter. To better exploit and fuse spatial-spectral features in the data, we design a core tensor fusion network that incorporates a spatial spectral attention mechanism for aligning and fusing features at different scales. Furthermore, to enhance the capacity in capturing global information, a Laplacian-based spatial-spectral manifold constraints is introduced in shared-decoders. Sufficient experiments have validated that this method enhances the accuracy and efficiency of hyperspectral and multispectral fusion on different remote sensing datasets. The source code is available at https://github.com/Shawn-H-Wang/DTDNML.
Paper Structure (18 sections, 27 equations, 12 figures, 8 tables)

This paper contains 18 sections, 27 equations, 12 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Driven Based Methods
  4. Deep Learning Based Methods
  5. Model Guided in DL Methods
  6. Problem Formulation
  7. Methodology
  8. Deep Tucker Decomposition Network
  9. Core Tensor Fusion Network
  10. Weighted Manifold Learning
  11. Joint Loss Function
  12. Experiment
  13. Datasets and Experiments Setup
  14. Experimental Results
  15. Ablation Studies
  16. ...and 3 more sections

Figures (12)

  • Figure 1: The architecture of the HSI-MSI blind fusion based on deep Tucker decomposition and manifold learning
  • Figure 2: The structure of Core Tensor Fusion Network.
  • Figure 3: Spectral-Spatial Attention Module
  • Figure 4: Illustration of fusion results on Pavia University dataset. First row: pseudo-RGB image (R:61, G:36, B:10). The second and third rows are heatmaps of RMSE and SAM between the ground truth and reconstructed respectively. The error map range are [0, 2.4] and [0, 2.6]. (a).GT, (b).CNMF5982386, (c).HySure7000523, (d).CSTF8359412, (e).uSDNuSDN, (f).CuCANet10.1007/978-3-030-58526-6_13, (g).HyCoNetHyCoNet, (h).MIAE9681709, (i).UDTN10115230, (j).DTDNML
  • Figure 5: Illustration of fusion results on Chikusei dataset. First row: pseudo-RGB image (R:56, G:16, B:36). The second and third rows are heatmaps of RMSE and SAM between the ground truth and reconstructed respectively. The error map range are [0, 1.4] and [0, 2.8]. (a).GT, (b).CNMF5982386, (c).HySure7000523, (d).CSTF8359412, (e).uSDNuSDN, (f).CuCANet10.1007/978-3-030-58526-6_13, (g).HyCoNetHyCoNet, (h).MIAE9681709, (i).UDTN10115230, (j).DTDNML
  • ...and 7 more figures