Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations

Ting Wang, Zipei Yan, Jizhou Li, Xile Zhao, Chao Wang, Michael Ng

TL;DR

Theoretically, the low-rank property and Lipschitz continuity are proved in the proposed continuous low-rank factorization (CLoRF), which significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining.

Abstract

The fusion of a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR). Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI), leveraging an appropriate image prior and likelihood computed from the discrepancy between the latent HSI and observed images. Low rankness stands out for preserving latent HSI characteristics through matrix factorization among the various priors. However, the primary limitation in previous studies lies in the generalization of a fusion model with fixed resolution scales, which necessitates retraining whenever output resolutions are changed. To overcome this limitation, we propose a novel continuous low-rank factorization (CLoRF) by integrating two neural representations into the matrix factorization, capturing spatial and spectral information, respectively. This approach enables us to harness both the low rankness from the matrix factorization and the continuity from neural representation in a self-supervised manner.Theoretically, we prove the low-rank property and Lipschitz continuity in the proposed continuous low-rank factorization. Experimentally, our method significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining. Code is available at https://github.com/wangting1907/CLoRF-Fusion.

Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations

TL;DR

Theoretically, the low-rank property and Lipschitz continuity are proved in the proposed continuous low-rank factorization (CLoRF), which significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining.

Abstract

The fusion of a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR). Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI), leveraging an appropriate image prior and likelihood computed from the discrepancy between the latent HSI and observed images. Low rankness stands out for preserving latent HSI characteristics through matrix factorization among the various priors. However, the primary limitation in previous studies lies in the generalization of a fusion model with fixed resolution scales, which necessitates retraining whenever output resolutions are changed. To overcome this limitation, we propose a novel continuous low-rank factorization (CLoRF) by integrating two neural representations into the matrix factorization, capturing spatial and spectral information, respectively. This approach enables us to harness both the low rankness from the matrix factorization and the continuity from neural representation in a self-supervised manner.Theoretically, we prove the low-rank property and Lipschitz continuity in the proposed continuous low-rank factorization. Experimentally, our method significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining. Code is available at https://github.com/wangting1907/CLoRF-Fusion.
Paper Structure (19 sections, 4 theorems, 29 equations, 16 figures, 10 tables)

This paper contains 19 sections, 4 theorems, 29 equations, 16 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Implicit Neural Representation
  4. HSI-MSI Fusion via Continuous Representation
  5. Proposed Method
  6. Problem Formulation
  7. Deep Continuous Low-rank Factorization Model
  8. Theoretical Analysis
  9. Experiments and Analysis
  10. Experimental Details
  11. Evaluation on Synthetic Data
  12. Arbitrary Resolution
  13. Ablation Study
  14. Evaluation on Real Dataset
  15. PAN-HSI Fusion
  16. ...and 4 more sections

Key Result

Theorem 1

Let $\mathbf{X} \in \mathbb{R}^{n_1 \times n_2}$, where $\mathrm{rank} (\mathbf{X})=K$, then there exists two matrices $\mathbf{U} \in \mathbb {R}^{n_1 \times K}$, $\mathbf{V} \in \mathbb {R}^{n_2 \times K}$ such that $\mathbf{X}=\mathbf{U} \mathbf{V}^T$.

Figures (16)

  • Figure 1: The pipeline of CLoRF for arbitrary resolution. (A) Train the CLoRF. (B) Use the trained CLoRF to infer arbitrary resolutions of HSIs with given spatial and spectral coordinates. (C) An example of CLoRF for super-resolution on the Pavia University ($336\times336\times93$). The CLoRF is trained given its LR-MSI ($168\times168\times4$) and HR-HSI ($42\times42\times50$), then infers the original resolution. Bicubic interpolation directly upsamples from LR. In the spectral domain, each band has distinct brightness values, and bicubic interpolation estimates the missing bands by referencing adjacent ones. As a result, this can cause discrepancies in the color representation of the interpolated image, such as in band 60, when compared to the GT image. (D) Visualize the spectrum of a random pixel from the results on (C).
  • Figure 2: Illustration of the proposed CLoRF for MSI-HSI fusion. The spatial coordinates and spectral coordinates of HR-MSI and LR-HSI are fed into the Spatial-INR $\Phi_{\theta}(\cdot)$ and Spectral-INR $\Psi_{\alpha}(\cdot)$ to generate coefficients and the bases, respectively. Thereafter, the generated coefficients and the bases are multiplied to recover the HR-HSI.
  • Figure 3: The first row shows the Pavia University image (1st band) of the estimated HR-HSI, and the second row shows the error map between the estimated image and GT.
  • Figure 4: The first row shows the Pavia Center image (8th band) of the estimated HR-HSI, and the second row shows the error map between the estimated image and GT.
  • Figure 5: The first row shows the Indian Pines image (160th band) of the estimated HR-HSI, and the second row shows the error map between the estimated image and GT.
  • ...and 11 more figures

Theorems & Definitions (9)

  • Theorem 1: rank factorization piziak1999full
  • Definition 1: sampled matrix set
  • Definition 2: matrix function rank
  • Proposition 2
  • Theorem 3: continuous low-rank factorization
  • Remark 1
  • Theorem 4: Lipschitz continuity
  • Remark 2
  • Remark 3