Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Equivariant Imaging for Self-supervised Hyperspectral Image Inpainting

Shuo Li, Mike Davies, Mehrdad Yaghoobi

TL;DR

Hyperspectral images often contain missing stripes due to platform motion and atmospheric effects, hindering downstream analysis. Hyper-EI formulates inpainting as recovering $\\boldsymbol{x}$ from the forward model $\\boldsymbol{y} = \\mathbf{M} \\boldsymbol{x} + \\boldsymbol{n}$ by enforcing an Equivariant Imaging (EI) constraint alongside a measurement-consistency objective, implemented with a spatio-spectral attention-augmented network. The paper introduces a novel self-supervised Hyper-EI algorithm and a spatio-spectral attention block with a low-rank spectral memory, demonstrating superior inpainting performance on three real HS datasets compared to existing self-supervised methods. This work offers a data-efficient, robust solution for HSI inpainting that generalizes across masks and resolutions and lays groundwork for extending to related HSI inverse problems such as denoising, compressive sensing, and super-resolution.

Abstract

Hyperspectral imaging (HSI) is a key technology for earth observation, surveillance, medical imaging and diagnostics, astronomy and space exploration. The conventional technology for HSI in remote sensing applications is based on the push-broom scanning approach in which the camera records the spectral image of a stripe of the scene at a time, while the image is generated by the aggregation of measurements through time. In real-world airborne and spaceborne HSI instruments, some empty stripes would appear at certain locations, because platforms do not always maintain a constant programmed attitude, or have access to accurate digital elevation maps (DEM), and the travelling track is not necessarily aligned with the hyperspectral cameras at all times. This makes the enhancement of the acquired HS images from incomplete or corrupted observations an essential task. We introduce a novel HSI inpainting algorithm here, called Hyperspectral Equivariant Imaging (Hyper-EI). Hyper-EI is a self-supervised learning-based method which does not require training on extensive datasets or access to a pre-trained model. Experimental results show that the proposed method achieves state-of-the-art inpainting performance compared to the existing methods.

Equivariant Imaging for Self-supervised Hyperspectral Image Inpainting

TL;DR

Hyperspectral images often contain missing stripes due to platform motion and atmospheric effects, hindering downstream analysis. Hyper-EI formulates inpainting as recovering from the forward model by enforcing an Equivariant Imaging (EI) constraint alongside a measurement-consistency objective, implemented with a spatio-spectral attention-augmented network. The paper introduces a novel self-supervised Hyper-EI algorithm and a spatio-spectral attention block with a low-rank spectral memory, demonstrating superior inpainting performance on three real HS datasets compared to existing self-supervised methods. This work offers a data-efficient, robust solution for HSI inpainting that generalizes across masks and resolutions and lays groundwork for extending to related HSI inverse problems such as denoising, compressive sensing, and super-resolution.

Abstract

Hyperspectral imaging (HSI) is a key technology for earth observation, surveillance, medical imaging and diagnostics, astronomy and space exploration. The conventional technology for HSI in remote sensing applications is based on the push-broom scanning approach in which the camera records the spectral image of a stripe of the scene at a time, while the image is generated by the aggregation of measurements through time. In real-world airborne and spaceborne HSI instruments, some empty stripes would appear at certain locations, because platforms do not always maintain a constant programmed attitude, or have access to accurate digital elevation maps (DEM), and the travelling track is not necessarily aligned with the hyperspectral cameras at all times. This makes the enhancement of the acquired HS images from incomplete or corrupted observations an essential task. We introduce a novel HSI inpainting algorithm here, called Hyperspectral Equivariant Imaging (Hyper-EI). Hyper-EI is a self-supervised learning-based method which does not require training on extensive datasets or access to a pre-trained model. Experimental results show that the proposed method achieves state-of-the-art inpainting performance compared to the existing methods.
Paper Structure (8 sections, 4 equations, 4 figures, 2 tables)

This paper contains 8 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contribution
  3. Proposed Method
  4. Hyper-EI Inpainting Algorithm
  5. Spatio-spectral Attention Blocks
  6. Implementation Details
  7. Experimental Results
  8. Conclusion

Figures (4)

  • Figure 1: Concept of equivariant imaging for HSI inpainting task problem. Path 1 and Path 2 represent the left-hand side and right-hand side of equation \ref{['EI_invariant']}, respectively. The composition of the inpainting operation with the reconstruction method $f$ (e,g. a neural network) should be equivariant to the rotation. Hence, both Path 1 and Path 2 yield identical results.
  • Figure 2: Training loss of the proposed Hyper-EI algorithm. the loss comprises the MC term and the EI regularization term.
  • Figure 3: Spatio-spectral U-Net used in the experiments.
  • Figure 4: Inpainting performance of Hyper-EI on different HS datasets. Specifically, the top two test samples were selected from the Chikusei dataset, the middle two from the Indian Pine dataset, and the bottom two from the Botswana dataset. From left to right: (1) Clean Image, (2) Input Corrupted Image, (3) DHP, (4) PnP-DIP, (5) R-DLRHyIn, (6) Hyper-EI.