Self-supervised Deep Hyperspectral Inpainting with the Plug and Play and Deep Image Prior Models

TL;DR

This work tackles hyperspectral image inpainting under noise and data loss by introducing LRS-PnP-DIP(1-Lip), a convergent algorithm that combines joint sparsity and low-rank priors with a deep image prior (DIP) and a plug-and-play denoiser within an ADMM framework. The authors provide a Lyapunov-based convergence analysis under mild assumptions, including a 1-Lipschitz DIP and a theta-averaged denoiser, and prove fixed-point convergence to a stable solution; they replace the traditional SVT low-rank step with a DIP to better capture non-linear spectral-spatial structures. Extensive experiments on Chikusei and Indian Pines demonstrate state-of-the-art inpainting performance with self-supervision (no training data) and competitive runtimes, highlighting the stability gains from the Lipschitz constraint. The approach is practically impactful for on-board hyperspectral processing and remote sensing pipelines, offering convergence guarantees while achieving high-fidelity reconstructions across varied mask patterns and spectral bands.

Abstract

Hyperspectral images are typically composed of hundreds of narrow and contiguous spectral bands, each containing information regarding the material composition of the imaged scene. However, these images can be affected by various sources of noise, distortions, or data loss, which can significantly degrade their quality and usefulness. This paper introduces a convergent guaranteed algorithm, LRS-PnP-DIP(1-Lip), which successfully addresses the instability issue of DHP that has been reported before. The proposed algorithm extends the successful joint low-rank and sparse model to further exploit the underlying data structures beyond the conventional and sometimes restrictive unions of subspace models. A stability analysis guarantees the convergence of the proposed algorithm under mild assumptions , which is crucial for its application in real-world scenarios. Extensive experiments demonstrate that the proposed solution consistently delivers visually and quantitatively superior inpainting results, establishing state-of-the-art performance.

Figures (9)

• Figure S1: Flow chart of the proposed LRS-PnP-DIP(1-Lip) algorithm. At each iteration, $\boldsymbol{\alpha}^{k+1}$$\boldsymbol{u}^{k+1}$ and $\boldsymbol{x}^{k+1}$ are sequentially updated. The 1-Lipschitz DIP is implemented by imposing Lipschitz constraints on all layers. We use red color to highlight the differences between this work and the LRS-PnP-DIP algorithm li2023self. The detailed design of the 1-Lipschitz DIP is placed in Appendix \ref{['appendix_1_Lip_DIP']}.
• Figure S2: Comparison of MPSNR value of LRS-PnP among different $\tau$ ($\tau = w_{s}/w_{lr}$) under different masks.
• Figure S3: Learning capability of DIP vs. number of input and output channels. Training is conducted with a single-band HS image, meaning that the input HSIs are processed; there is no correlation in the spectral domain. There is a significant performance gain when there are more input bands, indicating that the DIP with 2D convolution has the ability to exploit the correlation between channels.
• Figure S4: The amplitude of the singular value of the reconstructed image upon converge. The important singular values are captured and preserved via the 2D-convolution DIP, which is even more accurate than the traditional SVT projection.
• Figure S5: Empirical converge of LRS-PnP-DIP(1-Lip) with modified NLM denoiser and non-expansive/1-Lipschitz DIP. Top left, top right, and bottom left: successive difference of $\boldsymbol{x}$, $\boldsymbol{\lambda}_1$, and $\boldsymbol{\lambda}_2$ in the log scale, respectively. Bottom right: the inpainting MPSNR vs. the number of iterations.

Theorems & Definitions (9)

• Definition 1
• Definition 2
• Lemma 1
• Definition 3
• Definition 4
• Definition 5
• Lemma 2
• Lemma 3
• Theorem 1