HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models

TL;DR

This work tackles hyperspectral image restoration by exploiting a low-rank decomposition $\\mathcal{X}=\\mathcal{A} \\times_3 \\mathbf{E}$ and a pre-trained diffusion prior to synthesize the reduced image $\\mathcal{A}$. The coefficient matrix $\\mathbf{E}$ is robustly estimated from the degraded observation using SVD plus RRQR-based band index selection, while the reduced image is inferred through a conditional diffusion model guided by a data fidelity term and a TV penalty. A novel exponential diffusion schedule accelerates sampling by stabilizing early-noise decay and late-detail refinement, achieving about 5x faster denoising with minimal performance loss. The method demonstrates superior restoration quality across denoising, super-resolution, and inpainting tasks and offers practical speed advantages over prior approaches, making it suitable for diverse HSI applications with limited training data.

Abstract

Hyperspectral image (HSI) restoration aims at recovering clean images from degraded observations and plays a vital role in downstream tasks. Existing model-based methods have limitations in accurately modeling the complex image characteristics with handcraft priors, and deep learning-based methods suffer from poor generalization ability. To alleviate these issues, this paper proposes an unsupervised HSI restoration framework with pre-trained diffusion model (HIR-Diff), which restores the clean HSIs from the product of two low-rank components, i.e., the reduced image and the coefficient matrix. Specifically, the reduced image, which has a low spectral dimension, lies in the image field and can be inferred from our improved diffusion model where a new guidance function with total variation (TV) prior is designed to ensure that the reduced image can be well sampled. The coefficient matrix can be effectively pre-estimated based on singular value decomposition (SVD) and rank-revealing QR (RRQR) factorization. Furthermore, a novel exponential noise schedule is proposed to accelerate the restoration process (about 5$\times$ acceleration for denoising) with little performance decrease. Extensive experimental results validate the superiority of our method in both performance and speed on a variety of HSI restoration tasks, including HSI denoising, noisy HSI super-resolution, and noisy HSI inpainting. The code is available at https://github.com/LiPang/HIRDiff.

Figures (5)

• Figure 1: The overall framework of the proposed HIR-Diff. First, the coefficient matrix $\mathbf{E}$ is estimated from the degraded image using SVD and RRQR. Then, taking the degraded image and the estimated matrix $\mathbf{E}$ as conditions, the reduced image $\mathcal{A}$ is reconstructed with an improved pre-trained diffusion model that contains a newly designed guidance function. Finally, the clean image is restored from the product of the estimated $\mathbf{E}$ and $\mathcal{A}$.
• Figure 2: (a) The $\bar{\alpha}_t$ in the linear schedule, cosine schedule and our proposed exponential schedule. (b) The PSNR values throughout the diffusion process with different noise schedules. Linear schedule (*) and cosine schedule (*) denote the results when the guidance strength is enhanced.
• Figure 3: The visual result comparison of all the competing methods on the HSI restoration task.
• Figure 4: The reduced image $\mathcal{A}$ estimated by different methods. Ground Truth denotes the bands selected from the clean HSI.
• Figure 5: The visualization results of the estimated $\mathbf{E}$. Least Square and Ground Truth (LS) denote the coefficient matrix $\mathbf{E}$ estimated by employing the least square method with the observed image and the clean image, respectively. Ours and Ground Truth (SVD) denote the coefficient matrix $\mathbf{E}$ estimated using SVD and RRQR proposed in our work with the observed image and the clean image, respectively.

Theorems & Definitions (1)

• Remark 3.1