MP-HSIR: A Multi-Prompt Framework for Universal Hyperspectral Image Restoration

TL;DR

MP-HSIR addresses universal hyperspectral image restoration under unknown and diverse degradations by introducing a multi-prompt framework that fuses spectral, textual, and visual cues through a prompt-guided spatial-spectral transformer. The spectral prompt enables universal low-rank spectral pattern priors for robust spectral reconstruction, while the text-visual synergistic prompt encodes degradation information for controllable restoration. Comprehensive experiments across 9 restoration tasks, including all-in-one, generalization, and real-world cases, show MP-HSIR consistently surpasses all-in-one baselines and many task-specific methods, with strong spectral fidelity and restoration quality. The approach offers improved robustness, interpretability, and deployment feasibility, with public code and models supporting broad adoption in HSIs and related imaging domains.

Abstract

Hyperspectral images (HSIs) often suffer from diverse and unknown degradations during imaging, leading to severe spectral and spatial distortions. Existing HSI restoration methods typically rely on specific degradation assumptions, limiting their effectiveness in complex scenarios. In this paper, we propose \textbf{MP-HSIR}, a novel multi-prompt framework that effectively integrates spectral, textual, and visual prompts to achieve universal HSI restoration across diverse degradation types and intensities. Specifically, we develop a prompt-guided spatial-spectral transformer, which incorporates spatial self-attention and a prompt-guided dual-branch spectral self-attention. Since degradations affect spectral features differently, we introduce spectral prompts in the local spectral branch to provide universal low-rank spectral patterns as prior knowledge for enhancing spectral reconstruction. Furthermore, the text-visual synergistic prompt fuses high-level semantic representations with fine-grained visual features to encode degradation information, thereby guiding the restoration process. Extensive experiments on 9 HSI restoration tasks, including all-in-one scenarios, generalization tests, and real-world cases, demonstrate that MP-HSIR not only consistently outperforms existing all-in-one methods but also surpasses state-of-the-art task-specific approaches across multiple tasks. The code and models are available at https://github.com/ZhehuiWu/MP-HSIR.

Figures (15)

• Figure 1: PSNR comparison with the state-of-the-art all-in-one and task-specific methods across 9 tasks. Average results across all test datasets. Best viewed in color.
• Figure 2: Overview of HSI characteristics and prompt-based all-in-one restoration methods. (a) Differences between HSIs and RGB images. (b) Impacts of different degradations on spectral features. (c) Visual prompt-based models. (d) Textual prompt-based models. (e) Dual-modality prompt-based models. (f) The proposed model integrating spectral, textual, and visual prompts.
• Figure 3: (a) The architecture of the proposed MP-HSIR. (b) Prompt-Guided Spatial-Spectral Transformer Block (PGSSTB). (c) Design of the Prompt-Guided Spectral Self-Attention (PGSSA) Module. (d) Text-Visual Synergistic Prompt (TVSP) Module.
• Figure 4: [All-in-one] Visual comparison results on complex denoising, inpainting, and dehazing. The zoomed-in regions are highlighted with red boxes, the corresponding zoomed results are marked with green boxes, and the residual maps are outlined with yellow boxes.
• Figure 5: The spectral errors of the restored HSIs across multiple tasks. (a) Gaussian denoising. (b) Complex denoising. (c) Gaussian deblurring. (d) Super-resolution. (e) Inpainting. (f) Dehazing. (g) Band completion. (h) Poisson denoising. Zoom in for the best view.