PromptHSI: Universal Hyperspectral Image Restoration with Vision-Language Modulated Frequency Adaptation

TL;DR

PromptHSI tackles the challenge of restoring hyperspectral images under composite degradations by integrating frequency-domain priors with vision-language model guided prompts. The method introduces a frequency-aware feature modulation scheme that maps text-derived task descriptors to intensity and bias controllers, enabling adaptive restoration across hundreds of spectral bands. The architecture combines a U-shaped, residual-in-residual capable encoder–decoder with dual-branch spatial–spectral feature aggregation, and a Prompt-Guided Feature Modulation (PGFM) that links textual cues to frequency coefficients via an adapter over CLIP features. Evaluations on AVIRIS-derived composite degradations show that PromptHSI surpasses state-of-the-art AiO RGB approaches and task-specific HSI methods, while offering controllable restoration through prompts; a composite-degradation dataset is also introduced to standardize benchmarking in this domain. The paper provides a practical pathway for remote sensing applications by achieving higher spectral fidelity and robust restoration in real-world, multi-faceted degradation scenarios, with code released at the provided GitHub link. For example, the frequency-domain degradation model is expressed as $F(I_{degraded}) = (1+\lambda) \odot F(I_{clean}) + \mu$, illustrating how degradations shape the spectrum and guide restoration.

Abstract

Recent advances in All-in-One (AiO) RGB image restoration have demonstrated the effectiveness of prompt learning in handling multiple degradations within a single model. However, extending these approaches to hyperspectral image (HSI) restoration is challenging due to the domain gap between RGB and HSI features, information loss in visual prompts under severe composite degradations, and difficulties in capturing HSI-specific degradation patterns via text prompts. In this paper, we propose PromptHSI, the first universal AiO HSI restoration framework that addresses these challenges. By incorporating frequency-aware feature modulation, which utilizes frequency analysis to narrow down the restoration search space and employing vision-language model (VLM)-guided prompt learning, our approach decomposes text prompts into intensity and bias controllers that effectively guide the restoration process while mitigating domain discrepancies. Extensive experiments demonstrate that our unified architecture excels at both fine-grained recovery and global information restoration across diverse degradation scenarios, highlighting its significant potential for practical remote sensing applications. The source code is available at https://github.com/chingheng0808/PromptHSI.

Figures (18)

• Figure 1: Performance comparison between All-in-One and HSI restoration methods on the AVIRIS dataset. The proposed PromptHSI offers leading performance and controllability while retaining a compact model size.
• Figure 2: Illustration of the degraded HSI and description synthesis pipeline. Each degradation is activated with probability $p$ (set to 0.5 in all experiments). To ensure diversity and realism, each activated degradation randomly applies one of its sub-degradations to the input HSI. More details are provided in the Supplementary.
• Figure 3: Visualizations of the Fourier transforms of HSIs with specific degradations, displayed from top to bottom alongside the Fourier spectra of the residual images (obtained by subtracting the degraded images from the ground truth).
• Figure 4: A diagram illustrating our proposed strategy, where integrating prompt guidance with frequency modulation facilitates the adaptive selection of $\lambda$ and $\mu$, enhancing HSI restoration under various degradation types. This mechanism encourages the model to efficiently converge toward optimal parameters.
• Figure 5: The architecture of the proposed PromptHSI, which is designed to tackle the composite degradations within a single universal model via VLM-guided frequency modulation. First, we design a network based on U-net-like encoder-decoder architecture for capturing global spatial-spectral information, while combining plain-residual-in residual networks in the proposed PGFAB for spatial-spectral enhancement. Afterwards, text-prompt-guidance and frequency-aware feature modulation is proposed to jointly better capture the representation of composite degradation with controllability.