HSI-VAR: Rethinking Hyperspectral Restoration through Spatial-Spectral Visual Autoregression

TL;DR

This paper tackles hyperspectral image restoration under complex, mixed degradations by reframing the task as a visual autoregressive generation problem. The proposed HSI-VAR framework leverages next-scale prediction with a multi-scale VQVAE and introduces three key innovations: latent-condition alignment to bridge HQ latents and degraded conditions, degradation-aware guidance to efficiently manage multiple degradations, and a spatial-spectral adaptation module to preserve fine-grained structure in both domains. Empirical results across nine benchmarks show strong PSNR/SSIM gains and dramatic inference speedups (up to $95.5\times$ versus diffusion-based methods), while ablations demonstrate the value of each component for quality and efficiency. The approach offers a practical, scalable solution for real-world HSI restoration with robust structure preservation and perceptual quality.

Abstract

Hyperspectral images (HSIs) capture richer spatial-spectral information beyond RGB, yet real-world HSIs often suffer from a composite mix of degradations, such as noise, blur, and missing bands. Existing generative approaches for HSI restoration like diffusion models require hundreds of iterative steps, making them computationally impractical for high-dimensional HSIs. While regression models tend to produce oversmoothed results, failing to preserve critical structural details. We break this impasse by introducing HSI-VAR, rethinking HSI restoration as an autoregressive generation problem, where spectral and spatial dependencies can be progressively modeled rather than globally reconstructed. HSI-VAR incorporates three key innovations: (1) Latent-condition alignment, which couples semantic consistency between latent priors and conditional embeddings for precise reconstruction; (2) Degradation-aware guidance, which uniquely encodes mixed degradations as linear combinations in the embedding space for automatic control, remarkably achieving a nearly $50\%$ reduction in computational cost at inference; (3) A spatial-spectral adaptation module that refines details across both domains in the decoding phase. Extensive experiments on nine all-in-one HSI restoration benchmarks confirm HSI-VAR's state-of-the-art performance, achieving a 3.77 dB PSNR improvement on \textbf{\textit{ICVL}} and offering superior structure preservation with an inference speed-up of up to $95.5 \times$ compared with diffusion-based methods, making it a highly practical solution for real-world HSI restoration.

Figures (8)

• Figure 1: Regression v.s. Visual Autoregressive Modeling. Our HSI-VAR rethinks HSI restoration as scale-wise autoregressive generation, progressively modeling the structure dependencies.
• Figure 2: Illustration of the architecture of our HSI-VAR. (a) Structure of the multi-scale VQVAE. (b) HSI-VAR with the proposed three core components: feature alignment strategy, degradation-aware guidance and the spatial-spectral adaptation.
• Figure 3: Motivation for Alignment: Pre-trained encoder pocesses similar latents for degraded inputs even without extra training.
• Figure 4: Illustration of the inference pipeline of our HSI-VAR, which incorporates modules and processes features scale by scale.
• Figure 5: Motivation for DAG: Different degradations have not only the specificities but also the commonalities.