Efficient Hyperspectral Image Reconstruction Using Lightweight Separate Spectral Transformers

TL;DR

This work addresses the efficient reconstruction of hyperspectral images from compressive CASSI measurements. It introduces the Lightweight Separate Spectral Transformer (LSST), built from Separate Spectral Transformer Blocks (SSTB) and Lightweight Spatial Convolution Blocks (LSCB), and augments it with Focal Spectrum Loss to balance per-band reconstruction. Empirical results show LSST achieves superior accuracy with far fewer parameters and FLOPs than both CNN- and Transformer-based baselines across multiple datasets, including real CASSI data, demonstrating strong efficiency and generalization. The proposed divide-and-conquer spectral-spatial design enables practical deployment in resource-constrained scenarios and offers a solid foundation for future integrations with model-based unfolding approaches.

Abstract

Hyperspectral imaging (HSI) is essential across various disciplines for its capacity to capture rich spectral information. However, efficiently reconstructing hyperspectral images from compressive sensing measurements presents significant challenges. To tackle these, we adopt a divide-and-conquer strategy that capitalizes on the unique spectral and spatial characteristics of hyperspectral images. We introduce the Lightweight Separate Spectral Transformer (LSST), an innovative architecture tailored for efficient hyperspectral image reconstruction. This architecture consists of Separate Spectral Transformer Blocks (SSTB) for modeling spectral relationships and Lightweight Spatial Convolution Blocks (LSCB) for spatial processing. The SSTB employs Grouped Spectral Self-attention and a Spectrum Shuffle operation to effectively manage both local and non-local spectral relationships. Simultaneously, the LSCB utilizes depth-wise separable convolutions and strategic ordering to enhance spatial information processing. Furthermore, we implement the Focal Spectrum Loss, a novel loss weighting mechanism that dynamically adjusts during training to improve reconstruction across spectrally complex bands. Extensive testing demonstrates that our LSST achieves superior performance while requiring fewer FLOPs and parameters, underscoring its efficiency and effectiveness. The source code is available at: https://github.com/wcz1124/LSST.

Figures (12)

• Figure 1: Natural hyperspectral images display unique properties. (a) In each spectral band, the local neighborhoods in the image typically exhibit stronger spatial correlation. (b) Spectral bands that are closer in proximity often demonstrate stronger correlations compared to those that are more distant.
• Figure 2: Comparing accuracy and efficiency among various approaches. The circle's radius represents the number of model parameters.
• Figure 3: The overarching architecture of the LSST utilizes a U-shaped configuration, incorporating stacked Lightweight Separate Spectral Transformer Blocks (LSSTB). Each LSSTB comprises two primary components: the Separate Spectral Transformer Block (SSTB) and the Lightweight Spatial Convolution Block (LSCB), which are dedicated to efficiently modeling spectral and spatial relationships, respectively.
• Figure 4: The conceptual framework of the Separate Spectral Transformer Block (SSTB) utilizes Grouped Spectral Self-attention along with a parameter-free Spectrum Shuffle operation, effectively managing both local and non-local spectral relationships.
• Figure 5: Left: PSNR for reconstructed results across various frequency bands. Right: Focal Spectrum Loss with different focusing parameters.