Hyperspectral Image Denoising via Spatial-Spectral Recurrent Transformer

TL;DR

The paper tackles hyperspectral image denoising by leveraging two key domain knowledge components—global spectral correlation (GSC) and non-local spatial self-similarity (NSS)—in a unified framework. It introduces SSRT-UNet, which embeds a spatial-spectral recurrent transformer (SSRT) block with dual branches that interact through shared keys/values, enabling long-range spectral modelling across arbitrary band counts and non-local spatial feature aggregation guided by spectral context. The spectral branch uses an RNN-inspired transformer with gates to propagate information across bands, while the spatial branch exploits NSS under GSC guidance; bidirectional processing and shifted windowing further enhance long-range dependencies. Experiments on synthetic and real HSIs show state-of-the-art denoising performance, supported by extensive ablations that confirm the effectiveness of each component and the importance of network width, with practical implications for robust HSI restoration across diverse band counts.

Abstract

Hyperspectral images (HSIs) often suffer from noise arising from both intra-imaging mechanisms and environmental factors. Leveraging domain knowledge specific to HSIs, such as global spectral correlation (GSC) and non-local spatial self-similarity (NSS), is crucial for effective denoising. Existing methods tend to independently utilize each of these knowledge components with multiple blocks, overlooking the inherent 3D nature of HSIs where domain knowledge is strongly interlinked, resulting in suboptimal performance. To address this challenge, this paper introduces a spatial-spectral recurrent transformer U-Net (SSRT-UNet) for HSI denoising. The proposed SSRT-UNet integrates NSS and GSC properties within a single SSRT block. This block consists of a spatial branch and a spectral branch. The spectral branch employs a combination of transformer and recurrent neural network to perform recurrent computations across bands, allowing for GSC exploitation beyond a fixed number of bands. Concurrently, the spatial branch encodes NSS for each band by sharing keys and values with the spectral branch under the guidance of GSC. This interaction between the two branches enables the joint utilization of NSS and GSC, avoiding their independent treatment. Experimental results demonstrate that our method outperforms several alternative approaches. The source code will be available at https://github.com/lronkitty/SSRT.

Figures (12)

• Figure 1: Comparison of the modelling schemes of the existing methods (a) and ours (b). The existing methods use separate modules to explore NSS (non-local spatial self-similarity) and GSC (global spectral correlation) in an alternating and sequential manner. Our proposed method employs a single spatial-spectral recurrent transformer block for joint spatial-spectral modelling of the GSC and NSS to improve the recovery of HSIs.
• Figure 2: The overall architecture of the SSRT-UNet, where down and up denote $\text{downsampling}(\cdot)$ and $\text{upsampling}(\cdot)$, respectively.
• Figure 3: The structure of a typical SSRT. It consists of a spectral branch and a spatial branch, which interact with each other to achieve joint exploitation of the spatial and spectral properties of the HSI.
• Figure 4: The bidirectional SSRT block where $L$ is the total number of the SSRT blocks in a layer.
• Figure 5: Denoising results on the Houston 2018 HSI with the mixture noise. The false-color images are generated by combining bands 46, 23, and 1.