NukesFormers: Unpaired Hyperspectral Image Generation with Non-Uniform Domain Alignment

TL;DR

The paper addresses the challenge of unpaired RGB-HSI data for hyperspectral image generation by introducing Range-Null Space Decomposition (RND) to separate range-space constraints from null-space compensations. The authors propose NukesFormers, featuring Non-uniform KANs (Nuk-MSA) and a Dual-Dimensional Contrastive Prior Module (DCPM) to align geometric and spectral distributions across unpaired domains, while employing adversarial and non-degraded losses to stabilize training. Their approach demonstrates state-of-the-art UnHIG performance on NTIRE 2020/2022 and CAVE datasets, with ablations validating the importance of DCPM components and the Nukes design. The work advances practical UnHIG by enabling cross-domain spectral-spatial learning without GT, potentially improving deployment in engineering applications. All mathematical notation is presented with proper Delimiters, e.g., $D^ op D$, $(I - D^ op D)$, and the loss terms $\mathcal{L}_{cyc}$, $\mathcal{L}_{nde}$, $\mathcal{L}_{adv}$, $\mathcal{L}_{spec}$, and $\mathcal{L}_{geo}$.

Abstract

The inherent difficulty in acquiring accurately co-registered RGB-hyperspectral image (HSI) pairs has significantly impeded the practical deployment of current data-driven Hyperspectral Image Generation (HIG) networks in engineering applications. Gleichzeitig, the ill-posed nature of the aligning constraints, compounded with the complexities of mining cross-domain features, also hinders the advancement of unpaired HIG (UnHIG) tasks. In this paper, we conquer these challenges by modeling the UnHIG to range space interaction and compensations of null space through Range-Null Space Decomposition (RND) methodology. Specifically, the introduced contrastive learning effectively aligns the geometric and spectral distributions of unpaired data by building the interaction of range space, considering the consistent feature in degradation process. Following this, we map the frequency representations of dual-domain input and thoroughly mining the null space, like degraded and high-frequency components, through the proposed Non-uniform Kolmogorov-Arnold Networks. Extensive comparative experiments demonstrate that it establishes a new benchmark in UnHIG.

Figures (3)

• Figure 1: (a) The conventional framework of paired HIG, which utilizes ground truth (GT) to build direct pixel-wise consistency. (b) Our proposed NukesFormer for unpaired HIG without GT, which establishes indirect constraint with cycle-consistency and dual-dimensional contrastive prior module (DCPM).
• Figure 2: The overall framework of proposed NukesFormer of decomposition stream, where the reconstruction stream in (a) derives from another NukesFormer, leveraging shared parameters. (b) DCPM utilizes the dual-dimensional contrastive prior to pull ($\xrightarrow{}\xleftarrow{}$) similar geometric and spectral distributions and push ($\xleftarrow{}\xrightarrow{}$) different components. (c) The non-uniform B-Spline matrix and G-MSA are dedicated to capturing multi-frequency information from spectral dimension.
• Figure 3: Visual Comparison Results. Line 1-2: the RMSE error map of 20-th band on three distinct validation images in NTIRE 2020 Clean. Line 3-4: the RMSE error map of 21-st band on three distinct validation images in NTIRE 2022. Line 5-6: the RMSE error map of 5-th band on two distinct validation images in CAVE.