SpectralAdapt: Semi-Supervised Domain Adaptation with Spectral Priors for Human-Centered Hyperspectral Image Reconstruction

TL;DR

This work tackles the challenge of reconstructing hyperspectral images for human-centered healthcare under limited labeled data. It introduces SpectralAdapt, an SSDA framework that combines Spectral Density Masking (SDM) and Spectral Endmember Representation Alignment (SERA) within a Mean Teacher setup to align cross-domain spectral representations from general to human-centric data. SDM adaptively masks spectrally complex RGB regions to enhance inter-channel learning, while SERA anchors predictions to a dynamic bank of physically meaningful endmembers learned from labeled data. Across cross-domain reconstruction and downstream medical segmentation tasks, SpectralAdapt achieves superior spectral fidelity, better cross-domain generalization, and improved training stability, highlighting the practicality of SSDA for healthcare-focused HSI applications.

Abstract

Hyperspectral imaging (HSI) holds great potential for healthcare due to its rich spectral information. However, acquiring HSI data remains costly and technically demanding. Hyperspectral image reconstruction offers a practical solution by recovering HSI data from accessible modalities, such as RGB. While general domain datasets are abundant, the scarcity of human HSI data limits progress in medical applications. To tackle this, we propose SpectralAdapt, a semi-supervised domain adaptation (SSDA) framework that bridges the domain gap between general and human-centered HSI datasets. To fully exploit limited labels and abundant unlabeled data, we enhance spectral reasoning by introducing Spectral Density Masking (SDM), which adaptively masks RGB channels based on their spectral complexity, encouraging recovery of informative regions from complementary cues during consistency training. Furthermore, we introduce Spectral Endmember Representation Alignment (SERA), which derives physically interpretable endmembers from valuable labeled pixels and employs them as domain-invariant anchors to guide unlabeled predictions, with momentum updates ensuring adaptability and stability. These components are seamlessly integrated into SpectralAdapt, a spectral prior-guided framework that effectively mitigates domain shift, spectral degradation, and data scarcity in HSI reconstruction. Experiments on benchmark datasets demonstrate consistent improvements in spectral fidelity, cross-domain generalization, and training stability, highlighting the promise of SSDA as an efficient solution for hyperspectral imaging in healthcare.

Figures (7)

• Figure 1: Illustration of the domain adaptation scenario in HSI reconstruction under the SSDA paradigm. (a) Illustration of distribution shift between the source domain ($\mathcal{D}_S$) and target domain ($\mathcal{D}_T$), primarily reflecting covariate shift with similar conditional distributions. The source domain is fully labeled, while the target domain contains only a few labeled samples. (b) t-SNE visualization of features before and after adaptation. Our method aligns features across source (NTIRE2022) arad2022ntire and target (Hyper-Skin ng2024hyper) domain features using both labeled and unlabeled data.
• Figure 2: Overview of our proposed method. $\zeta$(weak) and $\zeta'$(strong) denote two stochastic augmentations applied to the same input before the student and teacher, $\theta$, $\theta'$ represents student and teacher model parameters respectively, where student and teacher model share the same architecture and $\theta'$ is updated through the Exponential Moving Average (EMA) of $\theta$. Spectral density masking (SDM) adaptively masks RGB regions based on spectral complexity to enhance spectral reasoning, and the spectral endmember representation alignment (SERA) aligns predictions with physically meaningful endmembers to improve domain transferability.
• Figure 3: Spectral Density Masking Illustration. We visualize three sampled regions from an RGB image (top left) and their corresponding reflectance spectra from the ground truth hyperspectral image (bottom left). The spectral bands are across the 400–700 nm range and color-coded into blue (400–500 nm), green (510–580 nm), and red (600–700 nm). Notably, higher angular deviations in the red region indicate greater spectral complexity, which motivates our Spectral Density Masking (SDM) strategy.
• Figure 4: Visualization of the downstream evaluation pipeline, which comprises three branches: RGB-based segmentation, HSI-based segmentation, and segmentation based on reconstructed HSI from RGB inputs. All branches utilize a shared segmentation backbone and generate pixel-wise predictions.
• Figure 5: Qualitative comparison of segmentation results on the Choledoch and HeiPorSPECTRAL datasets using three input modalities: RGB, reconstructed HSI (RS-HSI), and raw HSI. RS-HSI exhibits improved boundary accuracy and spatial consistency over RGB, closely matching the performance of raw HSI. Different colors in the prediction maps denote different semantic classes for each dataset.