SpectraMorph: Structured Latent Learning for Self-Supervised Hyperspectral Super-Resolution

TL;DR

SpectraMorph tackles hyperspectral super-resolution by reframing fusion as a structured unmixing problem. It builds a fixed endmember dictionary ${\mathbf E}$ from LR-HSI via NMF and uses a lightweight Latent Estimation Network to produce abundance-like activations ${\mathbf A}$ from MSI inputs, reconstructing HR-HSI through linear mixing ${\mathbf X} \approx {\mathbf A}{\mathbf E}$ in a self-supervised LR setting. A coarse spectral prior (CSP) stabilizes PAN-only inversion, while interpretability is preserved through the physically meaningful dictionary and latent maps; the model achieves strong results with training times under a minute and robust performance against misalignment and SRF uncertainty. Experiments on synthetic and real data show SpectraMorph outperforming most unsupervised baselines and remaining highly competitive with supervised methods, with notable gains in interpretability and efficiency. The approach enables practical, real-time HSR with a principled balance between physics-based structure and learnable flexibility.

Abstract

Hyperspectral sensors capture dense spectra per pixel but suffer from low spatial resolution, causing blurred boundaries and mixed-pixel effects. Co-registered companion sensors such as multispectral, RGB, or panchromatic cameras provide high-resolution spatial detail, motivating hyperspectral super-resolution through the fusion of hyperspectral and multispectral images (HSI-MSI). Existing deep learning based methods achieve strong performance but rely on opaque regressors that lack interpretability and often fail when the MSI has very few bands. We propose SpectraMorph, a physics-guided self-supervised fusion framework with a structured latent space. Instead of direct regression, SpectraMorph enforces an unmixing bottleneck: endmember signatures are extracted from the low-resolution HSI, and a compact multilayer perceptron predicts abundance-like maps from the MSI. Spectra are reconstructed by linear mixing, with training performed in a self-supervised manner via the MSI sensor's spectral response function. SpectraMorph produces interpretable intermediates, trains in under a minute, and remains robust even with a single-band (pan-chromatic) MSI. Experiments on synthetic and real-world datasets show SpectraMorph consistently outperforming state-of-the-art unsupervised/self-supervised baselines while remaining very competitive against supervised baselines.

Figures (5)

• Figure 1: SpectraMorph pipeline: the latent estimation network (LEN) produces a abundance-like latent estimate (ALLE) from a MSI pixel that is combined with a set of endmembers obtained from NMF (gray box) to estimate the corresponding HSI pixel. Training occurs at low resolution using a synthesized LR-MSI and the source LR-HSI. During inference, the same HSI pixel estimation process (orange box) is applied to the HR-MSI to obtain a HR-HSI estimate. Endmember dictionary is enlarged in the gray box.
• Figure 2: Endmember Signatures ${\mathbf E}$ obtained from the LR-HSI and Abundance-like latent estimate (ALLE) maps ${\mathbf A}$ obtained from the HR-MSI during inference on Washington DC Mall.
• Figure 3: The coarse spectral prior replicates the spectra of the LR-HSI ${\mathbf Y}$ to obtain a CSP ${\mathbf P}_{\mathbf Y}$ that serves as side information for inference from a panchromatic image. Here, ${\mathbf Y} \approx {\mathcal{D}}_2\bigl(\mathcal{H}({\mathbf X}, {\mathbf Q})\bigr)$ ($r=2$), hence the spectra of each pixel of ${\mathbf Y}$ is replicated $2\times2$ times to obtain ${\mathbf P}_Y$.
• Figure 4: Point Spread Functions used for Synthetic LR HSI generation
• Figure 5: UH SR results and corresponding spectra for two test scenes. (a, c) Super-resolved images with zoomed-in crops. (b, d) Spectral plots for selected regions.