Spectral Super-Resolution Neural Operator with Atmospheric Radiative Transfer Prior

TL;DR

This work tackles the ill-posed problem of spectral super-resolution by embedding atmospheric radiative transfer priors into a neural-operator framework. The proposed SSRNO uses a three-stage pipeline—ART-guided upsampling via Guidance Matrix Projection, a resolution-invariant Reconstruction Neural Operator with spectral-aware convolution, and a Refinement step enforcing a hard $S Y = X$ constraint—to enable continuous spectral reconstruction and zero-shot extrapolation. A theoretical optimality guarantee is provided for the GMP method, and experiments on AVIRIS-derived data show state-of-the-art accuracy and strong generalization across sites. The approach advances practical remote sensing by delivering physically plausible spectra and arbitrary-scale outputs, integrating domain knowledge with deep learning for robust SSR.

Abstract

Spectral super-resolution (SSR) aims to reconstruct hyperspectral images (HSIs) from multispectral observations, with broad applications in remote sensing. Data-driven methods are widely used, but they often overlook physical principles, leading to unrealistic spectra, particularly in atmosphere-affected bands. To address this challenge, we propose the Spectral Super-Resolution Neural Operator (SSRNO), which incorporates atmospheric radiative transfer (ART) prior into the data-driven procedure, yielding more physically consistent predictions. The proposed SSRNO framework consists of three stages: upsampling, reconstruction, and refinement. In the upsampling stage, we leverage prior information to expand the input multispectral image, producing a physically plausible hyperspectral estimate. Subsequently, we utilize a neural operator in the reconstruction stage to learn a continuous mapping across the spectral domain. Finally, the refinement stage imposes a hard constraint on the output HSI to eliminate color distortion. The upsampling and refinement stages are implemented via the proposed guidance matrix projection (GMP) method, and the reconstruction neural operator adopts U-shaped spectral-aware convolution (SAC) layers to capture multi-scale features. Moreover, we theoretically demonstrate the optimality of the GMP method. With the neural operator and ART priors, SSRNO also achieves continuous spectral reconstruction and zero-shot extrapolation. Various experiments validate the effectiveness and generalization ability of the proposed approach.

Figures (7)

• Figure 1: Comparison to the extraterrestrial irradiance $E_{on\lambda}$ and the direct beam radiation $E_{bn\lambda}$, where $E_{on\lambda}$ is based on the Air Mass 1.5 spectra and $E_{bn\lambda}$ is based on the 1976 U.S. Standard Atmosphere.
• Figure 2: Overall framework of the proposed method. The input MSI undergoes three main stages to output the final HSI $\hat{Y}$: (1) Upsampling, where we interpolate and extrapolate the input MSI to the upsampled HSI, (2) Reconstruction, where we employ the neural operator to transform the upsampled HSI to a finer HSI estimate, and (3) Refinement, where we impose a hard constraint on the final result to enforce zero reconstruction error and eliminate color distortion.
• Figure 3: Overview of the upsampling, reconstruction, and refinement stages. The upsampling and refinement stages both employ the GMP method, but with different objectives. In the upsampling stage, GMP incorporates prior knowledge to produce a physically plausible HSI estimate, whereas in the refinement stage, it projects the HSI onto the affine space defined by \ref{['eq:SY=X']}. The reconstruction stage employs a neural operator with a U-shaped structure and spectral-aware convolution layers to learn the mapping from upsampled HSIs to target HSIs across continuous spectral dimensions.
• Figure 4: Architecture of the spectral-aware convolution layer. The input $\boldsymbol{v}_t$ goes through two parallel streams: Fourier domain learning and spatial-spectral domain learning. Two outputs are added up, passed through the activation function, and result in the output $v_{t+1}$.
• Figure 5: Visual comparison of pseudo-RGB reconstruction results across different sites. Each row presents the pseudo-RGB images of one spectral reconstruction method applied to one scene from six different sites, while each column compares the pseudo-RGB images of various methods on the same scene within a single site. The MRAE reconstruction error is also calculated and presented on top of the image.

Theorems & Definitions (2)

• Theorem 1
• proof