Spectral Super-Resolution via Adversarial Unfolding and Data-Driven Spectrum Regularization: From Multispectral Satellite Data to NASA Hyperspectral Image

TL;DR

This study aims to achieve spectral super-resolution from 12-to-186 and unify the spatial resolution of Sentinel-2 data to 5 m and formulate a novel deep unfolding framework regularized by a data-driven spectrum prior from PriorNet, instead of relying on implicit deep priors as conventional deep unfolding does.

Abstract

The European Space Agency's Sentinel-2 satellite provides global multispectral coverage for remote sensing (RS) applications. However, limited spectral resolution (12 bands) and non-unified spatial resolution (60/20/10 m) restrict their practicality. In contrast, the high spectral-spatial resolution sensor (e.g., NASA's AVIRIS-NG) covers only the American region due to practical considerations. This raises a fundamental question: ``Can a global hyperspectral coverage be achieved by reconstructing Sentinel-2 data to NASA hyperspectral images?'' This study aims to achieve spectral super-resolution from 12-to-186 and unify the spatial resolution of Sentinel-2 data to 5 m. To enable a reliable and efficient reconstruction, we formulate a novel deep unfolding framework regularized by a data-driven spectrum prior from PriorNet, instead of relying on implicit deep priors as conventional deep unfolding does. Moreover, an adversarial term is integrated into the unfolded architecture, enabling the discriminator to guide the reconstruction in both the training and testing phases; we term this novel concept unfolding adversarial learning (UAL). Experiments show that our UALNet outperforms the next-best Transformer in PSNR, SSIM, and SAM, while requiring only 15% MACs and 20 times fewer parameters. The associated code will be publicly available at https://sites.google.com/view/chiahsianglin/software.

Figures (11)

• Figure 1: Performance-Params-MACs comparisons with spectral/spatial reconstruction models. The horizontal axis is computational complexity (measured in MACs ), the vertical axis indicates performance [reported as PSNR-over-SAM ratio $(\uparrow)$ to consider both spatial and spectral fidelities], while the circle radius corresponds to the network parameters (memory cost). When the performance stems from sophisticated architecture and model depth, it results in prohibitive computational complexity and parameters. Conversely, the proposed unfolding adversarial learning network (UALNet) achieves the highest performance with substantially lower MACs and Params with the explainable architecture.
• Figure 2: Visual comparison of the reference spectral cross-similarity matrix ${\bm A}{\bm A}^T$ and the spectral prior matrix ${\bm P}$ learned by the proposed PriorNet (see Supplementary Figure 1).
• Figure 3: The schematic pipeline of the proposed UALNet for the challenging Sentinel-2 ${\bm S}$ to AVIRIS-level HSI ${\bm A}$ transformation. To fulfill this goal, we first develop an efficient PriorNet (see Supplementary Figure 1) to provide the 5 m GSD spatial prior image ${\bm S}_u$ from the target multiresolution MSI ${\bm S}$, together with a spectral prior matrix ${\bm P}\approx{\bm A}{\bm A}^T$ that encodes the spectral cross-correlations. Subsequently, the adversarial learning process is formulated as a discriminator-maximization term (see Section \ref{['subsec: Criterion']}) and is solved by the Qusai-SB optimization framework (see Section \ref{['subsec: Quasi-ADMM']}). By unfolding the iterative Qusai-SB procedure (see Section \ref{['subsec: Unfolding']}), the proposed unfolding adversarial network (UALNet) can achieve an explainable architecture. Due to the space limitation, the details of the sub-modules are illustrated in the Supplementary Material.
• Figure 4: Comparison of spatial calibration between the 10 m GSD Sentinel-2 MSI and the 5 m GSD AVIRIS HSI in true-color composition.
• Figure 5: Qualitative comparisons between the estimated results and the corresponding GT, shown in true-color compositions (left) and the spectral signatures (right). The ROI is located near Okmulgee County, Eastern Oklahoma, USA, and was captured on Oct. 27, 2019.