EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution

TL;DR

This work tackles the data scarcity barrier in single-HSI-SR by bridging pre-trained RGB models to hyperspectral imagery through eigenimage-based spectral-spatial decoupling. It introduces EigenSR, a two-stage framework that first fine-tunes a pre-trained RGB model on spatial eigenimages and then performs SR in the eigenimage domain, reconstructing HSIs with the spectral basis and applying iterative spectral regularization to preserve spectral correlations. The approach leverages low-rank eigenimages to enable channel-wise SR with a spectral reconstruction step, achieving competitive or superior results to state-of-the-art methods across both spatial and spectral metrics and demonstrating robustness to spectral-agnostic unseen data. Comprehensive ablations show the benefits of eigenimage fine-tuning, the necessity of spectral constraints, and the trade-offs between inference time and fidelity, supported by experiments on multiple HSI datasets and remote sensing benchmarks.

Abstract

Single hyperspectral image super-resolution (single-HSI-SR) aims to improve the resolution of a single input low-resolution HSI. Due to the bottleneck of data scarcity, the development of single-HSI-SR lags far behind that of RGB natural images. In recent years, research on RGB SR has shown that models pre-trained on large-scale benchmark datasets can greatly improve performance on unseen data, which may stand as a remedy for HSI. But how can we transfer the pre-trained RGB model to HSI, to overcome the data-scarcity bottleneck? Because of the significant difference in the channels between the pre-trained RGB model and the HSI, the model cannot focus on the correlation along the spectral dimension, thus limiting its ability to utilize on HSI. Inspired by the HSI spatial-spectral decoupling, we propose a new framework that first fine-tunes the pre-trained model with the spatial components (known as eigenimages), and then infers on unseen HSI using an iterative spectral regularization (ISR) to maintain the spectral correlation. The advantages of our method lie in: 1) we effectively inject the spatial texture processing capabilities of the pre-trained RGB model into HSI while keeping spectral fidelity, 2) learning in the spectral-decorrelated domain can improve the generalizability to spectral-agnostic data, and 3) our inference in the eigenimage domain naturally exploits the spectral low-rank property of HSI, thereby reducing the complexity. This work bridges the gap between pre-trained RGB models and HSI via eigenimages, addressing the issue of limited HSI training data, hence the name EigenSR. Extensive experiments show that EigenSR outperforms the state-of-the-art (SOTA) methods in both spatial and spectral metrics.

Figures (9)

• Figure 1: The motivation. The lack of HSI training data is the main bottleneck for single-HSI-SR. The channel differences between RGB and HSI make it challenging to directly transfer RGB models to HSI. Inspired by spectral-spatial decoupling, we leverage spatial part of HSI, namely eigenimages, to bridge the gap between pre-trained RGB models and HSI, and constrain the spectral part to keep spectral fidelity.
• Figure 2: The flowchart of EigenSR. It consists of two stages. Stage 1 uses the pre-trained Transformer Body to fine-tune a single-channel model in the eigenimage domain. Stage 2 utilizes the fine-tuned model for inference with iterative spectral regularization on unseen LR HSI.
• Figure 3: Absolute spectral basis of the CAVE-beads image at different downsampling rates. (a) $| \mathbf{U}_{:, 1} |$ and (b) $| \mathbf{U}_{:, 2} |$.
• Figure 4: Visual results of SR $\times 4$. (a) ARAD_1K-Test. (b) CAVE. (c) Harvard. (d) DC Mall. We used band numbers (27, 14, 6) to generate the pseudo-color images for ARAD_1K, CAVE, and Harvard datasets, and band numbers (32, 14, 7) for DC Mall.
• Figure 5: SR $\times 4$ comparison in the spectral dimension on CAVE dataset. We denote the spatial coordinate as p(151, 321). Our method can provide more accurate spectral curves.