Onboard Hyperspectral Super-Resolution with Deep Pushbroom Neural Network

TL;DR

This work tackles the onboard hyperspectral image super-resolution problem by aligning the neural network design with pushbroom sensor acquisition. The authors introduce Deep Pushbroom Super-Resolution (DPSR), a lightweight, line-by-line architecture that uses Selective State Space Models (Mamba blocks) to retain memory of past acquired lines, enabling causal, real-time super-resolution with low memory footprint. DPSR achieves competitive image quality (e.g., near-state-of-the-art PSNR/SSIM/SAM) while reducing FLOPs per pixel by orders of magnitude (e.g., $\sim$ $31\times 10^3$ FLOPs/px for $4\times$ on HySpecNet-11k) and maintaining sub-GB memory for large frames, making onboard deployment feasible. Extensive experiments across HySpecNet-11k, Chikusei, Houston, and Pavia demonstrate favorable efficiency-quality trade-offs; ablations underscore the critical role of the Mamba memory and the bilinear residual in maintaining performance. The work highlights the practical potential of processing hyperspectral data in-flight, enabling real-time analytics and responsive sensing with future hardware optimizations.

Abstract

Hyperspectral imagers on satellites obtain the fine spectral signatures essential for distinguishing one material from another at the expense of limited spatial resolution. Enhancing the latter is thus a desirable preprocessing step in order to further improve the detection capabilities offered by hyperspectral images on downstream tasks. At the same time, there is a growing interest towards deploying inference methods directly onboard of satellites, which calls for lightweight image super-resolution methods that can be run on the payload in real time. In this paper, we present a novel neural network design, called Deep Pushbroom Super-Resolution (DPSR) that matches the pushbroom acquisition of hyperspectral sensors by processing an image line by line in the along-track direction with a causal memory mechanism to exploit previously acquired lines. This design greatly limits memory requirements and computational complexity, achieving onboard real-time performance, i.e., the ability to super-resolve a line in the time it takes to acquire the next one, on low-power hardware. Experiments show that the quality of the super-resolved images is competitive or even outperforms state-of-the-art methods that are significantly more complex.

Figures (13)

• Figure 1: DPSR spatially super-resolves a hyperspectral image line-by-line, by using line $y-1$ and line $y$, as well as a compact memory of past lines to super-resolve line $y-1$ in the along- and across-track directions.
• Figure 2: DPSR processes a line with an initial SFE Block followed by two CLLF Blocks composed of a NAFBlock to exact across-track and spectral features and a Mamba Block to provide context from previous lines. The architecture works as a residual correction to bilinear upsampling.
• Figure 3: Left: architecture of SFE Block. Its main purpose is to filter the initial input in order to remove noise ad extract shallow features that will be later refined by the subsequent CLFF Block. Right: architecture of Upsampler module. It leverages the Pixel Shuffle shi2016real and two convolutions to efficiently manage complexity.
• Figure 4: Qualitative comparison of methods on HySpecNet-11k test image 247 with SR factor $4\times$. Bands 43, 28, 10 were used as R, G, B.
• Figure 5: Mean (across bands) error heatmaps of methods on HySpecNet-11k test image 247 with SR factor $4\times$.