Scalable neural pushbroom architectures for real-time denoising of hyperspectral images onboard satellites

TL;DR

This work tackles real-time onboard denoising of hyperspectral pushbroom imagery under strict power and fault-tolerance constraints. It introduces a line-wise, causal denoising framework built as a mixture of $D$ lightweight denoisers, each on an independent accelerator, with fault-filtered aggregation via self-attention and a memory-enabled 1-D U-Net based on the Mamba Selective State-Space Model. A training strategy with a denoiser-subset sampling factor $\lambda$ enables flexible power scalability, while a fault-detection mechanism based on the variance of attention scores provides intrinsic resilience to radiation-induced faults. Experiments show the approach can operate in real time on low-power hardware (e.g., line time $t_{line}=4.34$ ms) with competitive denoising quality and robust fault-tolerance, highlighting practical impact for autonomous onboard processing in future Earth observation missions.

Abstract

The next generation of Earth observation satellites will seek to deploy intelligent models directly onboard the payload in order to minimize the latency incurred by the transmission and processing chain of the ground segment, for time-critical applications. Designing neural architectures for onboard execution, particularly for satellite-based hyperspectral imagers, poses novel challenges due to the unique constraints of this environment and imaging system that are largely unexplored by the traditional computer vision literature. In this paper, we show that this setting requires addressing three competing objectives, namely high-quality inference with low complexity, dynamic power scalability and fault tolerance. We focus on the problem of hyperspectral image denoising, which is a critical task to enable effective downstream inference, and highlights the constraints of the onboard processing scenario. We propose a neural network design that addresses the three aforementioned objectives with several novel contributions. In particular, we propose a mixture of denoisers that can be resilient to radiation-induced faults as well as allowing for time-varying power scaling. Moreover, each denoiser employs an innovative architecture where an image is processed line-by-line in a causal way, with a memory of past lines, in order to match the acquisition process of pushbroom hyperspectral sensors and greatly limit memory requirements. We show that the proposed architecture can run in real-time, i.e., process one line in the time it takes to acquire the next one, on low-power hardware and provide competitive denoising quality with respect to significantly more complex state-of-the-art models. We also show that the power scalability and fault tolerance objectives provide a design space with multiple tradeoffs between those properties and denoising quality.

Figures (13)

• Figure 1: The acquired line gets processed by each independent line-based denoiser, which makes use of a compact memory of past lines to refine it. The different denoised lines are then aggregated to form the output.
• Figure 2: Proposed design. a) overall architecture with a mixture of $D$ denoisers running in parallel: each denoiser is placed on an independent hardware accelerator that can be turned on/off for power scaling; the output features are then checked for faults and aggregated with an attention mechanism. b) individual denoiser architecture, which is a 1-D UNet processing a line with a memory of past lines, consisting of 1-D DASC Blocks chen2022simple and 1-D PA-SSM Blocks (c)).
• Figure 3: Runtime on Nvidia Jetson Orin Nano normalized for a $1 \times 1000 \times 66$ line. 4.34ms is the Line Acquisition Time of the PRISMA satellite.
• Figure 4: Denoising results on the Houston 2018 HSI with mixture noise. The false-color images are generated by combining bands 35th, 20th, and 5th.
• Figure 5: Denoising results on the Pavia city center with mixture noise, bands 65,45,25.