White-Box mHC: Electromagnetic Spectrum-Aware and Interpretable Stream Interactions for Hyperspectral Image Classification

TL;DR

This work tackles the interpretability gap in hyperspectral image classification by introducing ES-mHC, a spectrum-aware, partially white-box extension of Hyper-Connections (mHC). By splitting the HSI cube into four physically meaningful spectral streams (VIS, NIR, SWIR1, SWIR2) and enforcing manifold-constrained, doubly stochastic interaction matrices, the model reveals coherent spatial patterns and asymmetric inter-stream dynamics that align with material properties across wavelengths. The approach combines cluster-wise sequence scanning and a spectral-spatial Mamba block to expand feature width while maintaining stability, achieving state-of-the-art results on Indian Pines and providing mechanistic insights via visualizations of the hyper-connection matrices. Overall, ES-mHC shifts HSIC from a purely black-box predictor toward a structurally transparent framework with interpretable internal information flow and tangible connections to physical spectral groups.

Abstract

In hyperspectral image classification (HSIC), most deep learning models rely on opaque spectral-spatial feature mixing, limiting their interpretability and hindering understanding of internal decision mechanisms. We present physical spectrum-aware white-box mHC, named ES-mHC, a hyper-connection framework that explicitly models interactions among different electromagnetic spectrum groupings (residual stream in mHC) interactions using structured, directional matrices. By separating feature representation from interaction structure, ES-mHC promotes electromagnetic spectrum grouping specialization, reduces redundancy, and exposes internal information flow that can be directly visualized and spatially analyzed. Using hyperspectral image classification as a representative testbed, we demonstrate that the learned hyper-connection matrices exhibit coherent spatial patterns and asymmetric interaction behaviors, providing mechanistic insight into the model internal dynamics. Furthermore, we find that increasing the expansion rate accelerates the emergence of structured interaction patterns. These results suggest that ES-mHC transforms HSIC from a purely black-box prediction task into a structurally transparent, partially white-box learning process.

Figures (9)

• Figure 1: Illustration of the (A) model overview, (B) stream matrix, and (C) the impact of the expansion rate on the emergence of spatial pattern.
• Figure 2: Illustration of cluster-wise Spatial Mamba block in layer function $\mathcal{F}$. Clustering effect is found in $\mathcal{H}^{\text{res}}$ and used for reducing the token and sequence length. Take the expansion rate $n=2$ as an example.
• Figure 3: Overview of hyperspectral imaging. (A) Graphical illustration of the electromagnetic spectrum. (B) Expanded view of the typical wavelength regions captured in HSI: visible light (400-750 nm), near-infrared (NIR, 750-1400 nm), and shortwave infrared (SWIR, 1400-2500 nm). This figure comes from hong2025hyperspectral.
• Figure 4: Illustration of the four electromagnetic spectrum–aware sub-cubes. (a) VIS, (b) NIR, (c) SWIR1, (d) SWIR2.
• Figure 5: The Indian Pines classification map generated by different methods. (a) SSRN (b) SS-ConvNeXt (c) MTGAN (d) SSFTT (e) SSTN (f) GSC-ViT (g) MammbaHSI (h) 3DSS-Mamba (i) ES-mHC (j) False Color Image (k) Ground Truth. Some red circles are shown on the RGB image to illustrate the boundary preservation of our proposed model.