DTU-Net: A Multi-Scale Dilated Transformer Network for Nonlinear Hyperspectral Unmixing

TL;DR

DTU-Net tackles nonlinear hyperspectral unmixing by integrating a two-branch encoder that jointly captures multi-scale spatial and spectral information with a PPNMM-based decoder that models both linear and nonlinear mixing. The encoder uses a Dilated Transformer with MSDA and SWDA alongside 3D-CNN–CBAM features, enabling robust endmember and abundance estimation, while the decoder explicitly learns endmembers and pixel-wise nonlinear coefficients as part of the physical mixing model. Across synthetic and real datasets, DTU-Net consistently outperforms PPNMM-derived and other deep unmixing networks, offering improved accuracy and interpretability, particularly in complex scenes with multiple endmembers and nonlinear interactions. This approach advances practical HU by blending advanced attention-based spatial modeling with physically grounded nonlinear mixing, enhancing robustness and applicability to diverse imaging conditions.

Abstract

Transformers have shown significant success in hyperspectral unmixing (HU). However, challenges remain. While multi-scale and long-range spatial correlations are essential in unmixing tasks, current Transformer-based unmixing networks, built on Vision Transformer (ViT) or Swin-Transformer, struggle to capture them effectively. Additionally, current Transformer-based unmixing networks rely on the linear mixing model, which lacks the flexibility to accommodate scenarios where nonlinear effects are significant. To address these limitations, we propose a multi-scale Dilated Transformer-based unmixing network for nonlinear HU (DTU-Net). The encoder employs two branches. The first one performs multi-scale spatial feature extraction using Multi-Scale Dilated Attention (MSDA) in the Dilated Transformer, which varies dilation rates across attention heads to capture long-range and multi-scale spatial correlations. The second one performs spectral feature extraction utilizing 3D-CNNs with channel attention. The outputs from both branches are then fused to integrate multi-scale spatial and spectral information, which is subsequently transformed to estimate the abundances. The decoder is designed to accommodate both linear and nonlinear mixing scenarios. Its interpretability is enhanced by explicitly modeling the relationships between endmembers, abundances, and nonlinear coefficients in accordance with the polynomial post-nonlinear mixing model (PPNMM). Experiments on synthetic and real datasets validate the effectiveness of the proposed DTU-Net compared to PPNMM-derived methods and several advanced unmixing networks.

Figures (13)

• Figure 1: Illustration of Multi-Scale Dilated Attention (MSDA) R27: (a) The MSDA mechanism; (b) Structure of an MSDA block.
• Figure 2: Overview of the proposed Multi-Scale Dilated Transformer Unmixing Network (DTU-Net). DTU-Net consists of two main components: the Dilated-Transformer encoder and the PPNMM-based decoder. The encoder extracts multi-scale spatial and spectral features from the hyperspectral image through two distinct branches, which are then fused to generate the abundance tensor. The decoder applies a 2D convolutional layer to upsample the abundance tensor, with the weight matrix representing the endmember matrix, corresponding to the linear mixing part. The nonlinear coefficient tensor is learned by concatenating and transforming relevant features. Finally, the hyperspectral image is reconstructed by combining the linear and nonlinear mixing components using learned features and network parameters, following the PPNMM mechanism.
• Figure 3: Variation of metrics with different values of $C$ evaluated on the PPNMM synthetic image with SNR = 30 dB: (a) $\rm RMSE_{abun}$ (b) $\rm SAD_{end}$.
• Figure 4: Estimated endmembers by different unmixing methods on a PPNMM-generated dataset with $R = 4$, SNR = 30 dB.
• Figure 5: Abundance maps estimation results on a PPNMM-generated dataset with $R = 4$, SNR = 30 dB. From left to right: Ground Truth, PPNMM, CyCU-Net, DeepTrans, Swin-HU, UST-Net, PPNMM-AE, and the proposed DTU-Net.