LiDAR-Guided Cross-Attention Fusion for Hyperspectral Band Selection and Image Classification

TL;DR

This work addresses the high dimensionality and redundancy of hyperspectral data by introducing a LiDAR-guided cross-attention mechanism to select informative HSI bands for fusion with LiDAR. The method uses a transformer-inspired architecture with self-attention on each modality and a cross-attention step where LiDAR queries weight HSI bands, enabling end-to-end band selection and classification. Across three paired datasets (Houston 2013, Trento, MUUFL), the approach consistently outperforms traditional band-selection baselines and full-band fusion models, particularly when data augmentation is used, achieving state-of-the-art accuracy with a reduced spectral footprint. The results suggest practical benefits for real-time remote sensing applications by reducing data volume without sacrificing, and often enhancing, classification performance.

Abstract

The fusion of hyperspectral and LiDAR data has been an active research topic. Existing fusion methods have ignored the high-dimensionality and redundancy challenges in hyperspectral images, despite that band selection methods have been intensively studied for hyperspectral image (HSI) processing. This paper addresses this significant gap by introducing a cross-attention mechanism from the transformer architecture for the selection of HSI bands guided by LiDAR data. LiDAR provides high-resolution vertical structural information, which can be useful in distinguishing different types of land cover that may have similar spectral signatures but different structural profiles. In our approach, the LiDAR data are used as the "query" to search and identify the "key" from the HSI to choose the most pertinent bands for LiDAR. This method ensures that the selected HSI bands drastically reduce redundancy and computational requirements while working optimally with the LiDAR data. Extensive experiments have been undertaken on three paired HSI and LiDAR data sets: Houston 2013, Trento and MUUFL. The results highlight the superiority of the cross-attention mechanism, underlining the enhanced classification accuracy of the identified HSI bands when fused with the LiDAR features. The results also show that the use of fewer bands combined with LiDAR surpasses the performance of state-of-the-art fusion models.

Figures (10)

• Figure 1: Correlation coefficients between LiDAR and each hyperspectral band of Three Datasets. (a) Houston2013 HSI and LiDAR. (b) Trento HSI and LiDAR. (c) MUUFLE HSI and LIDAR. The horizontal axis shows the band index.
• Figure 2: The overall framework of the proposed LiDAR-guided cross-attention fusion for band selection.
• Figure 3: The detailed illustration of the the Proposed LiDAR-guided cross-attention fusion for band selection method. The network consists of four components: A) input and patch embedding module; B) self-attention module; C)cross-attention module; and D) MLP.
• Figure 4: Cross attention from LiDAR and HSI: LiDAR queries undergo linear transformation, interact with hyperspectral keys/values through scaled dot-product attention, concatenate across multiple heads, followed by a final linear projection for enriched data fusion.
• Figure 5: Visualisation of the Houston 2013 data set. Top: HSI; Middle: LiDAR data; Bottom: ground truth.