Spectral Discrepancy and Cross-modal Semantic Consistency Learning for Object Detection in Hyperspectral Image

TL;DR

Spectral Discrepancy and Cross-modal Semantic Consistency Learning (SDCM) addresses hyperspectral object detection by mitigating inter-band heterogeneity and leveraging spectral information. It introduces a three-module framework: Semantic Consistency Learning (SCL) to align inter-band spatial cues, Spectral Gated Generator (SGG) to filter redundant spectral data, and Spectral Discrepancy Aware (SDA) to inject spectral guidance into high-level features. The model uses two-stream encoders on band-selected visible and infrared bands, with cross-modal attention and a spectral decoder to predict bounding boxes and classes. Extensive experiments on HOD-1 and HOD3K show state-of-the-art performance, with notable gains in camouflaged and densely packed scenes. The work highlights the value of integrating spatial-spectral fusion and band-aware processing for robust hyperspectral detection.

Abstract

Hyperspectral images with high spectral resolution provide new insights into recognizing subtle differences in similar substances. However, object detection in hyperspectral images faces significant challenges in intra- and inter-class similarity due to the spatial differences in hyperspectral inter-bands and unavoidable interferences, e.g., sensor noises and illumination. To alleviate the hyperspectral inter-bands inconsistencies and redundancy, we propose a novel network termed \textbf{S}pectral \textbf{D}iscrepancy and \textbf{C}ross-\textbf{M}odal semantic consistency learning (SDCM), which facilitates the extraction of consistent information across a wide range of hyperspectral bands while utilizing the spectral dimension to pinpoint regions of interest. Specifically, we leverage a semantic consistency learning (SCL) module that utilizes inter-band contextual cues to diminish the heterogeneity of information among bands, yielding highly coherent spectral dimension representations. On the other hand, we incorporate a spectral gated generator (SGG) into the framework that filters out the redundant data inherent in hyperspectral information based on the importance of the bands. Then, we design the spectral discrepancy aware (SDA) module to enrich the semantic representation of high-level information by extracting pixel-level spectral features. Extensive experiments on two hyperspectral datasets demonstrate that our proposed method achieves state-of-the-art performance when compared with other ones.

Figures (10)

• Figure 1: Visualization of differences between bands in four group of images from four corresponding hyperspectral datasets: (a) Indian Pian (b) Pavia University (c) Sandiego (d) HOD-1 dataset. We select a portion of the bands in a fixed ratio to display, covering the visible wavelength range to the infrared wavelength range. The difference in band information increases as the band distance increase.
• Figure 2: Visualization of sample images in HOD-1 and HOD3K datasets. To better show the hyperspectral images, we transformed the hyperspectral images into pseudo-color images for display.
• Figure 3: An Overview of Spectral Discrepancy and Cross-modal semantic consistency learning (SDCM). Given a hyperspectral image, we filter out representative visible and infrared (IR) channels by band selection filters and provide the SCL module to mitigate inter-band heterogeneity. The spectral discrepancy aware module accepts input images from all channels and generates spectral information. Next, the spectral information is categorically characterized by cross-attentive guidance of high-level ($\boldsymbol{s}_5$) semantic information using the Transformer decoder. The SGG module is embedded after feature extraction to eliminate redundant information. Lastly, semantically consistent low-level features and refined high-level features are combined and predicted by FFN (feed-forward network) for bounding boxes and category projections.
• Figure 4: Illustration of the Semantic Consistency Learning (SCL) module.
• Figure 5: Illustration of the Spectral Gated Generator (SGG) module. The weights are obtained based on the assumption that all pixels in a single band follow the same distribution.