Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D-Convolution Guided Spectral-Spatial Transformer for Hyperspectral Image Classification

Shyam Varahagiri, Aryaman Sinha, Shiv Ram Dubey, Satish Kumar Singh

TL;DR

The paper tackles hyperspectral image classification by integrating CNN-derived spatial-spectral fusion with Vision Transformer architectures. It introduces a 3D-Convolution Guided Residual Module (CGRM) that fuses information across Transformer encoder blocks and replaces the CLS token with Global Average Pooling for final classification. Across Houston, MUUFL, and Botswana, the method consistently outperforms traditional classifiers, CNNs, RNNs, and state-of-the-art Transformer baselines, demonstrating the value of explicit spatial-spectral fusion in HSIs. The work provides reproducible code and insights into how encoder depth affects performance, suggesting dataset-dependent depth choices for optimal results.

Abstract

In recent years, Vision Transformers (ViTs) have shown promising classification performance over Convolutional Neural Networks (CNNs) due to their self-attention mechanism. Many researchers have incorporated ViTs for Hyperspectral Image (HSI) classification. HSIs are characterised by narrow contiguous spectral bands, providing rich spectral data. Although ViTs excel with sequential data, they cannot extract spectral-spatial information like CNNs. Furthermore, to have high classification performance, there should be a strong interaction between the HSI token and the class (CLS) token. To solve these issues, we propose a 3D-Convolution guided Spectral-Spatial Transformer (3D-ConvSST) for HSI classification that utilizes a 3D-Convolution Guided Residual Module (CGRM) in-between encoders to "fuse" the local spatial and spectral information and to enhance the feature propagation. Furthermore, we forego the class token and instead apply Global Average Pooling, which effectively encodes more discriminative and pertinent high-level features for classification. Extensive experiments have been conducted on three public HSI datasets to show the superiority of the proposed model over state-of-the-art traditional, convolutional, and Transformer models. The code is available at https://github.com/ShyamVarahagiri/3D-ConvSST.

3D-Convolution Guided Spectral-Spatial Transformer for Hyperspectral Image Classification

TL;DR

The paper tackles hyperspectral image classification by integrating CNN-derived spatial-spectral fusion with Vision Transformer architectures. It introduces a 3D-Convolution Guided Residual Module (CGRM) that fuses information across Transformer encoder blocks and replaces the CLS token with Global Average Pooling for final classification. Across Houston, MUUFL, and Botswana, the method consistently outperforms traditional classifiers, CNNs, RNNs, and state-of-the-art Transformer baselines, demonstrating the value of explicit spatial-spectral fusion in HSIs. The work provides reproducible code and insights into how encoder depth affects performance, suggesting dataset-dependent depth choices for optimal results.

Abstract

In recent years, Vision Transformers (ViTs) have shown promising classification performance over Convolutional Neural Networks (CNNs) due to their self-attention mechanism. Many researchers have incorporated ViTs for Hyperspectral Image (HSI) classification. HSIs are characterised by narrow contiguous spectral bands, providing rich spectral data. Although ViTs excel with sequential data, they cannot extract spectral-spatial information like CNNs. Furthermore, to have high classification performance, there should be a strong interaction between the HSI token and the class (CLS) token. To solve these issues, we propose a 3D-Convolution guided Spectral-Spatial Transformer (3D-ConvSST) for HSI classification that utilizes a 3D-Convolution Guided Residual Module (CGRM) in-between encoders to "fuse" the local spatial and spectral information and to enhance the feature propagation. Furthermore, we forego the class token and instead apply Global Average Pooling, which effectively encodes more discriminative and pertinent high-level features for classification. Extensive experiments have been conducted on three public HSI datasets to show the superiority of the proposed model over state-of-the-art traditional, convolutional, and Transformer models. The code is available at https://github.com/ShyamVarahagiri/3D-ConvSST.
Paper Structure (14 sections, 5 equations, 5 figures, 5 tables)

This paper contains 14 sections, 5 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Proposed 3D-ConvSST Architecture
  3. Convolutional Networks for Feature Extraction
  4. Tokenization and Positional Embedding
  5. Vision Transformer Encoder Module
  6. 3D-Convolution Guided Residual Module (3D-CGRM)
  7. Global Average Pool Classification
  8. Experimental Settings
  9. Experimental Results and Analysis
  10. Experimental Quantitative Results
  11. Visual Comparison
  12. Performance over Different Encoder Depths
  13. Ablation Analysis
  14. Conclusion

Figures (5)

  • Figure 1: Overall framework of the 3D-ConvSST Architecture (left) and the 3D-Convolution Guided Residual Module (right)
  • Figure 2: Visualization maps for the University of Houston (UH) HSI Dataset
  • Figure 3: Visualization maps for the MUUFL HSI Dataset
  • Figure 4: Visualization maps for the Botswana HSI Dataset
  • Figure 5: Effect of the 3D-ConvSST encoder depth.