Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coarse-Fine Spectral-Aware Deformable Convolution For Hyperspectral Image Reconstruction

Jincheng Yang, Lishun Wang, Miao Cao, Huan Wang, Yinping Zhao, Xin Yuan

TL;DR

This work tackles reconstructing 3D hyperspectral images from 2D CASSI measurements, addressing the limitations of CNNs in capturing long-range dependencies and the computational burden of Transformers. It introduces CFSDCN, a CNN-based architecture that combines grouped deformable convolutions for spatial non-locality with a Coarse-Fine Spectral-Aware Module to model both coarse spatial-spectral and fine spectral similarities. The key contributions are applying deformable convolutions to HSI-CASSI reconstruction for the first time, designing the Coarse-Fine Spectral-Aware Block, and demonstrating state-of-the-art performance with lower computational cost on simulated and real datasets. The approach yields accurate, perceptually pleasing HSIs and offers a practical, efficient solution for snapshot spectral imaging applications.

Abstract

We study the inverse problem of Coded Aperture Snapshot Spectral Imaging (CASSI), which captures a spatial-spectral data cube using snapshot 2D measurements and uses algorithms to reconstruct 3D hyperspectral images (HSI). However, current methods based on Convolutional Neural Networks (CNNs) struggle to capture long-range dependencies and non-local similarities. The recently popular Transformer-based methods are poorly deployed on downstream tasks due to the high computational cost caused by self-attention. In this paper, we propose Coarse-Fine Spectral-Aware Deformable Convolution Network (CFSDCN), applying deformable convolutional networks (DCN) to this task for the first time. Considering the sparsity of HSI, we design a deformable convolution module that exploits its deformability to capture long-range dependencies and non-local similarities. In addition, we propose a new spectral information interaction module that considers both coarse-grained and fine-grained spectral similarities. Extensive experiments demonstrate that our CFSDCN significantly outperforms previous state-of-the-art (SOTA) methods on both simulated and real HSI datasets.

Coarse-Fine Spectral-Aware Deformable Convolution For Hyperspectral Image Reconstruction

TL;DR

This work tackles reconstructing 3D hyperspectral images from 2D CASSI measurements, addressing the limitations of CNNs in capturing long-range dependencies and the computational burden of Transformers. It introduces CFSDCN, a CNN-based architecture that combines grouped deformable convolutions for spatial non-locality with a Coarse-Fine Spectral-Aware Module to model both coarse spatial-spectral and fine spectral similarities. The key contributions are applying deformable convolutions to HSI-CASSI reconstruction for the first time, designing the Coarse-Fine Spectral-Aware Block, and demonstrating state-of-the-art performance with lower computational cost on simulated and real datasets. The approach yields accurate, perceptually pleasing HSIs and offers a practical, efficient solution for snapshot spectral imaging applications.

Abstract

We study the inverse problem of Coded Aperture Snapshot Spectral Imaging (CASSI), which captures a spatial-spectral data cube using snapshot 2D measurements and uses algorithms to reconstruct 3D hyperspectral images (HSI). However, current methods based on Convolutional Neural Networks (CNNs) struggle to capture long-range dependencies and non-local similarities. The recently popular Transformer-based methods are poorly deployed on downstream tasks due to the high computational cost caused by self-attention. In this paper, we propose Coarse-Fine Spectral-Aware Deformable Convolution Network (CFSDCN), applying deformable convolutional networks (DCN) to this task for the first time. Considering the sparsity of HSI, we design a deformable convolution module that exploits its deformability to capture long-range dependencies and non-local similarities. In addition, we propose a new spectral information interaction module that considers both coarse-grained and fine-grained spectral similarities. Extensive experiments demonstrate that our CFSDCN significantly outperforms previous state-of-the-art (SOTA) methods on both simulated and real HSI datasets.
Paper Structure (12 sections, 8 equations, 5 figures, 3 tables)

This paper contains 12 sections, 8 equations, 5 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. MATHEMATICAL MODEL Of CASSI
  3. METHOD
  4. Overall Architecture
  5. Spatial-aware Deformable Convolution Module
  6. Coarse-Fine Spectral-Aware Module
  7. EXPERIMENTS
  8. Experimental Settings
  9. Quantitative Results
  10. Qualitative Results
  11. Ablation Study
  12. CONCLUSION

Figures (5)

  • Figure 1: Comparison of PSNR-Params-FLOPs with some previous SOTA methods.The vertical axis represents PSNR, and the horizontal axis represents FLOPs. The size of the scatter points is related to the parameters of the model.
  • Figure 2: A schematic diagram of CASSI.
  • Figure 3: Details of CFSDCN. (a) Overall structure diagram of CFSDCN. (b) Schematic diagram of Coarse-Fine Spectral-Aware Deformable Convolution (CFSConv). (c) Schematic diagram of Coarse-Fine Spectral-Aware Block (CFSAB).
  • Figure 4: Comparison of reconstruction results of simulation HSI datasets for 4 out of 28 spectral channels in Scene 2. 7 SOTA methods and our CFSDCN are included here. Zoom in for a better view.
  • Figure 5: Comparison of reconstruction results of real HSI datasets for 4 out of 28 spectral channels in Scene 2. 7 SOTA methods and our CFSDCN-M are included here. Zoom in for a better view.