Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HYDRA: HYbrid knowledge Distillation and spectral Reconstruction Algorithm for high channel hyperspectral camera applications

Christopher Thirgood, Oscar Mendez, Erin Ling, Jon Storey, Simon Hadfield

TL;DR

HYDRA tackles the challenge of generalizable spectral reconstruction from RGB to high-channel hyperspectral images by learning in a compact latent space $S'$. It introduces a cross-modal Teacher-Student framework, where a hyperspectral autoencoder (Teacher) learns $S'$ and an RGB-to-latent mapper (Student) is trained to translate RGB inputs into that latent domain, followed by a refinement stage that end-to-end tunes the Decoder. The three-stage training—Teacher encoding, RGB-to-latent mapping, and end-to-end refinement—yields state-of-the-art accuracy while reducing FLOPs, enabling robust SR across datasets with up to 204 channels. HYDRA’s combination of SE-enabled spectral compression, MDTA-based attention in a U-Net–style Student, and cross-modal distillation offers a practical, scalable solution for high-dimensional HSIs.

Abstract

Hyperspectral images (HSI) promise to support a range of new applications in computer vision. Recent research has explored the feasibility of generalizable Spectral Reconstruction (SR), the problem of recovering a HSI from a natural three-channel color image in unseen scenarios. However, previous Multi-Scale Attention (MSA) works have only demonstrated sufficient generalizable results for very sparse spectra, while modern HSI sensors contain hundreds of channels. This paper introduces a novel approach to spectral reconstruction via our HYbrid knowledge Distillation and spectral Reconstruction Architecture (HYDRA). Using a Teacher model that encapsulates latent hyperspectral image data and a Student model that learns mappings from natural images to the Teacher's encoded domain, alongside a novel training method, we achieve high-quality spectral reconstruction. This addresses key limitations of prior SR models, providing SOTA performance across all metrics, including an 18\% boost in accuracy, and faster inference times than current SOTA models at various channel depths.

HYDRA: HYbrid knowledge Distillation and spectral Reconstruction Algorithm for high channel hyperspectral camera applications

TL;DR

HYDRA tackles the challenge of generalizable spectral reconstruction from RGB to high-channel hyperspectral images by learning in a compact latent space . It introduces a cross-modal Teacher-Student framework, where a hyperspectral autoencoder (Teacher) learns and an RGB-to-latent mapper (Student) is trained to translate RGB inputs into that latent domain, followed by a refinement stage that end-to-end tunes the Decoder. The three-stage training—Teacher encoding, RGB-to-latent mapping, and end-to-end refinement—yields state-of-the-art accuracy while reducing FLOPs, enabling robust SR across datasets with up to 204 channels. HYDRA’s combination of SE-enabled spectral compression, MDTA-based attention in a U-Net–style Student, and cross-modal distillation offers a practical, scalable solution for high-dimensional HSIs.

Abstract

Hyperspectral images (HSI) promise to support a range of new applications in computer vision. Recent research has explored the feasibility of generalizable Spectral Reconstruction (SR), the problem of recovering a HSI from a natural three-channel color image in unseen scenarios. However, previous Multi-Scale Attention (MSA) works have only demonstrated sufficient generalizable results for very sparse spectra, while modern HSI sensors contain hundreds of channels. This paper introduces a novel approach to spectral reconstruction via our HYbrid knowledge Distillation and spectral Reconstruction Architecture (HYDRA). Using a Teacher model that encapsulates latent hyperspectral image data and a Student model that learns mappings from natural images to the Teacher's encoded domain, alongside a novel training method, we achieve high-quality spectral reconstruction. This addresses key limitations of prior SR models, providing SOTA performance across all metrics, including an 18\% boost in accuracy, and faster inference times than current SOTA models at various channel depths.
Paper Structure (21 sections, 12 equations, 6 figures, 6 tables)

This paper contains 21 sections, 12 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Learning for SR
  4. HSI Compression
  5. Teacher-Student Architectures
  6. Method
  7. Teacher model
  8. Encoder Structure
  9. Decoder Structure
  10. Student Model
  11. U-Net Pipeline
  12. MDTA blocks
  13. DGFN
  14. Training Procedure
  15. Training stage one:
  16. ...and 6 more sections

Figures (6)

  • Figure 1: (a) NTIRE-2022 (b) HySpecNet-11k (c) LIB-HSI. HYDRA excels in spectral reconstruction across various channel depths, balancing computational efficiency (FLOPS) with high performance (PSNR). Circle size indicates memory cost.
  • Figure 2: (a) The Teacher model, (b) the operation of an SE block.
  • Figure 3: Our Student model: A U-Net shaped Vision Transformer network in the channel domain, utilizing Multi-Dconv Head Transposed Attention (MDTA) and Dual-Gated-Dconv Feed-Forward Network (DGDF) modules. Green 3x3 blocks are depth-wise convolutions and blue blocks 1x1 are pixel-wise convolutions to capture spatial and spectral shapes.
  • Figure 4: (a) The traditional training method for SR. (b), (c) and (d) show the stages of the HYDRA training procedure.
  • Figure 5: MRAE heatmap errors for different datasets.
  • ...and 1 more figures