Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpectralZoom: Efficient Segmentation with an Adaptive Hyperspectral Camera

Jackson Arnold, Sophia Rossi, Chloe Petrosino, Ethan Mitchell, Sanjeev J. Koppal

TL;DR

A novel camera design and a vision transformer-based (ViT) algorithm that alleviate both the captured data footprint and the computational load for hyperspectral segmentation are proposed.

Abstract

Hyperspectral image segmentation is crucial for many fields such as agriculture, remote sensing, biomedical imaging, battlefield sensing and astronomy. However, the challenge of hyper and multi spectral imaging is its large data footprint. We propose both a novel camera design and a vision transformer-based (ViT) algorithm that alleviate both the captured data footprint and the computational load for hyperspectral segmentation. Our camera is able to adaptively sample image regions or patches at different resolutions, instead of capturing the entire hyperspectral cube at one high resolution. Our segmentation algorithm works in concert with the camera, applying ViT-based segmentation only to adaptively selected patches. We show results both in simulation and on a real hardware platform demonstrating both accurate segmentation results and reduced computational burden.

SpectralZoom: Efficient Segmentation with an Adaptive Hyperspectral Camera

TL;DR

A novel camera design and a vision transformer-based (ViT) algorithm that alleviate both the captured data footprint and the computational load for hyperspectral segmentation are proposed.

Abstract

Hyperspectral image segmentation is crucial for many fields such as agriculture, remote sensing, biomedical imaging, battlefield sensing and astronomy. However, the challenge of hyper and multi spectral imaging is its large data footprint. We propose both a novel camera design and a vision transformer-based (ViT) algorithm that alleviate both the captured data footprint and the computational load for hyperspectral segmentation. Our camera is able to adaptively sample image regions or patches at different resolutions, instead of capturing the entire hyperspectral cube at one high resolution. Our segmentation algorithm works in concert with the camera, applying ViT-based segmentation only to adaptively selected patches. We show results both in simulation and on a real hardware platform demonstrating both accurate segmentation results and reduced computational burden.
Paper Structure (16 sections, 12 equations, 6 figures, 3 tables)

This paper contains 16 sections, 12 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Camera Model
  4. Mirror Control
  5. Mirror Sample Speed
  6. Image Alignment
  7. Multispectral Segmentation
  8. Multispectral Segmenter
  9. Wandering Patch
  10. Wandering Patch Position Encoding
  11. Attention Mechanisms
  12. Efficiency Benefits
  13. Full Simulation Results
  14. Experiments with Camera Prototype
  15. Partially Simulated Results
  16. ...and 1 more sections

Figures (6)

  • Figure 1: Hardware Prototype
  • Figure 2: Raw imagery from our device: Here we show a few images captured outdoors using our mirror modulated linear pushbroom camera. The raw images are before our calibration and counter rotation is applied. After processing, the corrected images are the inputs into our wandering patch network.
  • Figure 3: Spectral zoom pipeline diagram. Our workflow is shown on the left, where the mirror modulates quickly across the field-of-view creating a low-resolution HSI image. This image is then sent to a attention network that produces an attention map. The camera then uses the attention map to select high-res patches. A small number of low-res patches from the low-res image plus high-res patches from the camera are combined into a ViT segmentor from strudel_segmenter_2021 that is modified to accept (a) HSI data and (b) variable patches.
  • Figure 4: Wandering patch encoding scheme
  • Figure 5: 512x512 input into the trainable sailency map with 100 wandering patches
  • ...and 1 more figures