Field Calibration of Hyperspectral Cameras for Terrain Inference

TL;DR

The paper tackles the challenge of obtaining illumination-invariant hyperspectral measurements for field robotics by introducing HYPER DRIVE, which combines VNIR and SWIR hyperspectral cameras with on-board spectrometers. It proposes a joint spectrometer–HSI calibration framework that uses a learned mapping $f$ and integration-time scaling to generate calibrated reflectance datacubes without calibration targets, supported by per-pixel regression models (MLR and MLP) and data augmentation. The system enables practical terrain inference, demonstrated through Vegetative Health via NDVI and Soil Moisture Content via a band index, achieving real-time processing on a mobile robot (roughly $30$ ms per datacube) and improved segmentation under varying illumination. The work advances field hyperspectral sensing by delivering a scalable, computationally efficient calibration workflow and actionable terrain metrics for autonomous navigation and decision-making.

Abstract

Intra-class terrain differences such as water content directly influence a vehicle's ability to traverse terrain, yet RGB vision systems may fail to distinguish these properties. Evaluating a terrain's spectral content beyond red-green-blue wavelengths to the near infrared spectrum provides useful information for intra-class identification. However, accurate analysis of this spectral information is highly dependent on ambient illumination. We demonstrate a system architecture to collect and register multi-wavelength, hyperspectral images from a mobile robot and describe an approach to reflectance calibrate cameras under varying illumination conditions. To showcase the practical applications of our system, HYPER DRIVE, we demonstrate the ability to calculate vegetative health indices and soil moisture content from a mobile robot platform.

Figures (7)

• Figure 1: Our hyperspectral imaging system, HYPER DRIVE, mounted to an off-road mobile robot. Images are collected from (a) a high resolution RGB camera, (b) a visible-near infrared (VNIR) hyperspectral camera, and (c) a shortwave infrared (SWIR) hyperspectral camera. We use combined (d) point spectrometers to measure a white-reference signal and calibrate the hyperspectral cameras.
• Figure 2: The likelihood of incident photon eliciting a digital count and sampling widths of the spectral sensors in HYPER DRIVE.
• Figure 3: To calibrate a datacube, the workflow takes in raw HSI hypercube data and spectral data from the down welling spectrometer (yellow), compares it to static calibration references (black), transforms it (blue), and passes it to the relevant algorithms (red).
• Figure 4: (a) Single digital number normalized white reference measure from the HYPER DRIVE spectrometer system with the sampled spectra of the VNIR HSI (green dots) and SWIR HSI (red dots). (b) Downsampling of raw spectra to the wavelengths associated with the (c) temporally synchronized hyperspectral image.
• Figure 5: (a) RGB image showing color checker (left) and Spectralon white reference target (right). The colored arrows correspond to the calibrated grayscale intensities seen in (f). (b) Raw demosaiced hyperspectral image with intensity in digital counts. Wavelengths increase from top-left to bottom right. (c) Reflectance calibrated image montage using the per-pixel MLP. Wavelengths increase from top-left to bottom right. (d) Raw digital count measurement from SWIR camera. (e) Reflectance calibrated image montage using the per-pixel MLP. Wavelengths increase from top-left to bottom right. (f) Reconstructed VNIR reflectance intensities for a 5-pixel neighborhood located in each target square. (g) Reconstructed SWIR reflectance intensities for Spectralon target in scene compared to naive min-max normalization.