Hyperspectral Neural Radiance Fields

TL;DR

This work tackles 3D hyperspectral reconstruction by extending Neural Radiance Fields to model wavelength-dependent radiance and transmittance. HS-NeRF conditions color radiance and volume density on wavelength using latent spectra, enabling continuous, spectrally aware rendering with equations for $c^λ$ and $σ^λ$ via dedicated decoders. They construct a dataset of nearly 2000 hyperspectral images across 8 scenes and 2 cameras, show improvements over RGB NeRF baselines through extensive ablations, and demonstrate practical applications in hyperspectral super-resolution and imaging sensor simulation. The results indicate fast, accurate volumetric hyperspectral reconstructions with broad potential impact for material analysis and sensor design.

Abstract

Hyperspectral Imagery (HSI) has been used in many applications to non-destructively determine the material and/or chemical compositions of samples. There is growing interest in creating 3D hyperspectral reconstructions, which could provide both spatial and spectral information while also mitigating common HSI challenges such as non-Lambertian surfaces and translucent objects. However, traditional 3D reconstruction with HSI is difficult due to technological limitations of hyperspectral cameras. In recent years, Neural Radiance Fields (NeRFs) have seen widespread success in creating high quality volumetric 3D representations of scenes captured by a variety of camera models. Leveraging recent advances in NeRFs, we propose computing a hyperspectral 3D reconstruction in which every point in space and view direction is characterized by wavelength-dependent radiance and transmittance spectra. To evaluate our approach, a dataset containing nearly 2000 hyperspectral images across 8 scenes and 2 cameras was collected. We perform comparisons against traditional RGB NeRF baselines and apply ablation testing with alternative spectra representations. Finally, we demonstrate the potential of hyperspectral NeRFs for hyperspectral super-resolution and imaging sensor simulation. We show that our hyperspectral NeRF approach enables creating fast, accurate volumetric 3D hyperspectral scenes and enables several new applications and areas for future study.

Figures (8)

• Figure 1: Instead of 3 color channels for each pixel, hyperspectral images have many color channels per pixel to measure the color spectrum for every pixel. In this work, we leverage recent advances in Neural Radiance Fields to create hyperspectral 3D scene representations. Left: an example hyperspectral image and spectrums at 2 points. Right: a hyperspectral NeRF and spectra.
• Figure 2: To handle hyperspectral data, we include a wavelength input to our network which predicts a scalar color intensity and a scalar transmittance. The network produces spectra for the color intensity and transmittance via the latent vectors $\bm{\Theta}_C$ and $\bm{\Theta}_\sigma$, respectively, and networks $\bm{C}(\lambda;\bm{\Theta}_C)$ and $\sigma(\lambda;\bm{\Theta}_\sigma)$ compute the value of the spectra at the queried wavelength.
• Figure 3: The Surface Optics SOC710-VP camera is mounted on a tripod and the sample of interest is placed on a turnable in a Macbeth SpectraLight lightbooth with a light gray background. The camera is roughly 2 meters away from the scene due to its shallow depth of field and narrow field of view.
• Figure 4: The BaySpec GoldenEye camera is held with a laboratory clamp and the sample of interest (here, an Anacampseros plastic plant) is placed on a turnable under tungsten halogen lighting. The camera is roughly 20 centimeters away from the scene thanks to its wide field of view.
• Figure 5: By visually inspecting the same image of the basil plant in several different wavelengths, it becomes obvious that the additional information afforded by HSI makes background removal significantly easier than in RGB images.