Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HS-3D-NeRF: 3D Surface and Hyperspectral Reconstruction From Stationary Hyperspectral Images Using Multi-Channel NeRFs

Kibon Ku, Talukder Z. Jubery, Adarsh Krishnamurthy, Baskar Ganapathysubramanian

TL;DR

HSI-SC-NeRF introduces a stationary-camera, multi-channel NeRF pipeline to jointly reconstruct 3D geometry and hyperspectral radiance from rotating agricultural objects, enabling high-throughput spectral 3D phenotyping. The method leverages ArUco-based pose estimation, a diffuse PTFE chamber, and a two-stage training regime to decouple geometry initialization from radiometric refinement across $204$ spectral bands. Quantitative results on apple, pear, and maize demonstrate strong spatial fidelity (e.g., high F-scores and sub-millimeter RMS errors) and wavelength-resolved spectral fidelity (low SAM and RMSE across bands), with loss-weight ablations guiding the optimal configuration. The approach offers a scalable pathway to integrate hyperspectral data into automated phenotyping and postharvest quality control workflows, while contributing a general framework for multi-channel spectral NeRF-based 3D reconstruction.

Abstract

Advances in hyperspectral imaging (HSI) and 3D reconstruction have enabled accurate, high-throughput characterization of agricultural produce quality and plant phenotypes, both essential for advancing agricultural sustainability and breeding programs. HSI captures detailed biochemical features of produce, while 3D geometric data substantially improves morphological analysis. However, integrating these two modalities at scale remains challenging, as conventional approaches involve complex hardware setups incompatible with automated phenotyping systems. Recent advances in neural radiance fields (NeRF) offer computationally efficient 3D reconstruction but typically require moving-camera setups, limiting throughput and reproducibility in standard indoor agricultural environments. To address these challenges, we introduce HSI-SC-NeRF, a stationary-camera multi-channel NeRF framework for high-throughput hyperspectral 3D reconstruction targeting postharvest inspection of agricultural produce. Multi-view hyperspectral data is captured using a stationary camera while the object rotates within a custom-built Teflon imaging chamber providing diffuse, uniform illumination. Object poses are estimated via ArUco calibration markers and transformed to the camera frame of reference through simulated pose transformations, enabling standard NeRF training on stationary-camera data. A multi-channel NeRF formulation optimizes reconstruction across all hyperspectral bands jointly using a composite spectral loss, supported by a two-stage training protocol that decouples geometric initialization from radiometric refinement. Experiments on three agricultural produce samples demonstrate high spatial reconstruction accuracy and strong spectral fidelity across the visible and near-infrared spectrum, confirming the suitability of HSI-SC-NeRF for integration into automated agricultural workflows.

HS-3D-NeRF: 3D Surface and Hyperspectral Reconstruction From Stationary Hyperspectral Images Using Multi-Channel NeRFs

TL;DR

HSI-SC-NeRF introduces a stationary-camera, multi-channel NeRF pipeline to jointly reconstruct 3D geometry and hyperspectral radiance from rotating agricultural objects, enabling high-throughput spectral 3D phenotyping. The method leverages ArUco-based pose estimation, a diffuse PTFE chamber, and a two-stage training regime to decouple geometry initialization from radiometric refinement across spectral bands. Quantitative results on apple, pear, and maize demonstrate strong spatial fidelity (e.g., high F-scores and sub-millimeter RMS errors) and wavelength-resolved spectral fidelity (low SAM and RMSE across bands), with loss-weight ablations guiding the optimal configuration. The approach offers a scalable pathway to integrate hyperspectral data into automated phenotyping and postharvest quality control workflows, while contributing a general framework for multi-channel spectral NeRF-based 3D reconstruction.

Abstract

Advances in hyperspectral imaging (HSI) and 3D reconstruction have enabled accurate, high-throughput characterization of agricultural produce quality and plant phenotypes, both essential for advancing agricultural sustainability and breeding programs. HSI captures detailed biochemical features of produce, while 3D geometric data substantially improves morphological analysis. However, integrating these two modalities at scale remains challenging, as conventional approaches involve complex hardware setups incompatible with automated phenotyping systems. Recent advances in neural radiance fields (NeRF) offer computationally efficient 3D reconstruction but typically require moving-camera setups, limiting throughput and reproducibility in standard indoor agricultural environments. To address these challenges, we introduce HSI-SC-NeRF, a stationary-camera multi-channel NeRF framework for high-throughput hyperspectral 3D reconstruction targeting postharvest inspection of agricultural produce. Multi-view hyperspectral data is captured using a stationary camera while the object rotates within a custom-built Teflon imaging chamber providing diffuse, uniform illumination. Object poses are estimated via ArUco calibration markers and transformed to the camera frame of reference through simulated pose transformations, enabling standard NeRF training on stationary-camera data. A multi-channel NeRF formulation optimizes reconstruction across all hyperspectral bands jointly using a composite spectral loss, supported by a two-stage training protocol that decouples geometric initialization from radiometric refinement. Experiments on three agricultural produce samples demonstrate high spatial reconstruction accuracy and strong spectral fidelity across the visible and near-infrared spectrum, confirming the suitability of HSI-SC-NeRF for integration into automated agricultural workflows.
Paper Structure (25 sections, 26 equations, 13 figures, 3 tables)

This paper contains 25 sections, 26 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hyperspectral Imaging in Agriculture and 3D Reconstruction
  4. Neural Radiance Fields for Spectral 3D Reconstruction
  5. Limitations of Moving-Camera NeRF in High-Throughput Settings
  6. HSI-SC-NeRF
  7. Materials and Methods
  8. Experimental Setup and Data Collection
  9. Data Processing
  10. NeRF-Based Hyperspectral PCD Reconstruction
  11. Evaluation Metrics
  12. Results and Discussion
  13. Example Applications
  14. Spatial Validation
  15. Spectral Validation
  16. ...and 10 more sections

Figures (13)

  • Figure 1: Motivation for hyperspectral imaging (HSI) in 3D. Two views of the same apple show viewpoint-dependent bruise visibility: one ROI is visible in view (A) but occluded in view (B), motivating HSI with 3D reconstruction for more complete surface assessment.
  • Figure 2: Workflow of the HSI-SC-NeRF pipeline. The process consists of three main steps: (A) Dataset Acquisition, where the experimental environment is set up and multi-view hyperspectral image data is collected using a stationary camera; (B) Data Preprocessing, involving white-reference spectral calibration and COLMAP-based pose estimation to ensure radiometric and geometric consistency; and (C) NeRF-Based HS PCD, where a multi-channel NeRF model is trained for scene representation, followed by hyperspectral point cloud reconstruction and refinement to generate high-quality 3D hyperspectral point clouds. The graphical abstract (bottom) illustrates the end-to-end pipeline: the imaging chamber with rotating turntable (A), an RGB timelapse frame from the acquisition session (B), the resulting hyperspectral point cloud at 480 nm (C), and per-region reflectance spectra extracted from the reconstructed point cloud (D).
  • Figure 3: Teflon studio chamber for uniform hyperspectral illumination. (A) The chamber is designed from PTFE (Teflon) sheets to provide diffuse illumination. (B) Inside the chamber and (C) the outside.
  • Figure 4: White reference (WR)--based spectral calibration. (A) Coarse WR region of interest (ROI) extraction. (B) Automatically generated central WR mask after 70th-percentile-based spatial deviation filtering. (C) Pseudo-RGB rendering of the raw hyperspectral cube before calibration. (D) Final spectrally calibrated result after WR-based normalization and reflectance clipping.
  • Figure 5: Camera poses and viewing directions estimated by COLMAP for 64 hyperspectral images. Black squares show camera positions, and red arrows indicate viewing directions toward the object center. (A) Pear, (B) Maize, and (C) Apple.
  • ...and 8 more figures