AgriNeRF: Neural Radiance Fields for Agriculture in Challenging Lighting Conditions

TL;DR

This work results in the development of a resilient NeRF, capable of performing well in visibly degraded scenarios, as well as a learnt cross-spectral representation that is used for automated fruit detection.

Abstract

Neural Radiance Fields (NeRFs) have shown significant promise in 3D scene reconstruction and novel view synthesis. In agricultural settings, NeRFs can serve as digital twins, providing critical information about fruit detection for yield estimation and other important metrics for farmers. However, traditional NeRFs are not robust to challenging lighting conditions, such as low-light, extreme bright light and varying lighting. To address these issues, this work leverages three different sensors: an RGB camera, an event camera and a thermal camera. Our RGB scene reconstruction shows an improvement in PSNR and SSIM by +2.06 dB and +8.3% respectively. Our cross-spectral scene reconstruction enhances downstream fruit detection by +43.0% in mAP50 and +61.1% increase in mAP50-95. The integration of additional sensors leads to a more robust and informative NeRF. We demonstrate that our multi-modal system yields high quality photo-realistic reconstructions under various tree canopy covers and at different times of the day. This work results in the development of a resilient NeRF, capable of performing well in visibly degraded scenarios, as well as a learnt cross-spectral representation, that is used for automated fruit detection.

Figures (5)

• Figure 1: AgriNeRF combines information from RGB, thermal and event cameras for resilient NeRF estimation in degraded lighting environments. Top: Sensor suite and input data. Left: Multi-modal sensor suite equipped with three different sensors. Right: RGB, thermal and event camera frames generated by sensor suite. These frames correspond to the physical camera location on the sensor suite. Bottom: Frames from the three cameras are used to generate RGB and Cross-Spectral NeRFs.
• Figure 2: Architecture of AgriNeRF and downstream fruit detection. We take frames from each of the cameras and sample features from coarse and fine TensorRF volumes. These features are then passed into the $MLP$s for the RGB and thermal modules. By optimizing over a regularization loss, we render a cross-spectral reconstruction. We generate two different NeRFs which are then leveraged for robust fruit detection.
• Figure 3: Multimodal sensor suite incorporating RGB, thermal and event cameras used to collect datasets. We also equip the sensor suite with a LiDAR and IMU.
• Figure 4: Qualitative comparison on Orchards and Garden datasets. Top: The Nerfacto reconstruction introduces an extra cloud, which is absent in the ground truth, while our method produces the most accurate reconstruction. Bottom: Our approach consistently delivers the highest fidelity, effectively capturing the fine details of the peppers with superior accuracy.
• Figure 5: Object detection comparison on Orchards and Garden datasets. For low-light sequences, our cross-spectral reconstruction is able to detect fruits not visible in the RGB reconstructions, thus showing highest overall performance and robustness in fruit detection.