Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

High-fidelity 3D Reconstruction of Plants using Neural Radiance Field

Kewei Hu, Ying Wei, Yaoqiang Pan, Hanwen Kang, Chao Chen

TL;DR

The experimental results show that NeRF demonstrates commendable performance in the synthesis of novel-view images and is able to achieve reconstruction results that are competitive with Reality Capture, a leading commercial software for 3D Multi-View Stereo (MVS)-based reconstruction.

Abstract

Accurate reconstruction of plant phenotypes plays a key role in optimising sustainable farming practices in the field of Precision Agriculture (PA). Currently, optical sensor-based approaches dominate the field, but the need for high-fidelity 3D reconstruction of crops and plants in unstructured agricultural environments remains challenging. Recently, a promising development has emerged in the form of Neural Radiance Field (NeRF), a novel method that utilises neural density fields. This technique has shown impressive performance in various novel vision synthesis tasks, but has remained relatively unexplored in the agricultural context. In our study, we focus on two fundamental tasks within plant phenotyping: (1) the synthesis of 2D novel-view images and (2) the 3D reconstruction of crop and plant models. We explore the world of neural radiance fields, in particular two SOTA methods: Instant-NGP, which excels in generating high-quality images with impressive training and inference speed, and Instant-NSR, which improves the reconstructed geometry by incorporating the Signed Distance Function (SDF) during training. In particular, we present a novel plant phenotype dataset comprising real plant images from production environments. This dataset is a first-of-its-kind initiative aimed at comprehensively exploring the advantages and limitations of NeRF in agricultural contexts. Our experimental results show that NeRF demonstrates commendable performance in the synthesis of novel-view images and is able to achieve reconstruction results that are competitive with Reality Capture, a leading commercial software for 3D Multi-View Stereo (MVS)-based reconstruction. However, our study also highlights certain drawbacks of NeRF, including relatively slow training speeds, performance limitations in cases of insufficient sampling, and challenges in obtaining geometry quality in complex setups.

High-fidelity 3D Reconstruction of Plants using Neural Radiance Field

TL;DR

The experimental results show that NeRF demonstrates commendable performance in the synthesis of novel-view images and is able to achieve reconstruction results that are competitive with Reality Capture, a leading commercial software for 3D Multi-View Stereo (MVS)-based reconstruction.

Abstract

Accurate reconstruction of plant phenotypes plays a key role in optimising sustainable farming practices in the field of Precision Agriculture (PA). Currently, optical sensor-based approaches dominate the field, but the need for high-fidelity 3D reconstruction of crops and plants in unstructured agricultural environments remains challenging. Recently, a promising development has emerged in the form of Neural Radiance Field (NeRF), a novel method that utilises neural density fields. This technique has shown impressive performance in various novel vision synthesis tasks, but has remained relatively unexplored in the agricultural context. In our study, we focus on two fundamental tasks within plant phenotyping: (1) the synthesis of 2D novel-view images and (2) the 3D reconstruction of crop and plant models. We explore the world of neural radiance fields, in particular two SOTA methods: Instant-NGP, which excels in generating high-quality images with impressive training and inference speed, and Instant-NSR, which improves the reconstructed geometry by incorporating the Signed Distance Function (SDF) during training. In particular, we present a novel plant phenotype dataset comprising real plant images from production environments. This dataset is a first-of-its-kind initiative aimed at comprehensively exploring the advantages and limitations of NeRF in agricultural contexts. Our experimental results show that NeRF demonstrates commendable performance in the synthesis of novel-view images and is able to achieve reconstruction results that are competitive with Reality Capture, a leading commercial software for 3D Multi-View Stereo (MVS)-based reconstruction. However, our study also highlights certain drawbacks of NeRF, including relatively slow training speeds, performance limitations in cases of insufficient sampling, and challenges in obtaining geometry quality in complex setups.
Paper Structure (32 sections, 24 equations, 11 figures, 2 tables)

This paper contains 32 sections, 24 equations, 11 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Plant phenotyping
  4. 2D imaging: Multi-view RGB Imaging
  5. 3D imaging: Geometry Extraction
  6. Problem Statement
  7. Rendering Novel Viewpoints with NeRF
  8. Extraction Geometry From NeRF
  9. Point-Cloud Extraction from NeRF
  10. Mesh Extraction from NeRF
  11. Methods
  12. Data Acquisition
  13. View-pose estimation
  14. Learning From NeRF
  15. Network Training
  16. ...and 17 more sections

Figures (11)

  • Figure 1: High-fidelity 2D imaging (a) and 3D imaging (b) plant phenotypes from NeRF.
  • Figure 2: Framework of phenotyping system via NeRF: (a) Data Preparation, (b) Network Training, (c) Images Rendering, (d) Geometry Extraction.
  • Figure 3: (a) 360° image capturing, (b) front views capturing.
  • Figure 4: Pipeline of Instant-NGP and Instant-NSR. A). Instant-NGP: Given a 3D point $\mathbf{x}$, The 1)hash grid corresponding to each level l in the voxel grid is interpolated to hash encoding, then the density and color values are predicted by the 2)MLPs of density and color, and the color of the pixel is calculated by 3)volumetric rendering. B). Instant-NSR: Compared to the previousNGP, both the 2) MLPs and the 3)volume rendering are based on SDF and employ an extra normal regularisation to strengthen the geometrical constraints in the network training.
  • Figure 5: Photographs of indoor and outdoor orchards: (a) pitahaya ochard, (b) grape ochard, (c) orange ochard, (d) fig ochard.
  • ...and 6 more figures