CropNeRF: A Neural Radiance Field-Based Framework for Crop Counting

TL;DR

CropNeRF presents a NeRF-based framework for exact crop counting via 3D instance segmentation in multi-view field scenes. It constructs a semantic NeRF to produce both an environment and a crop-specific point cloud, then partitions the crop cloud into super- and subclusters. Visibility and mask-consistency scores are computed to form robust affinities, which are merged through label propagation to yield precise counts without crop-specific tuning. Evaluations on cotton, apple, and pear datasets show superior counting accuracy compared to prior methods, and a public cotton plant dataset is released to foster further research.

Abstract

Rigorous crop counting is crucial for effective agricultural management and informed intervention strategies. However, in outdoor field environments, partial occlusions combined with inherent ambiguity in distinguishing clustered crops from individual viewpoints poses an immense challenge for image-based segmentation methods. To address these problems, we introduce a novel crop counting framework designed for exact enumeration via 3D instance segmentation. Our approach utilizes 2D images captured from multiple viewpoints and associates independent instance masks for neural radiance field (NeRF) view synthesis. We introduce crop visibility and mask consistency scores, which are incorporated alongside 3D information from a NeRF model. This results in an effective segmentation of crop instances in 3D and highly-accurate crop counts. Furthermore, our method eliminates the dependence on crop-specific parameter tuning. We validate our framework on three agricultural datasets consisting of cotton bolls, apples, and pears, and demonstrate consistent counting performance despite major variations in crop color, shape, and size. A comparative analysis against the state of the art highlights superior performance on crop counting tasks. Lastly, we contribute a cotton plant dataset to advance further research on this topic.

Figures (10)

• Figure 1: An overview of crop counting via 3D instance segmentation. The zoomed-in views depict clusters of cotton bolls, apples, and pears from a cotton plant, apple tree, and pear tree, respectively. The instance segmentation results for these clusters, obtained using CropNeRF, are shown in color on the right-hand side of the zoomed-in views.
• Figure 2: The crop counting pipeline. Multiple images of the target crop are captured from different viewpoints during the data collection step. In the data preprocessing stage, camera poses ($\mathcal{C}$) are extracted from the captured images ($\mathcal{I}$), and the crops are segmented into instance masks ($\mathcal{M}$). Semantic segmentation masks are then generated from the instance masks. The extracted camera poses, captured images, and semantic masks are used to train a semantic NeRF model during the 3D reconstruction stage. The trained model generates a semantic point cloud representing the crops ($\Pi_t$) and a point cloud representing the environment ($\Pi_e$). In the partitioning and merging step, the semantic point cloud is first divided into superclusters and then into subclusters ($S_i$). The affinity among subclusters is calculated based on subcluster visibility and mask consistency scores. Lastly, the subclusters are merged based on subcluster affinity to produce the final output.
• Figure 7: A visual representation of computing the mask reliability score. The score of the purple subcluster is calculated based on two different camera views. Only the visible projection area of the subcluster that overlaps with the masks (represented by the cyan and yellow boundaries) is considered in the numerator.
• Figure 8: The subcluster merging process. Top left: a single supercluster is partitioned into $10$ subclusters, each represented by a unique color. Top right: a weighted complete graph illustrates the affinities among subclusters, with positive and negative affinities colored in green and red, respectively. The width of each edge is proportional to its affinity value. Bottom right: the graph is partitioned into smaller subgraphs based on affinity scores via label propagation. Bottom left: the corresponding subclusters belonging to the same subgraph are merged.
• Figure 9: Cotton plant.