IPENS:Interactive Unsupervised Framework for Rapid Plant Phenotyping Extraction via NeRF-SAM2 Fusion

TL;DR

IPENS addresses the challenge of unsupervised, grain-level 3D plant segmentation by integrating NeRF-based radiance fields with SAM2-driven 2D segmentation and mask inverse rendering. The framework supports single-round, multi-target prompting (manual or YOLO-assisted) to lift 2D masks into high-quality 3D point clouds, with two post-processing stages and SSIM-guided rear-frame prompting to handle occlusions. On rice and wheat, IPENS achieves competitive mIoU and excellent phenotypic correlations (e.g., $R^2$ values up to 0.9956 for panicle volume and 1.00 for leaf area) while completing the pipeline in about 3 minutes without annotated data. This non-invasive, rapid approach offers substantial potential to accelerate intelligent breeding by delivering reliable 3D phenotypes from unlabeled imagery across multiple species.

Abstract

Advanced plant phenotyping technologies play a crucial role in targeted trait improvement and accelerating intelligent breeding. Due to the species diversity of plants, existing methods heavily rely on large-scale high-precision manually annotated data. For self-occluded objects at the grain level, unsupervised methods often prove ineffective. This study proposes IPENS, an interactive unsupervised multi-target point cloud extraction method. The method utilizes radiance field information to lift 2D masks, which are segmented by SAM2 (Segment Anything Model 2), into 3D space for target point cloud extraction. A multi-target collaborative optimization strategy is designed to effectively resolve the single-interaction multi-target segmentation challenge. Experimental validation demonstrates that IPENS achieves a grain-level segmentation accuracy (mIoU) of 63.72% on a rice dataset, with strong phenotypic estimation capabilities: grain volume prediction yields R2 = 0.7697 (RMSE = 0.0025), leaf surface area R2 = 0.84 (RMSE = 18.93), and leaf length and width predictions achieve R2 = 0.97 and 0.87 (RMSE = 1.49 and 0.21). On a wheat dataset,IPENS further improves segmentation accuracy to 89.68% (mIoU), with equally outstanding phenotypic estimation performance: spike volume prediction achieves R2 = 0.9956 (RMSE = 0.0055), leaf surface area R2 = 1.00 (RMSE = 0.67), and leaf length and width predictions reach R2 = 0.99 and 0.92 (RMSE = 0.23 and 0.15). This method provides a non-invasive, high-quality phenotyping extraction solution for rice and wheat. Without requiring annotated data, it rapidly extracts grain-level point clouds within 3 minutes through simple single-round interactions on images for multiple targets, demonstrating significant potential to accelerate intelligent breeding efficiency.

Figures (11)

• Figure 1: Overall Workflow of IPENS: Data Preparation, Model & Method, and Phenotyping Extraction.
• Figure 2: Data acquisition cube. Components inside (b) include:1:A 98-cell modular seedling tray in the breeding chamber 2:Rice or other crop. 3: Intel D435i camera. 4: Robotic arm. 5: Top and side soft-box lights.
• Figure 3: Data acquisition and instance segmentation
• Figure 4: IPENS Model. Given a radiance field trained on rice or wheat, the model first takes manual inputs or YOLO prompts as input. It then uses SAM2 to generate 2D masks for the image sequence. Based on the radiance field information, a mask inverse rendering process is performed iteratively, ultimately obtaining the 3D masks.
• Figure 5: Target Vanishing to Reappearance Process