In-Field 3D Wheat Head Instance Segmentation From TLS Point Clouds Using Deep Learning Without Manual Labels

Abstract

3D instance segmentation for laser scanning (LiDAR) point clouds remains a challenge in many remote sensing-related domains. Successful solutions typically rely on supervised deep learning and manual annotations, and consequently focus on objects that can be well delineated through visual inspection and manual labeling of point clouds. However, for tasks with more complex and cluttered scenes, such as in-field plant phenotyping in agriculture, such approaches are often infeasible. In this study, we tackle the task of in-field wheat head instance segmentation directly from terrestrial laser scanning (TLS) point clouds. To address the problem and circumvent the need for manual annotations, we propose a novel two-stage pipeline. To obtain the initial 3D instance proposals, the first stage uses 3D-to-2D multi-view projections, the Grounded SAM pipeline for zero-shot 2D object-centric segmentation, and multi-view label fusion. The second stage uses these initial proposals as noisy pseudo-labels to train a supervised 3D panoptic-style segmentation neural network. Our results demonstrate the feasibility of the proposed approach and show performance improvementsrelative to Wheat3DGS, a recent alternative solution for in-field wheat head instance segmentation without manual 3D annotations based on multi-view RGB images and 3D Gaussian Splatting, showcasing TLS as a competitive sensing alternative. Moreover, the results show that both stages of the proposed pipeline can deliver usable 3D instance segmentation without manual annotations, indicating promising, low-effort transferability to other comparable TLS-based point cloud segmentation tasks.

Figures (6)

• Figure 1: Flowchart of the implemented multi-view projection-based 3D instance segmentation (white: I/O, blue: data processing modules).
• Figure 2: Examples of intensity (top) and range (bottom) images generated by the per-scan 3D-to-2D projection module, with a few examples of 2D instance masks generated by the 2D instance segmentation module. For clarity, only a small subset of the masks is shown.
• Figure 3: Overview of experiment setup and data collection. Orange rectangles: (1) field phenotyping platform with a camera rig, (2) photogrammetric target (coded marker), (3) reference sphere, (4) TLS. Black rectangles: 7 wheat plots covered within a measurement campaign, from left to right - 5 training, 1 validation, 1 test / hold-out plot.
• Figure 4: Sparse reference 3D instance annotations (red) overlaid over $P_t$ point cloud of the test / hold-out plot.
• Figure 5: Detected 3D wheat head instances (random color per instance alignment) overlaid over $P_t$ point cloud of the hold-out (test) plot. A few side views are shown in Appendix A.