PLANesT-3D: A new annotated dataset for segmentation of 3D plant point clouds

TL;DR

PLANesT-3D introduces a new dataset of 34 annotated 3D plant color point clouds across three species (pepper, rose, ribes) acquired with low-cost SfM/MVS, enabling semantic and instance segmentation research. The authors propose SP-LSCnet, a partly unsupervised two-stage segmentation pipeline that maps 3D points to 2D via t-SNE to form superpoints, followed by a 3D classifier with adaptive local regions, and benchmark it against PointNet++ and RoseSegNet. Quantitative results show RoseSegNet achieving the highest MIoU across species, with SP-LSCnet providing competitive performance and enhanced interpretability due to its 2D visualization step. The dataset emphasizes diversity, noise, and illumination variability, offering a robust testbed for generalization and future panoptic segmentation work in 3D plant phenotyping. Overall, PLANesT-3D serves as a valuable resource for evaluating cross-species generalization, robustness to reconstruction artifacts, and the development of interpretable segmentation approaches in plant science.

Abstract

Creation of new annotated public datasets is crucial in helping advances in 3D computer vision and machine learning meet their full potential for automatic interpretation of 3D plant models. Despite the proliferation of deep neural network architectures for segmentation and phenotyping of 3D plant models in the last decade, the amount of data, and diversity in terms of species and data acquisition modalities are far from sufficient for evaluation of such tools for their generalization ability. To contribute to closing this gap, we introduce PLANesT-3D; a new annotated dataset of 3D color point clouds of plants. PLANesT-3D is composed of 34 point cloud models representing 34 real plants from three different plant species: \textit{Capsicum annuum}, \textit{Rosa kordana}, and \textit{Ribes rubrum}. Both semantic labels in terms of "leaf" and "stem", and organ instance labels were manually annotated for the full point clouds. PLANesT-3D introduces diversity to existing datasets by adding point clouds of two new species and providing 3D data acquired with the low-cost SfM/MVS technique as opposed to laser scanning or expensive setups. Point clouds reconstructed with SfM/MVS modality exhibit challenges such as missing data, variable density, and illumination variations. As an additional contribution, SP-LSCnet, a novel semantic segmentation method that is a combination of unsupervised superpoint extraction and a 3D point-based deep learning approach is introduced and evaluated on the new dataset. The advantages of SP-LSCnet over other deep learning methods are its modular structure and increased interpretability. Two existing deep neural network architectures, PointNet++ and RoseSegNet, were also tested on the point clouds of PLANesT-3D for semantic segmentation.

Figures (16)

• Figure 1: Estimated camera poses for 231 images of a pepper plant (a), 240 images of a rose plant (b), and 177 images of two ribes plants (c). Sample images for pepper (d), rose (e), and ribes (f) are also provided.
• Figure 2: Raw point clouds reconstructed via Agisoft Metashape Professional of a rose plant (a) and a ribes plant (c). Corresponding clouds including only 3D points of the target plants are given in (b) for rose, and in (d) for ribes.
• Figure 3: Sample point clouds from PLANesT-3D demonstrating the diversity in terms of the overall plant shape and architecture (a). Examples of dense foliage in the PLANesT-3D dataset (b).
• Figure 4: Diversity in leaf shape and texture (a). Diversity in stem shape (b).
• Figure 5: Instances of noisy and missing data due to self-occlusion and reconstruction errors.