3D Branch Point Cloud Completion for Robotic Pruning in Apple Orchards

TL;DR

This work addressed point cloud quality via a closed-loop approach, (Real2Sim)−1, Leveraging a Real-to-Simulation (Real2Sim) data generation pipeline, and generated simulated 3D apple trees based on realistically characterized apple tree information without manual parameterization that contributed to the precision and efficacy of robotic branch pruning.

Abstract

Robotic branch pruning is a significantly growing research area to cope with the shortage of labor force in the context of agriculture. One fundamental requirement in robotic pruning is the perception of detailed geometry and topology of branches. However, the point clouds obtained in agricultural settings often exhibit incompleteness due to several constraints, thereby restricting the accuracy of downstream robotic pruning. In this work, we addressed the issue of point cloud quality through a simulation-based deep neural network, leveraging a Real-to-Simulation (Real2Sim) data generation pipeline that not only eliminates the need for manual parameterization but also guarantees the realism of simulated data. The simulation-based neural network was applied to jointly perform point cloud completion and skeletonization on real-world partial branches, without additional real-world training. The Sim2Real qualitative completion and skeletonization results showed the model's remarkable capability for geometry reconstruction and topology prediction. Additionally, we quantitatively evaluated the Sim2Real performance by comparing branch-level trait characterization errors using raw incomplete data and complete data. The Mean Absolute Error (MAE) reduced by 75% and 8% for branch diameter and branch angle estimation, respectively, using the best complete data, which indicates the effectiveness of the Real2Sim data in a zero-shot generalization setting. The characterization improvements contributed to the precision and efficacy of robotic branch pruning.

Figures (5)

• Figure 1: The proposed $\text{(}\text{Real2Sim}\text{)}^{-1}$ (i.e., Real2Sim and Sim2Real) loop for deep learning models in the context of agriculture where large-scale datasets are usually unavailable and need enormous effort for development.
• Figure 2: Top: The developed joint completion and skeletonization model formulated as a GAN framework. Bottom: The illustration of the variance loss that aims to minimize the variance of the distance from surface points ($d_{1}, d_{2}, \cdots, d_{n}$) to the skeleton point to better constrain the geometry. The red dot represents the skeleton point for the cross-section.
• Figure 3: Branch representative from each dataset. The real-world data presents significant incompleteness and discontinuity, which is the major bottleneck for robotic perception. The NB data shows simple geometry and lacks intricate structures. The FB data shows significantly more realistic geometry and organic structure.
• Figure 4: Completion results generated by different models. The red points represent the complete points while the green points are the skeleton points. The first two rows show the partial branches while the last two rows show the partial and discontinued branches. The bottom blue and green boxes present the zoom-in completion results of the corresponding selected area in the second and third rows. We referred to the joint model as ours. GS stands for generative and skeleton losses and V for variance loss.
• Figure 5: Pruning map developed based on the characterization results using raw data and the complete data from Ours-GSV-FB.