Enhancing annotations for 5D apple pose estimation through 3D Gaussian Splatting (3DGS)

TL;DR

This work tackles occlusion challenges in 5D apple pose estimation by building a 3DGaussian Splatting (3DGS) reconstruction pipeline that enables per-fruit 3D annotations and automatic projection to images. By rendering additional views, the authors vastly increase training data (105 manual annotations yielded 28,191 labels), and evaluate how training occlusion rates and dataset size affect detection and pose estimation. Results show maximum benefits when training includes fruits with up to 95% occlusion, achieving a neutral F1 of 0.927 on original images and 0.970 on rendered images, with a typical position error around 7.13 mm; however, orientation estimation remains challenging, with large pitch/yaw errors and no clear learning signal for angle. Overall, the 3DGS-based pipeline substantially reduces annotation effort and provides a scalable way to study occlusion effects, though further work is needed to improve orientation prediction and generalize to diverse orchards.

Abstract

Automating tasks in orchards is challenging because of the large amount of variation in the environment and occlusions. One of the challenges is apple pose estimation, where key points, such as the calyx, are often occluded. Recently developed pose estimation methods no longer rely on these key points, but still require them for annotations, making annotating challenging and time-consuming. Due to the abovementioned occlusions, there can be conflicting and missing annotations of the same fruit between different images. Novel 3D reconstruction methods can be used to simplify annotating and enlarge datasets. We propose a novel pipeline consisting of 3D Gaussian Splatting to reconstruct an orchard scene, simplified annotations, automated projection of the annotations to images, and the training and evaluation of a pose estimation method. Using our pipeline, 105 manual annotations were required to obtain 28,191 training labels, a reduction of 99.6%. Experimental results indicated that training with labels of fruits that are $\leq95\%$ occluded resulted in the best performance, with a neutral F1 score of 0.927 on the original images and 0.970 on the rendered images. Adjusting the size of the training dataset had small effects on the model performance in terms of F1 score and pose estimation accuracy. It was found that the least occluded fruits had the best position estimation, which worsened as the fruits became more occluded. It was also found that the tested pose estimation method was unable to correctly learn the orientation estimation of apples.

Figures (20)

• Figure 1: Flowchart showing the proposed pipeline for pose estimation.
• Figure 2: Image from the trained 3DGS. The camera poses are shown using frustums in white. Not all camera poses are visible from this viewpoint.
• Figure 3: Steps to obtain fruit pose annotations.
• Figure 4: Examples of occlusion rates of different apples
• Figure 5: Figure showing the full 3DGS and the cropped 3DGS when using the bounding box of a tree.