YOLO11 and Vision Transformers based 3D Pose Estimation of Immature Green Fruits in Commercial Apple Orchards for Robotic Thinning

TL;DR

This work presents a field-ready 3D pose estimation pipeline for immature green apples by integrating YOLO11-based detection with Vision Transformer–driven depth estimation (DPT and Depth Anything V2). Through a rigorous comparison against YOLOv8 and subsequent RGB-to-RGB-D mapping, the study demonstrates that Depth Anything V2 provides the most accurate 3D pose length estimates, achieving the lowest RMSE/MAE among tested approaches. Field validation against caliper measurements on 70 fruitlets shows promise for robotic thinning in commercial orchards, with YOLO11n delivering real-time performance (as low as $2.7\,\mathrm{ms}$ per image) and robust detection under occlusion. The results highlight the practical potential of combining advanced object detection with monocular depth prediction to enable precise manipulation in dynamic orchard environments, potentially reducing manual labor and improving fruit quality.

Abstract

In this study, a robust method for 3D pose estimation of immature green apples (fruitlets) in commercial orchards was developed, utilizing the YOLO11(or YOLOv11) object detection and pose estimation algorithm alongside Vision Transformers (ViT) for depth estimation (Dense Prediction Transformer (DPT) and Depth Anything V2). For object detection and pose estimation, performance comparisons of YOLO11 (YOLO11n, YOLO11s, YOLO11m, YOLO11l and YOLO11x) and YOLOv8 (YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l and YOLOv8x) were made under identical hyperparameter settings among the all configurations. It was observed that YOLO11n surpassed all configurations of YOLO11 and YOLOv8 in terms of box precision and pose precision, achieving scores of 0.91 and 0.915, respectively. Conversely, YOLOv8n exhibited the highest box and pose recall scores of 0.905 and 0.925, respectively. Regarding the mean average precision at 50\% intersection over union (mAP@50), YOLO11s led all configurations with a box mAP@50 score of 0.94, while YOLOv8n achieved the highest pose mAP@50 score of 0.96. In terms of image processing speed, YOLO11n outperformed all configurations with an impressive inference speed of 2.7 ms, significantly faster than the quickest YOLOv8 configuration, YOLOv8n, which processed images in 7.8 ms. Subsequent integration of ViTs for the green fruit's pose depth estimation revealed that Depth Anything V2 outperformed Dense Prediction Transformer in 3D pose length validation, achieving the lowest Root Mean Square Error (RMSE) of 1.52 and Mean Absolute Error (MAE) of 1.28, demonstrating exceptional precision in estimating immature green fruit lengths. Integration of YOLO11 and Depth Anything Model provides a promising solution to 3D pose estimation of immature green fruits for robotic thinning applications. (YOLOv11 pose detection, YOLOv11 Pose, YOLOv11 Keypoints detection, YOLOv11 pose estimation)

Figures (13)

• Figure 1: Manual thinning of apple fruitlet in a commercial 'Scilate' apple orchard in Prosser, Washington State, USA, during 2024 growing season; (a) use of an aluminum ladder by farm workers to access fruit clusters in upper canopy region, demonstrating the physical demands and safety risks of traditional methods; b) manual thinning using a mechanized platform, which reduced the need for climbing ladders, but still required extensive manual efforts to reach fruitlet clusters; c) Showing the high-density examples of immature green apples in commercial orchards which needs thinning.
• Figure 2: Outlining of the research the methodology that includes various steps including data collection (using a robotic platform), deep learning model training and validation, and precise pose estimation of green fruitlets.
• Figure 3: YOLO11 Architecture used for immature green fruit detection
• Figure 4: Flowchart of the methodology used for depth mapping and validation using two Vision Transformer models. RGB images of apple trees with immature green apples (fruitlets), initially processed using YOLO11n for pose estimation, were transformed into RGB-D data by Dense Prediction Transformer (DPT) and Depth Anything V2. The depth maps generated were then used to create precise 3D point clouds, which were validated against ground truth measurements.
• Figure 5: (a) Workflow of the Dense Prediction Transformer (DPT) for generating 3D point clouds. DPT processes RGB images, originally captured using a consumer-grade camera, to produce RGB-D data, facilitating the creation of detailed 3D point clouds; (b) Workflow of Depth Anything V2, detailing each step from RGB image processing to depth map generation and subsequent 3D point cloud generation