AHPPEBot: Autonomous Robot for Tomato Harvesting based on Phenotyping and Pose Estimation

TL;DR

AHPPEBot addresses labor shortages in greenhouse tomato harvesting by integrating rapid phenotyping with seven-peduncle pose estimation to enable autonomous, selective peduncle cutting. The approach combines a multi-task YOLOv5 detector for truss and fruit maturity with adaptive DBScan-based association and a deep keypoint model (seven peduncle points) to guide path planning and the end-effector’s motion via bottom-up wrapping. In greenhouse experiments, the system achieved an harvesting success rate of 86.67% with an average time of 32.46 s, demonstrating robust, continuous harvesting capabilities and potential to reduce labor gaps in agriculture. The work highlights the value of coupling plant phenotyping with pose-based manipulation to improve selectivity, safety, and throughput in agricultural robotics.

Abstract

To address the limitations inherent to conventional automated harvesting robots specifically their suboptimal success rates and risk of crop damage, we design a novel bot named AHPPEBot which is capable of autonomous harvesting based on crop phenotyping and pose estimation. Specifically, In phenotyping, the detection, association, and maturity estimation of tomato trusses and individual fruits are accomplished through a multi-task YOLOv5 model coupled with a detection-based adaptive DBScan clustering algorithm. In pose estimation, we employ a deep learning model to predict seven semantic keypoints on the pedicel. These keypoints assist in the robot's path planning, minimize target contact, and facilitate the use of our specialized end effector for harvesting. In autonomous tomato harvesting experiments conducted in commercial greenhouses, our proposed robot achieved a harvesting success rate of 86.67%, with an average successful harvest time of 32.46 s, showcasing its continuous and robust harvesting capabilities. The result underscores the potential of harvesting robots to bridge the labor gap in agriculture.

Figures (3)

• Figure 1: AHPPEBot Overview: The robot perceives nearby tomato trusses and autonomously makes decisions based on various information such as arm workspace, ripeness, and pose, ultimately using a circular cutter to harvest the target.
• Figure 2: Automated Harvesting System Workflow: During the phenotyping, the number and ripeness of each tomato truss are determined (different color prediction frames indicate varying maturity levels). The pose detection captures the key points of the tomato trusses. By integrating and encoding both phenotypic and pose information, a target is selected. Subsequently, based on its pose, the robotic arm's path is planned, and harvesting is executed.
• Figure 3: A tomato truss is represented by a combination of 7 peduncle keypoints and multiple fruit keypoints. The harvesting process, termed "bottom-up wrapping," involves (a) initial positioning below the target; (b) slow ascent and translation along the peduncle curve to avoid collisions with the target until all fruits are fully enveloped; (c) bringing the edge of the end-effector close to the SP point; (d) rotating the end-effector, causing the peduncle to fall into the slot and be cut by the blade; (e) applying continued pressure through rotation; (f) the peduncle is severed and falls into the collection mesh. Key keypoints predominantly utilized in each stage are denoted in blue.