A Novel Approach to Tomato Harvesting Using a Hybrid Gripper with Semantic Segmentation and Keypoint Detection

TL;DR

The study tackles the challenge of gentle, selective tomato harvesting in cluttered environments by integrating a six-finger hybrid soft–rigid gripper with a latex basket and separator leaves, a micro-servo pedicel cutter, and a Detectron2-based perception pipeline. End-to-end autonomy is achieved through PSO-based trajectory planning for a 5-DOF arm and closed-loop PID grasp-force control using FSR feedback, enabling stable, low-force grasps at approx. 0.30 N. The approach is validated in lab experiments, achieving about 80% success over ten trials with an average cycle time of 24.3 s and grasp forces in the 0.20–0.50 N range. These results demonstrate a feasible end-to-end harvesting solution capable of operating in cluttered, delicate-fruit contexts and offer design and perception-control insights for scalable agricultural robotics.

Abstract

This paper presents an autonomous tomato-harvesting system built around a hybrid robotic gripper that combines six soft auxetic fingers with a rigid exoskeleton and a latex basket to achieve gentle, cage-like grasping. The gripper is driven by a servo-actuated Scotch--yoke mechanism, and includes separator leaves that form a conical frustum for fruit isolation, with an integrated micro-servo cutter for pedicel cutting. For perception, an RGB--D camera and a Detectron2-based pipeline perform semantic segmentation of ripe/unripe tomatoes and keypoint localization of the pedicel and fruit center under occlusion and variable illumination. An analytical model derived using the principle of virtual work relates servo torque to grasp force, enabling design-level reasoning about actuation requirements. During execution, closed-loop grasp-force regulation is achieved using a proportional--integral--derivative controller with feedback from force-sensitive resistors mounted on selected fingers to prevent slip and bruising. Motion execution is supported by Particle Swarm Optimization (PSO)--based trajectory planning for a 5-DOF manipulator. Experiments demonstrate complete picking cycles (approach, separation, cutting, grasping, transport, release) with an average cycle time of 24.34~s and an overall success rate of approximately 80\%, while maintaining low grasp forces (0.20--0.50~N). These results validate the proposed hybrid gripper and integrated vision--control pipeline for reliable harvesting in cluttered environments.

Figures (12)

• Figure 1: CAD model of the proposed hybrid gripper. Left: cutter mechanism actuated by a micro-servo (top), soft auxetic finger structures for gentle grasping (middle), and separator leaves forming a conical frustum during separation (bottom). Right: servo-actuated Scotch--yoke mechanism used to drive finger closure.
• Figure 2: Micro-servo-actuated cutter mechanism mounted on a separator leaf. The cutter is actuated when the separator leaves are fully closed and the pedicel is aligned within the cutting zone.
• Figure 3: Representative finger contact model used for static force and virtual-work-based torque analysis.
• Figure 4: Kinematic and geometric parameters used in the torque analysis. The reaction force arises through the rigid linkage supported by the auxetic structure during contact.
• Figure 5: Nonlinear variation of motor torque with the desired grasping force for the gripper fingers.