Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Novel Approach to Tomato Harvesting Using a Hybrid Gripper with Semantic Segmentation and Keypoint Detection

Shahid Ansari, Mahendra Kumar Gohil, Bishakh Bhattacharya

TL;DR

This work addresses the challenge of selectively harvesting soft fruits, specifically tomatoes, by integrating a novel hybrid gripper with soft auxetic fingers and a rigid exoskeleton, a depth/RGB vision system for semantic segmentation of ripeness, and keypoint detection of the pedicel and tomato center. The method couples a scotch-yoke driven actuator with six soft fingers and a latex basket to achieve conformable yet strong grasping, while a conical separator enables pre-grasp separation from neighboring fruits. A Detectron-2-based vision pipeline detects ripe status and keypoints; PSO-based trajectory planning translates perception into safe, efficient manipulation. Experimental results show ~80% success and an average cycle time of ~24.34 seconds, with future work aimed at improving pedicel cutting reliability and overall gripper optimization for robust field deployment.

Abstract

Current agriculture and farming industries are able to reap advancements in robotics and automation technology to harvest fruits and vegetables using robots with adaptive grasping forces based on the compliance or softness of the fruit or vegetable. A successful operation depends on using a gripper that can adapt to the mechanical properties of the crops. This paper proposes a new robotic harvesting approach for tomato fruit using a novel hybrid gripper with a soft caging effect. It uses its six flexible passive auxetic structures based on fingers with rigid outer exoskeletons for good gripping strength and shape conformability. The gripper is actuated through a scotch-yoke mechanism using a servo motor. To perform tomato picking operations through a gripper, a vision system based on a depth camera and RGB camera implements the fruit identification process. It incorporates deep learning-based keypoint detection of the tomato's pedicel and body for localization in an occluded and variable ambient light environment and semantic segmentation of ripe and unripe tomatoes. In addition, robust trajectory planning of the robotic arm based on input from the vision system and control of robotic gripper movements are carried out for secure tomato handling. The tunable grasping force of the gripper would allow the robotic handling of fruits with a broad range of compliance.

A Novel Approach to Tomato Harvesting Using a Hybrid Gripper with Semantic Segmentation and Keypoint Detection

TL;DR

This work addresses the challenge of selectively harvesting soft fruits, specifically tomatoes, by integrating a novel hybrid gripper with soft auxetic fingers and a rigid exoskeleton, a depth/RGB vision system for semantic segmentation of ripeness, and keypoint detection of the pedicel and tomato center. The method couples a scotch-yoke driven actuator with six soft fingers and a latex basket to achieve conformable yet strong grasping, while a conical separator enables pre-grasp separation from neighboring fruits. A Detectron-2-based vision pipeline detects ripe status and keypoints; PSO-based trajectory planning translates perception into safe, efficient manipulation. Experimental results show ~80% success and an average cycle time of ~24.34 seconds, with future work aimed at improving pedicel cutting reliability and overall gripper optimization for robust field deployment.

Abstract

Current agriculture and farming industries are able to reap advancements in robotics and automation technology to harvest fruits and vegetables using robots with adaptive grasping forces based on the compliance or softness of the fruit or vegetable. A successful operation depends on using a gripper that can adapt to the mechanical properties of the crops. This paper proposes a new robotic harvesting approach for tomato fruit using a novel hybrid gripper with a soft caging effect. It uses its six flexible passive auxetic structures based on fingers with rigid outer exoskeletons for good gripping strength and shape conformability. The gripper is actuated through a scotch-yoke mechanism using a servo motor. To perform tomato picking operations through a gripper, a vision system based on a depth camera and RGB camera implements the fruit identification process. It incorporates deep learning-based keypoint detection of the tomato's pedicel and body for localization in an occluded and variable ambient light environment and semantic segmentation of ripe and unripe tomatoes. In addition, robust trajectory planning of the robotic arm based on input from the vision system and control of robotic gripper movements are carried out for secure tomato handling. The tunable grasping force of the gripper would allow the robotic handling of fruits with a broad range of compliance.

Paper Structure

This paper contains 8 sections, 2 equations, 9 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. DESIGN
  3. Materials and components
  4. Torque requirement calculation
  5. Tomato Semantic Segmentation with Tomato Center and Pedicel Keypoint Detection
  6. Robotic Gripper Control Strategy for Tomato Handling: Workflow and Integration
  7. Experimental evaluation
  8. CONCLUSIONS

Figures (9)

  • Figure 1: The figure shows a 3D CAD model of the robotic gripper mechanical design with details of the main parts of the gripper, which remains active throughout the tomato harvesting process. The top left shows the cutting mechanism, which is actuated by a micro servo motor; the middle left shows the soft auxetic structure for gentle grasp; the bottom left shows the separator leaves in a closed state forming a conical frustum for tomato separation, and top right shows the servo actuated scotch yoke mechanism to drive the gripper fingers.
  • Figure 2: The figure shows the force of interaction of a single gripper finger as the gripper grasps the tomato with soft auxetic structure in contact during the picking operation for static force analysis
  • Figure 3: The figure shows the various kinematic parameters and the force acting on the rigid link of the finger, which is the reaction force coming from the auxetic structure support during the interaction with the tomato while grasping.
  • Figure 4: The figure shows the variation of motor torque required to provide the desired grasping force to the gripper fingers for holding the tomato
  • Figure 5: In this figure, the tomato picking operation is performed experimentally in the lab environment under controlled conditions.From (a-b): Gripper Approached towards the tomato cluster (c-d): The target tomato is separated from the cluster (e). The cutting action of the tomato pedicel is performed (f)Soft grasping of the tomato under the influence of auxetic fingers (g)Gripper departs towards the punnet (h): Tomato is released inside a punnet
  • ...and 4 more figures