Quantitative Hardness Assessment with Vision-based Tactile Sensing for Fruit Classification and Grasping

TL;DR

This work tackles non-destructive fruit hardness estimation for robotic harvesting by leveraging a vision-based tactile sensor to perform force decomposition and infer hardness from average normal force dynamics. The core framework attaches a high-frequency tactile sensor (DM-Tac) to a gripper, enabling decomposition into normal and shear forces and using $H = g\left( \dfrac{dF_z}{d\delta}, \dfrac{dF_z}{dt}, \delta_{\max} \right)$ (with a simplified form $H = g'(\dfrac{dF_z}{dt})$) to quantify hardness. Two operational criteria based on grasping distance and a normal-force threshold were proposed and validated across fruit varieties (cucumber, strawberry, grape) and ripeness tracking in mango and kiwi, demonstrating robust discrimination and non-destructive assessment. The approach enables rapid, safe, and adaptable robotic handling, with future work aimed at incorporating full 3D force data to enhance estimation and extend tasks in agricultural robotics.

Abstract

Accurate estimation of fruit hardness is essential for automated classification and handling systems, particularly in determining fruit variety, assessing ripeness, and ensuring proper harvesting force. This study presents an innovative framework for quantitative hardness assessment utilizing vision-based tactile sensing, tailored explicitly for robotic applications in agriculture. The proposed methodology derives normal force estimation from a vision-based tactile sensor, and, based on the dynamics of this normal force, calculates the hardness. This approach offers a rapid, non-destructive evaluation through single-contact interaction. The integration of this framework into robotic systems enhances real-time adaptability of grasping forces, thereby reducing the likelihood of fruit damage. Moreover, the general applicability of this approach, through a universal criterion based on average normal force dynamics, ensures its effectiveness across a wide variety of fruit types and sizes. Extensive experimental validation conducted across different fruit types and ripeness-tracking studies demonstrates the efficacy and robustness of the framework, marking a significant advancement in the domain of automated fruit handling.

Figures (9)

• Figure 1: Comparison of original and deformed images.
• Figure 2: Different grasp distance for old cucumber (very soft).
• Figure 3: Three kinds of fruits: cucumbers, strawberries, and grapes.
• Figure 4: Continue grasping with the same grasp distance. The normal force value only represents relative magnitude.
• Figure 5: Step grasping based on the normal force threshold.