FruitTouch: A Perceptive Gripper for Gentle and Scalable Fruit Harvesting

TL;DR

FruitTouch tackles the challenge of gentle, scalable autonomous fruit harvesting by integrating high-resolution vision-based tactile sensing into a compact, low-cost parallel-jaw gripper that uses a single central camera and mirror-based optics to cover both sensing surfaces. The perception pipeline reconstructs contact geometry, estimates 3D normal and shear forces, detects slip, and assesses softness in-hand to enable tactile-informed closed-loop control. Experimental results demonstrate accurate force estimation (normal $R^2=0.951$, shear $R^2=0.903$), reliable slip detection, and strong cross-fruit softness ranking, along with successful autonomous harvesting in lab settings. This work presents a scalable, market-ready approach to damage-free harvesting that could reduce labor dependence and improve harvest efficiency.

Abstract

The automation of fruit harvesting has gained increasing significance in response to rising labor shortages. A sensorized gripper is a key component of this process, which must be compact enough for confined spaces, able to stably grasp diverse fruits, and provide reliable feedback on fruit conditions for efficient harvesting. To address this need, we propose FruitTouch, a compact gripper that integrates high-resolution, vision-based tactile sensing through an optimized optical design. This configuration accommodates a wide range of fruit sizes while maintaining low cost and mechanical simplicity. Tactile images captured by an embedded camera provide rich information for real-time force estimation, slip detection, and softness prediction. We validate the gripper in real-world fruit harvesting experiments, demonstrating robust grasp stability and effective damage prevention.

Figures (7)

• Figure 1: Demonstration of the FruitTouch gripper harvesting cherry tomatoes. We designed the gripper for compactness, low cost, and scalability, while the perception system measures high-resolution contact geometry, 3D force, slip, and object softness. The design integrates mechanical efficiency with rich tactile sensing to enable reliable and efficient fruit harvesting.
• Figure 2: Mechanical and Optical Design of FruitTouch Gripper. (A) Mechanical Design. The gripper components are designed for scalability, enabling harvesting of fruits with varying sizes. We use gear-and-rack mechanism to provide actuation. Each finger consists of a soft silicone sensing surface supported by a transparent acrylic sheet, with three LED strips ensuring uniform illumination. A centrally mounted camera, in combination with mirrors, provides comprehensive coverage of both sensing surfaces. (B) Sensor Optical Design. The mirror configuration is optimized to maximize the sensing area while maintaining low distortion across different finger distances. Simulated outputs are shown for both the open and closed states of the gripper.
• Figure 3: Tactile Sensing Pipeline. (A) Contact-geometry reconstruction pipeline. Top: Raw camera output that contains reading from both fingers. (i) Unwarped contact frames on one sensor. (ii) Background-subtracted difference images. (iii) Estimated surface normals. (iv) Reconstructed 3D shape (shown in the form of a heightmap). (B) Example tactile images and the reconstructed shape for a cherry tomato, a strawberry, and a blackberry.
• Figure 4: (A) Softness Prediction Pipeline. We randomly segment tactile image sequences and the corresponding normal force signals from the pressing process of the same fruit, and extract embeddings using a shared backbone. A lightweight ranking head then predicts whether the test object is harder or softer than the reference. (B) Softness perception results. Each bar shows classification accuracy for a specific hardness within a fruit type (comparisons within the same fruit). The red dotted line marks random guess (50%).
• Figure 5: Setup and Results for Force Prediction.(A) Experimental setup with the gripper mounted on an ATI Nano Force/Torque sensor, which measures the ground-truth force. (B, C) Results of force estimation. The middle panel compares normal force ($F_n$), while the right panel compares shear force ($F_{\tau}$). For clarity, the magnitude of the shear force is used for visualization. The solid green line indicates the RANSAC fit, and the dotted red line denotes the identity line ($y{=}x$).