DexFruit: Dexterous Manipulation and Gaussian Splatting Inspection of Fruit

TL;DR

DexFruit advances gentle robotic manipulation of soft fruits by integrating optical tactile sensing with diffusion-policy learning, enabling robust grasping while minimizing damage. It introduces FruitSplat, a 3D Gaussian Splatting-based damage analysis pipeline that fuses 2D bruise/fruit masks into a quantitative 3D representation, facilitating post-manipulation quality assessment. Across 630 trials on strawberries, tomatoes, and blackberries, the approach achieves high grasp success and reduced bruising, with tactile feedback and modality switching yielding substantial gains over vision- or touch-only baselines. The work highlights practical implications for harvesting and post-harvest quality control, providing both a manipulation framework and an accessible damage visualization/quantification tool.

Abstract

DexFruit is a robotic manipulation framework that enables gentle, autonomous handling of fragile fruit and precise evaluation of damage. Many fruits are fragile and prone to bruising, thus requiring humans to manually harvest them with care. In this work, we demonstrate by using optical tactile sensing, autonomous manipulation of fruit with minimal damage can be achieved. We show that our tactile informed diffusion policies outperform baselines in both reduced bruising and pick-and-place success rate across three fruits: strawberries, tomatoes, and blackberries. In addition, we introduce FruitSplat, a novel technique to represent and quantify visual damage in high-resolution 3D representation via 3D Gaussian Splatting (3DGS). Existing metrics for measuring damage lack quantitative rigor or require expensive equipment. With FruitSplat, we distill a 2D strawberry mask as well as a 2D bruise segmentation mask into the 3DGS representation. Furthermore, this representation is modular and general, compatible with any relevant 2D model. Overall, we demonstrate a 92% grasping policy success rate, up to a 20% reduction in visual bruising, and up to an 31% improvement in grasp success rate on challenging fruit compared to our baselines across our three tested fruits. We rigorously evaluate this result with over 630 trials. Please checkout our website at https://dex-fruit.github.io .

Figures (5)

• Figure 1: Our framework, DexFruit, enables safe manipulation of fragile fruit using optical tactile feedback. We further present FruitSplat, a method for accurate 3D reconstruction, automated segmentation, and bruise localization in strawberries.
• Figure 2: Here we show the FruitSplat pipeline. Prior to training, the 3D Gaussian Splat the images are processed through 2 models. YOLO segments the bruising, while SAM2 segments the strawberry. These two additional masks are used to supervise additional Gaussian parameters.
• Figure 3: On the left (a) is the experimental setup. The two sensing modalities used are highlighted in red: the DenseTact optical tactile sensor and IntelRealsense D435i. On the right (b) is the force measuring apparatus which is used to quantify internal damage in the fruit. (c) Shows that apparatus we utilize for scanning the strawberries.
• Figure 4: We show the qualitative results of our method compared to the baselines. On the left, we show the same strawberries before and after the experiments along with the corresponding bruise mask. All the strawberry images are rendered from FruitSplat. Blackberries are shown on the right while tomatoes are not shown as they do not show visible damage.
• Figure 5: Above we show the success rates across methods and fruit size. Below on the left we quantify the internal fruit damage measured as a percent change in stiffness, while on the right we show the results from FruitSplat representing external damage.