Tactile-based Object Retrieval From Granular Media

TL;DR

GEOTACT tackles buried-object retrieval in granular media using a tactile-only, end-to-end reinforcement learning framework. By integrating a pushing behavior that reduces uncertainty and a curriculum that enables sim-to-real transfer, the approach achieves robust grasping and retrieval across unseen objects, demonstrated in both simulation and a real UR5 robot with DISCO tactile fingers. The work highlights significant practical impact for hazardous environments, showing zero-shot hardware transfer from tabletop training to granular media and achieving retrieval across a broad object set, including rigid, deformable, and articulated shapes. This tactile-driven method offers a promising path for autonomous search and extraction tasks where vision is unreliable or unavailable.

Abstract

We introduce GEOTACT, the first robotic system capable of grasping and retrieving objects of potentially unknown shapes buried in a granular environment. While important in many applications, ranging from mining and exploration to search and rescue, this type of interaction with granular media is difficult due to the uncertainty stemming from visual occlusion and noisy contact signals. To address these challenges, we use a learning method relying exclusively on touch feedback, trained end-to-end with simulated sensor noise. We show that our problem formulation leads to the natural emergence of learned pushing behaviors that the manipulator uses to reduce uncertainty and funnel the object to a stable grasp despite spurious and noisy tactile readings. We introduce a training curriculum that bootstraps learning in simulated granular environments, enabling zero-shot transfer to real hardware. Despite being trained only on seven objects with primitive shapes, our method is shown to successfully retrieve 35 different objects, including rigid, deformable, and articulated objects with complex shapes. Videos and additional information can be found at https://jxu.ai/geotact.

Figures (10)

• Figure 1: Retrieving buried object from granular media. In this task, an object is buried under granular beads. We mount a parallel gripper with two tactile fingers on a robot arm to retrieve the buried object using only tactile feedback. Our learned policy uses a series of pushing actions to funnel the buried object into a stable grasp and then successfully retrieve the target object.
• Figure 2: Two types of contact inside granular media: ubiquitous contact and pushing contact. Ubiquitous contacts are between the finger and the granular media, but pushing contacts are sensed only when the finger has started pushing the buried object.
• Figure 3: Curriculum training strategy. The numbering of the frames corresponds to their order within a single episode. We first pre-train our policy on a tabletop environment and then fine-tune it inside granular media. We then zero-shot deploy our policy on the real hardware for evaluation. This curriculum allows the policy to converge to a higher performance within a shorter training time.
• Figure 4: Parallel gripper with multi-curved tactile fingers. Our gripper is effective at exploring under granular media due to the DISCO tactile fingers' streamlined shapes and all-around sensing coverage.
• Figure 5: 35 objects used in our real-robot experiments. Our method is trained on a set of 7 objects. It is then evaluated quantitatively (10 attempts on each object) on both the set of 7 training objects and a set of 6 previously unseen objects. In additional testing, we also show that our method can work on an extended set of 22 unseen objects with at most three attempts per object. The images of training and unseen objects do not correctly reflect their relative sizes. See the demonstration videos for each object on our website for accurate size information.