Disambiguate Gripper State in Grasp-Based Tasks: Pseudo-Tactile as Feedback Enables Pure Simulation Learning

TL;DR

This paper tackles gripper state ambiguity in grasp-based manipulation by introducing pseudo-tactile feedback from a force-controlled gripper to disambiguate grasp state, enabling a binary gripper observation and pure simulation learning without extra hardware. A simple closed-loop controller converts empty close states into empty open states, removing misleading correlations between gripper state and success. The approach is validated on three real-world tasks, showing robust disturbance resilience and superior performance when trained entirely in simulation, leveraging admittance control to bridge sim-to-real gaps. Overall, the method demonstrates that pseudo-tactile feedback can unlock scalable, simulation-driven policy learning for robust grasp-based manipulation.

Abstract

Grasp-based manipulation tasks are fundamental to robots interacting with their environments, yet gripper state ambiguity significantly reduces the robustness of imitation learning policies for these tasks. Data-driven solutions face the challenge of high real-world data costs, while simulation data, despite its low costs, is limited by the sim-to-real gap. We identify the root cause of gripper state ambiguity as the lack of tactile feedback. To address this, we propose a novel approach employing pseudo-tactile as feedback, inspired by the idea of using a force-controlled gripper as a tactile sensor. This method enhances policy robustness without additional data collection and hardware involvement, while providing a noise-free binary gripper state observation for the policy and thus facilitating pure simulation learning to unleash the power of simulation. Experimental results across three real-world grasp-based tasks demonstrate the necessity, effectiveness, and efficiency of our approach.

Figures (5)

• Figure 1: (a) Policies for grasp-based tasks are susceptible to gripper disturbance (forcing the gripper to close at the moment marked by the red dashed box). While the robot does not grasp the object, the policy outputs the action command for the post-grasp phase. (b) We propose pseudo-tactile feedback to accurately reflect the grasp state, effectively disambiguating the gripper state.
• Figure 2: A force-controlled gripper can act as a tactile sensor providing pseudo-tactile information. We designed a closed-loop gripper controller with pseudo-tactile information as feedback to convert the empty close state into the empty open state, disambiguating gripper state.
• Figure 3: Once the gripper state is disambiguated, there is no need to perform gripper disturbance during data collection, and the policy can use the binary gripper state observation, enabling pure simulation training. We design a state-based expert policy for automatic data collection and apply real-to-sim techniques, randomization, and admittance control to mitigate the visual sim-to-real gap and reduce the stress caused by kinematic discrepancies.
• Figure 4: (a) Real-world oven-opening comparisons. We introduced a disturbance at the moment marked by the red dashed box by forcing the gripper to close. While the baseline models failed due to the disturbance, ours completed the task successfully. (b) One of rollouts of a policy trained using simulation data with gripper randomization in ablation studies. It successfully recovered from the empty grasp (1.5s-4s) and grasped the handle (4s-9s), but could not complete the task.
• Figure 5: Visualize some rollouts of our policy both in simulation and real world. The “little devil” symbol marks the moments when disturbances are introduced, and the checkmark symbol indicates the policy’s ability to recover from disturbance in the real world.