Sim2Real Manipulation on Unknown Objects with Tactile-based Reinforcement Learning

TL;DR

This paper proposes to perform Reinforcement Learning with only visual tactile sensing inputs on diverse objects in a physical simulator to enable the policy to generalize to unseen objects.

Abstract

Using tactile sensors for manipulation remains one of the most challenging problems in robotics. At the heart of these challenges is generalization: How can we train a tactile-based policy that can manipulate unseen and diverse objects? In this paper, we propose to perform Reinforcement Learning with only visual tactile sensing inputs on diverse objects in a physical simulator. By training with diverse objects in simulation, it enables the policy to generalize to unseen objects. However, leveraging simulation introduces the Sim2Real transfer problem. To mitigate this problem, we study different tactile representations and evaluate how each affects real-robot manipulation results after transfer. We conduct our experiments on diverse real-world objects and show significant improvements over baselines for the pivoting task. Our project page is available at https://tactilerl.github.io/.

Figures (5)

• Figure 1: We study the task of pivoting an object to a target angle with only tactile observations. Our tactile-based policy, trained with Reinforcement Learning purely in simulation, successfully transfers to the real robot without real-world data. The first row in each block visualizes the initial state of different episodes, while the second row demonstrates the final execution results.
• Figure 2: Setup. Two tactile sensors are mounted on the gripper's fingertips. Tactile readings are processed using our proposed approach and then serve as the input for training RL policies.
• Figure 3: Object categories, Visualization of angle estimation using PCA and failure case of PC policy. We train with diverse objects in simulation (left image) and evaluate with a real robot (right image). In (b), we illustrate PCA angle estimation: a success (left) and a failure (right), with the red line indicating the estimated orientation. For (c), we present a failure case of the Point Cloud policy.
• Figure 4: Training curves. We report the training curves for each task with two metrics: reward and success rate in the simulation. Given that the tactile policies exhibit similar reward and success rate trends, we present the results for Tac. RGB (Aug) for simplicity. Note that Oracle Angle achieves the best performance because it uses the ground truth object pose, which serves as an upper bound in the simulation experiment.
• Figure 5: Real and Simulation Experiment for the pivoting. We evaluated our pivoting task policy using tactile inputs in RGB, difference, and binary formats. The first two columns display the task's initial and final states on a real robot and in simulation. Rows two and three present tactile images from the start and end frames captured by both grippers. The last two rows sequentially showcase the RGB (RGB), difference (Diff), and binary (Binary) tactile images.