AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch

TL;DR

This paper presents AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch and finds rich multi-fingered tactile sensing can detect unstable grasps and provide a reactive behavior that improves the robustness of the policy.

Abstract

Human hands are capable of in-hand manipulation in the presence of different hand motions. For a robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training a dense tactile policy in simulation and present a sim-to-real method for rich tactile sensing to achieve zero-shot policy transfer. Our formulation allows the training of a unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. In our experiments, we highlight the benefit of capturing detailed contact information when handling objects of varying properties. Interestingly, we found rich multi-fingered tactile sensing can detect unstable grasps and provide a reactive behavior that improves the robustness of the policy. The project website can be found at https://maxyang27896.github.io/anyrotate/.

Figures (16)

• Figure 1: Setup: A 4-fingered 16-DoF tactile robot hand attached to a UR5 performing multi-axis in-hand object rotation (top), with experiments in six key hand orientations with respect to gravity: palm up, palm down, thumb up, thumb down, base up and base down (bottom).
• Figure 2: Overview of the approach. Left: The object rotation problem is formulated as an object reorientation to a moving goal. Auxiliary goal keypoints are used to define target poses about the chosen rotation axis. Right: training a policy using teacher-student policy distillation. The teacher is trained using privileged information with RL and the student aims to imitate the teacher's action given real-world observations. Privileged information and real-world observation are shown.
• Figure 3: Tactile prediction pipeline; a) tactile images are preprocessed to grey-scale filtered images, b) models extract explicit contact features, c) visualization of the tactile features: contact pose and contact force are represented by the center and area of the shaded circle respectively.
• Figure 4: Top: Simulation test object set from brahmbhatt2019contactdb. Bottom: Real everyday objects.
• Figure 5: Comparison of different tactile policies on test object sets in simulation. We report on average rotation achieved per episode (Rot) and average episode length (EpLen) for arbitrary rotation axis and hand direction.