Rotating without Seeing: Towards In-hand Dexterity through Touch

TL;DR

<3-5 sentence high-level summary>Touch Dexterity demonstrates that a multi-finger robot hand can achieve in-hand rotation using only touch, by leveraging a dense array of low-cost binary force sensors distributed over the palm, links, and fingertips. The authors train a reinforcement learning policy in a physics-based simulator (IsaacGym) with extensive domain randomization and transfer it directly to a real Allegro Hand, achieving rotation of unseen objects without vision. Through rigorous ablations and qualitative analyses, they show that tactile information is essential for robust, generalizable manipulation and that both palm and fingertip sensors contribute meaningfully. The work highlights a practical, Sim2Real-friendly path toward tactile-only dexterity and suggests avenues for denser sensing and more complex touch-driven tasks in the future.

Abstract

Tactile information plays a critical role in human dexterity. It reveals useful contact information that may not be inferred directly from vision. In fact, humans can even perform in-hand dexterous manipulation without using vision. Can we enable the same ability for the multi-finger robot hand? In this paper, we present Touch Dexterity, a new system that can perform in-hand object rotation using only touching without seeing the object. Instead of relying on precise tactile sensing in a small region, we introduce a new system design using dense binary force sensors (touch or no touch) overlaying one side of the whole robot hand (palm, finger links, fingertips). Such a design is low-cost, giving a larger coverage of the object, and minimizing the Sim2Real gap at the same time. We train an in-hand rotation policy using Reinforcement Learning on diverse objects in simulation. Relying on touch-only sensing, we can directly deploy the policy in a real robot hand and rotate novel objects that are not presented in training. Extensive ablations are performed on how tactile information help in-hand manipulation.Our project is available at https://touchdexterity.github.io.

Figures (11)

• Figure 1: We propose Touch Dexterity, a new dexterous manipulation system to perform in-hand object rotation with only touch sensing. On the left, we show our hardware setup with 16 FSR sensors attached to an Allegro hand. We train our policy in simulation on rotating diverse objects around different axes. Our trained policy can be directly transferred to the real robot hand and can rotate novel/unseen objects successfully.
• Figure 2: Two major functionalities of our sensors: sensing (i) the objects' in-hand position, and (ii) the critical contact during the dexterous manipulation process. Note that we use finger cots to increase the friction and we still have force-sensing resistors inside the finger cots.
• Figure 3: Overview of the control process. The state contains tactile information, joint position, previous target, and task information like rotation axis (not shown in the figure). The policy then uses the stacked state to get the relative action, and the next target joint position is calculated. The new target is then fed to a PD controller.
• Figure 4: Illustration of the calculation of rotation angle $\Delta \theta$: The object rotates alone Axis $k$ and here we visualize the rotation angle $\Delta \theta$ in the Normal Plane.
• Figure 5: The object sets used in our experiments. The full object set in the real world can be found in the supplementary material.