DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity

TL;DR

This paper introduces a multi-finger robot system designed to manipulate objects using the sense of touch, without relying on vision, to enable robots to perform blind manipulation by using tactile sensation to compensate for the information gap caused by the absence of vision.

Abstract

The sense of touch is an essential ability for skillfully performing a variety of tasks, providing the capacity to search and manipulate objects without relying on visual information. In this paper, we introduce a multi-finger robot system designed to manipulate objects using the sense of touch, without relying on vision. For tasks that mimic daily life, the robot uses its sense of touch to manipulate randomly placed objects in dark. The objective of this study is to enable robots to perform blind manipulation by using tactile sensation to compensate for the information gap caused by the absence of vision, given the presence of prior information. Training the policy through reinforcement learning in simulation and transferring the trained policy to the real environment, we demonstrate that blind manipulation can be applied to robots without vision. In addition, the experiments showcase the importance of tactile sensing in the blind manipulation tasks. Our project page is available at https://lee-kangwon.github.io/dextouch/

Figures (6)

• Figure 2: We propose DexTouch, a dexterous robotic system with tactile-based blind manipulation. The robotic system consisted of a UR5e arm and an AllegroHand with 16 attached touch sensors. Policies were trained in simulation and then deployed to the real environment without fine-tuning.
• Figure 3: Pipeline of the system. Left: visualization of the Allegro hand with tactile sensors (red indicators) in simulation and real-world. Right: visualization for all three tasks in simulation and training pipeline. The blue areas and arrows represent the random range of the target object. The state contains robot proprioception, tactile information, and task information like goal position and random range from origin state. Then the policy uses this state to get actions.
• Figure 4: The object sets used in grasping object task. The objects in red box are the unseen objects.
• Figure 5: Visualization of the reward for each task. Yellow arrows represent reward $r_{reach}$ based on the distance between each fingertip and the target object. Blue arrows represent reward $r_{execute}$ based on manipulative actions to perform each task. The total reward function is the sum of the two rewards.
• Figure 6: Training process depending on (a) the attachment and sensitivity of the tactile sensor and (b) the location of the tactile sensor and other type of sensor. The results are averaged on 3 seeds, and the shaded area indicates the standard deviation. The x-axis is the training steps. The y-axis of the upper row is the success rate for each task and the lower row is the episodic return.