Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation

Branden Romero, Hao-Shu Fang, Pulkit Agrawal, Edward Adelson

TL;DR

EyeSight Hand presents a fully actuated 7-DoF humanoid hand with integrated GelSim(ple) vision-based tactile sensors and a quasi-direct drive actuation scheme to achieve human-like strength and speed while remaining robust for large-scale data collection. The authors co-design kinematics, actuation, and tactile sensing to enable dense tactile feedback across the hand and implement an imitation-learning baseline using ACT CVAE with a vision dropout strategy to exploit tactile cues. Real-world experiments on bottle opening, plate pick-and-place, and plasticine cutting show that tactile sensing substantially improves task success rates, with dropout training further enhancing robustness. The work advances dexterous manipulation by bridging hardware design with tactile-enabled learning, offering a low-cost, data-friendly platform for robotics research.

Abstract

In this work, we introduce the EyeSight Hand, a novel 7 degrees of freedom (DoF) humanoid hand featuring integrated vision-based tactile sensors tailored for enhanced whole-hand manipulation. Additionally, we introduce an actuation scheme centered around quasi-direct drive actuation to achieve human-like strength and speed while ensuring robustness for large-scale data collection. We evaluate the EyeSight Hand on three challenging tasks: bottle opening, plasticine cutting, and plate pick and place, which require a blend of complex manipulation, tool use, and precise force application. Imitation learning models trained on these tasks, with a novel vision dropout strategy, showcase the benefits of tactile feedback in enhancing task success rates. Our results reveal that the integration of tactile sensing dramatically improves task performance, underscoring the critical role of tactile information in dexterous manipulation.

EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation

TL;DR

EyeSight Hand presents a fully actuated 7-DoF humanoid hand with integrated GelSim(ple) vision-based tactile sensors and a quasi-direct drive actuation scheme to achieve human-like strength and speed while remaining robust for large-scale data collection. The authors co-design kinematics, actuation, and tactile sensing to enable dense tactile feedback across the hand and implement an imitation-learning baseline using ACT CVAE with a vision dropout strategy to exploit tactile cues. Real-world experiments on bottle opening, plate pick-and-place, and plasticine cutting show that tactile sensing substantially improves task success rates, with dropout training further enhancing robustness. The work advances dexterous manipulation by bridging hardware design with tactile-enabled learning, offering a low-cost, data-friendly platform for robotics research.

Abstract

In this work, we introduce the EyeSight Hand, a novel 7 degrees of freedom (DoF) humanoid hand featuring integrated vision-based tactile sensors tailored for enhanced whole-hand manipulation. Additionally, we introduce an actuation scheme centered around quasi-direct drive actuation to achieve human-like strength and speed while ensuring robustness for large-scale data collection. We evaluate the EyeSight Hand on three challenging tasks: bottle opening, plasticine cutting, and plate pick and place, which require a blend of complex manipulation, tool use, and precise force application. Imitation learning models trained on these tasks, with a novel vision dropout strategy, showcase the benefits of tactile feedback in enhancing task success rates. Our results reveal that the integration of tactile sensing dramatically improves task performance, underscoring the critical role of tactile information in dexterous manipulation.
Paper Structure (22 sections, 1 equation, 6 figures)

This paper contains 22 sections, 1 equation, 6 figures.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND AND RELATED WORK
  3. Humanoid Robotic Hands
  4. Hands with Vision Based Tactile Sensors
  5. Learning for Dexterous Hand
  6. Mechanical Design
  7. Kinematics
  8. Actuation
  9. Electronics
  10. TACTILE SENSOR DESIGN
  11. Illumination
  12. Implementation Details
  13. EXPERIMENT SETUP
  14. Teleoperation
  15. Task Specification
  16. ...and 7 more sections

Figures (6)

  • Figure 0: (a-b) Comparison of the dimension of a human hand with the dimensions of the EyeSight Hand. (c-d) Diagram illustrating the kinematics of the (c) index/middle finger and (d) thumb as well as the placements of the motors and transmissions. Axis of rotation is shown with green arrows.
  • Figure 1: Simplified kinematics of the human hand human_kin.
  • Figure 2: (a) Dimension and exploded view of the GelSim(ple) sensor at the tip of the thumb. (b) Example raw outputs of the GelSim(ple) and the approximate simulation of the corresponding deformation.
  • Figure 3: Illustrations of the evaluation task. (a) The robot performs lid opening, where the robot must first approach the bottle, then constrain it with the index, middle finger, and palm, and then use the thumb to open the bottle. (b) The robot must approach a set of stack places, and then slide one over the other until it is able to grasp the plate. It then picks the plate and places next to the other plate. (c) The robot must remove a knife from a box, and use it to cut a plasticine roll fully.
  • Figure 4: Illustration of our input to the policy along with the architecture of the policy network.
  • ...and 1 more figures