Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continuous Wrist Control on the Hannes Prosthesis: a Vision-based Shared Autonomy Framework

Federico Vasile, Elisa Maiettini, Giulia Pasquale, Nicolò Boccardo, Lorenzo Natale

TL;DR

This work tackles the challenge of controlling the wrist in transradial prostheses during reach-to-grasp by introducing an eye-in-hand, vision-based shared-autonomy framework that continuously drives the wrist DoFs during the approach. The system comprises a three-phase pipeline (transport, rotation, grasping) where visual servoing keeps the target centered during transport, followed by a parts-based wrist rotation driven by object-part predictions, and final grasping controlled by EMG. Key contributions include the DINOv2Det-based object-parts segmentation for fine-grained wrist orientation, a synthetic data generation tool to support sim-to-real training, and a deployment on the Hannes prosthetic arm with real-time operation. The results demonstrate improved natural wrist motion and continuous control, with potential to reduce compensatory movements and user cognitive load in active prosthetic use, though validation with amputees remains future work.

Abstract

Most control techniques for prosthetic grasping focus on dexterous fingers control, but overlook the wrist motion. This forces the user to perform compensatory movements with the elbow, shoulder and hip to adapt the wrist for grasping. We propose a computer vision-based system that leverages the collaboration between the user and an automatic system in a shared autonomy framework, to perform continuous control of the wrist degrees of freedom in a prosthetic arm, promoting a more natural approach-to-grasp motion. Our pipeline allows to seamlessly control the prosthetic wrist to follow the target object and finally orient it for grasping according to the user intent. We assess the effectiveness of each system component through quantitative analysis and finally deploy our method on the Hannes prosthetic arm. Code and videos: https://hsp-iit.github.io/hannes-wrist-control.

Continuous Wrist Control on the Hannes Prosthesis: a Vision-based Shared Autonomy Framework

TL;DR

This work tackles the challenge of controlling the wrist in transradial prostheses during reach-to-grasp by introducing an eye-in-hand, vision-based shared-autonomy framework that continuously drives the wrist DoFs during the approach. The system comprises a three-phase pipeline (transport, rotation, grasping) where visual servoing keeps the target centered during transport, followed by a parts-based wrist rotation driven by object-part predictions, and final grasping controlled by EMG. Key contributions include the DINOv2Det-based object-parts segmentation for fine-grained wrist orientation, a synthetic data generation tool to support sim-to-real training, and a deployment on the Hannes prosthetic arm with real-time operation. The results demonstrate improved natural wrist motion and continuous control, with potential to reduce compensatory movements and user cognitive load in active prosthetic use, though validation with amputees remains future work.

Abstract

Most control techniques for prosthetic grasping focus on dexterous fingers control, but overlook the wrist motion. This forces the user to perform compensatory movements with the elbow, shoulder and hip to adapt the wrist for grasping. We propose a computer vision-based system that leverages the collaboration between the user and an automatic system in a shared autonomy framework, to perform continuous control of the wrist degrees of freedom in a prosthetic arm, promoting a more natural approach-to-grasp motion. Our pipeline allows to seamlessly control the prosthetic wrist to follow the target object and finally orient it for grasping according to the user intent. We assess the effectiveness of each system component through quantitative analysis and finally deploy our method on the Hannes prosthetic arm. Code and videos: https://hsp-iit.github.io/hannes-wrist-control.

Paper Structure

This paper contains 15 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. CONTROL ARCHITECTURE
  4. Shared-autonomy prosthetic grasping
  5. Visual servoing control
  6. Object parts segmentation
  7. DATASETS
  8. Real dataset annotation
  9. Synthetic dataset generation
  10. EXPERIMENTS
  11. The Hannes prosthetic device
  12. Visual servoing simulation
  13. Object parts segmentation
  14. APPLICATION ON THE HANNES PROSTHESIS
  15. CONCLUSIONS

Figures (4)

  • Figure 1: The phases of the prosthetic grasping pipeline.
  • Figure 2: The natural wrist motion (a-b) and a non-natural motion (c) as the user drives the arm around the object.
  • Figure 3: The 15 YCB objects with labeled object parts. The red and green masks encode the top and side grasps, respectively. The non-highlighted object parts are labeled as no grasp.
  • Figure 4: The trajectories generated by the visual servo schemes for two different WFE initial configurations (a-b). The Hannes arm (c).