Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

Federico Vasile, Elisa Maiettini, Giulia Pasquale, Astrid Florio, Nicolò Boccardo, Lorenzo Natale

TL;DR

This work tackles prosthetic grasping with multiple grasp types by using an eye-in-hand, vision-based approach to pre-shape classification from RGB sequences. It introduces a synthetic data generation pipeline with realistic, human-like arm trajectories and domain randomization to train models, addressing data scarcity and enhancing generalization. The study demonstrates that synthetic data can match or exceed real-data performance in robust, cross-condition scenarios and validates the pipeline on the Hannes prosthesis. Overall, the paper shows that allowing different grasps for distinct object parts and leveraging synthetic data improves control transparency and user satisfaction in shared-autonomy prosthetic systems.

Abstract

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-in-hand systems automatically control the hand pre-shaping before the grasp, based on visual input coming from a camera on the wrist. In this paper, we present an eye-in-hand learning-based approach for hand pre-shape classification from RGB sequences. Differently from previous work, we design the system to support the possibility to grasp each considered object part with a different grasp type. In order to overcome the lack of data of this kind and reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. We make publicly available the code and dataset to reproduce the presented results.

Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

TL;DR

This work tackles prosthetic grasping with multiple grasp types by using an eye-in-hand, vision-based approach to pre-shape classification from RGB sequences. It introduces a synthetic data generation pipeline with realistic, human-like arm trajectories and domain randomization to train models, addressing data scarcity and enhancing generalization. The study demonstrates that synthetic data can match or exceed real-data performance in robust, cross-condition scenarios and validates the pipeline on the Hannes prosthesis. Overall, the paper shows that allowing different grasps for distinct object parts and leveraging synthetic data improves control transparency and user satisfaction in shared-autonomy prosthetic systems.

Abstract

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-in-hand systems automatically control the hand pre-shaping before the grasp, based on visual input coming from a camera on the wrist. In this paper, we present an eye-in-hand learning-based approach for hand pre-shape classification from RGB sequences. Differently from previous work, we design the system to support the possibility to grasp each considered object part with a different grasp type. In order to overcome the lack of data of this kind and reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. We make publicly available the code and dataset to reproduce the presented results.
Paper Structure (10 sections, 4 figures, 3 tables)

This paper contains 10 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Synthetic Data Generation
  5. Experiments
  6. Experimental setup
  7. System design motivation
  8. Learning pre-shape classification from synthetic data
  9. Application on the Hannes prosthesis
  10. Conclusions

Figures (4)

  • Figure 1: Examples of the proposed multi grasp type per object annotation for the Mug and the Mustard (each object part is labeled with a different grasp type and pre-shape) (a) and of the table-top setup rendering (b).
  • Figure 2: Pictures of the developed wearable sensorized experimental setup and of the used Hannes prosthesis laffranchi2020.
  • Figure 3: Sample frames of the proposed synthetic dataset and of the collected real sequences for the different training and test sets.
  • Figure 4: Accuracy trends comparison over time during a grasping sequence of models trained with egocentric (blue) and eye-in-hand (pink) data. We report performance for single (dashed) and multi grasp (solid) objects.