TeleFMG: A Wearable Force-Myography Device for Natural Teleoperation of Multi-finger Robotic Hands

TL;DR

TeleFMG presents a low-cost wearable FMG system that maps forearm muscle signals to finger poses for natural teleoperation of a multi-finger robotic hand using a spatio-temporal Temporal Convolutional Network. By collecting synchronized FMG data and hand pose labels with a sensor glove, the approach achieves accurate finger-joint estimation (TCN mean error around $9.76^{\circ}$) and enables real-time teleoperation of gestures and grasping with a four-finger robot hand. The study demonstrates partial transfer to new users via fine-tuning with limited data (~10 minutes, ~5% of training data) and analyzes sensor importance, suggesting avenues for minimal-sensor designs and future enhancements such as thumb DOF, IMU integration, and haptic feedback for improved fidelity.

Abstract

Teleoperation enables a user to perform dangerous tasks (e.g., work in disaster zones or in chemical plants) from a remote location. Nevertheless, common approaches often provide cumbersome and unnatural usage. In this letter, we propose TeleFMG, an approach for teleoperation of a multi-finger robotic hand through natural motions of the user's hand. By using a low-cost wearable Force-Myography (FMG) device, musculoskeletal activities on the user's forearm are mapped to hand poses which, in turn, are mimicked by a robotic hand. The mapping is performed by a spatio-temporal data-based model based on the Temporal Convolutional Network. The model considers spatial positions of the sensors on the forearm along with temporal dependencies of the FMG signals. A set of experiments show the ability of a teleoperator to control a multi-finger hand through intuitive and natural finger motion. A robot is shown to successfully mimic the user's hand in object grasping and gestures. Furthermore, transfer to a new user is evaluated while showing that fine-tuning with a limited amount of new data significantly improves accuracy.

Figures (10)

• Figure 1: (a) The wearable TeleFMG device with the sensor glove are used to collect data in order to train a model for mapping musculoskeletal activities to hand poses. Then, teleoperation is demonstrated with a multi-finger robotic hand using the TeleFMG device in (b) pinch grasping of a small cube and (c) pointing gesture.
• Figure 2: The TeleFMG system including two bands with 14 FSR senors each, and the sensor glove for labeling FMG signals with hand poses in the data collection phase.
• Figure 3: An illustration of the TCN model which acquires spatio-temporal FMG signals and maps them to the pose of the human hand. The pose is mimicked by a robotic hand.
• Figure 4: Snapshots of data collection using the FMG device and the labeling glove in various hand poses.
• Figure 5: Snapshots of the 4-finger Allegro hand grasping a bottle using TeleFMG.