Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control

TL;DR

This work tackles robust prosthetic hand control by fusing EMG signals with vision-derived cues through a Bayesian evidence framework. It collects a synchronized, multimodal dataset with eye-tracking, a head-mounted camera, and dynamic EMG across 12 forearm channels while subjects perform 14 grasp gestures. The EMG pipeline employs unsupervised segmentation of dynamic motion and a 13-class gesture classifier, while the vision module uses copy-paste augmented YOLOv4-based grasp detection with background generalization. The fusion of EMG and vision evidence via P(L|M) and P(L|V) improves grasp-type accuracy during reach to 95.3% (and 96.8% with smoothing), demonstrating complementary strengths and the potential for robust real-time prosthetic hand control.

Abstract

Objective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Vision sensors are a major source of information about the environment state and can play a vital role in inferring feasible and intended gestures. However, visual evidence is also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. Multimodal evidence fusion using physiological and vision sensor measurements is a natural approach due to the complementary strengths of these modalities. Methods: In this paper, we present a Bayesian evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. We analyze individual and fused performance as a function of time as the hand approaches the object to grasp it. For this purpose, we have also developed novel data processing and augmentation techniques to train neural network components. Results: Our results indicate that, on average, fusion improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%. Conclusion: Our experimental data analyses demonstrate that EMG and visual evidence show complementary strengths, and as a consequence, fusion of multimodal evidence can outperform each individual evidence modality at any given time.

Figures (10)

• Figure 1: Proposed System Overview (eye-tracker from eyetracker).
• Figure 2: Selected $14$ gestures for the classification problem.
• Figure 3: Schematic of the EMG Signal Processing and Data Annotation Workflow. This figure outlines the sequence of processing steps applied to EMG data, starting with band-pass filtering, followed by RMS envelope computation, MVC normalization, and the application of sliding window techniques. The featured extracted at this stage is used in gesture classification. In phase classifier, unsupervised data annotation and alignment provides phases of object manipulation – rest, reach, grasp, and return outlined in the lower section which constitute the labeled activities in the dataset for machine learning model training and validation, with features such as RMS, MAV, and VAR extracted for analysis.
• Figure 4: Experiment timeline. The subject was given $5$ seconds to read the shown gesture before the first trial. Each trial lasted for $4$ seconds, repeated for $6$ trials without interruption. All EMG trials were segmented unsupervisedly into four sequences of reaching, grasping, returning and resting. The first three motion phases were labeled as gesture $l \in {\{1,...,13\}}$ corresponding to the target object, and the resting phase was tagged by the open-palm label $l=0$.
• Figure 5: Overall overview of background generalization using visual mask generation and copy-paste augmentation.