Online Adaptation for Myographic Control of Natural Dexterous Hand and Finger Movements

TL;DR

This work tackles the challenge of decoding continuous, multi-DoF hand and wrist movements from myographic signals to enable dexterous, biomimetic prosthesis control. It advances a regression-based framework using temporal models (TCN/LSTM) and online reinforcement to adapt in real time under minimally constrained training. The results show that sequential models outperform frame-wise baselines with RMSEs on the order of $8$–$11^{\circ}$ and latencies near $0$–$25$ ms, with reinforcement further improving accuracy and alignment to ground truth. Collectively, the study demonstrates a path toward fully restoring natural upper-limb function by combining advanced algorithms, flexible experimental paradigms, high-performance prostheses, and potential multi-modal sensing like sonomyography.

Abstract

One of the most elusive goals in myographic prosthesis control is the ability to reliably decode continuous positions simultaneously across multiple degrees-of-freedom. Goal: To demonstrate dexterous, natural, biomimetic finger and wrist control of the highly advanced robotic Modular Prosthetic Limb. Methods: We combine sequential temporal regression models and reinforcement learning using myographic signals to predict continuous simultaneous predictions of 7 finger and wrist degrees-of-freedom for 9 non-amputee human subjects in a minimally-constrained freeform training process. Results: We demonstrate highly dexterous 7 DoF position-based regression for prosthesis control from EMG signals, with significantly lower error rates than traditional approaches (p < 0.001) and nearly zero prediction response time delay (p < 0.001). Their performance can be continuously improved at any time using our freeform reinforcement process. Significance: We have demonstrated the most dexterous, biomimetic, and natural prosthesis control performance ever obtained from the surface EMG signal. Our reinforcement approach allowed us to abandon standard training protocols and simply allow the subject to move in any desired way while our models adapt. Conclusions: This work redefines the state-of-the-art in myographic decoding in terms of the reliability, responsiveness, and movement complexity available from prosthesis control systems. The present-day emergence and convergence of advanced algorithmic methods, experiment protocols, dexterous robotic prostheses, and sensor modalities represents a unique opportunity to finally realize our ultimate goal of achieving fully restorative natural upper-limb function for amputees.

Figures (8)

• Figure 1: (A) Experiment set-up and devices. A CTRL-kit armband was placed around the circumference of the subject's forearm to record EMG, while hand and wrist positional data were recorded with a CyberGlove. The vMPL interface provided subjects with a real-time display of their hand and wrist movements on a virtual prosthesis during all of our experiments. (B) Active movement types used in the standard experiment paradigm, wherein these movements are cued to the subject, and the subjects performs each by transitioning from a rest state into the movement and back to rest. Prediction models are then trained on this data and tested on similar data. (C). Freeform reinforcement paradigm, wherein the subject performs any types of movement (not limited the those in B) s/he prefers and the prediction models are tested in real-time and periodically updated with new information. This type of training is designed to gradually improve performance by adapting to the subject in real-time, resulting in better spatial and temporal alignment/agreement between ground truth and model predictions.
• Figure 2: Standard experiment offline regression output for a single subject when predicting using the following models: (A) SVR (B) LSTM, and (C) TCN. TCN is our preferred model so we show its simultaneous output for all 7 DoFs, whereas we only show the first 3 DoFs for SVR and LSTM. These prediction output profiles are provided to give a sense of the behavioral characteristics each model for granular comparison. For example, we note that the total error of LSTM was slightly lower than TCN, but LSTM prediction outputs also tended to be visibly noisier than TCN.
• Figure 3: Freeform experiment offline regression output for a single subject when predicting using the following models: (A) SVR (B) LSTM, and (C) TCN. TCN is our preferred model so we show its simultaneous output for all 7 DoFs, whereas we only show the first 3 DoFs for SVR and LSTM. These prediction output profiles are provided to give a sense of the behavioral characteristics each model for granular comparison. For example, we note that the total error of LSTM was slightly lower than TCN, but LSTM prediction outputs also tended to be visibly noisier than TCN.
• Figure 4: EMG features can be fed as fixed-length sequences into sequential prediction models such as (A) TCN (© 2019 IEEE) and (B) LSTM (© Wikimedia Commons). (C) Parameter-sweeps to determine optimal window sizes and sequence lengths. The inclusion of sequence data lowered error rates, and sequential models required much shorter feature windows to achieve superior performance than the typically-used 200 ms window for decoding EMG with frame-wise models. (D) RMSE and training time as a function of training epochs. We used 15 training epochs for TCN, and 40 training epochs for LSTM. Final parameters used in our experiments are shown in Table \ref{['model_params_table']}.
• Figure 5: The per-DoF angular RMSE, total angular RMSE, and $r^2$ coefficient of determination based on regression performance across subjects during (A) the standard experiment and (B) the freeform experiment. In both experiments, the average RMSE values were better for wrist (DoFs 1 and 2) and thumb (DoF 3) motions than they were for individual fingers (DoFs 4 to 7). In both experiments, the sequential models TCN and LSTM performed similarly, and they significantly outperformed the SVR frame-wise model.