High-density Electromyography for Effective Gesture-based Control of Physically Assistive Mobile Manipulators

TL;DR

The use of real-time myoelectric gesture recognition is evaluated using the easily-producible HDEMG device to enable precise control over the intricate mobility and manipulation functionalities of an 8 degree-of-freedom mobile manipulator.

Abstract

High-density electromyography (HDEMG) can detect myoelectric activity as control inputs to a variety of electronically-controlled devices. Furthermore, HDEMG sensors may be built into a variety of clothing, allowing for a non-intrusive myoelectric interface that is integrated into a user's routine. In our work, we introduce an easily-producible HDEMG device that interfaces with the control of a mobile manipulator to perform a range of household and physically assistive tasks. Mobile manipulators can operate throughout the home and are applicable for a spectrum of assistive and daily tasks in the home. We evaluate the use of real-time myoelectric gesture recognition using our device to enable precise control over the intricate mobility and manipulation functionalities of an 8 degree-of-freedom mobile manipulator. Our evaluation, involving 13 participants engaging in challenging self-care and household activities, demonstrates the potential of our wearable HDEMG system to control a mobile manipulator in the home.

Figures (14)

• Figure 1: System Overview. (A) Participant sitting with HDEMG sleeve on chair controlling a Stretch RE2 robot to move towards a table to perform an assistive task. The Intan RHD Recording Controller is behind the user and an SPI cable connects the sleeve to the recording controller. On the right, we show a close-up of participant wearing custom HDEMG sleeve, along with a diagram of an unrolled HDEMG device. Below, we show voltage signals over time seen during gesture performance. To the right of the signals, we show that different gestures have varying patterns of voltage root-mean-square (RMS) heatmaps. (B) These frames show different movements that the participant gestures in order to control the robot to bring a cereal bar close to their face. The corresponding gesture and sample RMS heatmaps from the participant's data are shown for each gesture and movement.
• Figure 1: Donning Procedure. The donning process for the wearable is shown here. This procedure takes 2-3 minutes for the full donning process.
• Figure 2: Confusion Matrices, SNR, and Test Accuracies. (A) Test accuracy confusion matrices before recalibration and after recalibration. The confusion matrix "Before Recalibration" shows some of the most commonly misclassified gestures after the shift in signals distributions have occurred. The "After Recalibration" matrix shows that accuracies are increased after the recalibration phase. (B) Differences between the initial session and the recalibration session. The mean SNR between the initial session and the recalibration session was not found to be statistically significant. The error bars show 95% confidence intervals for the mean. This shows the test accuracies for the initial dataset, before recalibration, and after recalibration for each participant.
• Figure 2: Electrode Impedances. The average impedances of electrodes in Ohms during the initial data collection phase and during the recalibration data collection phase is visualized using these heatmaps.
• Figure 3: Data Collection and Decoder Architecture. (A) Data collection GUI, with the first three frames showing a priming phase in which a translucent animation demonstrates the gesture to be performed next. The second set of frames shows an animation of the actual gesture performance where the participant is cued to perform simultaneously. The end of the second row shows the gesture being held, which is held for 3 seconds. The third set of frames shows the gesture returning to neutral, along with a gesture prompt change in text at the top to the next gesture to be performed. (B) Data processing and machine learning pipeline. Raw EMG data is filtered with a high pass filter at 120 Hz. A root mean square is then applied to extract magnitude information for each electrode. A z-score normalization is applied, followed by PCA for dimension reduction. The transformed input is then fed into a feedforward neural network for training and predicting gestures out of 10 classes.