Wearable and Ultra-Low-Power Fusion of EMG and A-Mode US for Hand-Wrist Kinematic Tracking

TL;DR

This work tackles robust hand-wrist kinematic tracking using a wearable, ultra-low-power fusion of EMG and A-mode ultrasound. It introduces a sub-50 mW platform that concurrently collects 8-channel EMG and 4-channel US on dry armbands and pairs it with a 23-DoF ground-truth framework based on the MANUS kinematic glove. The authors design modality-specific encoder–decoder networks with late-feature fusion and multitask learning to predict 20 hand and 3 wrist joint angles, achieving a fused RMSE of $10.6^\ m{\circ} \pm 2.0^ m{\circ}$ and $R^2=0.61\pm0.10$ in inter-session tests, outperforming EMG- or US-only baselines in certain scenarios. The platform demonstrates a practical path toward multi-day, high-dimensional hand monitoring, with future work focusing on larger subject cohorts and more extensive statistical validation.

Abstract

Hand gesture recognition based on biosignals has shown strong potential for developing intuitive human-machine interaction strategies that closely mimic natural human behavior. In particular, sensor fusion approaches have gained attention for combining complementary information and overcoming the limitations of individual sensing modalities, thereby enabling more robust and reliable systems. Among them, the fusion of surface electromyography (EMG) and A-mode ultrasound (US) is very promising. However, prior solutions rely on power-hungry platforms unsuitable for multi-day use and are limited to discrete gesture classification. In this work, we present an ultra-low-power (sub-50 mW) system for concurrent acquisition of 8-channel EMG and 4-channel A-mode US signals, integrating two state-of-the-art platforms into fully wearable, dry-contact armbands. We propose a framework for continuous tracking of 23 degrees of freedom (DoFs), 20 for the hand and 3 for the wrist, using a kinematic glove for ground-truth labeling. Our method employs lightweight encoder-decoder architectures with multi-task learning to simultaneously estimate hand and wrist joint angles. Experimental results under realistic sensor repositioning conditions demonstrate that EMG-US fusion achieves a root mean squared error of $10.6^\circ\pm2.0^\circ$, compared to $12.0^\circ\pm1^\circ$ for EMG and $13.1^\circ\pm2.6^\circ$ for US, and a R$^2$ score of $0.61\pm0.1$, with $0.54\pm0.03$ for EMG and $0.38\pm0.20$ for US.

Figures (3)

• Figure 1: a) Block diagram of EMG-US integrated Platform; b) Integrated Hardware c) EMG and US dry armbands d) Data collection framework e) Overview of the EMG-US-MANUS synchronization procedure. US signals are acquired by the WULPUS GUI, whereas glove data are streamed to the MANUS Core and then forwarded to the BioGUI, which also acquires EMG and the hardware synchronization signal from BioGAP. BioGAP and MANUS data are synchronized using the software-based trigger generated by the BioGUI, and then they are synchronized with WULPUS data using the hardware synchronization signal.
• Figure 2: Network Architectures for EMG and US, with late-feature fusion.
• Figure 3: Example of ground truth vs models' predictions (smoothed with a median filter, window of $\approx670\text{ms}$) over time for one acquisition set. Light grey shades highlight the benefits of fusion when either one or both modalities fail.