Bimanual High-Density EMG Control for In-Home Mobile Manipulation by a User with Quadriplegia

TL;DR

This work presents the first bimanual, fabric-integrated high-density EMG sleeve system for real-time control of a mobile manipulator by a user with quadriplegia and couples it with a shared-autonomy stack that integrates perception, navigation, and safety for in-home use. It details a 12-day in-home deployment of the system, including a five-day exploratory phase for data collection and classifier training and a seven-day finalized phase with a fixed configuration and added shared-autonomy modules. The main contributions are (i) the wearable bimanual HDEMG sleeves with dense forearm coverage and rapid retraining, (ii) the open-vocabulary perception-based automatic object alignment, room navigation, and LiDAR-based collision monitoring, and (iii) the first in-home evaluation demonstrating improved task times, reduced workload, and maintained usability for daily living and personalized tasks. The results suggest wearable neuromotor interfaces paired with shared autonomy can enable practical, user-driven independent mobile manipulation in real home environments, with implications for scalability and daily living independence for people with quadripelegia.

Abstract

Mobile manipulators in the home can enable people with cervical spinal cord injury (cSCI) to perform daily physical household tasks that they could not otherwise do themselves. However, paralysis in these users often limits access to traditional robot control interfaces such as joysticks or keyboards. In this work, we introduce and deploy the first system that enables a user with quadriplegia to control a mobile manipulator in their own home using bimanual high-density electromyography (HDEMG). We develop a pair of custom, fabric-integrated HDEMG forearm sleeves, worn on both arms, that capture residual neuromotor activity from clinically paralyzed degrees of freedom and support real-time gesture-based robot control. Second, by integrating vision, language, and motion planning modules, we introduce a shared autonomy framework that supports robust and user-driven teleoperation, with particular benefits for navigation-intensive tasks in home environments. Finally, to demonstrate the system in the wild, we present a twelve-day in-home user study evaluating real-time use of the wearable EMG interface for daily robot control. Together, these system components enable effective robot control for performing activities of daily living and other household tasks in a real home environment.

Figures (16)

• Figure 1: Flex PCB has 128 gold-plated electrodes soldered onto pads, folded and sewn into spandex fabric forearm sleeve, and attached onto both arms of the user. Three Velcro strips are also attached per sleeve in order to ensure electrodes make tight contact with the arm.
• Figure 2: (Left) Gestures used for robot control on both hands for the last seven days. We decrease the number of gestures for the right hand due to lower classification accuracy on the right hand for this user. (Right) Gestures are mapped to robot movements in three different modes. Modes are cycled between either through voice commands ("Hey Robot, Next Mode") or holding the wrist back gesture on both hands for 0.2 seconds.
• Figure 3: Illustrations of the three shared autonomy modules used: automatic object alignment, room navigation, and LiDAR-based collision avoidance.
• Figure 4: Task times for all tasks performed over the second week. (Left) Task times for cu task performed over the second week. This task is repeated each day. A least-squares trend line is shown, indicating a decrease of approximately 33 seconds per day. (Right) Task times for other tasks performed over the second week.
• Figure 5: Task times comparing pure teleoperation with auto-align. Auto-alignment shows a visually consistent decrease in task time, while pure teleoperation exhibits higher variability.