ProACT: An Augmented Reality Testbed for Intelligent Prosthetic Arms

TL;DR

ProACT presents an open-source augmented reality testbed to evaluate intelligent control for whole-arm prostheses by combining gaze-based intent estimation, motion planning, and low-level autonomy within an immersive AR environment. The platform integrates ROS-based robotics, Gazebo dynamics, Unity visualization, and multi-modal inputs (EMG, gaze, motion capture) using the MPL prosthetic arm to study four control methods ranging from direct to assisted control. Across two exploratory studies with non-amputee participants performing a modified Box-and-Blocks task, results show that gaze-assisted (C) and context-assisted (D) methods yield higher success and more consistent performance than direct control (A/B), with learning effects and substantial individual variability. These findings, together with the platform’s open-source nature, suggest that intelligent prosthesis control for complex whole-arm tasks is feasible and can be iteratively improved, with future work extending to amputees and richer feedback mechanisms.

Abstract

Upper-limb amputees face tremendous difficulty in operating dexterous powered prostheses. Previous work has shown that aspects of prosthetic hand, wrist, or elbow control can be improved through "intelligent" control, by combining movement-based or gaze-based intent estimation with low-level robotic autonomy. However, no such solutions exist for whole-arm control. Moreover, hardware platforms for advanced prosthetic control are expensive, and existing simulation platforms are not well-designed for integration with robotics software frameworks. We present the Prosthetic Arm Control Testbed (ProACT), a platform for evaluating intelligent control methods for prosthetic arms in an immersive (Augmented Reality) simulation setting. We demonstrate the use of ProACT through preliminary studies, with non-amputee participants performing an adapted Box-and-Blocks task with and without intent estimation. We further discuss how our observations may inform the design of prosthesis control methods, as well as the design of future studies using the platform. To the best of our knowledge, this constitutes the first study of semi-autonomous control for complex whole-arm prostheses, the first study including sequential task modeling in the context of wearable prosthetic arms, and the first testbed of its kind. Towards the goal of supporting future research in intelligent prosthetics, the system is built upon on existing open-source frameworks for robotics, and is available at https://arm.stanford.edu/proact.

Figures (8)

• Figure 1: (a) Components of the testbed, showing a non-amputee participant (pictured with permission) and virtual components (the arm, , and selection marker) as they would be seen by the participant. (b) The first-person view of the . (c) The mixed-reality scene is localized with respect to the room, as explained in detail in Section \ref{['sec:sysarch']}.
• Figure 2: The four control methods (A-D) mapping inputs to motion.
• Figure 3: The image plane is defined as a plane normal to the gaze vector. Each block is assigned a probability based on its distance from the gaze target in the image plane, thus: a bivariate Gaussian function centered at the gaze target is evaluated at the centers of all blocks, and the resultant values normalized by their sum. This simplistic calculation is sufficient for our aims in this study.
• Figure 4: The outcome shown for each individual block in the experiment (Study 1).
• Figure 5: Survey rating results after each method (Study 1).