Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeuroManip: Prosthetic Hand Manipulation System Based on EMG and Eye Tracking Powered by the Neuromorphic Processor AltAi

Roman Akinshin, Elizaveta Lopatina, Kirill Bogatikov, Nikolai Kiz, Anna V. Makarova, Mikhail Lebedev, Miguel Altamirano Cabrera, Dzmitry Tsetserukou, Valerii Kangler

TL;DR

The paper presents a context-aware upper-limb prosthesis control system that fuses sEMG with gaze-guided vision, deploying a spiking neural network on the AltAi neuromorphic processor for real-time, low-power EMG decoding. By constraining the gesture space using gaze context and object detection, the approach achieves high accuracy (up to 95%) and eliminates unsafe grasps while maintaining energy efficiency suitable for wearable use. Comparisons show the neuromorphic implementation achieves near-GPU latency at a fraction of the power, enabling lighter hardware and reduced fatigue in users. The work outlines a practical path toward fully neuromorphic, wearable prosthetic control with plans for on-chip learning and broader validation.

Abstract

This paper presents a novel neuromorphic control architecture for upper-limb prostheses that combines surface electromyography (sEMG) with gaze-guided computer vision. The system uses a spiking neural network deployed on the neuromorphic processor AltAi to classify EMG patterns in real time while an eye-tracking headset and scene camera identify the object within the user's focus. In our prototype, the same EMG recognition model that was originally developed for a conventional GPU is deployed as a spiking network on AltAi, achieving comparable accuracy while operating in a sub-watt power regime, which enables a lightweight, wearable implementation. For six distinct functional gestures recorded from upper-limb amputees, the system achieves robust recognition performance comparable to state-of-the-art myoelectric interfaces. When the vision pipeline restricts the decision space to three context-appropriate gestures for the currently viewed object, recognition accuracy increases to roughly 95% while excluding unsafe, object-inappropriate grasps. These results indicate that the proposed neuromorphic, context-aware controller can provide energy-efficient and reliable prosthesis control and has the potential to improve safety and usability in everyday activities for people with upper-limb amputation.

NeuroManip: Prosthetic Hand Manipulation System Based on EMG and Eye Tracking Powered by the Neuromorphic Processor AltAi

TL;DR

The paper presents a context-aware upper-limb prosthesis control system that fuses sEMG with gaze-guided vision, deploying a spiking neural network on the AltAi neuromorphic processor for real-time, low-power EMG decoding. By constraining the gesture space using gaze context and object detection, the approach achieves high accuracy (up to 95%) and eliminates unsafe grasps while maintaining energy efficiency suitable for wearable use. Comparisons show the neuromorphic implementation achieves near-GPU latency at a fraction of the power, enabling lighter hardware and reduced fatigue in users. The work outlines a practical path toward fully neuromorphic, wearable prosthetic control with plans for on-chip learning and broader validation.

Abstract

This paper presents a novel neuromorphic control architecture for upper-limb prostheses that combines surface electromyography (sEMG) with gaze-guided computer vision. The system uses a spiking neural network deployed on the neuromorphic processor AltAi to classify EMG patterns in real time while an eye-tracking headset and scene camera identify the object within the user's focus. In our prototype, the same EMG recognition model that was originally developed for a conventional GPU is deployed as a spiking network on AltAi, achieving comparable accuracy while operating in a sub-watt power regime, which enables a lightweight, wearable implementation. For six distinct functional gestures recorded from upper-limb amputees, the system achieves robust recognition performance comparable to state-of-the-art myoelectric interfaces. When the vision pipeline restricts the decision space to three context-appropriate gestures for the currently viewed object, recognition accuracy increases to roughly 95% while excluding unsafe, object-inappropriate grasps. These results indicate that the proposed neuromorphic, context-aware controller can provide energy-efficient and reliable prosthesis control and has the potential to improve safety and usability in everyday activities for people with upper-limb amputation.
Paper Structure (12 sections, 4 figures)

This paper contains 12 sections, 4 figures.

Table of Contents

  1. Introduction
  2. System Architecture
  3. Prosthetic Hand
  4. EMG
  5. Computer Vision
  6. System Evaluation
  7. Neural network deployment on GPU and neuromorphic hardware
  8. Experimental Design
  9. Measures
  10. Results
  11. Conclusion and Future Work
  12. Acknowledgment

Figures (4)

  • Figure 1: System architecture and control flow. (a) The prosthetic hand supports n pre-programmed functional gestures. (b) The vision subsystem processes stereo RGB video and gaze position from the eyeglass-mounted eye tracker. Object detection via YOLO classifier identifies scene objects; the gaze intersection determines the currently fixated object and automatically predicts context-appropriate grasps, reducing the action space from n to k candidate gestures (k < n). (c) The myoelectric armband decodes muscle signals to select the final gesture from the restricted subset of k candidates. (d) User confirms selection via EMG activation, triggering prosthetic hand execution of the chosen grasp.
  • Figure 2: Process of EMG data collection
  • Figure 3: Comparison of NASA-TLX scores across three weight conditions. Error bars represent standard deviation.
  • Figure 4: Task performance and fatigue across distal mass conditions (a) Completion time for the block-transfer task. (b) Fatigue Index, defined as the change in completion time between the third and first trial within each condition