Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Controlled Sensorized Prosthetic Hand

Md Abdul Baset Sarker, Juan Pablo S. Sola, Aaron Jones, Evan Laing, Ernesto Sola-Thomas, Masudul H. Imtiaz

TL;DR

The paper addresses the challenge of upper-limb prosthetics that rely on training-intensive EMG interfaces by introducing a vision-controlled sensorized hand built with a wrist-mounted camera, five fingertip FSRs, a distance sensor, and an accelerometer. It presents a compact hardware stack powered by a Coral Dev Board Mini running EfficientDet-Lite0 for real-time grasp detection and gesture-based release, with an ATmega328P motor controller and five SG90 servos actuating tendon-wired fingers. The work contributes a 3D-printed PLA prototype with open-source firmware, validated through real-time object detection at 9 FPS and successful grasp/lift/drop interactions on two objects, achieving around 90% accuracy in 20 trials and about 1 hour of operation per charge. This approach promises a training-light, potentially more reliable alternative to EMG-based prosthetics, with open-source hardware/software and pathways for fitting across ages and arm sizes, enhancing accessibility for amputees worldwide.

Abstract

This paper presents a sensorized vision-enabled prosthetic hand aimed at replicating a natural hand's performance, functionality, appearance, and comfort. The design goal was to create an accessible substitution with a user-friendly interface requiring little to no training. Our mechanical hand uses a camera and embedded processors to perform most of these tasks. The interfaced pressure sensor is used to get pressure feedback and ensure a safe grasp of the object; an accelerometer is used to detect gestures and release the object. Unlike current EMG-based designs, the prototyped hand does not require personalized training. The details of the design, trade-offs, results, and informing the next iteration are presented in this paper.

Vision Controlled Sensorized Prosthetic Hand

TL;DR

The paper addresses the challenge of upper-limb prosthetics that rely on training-intensive EMG interfaces by introducing a vision-controlled sensorized hand built with a wrist-mounted camera, five fingertip FSRs, a distance sensor, and an accelerometer. It presents a compact hardware stack powered by a Coral Dev Board Mini running EfficientDet-Lite0 for real-time grasp detection and gesture-based release, with an ATmega328P motor controller and five SG90 servos actuating tendon-wired fingers. The work contributes a 3D-printed PLA prototype with open-source firmware, validated through real-time object detection at 9 FPS and successful grasp/lift/drop interactions on two objects, achieving around 90% accuracy in 20 trials and about 1 hour of operation per charge. This approach promises a training-light, potentially more reliable alternative to EMG-based prosthetics, with open-source hardware/software and pathways for fitting across ages and arm sizes, enhancing accessibility for amputees worldwide.

Abstract

This paper presents a sensorized vision-enabled prosthetic hand aimed at replicating a natural hand's performance, functionality, appearance, and comfort. The design goal was to create an accessible substitution with a user-friendly interface requiring little to no training. Our mechanical hand uses a camera and embedded processors to perform most of these tasks. The interfaced pressure sensor is used to get pressure feedback and ensure a safe grasp of the object; an accelerometer is used to detect gestures and release the object. Unlike current EMG-based designs, the prototyped hand does not require personalized training. The details of the design, trade-offs, results, and informing the next iteration are presented in this paper.
Paper Structure (12 sections, 5 figures)

This paper contains 12 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Design Consideration
  3. Prototype Description
  4. Mechanical Design
  5. Electronic Hardware
  6. Coral dev board mini
  7. Camera and Sensors
  8. Motor Controller and Motors
  9. Training and object detection
  10. System Validation
  11. Discussion
  12. Conclusion

Figures (5)

  • Figure 1: (left) shows the front side of the working prototype of Vision Controlled Sensorized Prosthetic hand: (a) Camera, (b) Distance sensor, (c) Pressure sensor (FSR), (d) accelerometer (hidden inside),(e) Tendon wire. and the image (right) is the backside showing the electronic circuit and servo motor assembly
  • Figure 2: 3D Design of Vision-Enabled Sensorized Prosthetic Hand
  • Figure 3: Block diagram of the system
  • Figure 4: Object detected when the hand is closer to the object
  • Figure 5: Grasping the object with the prosthetic hand