A Powered Prosthetic Hand with Vision System for Enhancing the Anthropopathic Grasp

TL;DR

This work tackles the challenge of achieving natural, anthropomorphic grasping for prosthetic hands using vision-only data. It introduces two key components: SG-GM, which builds gesture transformation functions from geometric hand–object interactions, and MTR-GIE, which estimates grasping intent in multi-object environments from hand trajectories. The approach yields high anthropomorphism (average $R^2=0.911$, $RMSE=2.47^ ext{o}$), strong single-object success ($ ext{mean }Suc=95.43 ext{%}$) and robust multi-object intent estimation accuracy ($ ext{mean }Acc=94.35 ext{%}$), with an average grasping duration of $3.07 ext{s}$ in prosthetic hands compared to $1.47 ext{s}$ for natural hands. Collectively, SG-GM and MTR-GIE enable fast, visually guided, anthropomorphic grasping in both single- and multi-object settings, offering a practical pathway to more usable, self-esteem-supporting prosthetic hands and informing future vision-based control research.

Abstract

The anthropomorphism of grasping process significantly benefits the experience and grasping efficiency of prosthetic hand wearers. Currently, prosthetic hands controlled by signals such as brain-computer interfaces (BCI) and electromyography (EMG) face difficulties in precisely recognizing the amputees' grasping gestures and executing anthropomorphic grasp processes. Although prosthetic hands equipped with vision systems enables the objects' feature recognition, they lack perception of human grasping intention. Therefore, this paper explores the estimation of grasping gestures solely through visual data to accomplish anthropopathic grasping control and the determination of grasping intention within a multi-object environment. To address this, we propose the Spatial Geometry-based Gesture Mapping (SG-GM) method, which constructs gesture functions based on the geometric features of the human hand grasping processes. It's subsequently implemented on the prosthetic hand. Furthermore, we propose the Motion Trajectory Regression-based Grasping Intent Estimation (MTR-GIE) algorithm. This algorithm predicts pre-grasping object utilizing regression prediction and prior spatial segmentation estimation derived from the prosthetic hand's position and trajectory. The experiments were conducted to grasp 8 common daily objects including cup, fork, etc. The experimental results presented a similarity coefficient $R^{2}$ of grasping process of 0.911, a Root Mean Squared Error ($RMSE$) of 2.47\degree, a success rate of grasping of 95.43$\%$, and an average duration of grasping process of 3.07$\pm$0.41 s. Furthermore, grasping experiments in a multi-object environment were conducted. The average accuracy of intent estimation reached 94.35$\%$. Our methodologies offer a groundbreaking approach to enhance the prosthetic hand's functionality and provides valuable insights for future research.

Figures (8)

• Figure 1: Overview of the intelligent prosthetic hand system. (a) Diagram A represents the workflow chart of the dynamic grasping gesture modeling module. (b) Diagram B represents the workflow chart of the prosthetic hand control module. The dynamic grasping gesture modeling module first extracts the geometric features of the hand and the objects as presented in sub-diagram A1. Then, it maps the hand posture angles into spatiotemporal gesture transformation functions as presented in sub-diagram A2. Finally, it stores the object properties and gesture functions into the gesture model library as presented in sub-diagram A3. The prosthetic hand control module starts by quickly identifying the target object using an intent estimation algorithm as presented in sub-diagram B1. It then retrieves the corresponding gesture function from the gesture model library in sub-diagram A3. Based on the information in the gesture function, the prosthetic hand is guided to perform natural grasping gestures during the movement as presented in sub-diagram B2.
• Figure 2: Pipeline of the SG-GM method. In the object detection and 3D reconstruction part as presented in sub-diagram A, the process consists of 2D object detection, 3D mapping, and background removal in order to obtain 3D model parameters and object positions. In the hand detection and gesture estimation part as presented in sub-diagram B, the steps involve 2D hand detection, reshaping the hand bounding box, and gesture estimation to obtain the gesture and hand positions for each frame. Subsequently, the information of the hand and the object is associated, and the gesture transformation functions that are spatiotemporally related to the object are derived as presented in sub-diagram C. The sequence $\boldsymbol{D}=\{D_1, D_2, D_3, \ldots, D_n\}$ represents the hand-object distance during the hand's movement. In the gesture mapping section, the horizontal axis represents the spatial distance progress of the hand from the starting moving position to the final grasping position.
• Figure 3: Diagram of the MTR-GIE algorithm for estimating grasping intent. The coordinate in the figure represents the world coordinate. The red points indicate the position of the wrist during the movement of the prosthetic hand. The MTR-GIE algorithm predicts the future movement trajectory based on the historical positions of the prosthetic hand during the grasping process. The intersection of the regression plane 1 marked in green and the regression plane 2 marked in blue in the figure represents the predicted regression line. This regression line is used as the predicted trajectory. The plane marked in orange on the predicted trajectory serves as a spatial separation plane to separate the objects in the left space (for right-hand prosthetic). The object closest to the left space is then estimated as the target object.
• Figure 4: Grasping gesture of participants for constructing the library of the gesture models. The small image in the upper left corner demonstrates that how the camera worn on the head captures object. It demonstrates the natural hand gestures during the process of grasping different objects. Different grasping gestures are adopted to grasp eight objects. The system models the gestures based on the visual information captured by the camera.
• Figure 5: Grasping objects of the prosthetic hand. (a) Single-object scenario and the corresponding prosthetic hand gestures for grasping eight different objects. (b) Multi-object scenario. The initial positions of the prosthetic hands are identical, and there is an equal horizontal spacing between the objects.