Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation

TL;DR

PiMForce is presented, a novel framework that enhances hand pressure estimation by leveraging 3D hand posture information to augment forearm surface electromyography (sEMG) signals to enable accurate and robust whole-hand pressure measurements under diverse hand-object interactions.

Abstract

We present PiMForce, a novel framework that enhances hand pressure estimation by leveraging 3D hand posture information to augment forearm surface electromyography (sEMG) signals. Our approach utilizes detailed spatial information from 3D hand poses in conjunction with dynamic muscle activity from sEMG to enable accurate and robust whole-hand pressure measurements under diverse hand-object interactions. We also developed a multimodal data collection system that combines a pressure glove, an sEMG armband, and a markerless finger-tracking module. We created a comprehensive dataset from 21 participants, capturing synchronized data of hand posture, sEMG signals, and exerted hand pressure across various hand postures and hand-object interaction scenarios using our collection system. Our framework enables precise hand pressure estimation in complex and natural interaction scenarios. Our approach substantially mitigates the limitations of traditional sEMG-based or vision-based methods by integrating 3D hand posture information with sEMG signals. Video demos, data, and code are available online.

Figures (23)

• Figure 1: Our sensing framework (PiMForce) leverages 3D hand posture information along with sEMG data to enable a whole-hand pressure estimation during various hand-object interactions. We support real-time pressure estimation on the fingertips and palm regions based on RGB image and sEMG inputs. The intensity of each node's color indicates the pressure level.
• Figure 2: Our multimodal hand pressure estimation architecture enhances sEMG data by embedding 3D hand pose information. We train the model using a classification-regression joint loss to improve hand pressure estimation.
• Figure 3: Quantitative evaluation of the user-independent model, showing the posture-wise performance on the estimation of hand pressure. The error bars indicate standard error.
• Figure 4: (a) Qualitative results in the absence of a pressure glove. The 3D Hand Pose Estimation yu2023acr represents 3D hand posture, including hand occlusion, using the 3D hand detector. The PressureVision++ grady2022pressurevision column shows the pressure estimation of fingertips. The red rectangles indicate the instances of pressure estimation failure due to hand occlusion. The proposed multimodal framework shows robust whole-hand pressure estimation for diverse hand-object interactions. (b) Illustration of the demo video footage showing robust hand pressure estimation with varying hand postures, pressure levels, and interacting objects.
• Figure 5: We added position tracking sensors at the knuckles and 5 fingertip pressure sensors. Our glove records occlusion-free hand-tracking data with exerted hand pressures at 9 regions (5 fingertips and 4 palm regions).