Contact-Aware Refinement of Human Pose Pseudo-Ground Truth via Bioimpedance Sensing

TL;DR

BioTUCH introduces a novel, multi-modal framework that leverages wrist-to-wrist bioimpedance sensing to detect self-contact and refine arm poses in SMPL-X during monocular video. By applying a contact-aware optimization to frames where contact is detected, it improves pose accuracy (notably at the wrist) and contact reliability across multiple visual estimators. The work provides a new dataset with synchronized RGB, bioimpedance, and mocap ground truth, plus a miniature wearable sensor to enable large-scale, in-the-wild data collection. Together, these contributions offer a scalable path toward more accurate 3D human pose in scenarios involving self-contact.

Abstract

Capturing accurate 3D human pose in the wild would provide valuable data for training pose estimation and motion generation methods. While video-based estimation approaches have become increasingly accurate, they often fail in common scenarios involving self-contact, such as a hand touching the face. In contrast, wearable bioimpedance sensing can cheaply and unobtrusively measure ground-truth skin-to-skin contact. Consequently, we propose a novel framework that combines visual pose estimators with bioimpedance sensing to capture the 3D pose of people by taking self-contact into account. Our method, BioTUCH, initializes the pose using an off-the-shelf estimator and introduces contact-aware pose optimization during measured self-contact: reprojection error and deviations from the input estimate are minimized while enforcing vertex proximity constraints. We validate our approach using a new dataset of synchronized RGB video, bioimpedance measurements, and 3D motion capture. Testing with three input pose estimators, we demonstrate an average of 11.7% improvement in reconstruction accuracy. We also present a miniature wearable bioimpedance sensor that enables efficient large-scale collection of contact-aware training data for improving pose estimation and generation using BioTUCH. Code and data are available at biotuch.is.tue.mpg.de

Figures (8)

• Figure 1: Human pose estimation methods struggle to reconstruct self-contact along the camera's viewing axis. BioTUCH uses sharp changes in the bioimpedance signal measured between the wrists to estimate the beginning and end of such contacts. For all frames between these points (red segments of the signal), BioTUCH optimizes the results of off-the-shelf methods (such as Multi-HMR) to create plausible contact. Five selected poses (at the indicated time points) and their reconstructions are shown, with zoomed-in views of the contact regions.
• Figure 2: Wearable setup. We created a miniaturized sensor that consists of a processor, shown here attached to the wearer's shirt, and two electrodes embedded in the blue bracelets on their wrists. All components can be hidden beneath clothing.
• Figure 3: Sample bioimpedance signal showing four self-contact gestures over time. The bioimpedance magnitude begins to change when the person moves from the resting pose. The start of each self-contact triggers an abrupt drop in bioimpedance that is detected by our algorithm and marked with $\circ$. The thick red segments show the duration of the self-contacts, and their estimated ends are marked with $\times$.
• Figure 4: Sample contacts that cause a significant change in the bioimpedance of the user. Any self-contact (direct on skin or through body hair) greatly impacts the wrist-to-wrist bioimpedance signal. The videos from which these images were taken and their ground-truth SMPL-X meshes are part of our newly collected dataset.
• Figure 5: Robustness evaluation. The plot shows PA-V2V errors before ($x$-axis) and after ($y$-axis) applying BioTUCH to the three input methods. Points below the dashed black line indicate improvement. The $R^2$ values show how well the fitted curves explain the variance in the data.