DexAvatar: 3D Sign Language Reconstruction with Hand and Body Pose Priors

TL;DR

DexAvatar addresses the lack of accurate 3D data for sign language by introducing SignBPoser and SignHPoser, sign-language-aware priors trained on mocap data to regularize body and hand articulations. Framed as an optimization over the SMPL-X model, the method integrates these priors with reprojection, temporal, and biomechanical constraints to produce stable, bio-mechanically plausible 3D signing avatars from monocular videos. Empirical results on the SGNify dataset show DexAvatar achieving state-of-the-art performance, with pronounced improvements in upper-body and hand reconstructions, and ablations confirming the effectiveness of the priors and preprocessing. The work highlights the importance of domain-specific priors for sign language and offers a path toward scalable, realistic 3D signing avatars in real-world, in-the-wild settings.

Abstract

The trend in sign language generation is centered around data-driven generative methods that require vast amounts of precise 2D and 3D human pose data to achieve an acceptable generation quality. However, currently, most sign language datasets are video-based and limited to automatically reconstructed 2D human poses (i.e., keypoints) and lack accurate 3D information. Furthermore, existing state-of-the-art for automatic 3D human pose estimation from sign language videos is prone to self-occlusion, noise, and motion blur effects, resulting in poor reconstruction quality. In response to this, we introduce DexAvatar, a novel framework to reconstruct bio-mechanically accurate fine-grained hand articulations and body movements from in-the-wild monocular sign language videos, guided by learned 3D hand and body priors. DexAvatar achieves strong performance in the SGNify motion capture dataset, the only benchmark available for this task, reaching an improvement of 35.11% in the estimation of body and hand poses compared to the state-of-the-art. The official website of this work is: https://github.com/kaustesseract/DexAvatar.

Figures (15)

• Figure 1: DexAvatar recovers bio-mechanically accurate 3D hand and body poses from monocular sign language videos.
• Figure 2: Overview of the DexAvatar pipeline. Given a set of input frames, we first run SMPLerX cai2023smpler and HaMeR pavlakos2024reconstructing to obtain initial body and hand pose estimates. We then refine these estimates by fitting to a 2D joint, using Sapiens khirodkar2024sapiens for body keypoints and HaMeR for hand keypoints by minimizing the reprojection error ($\mathcal{L}_{\text{joint }}$) to the detected joints $\mathcal{K}_i$. To generate plausible hand and body articulations, we constrain poses to learned manifolds, where SignBPoser maps a body latent $\zeta$ to $\theta_b$, and SignHPoser maps independent left and right latents $\epsilon^{\ell}$ and $\epsilon^{r}$ to $\theta_h$. Finally, bio-mechanical constraints enforce physically plausible articulation, producing accurate 3D signing avatars.
• Figure 3: Bio-mechanical body filter. For body data from how2sign, we enforce joint range of motion and signer space constraints on shoulders, elbows/forearm, and wrists. Frames that violate these envelopes are rejected, and only plausible body poses are retained for training.
• Figure 4: Bio-mechanical hand rectifier. For raw mocap hand data, we correct implausible joint configurations by enforcing per-joint limits on bending, splaying, and twisting (15 hand joints). The rectifier outputs corrected hand geometry for training.
• Figure 5: Qualitative results. We compare 3D holistic human mesh reconstruction methods on SGNify forte2023reconstructing evaluation dataset. DexAvatar produces significantly better SL reconstructions with plausible body and hand poses.