Capturing Head Avatar with Hand Contacts from a Monocular Video

TL;DR

This work tackles reconstructing photoreal head avatars from monocular video while capturing natural hand-face interactions. It introduces a two-stage pipeline: preprocessing that jointly aligns hand and face meshes using a depth-order loss $L_{\text{order}}$ and a contact regularization $L_{\text{contact}}$, followed by neural implicit avatar reconstruction with a PCA-guided non-rigid deformation prior learned from hand-face interaction data. The face is represented by a trio of neural networks ($f_{\sigma_g}$, $f_{\sigma_d}$, $f_{\sigma_r}$) with surface points mapped to a canonical space via $LBS^{-1}$, and the hand is modeled through an occupancy-based representation and texture MLP; non-rigid deformations are produced by $f_{\sigma_n}$ and constrained by a PCA prior $\mathcal{N}^{\text{GT}}$ and a contact loss to prevent interpenetration. Training optimizes a multi-term objective, incorporating RGB, mask, FLAME priors, and contact/regularization terms into $\mathcal{L}_{\text{total}}$, enabling robust learning from monocular RGB video alone. Evaluations on real iPhone RGB(D) data and a synthetic dataset show improved appearance fidelity, accurate hand-face geometry, and physically plausible deformations compared with state-of-the-art baselines, demonstrating practical potential for telepresence and AR/VR applications.

Abstract

Photorealistic 3D head avatars are vital for telepresence, gaming, and VR. However, most methods focus solely on facial regions, ignoring natural hand-face interactions, such as a hand resting on the chin or fingers gently touching the cheek, which convey cognitive states like pondering. In this work, we present a novel framework that jointly learns detailed head avatars and the non-rigid deformations induced by hand-face interactions. There are two principal challenges in this task. First, naively tracking hand and face separately fails to capture their relative poses. To overcome this, we propose to combine depth order loss with contact regularization during pose tracking, ensuring correct spatial relationships between the face and hand. Second, no publicly available priors exist for hand-induced deformations, making them non-trivial to learn from monocular videos. To address this, we learn a PCA basis specific to hand-induced facial deformations from a face-hand interaction dataset. This reduces the problem to estimating a compact set of PCA parameters rather than a full spatial deformation field. Furthermore, inspired by physics-based simulation, we incorporate a contact loss that provides additional supervision, significantly reducing interpenetration artifacts and enhancing the physical plausibility of the results. We evaluate our approach on RGB(D) videos captured by an iPhone. Additionally, to better evaluate the reconstructed geometry, we construct a synthetic dataset of avatars with various types of hand interactions. We show that our method can capture better appearance and more accurate deforming geometry of the face than SOTA surface reconstruction methods.

Figures (6)

• Figure 1: Given an RGB video capturing hand-face interactions, our method automatically aligns tracked hand and face meshes, reconstructs high-fidelity 3D surfaces, and renders photorealistic textures. We further model contact-induced non-rigid deformations through learned blendshape fields guided by a non-rigid deformation PCA prior derived from hand-face interaction data.
• Figure 2: Pipeline of our method Our framework operates through two stages: Preprocessing aligns separately tracked FLAME (face) and MANO (hand) meshes into a unified coordinate system via joint optimization of depth ordering loss and contact loss. Reconstruction learns neural deformation fields for the head avatar, with a contact-specific non-rigid deformation network. This specialized component, regularized by contact losses, explicitly models facial surface deformation induced by hand-face interaction.
• Figure 3: Qualitative Comparison with SOTA on Captured Videos Reconstruction results from captured video sequences comparing our method with SOTA baselines. Our method trains exclusively on RGB video input, whereas NDR and Morpheus require LiDAR-derived depth maps. Qualitative comparisons demonstrate our approach achieves superior hand and face reconstruction fidelity while faithfully recovering physically plausible facial deformations from hand interactions. The final columns visualize our learned blendshape fields alongside photorealistic rendering outputs.
• Figure 4: Qualitative Comparison with SOTA on Synthetic Videos Reconstruction results from synthetic video sequences comparing our method with SOTA baselines. To ensure comparative fairness, we use the depth information in our method as NDR and Morpheus. Our approach achieves superior reconstruction of both hand geometry and facial features while maintaining physically consistent non-rigid deformations from hand-face interactions. The final column presents ground truth normal maps for reference, demonstrating our method's ability to recover intricate surface details.
• Figure 5: Ablation Study in the Preprocessing Stage The contact loss guides the hand mesh toward the surface of the face mesh to establish contact, as illustrated in Figure \ref{['fig:ablation_preprocess']}. Regions of contact between the meshes are visualized as red pixels on the face mesh. The depth order loss plays a critical role in ensuring plausible interactions by preventing excessive interpenetration of the meshes. We also compare our method with Pixie feng2021collaborative in the last column.