NePHIM: A Neural Physics-Based Head-Hand Interaction Model

TL;DR

A novel volumetric and physics‐based interaction simulation that incorporates temporal effects such as collision paths, respects anatomical constraints, and can detect and simulate skin pulling can achieve more natural‐looking interaction animations and take a step towards greater realism.

Abstract

Due to the increasing use of virtual avatars, the animation of head-hand interactions has recently gained attention. To this end, we present a novel volumetric and physics-based interaction simulation. In contrast to previous work, our simulation incorporates temporal effects such as collision paths, respects anatomical constraints, and can detect and simulate skin pulling. As a result, we can achieve more natural-looking interaction animations and take a step towards greater realism. However, like most complex and computationally expensive simulations, ours is not real-time capable even on high-end machines. Therefore, we train small and efficient neural networks as accurate approximations that achieve about 200 FPS on consumer GPUs, about 50 FPS on CPUs, and are learned in less than four hours for one person. In general, our focus is not to generalize the approximation networks to low-resolution head models but to adapt them to more detailed personalized avatars. Nevertheless, we show that these networks can learn to approximate our head-hand interaction model for multiple identities while maintaining computational efficiency. Since the quality of the simulations can only be judged subjectively, we conducted a comprehensive user study which confirms the improved realism of our approach. In addition, we provide extensive visual results and inspect the neural approximations quantitatively. All data used in this work has been recorded with a multi--view camera rig and will be made available upon publication. We will also publish relevant implementations.

Figures (8)

• Figure 2: Overview of the three stages of our approach. a) Data capturing as described in Section \ref{['sec:res:data']}. b) All steps of our physics-based simulation $\texttt{phy}$ as explained in Section \ref{['sec:mtd:sim']}. c) Efficient neural approximation net of phy as explained in Section \ref{['sec:mtd:net']}.
• Figure 3: a) Visualization of pushing as described in Section \ref{['sec:mtd:sim:fi']} and Algorithm \ref{['alg:push']}. Here, $\epsilon$ is a substep between the time steps $t-1$ and $t$. b) Illustration of a finger cylinder with radius $r$, length $l$, and an exemplary ridge shape that is used for pulling as described in Algorithm \ref{['alg:pull']}.
• Figure 4: An overview of the efficient network architecture of net. Basically, a simple MLP with only 65.536 parameters.
• Figure 5: The figure shows examples of our simulation $\texttt{phy}$ and compares them to the tracked surfaces as well as the simulation of Decafshimada2023decaf. In the top left, for example, the advantage of simulating the skull becomes apparent near the cheekbone. In the top right image, a pulling interaction is shown and the lower images demonstrate the importance of collision paths.
• Figure 6: The figure shows examples of our simulation $\texttt{phy}$ along with the tracked surfaces as well as the learned neural approximation $\texttt{net}$ (trained on all identities in our dataset). As can be recognized, the quality of the approximation is independent of whether it is a pushing or a pulling interaction.