Learning a Generalized Physical Face Model From Data

TL;DR

This work aims to make physics-based facial animation more accessible by proposing a generalized physical face model that the authors learn from a large 3D face dataset that can be quickly fit to any unseen identity and produce a ready-to-animate physical face model automatically.

Abstract

Physically-based simulation is a powerful approach for 3D facial animation as the resulting deformations are governed by physical constraints, allowing to easily resolve self-collisions, respond to external forces and perform realistic anatomy edits. Today's methods are data-driven, where the actuations for finite elements are inferred from captured skin geometry. Unfortunately, these approaches have not been widely adopted due to the complexity of initializing the material space and learning the deformation model for each character separately, which often requires a skilled artist followed by lengthy network training. In this work, we aim to make physics-based facial animation more accessible by proposing a generalized physical face model that we learn from a large 3D face dataset. Once trained, our model can be quickly fit to any unseen identity and produce a ready-to-animate physical face model automatically. Fitting is as easy as providing a single 3D face scan, or even a single face image. After fitting, we offer intuitive animation controls, as well as the ability to retarget animations across characters. All the while, the resulting animations allow for physical effects like collision avoidance, gravity, paralysis, bone reshaping and more.

Figures (16)

• Figure 1: Overview of our model. Driven by identity $\beta$ and expression $\gamma$ latent codes, $\mathcal{N}_C$ learns to deform a canonical material space $\Omega^C$ to be identity-specific, $\Omega^{0,\beta}$. Then $\mathcal{N}_e$ learns to deform the material space to match a given expression, $\Omega^{\gamma,\beta}$. The latent codes are parameterized by a common 3DMM ($\hat{\beta}$, $\hat{\gamma}$). The network is trained with physically-inspired constraints so that, after discretization, a simulator will produce physics-based facial animation.
• Figure 2: Detailed architecture of the identity branch. The size of the inputs, the latent codes, and the output are shown on the links.
• Figure 3: Model fitting to a 3D scan. Seven examples are shown for different identities and different expressions. After fitting, the predicted actuations and bones allow to simulate the final facial skin mesh, which matches the input with a very low error.
• Figure 4: Model fitting to a single face image. Seven different identities and expressions are shown. After fitting, the predicted actuations and bones allow to simulate the final facial skin mesh. The result is an animatable physical model from a very lightweight input.
• Figure 5: Collision handling. Our physical model is able to accurately detect and resolve interpenetrating geometries.