OT-Talk: Animating 3D Talking Head with Optimal Transportation

TL;DR

OT-Talk addresses the challenge of aligning speech with 3D facial dynamics by treating head meshes as probability measures via oriented varifolds and optimizing with a sliced Wasserstein distance. It combines Hubert-based audio features, a temporal transformer, and Chebyshev graph convolution to learn geometry-aware mesh embeddings, with an auto-encoder pipeline predicting framewise vertex displacements. The key contributions include the first application of optimal transport to talking-head animation, a ChebNet mesh representation, and strong empirical results on VOCASET and Multiface, reinforced by a user perception study. This approach yields smoother, more accurately synchronized 3D talking heads, with potential impact on AR/VR, gaming, and film production; it also opens avenues for diffusion-model-based extensions to capture richer expressive styles. $L = L_r + \beta_1 L_v + \beta_2 L_{SW} + \beta_3 ||W||_2$ demonstrates how reconstruction, temporal coherence, and distributional matching jointly drive high-quality animations.

Abstract

Animating 3D head meshes using audio inputs has significant applications in AR/VR, gaming, and entertainment through 3D avatars. However, bridging the modality gap between speech signals and facial dynamics remains a challenge, often resulting in incorrect lip syncing and unnatural facial movements. To address this, we propose OT-Talk, the first approach to leverage optimal transportation to optimize the learning model in talking head animation. Building on existing learning frameworks, we utilize a pre-trained Hubert model to extract audio features and a transformer model to process temporal sequences. Unlike previous methods that focus solely on vertex coordinates or displacements, we introduce Chebyshev Graph Convolution to extract geometric features from triangulated meshes. To measure mesh dissimilarities, we go beyond traditional mesh reconstruction errors and velocity differences between adjacent frames. Instead, we represent meshes as probability measures and approximate their surfaces. This allows us to leverage the sliced Wasserstein distance for modeling mesh variations. This approach facilitates the learning of smooth and accurate facial motions, resulting in coherent and natural facial animations. Our experiments on two public audio-mesh datasets demonstrate that our method outperforms state-of-the-art techniques both quantitatively and qualitatively in terms of mesh reconstruction accuracy and temporal alignment. In addition, we conducted a user perception study with 20 volunteers to further assess the effectiveness of our approach.

Figures (8)

• Figure 1: System Pipeline
• Figure 2: Network Architecture
• Figure 7: Scores on VOCASET Dataset
• Figure 8: Scores on Multiface Dataset
• Figure : GT OT-Talk FaceFormer CodeTalker VOCA