PTalker: Personalized Speech-Driven 3D Talking Head Animation via Style Disentanglement and Modality Alignment

TL;DR

PTalker tackles the challenge of personalized, speech-driven 3D talking head animation by jointly disentangling speaking style from audio content and motion, and by aligning audio with 3D mesh across spatial, temporal, and feature domains. It introduces a graph-based spatial encoder, audio and motion style/content encoders, an identity-aware style fusion, and a three-level modality alignment with dedicated losses to enforce both local and global cross-modal coherence. Key contributions include comprehensive disentanglement constraints, a three-level alignment strategy, and extensive evaluations showing superior lip-sync accuracy and style fidelity on VOCASET and BIWI, with demonstrated generalization to unseen speakers and languages. The work advances realistic, identity-consistent talking heads with practical implications for avatars, VR, and streaming, while also outlining future work on disentanglement with limited data and improved cross-subject generalization.

Abstract

Speech-driven 3D talking head generation aims to produce lifelike facial animations precisely synchronized with speech. While considerable progress has been made in achieving high lip-synchronization accuracy, existing methods largely overlook the intricate nuances of individual speaking styles, which limits personalization and realism. In this work, we present a novel framework for personalized 3D talking head animation, namely "PTalker". This framework preserves speaking style through style disentanglement from audio and facial motion sequences and enhances lip-synchronization accuracy through a three-level alignment mechanism between audio and mesh modalities. Specifically, to effectively disentangle style and content, we design disentanglement constraints that encode driven audio and motion sequences into distinct style and content spaces to enhance speaking style representation. To improve lip-synchronization accuracy, we adopt a modality alignment mechanism incorporating three aspects: spatial alignment using Graph Attention Networks to capture vertex connectivity in the 3D mesh structure, temporal alignment using cross-attention to capture and synchronize temporal dependencies, and feature alignment by top-k bidirectional contrastive losses and KL divergence constraints to ensure consistency between speech and mesh modalities. Extensive qualitative and quantitative experiments on public datasets demonstrate that PTalker effectively generates realistic, stylized 3D talking heads that accurately match identity-specific speaking styles, outperforming state-of-the-art methods. The source code and supplementary videos are available at: PTalker.

Figures (5)

• Figure 1: Demonstrations of diverse speaking styles, highlighting variations in mouth opening and closing amplitudes (outlined in red and green borders, respectively), and lip shape changes and plumpness, such as lip pouting or curling (outlined in blue border).
• Figure 2: Illustration of PTalker. The input motion sequence $M_{1:T}$ is first encoded by the graph attention encoder $E_G$ to produce $G_{1:T}$. Then, the motion style encoder $E_{m}^{s}$ and the motion content encoder $E_{m}^{c}$ extract the motion style $s_m$ and content features $c_{1:T}$ from $G_{1:T}$. The synchronized raw speech $\chi$ is encoded into audio features $a_{1:T}$ and audio style $s_a$ by the audio content encoder $E_{a}^{c}$ and the audio style encoder $E_{a}^{s}$. The identity style encoder $E_{p}^{s}$ encodes the one-hot vector into the personal style $s_p$, where $s_p$, combined with $s_m$ and $s_a$, produces the final style code $s$. The motion decoder $\mathcal{D}$ reconstructs the facial motions $\hat{M}_{1:T}$ by combining $s$ with $c_{1:T}$ and $a_{1:T}$. During inference, $E_{m}^{c}$ and $E_{m}^{s}$ encode the generated frame sequence from the driving audio to obtain the motion style and content. All dashed and solid lines in the figure together constitute the training process, while excluding the dashed lines represents the inference process.
• Figure 3: Visualization results on the VOCASET-Test (left) and the BIWI-Test-B (right). We compare the mouth movements on some key syllables generated by our method against competitors.
• Figure 4: Visual comparison results under different personal styles with the same sentence by the t-SNE on BIWI-Test-B.
• Figure 5: Visual comparisons of the generalization evaluation using an Italian speech on the VOCASET dataset.