SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting

TL;DR

SyncTalk++ tackles the devil of talking-head realism by integrating Gaussian Splatting with a multi-component synchronization framework. The Face-Sync Controller, Head-Sync Stabilizer, and Dynamic Portrait Renderer jointly ensure identity preservation, lip-sync, facial expressions, and stable head poses, while the OOD Audio Expression Generator and Torso Restorer enhance robustness to out-of-distribution audio and head-torso continuity. The method delivers high fidelity at real-time speeds (up to 101 FPS) and outperforms prior state-of-the-art approaches in both quantitative metrics and user studies. Ethical considerations are discussed to mitigate potential misuse of hyperrealistic synthesis. Overall, the work provides a scalable, high-quality pipeline for practical, synchronized talking-head generation with robust cross-domain applicability.

Abstract

Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a fundamental flaw, leading to unrealistic results. To address the critical issue of synchronization, identified as the ''devil'' in creating realistic talking heads, we introduce SyncTalk++, which features a Dynamic Portrait Renderer with Gaussian Splatting to ensure consistent subject identity preservation and a Face-Sync Controller that aligns lip movements with speech while innovatively using a 3D facial blendshape model to reconstruct accurate facial expressions. To ensure natural head movements, we propose a Head-Sync Stabilizer, which optimizes head poses for greater stability. Additionally, SyncTalk++ enhances robustness to out-of-distribution (OOD) audio by incorporating an Expression Generator and a Torso Restorer, which generate speech-matched facial expressions and seamless torso regions. Our approach maintains consistency and continuity in visual details across frames and significantly improves rendering speed and quality, achieving up to 101 frames per second. Extensive experiments and user studies demonstrate that SyncTalk++ outperforms state-of-the-art methods in synchronization and realism. We recommend watching the supplementary video: https://ziqiaopeng.github.io/synctalk++.

Figures (13)

• Figure 1: The proposed SyncTalk++ uses 3D Gaussian Splatting for rendering. It can generate synchronized lip movements, facial expressions, and more stable head poses, and features faster rendering speeds while applying to high-resolution talking videos.
• Figure 2: Overview of SyncTalk++. Given a cropped reference video of a talking head and the corresponding speech, SyncTalk++ can extract the Lip Feature $f_l$, Expression Feature $f_e$, and Head Pose $(R, T)$ through two synchronization modules $(a)$ and $(b)$. Then, Gaussian Splatting is used to model and deform the head, producing a talking head video. The OOD Audio Expression Generator and Torso Restorer can generate speech-matched facial expressions and repair artifacts at head-torso junctions.
• Figure 3: Visualization of reconstruction quality. The Audio-Visual Encoder effectively captures and reconstructs lip movements.
• Figure 4: Facial Animation Capturer. We use 3D facial blendshape coefficients to capture the expressions of characters.
• Figure 5: Overview of Gaussian Rendering. Canonical Gaussian fields utilize a triplane representation to encode 3D head features, which are processed by the MLP to yield canonical parameters. These parameters are then integrated with lip feature, expression feature, and head pose parameters in the deformable Gaussian fields. This design facilitates the generation of high-fidelity talking head, achieving realistic and dynamic facial animations.