GenSync: A Generalized Talking Head Framework for Audio-driven Multi-Subject Lip-Sync using 3D Gaussian Splatting

TL;DR

GenSync introduces a unified multi-speaker lip-sync framework built on 3D Gaussian Splatting that disentangles identity from audio to support multiple speakers without per-identity training. The Identity-Aware Disentanglement Module and Fused Spatial-Audio Attention Network enable audio-driven, identity-conditioned deformations via a canonical 3D face representation. Empirical results show competitive lip-sync accuracy and visual quality with dramatically reduced training time (about 6.8× faster) and robust performance under identity switching and novel audio distributions. This approach significantly improves scalability for multi-identity talking head synthesis with practical implications for real-time audiovisual production and virtual avatar applications.

Abstract

We introduce GenSync, a novel framework for multi-identity lip-synced video synthesis using 3D Gaussian Splatting. Unlike most existing 3D methods that require training a new model for each identity , GenSync learns a unified network that synthesizes lip-synced videos for multiple speakers. By incorporating a Disentanglement Module, our approach separates identity-specific features from audio representations, enabling efficient multi-identity video synthesis. This design reduces computational overhead and achieves 6.8x faster training compared to state-of-the-art models, while maintaining high lip-sync accuracy and visual quality.

Figures (4)

• Figure 1: Overview of GenSync's pipeline. The Deformation Module decouples representations from the audio features ($a$) and the identity vector ($i$) using a multiplicative transform (Eqn. \ref{['eq:mult']}). The Fused Spatial-Audio Attention Module computes cross attention among the canonical features $f(\mu_c)$ and the concatenation of the eye features ($e$), viewpoint ($v$), and the output of the Disentanglement Module. This is used to compute the deformations using an MLP $F_d$, which dynamically deforms the static canonical face output by the canonical network $F_c$cho2024gaussiantalker. Multiple identities are shown here for illustration purposes.
• Figure 2: Comparative results between GenSync (ours) and GaussianTalker (baseline) for frame-wise images from a rendered video. The red-highlighted text indicates the current syllable being spoken. GenSync utilizes a single shared model across all identities, whereas the baseline requires separately trained models. Despite this, GenSync achieves performance comparable to GaussianTalker.
• Figure 3: GenSync's output for Identity A and Identity B. We use A's learned identity embedding to generate B's video and we see the speaking style (mouth more open) transferred from A to B.
• Figure 4: GenSync's output for Identity B, C and D with Identity A's audio (novel audio distribution) for the same time step. GenSync shows robustness to distribution shifts in the driving audio. For instance, the model generates plausible lip movement for male speakers C and D even when the driving audio is from female speaker A.