KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution

TL;DR

KeySync tackles leakage-free, high-resolution lip synchronization by combining a leakage-aware, two-stage latent-diffusion model with automatic occlusion handling. It introduces a leakage-proof masking strategy and an inference-time segmentation-based occlusion removal to minimize expression leakage and artifacts in the presence of occluders. The approach achieves state-of-the-art lip reconstruction and cross-synchronization at $512 × 512$, introduces LipLeak as a direct leakage metric, and is validated through extensive quantitative and human studies, including ablations on architecture, audio backbone, and masking. Overall, KeySync significantly improves realism and synchronization quality, with strong practical implications for automated dubbing and cross-language avatar applications.

Abstract

Lip synchronization, known as the task of aligning lip movements in an existing video with new input audio, is typically framed as a simpler variant of audio-driven facial animation. However, as well as suffering from the usual issues in talking head generation (e.g., temporal consistency), lip synchronization presents significant new challenges such as expression leakage from the input video and facial occlusions, which can severely impact real-world applications like automated dubbing, but are often neglected in existing works. To address these shortcomings, we present KeySync, a two-stage framework that succeeds in solving the issue of temporal consistency, while also incorporating solutions for leakage and occlusions using a carefully designed masking strategy. We show that KeySync achieves state-of-the-art results in lip reconstruction and cross-synchronization, improving visual quality and reducing expression leakage according to LipLeak, our novel leakage metric. Furthermore, we demonstrate the effectiveness of our new masking approach in handling occlusions and validate our architectural choices through several ablation studies. Code and model weights can be found at https://antonibigata.github.io/KeySync.

Figures (21)

• Figure 1: KeySync's contributions. Unlike existing methods, KeySync generates high-resolution lip-synced videos that are closely aligned with the driving audio while minimizing leakage from the input video and seamlessly handling facial occlusions.
• Figure 2: Overview of the KeySync framework. KeySync consists of two stages, both of which involve generating video using latent diffusion conditioned on an input video and audio, differing only in the reference frames selection, as described in (b). During keyframe generation, the model receives an identity frame $x_{\text{id}}$, which is repeated and concatenated with the noised video input. During interpolation, the model is conditioned on two successive keyframes $z_{i}$ and $z_{i+1}$, along with intermediate learnable embeddings $z_m$. Both stages integrate audio embeddings $a$ from HuBERT hubert. In (c), we illustrate our occlusion handling pipeline, which we apply during inference.
• Figure 3: Qualitative comparison with other works. The top row (“Target lips”) shows lip movements corresponding directly to the provided audio input, and can therefore be seen as the target for the lips in the generated videos.
• Figure 4: Qualitative leakage comparison. We condition the models on silent audio and non-silent video (first row).
• Figure 5: MAR over time. If MAR exceeds the threshold, the mouth is considered open, indicating leakage.