SyncAnyone: Implicit Disentanglement via Progressive Self-Correction for Lip-Syncing in the wild

TL;DR

SyncAnyone addresses the challenge of audio-driven lip-syncing in-the-wild by moving beyond mask-inpainting limitations. It introduces Progressive Self-Correction (PSC), a two-stage training framework where Stage 1 learns a robust mask-inpainting motion model to generate audio-driven lip edits, and Stage 2 trains a fast mask-free lip-editing model using pseudo-paired data, guided by a background restoration objective to preserve identity and background. The method combines Flow Matching with a Diffusion Transformer backbone and a carefully designed I2V conditioning scheme, achieving efficient inference and strong background fidelity. Extensive experiments on diverse, multilingual in-the-wild data demonstrate state-of-the-art results in visual quality, temporal coherence, and identity preservation, highlighting the practical impact for dubbing, avatars, and cross-lingual content. The work also provides a scalable workflow for implicit disentanglement via data-driven restoration, reducing reliance on controlled datasets and enabling robust performance across challenging scenarios.

Abstract

High-quality AI-powered video dubbing demands precise audio-lip synchronization, high-fidelity visual generation, and faithful preservation of identity and background. Most existing methods rely on a mask-based training strategy, where the mouth region is masked in talking-head videos, and the model learns to synthesize lip movements from corrupted inputs and target audios. While this facilitates lip-sync accuracy, it disrupts spatiotemporal context, impairing performance on dynamic facial motions and causing instability in facial structure and background consistency. To overcome this limitation, we propose SyncAnyone, a novel two-stage learning framework that achieves accurate motion modeling and high visual fidelity simultaneously. In Stage 1, we train a diffusion-based video transformer for masked mouth inpainting, leveraging its strong spatiotemporal modeling to generate accurate, audio-driven lip movements. However, due to input corruption, minor artifacts may arise in the surrounding facial regions and the background. In Stage 2, we develop a mask-free tuning pipeline to address mask-induced artifacts. Specifically, on the basis of the Stage 1 model, we develop a data generation pipeline that creates pseudo-paired training samples by synthesizing lip-synced videos from the source video and random sampled audio. We further tune the stage 2 model on this synthetic data, achieving precise lip editing and better background consistency. Extensive experiments show that our method achieves state-of-the-art results in visual quality, temporal coherence, and identity preservation under in-the wild lip-syncing scenarios.

Figures (5)

• Figure 1: Given any video and audio input, SyncAnyone can modify the mouth movements of characters in the video to synchronize with the audio. Additionally, SyncAnyone is capable of handling challenging scenarios such as large poses, background changes, occlusions, scene cuts, and diverse styles.
• Figure 2: The overall framework of SyncAnyone. The left panel illustrates our two-stage Progressive Self-Correction (PSC) training paradigm for modifying a source video's lip movements to match a target audio. In Stage 1, a multi-reference mask inpainting model is trained for robust synthesis. In Stage 2, this model is leveraged to create a pseudo-paired dataset, which in turn supervises the training of our final, efficient mask-free model. The right panel details the specific network architecture employed in our framework.
• Figure 3: Qualitative comparison of our method and other methods under different scenarios. Zoom in for better visualization.
• Figure 4: Lip-syncing results for different syllables.
• Figure 5: The comparison of the 1st and 2nd stage of our model. The regions displayed within the red box are the blurred areas in the stage-one results.