OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers

TL;DR

OmniSync tackles universal lip synchronization by removing the need for reference frames and explicit masks, addressing identity preservation, pose variation, and stylization in diverse visuals. It introduces a diffusion-transformer-based mask-free editing framework, a flow-matching progressive noise initialization to stabilize inference, and a dynamic spatiotemporal CFG to balance audio conditioning with visual fidelity. The work also provides the AIGC-LipSync Benchmark to evaluate lip-sync performance in AI-generated content across real, stylized, and non-human subjects. Experimental results show significant gains in both visual quality and lip-sync accuracy over state-of-the-art methods, including in challenging AI-generated scenarios. These contributions enable more reliable, scalable lip synchronization for film dubbing, digital avatars, and downstream AI video workflows.

Abstract

Lip synchronization is the task of aligning a speaker's lip movements in video with corresponding speech audio, and it is essential for creating realistic, expressive video content. However, existing methods often rely on reference frames and masked-frame inpainting, which limit their robustness to identity consistency, pose variations, facial occlusions, and stylized content. In addition, since audio signals provide weaker conditioning than visual cues, lip shape leakage from the original video will affect lip sync quality. In this paper, we present OmniSync, a universal lip synchronization framework for diverse visual scenarios. Our approach introduces a mask-free training paradigm using Diffusion Transformer models for direct frame editing without explicit masks, enabling unlimited-duration inference while maintaining natural facial dynamics and preserving character identity. During inference, we propose a flow-matching-based progressive noise initialization to ensure pose and identity consistency, while allowing precise mouth-region editing. To address the weak conditioning signal of audio, we develop a Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG) mechanism that adaptively adjusts guidance strength over time and space. We also establish the AIGC-LipSync Benchmark, the first evaluation suite for lip synchronization in diverse AI-generated videos. Extensive experiments demonstrate that OmniSync significantly outperforms prior methods in both visual quality and lip sync accuracy, achieving superior results in both real-world and AI-generated videos.

Figures (9)

• Figure 1: OmniSync demonstrates universal lip synchronization capabilities, effectively handling facial occlusion, while maintaining visual consistency and generating accurate lip movements.
• Figure 2: Overview of OmniSync. A mask-free training paradigm employs timestep-dependent sampling to predict the lip-synchronized targets $V_{ab}$. During inference, progressive noise initialization and dynamic spatiotemporal CFG ensure consistent head pose and precise lip synchronization.
• Figure 3: Qualitative comparison with previous methods across diverse subjects and phonemes. Our approach produces more accurate lip synchronization and better identity preservation.
• Figure 4: Ablation study for timestep-dependent sampling strategy and different CFG settings.
• Figure 5: Comparison with portrait animation methods. Visual comparison between our OmniSync framework and other approaches (EchoMimic, Hallo3, and Sonic), demonstrating our method's superior ability to preserve identity and natural speaking style while maintaining accurate lip synchronization.