From Inpainting to Editing: A Self-Bootstrapping Framework for Context-Rich Visual Dubbing

TL;DR

This work tackles the data scarcity problem in audio-driven visual dubbing by reframing the task as context-rich video editing rather than mask-based inpainting. It introduces X-Dub, a self-bootstrapping framework where a DiT-based generator creates lip-varied companion videos to form frame-aligned training pairs, and a DiT-based editor learns mask-free dubbing from these pairs using complete visual context and audio guidance. A timestep-adaptive multi-phase learning strategy with LoRA experts disentangles global structure, lip motion, and texture across diffusion timesteps, enabling stable training and superior lip synchronization and identity preservation. The ContextDubBench benchmark and comprehensive experiments demonstrate state-of-the-art performance, robustness to in-the-wild conditions, and practical viability for real-world dubbing applications.

Abstract

Audio-driven visual dubbing aims to synchronize a video's lip movements with new speech, but is fundamentally challenged by the lack of ideal training data: paired videos where only a subject's lip movements differ while all other visual conditions are identical. Existing methods circumvent this with a mask-based inpainting paradigm, where an incomplete visual conditioning forces models to simultaneously hallucinate missing content and sync lips, leading to visual artifacts, identity drift, and poor synchronization. In this work, we propose a novel self-bootstrapping framework that reframes visual dubbing from an ill-posed inpainting task into a well-conditioned video-to-video editing problem. Our approach employs a Diffusion Transformer, first as a data generator, to synthesize ideal training data: a lip-altered companion video for each real sample, forming visually aligned video pairs. A DiT-based audio-driven editor is then trained on these pairs end-to-end, leveraging the complete and aligned input video frames to focus solely on precise, audio-driven lip modifications. This complete, frame-aligned input conditioning forms a rich visual context for the editor, providing it with complete identity cues, scene interactions, and continuous spatiotemporal dynamics. Leveraging this rich context fundamentally enables our method to achieve highly accurate lip sync, faithful identity preservation, and exceptional robustness against challenging in-the-wild scenarios. We further introduce a timestep-adaptive multi-phase learning strategy as a necessary component to disentangle conflicting editing objectives across diffusion timesteps, thereby facilitating stable training and yielding enhanced lip synchronization and visual fidelity. Additionally, we propose ContextDubBench, a comprehensive benchmark dataset for robust evaluation in diverse and challenging practical application scenarios.

Figures (14)

• Figure 1: Moving beyond mask-inpainting, X-Dub redefines visual dubbing as context-rich, full-reference video-to-video editing, which yields precise lip synchronization and faithful identity preservation, even in challenging scenarios with occlusions and dynamic lighting.
• Figure 2: Overview of X-Dub, our self-bootstrapping dubbing framework. At its core, our paradigm employs a DiT generator to create a lip-altered counterpart for each video, forming a context-rich pair with the original (left). A DiT editor then learns mask-free, video-to-video dubbing directly from these ideal pairs, leveraging the complete visual context to ensure accurate lip sync and identity preservation (middle). This contextual learning is further refined by our timestep-adaptive multi-phase learning (right), which aligns different diffusion stages with learning distinct information: global structure, lip movements, and texture details, respectively.
• Figure 3: Conditioning mechanisms for our DiT-based framework. Reference conditions (full contextual video frames for editor; a single reference frame for generator) and the target video are concatenated into a unified sequence for 3D self-attention. Audio is injected via cross-attention.
• Figure 4: Qualitative comparisons across diverse scenarios. Lip-sync errors are marked with yellow, visual artifacts with blue, and lip leakage during silence with red. "ERROR" indicates runtime failure from missing 3DMM or landmarks despite best efforts. Our method exhibits robust performance with superior lip accuracy and identity consistency. Please zoom in for details.
• Figure 5: Ablations on reference video conditioning and multi-phase learning. Replacing our frame-wise token-sequence concatenation (abbr. concat) with channel concat for reference conditioning causes conflicts and thus lowers lip-sync accuracy. Removing the lip or texture phase degrades lip synchronization and detailed texture fidelity, respectively. Please zoom in for details.