JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion

TL;DR

JUST-DUB-IT reframes video dubbing as a single, joint audio–visual generation task by adapting a pretrained audio–visual diffusion backbone with a lightweight LoRA. It synthesizes paired multilingual supervision via language-switching and inpainting, and employs context-aligned positional encoding with modality-isolated cross-attention to preserve identity and synchronize lip motion with translated speech. The approach demonstrates robust audiovisual coherence, accurate lip synchronization, and resilience to real-world dynamics, outperforming modular baselines on standard and unconstrained datasets. This work highlights the potential of strong audio–visual priors for holistic multimodal editing tasks and paves the way for more integrated dubbing systems that respect scene context and paralinguistic cues.

Abstract

Audio-Visual Foundation Models, which are pretrained to jointly generate sound and visual content, have recently shown an unprecedented ability to model multi-modal generation and editing, opening new opportunities for downstream tasks. Among these tasks, video dubbing could greatly benefit from such priors, yet most existing solutions still rely on complex, task-specific pipelines that struggle in real-world settings. In this work, we introduce a single-model approach that adapts a foundational audio-video diffusion model for video-to-video dubbing via a lightweight LoRA. The LoRA enables the model to condition on an input audio-video while jointly generating translated audio and synchronized facial motion. To train this LoRA, we leverage the generative model itself to synthesize paired multilingual videos of the same speaker. Specifically, we generate multilingual videos with language switches within a single clip, and then inpaint the face and audio in each half to match the language of the other half. By leveraging the rich generative prior of the audio-visual model, our approach preserves speaker identity and lip synchronization while remaining robust to complex motion and real-world dynamics. We demonstrate that our approach produces high-quality dubbed videos with improved visual fidelity, lip synchronization, and robustness compared to existing dubbing pipelines.

Figures (12)

• Figure 1: Video dubbing via joint audio–visual generation.Top: an input video with spoken dialogue in the source language. Bottom: the same video dubbed into a target language, generated by our trained model built on top of an audio–visual foundation backbone. Translated speech and lip motion are produced jointly, while the visual context (such as scene dynamics, face expressions, and body movements), speaker identity, and non-speech events (e.g., pauses, background sounds) are preserved.
• Figure 2: Pipeline for Generating Paired Audio-Visual Dubbing Data. The pipeline consists of two stages. First, the audio–visual generation model produces a contiguous sequence containing a context clip (e.g., spoken English) followed by a target clip (e.g., spoken French). Second, the audio and lip-region video of the target clip are masked, and the same unified model is used in an inpainting setting to regenerate the masked content, conditioned on the context clip and a new text prompt (e.g., re-dubbing the target into English).
• Figure 3: The Identity–Pronunciation Trade-off. Naïve audio inpainting reveals a fundamental conflict between preserving speaker identity and achieving linguistically correct pronunciation. When denoising from scratch (Left), the model exhibits voice drift, failing to preserve the speaker’s vocal identity. When conditioning on the source audio to maintain identity (Middle), phonetic and prosodic patterns leak across languages, resulting in prosody leakage. Our approach (Right) resolves this trade-off by conditioning generation on a reference clip that preserves speaker identity while exhibiting the target-language phonetic style.
• Figure 4: Model Training. Our framework follows an in-context generation paradigm where clean context audio-visual pairs are concatenated with noised target pairs. We fine-tune only LoRA adapters while keeping a pre-trained Audio-Visual (AV) Diffusion Transformer frozen. Conditioned on a text prompt (e.g. "The person is speaking in French"), the model learns to propagate edits from the context while maintaining temporal synchronization between audio and video. We introduce a modality-specific masking strategy in AV cross-attention, ensuring that noisy audio attends only to noisy video and vice versa, since conditioning a noisy signal on clean context from the other modality leads to signal leakage and conflicting guidance, which this masking prevents.
• Figure 5: User Study Results. We compare our method against LatentSync and HeyGen through a user study, evaluating Lip Synchronization, Prompt Adherence, and Overall Quality. Results indicate that participants prefer our method over baselines across all evaluated metrics.