EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers

TL;DR

EditYourself introduces a diffusion-transformer framework for transcript-driven editing of talking head videos, enabling addition, removal, and retiming of speech while preserving lip-sync, identity, and motion coherence. Built on a generalized video diffusion backbone, it adds cross-modal audio conditioning, latent-space visual dialog editing, and long-inference strategies including Forward–Backward RoPE conditioning and TeaCache-aware inference to maintain consistency over long sequences. The method achieves state-of-the-art lip synchronization and visual fidelity on V2V and I2V benchmarks, while delivering practical runtimes through VAE tiling, quantization, and distributed attention. By operating in latent space and coupling audio-guided edits with transcript diffing, EditYourself offers a viable tool for professional post-production and personalized content variants, though it necessitates careful ethical guardrails to prevent misuse in synthetic media generation.

Abstract

Current generative video models excel at producing novel content from text and image prompts, but leave a critical gap in editing existing pre-recorded videos, where minor alterations to the spoken script require preserving motion, temporal coherence, speaker identity, and accurate lip synchronization. We introduce EditYourself, a DiT-based framework for audio-driven video-to-video (V2V) editing that enables transcript-based modification of talking head videos, including the seamless addition, removal, and retiming of visually spoken content. Building on a general-purpose video diffusion model, EditYourself augments its V2V capabilities with audio conditioning and region-aware, edit-focused training extensions. This enables precise lip synchronization and temporally coherent restructuring of existing performances via spatiotemporal inpainting, including the synthesis of realistic human motion in newly added segments, while maintaining visual fidelity and identity consistency over long durations. This work represents a foundational step toward generative video models as practical tools for professional video post-production.

Figures (13)

• Figure 1: EditYourself is a multipurpose lip-syncing video diffusion model designed for transcription-based dialog editing, capable of lip-syncing from a single frame or an existing video, and seamlessly editing the video to match the new script.
• Figure 2: Our proposed pipeline. A global audio projection layer and audio cross-attention layers are added to the network's architecture. For V2V lip syncing, we apply noise to tokens corresponding to the mouth area and task the model with spatio-temporally inpainting them.
• Figure 3: In order to train the audio attention layers, we fully noise the tokens corresponding to mouth region throughout the training sample. We retain clean latents of the first frame, similar to image-to-video training in LTX. The model learns to in-paint the mouth through time and space using the audio, and the initial mouth shape as conditions.
• Figure 4: V2V Inference Modes. Adjusting the mask $\mathbf{M}$ in inference enables different synchronization levels: Lip for mouth-only sync, Face for expressions, and Head to synthesize new head dynamics matching the audio prosody.
• Figure 5: Example of a script-driven temporal edit, illustrating the complexity of $\text{V2V}$ operations. New content is highlighted in green, and a redaction is shown with a red strike-through, accompanied by the required duration change for each operation. Two addition operations and a removal operation are needed to account for these edits.