OmniEdit: A Training-free framework for Lip Synchronization and Audio-Visual Editing

TL;DR

This approach reformulates the editing paradigm by substituting the edit sequence in FlowEdit with the target sequence, yielding an unbiased estimation of the desired output, and by removing stochastic elements from the generation process, establishes a smooth and stable editing trajectory.

Abstract

Lip synchronization and audio-visual editing have emerged as fundamental challenges in multimodal learning, underpinning a wide range of applications, including film production, virtual avatars, and telepresence. Despite recent progress, most existing methods for lip synchronization and audio-visual editing depend on supervised fine-tuning of pre-trained models, leading to considerable computational overhead and data requirements. In this paper, we present OmniEdit, a training-free framework designed for both lip synchronization and audio-visual editing. Our approach reformulates the editing paradigm by substituting the edit sequence in FlowEdit with the target sequence, yielding an unbiased estimation of the desired output. Moreover, by removing stochastic elements from the generation process, we establish a smooth and stable editing trajectory. Extensive experimental results validate the effectiveness and robustness of the proposed framework. Code is available at https://github.com/l1346792580123/OmniEdit.

Figures (4)

• Figure 1: Overview of OmniEdit.(a) Conditioned on the target audio, OmniEdit leverages a pre-trained audio-to-video diffusion model to synchronize the lip movements in the source video with the target audio signal. (b) Utilizing an audio–visual generation model, OmniEdit performs concurrent modification of audio and video modalities according to target prompt.
• Figure 2: Qualitative results of our proposed method and other methods. Our method achieves more accurate lip synchronization and produces clearer dental details. Please zoom in to observe the fine-grained improvements.
• Figure 3: Ablation study for edit sequence and random noise. Edit sequence and stochastic noise injection tends to produce blurred dental details, whereas our method is capable of generating sharper and more clearly defined teeth.
• Figure 4: Qualitative results of OmniEdit for Audio-visual Editing. Our approach supports prompt-based manipulation of diverse attributes—including age (a), gender (b), person (c), emotion (d), behaviors (e), and even car categories (f), while jointly generating audio and video in a temporally synchronized and semantically consistent manner.