Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits

TL;DR

The paper tackles the challenge of maintaining audio-visual coherence during edits by proposing a sequential pipeline: first performing high-quality video edits, then editing the audio to align with those changes. It introduces a video-to-audio model that conditions on source audio, target video, and a text prompt, extending MMAudio with conditional audio input and training-time data augmentation. A hierarchical acoustic feature representation, detail-temporal masking, and adaptive conditioning enable selective preservation of the original audio structure based on edit difficulty, improving audio-visual alignment and content fidelity. Extensive experiments on AvED-Bench demonstrate superior alignment and structure preservation over baselines, with ablations confirming the importance of each component. The work enables higher-frame-rate, coherent edits and suggests future extensions to more editing tasks such as object manipulation.

Abstract

We introduce a novel pipeline for joint audio-visual editing that enhances the coherence between edited video and its accompanying audio. Our approach first applies state-of-the-art video editing techniques to produce the target video, then performs audio editing to align with the visual changes. To achieve this, we present a new video-to-audio generation model that conditions on the source audio, target video, and a text prompt. We extend the model architecture to incorporate conditional audio input and propose a data augmentation strategy that improves training efficiency. Furthermore, our model dynamically adjusts the influence of the source audio based on the complexity of the edits, preserving the original audio structure where possible. Experimental results demonstrate that our method outperforms existing approaches in maintaining audio-visual alignment and content integrity.

Figures (10)

• Figure 1: An overview of the proposed pipeline.
• Figure 2: Hierarchical acoustic features.
• Figure 3: The architecture of our model. The major modifications from MMAudio are shown in purple.
• Figure 4: Detail-temporal masking of the acoustic features during training.
• Figure 5: The adaptive conditioning based on the editability score.