AV-Edit: Multimodal Generative Sound Effect Editing via Audio-Visual Semantic Joint Control

TL;DR

AV-Edit tackles the challenge of fine-grained sound effect editing in videos by jointly modeling audio, visual, and textual semantics. It introduces CAV-MAE-Edit, a contrastive audio–visual masked autoencoder that learns aligned cross-modal representations through spectrogram segmentation and audio mixing, paired with a multimodal diffusion transformer MM-DiT that applies correlation-based feature gating to edit audio synchronously with video content. The approach is validated on a new video-driven benchmark VGG-Edit and shows state-of-the-art performance in sound effect editing and competitive audio generation across multiple metrics. The work enables visually guided audio edits for video workflows, though it notes that preserving the exact original audio without distortion remains an area for improvement.

Abstract

Sound effect editing-modifying audio by adding, removing, or replacing elements-remains constrained by existing approaches that rely solely on low-level signal processing or coarse text prompts, often resulting in limited flexibility and suboptimal audio quality. To address this, we propose AV-Edit, a generative sound effect editing framework that enables fine-grained editing of existing audio tracks in videos by jointly leveraging visual, audio, and text semantics. Specifically, the proposed method employs a specially designed contrastive audio-visual masking autoencoder (CAV-MAE-Edit) for multimodal pre-training, learning aligned cross-modal representations. These representations are then used to train an editorial Multimodal Diffusion Transformer (MM-DiT) capable of removing visually irrelevant sounds and generating missing audio elements consistent with video content through a correlation-based feature gating training strategy. Furthermore, we construct a dedicated video-based sound editing dataset as an evaluation benchmark. Experiments demonstrate that the proposed AV-Edit generates high-quality audio with precise modifications based on visual content, achieving state-of-the-art performance in the field of sound effect editing and exhibiting strong competitiveness in the domain of audio generation.

Figures (7)

• Figure 1: Overview of the AV-Edit framework. The pre-trained CAV-MAE-Edit encoder extracts joint audio–visual features, which are then fed into a multimodal diffusion model to generate the edited audio.
• Figure 2: Overview of CAV-MAE-Edit network. The single-modal encoders encode the visual and audio inputs separately. The multi-modal encoder and decoder process the joint embeddings of vision and audio.
• Figure 3: The spectrograms of generated audios.
• Figure 4: Examples of three editing manipulations: add, remove and replace.
• Figure B: Curves of different loss functions during CAV-MAE-Edit training process