SoundBrush: Sound as a Brush for Visual Scene Editing

TL;DR

SoundBrush investigates sound-conditioned visual editing by extending Latent Diffusion Models (LDM) to treat audio as an editing signal. It builds a large sound-scene paired dataset, including fully synthetic and real-data subsets, and learns a mapping from audio features to textual tokens in the LDM space, enabling edits guided by in-the-wild sounds. The approach combines $L_{ ext{LDM}}$, $L_{ ext{NCE}}$, and $|V^A|_1$ objectives with LoRA fine-tuning, achieving superior 2D editing quality and enabling 3D scene manipulation via InstructNeRF2NeRF, as shown by both objective metrics and user studies. This work broadens multimodal editing by leveraging auditory cues, with practical implications for sound-driven content creation and 3D visualization.

Abstract

We propose SoundBrush, a model that uses sound as a brush to edit and manipulate visual scenes. We extend the generative capabilities of the Latent Diffusion Model (LDM) to incorporate audio information for editing visual scenes. Inspired by existing image-editing works, we frame this task as a supervised learning problem and leverage various off-the-shelf models to construct a sound-paired visual scene dataset for training. This richly generated dataset enables SoundBrush to learn to map audio features into the textual space of the LDM, allowing for visual scene editing guided by diverse in-the-wild sound. Unlike existing methods, SoundBrush can accurately manipulate the overall scenery or even insert sounding objects to best match the audio inputs while preserving the original content. Furthermore, by integrating with novel view synthesis techniques, our framework can be extended to edit 3D scenes, facilitating sound-driven 3D scene manipulation. Demos are available at https://soundbrush.github.io/.

Figures (13)

• Figure 1: Sound-guided visual scene editing. We propose SoundBrush, a model that can modify visual scenes by adding sounding objects and adjusting the scene to align with the input sound (left). Furthermore, it can be extended to edit 3D visual scenes using the input sound (right).
• Figure 2: Our proposed approach. We start by designing an automatic dataset-construction pipeline as in (a). The dataset is constructed with a fully synthetic subset, involving synthetically generated image pairs paired with audio, and real data involved subset, involving real audio and images. Using this dataset, we train SoundBrush to effectively translate audio features into audio tokens, which can then be used as control signals for the image editing latent diffusion model as described in (b).
• Figure 3: Qualitative comparison. We compare our model with existing sound-guided visual scene editing methods and demonstrate that our model can edit visual scenes using diverse in-the-wild audio while preserving the rest of the content unchanged.
• Figure 4: Editing results by different audio intensities. SoundBrush captures the intensity differences in the audio and reflects these changes in the edited images.
• Figure 5: Quantitative comparison and user study. We compare SoundBrush with existing methods and demonstrate that it outperforms them in overall metrics (a). We also provide human evaluation results that align with the quantitative comparisons (b).