SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models

TL;DR

SonicDiffusion addresses the gap in multimodal conditioning for large diffusion models by enabling audio-driven image generation and editing. It introduces an Audio Projector that maps audio clips to tokens aligned with text-space embeddings and injects them through gated cross-attention adapters into a frozen Stable Diffusion backbone, enabling parameter-efficient training. The method employs a two-stage training pipeline with contrastive and MSE losses for audio-text alignment and a DDPM-based loss for cross-modal conditioning, achieving state-of-the-art semantic fidelity (AIS, AIC, IIS) and image quality (FID) across diverse datasets, including landscape and audio-visual scenes. Overall, SonicDiffusion demonstrates robust audio-visual alignment and editable image synthesis with practical potential for multimodal content creation and editing.

Abstract

We are witnessing a revolution in conditional image synthesis with the recent success of large scale text-to-image generation methods. This success also opens up new opportunities in controlling the generation and editing process using multi-modal input. While spatial control using cues such as depth, sketch, and other images has attracted a lot of research, we argue that another equally effective modality is audio since sound and sight are two main components of human perception. Hence, we propose a method to enable audio-conditioning in large scale image diffusion models. Our method first maps features obtained from audio clips to tokens that can be injected into the diffusion model in a fashion similar to text tokens. We introduce additional audio-image cross attention layers which we finetune while freezing the weights of the original layers of the diffusion model. In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods to enable audio conditioned image editing. We demonstrate our method on a wide range of audio and image datasets. We perform extensive comparisons with recent methods and show favorable performance.

Figures (10)

• Figure 1: An overview of our proposed SonicDiffusion model. Our framework allows for two core functionalities: (1) audio-driven image generation, and (2) audio-guided image editing. In (1), both sound inputs and optional text prompts are tokenized, guiding the denoising process via text-to-image and audio-to-image cross-attention layers. For (2), the process begins with the inversion of the input image using DDIM. Subsequently, extracted spatial features and self-attention maps are integrated into the generation process, complemented by audio-conditioned cross-attention maps to obtain the desired changes.
• Figure 2: Training pipeline of SonicDiffusion comprises two distinct stages: (1) aligning audio features with CLIP's semantic space, and (2) sound-driven parameter-efficient tuning of the Stable Diffusion (SD) model. In stage (1), the audio projector module is trained to transform audio clips into semantically rich tokens, employing MSE and contrastive loss functions. Stage (2) involves integrating gated cross-attention layers, which facilitate interaction between image and audio modalities, into the existing SD framework. Only these newly added layers, alongside the audio projector module, are adjusted to enable audio-conditioned image synthesis.
• Figure 3: Audio Projector utilizes CLAP audio encoder clap for initial feature extraction, followed by a mapper with 1D convolutions and deconvolutions. Four self-attention layers are then integrated to enhance learning effectiveness of the projector.
• Figure 4: Gated cross-attention module. Our method guides the diffusion process by gated cross-attention and dense feed-forward layers added after pre-trained text-conditioned layers, using audio tokens as keys/values and ensuring training stability and quality.
• Figure 5: Comparison against the state-of-the-art audio-driven image synthesis methods. Our model generates images that closely align with the semantics of the input audio clips, surpassing all other existing methods in performance and fidelity.