MACS: Multi-source Audio-to-image Generation with Contextual Significance and Semantic Alignment

TL;DR

MACS addresses the realistic scenario of multi-source audio-to-image generation by first separating mixed audio into constituent sources and then synthesizing images conditioned on the separated embeddings. It introduces a weakly supervised multi-source sound separation module with a mixture-of-mixtures training objective, CLAP-based audio-text alignment via ranking and contrastive losses, and a decoupled cross-attention adapter to fuse multiple audio signals during diffusion. The approach is evaluated across multi-source, mixed-source, and single-source tasks on LLP-multi, AudioSet-Eval, and Landscape, where MACS achieves state-of-the-art or competitive results and demonstrates notable qualitative improvements in semantic fidelity and visual realism. This work advances cross-modal synthesis in realistic auditory scenes and provides a scalable framework for integrating multiple audio cues into diffusion-based image generation.

Abstract

Propelled by the breakthrough in deep generative models, audio-to-image generation has emerged as a pivotal cross-modal task that converts complex auditory signals into rich visual representations. However, previous works only focus on single-source audio inputs for image generation, ignoring the multi-source characteristic in natural auditory scenes, thus limiting the performance in generating comprehensive visual content. To bridge this gap, we propose a method called MACS to conduct multi-source audio-to-image generation. To our best knowledge, this is the first work that explicitly separates multi-source audio to capture the rich audio components before image generation. MACS is a two-stage method. In the first stage, multi-source audio inputs are separated by a weakly supervised method, where the audio and text labels are semantically aligned by casting into a common space using the large pre-trained CLAP model. We introduce a ranking loss to consider the contextual significance of the separated audio signals. In the second stage, effective image generation is achieved by mapping the separated audio signals to the generation condition using only a trainable adapter and a MLP layer. We preprocess the LLP dataset as the first full multi-source audio-to-image generation benchmark. The experiments are conducted on multi-source, mixed-source, and single-source audio-to-image generation tasks. The proposed MACS outperforms the current state-of-the-art methods in 17 out of the 21 evaluation indexes on all tasks and delivers superior visual quality.

Figures (6)

• Figure 1: Qualitative and Quantitative Comparison of MACS and Other SOTA Methods.Upper: Generated images from single-source and multi-source audio datasets. Lower: Normalized radar maps (Left: single-source; Right: multi-source). FID, CLIP-FID, and KID are inverted.
• Figure 2: An overview of the proposed two-stage MACS architecture.Stage 1: A Multi-source Sound Separation (MSS) model decomposes audio mixtures into sub-audios using reconstruction loss. The separated audios are embedded with CLAP, guided by contrastive and ranking losses to ensure audio-text semantic alignment and contextual significance. Stage 2: A diffusion-based generator uses a decoupled cross-attention module to integrate audio embeddings and produce high-quality, semantically accurate images. MACS enables scalable MSS pre-training and image generation with fewer trainable layers.
• Figure 3: The average standard deviation of the similarity scores between the text labels of each audio mixture and its separations across three datasets.
• Figure 4: Ranking loss helps sort the contextual significance. Embeddings in higher ranking contain more important semantic information for accurate image generation.
• Figure 5: Generated images from interpolations between two audio clips (dog bark and motor vehicle).