SeeingSounds: Learning Audio-to-Visual Alignment via Text

TL;DR

SeeingSounds addresses audio-to-visual generation without requiring paired data by grounding audio in language and vision through a tri-modal alignment that uses frozen encoders and lightweight adapters. It maps audio to semantic text via a frozen language model and grounds text to vision through a CLIP-like encoder, while keeping the diffusion backbone frozen; only adapters are trained. This enables controllable, interpretable generation where audio transformations translate into textual prompts guiding the visuals. Empirical results across VEGAS, VGGSound, Landscape+Into the Wild, RAVDESS, and zero-shot ESC-50 demonstrate state-of-the-art performance and strong generalization, with efficient training and the ability to compose scenes from mixed audio sources.

Abstract

We introduce SeeingSounds, a lightweight and modular framework for audio-to-image generation that leverages the interplay between audio, language, and vision-without requiring any paired audio-visual data or training on visual generative models. Rather than treating audio as a substitute for text or relying solely on audio-to-text mappings, our method performs dual alignment: audio is projected into a semantic language space via a frozen language encoder, and, contextually grounded into the visual domain using a vision-language model. This approach, inspired by cognitive neuroscience, reflects the natural cross-modal associations observed in human perception. The model operates on frozen diffusion backbones and trains only lightweight adapters, enabling efficient and scalable learning. Moreover, it supports fine-grained and interpretable control through procedural text prompt generation, where audio transformations (e.g., volume or pitch shifts) translate into descriptive prompts (e.g., "a distant thunder") that guide visual outputs. Extensive experiments across standard benchmarks confirm that SeeingSounds outperforms existing methods in both zero-shot and supervised settings, establishing a new state of the art in controllable audio-to-visual generation.

Figures (4)

• Figure 1: Overview of audio-text and visual modal alignment in Seeing Sounds. During training (left), audio features $E_A$ are aligned to both language and vision-language spaces via lightweight adapters ($A_T$ and $A_V$) and projected (through $P_T$ and $P_V$) using MSE losses against frozen text ($E_T$)and vision-text ($E_V$) encoders. At inference (right), the audio-text-vision-aligned projection conditions a frozen diffusion model for audio-driven image generation, enabling controllable and paired-free synthesis.
• Figure 2: Qualitative results on VEGAS (5-class subset). Our model yields semantically accurate generations aligned with the input audio category. Images adapted from sound2scene2023 to include our results.
• Figure 3: Volume controllability: controllable generation based on audio intensity. Lower volume yields smaller or distant visual representations.
• Figure 4: Mixed-class controllability. Each cell visualizes a scene combining the row and column audio classes (e.g. helicopter flying + rail transport).