Seeing What You Say: Expressive Image Generation from Speech

TL;DR

VoxStudio addresses the problem of expressive image generation from speech by proposing an end-to-end S2I model that directly maps spoken descriptions to visuals while preserving paralinguistic cues. It introduces the Speech Information Bottleneck (SIB) to condense long speech embeddings into compact tokens that are still rich in semantic and affective information, enabling effective cross-attention conditioning in a diffusion-based image generator. The VoxEmoset dataset provides a scalable, emotion-rich benchmark synthesized from multiple emotion corpora to train and evaluate the model. Experiments on SpokenCOCO, Flickr8kAudio, and VoxEmoset show VoxStudio achieves strong emotional fidelity and competitive semantic alignment, demonstrating practical potential for voice-driven image synthesis and editing.

Abstract

This paper proposes VoxStudio, the first unified and end-to-end speech-to-image model that generates expressive images directly from spoken descriptions by jointly aligning linguistic and paralinguistic information. At its core is a speech information bottleneck (SIB) module, which compresses raw speech into compact semantic tokens, preserving prosody and emotional nuance. By operating directly on these tokens, VoxStudio eliminates the need for an additional speech-to-text system, which often ignores the hidden details beyond text, e.g., tone or emotion. We also release VoxEmoset, a large-scale paired emotional speech-image dataset built via an advanced TTS engine to affordably generate richly expressive utterances. Comprehensive experiments on the SpokenCOCO, Flickr8kAudio, and VoxEmoset benchmarks demonstrate the feasibility of our method and highlight key challenges, including emotional consistency and linguistic ambiguity, paving the way for future research.

Figures (19)

• Figure 1: Generated images by VoxStudio from spoken descriptions.
• Figure 2: (a) The cascaded system consisting of ASR and T2I, and (b) audio-to-text feature mapping-based methods soundbrushspeechclip+ limits in cost than (c) VoxStudio (ours). The diffusion process is excluded from GFLOPs and time computations. The parameters of the image generator are also excluded from Params.
• Figure 3: VoxStudio encodes an utterance to generate an expressive image. SIB compresses speech embedding into compact semantic tokens to condition the image generator; training the generator is an optional choice. Trainable parts are optimized by the diffusion loss.
• Figure 4: Examples from SpokenCOCO (neutral tone) and VoxEmoset (expressive tone).
• Figure 5: Qualitative comparison between SD using text prompts and VoxStudio using speech prompts.