Speak the Art: A Direct Speech to Image Generation Framework

TL;DR

The paper tackles direct speech-to-image generation, a task lagging behind text-to-image due to weak speech representations and unstable training. It proposes STA, a two-stage framework where a speech encoding network, guided by a large image-text model, produces embeddings that condition a VQ-Diffusion image generator, replacing GANs for improved stability and diversity. A multilingual extension (MSTA) demonstrates English and Arabic support with performance comparable to monolingual setups. Results on CUB-200, Oxford-102, and Flickr8K show STA achieving state-of-the-art or near-state-of-the-art performance, significantly reducing the gap to text-to-image methods and enabling unwritten-language applicability.

Abstract

Direct speech-to-image generation has recently shown promising results. However, compared to text-to-image generation, there is still a large gap to enclose. Current approaches use two stages to tackle this task: speech encoding network and image generative adversarial network (GAN). The speech encoding networks in these approaches produce embeddings that do not capture sufficient linguistic information to semantically represent the input speech. GANs suffer from issues such as non-convergence, mode collapse, and diminished gradient, which result in unstable model parameters, limited sample diversity, and ineffective generator learning, respectively. To address these weaknesses, we introduce a framework called \textbf{Speak the Art (STA)} which consists of a speech encoding network and a VQ-Diffusion network conditioned on speech embeddings. To improve speech embeddings, the speech encoding network is supervised by a large pre-trained image-text model during training. Replacing GANs with diffusion leads to more stable training and the generation of diverse images. Additionally, we investigate the feasibility of extending our framework to be multilingual. As a proof of concept, we trained our framework with two languages: English and Arabic. Finally, we show that our results surpass state-of-the-art models by a large margin.

Figures (6)

• Figure 1: The whole framework of STA. It consists of two stages: a speech encoding network to extract embeddings and a VQ-Diffusion model to generate high-quality images. Text is only used for illustration.
• Figure 2: The architecture of the speech encoding network.
• Figure 3: (a) An example of a multilingual dataset. (b) The architecture of VQ-Diffusion model.
• Figure 4: Visual comparison on CUB-200 and Oxford-102 datasets.
• Figure 5: Visual comparison on Flickr8k dataset.