Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separator

TL;DR

This work introduces AV-GAS, an audio-visual generation and separation framework that generates images from mixed soundscapes and supports both a unified image representing all present classes and separate images for each class. The method uses a two-module architecture consisting of an audio-visual separator (ResNet-18-based) and a BigGAN-based image generator, trained with synthetic two-class mixtures and aligned to ground-truth audio and visual representations via InfoNCE losses. It also defines the Class Representation Score (CRS) and a modified R@K metric to evaluate mixed-audio generation, reporting improvements over the Sound2Scene baseline on the VGGSound dataset. The results demonstrate that AV-GAS can produce plausible multi-class scenes from soundscapes and can generate separated visuals for each class, offering new capabilities for audio-visual synthesis and separation with potential applications in education and healthcare.

Abstract

Recent audio-visual generative models have made substantial progress in generating images from audio. However, existing approaches focus on generating images from single-class audio and fail to generate images from mixed audio. To address this, we propose an Audio-Visual Generation and Separation model (AV-GAS) for generating images from soundscapes (mixed audio containing multiple classes). Our contribution is threefold: First, we propose a new challenge in the audio-visual generation task, which is to generate an image given a multi-class audio input, and we propose a method that solves this task using an audio-visual separator. Second, we introduce a new audio-visual separation task, which involves generating separate images for each class present in a mixed audio input. Lastly, we propose new evaluation metrics for the audio-visual generation task: Class Representation Score (CRS) and a modified R@K. Our model is trained and evaluated on the VGGSound dataset. We show that our method outperforms the state-of-the-art, achieving 7% higher CRS and 4% higher R@2* in generating plausible images with mixed audio.

Figures (6)

• Figure 1: Comparison between our approach and existing methods. Our approach processes a mixed audio input to generate images, whereas existing methods generate images given single-class audio and fail to generate plausible images given mixed audio. Our method can be used for two tasks: first, to generate an image containing all classes present in the audio (task1: audio-visual generation (coloured in red)); second, to generate a separate image for each object present in the audio (task2: audio-visual separation (coloured in blue)). Our model is the first that can generate single images or multiple class-separated images from mixed audio.
• Figure 2: Our audio-visual training input example. We create new audio-visual training input tuples using VGGSound Chen20VggSound to process a mixed audio signal. Since our task aims to generate images from mixed audio, we mix two audio classes and create the audio-visual training input tuples with the mixed audio, two ground truth separated audio for sound separation and two ground truth separated images that cannot be mixed like audio.
• Figure 3: Overview of our approach. Our approach includes two modules: the separation module (for training) and the generation module (for inference). In the separation module, audio-visual training input tuples extracted from video samples are used for training. The audio-visual separator learns to distinguish objects in mixed audio using contrastive learning loss without class labels. The audio and/or image embeddings extracted from the pre-trained encoder are compared to the embeddings from the audio-visual separator. In the generation module, the two embeddings extracted from the audio-visual separator are used for the two types of image generation. Each embedding itself generates separate images (task: audio-visual separation (coloured in blue)) with the image generator, and if two embeddings are combined with the control parameter $\lambda$, mixed images are generated (task2: audio-visual generation (coloured in red)).
• Figure 4: Results of audio-visual generation (generating images given mixed audio) Existing methods av_vae_pedersoli2022estimating(Baseline)av_gan7_fanzeres2022soundtoimagination(S2I)sung2023sound focus on generating images from a single-class audio input, whereas our method focuses on generating images from a mixed audio input. We show the results from the state-of-the-art sung2023sound and ours based on realistic and unrealistic mixed sounds. The results given a mixed audio input show how well our model generates images given mixed audio. The first two columns in each part are the results of the state-of-the-art sung2023sound and ours (AV-GAS) when the same mixed audio is given. The third and fourth columns show other images generated by our approach.
• Figure 5: Two types of images generated from our model. Our model generates two types of images: mixed images (task: audio-visual generation, highlighted in red) and separated images (task: audio-visual separation, highlighted in blue). Each row shows when the same mixed audio is given to our model and the state-of-the-art sung2023sound. The existing method fails to plausibly generate images given mixed audio, whereas our approach generates images containing all sound classes. Moreover, our model can generate a separate image for each audio class which is not feasible in previous approaches.