Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Audio Generation Diversity with Visual Information

Zeyu Xie, Baihan Li, Xuenan Xu, Mengyue Wu, Kai Yu

TL;DR

This work proposes a clustering-based method, leveraging visual information to guide the model in generating distinct audio content within each category, and results indicate that extra visual input can largely enhance audio generation diversity.

Abstract

Audio and sound generation has garnered significant attention in recent years, with a primary focus on improving the quality of generated audios. However, there has been limited research on enhancing the diversity of generated audio, particularly when it comes to audio generation within specific categories. Current models tend to produce homogeneous audio samples within a category. This work aims to address this limitation by improving the diversity of generated audio with visual information. We propose a clustering-based method, leveraging visual information to guide the model in generating distinct audio content within each category. Results on seven categories indicate that extra visual input can largely enhance audio generation diversity. Audio samples are available at https://zeyuxie29.github.io/DiverseAudioGeneration.

Enhancing Audio Generation Diversity with Visual Information

TL;DR

This work proposes a clustering-based method, leveraging visual information to guide the model in generating distinct audio content within each category, and results indicate that extra visual input can largely enhance audio generation diversity.

Abstract

Audio and sound generation has garnered significant attention in recent years, with a primary focus on improving the quality of generated audios. However, there has been limited research on enhancing the diversity of generated audio, particularly when it comes to audio generation within specific categories. Current models tend to produce homogeneous audio samples within a category. This work aims to address this limitation by improving the diversity of generated audio with visual information. We propose a clustering-based method, leveraging visual information to guide the model in generating distinct audio content within each category. Results on seven categories indicate that extra visual input can largely enhance audio generation diversity. Audio samples are available at https://zeyuxie29.github.io/DiverseAudioGeneration.
Paper Structure (12 sections, 3 equations, 2 figures, 3 tables)

This paper contains 12 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Vision-guided audio generation
  3. Modal Fusion
  4. Audio Representation
  5. Token Prediction
  6. Experiment Setup
  7. Generation Framework
  8. Evaluation Metrics
  9. Result
  10. Diversity Enhancement
  11. Overall Quality
  12. Conclusion

Figures (2)

  • Figure 1: Vision-guided generated samples.
  • Figure 2: The diagram of audio generation framework. The input condition is processed via a modal fusion module. Red and green arrows represent the model training, including 1) red: VAE/VQ-VAE learning for audio representation, and 2) green: token prediction based on input conditions. Blue arrows show the inference process: the token prediction model predicts the audio representation, which is then restored to a spectrogram by the VAE/VQ-VAE decoder. Finally, the audio is generated by a vocoder.