Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text-to-Audio Generation Synchronized with Videos

Shentong Mo, Jing Shi, Yapeng Tian

TL;DR

The paper tackles the misalignment between audio and video in text-to-audio generation by introducing T2AV-Bench and a video-conditioned diffusion framework, T2AV. It advances the field with three new metrics for visual alignment and temporal consistency, and a model that fuses visual semantics into TTA via Visual-aligned CLAP and Audio-Visual ControlNet. Empirical results on AudioCaps and T2AV-Bench demonstrate state-of-the-art synchronization and audio fidelity, reinforced by thorough ablations on data scale and conditioning mechanisms. This work enables more faithful audio generation that tracks visual content, with implications for audiovisual content creation and retrieval systems.

Abstract

In recent times, the focus on text-to-audio (TTA) generation has intensified, as researchers strive to synthesize audio from textual descriptions. However, most existing methods, though leveraging latent diffusion models to learn the correlation between audio and text embeddings, fall short when it comes to maintaining a seamless synchronization between the produced audio and its video. This often results in discernible audio-visual mismatches. To bridge this gap, we introduce a groundbreaking benchmark for Text-to-Audio generation that aligns with Videos, named T2AV-Bench. This benchmark distinguishes itself with three novel metrics dedicated to evaluating visual alignment and temporal consistency. To complement this, we also present a simple yet effective video-aligned TTA generation model, namely T2AV. Moving beyond traditional methods, T2AV refines the latent diffusion approach by integrating visual-aligned text embeddings as its conditional foundation. It employs a temporal multi-head attention transformer to extract and understand temporal nuances from video data, a feat amplified by our Audio-Visual ControlNet that adeptly merges temporal visual representations with text embeddings. Further enhancing this integration, we weave in a contrastive learning objective, designed to ensure that the visual-aligned text embeddings resonate closely with the audio features. Extensive evaluations on the AudioCaps and T2AV-Bench demonstrate that our T2AV sets a new standard for video-aligned TTA generation in ensuring visual alignment and temporal consistency.

Text-to-Audio Generation Synchronized with Videos

TL;DR

The paper tackles the misalignment between audio and video in text-to-audio generation by introducing T2AV-Bench and a video-conditioned diffusion framework, T2AV. It advances the field with three new metrics for visual alignment and temporal consistency, and a model that fuses visual semantics into TTA via Visual-aligned CLAP and Audio-Visual ControlNet. Empirical results on AudioCaps and T2AV-Bench demonstrate state-of-the-art synchronization and audio fidelity, reinforced by thorough ablations on data scale and conditioning mechanisms. This work enables more faithful audio generation that tracks visual content, with implications for audiovisual content creation and retrieval systems.

Abstract

In recent times, the focus on text-to-audio (TTA) generation has intensified, as researchers strive to synthesize audio from textual descriptions. However, most existing methods, though leveraging latent diffusion models to learn the correlation between audio and text embeddings, fall short when it comes to maintaining a seamless synchronization between the produced audio and its video. This often results in discernible audio-visual mismatches. To bridge this gap, we introduce a groundbreaking benchmark for Text-to-Audio generation that aligns with Videos, named T2AV-Bench. This benchmark distinguishes itself with three novel metrics dedicated to evaluating visual alignment and temporal consistency. To complement this, we also present a simple yet effective video-aligned TTA generation model, namely T2AV. Moving beyond traditional methods, T2AV refines the latent diffusion approach by integrating visual-aligned text embeddings as its conditional foundation. It employs a temporal multi-head attention transformer to extract and understand temporal nuances from video data, a feat amplified by our Audio-Visual ControlNet that adeptly merges temporal visual representations with text embeddings. Further enhancing this integration, we weave in a contrastive learning objective, designed to ensure that the visual-aligned text embeddings resonate closely with the audio features. Extensive evaluations on the AudioCaps and T2AV-Bench demonstrate that our T2AV sets a new standard for video-aligned TTA generation in ensuring visual alignment and temporal consistency.
Paper Structure (14 sections, 4 equations, 3 figures, 8 tables)

This paper contains 14 sections, 4 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. T2AV-Bench & Metrics
  4. Benchmark Details
  5. Metrics Details
  6. Metrics Validation
  7. Method
  8. Preliminaries
  9. Visual-aligned CLAP
  10. Audio-Visual ControlNet
  11. Experiments
  12. Experimental setup
  13. Comparison to Prior Work
  14. Conclusion

Figures (3)

  • Figure 1: Comparison of our T2AV with state-of-the-art methods on the proposed T2AV-Bench in terms of FAD, FAVD, FATD, and FA(VT)D for video-aligned text-to-audio generation. Our method significantly outperforms previous baselines in terms of all metrics (lower is better).
  • Figure 2: Illustration of the proposed framework for Text-to-audio generation aligned with videos (T2AV). The Audio-Visual ControlNet aggregates temporal video features as the condition in the text-based latent diffusion models. Then, a contrastive language-audio pretraining objective across each temporal location is applied to match visual-aligned text embeddings with audio features. After visual-aligned CLAP pre-training, we directly extract text embeddings with Audio-Visual ControlNet as the condition for latent diffusion models to achieve Text-to-Audio generation with visual alignment and temporal consistency.
  • Figure 3: Qualitative comparisons with AudioLDM on video-aligned TTA generation. The proposed T2AV produces more accurate and aligned audio for target videos.