TAVGBench: Benchmarking Text to Audible-Video Generation

TL;DR

This work defines the Text to Audible-Video Generation (TAVG) task and introduces TAVGBench, a large-scale benchmark with over 1.7 million audio-visual pairs and an automatic annotation pipeline to describe both audio and video components. It proposes AVHScore to quantify audio-visual alignment and presents TAVDiffusion, a two-stream latent diffusion baseline that uses cross-attention and a contrastive alignment loss to jointly generate coherent audio and video from text. Across extensive experiments, TAVDiffusion demonstrates strong performance on conventional metrics and the proposed AVHScore, highlighting the dataset's utility for advancing audible-video generation research. The work also outlines potential applications and future directions, including integrating a multimodal diffusion transformer for unified generation.

Abstract

The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both audio and video elements. To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 thousand hours. We propose an automatic annotation pipeline to ensure each audible video has detailed descriptions for both its audio and video contents. We also introduce the Audio-Visual Harmoni score (AVHScore) to provide a quantitative measure of the alignment between the generated audio and video modalities. Additionally, we present a baseline model for TAVG called TAVDiffusion, which uses a two-stream latent diffusion model to provide a fundamental starting point for further research in this area. We achieve the alignment of audio and video by employing cross-attention and contrastive learning. Through extensive experiments and evaluations on TAVGBench, we demonstrate the effectiveness of our proposed model under both conventional metrics and our proposed metrics.

Figures (6)

• Figure 1: Comparison of the proposed TAVG task with existing generation tasks. (a) Text to video generation (TVG) generates the corresponding videos through text descriptions. (b) Text to audio generation (TAG) generates the corresponding audio through text descriptions. (c) The proposed TAVG task generates audio-visual content based on text descriptions of both audio and video elements.
• Figure 2: Overview of the annotation pipeline. We employ BLIP2 for video descriptions and WavCaps for audio descriptions. The descriptions are further refined using ChatGPT, resulting in the final detailed audible video description.
• Figure 3: Data samples. The video (we give three frames for each video clip), audio, and the corresponding generated captions. We highlight the video caption in black and the audio caption in blue.
• Figure 4: Overview of the TAVDiffusion training (left) and inference (right) stages. We develop a two-stream architecture. During the training phase, we randomly select a timestep $t$ and employ diffusion loss to guide the single-step denoising. In the inference phase, iterative denoising is conducted to finally produce an audible video.
• Figure 5: Qualitative comparison. We compare our TAVDiffusion model with methods (1) and (2). Given the inferior visual quality of method (3) in our task (see Table \ref{['tab:results']}), we exclude it from the qualitative comparison. Best viewed on screen.