T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation

TL;DR

T2AV-Compass tackles fragmented evaluation in text-to-audio-video generation by introducing a taxonomy-driven, 500-prompt benchmark and a unified dual-level evaluation framework that combines objective signal metrics with a reasoning-first MLLM-based judge. The benchmark emphasizes cross-modal alignment, temporal synchronization, instruction following, and perceptual realism, and undergoes extensive testing across 11 representative T2AV systems, revealing a persistent Audio Realism Bottleneck and gaps between open-source and closed-source models. By integrating data construction, real-world grounding via video inversion, and rich diagnostic signals (including compositional prompts and QA-based evaluation), the work provides a practical, diagnostic foundation to push forward both evaluation and modeling in T2AV generation. The framework highlights the potential benefits of native audiovisual diffusion approaches and ongoing human-in-the-loop improvements to bridge the realism gap and enhance cross-modal fidelity.

Abstract

Text-to-Audio-Video (T2AV) generation aims to synthesize temporally coherent video and semantically synchronized audio from natural language, yet its evaluation remains fragmented, often relying on unimodal metrics or narrowly scoped benchmarks that fail to capture cross-modal alignment, instruction following, and perceptual realism under complex prompts. To address this limitation, we present T2AV-Compass, a unified benchmark for comprehensive evaluation of T2AV systems, consisting of 500 diverse and complex prompts constructed via a taxonomy-driven pipeline to ensure semantic richness and physical plausibility. Besides, T2AV-Compass introduces a dual-level evaluation framework that integrates objective signal-level metrics for video quality, audio quality, and cross-modal alignment with a subjective MLLM-as-a-Judge protocol for instruction following and realism assessment. Extensive evaluation of 11 representative T2AVsystems reveals that even the strongest models fall substantially short of human-level realism and cross-modal consistency, with persistent failures in audio realism, fine-grained synchronization, instruction following, etc. These results indicate significant improvement room for future models and highlight the value of T2AV-Compass as a challenging and diagnostic testbed for advancing text-to-audio-video generation.

Figures (6)

• Figure 1: Overview of T2AV-Compass analysis and evaluation taxonomy. (a) Radial comparison of representative T2AV models under our evaluation suite. (b) Prompt token-length distribution. (c--d) Semantic diversity of video/audio prompts quantified via embedding similarity (higher indicates broader coverage). (e) Hierarchical distribution of evaluation dimensions, clearly organizing objective metrics and MLLM-based assessments across video, audio, and cross-modal alignment.
• Figure 2: Data construction and checklist-based evaluation generation. The prompt suite is constructed from (1) curated community prompts with semantic deduplication (cos $\ge$ 0.8), clustering-based sampling, LLM rewriting, and human refinement, and (2) a video-inversion stream using filtered 4–10s YouTube clips with dense captioning and manual verification. The finalized prompts are then converted into two types of checklists: instruction-alignment checks via slot extraction and dimension mapping, and perceptual-realism checks for video/audio quality.
• Figure 3: Dataset statistics of T2AV-Compass. (a) Category distributions over five annotation dimensions (Content Genre, Primary Subject, Event Scenario, Sound Category, and Camera Motion). (b) Distributions of audiovisual complexity factors, including Visual Subject Count, Event Temporal Structure, Audio Spatial Composition, and Audio Temporal Composition.
• Figure 4: Illustration of the subjective evaluation framework in T2AV-Compass. Unlike traditional metrics, our protocol provides interpretable diagnosis through two distinct tracks: (Top) Instruction following is evaluated via rigorous Q&A checklist pairs, ensuring semantic alignment in complex scenarios like social interactions and sound effects. (Bottom) Realism scrutinizes perceptual quality, rewarding fine-grained details (e.g., fur texture) while explicitly penalizing visual hallucinations (e.g., two-headed dog) or audio dissonance. The examples demonstrate the judge's ability to discern model capabilities (Veo-3.1 vs. Ovi-1.1) with grounded evidence.
• Figure 5: Macro-level comparison across six evaluation dimensions. We report the averaged Video Instruction-Following score (Video IF, Avg.) of five representative models (Veo-3.1, Wan-2.5, Ovi-1.1, PixVerse-V5.5, and Sora-2) on Aesthetics, Attribute, Cinematography, Dynamics, Relations, and World. Overall, Veo-3.1 and Wan-2.5 form the top tier with consistently strong performance; Sora-2 is competitive on Attribute and Cinema but lags on Dynamics; PixVerse exhibits mid-range performance across most dimensions; and Ovi-1.1 shows the lowest scores, with the largest gaps on Dynamics and World.