MTAVG-Bench: A Comprehensive Benchmark for Evaluating Multi-Talker Dialogue-Centric Audio-Video Generation

TL;DR

MTAVG-Bench introduces a comprehensive benchmark for evaluating multi-talker text-to-audio-video generation, addressing structural and cross-modal failures beyond perceptual realism. It builds a four-level evaluation framework—Signal Fidelity, Attribute Consistency, Social Interaction, and Cinematic Expression—with nine fine-grained dimensions and a semi-automatic data-generation pipeline that yields 1.8k videos and 2.4k diagnostic QA pairs. Through evaluation of twelve models, including Gemini 3 Pro, MTAVG-Bench reveals that perceptual quality often outpaces social-structural coherence, with speaker identity, turn-taking, and audiovisual grounding as key bottlenecks; Gemini 3 Pro shows the strongest overall performance, particularly in interaction and cinematic metrics. The benchmark enables precise failure diagnosis and targeted video refinement, offering a foundation for advances in reliable, controllable multi-talker audiovisual generation and multimodal learning systems.

Abstract

Recent advances in text-to-audio-video (T2AV) generation have enabled models to synthesize audio-visual videos with multi-participant dialogues. However, existing evaluation benchmarks remain largely designed for human-recorded videos or single-speaker settings. As a result, potential errors that occur in generated multi-talker dialogue videos, such as identity drift, unnatural turn transitions, and audio-visual misalignment, cannot be effectively captured and analyzed. To address this issue, we introduce MTAVG-Bench, a benchmark for evaluating audio-visual multi-speaker dialogue generation. MTAVG-Bench is built via a semi-automatic pipeline, where 1.8k videos are generated using multiple popular models with carefully designed prompts, yielding 2.4k manually annotated QA pairs. The benchmark evaluates multi-speaker dialogue generation at four levels: audio-visual signal fidelity, temporal attribute consistency, social interaction, and cinematic expression. We benchmark 12 proprietary and open-source omni-models on MTAVG-Bench, with Gemini 3 Pro achieving the strongest overall performance, while leading open-source models remain competitive in signal fidelity and consistency. Overall, MTAVG-Bench enables fine-grained failure analysis for rigorous model comparison and targeted video generation refinement.

Figures (12)

• Figure 1: MTAVG-Bench is a benchmark for evaluating text-to-audio-video (T2AV) models on multi-talker dialogue generation, built by synthesizing dialogue-driven videos from structured prompts and collecting human annotations based on carefully defined fine-grained evaluation dimensions. It features a four-level evaluation framework and diverse multi-choice and pairwise questions that assess signal quality, consistency, social interaction, and cinematic expression, with a focus on failure mode in cinematic speaker-centric dialogue video generation.
• Figure 2: Data distribution of MTAVG-Bench.
• Figure 3: MTAVG-Bench construction and annotation pipeline. Multi-speaker dialogues are first rewritten by an LLM into structured prompts and used to generate multi-talker audio-visual clips with Veo 3.1, Wan 2.5, and Sora 2. The generated videos are analyzed to discover fine-grained failure cases, which are systematically mapped to a unified set of failure/evaluation dimensions. Based on this failure-dimension mapping, a failure-aware QA generator produces diverse evaluation questions that are further validated and refined by human experts.
• Figure 4: Qualitative Result for Instruction-Following under the Turn-Taking Logic Dimension.
• Figure 5: The specific system prompt for decomposing text descriptions into hierarchical semantic levels.