Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?

Susan Liang, Chao Huang, Filippos Bellos, Yolo Yunlong Tang, Qianxiang Shen, Jing Bi, Luchuan Song, Zeliang Zhang, Jason Corso, Chenliang Xu

TL;DR

This work tackles the challenge of evaluating text-conditioned audio–video generation, a tri-modal task lacking reliable, scalable metrics. It introduces Omni-Judge, an omni-modal LLM framework that jointly reasons over text, video, and audio to score nine perceptual and alignment dimensions while providing interpretable explanations. Through a VidProM-based prompt set and outputs from Sora 2 and Veo 3, Omni-Judge achieves correlations with human judgments comparable to traditional metrics and notably excels at semantic alignment tasks such as audio-text, video-text, and tri-modal coherence, though it lags on high-FPS perceptual metrics due to limited temporal resolution. The approach demonstrates the potential of unified omni-modal evaluators for multi-modal generation and highlights current limitations that guide future improvements in temporal sensitivity and reasoning-based feedback.

Abstract

State-of-the-art text-to-video generation models such as Sora 2 and Veo 3 can now produce high-fidelity videos with synchronized audio directly from a textual prompt, marking a new milestone in multi-modal generation. However, evaluating such tri-modal outputs remains an unsolved challenge. Human evaluation is reliable but costly and difficult to scale, while traditional automatic metrics, such as FVD, CLAP, and ViCLIP, focus on isolated modality pairs, struggle with complex prompts, and provide limited interpretability. Omni-modal large language models (omni-LLMs) present a promising alternative: they naturally process audio, video, and text, support rich reasoning, and offer interpretable chain-of-thought feedback. Driven by this, we introduce Omni-Judge, a study assessing whether omni-LLMs can serve as human-aligned judges for text-conditioned audio-video generation. Across nine perceptual and alignment metrics, Omni-Judge achieves correlation comparable to traditional metrics and excels on semantically demanding tasks such as audio-text alignment, video-text alignment, and audio-video-text coherence. It underperforms on high-FPS perceptual metrics, including video quality and audio-video synchronization, due to limited temporal resolution. Omni-Judge provides interpretable explanations that expose semantic or physical inconsistencies, enabling practical downstream uses such as feedback-based refinement. Our findings highlight both the potential and current limitations of omni-LLMs as unified evaluators for multi-modal generation.

Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?

TL;DR

This work tackles the challenge of evaluating text-conditioned audio–video generation, a tri-modal task lacking reliable, scalable metrics. It introduces Omni-Judge, an omni-modal LLM framework that jointly reasons over text, video, and audio to score nine perceptual and alignment dimensions while providing interpretable explanations. Through a VidProM-based prompt set and outputs from Sora 2 and Veo 3, Omni-Judge achieves correlations with human judgments comparable to traditional metrics and notably excels at semantic alignment tasks such as audio-text, video-text, and tri-modal coherence, though it lags on high-FPS perceptual metrics due to limited temporal resolution. The approach demonstrates the potential of unified omni-modal evaluators for multi-modal generation and highlights current limitations that guide future improvements in temporal sensitivity and reasoning-based feedback.

Abstract

State-of-the-art text-to-video generation models such as Sora 2 and Veo 3 can now produce high-fidelity videos with synchronized audio directly from a textual prompt, marking a new milestone in multi-modal generation. However, evaluating such tri-modal outputs remains an unsolved challenge. Human evaluation is reliable but costly and difficult to scale, while traditional automatic metrics, such as FVD, CLAP, and ViCLIP, focus on isolated modality pairs, struggle with complex prompts, and provide limited interpretability. Omni-modal large language models (omni-LLMs) present a promising alternative: they naturally process audio, video, and text, support rich reasoning, and offer interpretable chain-of-thought feedback. Driven by this, we introduce Omni-Judge, a study assessing whether omni-LLMs can serve as human-aligned judges for text-conditioned audio-video generation. Across nine perceptual and alignment metrics, Omni-Judge achieves correlation comparable to traditional metrics and excels on semantically demanding tasks such as audio-text alignment, video-text alignment, and audio-video-text coherence. It underperforms on high-FPS perceptual metrics, including video quality and audio-video synchronization, due to limited temporal resolution. Omni-Judge provides interpretable explanations that expose semantic or physical inconsistencies, enabling practical downstream uses such as feedback-based refinement. Our findings highlight both the potential and current limitations of omni-LLMs as unified evaluators for multi-modal generation.
Paper Structure (25 sections, 16 figures, 2 tables)

This paper contains 25 sections, 16 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Metrics for Content Generation
  4. LLM-Based Evaluation for Video and Images
  5. Omni-Modal LLMs
  6. Method
  7. Prompt Collection and Analysis
  8. Video Generation
  9. Human Evaluation Protocol
  10. Human Preference Analysis
  11. Omni-Judge Framework
  12. Framework.
  13. System Instruction.
  14. Judgment Example.
  15. Experiments
  16. ...and 10 more sections

Figures (16)

  • Figure 1: Statistical analysis of the collected 300 prompts from VidProM, illustrating the distributions across linguistic length, film genres, main subjects, visual styles, camera techniques, and sound dimensions.
  • Figure 2: (a) Comparison on color consistency. Sora 2 shows noticeable color jittering and brightness shifts (visible in optical flow), while Veo 3 produces sharper and more stable frames with smoother temporal consistency. (b) Comparison on motion continuity. Sora 2 exhibits frame freezing with static optical flow, whereas Veo 3 maintains fluent motion and higher temporal stability across frames.
  • Figure 3: (a) Examples of audio-video desynchronization. Sora 2 produces speech without matching lip motion (top), while Veo 3 generates keyboard typing sounds without corresponding hand movement (bottom), illustrating temporal misalignment between sound and action. (b) Examples of ambiguous audio generation. The models adopt different strategies to match sound with visuals: narration describing the scene (top), and background music expressing emotion and tone (bottom).
  • Figure 4: Overview of the Omni-Judge framework. (a) Each metric is defined by a task-specific instruction and the required modality inputs. The omni-LLM (Qwen3-Omni) jointly processes text, video, and audio to output a score and explanation. (b) Example of the audio-video-text alignment metric, where Omni-Judge reasons step by step across modalities to assess coherence and produce an interpretable score.
  • Figure 5: Ablation on model variants and different FPS values. We compare instruction and reasoning models across different frame rates for video quality and audio-video synchronization. Reasoning models perform better overall, and FPS affects performance without a strictly monotonic trend.
  • ...and 11 more figures