Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Judge Model for Large-scale Multimodality Benchmarks

Min-Han Shih, Yu-Hsin Wu, Yu-Wei Chen

TL;DR

The work introduces a dedicated multimodal Judge Model to evaluate large-scale multimodal benchmarks spanning text, audio, image, and video. By aggregating judgments, verifying reasoning, and providing diagnostic feedback, the Judge offers an explainable evaluation pipeline that aligns closely with human annotators on 280 multimodal samples. The framework compares Gemini-2.5, Phi-4, and Qwen-2.5 across diverse modalities, revealing cross-modal robustness and modality-specific weaknesses. It also demonstrates the potential to reuse judge outputs as supervision signals for future model tuning, enabling scalable and interpretable assessments for multimodal AI systems. Overall, the approach provides a reproducible, ground-truth-aware evaluation strategy with actionable feedback for advancing multimodal reasoning capabilities.

Abstract

We propose a dedicated multimodal Judge Model designed to provide reliable, explainable evaluation across a diverse suite of tasks. Our benchmark spans text, audio, image, and video modalities, drawing from carefully sampled public datasets with fixed seeds to ensure reproducibility and minimize train test leakage. Instead of simple scoring, our framework aggregates multimodal judgments, analyzes the quality and reasoning consistency of model outputs, and generates diagnostic feedback. We evaluate several MLLMs, including Gemini 2.5, Phi 4, and Qwen 2.5, across 280 multimodal samples and compare judge model assessments with human annotators. Results show strong alignment between the Judge Model and human scores, demonstrating its potential as a scalable, interpretable evaluation pipeline for future multimodal AI research.

Judge Model for Large-scale Multimodality Benchmarks

TL;DR

The work introduces a dedicated multimodal Judge Model to evaluate large-scale multimodal benchmarks spanning text, audio, image, and video. By aggregating judgments, verifying reasoning, and providing diagnostic feedback, the Judge offers an explainable evaluation pipeline that aligns closely with human annotators on 280 multimodal samples. The framework compares Gemini-2.5, Phi-4, and Qwen-2.5 across diverse modalities, revealing cross-modal robustness and modality-specific weaknesses. It also demonstrates the potential to reuse judge outputs as supervision signals for future model tuning, enabling scalable and interpretable assessments for multimodal AI systems. Overall, the approach provides a reproducible, ground-truth-aware evaluation strategy with actionable feedback for advancing multimodal reasoning capabilities.

Abstract

We propose a dedicated multimodal Judge Model designed to provide reliable, explainable evaluation across a diverse suite of tasks. Our benchmark spans text, audio, image, and video modalities, drawing from carefully sampled public datasets with fixed seeds to ensure reproducibility and minimize train test leakage. Instead of simple scoring, our framework aggregates multimodal judgments, analyzes the quality and reasoning consistency of model outputs, and generates diagnostic feedback. We evaluate several MLLMs, including Gemini 2.5, Phi 4, and Qwen 2.5, across 280 multimodal samples and compare judge model assessments with human annotators. Results show strong alignment between the Judge Model and human scores, demonstrating its potential as a scalable, interpretable evaluation pipeline for future multimodal AI research.
Paper Structure (27 sections, 3 figures, 8 tables)

This paper contains 27 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning LLMs
  4. LLM-as-the-Judge
  5. Multimodal Benchmark
  6. Datasets
  7. Text-to-Text Datasets
  8. Text-to-Text Tasks
  9. Audio-Language Datasets
  10. Audio-Language Tasks
  11. Video Datasets
  12. Video Tasks
  13. Iamge Datasets
  14. Iamge Tasks
  15. Methodology
  16. ...and 12 more sections

Figures (3)

  • Figure 1: System architecture of the MLLMs evaluation framework. The framework takes multimodal inputs and task instructions to query multiple tested MLLMs. Each model produces an answer along with its justification. A judge model, following evaluation instructions, assesses the responses and outputs scores (0–5) and error analyses.
  • Figure 2: Human vs. Judge alignment across MLLMs.
  • Figure 3: The input chart example for image case study.