Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation

Zeyu Chen, Huanjin Yao, Ziwang Zhao, Min Yang

TL;DR

This work introduces M-JudgeBench, a ten-dimensional capability-oriented benchmark designed to comprehensively assess the judgment abilities of MLLMs, and proposes Judge-MCTS, a data construction framework generating pairwise reasoning trajectories with various correctness and length.

Abstract

Using Multimodal Large Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains. Evaluating the capability and reliability of MLLM-as-a-judge systems is therefore essential for ensuring trustworthy assessment. Existing judge benchmarks categorize samples by task types but fail to capture the fundamental judgment capabilities required for reliable evaluation. In this work, we introduce M-JudgeBench, a ten-dimensional capability-oriented benchmark designed to comprehensively assess the judgment abilities of MLLMs. Our benchmark decomposes evaluation into pairwise Chain-of-Thought (CoT) comparison, length bias avoidance, and process error detection tasks, jointly covering ten fine-grained subtasks. This design enables diagnosis of model reliability across reasoning styles, response lengths, and cross-model variations. Systematic evaluation uncovers the systematic weaknesses in existing MLLM-as-a-judge systems. To address this issue, we further propose Judge-MCTS, a data construction framework generating pairwise reasoning trajectories with various correctness and length. Using Judge-MCTS, we construct an MCTS-augmented dataset and train M-Judger, a series of strong judge models. Extensive experiments demonstrate the superiority of M-Judger on existing judge benchmarks as well as M-JudgeBench. Overall, our work establishes a more principled foundation for evaluating MLLM-as-a-judge through M-JudgeBench and Judge-MCTS framework, paving the way for future research on judge model evaluation and capability-driven judge training.

Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation

TL;DR

This work introduces M-JudgeBench, a ten-dimensional capability-oriented benchmark designed to comprehensively assess the judgment abilities of MLLMs, and proposes Judge-MCTS, a data construction framework generating pairwise reasoning trajectories with various correctness and length.

Abstract

Using Multimodal Large Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains. Evaluating the capability and reliability of MLLM-as-a-judge systems is therefore essential for ensuring trustworthy assessment. Existing judge benchmarks categorize samples by task types but fail to capture the fundamental judgment capabilities required for reliable evaluation. In this work, we introduce M-JudgeBench, a ten-dimensional capability-oriented benchmark designed to comprehensively assess the judgment abilities of MLLMs. Our benchmark decomposes evaluation into pairwise Chain-of-Thought (CoT) comparison, length bias avoidance, and process error detection tasks, jointly covering ten fine-grained subtasks. This design enables diagnosis of model reliability across reasoning styles, response lengths, and cross-model variations. Systematic evaluation uncovers the systematic weaknesses in existing MLLM-as-a-judge systems. To address this issue, we further propose Judge-MCTS, a data construction framework generating pairwise reasoning trajectories with various correctness and length. Using Judge-MCTS, we construct an MCTS-augmented dataset and train M-Judger, a series of strong judge models. Extensive experiments demonstrate the superiority of M-Judger on existing judge benchmarks as well as M-JudgeBench. Overall, our work establishes a more principled foundation for evaluating MLLM-as-a-judge through M-JudgeBench and Judge-MCTS framework, paving the way for future research on judge model evaluation and capability-driven judge training.
Paper Structure (42 sections, 1 equation, 3 figures, 6 tables)

This paper contains 42 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. M-JudgeBench Construction
  3. Capability-Oriented Evaluation Framework
  4. Data Generation Steps
  5. Pairwise CoT Comparison
  6. Length Bias Avoidance
  7. Process Error Detection
  8. Statistics of M-JudgeBench
  9. M-Judger Training
  10. Open-Source Training Data Collection
  11. Judge-MCTS Data Construction Framework
  12. Reasoning Rollout via MCTS
  13. Pairwise Data Construction
  14. M-Judger Training Methodology
  15. Stage1. Supervised Fine-Tuning (SFT).
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Comparison between existing judge benchmarks and M-JudgeBench. M-JudgeBench is designed with an emphasis on evaluating judgment capabilities.
  • Figure 2: Overview of M-JudgeBench. The figure illustrates the data construction methods and resulting task types in M-JudgeBench. Result-error pairs are derived from rollouts of different models with varied temperatures and reasoning lengths, while process-error data are produced by controlled noise injection preserving correct answers. These yield three main task types and ten subtasks in total.
  • Figure 3: Data statistics of M-JudgeBench.