Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge

Junjie Wu, Xuan Kan, Zihao He, Shunwen Tan, Bo Pan, Kaitai Zhang

Abstract

Multimodal Large Language Models (MLLMs) have been widely adopted as MLLM-as-a-Judges due to their strong alignment with human judgment across various visual tasks. However, most existing judge models are optimized for single-task scenarios and struggle to generalize to diverse contexts, which is a critical requirement for reliable evaluation. To address this limitation, we propose Multi-Task Reinforcement Learning for MLLM-as-a-Judge (MT-RL-Judge), a framework that jointly optimizes the judge model across multiple tasks, leveraging the generalization capabilities of RL. Experimental results against several strong baselines demonstrate that MT-RL-Judge outperforms strong baselines in both judgment consistency and correlation with human preferences. Furthermore, our approach exhibits robust generalization on out-of-distribution tasks, further validating its effectiveness.

Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge

Abstract

Multimodal Large Language Models (MLLMs) have been widely adopted as MLLM-as-a-Judges due to their strong alignment with human judgment across various visual tasks. However, most existing judge models are optimized for single-task scenarios and struggle to generalize to diverse contexts, which is a critical requirement for reliable evaluation. To address this limitation, we propose Multi-Task Reinforcement Learning for MLLM-as-a-Judge (MT-RL-Judge), a framework that jointly optimizes the judge model across multiple tasks, leveraging the generalization capabilities of RL. Experimental results against several strong baselines demonstrate that MT-RL-Judge outperforms strong baselines in both judgment consistency and correlation with human preferences. Furthermore, our approach exhibits robust generalization on out-of-distribution tasks, further validating its effectiveness.
Paper Structure (23 sections, 5 equations, 12 figures, 5 tables)

This paper contains 23 sections, 5 equations, 12 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Our Method
  4. Problem Formulation
  5. MLLM-as-a-Judge.
  6. Unified MLLM-as-a-Judge.
  7. RL-based MLLM-as-a-Judge with Reasoning.
  8. MT-RL-Judge
  9. Reward Function.
  10. Training Objective.
  11. Experiments
  12. Datasets
  13. Setting
  14. Main Results
  15. RL Enhances MLLM-as-a-Judge.
  16. ...and 8 more sections

Figures (12)

  • Figure 1: Prompt for SFT on Unsafe Bench.
  • Figure 2: Prompt for SFT on AGIN-Tech.
  • Figure 3: Prompt for SFT on AGIN-Rat.
  • Figure 4: Prompt for SFT on AGIN-Nat.
  • Figure 5: Prompt for SFT on SeeTrue.
  • ...and 7 more figures