Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

Zhengxian Wu, Kai Shi, Chuanrui Zhang, Zirui Liao, Jun Yang, Ni Yang, Qiuying Peng, Luyuan Zhang, Hangrui Xu, Tianhuang Su, Zhenyu Yang, Haonan Lu, Haoqian Wang

Abstract

Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.

When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning

Abstract

Recent progress in multimodal large language models has led to strong performance on reasoning tasks, but these improvements largely rely on high-quality annotated data or teacher-model distillation, both of which are costly and difficult to scale. To address this, we propose an unsupervised self-evolution training framework for multimodal reasoning that achieves stable performance improvements without using human-annotated answers or external reward models. For each input, we sample multiple reasoning trajectories and jointly model their within group structure. We use the Actor's self-consistency signal as a training prior, and introduce a bounded Judge based modulation to continuously reweight trajectories of different quality. We further model the modulated scores as a group level distribution and convert absolute scores into relative advantages within each group, enabling more robust policy updates. Trained with Group Relative Policy Optimization (GRPO) on unlabeled data, our method consistently improves reasoning performance and generalization on five mathematical reasoning benchmarks, offering a scalable path toward self-evolving multimodal models. The code are available at https://github.com/OPPO-Mente-Lab/LLM-Self-Judge.
Paper Structure (33 sections, 29 equations, 7 figures, 14 tables, 1 algorithm)

This paper contains 33 sections, 29 equations, 7 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Multi-modal Reasoning
  4. Self-Evolving In Large Language Models
  5. Method
  6. Consistency-Based Initial Reward for the Actor
  7. Calibrating Consistency Rewards with a Judge
  8. Distributional Modeling of the Final Reward
  9. Experiments
  10. Datasets and Training Details
  11. Training Data.
  12. Evaluation Benchmarks.
  13. Training Setup.
  14. Experimental Results
  15. Main Results.
  16. ...and 18 more sections

Figures (7)

  • Figure 1: Limitation of majority voting in unsupervised self-evolution.Right: An example where the most frequent answer is incorrect. Majority voting reinforces this dominant error, while our method favors higher-quality reasoning paths through Judge modulation.Left: Results on MathVisionmathvision and DynaMathDynamath show that our approach consistently outperforms majority-voting-based self-training.
  • Figure 2: Overview of the proposed unsupervised self-evolution framework.The Actor generates multiple reasoning trajectories for the same input, while a frozen Judge provides bounded score modulation. The final rewards are optimized in a group-wise, distributional manner to enable stable policy updates without external supervision.
  • Figure 3: Training dynamics on MMR1mmr1.The figure compares majority voting, supervised reinforcement learning, and our method during training, in terms of validation accuracy on MathVision, actor entropy, and average response length.
  • Figure 4: Ablation training dynamics on MMR1mmr1.We compare Self-Consistency, Judge-only, and the full method in terms of validation accuracy on MathVision, actor entropy, and average response length during training.
  • Figure 5: A case study on Geo3Kgeo3k
  • ...and 2 more figures