Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models

Shidong Yang, Tongwen Huang, Hao Wen, Yong Wang, Li Chen, Xiangxiang Chu

TL;DR

The paper addresses data noise and training inefficiency in multimodal reward models by introducing Entropy-Guided Training (EGT). It establishes that response entropy correlates with accuracy, using this as an unsupervised proxy for sample difficulty and annotation noise, and couples entropy-driven data curation with a low-to-high entropy curriculum in reinforcement learning. The approach includes generating high-quality reasoning trajectories, pruning high-entropy samples, and training with an entropy-aware reward, achieving state-of-the-art results across three benchmarks with significantly reduced data requirements. Overall, EGT offers a practical, scalable method to strengthen multimodal reasoning reward models and improve alignment with human preferences.

Abstract

Multimodal reward models are crucial for aligning multimodal large language models with human preferences. Recent works have incorporated reasoning capabilities into these models, achieving promising results. However, training these models suffers from two critical challenges: (1) the inherent noise in preference datasets, which degrades model performance, and (2) the inefficiency of conventional training methods, which ignore the differences in sample difficulty. In this paper, we identify a strong correlation between response entropy and accuracy, indicating that entropy can serve as a reliable and unsupervised proxy for annotation noise and sample difficulty. Based on this insight, we propose a novel Entropy-Guided Training (EGT) approach for multimodal reasoning reward models, which combines two strategies: (1) entropy-guided data curation to mitigate the impact of unreliable samples, and (2) an entropy-guided training strategy that progressively introduces more complex examples. Extensive experiments across three benchmarks show that the EGT-trained model consistently outperforms state-of-the-art multimodal reward models.

Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models

TL;DR

The paper addresses data noise and training inefficiency in multimodal reward models by introducing Entropy-Guided Training (EGT). It establishes that response entropy correlates with accuracy, using this as an unsupervised proxy for sample difficulty and annotation noise, and couples entropy-driven data curation with a low-to-high entropy curriculum in reinforcement learning. The approach includes generating high-quality reasoning trajectories, pruning high-entropy samples, and training with an entropy-aware reward, achieving state-of-the-art results across three benchmarks with significantly reduced data requirements. Overall, EGT offers a practical, scalable method to strengthen multimodal reasoning reward models and improve alignment with human preferences.

Abstract

Multimodal reward models are crucial for aligning multimodal large language models with human preferences. Recent works have incorporated reasoning capabilities into these models, achieving promising results. However, training these models suffers from two critical challenges: (1) the inherent noise in preference datasets, which degrades model performance, and (2) the inefficiency of conventional training methods, which ignore the differences in sample difficulty. In this paper, we identify a strong correlation between response entropy and accuracy, indicating that entropy can serve as a reliable and unsupervised proxy for annotation noise and sample difficulty. Based on this insight, we propose a novel Entropy-Guided Training (EGT) approach for multimodal reasoning reward models, which combines two strategies: (1) entropy-guided data curation to mitigate the impact of unreliable samples, and (2) an entropy-guided training strategy that progressively introduces more complex examples. Extensive experiments across three benchmarks show that the EGT-trained model consistently outperforms state-of-the-art multimodal reward models.
Paper Structure (11 sections, 4 equations, 4 figures, 5 tables)

This paper contains 11 sections, 4 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Task Definition
  4. Enhancing RM with Reasoning Capabilities
  5. Entropy-Guided Data Curation
  6. Data-Efficient Training
  7. Experiments
  8. Experiments Setup
  9. Main Results
  10. Analysis
  11. Conclusion

Figures (4)

  • Figure 1: Response entropy can serve as a proxy for challenging and noisy samples. Left: An ambiguous sample results in a high-entropy output distribution. Right: A sample with a clear factual error allows for a confident, low-entropy decision.
  • Figure 2: Correlation between response entropy and accuracy on a large-scale (80,000 samples) preference dataset. Samples are binned by their response entropy. The accuracy rate per bin reveals a clear downward trend: higher entropy correlates with lower accuracy.
  • Figure 3: Overview of our proposed entropy-guided data-efficient training method. The process consists of three stages: (1) Reasoning Enhancement, where an instruction model is fine-tuned on high-quality reasoning trajectories; (2) Entropy-Guided Curation, where the reasoning-enhanced model prunes a preference dataset by identifying high-entropy samples. In this stage, the entropy probed by the reasoning reward model serves as a proxy for sample difficulty and noise; and (3) Data-Efficient Training, where the final model is trained on the curated dataset via reinforcement learning, following an easy-to-hard progression where samples are introduced in order of increasing entropy.
  • Figure 4: Performance comparison with different data scales.