Boosting MLLM Reasoning with Text-Debiased Hint-GRPO

TL;DR

Confronts two bottlenecks in group-relative policy optimization for multimodal LLM reasoning: low data utilization and text-bias. Proposes Hint-GRPO to adaptively inject reasoning hints and text-bias calibration to emphasize image information at test time. Empirically, the method yields substantial improvements across three base MLLMs and eleven datasets, outperforming PRM and existing GRPO approaches, with notable gains in geometry reasoning. The work provides open-source code and contributes dataset preparation strategies for more robust MLLM reasoning evaluation.

Abstract

MLLM reasoning has drawn widespread research for its excellent problem-solving capability. Current reasoning methods fall into two types: PRM, which supervises the intermediate reasoning steps, and ORM, which supervises the final results. Recently, DeepSeek-R1 has challenged the traditional view that PRM outperforms ORM, which demonstrates strong generalization performance using an ORM method (i.e., GRPO). However, current MLLM's GRPO algorithms still struggle to handle challenging and complex multimodal reasoning tasks (e.g., mathematical reasoning). In this work, we reveal two problems that impede the performance of GRPO on the MLLM: Low data utilization and Text-bias. Low data utilization refers to that GRPO cannot acquire positive rewards to update the MLLM on difficult samples, and text-bias is a phenomenon that the MLLM bypasses image condition and solely relies on text condition for generation after GRPO training. To tackle these problems, this work proposes Hint-GRPO that improves data utilization by adaptively providing hints for samples of varying difficulty, and text-bias calibration that mitigates text-bias by calibrating the token prediction logits with image condition in test-time. Experiment results on three base MLLMs across eleven datasets demonstrate that our proposed methods advance the reasoning capability of original MLLM by a large margin, exhibiting superior performance to existing MLLM reasoning methods. Our code is available at https://github.com/hqhQAQ/Hint-GRPO.

Figures (8)

• Figure 1: Low data utilization of GRPO: If all answers are incorrect, the zero loss gradients ($\nabla_\theta\mathcal{L}=\mathbf{0}$) will invalidate the sample.
• Figure 2: Text-bias of GRPO, where the MLLM ignores real image and uses its imagined image from text to generate outputs.
• Figure 3: Data utilization rate & reward of GRPO & Hint-GRPO.
• Figure 4: Framework of Hint-GRPO and text-bias calibration. Specifically, Hint-GRPO adaptively provides hints to the samples and selects the most eligible group for training, mitigating the low data utilization problem. Text-bias calibration increases the intensity of image condition in test-time, alleviating the text-bias problem. Note that "$<$BoS$>$" denotes the beginning-of-sequence token in MLLM.
• Figure 5: (a) Qwen2-VL-7B's test accuracy w/ & w/o image in GRPO training. (b) The ratio of two types of invalid samples.