Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning

TL;DR

CalibRL is proposed, a hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms that help alleviate the distributional mismatch between the model's policy and expert trajectories, thereby achieving a more stable balance between exploration and exploitation.

Abstract

Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the enormous state space of MLLM and sparse rewards often leads to entropy collapse, policy degradation, or over-exploitation of suboptimal behaviors. This necessitates an exploration strategy that maintains productive stochasticity while avoiding the drawbacks of uncontrolled random sampling, yielding inefficient exploration. In this paper, we propose CalibRL, a hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms. First, a distribution-aware advantage weighting scales updates by group rareness to calibrate the distribution, therefore preserving exploration. Meanwhile, the asymmetric activation function (LeakyReLU) leverages the expert knowledge as a calibration baseline to moderate overconfident updates while preserving their corrective direction. CalibRL increases policy entropy in a guided manner and clarifies the target distribution by estimating the on-policy distribution through online sampling. Updates are driven by these informative behaviors, avoiding convergence to erroneous patterns. Importantly, these designs help alleviate the distributional mismatch between the model's policy and expert trajectories, thereby achieving a more stable balance between exploration and exploitation. Extensive experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements, validating the effectiveness of our controllable hybrid-policy RLVR training. Code is available at https://github.com/zhh6425/CalibRL.

Figures (6)

• Figure 1: Entropy, reward, and accuracy curves of different methods. We split the entropy comparison into two panels for clarity.
• Figure 2: Performance comparison showing relative changes from GRPO baseline across in-domain geometry and out-of-domain reasoning tasks.
• Figure 3: Entropy evolution during training for different $\alpha$ values in our framework. We split the comparison into two panels for clarity. The curves demonstrate how $\alpha$ controls exploration strength.
• Figure 4: Relationship between $|\hat{A}_i|$ and reward frequency in a group of 10 samples.
• Figure 5: A case of CalibRL compared with baselines GRPO and SFT+GRPO.