Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

TL;DR

This work addresses the challenge in RLVR where base-model capability limits both the discovery of correct reasoning trajectories and the diversity of exploration. It introduces MENTOR, a framework that injects expert guidance only at high-uncertainty decision points via a mixed-policy rollout and a modified on-policy GRPO, leveraging speculative sampling for efficiency. Empirical results across multiple backbones and math benchmarks show consistent improvements and reduced entropy collapse, with analyses demonstrating selective absorption of expert knowledge and enhanced reasoning diversity. The approach offers a practical path to scaling RLVR to diverse models and tasks while preserving autonomous exploration and robust generalization.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.

Figures (4)

• Figure 1: Illustration of MENTOR framework. By providing expert guidance only at critical decision points, MENTOR steers reasoning trajectories while preserving the policy’s own exploration, thereby avoiding the constraints of fixed expert trajectories and achieving more effective and diverse exploration in RL training.
• Figure 2: Training dynamics of MENTOR compared with On-policy RL. MENTOR mitigates entropy collapse, and its response length dynamics reflect a shift from learning to understanding, thereby achieving higher performance.
• Figure 3: The occurrence rate of high-frequency reasoning tokens under different training methods. MENTOR absorbs the essence of expert trajectories such as verify, while avoiding over-imitation of redundant tokens like okay or wait.
• Figure 4: Pass@32 performance of Qwen2.5-7B under different methods. MENTOR improves the model’s reasoning diversity beyond other baselines.