RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents

TL;DR

This work tackles the sparse-reward problem in multi-turn tool calling by identifying a gradient-collapse risk in standard GRPO when starting from highly peaked SFT policies. It introduces RC-GRPO, a two-stage approach: Stage 1 fine-tunes a Reward-Conditioned Trajectory Policy (RCTP) that maps discrete reward tokens to distinct trajectory qualities, and Stage 2 performs reward-conditioned GRPO to inject controlled diversity within each rollout group via tokens like $<|high\_reward|>$ and $<|low\_reward|>$. The method uses a trajectory-level reward $R(\tau)$ composed of $R_{state}$ and $R_{action}$, with within-group advantages computed as $A_j = (R(\tau_j)-\mu_g)/(\sigma_g + \epsilon_{stab})$, and optimizes with a PPO-style clipped objective plus KL regularization. Empirically, RC-GRPO improves BFCLv4 performance for open-weight models (notably achieving 85.0% on Qwen2.5-7B-Instruct) and maintains informative updates through variance guarantees, while analyses reveal gains arise from better between-mode separation rather than mere entropy increases. The work suggests a principled variance-based view of how reward-conditioned rollout generation stabilizes learning for complex, partially observable tool-calling tasks with practical implications for building robust, controllable agent policies.

Abstract

Multi-turn tool calling is challenging for Large Language Models (LLMs) because rewards are sparse and exploration is expensive. A common recipe, SFT followed by GRPO, can stall when within-group reward variation is low (e.g., more rollouts in a group receive the all 0 or all 1 reward), making the group-normalized advantage uninformative and yielding vanishing updates. To address this problem, we propose RC-GRPO (Reward-Conditioned Group Relative Policy Optimization), which treats exploration as a controllable steering problem via discrete reward tokens. We first fine-tune a Reward-Conditioned Trajectory Policy (RCTP) on mixed-quality trajectories with reward goal special tokens (e.g., <|high_reward|>, <|low_reward|>) injected into the prompts, enabling the model to learn how to generate distinct quality trajectories on demand. Then during RL, we sample diverse reward tokens within each GRPO group and condition rollouts on the sampled token to improve within-group diversity, improving advantage gains. On the Berkeley Function Calling Leaderboard v4 (BFCLv4) multi-turn benchmark, our method yields consistently improved performance than baselines, and the performance on Qwen-2.5-7B-Instruct even surpasses all closed-source API models.

Figures (2)

• Figure 1: Overview of RC-GRPO. (Top) Standard GRPO optimization from an SFT initialization can sharply reduce rollout diversity, yielding low within-group reward variance and a weak/vanishing advantage signal. (Bottom) Our RC-GRPO conditions rollouts on discrete reward tokens and samples diverse tokens within each group, explicitly injecting variance and producing informative advantages.
• Figure 2: Training dynamics for Qwen2.5-7B on BFCLv4. We plot a proxy for within-group diversity (the gap between the maximum and minimum advantage within each sampled group) together with the training reward over time.

Theorems & Definitions (14)

• Definition 4.1: GRPO Advantage Collapse
• Proposition 4.2: Vanishing Gradient in Peaked Policies
• proof : Proof Sketch
• Proposition 4.3: Variance Guarantee via Reward Conditioning
• proof : Proof Sketch
• Remark 4.4: Convergence Implication
• Definition 1.1: Probability Space
• Lemma 1.2: KL-Probability Bound
• proof : Proof of Lemma \ref{['lem:kl_bound']}
• proof : Proof of Proposition \ref{['prop:collapse']}