Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design

TL;DR

The paper addresses the challenge of learning long-horizon reasoning in LLM-based agents by introducing turn-level rewards for multi-turn reinforcement learning. It develops two multi-turn extensions, MT-GRPO and MT-PPO, enabling fine-grained credit assignment across turns, and demonstrates their effectiveness through a reasoning-augmented search case study with verifiable rewards and LLM-based evaluation. Empirical results show that turn-level rewards yield greater training stability, faster convergence, and higher accuracy, including near-perfect format correctness across tasks. The work highlights the broad potential of turn-level feedback to improve autonomous tool-using LLM agents beyond search applications.

Abstract

This paper investigates Reinforcement Learning (RL) approaches to enhance the reasoning capabilities of Large Language Model (LLM) agents in long-horizon, multi-turn scenarios. Although RL algorithms such as Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO) have been widely applied to train multi-turn LLM agents, they typically rely only on sparse outcome rewards and lack dense intermediate signals across multiple decision steps, limiting their performance on complex reasoning tasks. To bridge this gap, we present the first systematic study of \textit{turn-level reward design} for multi-turn RL algorithms and agent applications. By integrating turn-level rewards, we extend GRPO and PPO to their respective multi-turn variants, enabling fine-grained credit assignment. We conduct case studies on multi-turn reasoning-augmented search agents, where we carefully design two types of turn-level rewards: verifiable and LLM-as-judge. Our experiments on multi-turn search tasks demonstrate that incorporating well-designed turn-level rewards enables RL algorithms to significantly outperform baseline methods with trajectory-level rewards. Both training and validation reward curves illustrate that our method achieves \textit{greater stability}, \textit{faster convergence}, and \textit{higher accuracy}. Numerical results across diverse question-answering datasets further show that our approach consistently delivers highest answer correctness and 100\% format correctness.

Figures (12)

• Figure 1: Curves for different training reward components during training with various algorithms. GRPO-OR means GRPO with outcome rewards while GPRO-MR means GRPO with merged outcome and intermediate rewards. Each plot shows the training reward score over training steps. Dotted lines represent the average reward across 10 runs, while solid lines show trends smoothed using the Exponential Moving Average (EMA).
• Figure 2: Overview of the multi-turn reasoning-augmented search agent pipeline. Given a system prompt and a question, each iteration of the LLM-based search agent proceeds as follows: (1) The agent begins with reasoning, analyzing the current context to identify missing information. (2) It then formulates a search query to retrieve relevant information from an external database, which is integrated into the evolving context. (3) This cycle continues until the agent judges that the context is sufficient, at which point it performs a final round of reasoning to generate the answer.
• Figure 3: Training reward curves recorded during training for PPO baselines and MT-PPO on the NQ and HotpotQA datasets. The rewards include answer correctness, format correctness, and retrieval correctness. Solid lines show mean reward values, while shaded regions indicate variability across five independent runs.
• Figure 4: Ablation studies on (1) the search count reward $\lambda_s$ and (2) the maximum number of turns $N_{\max}$ on the NQ dataset. The left panel reports answer correctness, the middle panel shows the average number of turns, and the right panel illustrates accuracy under different $N_{\max}$ settings.
• Figure 5: Validate reward curves recorded during training for PPO baselines and MT-PPO on the NQ and HotpotQA datasets. The rewards include answer correctness, format correctness, and retrieval correctness. Solid lines show mean reward values, while shaded regions indicate variability across five independent runs.