From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, Weipeng Chen
TL;DR
This work tackles the challenge of sparse reward signals in training LLM-based agents by proposing StepAgent, a step-wise reinforcement learning framework that mimics novice-to-expert learning through inspection and reflection. The inspection stage enables the agent to identify capability gaps by comparing stepwise expert actions, while the reflection stage employs two strategies—implicit-reward reinforcement learning and inverse reinforcement learning—to align the agent’s policy with the expert’s distribution using step-wise feedback. Theoretical analysis shows convergence of the agent’s action distribution toward the expert’s distribution across training cycles, and extensive experiments across web, embodied, and multi-hop QA tasks demonstrate that StepAgent consistently outperforms strong baselines, with notable gains on complex multi-hop reasoning. The approach reduces reliance on manually annotated step-wise rewards and offers a scalable path to improving LLM agent performance in interactive environments. The results suggest step-wise supervision can substantially enhance learning efficiency and decision quality in autonomous AI agents.
Abstract
The outstanding capabilities of large language models (LLMs) render them a crucial component in various autonomous agent systems. While traditional methods depend on the inherent knowledge of LLMs without fine-tuning, more recent approaches have shifted toward the reinforcement learning strategy to further enhance agents' ability to solve complex interactive tasks with environments and tools. However, previous approaches are constrained by the sparse reward issue, where existing datasets solely provide a final scalar reward for each multi-step reasoning chain, potentially leading to ineffectiveness and inefficiency in policy learning. In this paper, we introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. Inheriting the spirit of novice-to-expert theory, we first compare the actions of the expert and the agent to automatically generate intermediate rewards for fine-grained optimization. Additionally, we propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment. Further theoretical analysis demonstrates that the action distribution of the agent can converge toward the expert action distribution over multiple training cycles. Experimental results across various datasets indicate that StepAgent outperforms existing baseline methods.