SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution

TL;DR

Stepwise Progress Attribution (SPA), a general reward redistribution framework that decomposes the final reward into stepwise contributions, each reflecting its incremental progress toward overall task completion, is proposed.

Abstract

Reinforcement learning (RL) holds significant promise for training LLM agents to handle complex, goal-oriented tasks that require multi-step interactions with external environments. However, a critical challenge when applying RL to these agentic tasks arises from delayed rewards: feedback signals are typically available only after the entire task is completed. This makes it non-trivial to assign delayed rewards to earlier actions, providing insufficient guidance regarding environmental constraints and hindering agent training. In this work, we draw on the insight that the ultimate completion of a task emerges from the cumulative progress an agent makes across individual steps. We propose Stepwise Progress Attribution (SPA), a general reward redistribution framework that decomposes the final reward into stepwise contributions, each reflecting its incremental progress toward overall task completion. To achieve this, we train a progress estimator that accumulates stepwise contributions over a trajectory to match the task completion. During policy optimization, we combine the estimated per-step contribution with a grounding signal for actions executed in the environment as the fine-grained, intermediate reward for effective agent training. Extensive experiments on common agent benchmarks (including Webshop, ALFWorld, and VirtualHome) demonstrate that SPA consistently outperforms the state-of-the-art method in both success rate (+2.5\% on average) and grounding accuracy (+1.9\% on average). Further analyses demonstrate that our method remarkably provides more effective intermediate rewards for RL training. Our code is available at https://github.com/WangHanLinHenry/SPA-RL-Agent.

Figures (6)

• Figure 1: Overview of our proposed Stepwise Progress Attribution (SPA) framework for training LLM agents with RL.
• Figure 2: Performance comparison of various methods across different evaluation metrics and environments. "WS" and "VH" denote the Webshop and VirtualHome environments, respectively. In the WebShop environment, where the action set is predefined for each step and provided by the environment, the grounding accuracy consistently reaches 100%, owing to the powerful instruction-following capabilities of LLMs, and thus we do not report it here.
• Figure 3: Performance with different intermediate rewards in the ALFWorld environment.
• Figure 4: Relative performance improvement between our SPA and PPO at different step intervals.
• Figure 5: Inference prompt template for ALFWorld and VirtualHome environment.