Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning

Yu Xinmiao, Zhang Liwen, Feng Xiaocheng, Jiang Yong, Qin Bing, Xie Pengjun, Zhou Jingren

TL;DR

This work identifies the plan anchor phenomenon in long-horizon web reasoning, where the initial planning step strongly determines downstream success. It introduces Anchor-GRPO, a two-stage RL framework that decouples planning and execution: Stage 1 uses Plan Rubrics Learner-derived dense rewards to optimize the first-step plan, while Stage 2 trains the executor with sparse rewards to align behavior with the initial plan. The approach yields state-of-the-art results across four benchmarks and scales effectively from 3B to 30B parameter models, with clear evidence that dense, rubric-based planning signals improve long-horizon reliability and tool efficiency. These findings suggest principled planning, guided by explicit reasoning standards, is key to building stable, capable web agents at scale.

Abstract

Large Language Model(LLM)-based agents have shown strong capabilities in web information seeking, with reinforcement learning (RL) becoming a key optimization paradigm. However, planning remains a bottleneck, as existing methods struggle with long-horizon strategies. Our analysis reveals a critical phenomenon, plan anchor, where the first reasoning step disproportionately impacts downstream behavior in long-horizon web reasoning tasks. Current RL algorithms, fail to account for this by uniformly distributing rewards across the trajectory. To address this, we propose Anchor-GRPO, a two-stage RL framework that decouples planning and execution. In Stage 1, the agent optimizes its first-step planning using fine-grained rubrics derived from self-play experiences and human calibration. In Stage 2, execution is aligned with the initial plan through sparse rewards, ensuring stable and efficient tool usage. We evaluate Anchor-GRPO on four benchmarks: BrowseComp, BrowseComp-Zh, GAIA, and XBench-DeepSearch. Across models from 3B to 30B, Anchor-GRPO outperforms baseline GRPO and First-step GRPO, improving task success and tool efficiency. Notably, WebAnchor-30B achieves 46.0% pass@1 on BrowseComp and 76.4% on GAIA. Anchor-GRPO also demonstrates strong scalability, getting higher accuracy as model size and context length increase.

WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning

TL;DR

This work identifies the plan anchor phenomenon in long-horizon web reasoning, where the initial planning step strongly determines downstream success. It introduces Anchor-GRPO, a two-stage RL framework that decouples planning and execution: Stage 1 uses Plan Rubrics Learner-derived dense rewards to optimize the first-step plan, while Stage 2 trains the executor with sparse rewards to align behavior with the initial plan. The approach yields state-of-the-art results across four benchmarks and scales effectively from 3B to 30B parameter models, with clear evidence that dense, rubric-based planning signals improve long-horizon reliability and tool efficiency. These findings suggest principled planning, guided by explicit reasoning standards, is key to building stable, capable web agents at scale.

Abstract

Large Language Model(LLM)-based agents have shown strong capabilities in web information seeking, with reinforcement learning (RL) becoming a key optimization paradigm. However, planning remains a bottleneck, as existing methods struggle with long-horizon strategies. Our analysis reveals a critical phenomenon, plan anchor, where the first reasoning step disproportionately impacts downstream behavior in long-horizon web reasoning tasks. Current RL algorithms, fail to account for this by uniformly distributing rewards across the trajectory. To address this, we propose Anchor-GRPO, a two-stage RL framework that decouples planning and execution. In Stage 1, the agent optimizes its first-step planning using fine-grained rubrics derived from self-play experiences and human calibration. In Stage 2, execution is aligned with the initial plan through sparse rewards, ensuring stable and efficient tool usage. We evaluate Anchor-GRPO on four benchmarks: BrowseComp, BrowseComp-Zh, GAIA, and XBench-DeepSearch. Across models from 3B to 30B, Anchor-GRPO outperforms baseline GRPO and First-step GRPO, improving task success and tool efficiency. Notably, WebAnchor-30B achieves 46.0% pass@1 on BrowseComp and 76.4% on GAIA. Anchor-GRPO also demonstrates strong scalability, getting higher accuracy as model size and context length increase.
Paper Structure (55 sections, 8 equations, 5 figures, 2 tables)

This paper contains 55 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Agentic Web Reasoning.
  4. Tool Design.
  5. Methodology
  6. Anchor-GRPO Overview
  7. Stage 1: Anchor Plan Optimization
  8. Plan Rubrics Learner
  9. Insight Extraction
  10. Rubrics Optimization
  11. Stage 1: Anchor Plan Optimization with Plan Rubrics
  12. Plan Rubrics Reward Definition
  13. Objective Function for Plan Optimization
  14. LLM Masking for First-Step Update
  15. Stage 2: Trajectory Level Executor Optimization
  16. ...and 40 more sections

Figures (5)

  • Figure 1: Illustrating the plan anchor phenomenon, the critical role of the first step in task accuracy, and the use of plan rubrics to guide optimization.
  • Figure 2: Anchor-GRPO framework. Stage 1 optimizes the initial plan using the Rubrics Model, providing dense rewards. Stage 2 refines the trajectory with sparse rewards, ensuring alignment with the plan.
  • Figure 3: Overview of the Plan Rubrics Learner and verification process, illustrating how WebAnchor collects experiences, extracts insights, iteratively optimizes rubrics, and verifies plan quality with human feedback.
  • Figure 4: Performance and behavior analysis of Anchor-GRPO versus baselines, including training convergence, tool efficiency, and reward robustness across model scales.
  • Figure 5: Scaling of Anchor-GRPO