ExpSeek: Self-Triggered Experience Seeking for Web Agents

TL;DR

ExpSeek introduces a self-triggered, step-level experience-seeking framework for web agents that leverages step entropy to decide when to retrieve guided experience. It builds an experience base of triplets (Behavior, Mistake, Guidance) and uses an experience model to generate contextually relevant guidance at inference time, controlled by bootstrap-derived thresholds for process and answer steps. Across four web-agent benchmarks with 8B and 32B backbones, ExpSeek achieves substantial absolute gains over passive baselines and demonstrates cross-task transferability, indicating that entropy can serve as a practical self-trigger signal for proactive guidance. The work provides a principled approach to balancing exploration and exploitation in open-world reasoning, with implications for scalable, adaptive agent systems.

Abstract

Experience intervention in web agents emerges as a promising technical paradigm, enhancing agent interaction capabilities by providing valuable insights from accumulated experiences. However, existing methods predominantly inject experience passively as global context before task execution, struggling to adapt to dynamically changing contextual observations during agent-environment interaction. We propose ExpSeek, which shifts experience toward step-level proactive seeking: (1) estimating step-level entropy thresholds to determine intervention timing using the model's intrinsic signals; (2) designing step-level tailor-designed experience content. Experiments on Qwen3-8B and 32B models across four challenging web agent benchmarks demonstrate that ExpSeek achieves absolute improvements of 9.3% and 7.5%, respectively. Our experiments validate the feasibility and advantages of entropy as a self-triggering signal, reveal that even a 4B small-scale experience model can significantly boost the performance of larger agent models.

Figures (9)

• Figure 1: Comparison of experience intervention frameworks. Panel A shows the traditional global passive injection of experience, while we extend the framework to Panel B, where the agent proactively seeks guidance at each step based on its own signals.
• Figure 2: The overall architecture of ExpSeek, including experience base construction and actively seeking experience guidance during inference. The step entropy threshold calculation process is not depicted here.
• Figure 3: Entropy distributions of process and answer steps on $\mathcal{D}_{train}$ for Qwen3-8B, with fitted logistic regression curves. Green zone indicates no intervention during inference, red indicates intervention, and yellow indicates probabilistic intervention.
• Figure 4: Entropy distributions of process and answer steps for Qwen3-8B before and after applying ExpSeek across all benchmarks. Results for Qwen3-32B are provided in Figure \ref{['fig:qwen3_32b_six']}.
• Figure 5: Scaling Law of experience model $\mathcal{M}_e$.