Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Planning Framework for LLM-Based Web Agents

Orit Shahnovsky, Rotem Dror

Abstract

Developing autonomous agents for web-based tasks is a core challenge in AI. While Large Language Model (LLM) agents can interpret complex user requests, they often operate as black boxes, making it difficult to diagnose why they fail or how they plan. This paper addresses this gap by formally treating web tasks as sequential decision-making processes. We introduce a taxonomy that maps modern agent architectures to traditional planning paradigms: Step-by-Step agents to Breadth-First Search (BFS), Tree Search agents to Best-First Tree Search, and Full-Plan-in-Advance agents to Depth-First Search (DFS). This framework allows for a principled diagnosis of system failures like context drift and incoherent task decomposition. To evaluate these behaviors, we propose five novel evaluation metrics that assess trajectory quality beyond simple success rates. We support this analysis with a new dataset of 794 human-labeled trajectories from the WebArena benchmark. Finally, we validate our evaluation framework by comparing a baseline Step-by-Step agent against a novel Full-Plan-in-Advance implementation. Our results reveal that while the Step-by-Step agent aligns more closely with human gold trajectories (38% overall success), the Full-Plan-in-Advance agent excels in technical measures such as element accuracy (89%), demonstrating the necessity of our proposed metrics for selecting appropriate agent architectures based on specific application constraints.

AI Planning Framework for LLM-Based Web Agents

Abstract

Developing autonomous agents for web-based tasks is a core challenge in AI. While Large Language Model (LLM) agents can interpret complex user requests, they often operate as black boxes, making it difficult to diagnose why they fail or how they plan. This paper addresses this gap by formally treating web tasks as sequential decision-making processes. We introduce a taxonomy that maps modern agent architectures to traditional planning paradigms: Step-by-Step agents to Breadth-First Search (BFS), Tree Search agents to Best-First Tree Search, and Full-Plan-in-Advance agents to Depth-First Search (DFS). This framework allows for a principled diagnosis of system failures like context drift and incoherent task decomposition. To evaluate these behaviors, we propose five novel evaluation metrics that assess trajectory quality beyond simple success rates. We support this analysis with a new dataset of 794 human-labeled trajectories from the WebArena benchmark. Finally, we validate our evaluation framework by comparing a baseline Step-by-Step agent against a novel Full-Plan-in-Advance implementation. Our results reveal that while the Step-by-Step agent aligns more closely with human gold trajectories (38% overall success), the Full-Plan-in-Advance agent excels in technical measures such as element accuracy (89%), demonstrating the necessity of our proposed metrics for selecting appropriate agent architectures based on specific application constraints.
Paper Structure (33 sections, 5 equations, 5 figures, 10 tables)

This paper contains 33 sections, 5 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI Planning
  4. LLM-Based Agents
  5. Evaluation of Web-Agents
  6. Planning-Based Web Agent
  7. Web Agent Taxonomy Inspired by Traditional Planning
  8. Implementation of Full-Plan-in-Advance Agents
  9. Webpage Representation
  10. Plan Generation and Execution
  11. Evaluation Metrics
  12. LLM-as-A-Judge in Our Work
  13. Experimental Setting
  14. Dataset
  15. Web Agents
  16. ...and 18 more sections

Figures (5)

  • Figure 1: An example step from task 40 illustrating the agent’s decision-making process. The pink section, labeled A represents the previous action, the top gray section, labeled B details the agent’s reasoning process, the bottom gray section, labeled C, contains meta data, which we did not include in our analysis, and the yellow section, labeled D indicates the next action to be performed.
  • Figure 2: Success rates of Step-by-Step agent and Full-Plan-in-Advance agent on the WebArena benchmark divided to success on each domain.
  • Figure 3: Results of the proposed evaluation metrics of Step-by-Step agent and Full-Plan-in-Advance agent on the WebArena benchmark averaged over all domains.
  • Figure 4: A step performed by the Full-Plan-in-Advance agent for task number 727, where the reasoning phase suggests scrolling in the correct format of the instruction, but the actual action taken in 'None'. Task 727: DisLike all submissions created by PatientBuilder499 in subreddit videos.
  • Figure 5: A step performed by the Full-Plan-in-Advance agent for task 392, where the reasoning phase suggests scrolling but does not follow the required action format, causing the actual executed action to be 'None'. Task 392: Post "Good idea" for the merge request related to color ulitity in a11yproject.com project.

Theorems & Definitions (5)

  • definition 1: Recovery Rate
  • definition 2: Repetitiveness Rate
  • definition 3: Step Success Rate
  • definition 4: Partial Success Rate
  • definition 5: Element Accuracy Rate