Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BEAP-Agent: Backtrackable Execution and Adaptive Planning for GUI Agents

Ziyu Lu, Tengjin Weng, Yiying Yang, Yuhang Zhao, Xinxin Huang, Wenhao Jiang

TL;DR

The paper tackles GUI agents failing on long-horizon tasks due to sparse rewards and limited backtracking. It introduces BEAP-Agent, a DFS-based framework with Planner, Executor, and Tracker to enable long-range backtracking and dynamic task tracking over a state-space $S$ with actions $A(s)$ and transitions $T(s,a)$. Key contributions include formalizing GUI task execution as a state-space tree search, implementing a multi-component agent that can re-plan after backtracking, and demonstrating improved performance on the OSWorld benchmark with a 28.2% accuracy under 50 steps, along with ablation evidence validating backtracking and tracking components. The work advances robust GUI task execution by enabling deeper exploration and recovery, suggesting potential for integrating diverse models to further enhance grounding and planning in interactive environments.

Abstract

GUI agents are designed to automate repetitive tasks and enhance productivity. However, existing GUI agents struggle to recover once they follow an incorrect exploration path, often leading to task failure. In this work, we model GUI task execution as a DFS process and propose BEAP-Agent, a DFS-based framework that supports long-range, multi-level state backtracking with dynamic task tracking and updating. The framework consists of three collaborative components: Planner, Executor, and Tracker. Together, they enable effective task exploration and execution. BEAP-Agent fills the gap in systematic backtracking mechanisms for GUI agents, offering a systematic solution for long-horizon task exploration. We conducted a systematic evaluation on the OSWorld benchmark, where BEAP-Agent achieved an accuracy of 28.2%, validating the effectiveness of the proposed method.

BEAP-Agent: Backtrackable Execution and Adaptive Planning for GUI Agents

TL;DR

The paper tackles GUI agents failing on long-horizon tasks due to sparse rewards and limited backtracking. It introduces BEAP-Agent, a DFS-based framework with Planner, Executor, and Tracker to enable long-range backtracking and dynamic task tracking over a state-space with actions and transitions . Key contributions include formalizing GUI task execution as a state-space tree search, implementing a multi-component agent that can re-plan after backtracking, and demonstrating improved performance on the OSWorld benchmark with a 28.2% accuracy under 50 steps, along with ablation evidence validating backtracking and tracking components. The work advances robust GUI task execution by enabling deeper exploration and recovery, suggesting potential for integrating diverse models to further enhance grounding and planning in interactive environments.

Abstract

GUI agents are designed to automate repetitive tasks and enhance productivity. However, existing GUI agents struggle to recover once they follow an incorrect exploration path, often leading to task failure. In this work, we model GUI task execution as a DFS process and propose BEAP-Agent, a DFS-based framework that supports long-range, multi-level state backtracking with dynamic task tracking and updating. The framework consists of three collaborative components: Planner, Executor, and Tracker. Together, they enable effective task exploration and execution. BEAP-Agent fills the gap in systematic backtracking mechanisms for GUI agents, offering a systematic solution for long-horizon task exploration. We conducted a systematic evaluation on the OSWorld benchmark, where BEAP-Agent achieved an accuracy of 28.2%, validating the effectiveness of the proposed method.
Paper Structure (10 sections, 6 equations, 3 figures, 1 table)

This paper contains 10 sections, 6 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Backtracking Mechanism under DFS Modeling
  4. BEAP-Agent
  5. Experiments
  6. Experimental Setup
  7. Main Result
  8. Ablation Study
  9. Experimental Analysis
  10. Conclusion

Figures (3)

  • Figure 1: Comparison of GUI exploration trajectories. The left side illustrates single-step backtracking, which fails to recover from error. The right side demonstrates multi-step, long-distance backtracking based on DFS modeling, which successfully completes the task. Dashed states denote unexplored yet explorable regions of the search space.
  • Figure 2: The overview of BEAP-Agent. The lower right part is the BEAP-Agent pipeline, and the rest is the data display of each module. Each module is different from normal mode and backtrack mode, where Tracker is responsible for switching between modes.
  • Figure 3: Accuracy comparison across domains between BEAP-Agent and baseline.