ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory

TL;DR

ActionEngine is proposed, a training-free framework that transitions from reactive execution to programmatic planning through a novel two-agent architecture: a Crawling Agent that constructs an updatable state-machine memory of the GUIs through offline exploration, and an Execution Agent that leverages this memory to synthesize complete, executable Python programs for online task execution.

Abstract

Existing Graphical User Interface (GUI) agents operate through step-by-step calls to vision language models--taking a screenshot, reasoning about the next action, executing it, then repeating on the new page--resulting in high costs and latency that scale with the number of reasoning steps, and limited accuracy due to no persistent memory of previously visited pages. We propose ActionEngine, a training-free framework that transitions from reactive execution to programmatic planning through a novel two-agent architecture: a Crawling Agent that constructs an updatable state-machine memory of the GUIs through offline exploration, and an Execution Agent that leverages this memory to synthesize complete, executable Python programs for online task execution. To ensure robustness against evolving interfaces, execution failures trigger a vision-based re-grounding fallback that repairs the failed action and updates the memory. This design drastically improves both efficiency and accuracy: on Reddit tasks from the WebArena benchmark, our agent achieves 95% task success with on average a single LLM call, compared to 66% for the strongest vision-only baseline, while reducing cost by 11.8x and end-to-end latency by 2x. Together, these components yield scalable and reliable GUI interaction by combining global programmatic planning, crawler-validated action templates, and node-level execution with localized validation and repair.

Figures (6)

• Figure 1: Reactive vs. Programmatic Paradigms. Our approach (bottom) replaces $O(N)$ visual inferences with a single $O(1)$ planning phase, drastically reducing latency compared to the reactive baseline (top).
• Figure 2: System architecture with two specialized agents. The Crawling Agent (top, offline) explores the website/app to construct and maintain the state-machine memory. The Execution Agent (bottom, online) uses this memory to generate a high-level program sketch, ground it into executable UI calls through graph search, and compile and execute the plan. When execution fails, an MLLM-based fallback recovers and updates the memory, while a Python validator handles logic errors.
• Figure 3: Illustration of our State-Machine Graph model. The graph shows a subset of states and transitions in a Reddit-like website, where nodes are States and edges are Operations (black: UI manipulation, red: data collection).
• Figure 4: The Home Page State is composed of four Atoms that represents the navigation bar, the search bar, post type selection and filter.
• Figure 5: State template equivalence across dynamic page instances.