Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents

Zikang Liu, Junyi Li, Wayne Xin Zhao, Dawei Gao, Yaliang Li, Ji-rong Wen

TL;DR

PAL-UI tackles memory bottlenecks in vision-based GUI agents by enabling active look-back through a dedicated retrieval tool combined with dual-level history summaries. It trains on 8.6K step-level trajectories and demonstrates that retrieving targeted past observations during planning yields strong improvements on mobile GUI benchmarks and transfers to web interfaces. The approach achieves state-of-the-art results with A100-scale training, and analyses show the method balances efficiency and information retention while preserving generalization across domains. This work highlights active memory retrieval as a powerful mechanism for robust, long-horizon GUI planning in multimodal systems.

Abstract

Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) promise human-like interaction with software applications, yet long-horizon tasks remain challenging due to memory limitations. Existing approaches either truncate history or rely on simple textual summaries, which risk losing critical information when past visual details become necessary for future decisions. In this paper, we propose \textbf{PAL-UI} (\textbf{P}lanning with \textbf{A}ctive \textbf{L}ook-back), a novel framework that enables GUI agents to adaptively retrieve past observations when required. PAL-UI combines a dual-level summarization agent, capturing both observation-level cues and action-level outcomes, with a dedicated retrieval tool that allows the agent to recall specific historical screenshots during planning. We curate a step-level instruction dataset of 8.6K samples from mobile GUI navigation trajectories and train \textbf{PAL-UI-3B} and \textbf{PAL-UI-7B} models based on Qwen2.5-VL. Extensive experiments demonstrate that PAL-UI significantly outperforms baseline models and prior methods in mobile GUI navigation tasks, even under data-efficient settings. Moreover, PAL-UI exhibits strong cross-domain generalization, achieving notable improvements in web navigation without additional training. Our work highlights the potential of active memory retrieval for long-horizon planning capabilities of vision-based GUI agents.

PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents

TL;DR

PAL-UI tackles memory bottlenecks in vision-based GUI agents by enabling active look-back through a dedicated retrieval tool combined with dual-level history summaries. It trains on 8.6K step-level trajectories and demonstrates that retrieving targeted past observations during planning yields strong improvements on mobile GUI benchmarks and transfers to web interfaces. The approach achieves state-of-the-art results with A100-scale training, and analyses show the method balances efficiency and information retention while preserving generalization across domains. This work highlights active memory retrieval as a powerful mechanism for robust, long-horizon GUI planning in multimodal systems.

Abstract

Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) promise human-like interaction with software applications, yet long-horizon tasks remain challenging due to memory limitations. Existing approaches either truncate history or rely on simple textual summaries, which risk losing critical information when past visual details become necessary for future decisions. In this paper, we propose \textbf{PAL-UI} (\textbf{P}lanning with \textbf{A}ctive \textbf{L}ook-back), a novel framework that enables GUI agents to adaptively retrieve past observations when required. PAL-UI combines a dual-level summarization agent, capturing both observation-level cues and action-level outcomes, with a dedicated retrieval tool that allows the agent to recall specific historical screenshots during planning. We curate a step-level instruction dataset of 8.6K samples from mobile GUI navigation trajectories and train \textbf{PAL-UI-3B} and \textbf{PAL-UI-7B} models based on Qwen2.5-VL. Extensive experiments demonstrate that PAL-UI significantly outperforms baseline models and prior methods in mobile GUI navigation tasks, even under data-efficient settings. Moreover, PAL-UI exhibits strong cross-domain generalization, achieving notable improvements in web navigation without additional training. Our work highlights the potential of active memory retrieval for long-horizon planning capabilities of vision-based GUI agents.

Paper Structure

This paper contains 32 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GUI Agents
  4. Memory Management for GUI Agents
  5. MLLM-based Tool-use Agents
  6. Problem Formulation
  7. Approach
  8. Tool Construction and Summary Agent
  9. Observation-level Captioner.
  10. Action-level Validator.
  11. Memory Retrieval Tool.
  12. Active Look-back Planning Process
  13. Training Data Construction
  14. Seed Trajectory Collection.
  15. Tool-Use Calling Curation.
  16. ...and 17 more sections

Figures (2)

  • Figure 1: The illustration of our proposed PAL-UI agent. We utilize an observation-level captioner and an action-level validator for memory compression. Then, equipped with a retrieval tool, the agent is able to actively recall detailed visual information from past memory at inference time.
  • Figure 2: Behavior of active look-back.