Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ScaleTrack: Scaling and back-tracking Automated GUI Agents

Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, Lin Ma

TL;DR

ScaleTrack addresses two key limitations of automated GUI agents: limited grounding data and the absence of back-tracking in planning. It scales grounding data by unifying diverse sources into a fixed training template and introduces a back-tracking mechanism that jointly predicts the next action and the historical actions leading to the current state, enabling better interpretation of GUI evolution. The method uses a two-stage training regime around a base MLLM (Qwen2-VL-7B) and demonstrates improvements across grounding, offline and online task settings on benchmarks like ScreenSpot, AndroidControl, GUI Odyssey, AndroidWorld, and MobileMiniWob. The approach enhances generalization across cross-platform GUI tasks and offers practical benefits for robust, automated task execution in real-world digital environments, with data and code to be released publicly.

Abstract

Automated GUI agents aims to facilitate user interaction by automatically performing complex tasks in digital environments, such as web, mobile, desktop devices. It receives textual task instruction and GUI description to generate executable actions (\emph{e.g.}, click) and operation boxes step by step. Training a GUI agent mainly involves grounding and planning stages, in which the GUI grounding focuses on finding the execution coordinates according to the task, while the planning stage aims to predict the next action based on historical actions. However, previous work suffers from the limitations of insufficient training data for GUI grounding, as well as the ignorance of backtracking historical behaviors for GUI planning. To handle the above challenges, we propose ScaleTrack, a training framework by scaling grounding and backtracking planning for automated GUI agents. We carefully collected GUI samples of different synthesis criterions from a wide range of sources, and unified them into the same template for training GUI grounding models. Moreover, we design a novel training strategy that predicts the next action from the current GUI image, while also backtracking the historical actions that led to the GUI image. In this way, ScaleTrack explains the correspondence between GUI images and actions, which effectively describes the evolution rules of the GUI environment. Extensive experimental results demonstrate the effectiveness of ScaleTrack. Data and code will be available at url.

ScaleTrack: Scaling and back-tracking Automated GUI Agents

TL;DR

ScaleTrack addresses two key limitations of automated GUI agents: limited grounding data and the absence of back-tracking in planning. It scales grounding data by unifying diverse sources into a fixed training template and introduces a back-tracking mechanism that jointly predicts the next action and the historical actions leading to the current state, enabling better interpretation of GUI evolution. The method uses a two-stage training regime around a base MLLM (Qwen2-VL-7B) and demonstrates improvements across grounding, offline and online task settings on benchmarks like ScreenSpot, AndroidControl, GUI Odyssey, AndroidWorld, and MobileMiniWob. The approach enhances generalization across cross-platform GUI tasks and offers practical benefits for robust, automated task execution in real-world digital environments, with data and code to be released publicly.

Abstract

Automated GUI agents aims to facilitate user interaction by automatically performing complex tasks in digital environments, such as web, mobile, desktop devices. It receives textual task instruction and GUI description to generate executable actions (\emph{e.g.}, click) and operation boxes step by step. Training a GUI agent mainly involves grounding and planning stages, in which the GUI grounding focuses on finding the execution coordinates according to the task, while the planning stage aims to predict the next action based on historical actions. However, previous work suffers from the limitations of insufficient training data for GUI grounding, as well as the ignorance of backtracking historical behaviors for GUI planning. To handle the above challenges, we propose ScaleTrack, a training framework by scaling grounding and backtracking planning for automated GUI agents. We carefully collected GUI samples of different synthesis criterions from a wide range of sources, and unified them into the same template for training GUI grounding models. Moreover, we design a novel training strategy that predicts the next action from the current GUI image, while also backtracking the historical actions that led to the GUI image. In this way, ScaleTrack explains the correspondence between GUI images and actions, which effectively describes the evolution rules of the GUI environment. Extensive experimental results demonstrate the effectiveness of ScaleTrack. Data and code will be available at url.
Paper Structure (21 sections, 2 equations, 3 figures, 7 tables)

This paper contains 21 sections, 2 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal LLMs
  4. GUI Agents
  5. Method
  6. Formulation
  7. Data Scale Across Different Domains and Types
  8. Unified Data Format
  9. Merge Data Sources
  10. Enhancing Action Understanding with back-tracking
  11. From Predicting the Future to back-tracking
  12. Different Expressions of Previous Actions
  13. Experiments
  14. Training Details
  15. Training Data
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Difference of forward-planning and back-tracking.
  • Figure 2: Overall description of our proposed ScaleTrack in processing task instruction and generating actions via forward-planning and back-tracking, as well as the format of training data
  • Figure 3: Scaling curve of ScaleTrack-7B on ScreenSpot.