Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continual GUI Agents

Ziwei Liu, Borui Kang, Hangjie Yuan, Zixiang Zhao, Wei Li, Yifan Zhu, Tao Feng

TL;DR

The paper addresses the problem of GUI grounding under domain and resolution shifts by proposing Continual GUI Agents. It introduces GUI-AiF, a reinforcement fine-tuning framework that adds two anchoring rewards, APR-iF and ARR-iF, to encourage exploration of diverse interaction points and region scales, integrated with GRPO and KL regularization to mitigate forgetting. Empirical results on ScreenSpot-V1, V2, and Pro show that GUI-AiF outperforms SFT and existing RFT baselines, with ablations confirming the complementary benefits of APR-iF and ARR-iF and revealing forward transfer benefits. This work establishes the first continual GUI agent framework and demonstrates the potential of reinforcement fine-tuning to sustain grounding performance across evolving GUIs, enabling more robust real-world automation across platforms and resolutions.

Abstract

As digital environments (data distribution) are in flux, with new GUI data arriving over time-introducing new domains or resolutions-agents trained on static environments deteriorate in performance. In this work, we introduce Continual GUI Agents, a new task that requires GUI agents to perform continual learning under shifted domains and resolutions. We find existing methods fail to maintain stable grounding as GUI distributions shift over time, due to the diversity of UI interaction points and regions in fluxing scenarios. To address this, we introduce GUI-Anchoring in Flux (GUI-AiF), a new reinforcement fine-tuning framework that stabilizes continual learning through two novel rewards: Anchoring Point Reward in Flux (APR-iF) and Anchoring Region Reward in Flux (ARR-iF). These rewards guide the agents to align with shifting interaction points and regions, mitigating the tendency of existing reward strategies to over-adapt to static grounding cues (e.g., fixed coordinates or element scales). Extensive experiments show GUI-AiF surpasses state-of-the-art baselines. Our work establishes the first continual learning framework for GUI agents, revealing the untapped potential of reinforcement fine-tuning for continual GUI Agents.

Continual GUI Agents

TL;DR

The paper addresses the problem of GUI grounding under domain and resolution shifts by proposing Continual GUI Agents. It introduces GUI-AiF, a reinforcement fine-tuning framework that adds two anchoring rewards, APR-iF and ARR-iF, to encourage exploration of diverse interaction points and region scales, integrated with GRPO and KL regularization to mitigate forgetting. Empirical results on ScreenSpot-V1, V2, and Pro show that GUI-AiF outperforms SFT and existing RFT baselines, with ablations confirming the complementary benefits of APR-iF and ARR-iF and revealing forward transfer benefits. This work establishes the first continual GUI agent framework and demonstrates the potential of reinforcement fine-tuning to sustain grounding performance across evolving GUIs, enabling more robust real-world automation across platforms and resolutions.

Abstract

As digital environments (data distribution) are in flux, with new GUI data arriving over time-introducing new domains or resolutions-agents trained on static environments deteriorate in performance. In this work, we introduce Continual GUI Agents, a new task that requires GUI agents to perform continual learning under shifted domains and resolutions. We find existing methods fail to maintain stable grounding as GUI distributions shift over time, due to the diversity of UI interaction points and regions in fluxing scenarios. To address this, we introduce GUI-Anchoring in Flux (GUI-AiF), a new reinforcement fine-tuning framework that stabilizes continual learning through two novel rewards: Anchoring Point Reward in Flux (APR-iF) and Anchoring Region Reward in Flux (ARR-iF). These rewards guide the agents to align with shifting interaction points and regions, mitigating the tendency of existing reward strategies to over-adapt to static grounding cues (e.g., fixed coordinates or element scales). Extensive experiments show GUI-AiF surpasses state-of-the-art baselines. Our work establishes the first continual learning framework for GUI agents, revealing the untapped potential of reinforcement fine-tuning for continual GUI Agents.
Paper Structure (18 sections, 6 equations, 12 figures, 6 tables)

This paper contains 18 sections, 6 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. GUI Agents
  4. Continual Post-Training
  5. Method
  6. Problem Formation
  7. GUI-AiF Rewards
  8. Reinforcement Fine-tuning with GUI-AiF
  9. Experiments
  10. Continual GUI Agents Implementation
  11. Baselines and Training Details
  12. Main Results
  13. Discussion
  14. Conclusion
  15. Upper Boundary of GUI-AiF
  16. ...and 3 more sections

Figures (12)

  • Figure 1: Workflow. Continual GUI Agents operate under two evolving scenarios: domain-in-flux (e.g., from Mobile OS to Web OS) and resolution-in-flux (e.g., scaling from 1080p to 4K).
  • Figure 2: Overview. The Above shows the Continual GUI Agents pipeline. GUI-AiF is proposed to advance the RFT paradigm in this setting by shaping grounding rewards: APR-iF () encourages diverse interaction points, while ARR-iF () refines element regions. Together, GUI-AiF enables agents to better adapt to varying interaction locations and element scales.
  • Figure 3: Hyperparameter sensitivity analysis for $\alpha$ and $\gamma$. Performance peaks at ($\alpha$, $\gamma$) = (1,1) on three benchmarks.
  • Figure 4: Forward transfer evaluation of GUI-AiF.
  • Figure 5: Reward value analysis within two scenarios. (a)/(c) show reward statistics, while (b)/(d) show their trend distributions.
  • ...and 7 more figures