Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning

Longxi Gao, Li Zhang, Pengzhi Gao, Wei Liu, Jian Luan, Mengwei Xu

TL;DR

GUI-Shift introduces a self-supervised reinforcement learning framework that leverages the K-step GUI Transition task to learn GUI dynamics from unlabeled trajectories. By applying group-relative policy optimization with a rule-based reward, it trains VLMs to predict initial actions that cause state transitions, while data filtering ensures high-quality supervision. The approach yields consistent improvements across GUI automation and grounding benchmarks and reduces reliance on expensive human annotations, highlighting the practicality of self-supervised RL for scalable GUI agent training.

Abstract

Training effective Vision-Language Models (VLMs) for GUI agents typically depends on large-scale annotated datasets, whose collection is both labor-intensive and error-prone. We introduce K-step GUI Transition, a self-supervised inverse dynamics task in which VLMs learn GUI dynamics by predicting the initial action that causes a transition between two GUI states. This approach eliminates the need for natural language instructions and enables scalable dataset construction from existing GUI trajectories or automated exploration. Building on this task, we propose GUI-Shift, a reinforcement learning (RL) framework that combines rule-based optimization with data filtering to improve VLM performance. We conduct extensive experiments using multiple VLM backbones across four benchmarks, spanning GUI task automation (AndroidControl, GUI Odyssey) and GUI grounding (ScreenSpot-v2, ScreenSpot-Pro). Our results show that training on GUI-Shift generalizes well to both GUI automation and grounding tasks, yielding up to an 11.2% increase in GUI automation accuracy. This study underscores the potential of self-supervised RL to leverage unlabeled GUI trajectories and offers a scalable alternative to training with annotated samples.

GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning

TL;DR

GUI-Shift introduces a self-supervised reinforcement learning framework that leverages the K-step GUI Transition task to learn GUI dynamics from unlabeled trajectories. By applying group-relative policy optimization with a rule-based reward, it trains VLMs to predict initial actions that cause state transitions, while data filtering ensures high-quality supervision. The approach yields consistent improvements across GUI automation and grounding benchmarks and reduces reliance on expensive human annotations, highlighting the practicality of self-supervised RL for scalable GUI agent training.

Abstract

Training effective Vision-Language Models (VLMs) for GUI agents typically depends on large-scale annotated datasets, whose collection is both labor-intensive and error-prone. We introduce K-step GUI Transition, a self-supervised inverse dynamics task in which VLMs learn GUI dynamics by predicting the initial action that causes a transition between two GUI states. This approach eliminates the need for natural language instructions and enables scalable dataset construction from existing GUI trajectories or automated exploration. Building on this task, we propose GUI-Shift, a reinforcement learning (RL) framework that combines rule-based optimization with data filtering to improve VLM performance. We conduct extensive experiments using multiple VLM backbones across four benchmarks, spanning GUI task automation (AndroidControl, GUI Odyssey) and GUI grounding (ScreenSpot-v2, ScreenSpot-Pro). Our results show that training on GUI-Shift generalizes well to both GUI automation and grounding tasks, yielding up to an 11.2% increase in GUI automation accuracy. This study underscores the potential of self-supervised RL to leverage unlabeled GUI trajectories and offers a scalable alternative to training with annotated samples.
Paper Structure (16 sections, 4 equations, 3 figures, 10 tables)

This paper contains 16 sections, 4 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mobile GUI Agents
  4. Rule-based Reinforcement Learning
  5. Methodology
  6. Preliminaries
  7. Reward Design
  8. GUI-Shift Framework
  9. Experiments
  10. Experimental Setup
  11. Benchmarks and Results
  12. Ablation Study
  13. Conclusion
  14. Training Hyper-parameters
  15. Task Automation Results with and without Data Filtering
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overview of the GUI-Shift framework. Left: $K$-step GUI Transition replaces annotated instructions with the target state $S_{t+k}$, enabling scalable data construction through automated offline exploration. Middle: The model learns GUI dynamics by predicting the action that causes the transition. Right: GUI-Shift achieves self-supervised training by applying GRPO to GUI Transition.
  • Figure 2: Impact of Data filtering. Each model is fine-tuned on 2K $K$-step GUI Transition samples. Filtered data are more informative and challenging, and outperform unfiltered ones.
  • Figure 3: Comparison of training algorithms for the $K$-step GUI Transition task ($k \in \{1,2,3,4\}$). Qwen2.5-VL-7B and InternVL3-8B are fine-tuned with 2K samples for each $k$ and evaluated on AndroidControl. GRPO provides notable performance gains over SFT for all models and settings.