Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning

Jiahao Li, Kaer Huang

TL;DR

The paper addresses how GUI agents can autonomously operate digital devices through GUIs by leveraging foundation models and reinforcement learning. It formalizes GUI tasks as the MDP $M=\{S,A,T,r,\\u03b3}$ with screen captures as states and UI actions as decisions, then surveys modular architectures (Perception/Planning/Acting) and three training paradigms (Prompt-based, SFT, RL). It catalogs representative LLM- and MLLM-based agents, detailing advances in multimodal perception, reasoning-driven planning (including CoT-style approaches and dynamic replanning), and adaptive action generation with cross-platform generalization. The discussion identifies challenges in perception robustness, long-horizon reasoning, data efficiency, and evaluation, offering future directions such as semantic grounding, continual learning, and human-in-the-loop feedback to enable robust, real-world GUI automation.

Abstract

Graphical User Interface (GUI) agents, driven by Multi-modal Large Language Models (MLLMs), have emerged as a promising paradigm for enabling intelligent interaction with digital systems. This paper provides a structured survey of recent advances in GUI agents, focusing on architectures enhanced by Reinforcement Learning (RL). We first formalize GUI agent tasks as Markov Decision Processes and discuss typical execution environments and evaluation metrics. We then review the modular architecture of (M)LLM-based GUI agents, covering Perception, Planning, and Acting modules, and trace their evolution through representative works. Furthermore, we categorize GUI agent training methodologies into Prompt-based, Supervised Fine-Tuning (SFT)-based, and RL-based approaches, highlighting the progression from simple prompt engineering to dynamic policy learning via RL. Our summary illustrates how recent innovations in multimodal perception, decision reasoning, and adaptive action generation have significantly improved the generalization and robustness of GUI agents in complex real-world environments. We conclude by identifying key challenges and future directions for building more capable and reliable GUI agents.

A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning

TL;DR

The paper addresses how GUI agents can autonomously operate digital devices through GUIs by leveraging foundation models and reinforcement learning. It formalizes GUI tasks as the MDP with screen captures as states and UI actions as decisions, then surveys modular architectures (Perception/Planning/Acting) and three training paradigms (Prompt-based, SFT, RL). It catalogs representative LLM- and MLLM-based agents, detailing advances in multimodal perception, reasoning-driven planning (including CoT-style approaches and dynamic replanning), and adaptive action generation with cross-platform generalization. The discussion identifies challenges in perception robustness, long-horizon reasoning, data efficiency, and evaluation, offering future directions such as semantic grounding, continual learning, and human-in-the-loop feedback to enable robust, real-world GUI automation.

Abstract

Graphical User Interface (GUI) agents, driven by Multi-modal Large Language Models (MLLMs), have emerged as a promising paradigm for enabling intelligent interaction with digital systems. This paper provides a structured survey of recent advances in GUI agents, focusing on architectures enhanced by Reinforcement Learning (RL). We first formalize GUI agent tasks as Markov Decision Processes and discuss typical execution environments and evaluation metrics. We then review the modular architecture of (M)LLM-based GUI agents, covering Perception, Planning, and Acting modules, and trace their evolution through representative works. Furthermore, we categorize GUI agent training methodologies into Prompt-based, Supervised Fine-Tuning (SFT)-based, and RL-based approaches, highlighting the progression from simple prompt engineering to dynamic policy learning via RL. Our summary illustrates how recent innovations in multimodal perception, decision reasoning, and adaptive action generation have significantly improved the generalization and robustness of GUI agents in complex real-world environments. We conclude by identifying key challenges and future directions for building more capable and reliable GUI agents.
Paper Structure (17 sections, 2 figures)

This paper contains 17 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Benchmarks
  4. (M)LLM-based GUI Agent Architectures
  5. Perception
  6. Planning
  7. Acting
  8. (M)LLM-based GUI Agent Taxonomy
  9. Modal Taxonomy
  10. LLM-based Agent
  11. MLLM-based Agent
  12. Training Taxonomy
  13. Prompt-based Methods
  14. SFT-based Method
  15. RL-based Method
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Architecture of GUI agents powered by (Multi-modal) Large Language Models. Comprising three interconnected modules: Perception, Planning, and Acting. The Perception module is responsible for perceiving and understanding the GUI state, the Planning module formulates high-level action strategies, and the Acting module executes operations, collectively enabling intelligent interaction with the GUI.
  • Figure 2: Taxonomy of (M)LLM-based GUI Agents. This figure presents a classification of GUI agents based on their underlying model modalities and training methodologies.