Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Trustworthy GUI Agents: A Survey

Yucheng Shi, Wenhao Yu, Wenlin Yao, Wenhu Chen, Ninghao Liu

TL;DR

This survey examines the trustworthiness of GUI agents in five critical dimensions: security vulnerabilities, reliability in dynamic environments, transparency and explainability, ethical considerations, and evaluation methodologies.

Abstract

GUI agents, powered by large foundation models, can interact with digital interfaces, enabling various applications in web automation, mobile navigation, and software testing. However, their increasing autonomy has raised critical concerns about their security, privacy, and safety. This survey examines the trustworthiness of GUI agents in five critical dimensions: security vulnerabilities, reliability in dynamic environments, transparency and explainability, ethical considerations, and evaluation methodologies. We also identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making, and a lack of realistic evaluation benchmarks. These issues not only hinder real-world deployment but also call for comprehensive mitigation strategies beyond task success. As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential. This survey provides a foundation for advancing trustworthy GUI agents through systematic understanding and future research.

Towards Trustworthy GUI Agents: A Survey

TL;DR

This survey examines the trustworthiness of GUI agents in five critical dimensions: security vulnerabilities, reliability in dynamic environments, transparency and explainability, ethical considerations, and evaluation methodologies.

Abstract

GUI agents, powered by large foundation models, can interact with digital interfaces, enabling various applications in web automation, mobile navigation, and software testing. However, their increasing autonomy has raised critical concerns about their security, privacy, and safety. This survey examines the trustworthiness of GUI agents in five critical dimensions: security vulnerabilities, reliability in dynamic environments, transparency and explainability, ethical considerations, and evaluation methodologies. We also identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making, and a lack of realistic evaluation benchmarks. These issues not only hinder real-world deployment but also call for comprehensive mitigation strategies beyond task success. As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential. This survey provides a foundation for advancing trustworthy GUI agents through systematic understanding and future research.

Paper Structure

This paper contains 26 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Foundation of GUI Agents
  3. Security and Privacy
  4. Attacks and Vulnerabilities
  5. Privacy Risks
  6. Defenses and Mitigation Strategies
  7. Future Directions
  8. Reliability and Harmlessness
  9. Reliability
  10. Content Safety
  11. Alignment with Human Values
  12. Future Directions
  13. Explainability and Transparency
  14. Techniques for Explaining Agent Behavior
  15. Transparency in Decision-Making
  16. ...and 11 more sections

Figures (2)

  • Figure 1: An evolutionary tree of research on trustworthy GUI agents. Each branch represents a research direction, with notable works color-coded by their focus area, demonstrating how the field has evolved toward more comprehensive trustworthiness considerations. This figure is adapted from https://github.com/Mooler0410/LLMsPracticalGuide.
  • Figure 2: Overview of key dimensions for building trustworthy GUI agents, highlighting foundational abilities, evaluation metrics, security threats, reliability, harmfulness, explainability, transparency, and ethical implications.