GUITester: Enabling GUI Agents for Exploratory Defect Discovery

TL;DR

This work tackles the challenge of autonomous exploratory GUI testing by identifying two core issues—goal-oriented masking and execution-bias attribution—that hinder defect discovery by existing GUI agents. It introduces GUITestBench, the first interactive benchmark for exploratory GUI defects, and GUITester, a multi-agent framework with a Planning Execution Module and a Hierarchical Reflection Module that proactively probes defects and robustly attributes anomalies. Empirical results show that GUITester significantly improves defect detection over strong baselines, achieving up to 48.90% F1 on GUITestBench and demonstrating practical viability on real apps. The approach provides a foundation for scalable GUI quality assurance by enabling autonomous exploration, defect exposure, and accurate defect reporting in dynamic mobile applications.

Abstract

Exploratory GUI testing is essential for software quality but suffers from high manual costs. While Multi-modal Large Language Model (MLLM) agents excel in navigation, they fail to autonomously discover defects due to two core challenges: \textit{Goal-Oriented Masking}, where agents prioritize task completion over reporting anomalies, and \textit{Execution-Bias Attribution}, where system defects are misidentified as agent errors. To address these, we first introduce \textbf{GUITestBench}, the first interactive benchmark for this task, featuring 143 tasks across 26 defects. We then propose \textbf{GUITester}, a multi-agent framework that decouples navigation from verification via two modules: (i) a \textit{Planning-Execution Module (PEM)} that proactively probes for defects via embedded testing intents, and (ii) a \textit{Hierarchical Reflection Module (HRM)} that resolves attribution ambiguity through interaction history analysis. GUITester achieves an F1-score of 48.90\% (Pass@3) on GUITestBench, outperforming state-of-the-art baselines (33.35\%). Our work demonstrates the feasibility of autonomous exploratory testing and provides a robust foundation for future GUI quality assurance~\footnote{Our code is now available in~\href{https://github.com/ADaM-BJTU/GUITestBench}{https://github.com/ADaM-BJTU/GUITestBench}}.

Figures (18)

• Figure 1: A GUI defect where "Clear Completed" fails to remove the completed tasks. Existing approaches either bypass the defect by attempting alternative paths, or misattribute it to agent execution errors, while GUITester accurately captures the anomaly and attributes it as a genuine GUI defect.
• Figure 2: Example of a Navigation Logic Error defect with two task types. The defect-oriented task explicitly guides the agent to trigger the defect, while the exploratory-oriented task may encounter it during exploration. After clicking "Navigate Up", the app returns to the home page (actual) instead of the searching list (expected).
• Figure 3: Defects distribution of GUITestBench
• Figure 4: GUITester architecture. The system consists of four agents: (1) a Planner Agent for subtask planning and test intent generation; (2) an Executor Agent for GUI operation; (3) a Monitor Agent for capturing anomalies and controlling the execution process; and (4) a Reflector Agent for anomalies attribution and providing further adjustment for planning suggestions. The workflow are shown in Appendix \ref{['appsec:algorithm']}.
• Figure 5: A defect detected by GUITester on ESPN (v8.6.0, November 2025). The "Learn More" button in the Cast panel is visually present but functionally non-responsive, failing to navigate to the expected support page.