Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AUITestAgent: Automatic Requirements Oriented GUI Function Testing

Yongxiang Hu, Xuan Wang, Yingchuan Wang, Yu Zhang, Shiyu Guo, Chaoyi Chen, Xin Wang, Yangfan Zhou

TL;DR

AUITestAgent advances GUI testing for mobile apps by automatically converting natural-language requirements into GUI interactions and semantic function verifications. It introduces a two-module architecture with dynamic, multi-agent orchestration (Specific steps and Concise expression patterns) and a multi-dimensional data extraction approach to analyze interaction traces for verification. Empirical results show higher interaction quality (TC/CS/CT/SE) and robust verification performance (Oracle Acc. ~0.90, 90% recall in anomaly detection), with practical impact demonstrated by Meituan deployments that found new bugs and saved regression effort. This work offers a scalable, industry-applicable solution to automate GUI function testing in rapidly evolving commercial apps.

Abstract

The Graphical User Interface (GUI) is how users interact with mobile apps. To ensure it functions properly, testing engineers have to make sure it functions as intended, based on test requirements that are typically written in natural language. While widely adopted manual testing and script-based methods are effective, they demand substantial effort due to the vast number of GUI pages and rapid iterations in modern mobile apps. This paper introduces AUITestAgent, the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification. Since test requirements typically contain interaction commands and verification oracles. AUITestAgent can extract GUI interactions from test requirements via dynamically organized agents. Then, AUITestAgent employs a multi-dimensional data extraction strategy to retrieve data relevant to the test requirements from the interaction trace and perform verification. Experiments on customized benchmarks demonstrate that AUITestAgent outperforms existing tools in the quality of generated GUI interactions and achieved the accuracy of verifications of 94%. Moreover, field deployment in Meituan has shown AUITestAgent's practical usability, with it detecting 4 new functional bugs during 10 regression tests in two months.

AUITestAgent: Automatic Requirements Oriented GUI Function Testing

TL;DR

AUITestAgent advances GUI testing for mobile apps by automatically converting natural-language requirements into GUI interactions and semantic function verifications. It introduces a two-module architecture with dynamic, multi-agent orchestration (Specific steps and Concise expression patterns) and a multi-dimensional data extraction approach to analyze interaction traces for verification. Empirical results show higher interaction quality (TC/CS/CT/SE) and robust verification performance (Oracle Acc. ~0.90, 90% recall in anomaly detection), with practical impact demonstrated by Meituan deployments that found new bugs and saved regression effort. This work offers a scalable, industry-applicable solution to automate GUI function testing in rapidly evolving commercial apps.

Abstract

The Graphical User Interface (GUI) is how users interact with mobile apps. To ensure it functions properly, testing engineers have to make sure it functions as intended, based on test requirements that are typically written in natural language. While widely adopted manual testing and script-based methods are effective, they demand substantial effort due to the vast number of GUI pages and rapid iterations in modern mobile apps. This paper introduces AUITestAgent, the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification. Since test requirements typically contain interaction commands and verification oracles. AUITestAgent can extract GUI interactions from test requirements via dynamically organized agents. Then, AUITestAgent employs a multi-dimensional data extraction strategy to retrieve data relevant to the test requirements from the interaction trace and perform verification. Experiments on customized benchmarks demonstrate that AUITestAgent outperforms existing tools in the quality of generated GUI interactions and achieved the accuracy of verifications of 94%. Moreover, field deployment in Meituan has shown AUITestAgent's practical usability, with it detecting 4 new functional bugs during 10 regression tests in two months.
Paper Structure (22 sections, 3 equations, 5 figures, 6 tables)

This paper contains 22 sections, 3 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Practices of GUI Function Testing
  3. Tool Design and Testing Workflow
  4. Overview
  5. Natural Language Driven GUI Interaction
  6. Specific steps pattern
  7. Concise expression pattern
  8. Function Verification
  9. Simplifying Test Oracles
  10. Multi-dimensional Data Extraction
  11. Function Verification
  12. Evaluation
  13. Evaluation Setup
  14. Dataset
  15. Baselines
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Practical Testing Requirements
  • Figure 2: Overview of AUITestAgent
  • Figure 3: Workflow of GUI Interaction Module
  • Figure 4: Workflow of Function Verification Module
  • Figure 5: Prompt of RQ2 Baseline