Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XUAT-Copilot: Multi-Agent Collaborative System for Automated User Acceptance Testing with Large Language Model

Zhitao Wang, Wei Wang, Zirao Li, Long Wang, Can Yi, Xinjie Xu, Luyang Cao, Hanjing Su, Shouzhi Chen, Jun Zhou

TL;DR

This paper tackles automating user acceptance testing for WeChat Pay by introducing XUAT-Copilot, an LLM-powered multi-agent system that automates test-script generation. It decomposes the task into three agents (Operation, Parameter Selection, Inspection) plus Perception and Rewriting modules, enabling collaborative planning, parameterization, and verification of GUI-driven actions. The approach demonstrates substantial performance gains over single-agent baselines and is deployed in a formal WeChat Pay testing environment, reducing manual scripting workload. Overall, the work highlights the feasibility and practical impact of autonomous, LLM-guided UAT automation with real-world applicability and plans for further generalization.

Abstract

In past years, we have been dedicated to automating user acceptance testing (UAT) process of WeChat Pay, one of the most influential mobile payment applications in China. A system titled XUAT has been developed for this purpose. However, there is still a human-labor-intensive stage, i.e, test scripts generation, in the current system. Therefore, in this paper, we concentrate on methods of boosting the automation level of the current system, particularly the stage of test scripts generation. With recent notable successes, large language models (LLMs) demonstrate significant potential in attaining human-like intelligence and there has been a growing research area that employs LLMs as autonomous agents to obtain human-like decision-making capabilities. Inspired by these works, we propose an LLM-powered multi-agent collaborative system, named XUAT-Copilot, for automated UAT. The proposed system mainly consists of three LLM-based agents responsible for action planning, state checking and parameter selecting, respectively, and two additional modules for state sensing and case rewriting. The agents interact with testing device, make human-like decision and generate action command in a collaborative way. The proposed multi-agent system achieves a close effectiveness to human testers in our experimental studies and gains a significant improvement of Pass@1 accuracy compared with single-agent architecture. More importantly, the proposed system has launched in the formal testing environment of WeChat Pay mobile app, which saves a considerable amount of manpower in the daily development work.

XUAT-Copilot: Multi-Agent Collaborative System for Automated User Acceptance Testing with Large Language Model

TL;DR

This paper tackles automating user acceptance testing for WeChat Pay by introducing XUAT-Copilot, an LLM-powered multi-agent system that automates test-script generation. It decomposes the task into three agents (Operation, Parameter Selection, Inspection) plus Perception and Rewriting modules, enabling collaborative planning, parameterization, and verification of GUI-driven actions. The approach demonstrates substantial performance gains over single-agent baselines and is deployed in a formal WeChat Pay testing environment, reducing manual scripting workload. Overall, the work highlights the feasibility and practical impact of autonomous, LLM-guided UAT automation with real-world applicability and plans for further generalization.

Abstract

In past years, we have been dedicated to automating user acceptance testing (UAT) process of WeChat Pay, one of the most influential mobile payment applications in China. A system titled XUAT has been developed for this purpose. However, there is still a human-labor-intensive stage, i.e, test scripts generation, in the current system. Therefore, in this paper, we concentrate on methods of boosting the automation level of the current system, particularly the stage of test scripts generation. With recent notable successes, large language models (LLMs) demonstrate significant potential in attaining human-like intelligence and there has been a growing research area that employs LLMs as autonomous agents to obtain human-like decision-making capabilities. Inspired by these works, we propose an LLM-powered multi-agent collaborative system, named XUAT-Copilot, for automated UAT. The proposed system mainly consists of three LLM-based agents responsible for action planning, state checking and parameter selecting, respectively, and two additional modules for state sensing and case rewriting. The agents interact with testing device, make human-like decision and generate action command in a collaborative way. The proposed multi-agent system achieves a close effectiveness to human testers in our experimental studies and gains a significant improvement of Pass@1 accuracy compared with single-agent architecture. More importantly, the proposed system has launched in the formal testing environment of WeChat Pay mobile app, which saves a considerable amount of manpower in the daily development work.
Paper Structure (23 sections, 2 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 23 sections, 2 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Automated UAT and XUAT System
  4. Large Language Models
  5. Autonomous Agent and LLM-Based Agent
  6. XUAT-Copilot
  7. Problem Statement
  8. Perception Module
  9. Rewriting Module
  10. Multi-Agent Collaborative Framework
  11. Multi-Agent Instruct2Act Algorithm
  12. Skill Library
  13. Prompt Structure
  14. Operation Agent
  15. Parameter Selection Agent
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The overview of XUAT system and XUAT-Copilot system
  • Figure 2: The proposed multi-agent collaborative framework
  • Figure 3: Image Processing Pipeline For Widgets With Hyperlink