Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Automated Crowdsourced Testing via Personified-LLM

Shengcheng Yu, Yuchen Ling, Chunrong Fang, Zhenyu Chen, Chunyang Chen

Abstract

The rapid proliferation and increasing complexity of software demand robust quality assurance, with graphical user interface (GUI) testing playing a pivotal role. Crowdsourced testing has proven effective in this context by leveraging the diversity of human testers to achieve rich, scenario-based coverage across varied devices, user behaviors, and usage environments. In parallel, automated testing, particularly with the advent of large language models (LLMs), offers significant advantages in controllability, reproducibility, and efficiency, enabling scalable and systematic exploration. However, automated approaches often lack the behavioral diversity characteristic of human testers, limiting their capability to fully simulate real-world testing dynamics. To address this gap, we present PersonaTester, a novel personified-LLM-based framework designed to automate crowdsourced GUI testing. By injecting representative personas, defined along three orthogonal dimensions: testing mindset, exploration strategy, and interaction habit, into LLM-based agents, PersonaTester enables the simulation of diverse human-like testing behaviors in a controllable and repeatable manner. Experimental results demonstrate that PersonaTester faithfully reproduces the behavioral patterns of real crowdworkers, exhibiting strong intra-persona consistency and clear inter-persona variability (117.86% -- 126.23% improvement over the baseline). Moreover, persona-guided testing agents consistently generate more effective test events and trigger more crashes (100+) and functional bugs (11) than the baseline without persona, thus substantially advancing the realism and effectiveness of automated crowdsourced GUI testing.

Towards Automated Crowdsourced Testing via Personified-LLM

Abstract

The rapid proliferation and increasing complexity of software demand robust quality assurance, with graphical user interface (GUI) testing playing a pivotal role. Crowdsourced testing has proven effective in this context by leveraging the diversity of human testers to achieve rich, scenario-based coverage across varied devices, user behaviors, and usage environments. In parallel, automated testing, particularly with the advent of large language models (LLMs), offers significant advantages in controllability, reproducibility, and efficiency, enabling scalable and systematic exploration. However, automated approaches often lack the behavioral diversity characteristic of human testers, limiting their capability to fully simulate real-world testing dynamics. To address this gap, we present PersonaTester, a novel personified-LLM-based framework designed to automate crowdsourced GUI testing. By injecting representative personas, defined along three orthogonal dimensions: testing mindset, exploration strategy, and interaction habit, into LLM-based agents, PersonaTester enables the simulation of diverse human-like testing behaviors in a controllable and repeatable manner. Experimental results demonstrate that PersonaTester faithfully reproduces the behavioral patterns of real crowdworkers, exhibiting strong intra-persona consistency and clear inter-persona variability (117.86% -- 126.23% improvement over the baseline). Moreover, persona-guided testing agents consistently generate more effective test events and trigger more crashes (100+) and functional bugs (11) than the baseline without persona, thus substantially advancing the realism and effectiveness of automated crowdsourced GUI testing.
Paper Structure (29 sections, 7 figures, 2 tables)

This paper contains 29 sections, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminary Study
  3. Persona Investigation from Real Crowdsourced Testing
  4. Illustrative Example
  5. Methodology
  6. Personification for LLM
  7. GUI State Understanding
  8. LLM Decision-Making for GUI Test Generation
  9. Operation Execution & Check
  10. Experiment
  11. Experimental Setting
  12. Research Questions (RQ)
  13. Data Preparation
  14. Baseline Construction
  15. Experimental Configuration
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Motivating Example: Exploration Trace of Personified-LLM Agents with Different Personas
  • Figure 2: Overview of PersonaTester Workflow
  • Figure 3: Example of Prompt for LLM Personification ( Persona F)
  • Figure 4: RQ1.1: Intra-Cluster Cohesion
  • Figure 5: RQ1.2: Inter-Cluster Separation
  • ...and 2 more figures