Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing

Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, Qing Wang

TL;DR

This paper tackles the difficulty of generating meaningful text inputs for automated mobile GUI testing. It introduces QTypist, a GPT-3-based system that treats input generation as a cloze-style task and uses context-aware prompting plus prompt tuning to produce semantic values. The approach leverages view hierarchy data to extract widget, local, and global context and constructs prompts to guide the LLM. Evaluations across 106 apps show substantial improvements in passing rate and coverage, and integration with existing testing tools yields dramatically higher activity/page counts and bugs found. The work provides a data-driven methodology and dataset for future research in semantic input generation.

Abstract

Automated GUI testing is widely used to help ensure the quality of mobile apps. However, many GUIs require appropriate text inputs to proceed to the next page which remains a prominent obstacle for testing coverage. Considering the diversity and semantic requirement of valid inputs (e.g., flight departure, movie name), it is challenging to automate the text input generation. Inspired by the fact that the pre-trained Large Language Model (LLM) has made outstanding progress in text generation, we propose an approach named QTypist based on LLM for intelligently generating semantic input text according to the GUI context. To boost the performance of LLM in the mobile testing scenario, we develop a prompt-based data construction and tuning method which automatically extracts the prompts and answers for model tuning. We evaluate QTypist on 106 apps from Google Play and the result shows that the passing rate of QTypist is 87%, which is 93% higher than the best baseline. We also integrate QTypist with the automated GUI testing tools and it can cover 42% more app activities, 52% more pages, and subsequently help reveal 122% more bugs compared with the raw tool.

Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing

TL;DR

This paper tackles the difficulty of generating meaningful text inputs for automated mobile GUI testing. It introduces QTypist, a GPT-3-based system that treats input generation as a cloze-style task and uses context-aware prompting plus prompt tuning to produce semantic values. The approach leverages view hierarchy data to extract widget, local, and global context and constructs prompts to guide the LLM. Evaluations across 106 apps show substantial improvements in passing rate and coverage, and integration with existing testing tools yields dramatically higher activity/page counts and bugs found. The work provides a data-driven methodology and dataset for future research in semantic input generation.

Abstract

Automated GUI testing is widely used to help ensure the quality of mobile apps. However, many GUIs require appropriate text inputs to proceed to the next page which remains a prominent obstacle for testing coverage. Considering the diversity and semantic requirement of valid inputs (e.g., flight departure, movie name), it is challenging to automate the text input generation. Inspired by the fact that the pre-trained Large Language Model (LLM) has made outstanding progress in text generation, we propose an approach named QTypist based on LLM for intelligently generating semantic input text according to the GUI context. To boost the performance of LLM in the mobile testing scenario, we develop a prompt-based data construction and tuning method which automatically extracts the prompts and answers for model tuning. We evaluate QTypist on 106 apps from Google Play and the result shows that the passing rate of QTypist is 87%, which is 93% higher than the best baseline. We also integrate QTypist with the automated GUI testing tools and it can cover 42% more app activities, 52% more pages, and subsequently help reveal 122% more bugs compared with the raw tool.
Paper Structure (9 sections, 4 figures, 1 table)

This paper contains 9 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. motivational study and background
  3. Motivational Study
  4. Data Collection
  5. The Categories of Text Inputs
  6. Background of Large Language Model
  7. Approach
  8. Context-aware Input Prompt Generation
  9. Context Extraction

Figures (4)

  • Figure 1: Example of text input in Android app.
  • Figure 2: Example of 5 categories of text input.
  • Figure 3: The model structure of GPT-3.
  • Figure 4: Overview of QTypist.