Practical, Automated Scenario-based Mobile App Testing

TL;DR

ScenTest addresses the mismatch between automated mobile app testing and business logic by integrating GUI image understanding with event knowledge graphs constructed from crowdsourced test reports to guide scenario-based test generation. It builds EKGs with four entity types and five relation types, extracting and coreferencing information from GUI screenshots and textual steps, then uses GUI analysis to query the EKG and drive a depth-first, sub-scenario guided exploration. Across eight representative scenarios and 124 apps, ScenTest demonstrates high correctness, reliability, and adequacy, uncovering a substantial set of real-world bugs and outperforming baseline automated testing tools in scenario-focused testing. The approach offers a practical, scalable complement to existing automation, supported by a large, replicable dataset and a reproducible tool package.

Abstract

The importance of mobile application (app) quality insurance is increasing with the rapid development of the mobile Internet. Automated test generation approaches, as a dominant direction of app quality insurance, follow specific models or strategies, targeting at optimizing the code coverage. Such approaches lead to a huge gap between testing execution and app business logic. Test scripts developed by human testers consider business logic by focusing on testing scenarios. Due to the GUI-intensive feature of mobile apps, human testers always understand app GUI to organize test scripts for scenarios. This inspires us to utilize domain knowledge from app GUI understanding for scenario-based test generation. In this paper, we propose a novel approach, ScenTest, for scenario-based mobile app testing with event knowledge graph (EKG) via GUI image understanding. ScenTest tries to start automated testing by imitating human practices and integrating domain knowledge into scenario-based mobile app testing, realizing fully automated testing on target testing scenarios for the first time. ScenTest extracts four kinds of entities and five kinds of corresponding relationships from crowdsourced test reports, where the test events and app GUI information are presented, and constructs the EKGs for specific scenarios. Then, ScenTest conducts test generation for specific scenarios on different apps with the guidance of EKG with the combination consideration of app current state and testing context. We conduct an evaluation on ScenTest on different aspects. The results show that the test generation of ScenTest on the basis of EKG is effective, and ScenTest can reveal 80+ distinct real-world bugs in specific scenarios compared with representative baselines.

Figures (5)

• Figure 1: Illustrative Example: Login Scenario Process in Two Different Crowdsourced Test Reports
• Figure 2: Illustrative Example: Event Knowledge Graph
• Figure 3: Illustrative Example: Test Generation
• Figure 4: ScenTest Framework
• Figure 5: Entities and relationships defined in the EKG for scenario-based mobile app testing.