Advancing Mobile UI Testing by Learning Screen Usage Semantics

TL;DR

The paper addresses the limited coverage and interpretability of Automated Input Generation (AIG) tools in mobile UI testing, especially on complex screens and high-level use cases. It proposes learning-based screen usage semantics via a multimodal UI classifier (Aurora) to understand and navigate screens, complemented by prompt-engineering to identify covered use cases. A preliminary result shows Aurora achieving a 19.6% coverage gain across 17 Android apps by recognizing screen categories and applying targeted navigation heuristics. The work aims to enhance test coverage, provide interpretable testing signals, and improve usability for elderly users, thereby bridging automated testing with usability improvements.

Abstract

The demand for quality in mobile applications has increased greatly given users' high reliance on them for daily tasks. Developers work tirelessly to ensure that their applications are both functional and user-friendly. In pursuit of this, Automated Input Generation (AIG) tools have emerged as a promising solution for testing mobile applications by simulating user interactions and exploring app functionalities. However, these tools face significant challenges in navigating complex Graphical User Interfaces (GUIs), and developers often have trouble understanding their output. More specifically, AIG tools face difficulties in navigating out of certain screens, such as login pages and advertisements, due to a lack of contextual understanding which leads to suboptimal testing coverage. Furthermore, while AIG tools can provide interaction traces consisting of action and screen details, there is limited understanding of its coverage of higher level functionalities, such as logging in, setting alarms, or saving notes. Understanding these covered use cases are essential to ensure comprehensive test coverage of app functionalities. Difficulty in testing mobile UIs can lead to the design of complex interfaces, which can adversely affect users of advanced age who often face usability barriers due to small buttons, cluttered layouts, and unintuitive navigation. There exists many studies that highlight these issues, but automated solutions for improving UI accessibility needs more attention. This research seeks to enhance automated UI testing techniques by learning the screen usage semantics of mobile apps and helping them navigate more efficiently, offer more insights about tested functionalities and also improve the usability of a mobile app's interface by identifying and mitigating UI design issues.