GUIWatcher: Automatically Detecting GUI Lags by Analyzing Mobile Application Screencasts

TL;DR

This paper tackles the problem of detecting GUI lag in mobile apps from the end-user perspective, where traditional system metrics may miss perceptible unresponsiveness. It introduces GUIWatcher, a two-component framework that analyzes screencasts recorded during automated GUI testing, using SSIM to extract static frames and a fine-tuned YOLO detector to classify loading placeholders, to identify janky, long loading, and frozen frames. Se- vere lags are prioritized with thresholding, and the system outputs detailed bug reports; evaluated on real-world CI data achieving average precision 0.91 and recall 0.96. Deployed in production, GUIWatcher demonstrates practical value by helping developers focus on high-impact GUI performance issues and enabling continuous GUI performance monitoring across platforms.

Abstract

The Graphical User Interface (GUI) plays a central role in mobile applications, directly affecting usability and user satisfaction. Poor GUI performance, such as lag or unresponsiveness, can lead to negative user experience and decreased mobile application (app) ratings. In this paper, we present GUIWatcher, a framework designed to detect GUI lags by analyzing screencasts recorded during mobile app testing. GUIWatcher uses computer vision techniques to identify three types of lag-inducing frames (i.e., janky frames, long loading frames, and frozen frames) and prioritizes the most severe ones that significantly impact user experience. Our approach was evaluated using real-world mobile application tests, achieving high accuracy in detecting GUI lags in screencasts, with an average precision of 0.91 and recall of 0.96. The comprehensive bug reports generated from the lags detected by GUIWatcher help developers focus on the more critical issues and debug them efficiently. Additionally, GUIWatcher has been deployed in a real-world production environment, continuously monitoring app performance and successfully identifying critical GUI performance issues. By offering a practical solution for identifying and addressing GUI lags, GUIWatcher contributes to enhancing user satisfaction and the overall quality of mobile apps.

Figures (6)

• Figure 1: A sequence of frames in a recorded screencast, with each frame annotated by index (e.g, $f_{1}$) and PTS.
• Figure 2: Three types of GUI lags.
• Figure 3: The overall architecture of GUIWatcher.
• Figure 4: An illustration of GUI lag detection.
• Figure 5: Examples of frames containing placeholders.